# LXC Management with Salt

Note

This walkthrough assumes basic knowledge of Salt. To get up to speed, check out the Salt Walkthrough.

## Dependencies

Manipulation of LXC containers in Salt requires the minion to have an LXC version of at least 1.0 (an alpha or beta release of LXC 1.0 is acceptable). The following distributions are known to have new enough versions of LXC packaged:

- RHEL/CentOS 6 and later (via EPEL)

- Fedora (All non-EOL releases)

- Debian 8.0 (Jessie)

- Ubuntu 14.04 LTS and later (LXC templates are packaged separately as lxc-templates, it is recommended to also install this package)

- openSUSE 13.2 and later

## Profiles

Profiles allow for a sort of shorthand for commonly-used configurations to be defined in the minion config file, grains, pillar, or the master config file. The profile is retrieved by Salt using the `config.get` function, which looks in those locations, in that order. This allows for profiles to be defined centrally in the master config file, with several options for overriding them (if necessary) on groups of minions or individual minions.

There are two types of profiles:

> - One for defining the parameters used in container creation/clone.

> - One for defining the container's network interface(s) settings.

### Container Profiles

LXC container profiles are defined underneath the `lxc.container_profile` config option:

    lxc.container_profile:
      centos:
        template: centos
        backing: lvm
        vgname: vg1
        lvname: lxclv
        size: 10G
      centos_big:
        template: centos
        backing: lvm
        vgname: vg1
        lvname: lxclv
        size: 20G

Profiles are retrieved using the `config.get` function, with the recurse merge strategy. This means that a profile can be defined at a lower level (for example, the master config file) and then parts of it can be overridden at a higher level (for example, in pillar data). Consider the following container profile data:

In the Master config file:

    lxc.container_profile:
      centos:
        template: centos
        backing: lvm
        vgname: vg1
        lvname: lxclv
        size: 10G

In the Pillar data

    lxc.container_profile:
      centos:
        size: 20G

Any minion with the above Pillar data would have the size parameter in the centos profile overridden to 20G, while those minions without the above Pillar data would have the 10G size value. This is another way of achieving the same result as the centos_big profile above, without having to define another whole profile that differs in just one value.

Note

In the 2014.7.x release cycle and earlier, container profiles are defined under `lxc.profile`. This parameter will still work in version 2015.5.0, but is deprecated and will be removed in a future release. Please note however that the profile merging feature described above will only work with profiles defined under `lxc.container_profile`, and only in versions 2015.5.0 and later.

Additionally, in version 2015.5.0 container profiles have been expanded to support passing template-specific CLI options to `lxc.create`. Below is a table describing the parameters which can be configured in container profiles:

Parameter | 2015.5.0 and Newer | 2014.7.x and Earlier
--- | --- | ---
template1 | Yes | Yes
options1 | Yes | No
image1 | Yes | Yes
backing | Yes | Yes
snapshot2 | Yes | Yes
lvname1 | Yes | Yes
fstype1 | Yes | Yes
size | Yes | Yes

1. Parameter is only supported for container creation, and will be ignored if the profile is used when cloning a container.

2. Parameter is only supported for container cloning, and will be ignored if the profile is used when not cloning a container.

### Network Profiles

LXC network profiles are defined defined underneath the `lxc.network_profile` config option. By default, the module uses a DHCP based configuration and try to guess a bridge to get connectivity.

Warning

on pre 2015.5.2, you need to specify explicitly the network bridge

    lxc.network_profile:
      centos:
        eth0:
          link: br0
          type: veth
          flags: up
      ubuntu:
        eth0:
          link: lxcbr0
          type: veth
          flags: up

As with container profiles, network profiles are retrieved using the `config.get` function, with the recurse merge strategy. Consider the following network profile data:

In the Master config file:

    lxc.network_profile:
      centos:
        eth0:
          link: br0
          type: veth
          flags: up

In the Pillar data

    lxc.network_profile:
      centos:
        eth0:
          link: lxcbr0

Any minion with the above Pillar data would use the lxcbr0 interface as the bridge interface for any container configured using the centos network profile, while those minions without the above Pillar data would use the br0 interface for the same.

Note

In the 2014.7.x release cycle and earlier, network profiles are defined under `lxc.nic`. This parameter will still work in version 2015.5.0, but is deprecated and will be removed in a future release. Please note however that the profile merging feature described above will only work with profiles defined under `lxc.network_profile`, and only in versions 2015.5.0 and later.

The following are parameters which can be configured in network profiles. These will directly correspond to a parameter in an LXC configuration file (see `man 5 lxc.container.conf`).

- type - Corresponds to lxc.network.type

- link - Corresponds to lxc.network.link

- flags - Corresponds to lxc.network.flags

Interface-specific options (MAC address, IPv4/IPv6, etc.) must be passed on a container-by-container basis, for instance using the `nic_opts` argument to `lxc.create`:

    salt myminion lxc.create container1 profile=centos network_profile=centos nic_opts='{eth0: {ipv4: 10.0.0.20/24, gateway: 10.0.0.1}}'

Warning

The `ipv4`, `ipv6`, `gateway`, and `link` (bridge) settings in network profiles / nic_opts will only work if the container doesn't redefine the network configuration (for example in `/etc/sysconfig/network-scripts/ifcfg-<interface_name>` on RHEL/CentOS, or `/etc/network/interfaces` on Debian/Ubuntu/etc.). Use these with caution. The container images installed using the `download` template, for instance, typically are configured for eth0 to use DHCP, which will conflict with static IP addresses set at the container level.

Note

For LXC < 1.0.7 and DHCP support, set `ipv4.gateway: 'auto'` is your network profile, ie.:

    lxc.network_profile.nic:
      debian:
        eth0:
          link: lxcbr0
          ipv4.gateway: 'auto'

### Old lxc support (<1.0.7)

With saltstack 2015.5.2 and above, normally the setting is autoselected, but before, you'll need to teach your network profile to set lxc.network.ipv4.gateway to auto when using a classic ipv4 configuration.

Thus you'll need

    lxc.network_profile.foo:
      etho:
        link: lxcbr0
        ipv4.gateway: auto

### Tricky network setups Examples

This example covers how to make a container with both an internal ip and a public routable ip, wired on two veth pairs.

The another interface which receives directly a public routable ip can't be on the first interface that we reserve for private inter LXC networking.

    lxc.network_profile.foo:
      eth0: {gateway: null, bridge: lxcbr0}
      eth1:
        # replace that by your main interface
        'link': 'br0'
        'mac': '00:16:5b:01:24:e1'
        'gateway': '2.20.9.14'
        'ipv4': '2.20.9.1'

## Creating a Container on the CLI

### From a Template

LXC is commonly distributed with several template scripts in /usr/share/lxc/templates. Some distros may package these separately in an lxc-templates package, so make sure to check if this is the case.

There are LXC template scripts for several different operating systems, but some of them are designed to use tools specific to a given distribution. For instance, the `ubuntu` template uses deb_bootstrap, the `centos` template uses yum, etc., making these templates impractical when a container from a different OS is desired.

The `lxc.create` function is used to create containers using a template script. To create a CentOS container named `container1` on a CentOS minion named `mycentosminion`, using the `centos` LXC template, one can simply run the following command:

    salt mycentosminion lxc.create container1 template=centos

For these instances, there is a `download` template which retrieves minimal container images for several different operating systems. To use this template, it is necessary to provide an `options` parameter when creating the container, with three values:

1. dist - the Linux distribution (i.e. `ubuntu` or `centos`)

2. release - the release name/version (i.e. `trusty` or `6`)

3. arch - CPU architecture (i.e. `amd64` or `i386`)

The `lxc.images` function (new in version 2015.5.0) can be used to list the available images. Alternatively, the releases can be viewed on http://images.linuxcontainers.org/images/. The images are organized in such a way that the dist, release, and arch can be determined using the following URL format: `http://images.linuxcontainers.org/images/dist/release/arch`. For example, `http://images.linuxcontainers.org/images/centos/6/amd64` would correspond to a dist of `centos`, a release of `6`, and an arch of `amd64`.

Therefore, to use the `download` template to create a new 64-bit CentOS 6 container, the following command can be used:

    salt myminion lxc.create container1 template=download options='{dist: centos, release: 6, arch: amd64}'

Note

These command-line options can be placed into a container profile, like so:

    lxc.container_profile.cent6:
      template: download
      options:
        dist: centos
        release: 6
        arch: amd64

The `options` parameter is not supported in profiles for the 2014.7.x release cycle and earlier, so it would still need to be provided on the command-line.

### Cloning an Existing Container

To clone a container, use the `lxc.clone` function:

    salt myminion lxc.clone container2 orig=container1

### Using a Container Image

While cloning is a good way to create new containers from a common base container, the source container that is being cloned needs to already exist on the minion. This makes deploying a common container across minions difficult. For this reason, Salt's `lxc.create` is capable of installing a container from a tar archive of another container's rootfs. To create an image of a container named `cent6`, run the following command as root:

    tar czf cent6.tar.gz -C /var/lib/lxc/cent6 rootfs

Note

Before doing this, it is recommended that the container is stopped.

The resulting tarball can then be placed alongside the files in the salt fileserver and referenced using a `salt://` URL. To create a container using an image, use the `image` parameter with `lxc.create`:

    salt myminion lxc.create new-cent6 image=salt://path/to/cent6.tar.gz

Note

Making images of containers with LVM backing

For containers with LVM backing, the rootfs is not mounted, so it is necessary to mount it first before creating the tar archive. When a container is created using LVM backing, an empty `rootfs` dir is handily created within `/var/lib/lxc/container_name`, so this can be used as the mountpoint. The location of the logical volume for the container will be `/dev/vgname/lvname`, where `vgname` is the name of the volume group, and `lvname` is the name of the logical volume. Therefore, assuming a volume group of `vg1`, a logical volume of `lxc-cent6`, and a container name of `cent6`, the following commands can be used to create a tar archive of the rootfs:

    mount /dev/vg1/lxc-cent6 /var/lib/lxc/cent6/rootfs
    tar czf cent6.tar.gz -C /var/lib/lxc/cent6 rootfs
    umount /var/lib/lxc/cent6/rootfs

Warning

One caveat of using this method of container creation is that `/etc/hosts` is left unmodified. This could cause confusion for some distros if salt-minion is later installed on the container, as the functions that determine the hostname take `/etc/hosts` into account.

Additionally, when creating an rootfs image, be sure to remove `/etc/salt/minion_id` and make sure that `id` is not defined in `/etc/salt/minion`, as this will cause similar issues.

## Initializing a New Container as a Salt Minion

The above examples illustrate a few ways to create containers on the CLI, but often it is desirable to also have the new container run as a Minion. To do this, the `lxc.init` function can be used. This function will do the following:

1. Create a new container

2. Optionally set password and/or DNS

3. Bootstrap the minion (using either salt-bootstrap or a custom command)

By default, the new container will be pointed at the same Salt Master as the host machine on which the container was created. It will then request to authenticate with the Master like any other bootstrapped Minion, at which point it can be accepted.

    salt myminion lxc.init test1 profile=centos
    salt-key -a test1

For even greater convenience, the `LXC runner` contains a runner function of the same name (`lxc.init`), which creates a keypair, seeds the new minion with it, and pre-accepts the key, allowing for the new Minion to be created and authorized in a single step:

    salt-run lxc.init test1 host=myminion profile=centos

## Running Commands Within a Container

For containers which are not running their own Minion, commands can be run within the container in a manner similar to using (`cmd.run <salt.modules.cmdmod.run`). The means of doing this have been changed significantly in version 2015.5.0 (though the deprecated behavior will still be supported for a few releases). Both the old and new usage are documented below.

### 2015.5.0 and Newer

New functions have been added to mimic the behavior of the functions in the `cmd` module. Below is a table with the `cmd` functions and their `lxc` module equivalents:

Description | `cmd` module | `lxc` module
--- | --- | ---
Run a command and get all output | `cmd.run` | `lxc.run`
Run a command and get just stdout | `cmd.run_stdout` | `lxc.run_stdout`
Run a command and get just stderr | `cmd.run_stderr` | `lxc.run_stderr`
Run a command and get just the retcode | `cmd.retcode` | `lxc.retcode`
Run a command and get all information | `cmd.run_all` | `lxc.run_all`

### 2014.7.x and Earlier

Earlier Salt releases use a single function (`lxc.run_cmd`) to run commands within containers. Whether stdout, stderr, etc. are returned depends on how the function is invoked.

To run a command and return the stdout:

    salt myminion lxc.run_cmd web1 'tail /var/log/messages'

To run a command and return the stderr:

    salt myminion lxc.run_cmd web1 'tail /var/log/messages' stdout=False stderr=True

To run a command and return the retcode:

    salt myminion lxc.run_cmd web1 'tail /var/log/messages' stdout=False stderr=False

To run a command and return all information:

    salt myminion lxc.run_cmd web1 'tail /var/log/messages' stdout=True stderr=True

## Container Management Using salt-cloud

Salt cloud uses under the hood the salt runner and module to manage containers, Please look at this chapter

## Container Management Using States

Several states are being renamed or otherwise modified in version 2015.5.0. The information in this tutorial refers to the new states. For 2014.7.x and earlier, please refer to the `documentation for the LXC states`.

### Ensuring a Container Is Present

To ensure the existence of a named container, use the `lxc.present` state. Here are some examples:

    # Using a template
    web1:
      lxc.present:
        - template: download
        - options:
            dist: centos
            release: 6
            arch: amd64

    # Cloning
    web2:
      lxc.present:
        - clone_from: web-base

    # Using a rootfs image
    web3:
      lxc.present:
        - image: salt://path/to/cent6.tar.gz

    # Using profiles
    web4:
      lxc.present:
        - profile: centos_web
        - network_profile: centos

Warning

The `lxc.present` state will not modify an existing container (in other words, it will not re-create the container). If an `lxc.present` state is run on an existing container, there will be no change and the state will return a `True` result.

The `lxc.present` state also includes an optional `running` parameter which can be used to ensure that a container is running/stopped. Note that there are standalone `lxc.running` and `lxc.stopped` states which can be used for this purpose.

### Ensuring a Container Does Not Exist

To ensure that a named container is not present, use the `lxc.absent` state. For example:

    web1:
      lxc.absent

### Ensuring a Container is Running/Stopped/Frozen

Containers can be in one of three states:

- running - Container is running and active

- frozen - Container is running, but all process are blocked and the container is essentially non-active until the container is "unfrozen"

- stopped - Container is not running

Salt has three states (`lxc.running`, `lxc.frozen`, and `lxc.stopped`) which can be used to ensure a container is in one of these states:

    web1:
      lxc.running

    # Restart the container if it was already running
    web2:
      lxc.running:
        - restart: True

    web3:
      lxc.stopped

    # Explicitly kill all tasks in container instead of gracefully stopping
    web4:
      lxc.stopped:
        - kill: True

    web5:
      lxc.frozen

    # If container is stopped, do not start it (in which case the state will fail)
    web6:
      lxc.frozen:
        - start: False