This is Gentoo's testing wiki. It is a non-operational environment and its textual content is outdated.
Please visit our production wiki at https://wiki.gentoo.org
LXD
Not to be confused with LXDE.
LXD is is a next generation system container manager. The core of LXD is a privileged daemon which exposes a REST API over a local unix socket as well as over the network (if enabled). LXD isn't a rewrite of LXC, in fact it's building on top of LXC to provide a new, better user experience. Under the hood, LXD uses LXC through liblxc and its Go binding to create and manage the containers. It's basically an alternative to LXC's tools and distribution template system with the added features that come from being controllable over the network.
For those new to container technology, it would be good to first read the "Virtualization Concepts" section of the LXC article.
Key features of LXD include:
- It prefers to launch unprivileged containers (secure by default).
- A command-line client (lxc) interacts with a daemon (lxd).
- Configuration is made intuitive and scriptable through cascading profiles.
- Configuration changes are performed with the lxc command (not config files).
- Multiple hosts can be federated together (with a certificate-based trust system).
- A federated setup means that containers can be launched on remote machines and live-migrated between hosts (using CRIU technology).
- It is usable as a standalone hypervisor or integrated with Openstack nova
Quick start (single host)
Prepare the system
Kernel configuration
It is a good idea to have most kernel flags required by app-emulation/lxc and sys-process/criu.
root #ebuild /usr/portage/app-emulation/lxc/lxc-1.1.2.ebuild setup... * Checking for suitable kernel configuration options... * CONFIG_NETLINK_DIAG: needed for lxc-checkpoint * CONFIG_PACKET_DIAG: needed for lxc-checkpoint * CONFIG_INET_UDP_DIAG: needed for lxc-checkpoint * CONFIG_INET_TCP_DIAG: needed for lxc-checkpoint * CONFIG_UNIX_DIAG: needed for lxc-checkpoint * CONFIG_CHECKPOINT_RESTORE: needed for lxc-checkpoint * Please check to make sure these options are set correctly. * Failure to do so may cause unexpected problems. |ebuild /usr/portage/sys-process/criu/criu-1.6.1.ebuild setup ...
Do you have plans for running systemd-based unprivileged containers? You will probably need to enable the "Gentoo Linux -> Support for init systems, system and service managers -> systemd" (CONFIG_GENTOO_LINUX_INIT_SYSTEMD)
Emerge
root #emerge --ask lxdAuthorize a non-privileged user
root #usermod --append --groups lxd erikThis will allow a non-root user to interact with the control socket which is owned by the lxd unix group. For the group change to take effect, users may need to log out and log back in again.
Configure subuid/subgid
Ensure that you have the files /etc/subuid and /etc/subgid present
root #touch /etc{/subuid,/subgid}
root #usermod --add-subuids 1000000-1065535 root
root #usermod --add-subgids 1000000-1065535 root
In this setup, the user 0-65535 on the container will actually be seen on the host system as user 1000000+uid and 1000000+gid. This protects the host system, since if any container managed to break out of its sandboxed namespace, it could interact with the host system only as a process with an unknown, very high UID/GID.
Usermod is part of sys-apps/shadow which is needed for the subuid/subgid functionality.
Start the daemon
For OpenRC users:
root #rc-service lxd startA systemd unit file has also been installed.
/etc/conf.d/lxd has a few available options related to debug output, but the defaults are adequate for this quick start.'
Setup storage and networking
root #lxd initWould you like to use LXD clustering? (yes/no) [default=no]: Do you want to configure a new storage pool (yes/no) [default=yes]? Name of the new storage pool [default=default]: Name of the storage backend to use (dir, lvm) [default=dir]: Would you like to connect to a MAAS server? (yes/no) [default=no]: Would you like to create a new network bridge (yes/no) [default=yes]? no Would you like to configure LXD to use an existing bridge or host interface? (yes/no) [default=no]: Would you like LXD to be available over the network? (yes/no) [default=no]: Would you like stale cached images to be updated automatically? (yes/no) [default=yes]: Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]:
Configure the bridge
If a new bridge was created by lxd init, start it now.
root #rc-service net.lxcbr0 startIf desired, the bridge can be configured to come up automatically in the runlevel.
root #rc-update add net.lxcbr0 defaultLaunch a container
Add an image repository at a remote called "images":
user $lxc remote add images https://images.linuxcontainers.orgThis is an untrusted remote, which can be a source of images that have been published with the --public flag. Trusted remotes are also possible, and are used as container hosts and also to serve private images. This specific remote is not special to LXD; organizations may host their own images.
user $lxc image list images:+--------------------------------+--------------+--------+-------------------------+---------+-------------------------------+ | ALIAS | FINGERPRINT | PUBLIC | DESCRIPTION | ARCH | UPLOAD DATE | +--------------------------------+--------------+--------+-------------------------+---------+-------------------------------+ | | 3ae185265c53 | yes | Centos 6 (amd64) | x86_64 | Aug 29, 2015 at 10:17pm (CDT) | | | 369ac13f390e | yes | Centos 6 (amd64) | x86_64 | Sep 3, 2015 at 12:17pm (CDT) | | centos/6/amd64 (1 more) | 8e54c679f1c2 | yes | Centos 6 (amd64) | x86_64 | Sep 3, 2015 at 10:17pm (CDT) | | | 755542362bbb | yes | Centos 6 (i386) | i686 | Aug 29, 2015 at 10:19pm (CDT) | | | b4d26dbc6567 | yes | Centos 6 (i386) | i686 | Sep 3, 2015 at 12:20pm (CDT) | | centos/6/i386 (1 more) | 21eeba48a2d4 | yes | Centos 6 (i386) | i686 | Sep 3, 2015 at 10:19pm (CDT) | | | 9fe7ffdbc0ae | yes | Centos 7 (amd64) | x86_64 | Aug 29, 2015 at 10:22pm (CDT) | | | d750b910e62d | yes | Centos 7 (amd64) | x86_64 | Sep 3, 2015 at 12:23pm (CDT) | | centos/7/amd64 (1 more) | 06c4e5c21707 | yes | Centos 7 (amd64) | x86_64 | Sep 3, 2015 at 10:22pm (CDT) | | | ee229d68be51 | yes | Debian jessie (amd64) | x86_64 | Aug 29, 2015 at 6:29pm (CDT) | | | 69e457e1f4ab | yes | Debian jessie (amd64) | x86_64 | Sep 2, 2015 at 6:34pm (CDT) | | debian/jessie/amd64 (1 more) | 2ddd14ff9422 | yes | Debian jessie (amd64) | x86_64 | Sep 3, 2015 at 6:30pm (CDT) | | | 9fac01d1e773 | yes | Debian jessie (armel) | armv7l | Aug 31, 2015 at 7:24pm (CDT) | | | 67f4fedafd2f | yes | Debian jessie (armel) | armv7l | Sep 1, 2015 at 7:24pm (CDT) | ...
There are Gentoo images in the list, although they are not maintained by the Gentoo project. LXC users may recognize these images as the same ones available using the "download" template.
user $lxc launch images:centos/6/amd64 mycentos6Creating mycentos6 done. Starting mycentos6 done.
user $lxc list+-----------+---------+----------------+------+-----------+-----------+ | NAME | STATE | IPV4 | IPV6 | EPHEMERAL | SNAPSHOTS | +-----------+---------+----------------+------+-----------+-----------+ | mycentos6 | RUNNING | 192.168.43.240 | | NO | 0 | +-----------+---------+----------------+------+-----------+-----------+
A shell can be run in the container's context.
user $lxc exec mycentos6 /bin/bash[root@mycentos6 ~]# ps faux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 428 0.0 0.0 11500 2648 ? Ss 16:13 0:00 /bin/bash root 440 0.0 0.0 13380 1888 ? R+ 16:13 0:00 \_ ps faux root 1 0.0 0.0 19292 2432 ? Ss 16:03 0:00 /sbin/init root 188 0.0 0.0 4124 1316 console Ss+ 16:03 0:00 /sbin/mingetty --nohangup console root 228 0.0 0.0 9180 1392 ? Ss 16:03 0:00 /sbin/dhclient -H mycentos6 -1 -q -lf /var/lib/dhclient/dhclient-eth0.leases -pf /var/run/dhc root 278 0.0 0.0 171372 2544 ? Sl 16:03 0:00 /sbin/rsyslogd -i /var/run/syslogd.pid -c 5 root 439 0.0 0.0 4120 1472 ? Ss 16:13 0:00 /sbin/mingetty --nohangup /dev/tty[1-6]
While the container sees its processes as running as the root user, running ps on the host shows the processes running as UID 1000000. This is the advantage of unprivileged containers: root is only root in the container, and is nobody special in the host. It is possible to manipulate the subuid/subgid maps to allow containers access to host resources (for example, write to the host's X socket) but this must be explicitly allowed.
Configuration
Configuration of containers is managed with the lxc config and lxc profile commands. The two commands provide largely the same capabilities, but lxc config acts on single containers while lxc profile configures a profile which can be used across multiple containers.
Importantly, containers can be launched with multiple profiles. The profiles have a cascading effect so that a profile specified later in the list can add, remove, and override configuration values that were specified in a earlier profile. This can allow for complex setups where groups of containers can be specified which share some properties but not others.
user $lxc profile listdefault migratable
user $lxc profile show defaultname: default
config: {}
devices:
eth0:
nictype: bridged
parent: lxcbr0
type: nicThe default profile is applied if no profile is specified on the command line. In the quick start, the lxc launch omitted the profile, and so was equivalent to:
user $lxc launch images:centos/6/amd64 mycentos6 --profile defaultNotice that that the default profile only specifies that a container should have a single NIC which is bridged onto an existing bridge lxcbr0. So, having a bridge with that name is not a hard requirement, it just happens to be named in the default profile.
Available configuration includes limits on memory and CPU cores, and also devices including NICs, bind mounts, and block/character device nodes.
Configuration is documented in /usr/share/doc/lxd-0.16/specs/configuration.md (substitute the correct version of course).
All documented configuration options are not yet implemented, and only by inspecting source code can a user know which devices might be expected to work. For example, at the time of this writing:
- The only legal nictype for an interface is "bridged"
- The "disk" device type can only perform bind mounts, and only on a directory.
- Setting limits.memory seems not to work, although limits.cpus does
Example
Here a container is launched with the default profile and also a "cpusandbox" profile which imposes a limit of one CPU core. A directory on the host is also bind-mounted into the container using the container-specific lxc config command.
First, prepare a reusable profile.
user $lxc profile create cpusandboxProfile cpusandbox created
user $lxc profile set cpusandbox limits.cpus 1lxc config requires a container name, so a container is initialized.
user $lxc init 8e54c679f1c2 confexample --profile default --profile cpusandboxCreating confexample done.
The 8e54c679f1c2 argument represents an image fingerprint obtained with
lxc image list. It's possible to give friendly aliases to images.In the quick start above the
lxc launch was invoked, which is a shorthand to lxc init followed by lxc start. The latter is used in this example so that lxc config can be invoked before the container is first started.In this example a host directory /tmp/shared is bind-mounted into the container at /tmp. While this could be configured in a profile, instead it will be considered an exclusive feature for that container.
user $mkdir /tmp/shareduser $touch /tmp/shared/hello.txtSet the directory to be owned by the container's root user (really UID 1000000 in the host).
user $sudo chown -R 1000000:1000000 /tmp/shareduser $lxc config device add confexample sharedtmp disk path=/tmp source=/tmp/sharedDevice sharedtmp added to confexample
user $lxc start confexampleuser $lxc exec confexample ls /tmphello.txt
Multi-host setup
Two hosts on a network, alpha and beta, are running the lxd daemon. The goal is to run commands on alpha which can manipulate containers and images on either alpha or beta.
Resolvable names are prepared for readability but it also works to use IP addresses.
Setup
Configure the daemon on the remote to listen over HTTPS instead of the default local Unix socket.
beta $lxc config set core.https_address beta:8443Restart the daemon after this step, and be sure that the firewall will accept incoming connections as specified.
On beta configure a trust password, which is only used until certificates are exchanged.
beta $lxc config set core.trust_password Sieth9gei7ahm2raAdd the beta remote to alpha.
alpha $lxc remote add beta https://beta:8443Certificate fingerprint: 7a 76 5b c6 c6 eb 4e db 20 7f 31 bb 1d 11 ca 2d c5 d8 7d cf 41 c0 a0 1f aa 8b c3 f0 18 79 d3 a3 ok (y/n)? y Admin password for beta: Client certificate stored at server: beta
Result
It is now possible to perform actions on beta from alpha using the remote: syntax
alpha $lxc list beta:alpha $lxc remote list beta:alpha $lxc launch beta:centos6 beta:container0alpha $lxc info beta:container0To copy containers or images, the source ("from") host must have its daemon listening via HTTPS not Unix socket.
alpha $lxc image copy beta:gentlean local:alpha $lxc image copy centos6 beta:error: The source remote isn't available over the network
As long as commands were run from alpha, it was never necessary to add alpha as a remote to either host. It was also not necessary to change alpha's core.https_address config setting to use HTTPS instead of Unix socket unless it is the source of a container or image copy.
Configuration settings and profiles are only set on the local host. They can be copied to remotes but this is manual and error-prone unless configuration managment tools are used to propagate these values. Consider requiring all commands to be run from a single host with its local config database for ease of administration.
Advanced features
Live migration
TODO
Automatic BTRFS integration
When LXD detects that /var/lib/lxd is on a Btrfs filesystem, it uses Btrfs' snapshot capabilities to ensure that images, containers and snapshots share blocks as much as possible. No user action is required to enable this behavior.
When the container was launched in the Quick Start section above LXD created subvolumes for the image and container. The container filesystem is a copy-on-write snapshot of the image.
root #btrfs subvolume list /ID 330 gen 4518 top level 5 path var/lib/lxd/images/8e54c679f1c293f909c66097d97de23c66a399d2dc396ade92b3b6aae1c7 ID 331 gen 4595 top level 5 path var/lib/lxd/containers/mycentos6
root #btrfs subvolume show /var/lib/lxd/images/8e54c679f1c293f909c66097d97de23c66a399d2dc396ade92b3b6a
/var/lib/lxd/images/8e54c679f1c293f909c66097d97de23c66a399d2dc396ade92b3b6aae1c732fe.btrfs
Name: 8e54c679f1c293f909c66097d97de23c66a399d2dc396ade92b3b6aae1c732fe.btrfs
UUID: 5530510e-2007-f146-9e0b-8c05480d63de
Parent UUID: -
Received UUID: -
Creation time: 2015-09-04 15:03:32 -0500
Subvolume ID: 330
Generation: 4518
Gen at creation: 4517
Parent ID: 5
Top level ID: 5
Flags: -
Snapshot(s):
var/lib/lxd/containers/mycentos6
Making a snapshot of the running container filesystem creates another copy-on-write snapshot.
user $lxc snapshot mycentos6 firstsnaproot #btrfs subvolume list /ID 330 gen 4518 top level 5 path var/lib/lxd/images/8e54c679f1c293f909c66097d97de23c66a399d2dc396ade92b3b6aae1c7 ID 331 gen 4595 top level 5 path var/lib/lxd/containers/mycentos6 ID 332 gen 4584 top level 5 path var/lib/lxd/snapshots/mycentos6/firstsnap
root #btrfs subvolume show /var/lib/lxd/containers/mycentos6/var/lib/lxd/containers/mycentos6
Name: mycentos6
UUID: fe6bfd65-d911-e449-a1df-be42d2997f4a
Parent UUID: 5530510e-2007-f146-9e0b-8c05480d63de
Received UUID: -
Creation time: 2015-09-04 15:03:39 -0500
Subvolume ID: 331
Generation: 4595
Gen at creation: 4518
Parent ID: 5
Top level ID: 5
Flags: -
Snapshot(s):
var/lib/lxd/snapshots/mycentos6/firstsnap
/dev/lxd/sock
A socket is bind-mounted into the container at /dev/lxd/sock. It serves no critical purpose, but is available to users as a means to query configuration information about the container.
Troubleshooting
Cgroups inside Containers with OpenRC
As of 26th November 2017 there is an open bug with OpenRC (at least inside the container). Informations on the state of fixing and a workaround (or possible fix) can be found here: https://discuss.linuxcontainers.org/t/running-lxd-an-openrc-container-on-a-openrc-system-trouble-with-cgroups/843 (includes patching and init file). Setting rc_sys="" inside the container works, too, but might break other things.
Proceed at own risk and create backups ;-)
Containers not starting under OpenRC
Recent OpenRC release brought "unified" cgroups and LXD doesn't seem to like that (it runs well under systemd which also has unified cgroups though...). The workaround for that is to disable unified cgroups. You do that by editing /etc/rc.conf and setting rc_cgroup_mode="legacy".
Running systemd based containers on OpenRC hosts
To support systemd for e.g. ubuntu container there are two distinct changes necessary: a) Mount the host's cgroups automagically into the container
user $lxc config set <container> raw.lxc 'lxc.mount.auto = cgroup'b) Create the system cgroup directory on the host and mount the cgroup there:
root #mkdir -p /sys/fs/cgroup/systemd
root #mount -t cgroup -o none,name=systemd systemd /sys/fs/cgroup/systemd
For more details take a look a the upstream issue on github.com
See also
- LXC — a virtualization system, making use of the "cgroups" feature.