Escape Vagrant using Vagrant

Cole Brumley
Senior Software Engineer

We’ve spent a lot of time at Maryville thinking about how to marry the worlds of containers and virtual machines. In theory, all the components exist right now to create a container that’s pretty darn close to a VM: We can run full init systems (including systemd) in unprivileged containers, have several software-defined overlay networks to choose from, cgroups to allocate physical resources, and snapshot/restore built into the kernel.

I frequently need to work with lower level system bits that don’t translate very well into a docker workflow, but of course I still want the speed benefits of containers for local testing. VMs are okay I guess, but working with VirtualBox directly leaves something to be desired and cloud providers are too expensive and remote. Vagrant is close to what I’m after, but still uses large VM images that take a long time to pull and start.

We took a whack at the idea of a chubby-container last year using Docker, CRIU, and Weave with pretty good success. Unfortunately, there seems to be a lack of enthusiasm for these features from the Docker maintainers, so the idea was shelved after the proof-of-concept. Luckily we stumbled onto a ready alternative shortly thereafter.

Despite thin documentation and a clunky CLI interface, LXD is my weapon of choice for quickly spinning up and tearing down test systems. It’s perfect for the job since its stated goal is to be a container “hypervisor” rather than an application platform. The LXC/D teams promote the idea of a distinction between system and application containers; LXD providing a daemon for the former and docker for the latter. I can dig it, my use case seems like a pretty clear example of that distinction in the wild.

I use LXD like docker-machine, meaning a single, fairly large Vagrant VM that’s controlled by a local client. Again, not really escaping the confines of a VM, but close enough to give that down-home localhost feel. My home directory which contains my source is mounted inside the VM, just like docker-machine. We’ll need VirtualBox, Vagrant, and Go installed before starting.

Make sure $GOPATH/bin is in your $PATH

# Clone and build the client binary
go get github.com/lxc/lxd
cd $GOPATH/src/github.com/lxc/lxd
make client

Now we can start up the Vagrant machine and configure LXD:

# The LXD repo provides a Vagrant box.
# Time to grab a coffee.
vagrant up

# Configure LXD
vagrant ssh -- <<EOF
sudo /home/vagrant/go/bin/lxd init --auto \
    --network-address 0.0.0.0 \
    --network-port 443 \
    --trust-password lxd && \
sudo service lxd restart
EOF

Now we need to point our freshly compiled client at the VM:

lxc remote add vagrant 127.0.0.1:8443 --accept-certificate --password lxd
lxc remote set-default vagrant

Ta-da! We are now running an LXD implementation of docker-machine. By way of an example, let’s create a container to run docker in, with an associated DHCP network:

lxc network create docker0 ipv4.address=auto ipv4.dhcp=true ipv4.routing=true
lxc launch ubuntu:xenial docker -p default -p docker -n docker0
lxc exec docker -- apt-get -y update
lxc exec docker -- apt-get -y install docker.io

Now we can either exec into that container to run our docker commands, or we can add a port map to the Vagrantfile and use a local docker binary like a traditional docker-machine configuration would.

One of my favorite features is the ability to define cloud-init user, vendor, or meta data as part of the container launch command. Add -c user.user-data= to your lxc launch command line to instantly test your cloud config settings. Pretty sweet, but the feature depends on the image supporting it which isn’t guaranteed, although most of the images I’ve tried do.

Since I use Visual Studio Code for most of my development, adding a slick VSCode task results in a mostly tolerable automation development workflow. Huzzah! Here’s a contrived VSCode tasks file you could use to trigger things in containers:

{
"version": "0.1.0",
"command": "bash",
"args": ["-c"],
"isShellCommand": true,
"showOutput": "always",
"suppressTaskName": true,
"tasks": [
    {
        "taskName": "sync",
        "args": ["lxc file push -r -p \"$(pwd)/\" $CONTAINER/opt/work/"]
    },
    {
        "taskName": "test",
        "args": ["lxc exec $CONTAINER -- make -C /opt/work test"]
    },
    {
        "taskName": "buildimg",
        "args": ["lxc exec $CONTAINER -- docker build -t test -f /opt/work/Dockerfile /opt/work/"]
    }
]
}

Of course once things get complicated (and they always do) you’ll want to delegate these tasks out to makefiles or scripts