Three Test Jails on a 512 MB Free-Tier VPS

Table of Contents

Someone opened an issue against ansible-jailexec, the Ansible connection plugin for FreeBSD jails I wrote about at the end of last year, with an admirably small reproducer: a copy task works, the same copy task with a loop breaks the play. First item fine, everything after it dead.

I found the cause by reading ansible-core source. Ansible rebuilds a connection plugin’s option dictionary before every task — and inside a loop, before every single item. My plugin redirects SSH at the jail host by overriding the host option exactly once, in _connect(). The option reload wiped that redirect, _connect() short-circuits on an established connection so nothing put it back, and from item two onward Ansible dutifully tried to SSH to the jail name itself.

The unit tests confirmed the fix. That is not the same as knowing it works. The whole point of a connection plugin is the part that talks to a real machine, and my mocks are exactly as correct as my understanding of ansible-core, which had just been demonstrated to be incomplete.

So I needed test jails. Real ones.

The Box

What I had lying around was a Vultr free-tier instance that has been running nothing but sshd and a WireGuard interface for 51 days:

CPU:    1 × Intel Xeon (Skylake, IBRS)
RAM:    496 MB physical, 512 MB swap
Disk:   9.2 GB UFS, 1.3 GB used
OS:     FreeBSD 15.1-RELEASE, installed from pkgbase

Half a gigabyte of RAM and a single core. Twelve years ago this was a respectable server; today it is below the minimum system requirements of several “lightweight” container platforms. It is, however, entirely sufficient for running three jails that do nothing but wait to have commands executed in them.

The install was pkgbase, which turned out to be the detail that made the whole exercise pleasant.

Why pkgbase Makes Test Jails Cheap

The traditional way to populate a jail is to fetch base.txz and extract it. That gives you the full base system — every binary, every library, /usr/share/doc, the lot — for roughly 1 GB per jail before you install anything of your own.

With pkgbase, the base system is a set of ordinary packages, and you can install a subset of them into a directory:

# A minimal pkgbase install ships neither of these.
pkg install -y FreeBSD-jail doas

mkdir -p /jails/smokeroot
pkg --rootdir /jails/smokeroot install -r FreeBSD-base -y FreeBSD-runtime

FreeBSD-runtime pulls in twelve packages and produces this:

33M     /jails/smokeroot

Thirty-three megabytes for a jail with a working /bin/sh, cat, ls, ps, hostname, pw and the rest of the essential userland. No documentation, no debugging symbols, no sendmail, no games. For a jail whose entire purpose is to be the far end of a jexec call, that is precisely the right amount of operating system.

My plugin needs a Python interpreter in the jail as well, because that is where Ansible modules actually run:

pkg --rootdir /jails/smokeroot install -y python3

That is the expensive part, and it is worth seeing the two numbers next to each other:

The base system is 8 % of the jail. The Python stack is the other 92 %. I find that funny every time.

Sharp edge one: the fingerprints live inside the rootdir

The first pkg --rootdir run failed:

pkg: Error opening the trusted directory /usr/share/keys/pkgbase-15/trusted
pkg: Error loading trusted certificates
pkg: Repository FreeBSD-base cannot be opened. 'pkg update' required

The path in that message is misleading. The keys exist at /usr/share/keys/pkgbase-15/trusted on the host — that is how the host installed itself. But pkg resolves the fingerprint path relative to the rootdir, so it was looking inside an empty jail directory and reporting the path without the prefix. Copy the keys in first:

mkdir -p /jails/smokeroot/usr/share/keys
cp -R /usr/share/keys/ /jails/smokeroot/usr/share/keys/

Sharp edge two: nobody runs ldconfig for you

With the jails built and started, Ansible got as far as executing a module and then:

ld-elf.so.1: Shared object "libpython3.12.so.1.0" not found, required by "python3"

pkg --rootdir installs files. It does not run the post-install steps that would normally happen inside the system being modified, and ldconfig is one of them. The hints file in each jail was empty, so the dynamic linker had never heard of /usr/local/lib:

jexec smokeroot /sbin/ldconfig -m /usr/local/lib

Both of these are obvious in hindsight and neither is documented anywhere near where you need it, which is my excuse for writing them down.

The Smallest jail.conf That Does Anything

Here is the entire jail configuration for all three jails:

exec.clean;
mount.devfs;
persist;
path = "/jails/$name";
host.hostname = "$name";

smokeroot { }
smokeuser { }
smokealias { }

No exec.start. No /etc/rc. No IP address, no vnet, no ip4.addr, not even a loopback alias. These jails run zero processes when idle. persist is what keeps them alive without a single running program in them, and that is all jexec requires: a jail that exists.

mount.devfs is the one line you cannot drop for these targets. Python wants /dev/urandom at startup, and without devfs the interpreter fails in a way that looks nothing like a missing device node.

Networking is absent by design. The packages are installed from the host with pkg --rootdir, so nothing inside the jails ever needs to reach the internet. A test target that cannot phone home is a feature.

Two more lines in /etc/rc.conf make them survive a reboot:

jail_enable="YES"
jail_list="smokeroot smokeuser smokealias"

Incidentally, this box has no sysrc — a minimal pkgbase install does not include it. Appending to rc.conf with grep -q || echo >> works fine and is a useful reminder of how thin “minimal” really is.

Three Jails, Three Code Paths

Rather than one generic jail, I built three, each aimed at a specific thing the plugin can get wrong:

The third one deserves a note. Using a TEST-NET address as a tripwire is cheap and effective: without a deliberately configured local route, a connection to 192.0.2.99 cannot succeed. Any code path that reaches for ansible_host instead of ansible_jail_host therefore produces an immediate, unambiguous failure rather than a subtle wrong answer.

Building the second and third jail took no downloads at all — cp -a from the first is faster than re-fetching packages, and on this box, considerably faster.

The gotcha that will bite anyone running Ansible against a jail

The first full run against smokeuser failed like this:

Failed to create temporary directory ...
Failed command was: ( umask 77 && mkdir -p "` echo ~chofstede/.ansible/tmp `" ...

Ansible builds its temporary directory path from the home directory of the SSH user. That user exists on the jail host. It does not exist inside the jail, and a non-root ansible_jail_user cannot create directories in the jail’s /. The fix is one inventory variable:

ansible_remote_tmp=/tmp/.ansible-tmp

This has nothing to do with my plugin specifically — it applies to any setup where the connection lands somewhere the SSH user does not exist. It is now in the plugin’s troubleshooting table, because I would rather nobody else spend twenty minutes on it.

Does the Test Actually Catch the Bug?

A regression test that has never seen the regression is decoration. So before declaring victory I took a copy of the plugin, stripped out the fix, and ran the loop against a real jail:

changed: [smokeroot] => (item=one)
failed:  [smokeroot] (item=two)   => ssh: Could not resolve hostname smokeroot
failed:  [smokeroot] (item=three) => ssh: Could not resolve hostname smokeroot

There it is. Item one succeeds while the redirect from _connect() is still in place; the option reload before item two throws it away; SSH goes looking for a host called smokeroot and finds nothing. Exactly the shape the reporter described, and exactly what reading the ansible-core source predicted.

With the fix restored, all three jails pass all twenty tasks of the smoke test, including the loop.

What It Cost

Before the build, the filesystem used 1.3 GB. After three jails with a Python interpreter each:

Filesystem      Size    Used   Avail Capacity  Mounted on
/dev/vtbd0p2    9.2G    2.4G    6.0G    29%    /

Mem: 55M Active, 188M Inact, 148M Wired, 71M Free

That is roughly 1.1 GB for all three test targets together. More importantly, the idle jails add no processes at all. The host’s process count rises and falls with SSH sessions, but the jails contribute nothing until a task runs in one; when a play does run, the peak load is one Python interpreter at a time.

There is a small irony in the privilege escalation, too. This host uses mdo(1), which I have written about before and which has replaced sudo on most of my machines. My own plugin only supports doas, sudo or nothing at all — so to test it I installed doas on a box that had been happily living without it. Adding mdo as a fourth choice is now on the list.

Takeaways

• pkgbase makes disposable jails genuinely disposable. 33 MB and twelve packages for a working userland changes what is worth spinning up.
• A jail does not need to run anything. persist plus mount.devfs and a path is a complete, useful test target. exec.start and /etc/rc are for jails that provide services.
• pkg --rootdir does two things you must do yourself: copy the repository fingerprints into the rootdir, and run ldconfig afterwards.
• Free-tier hardware is underrated for test infrastructure. Half a gig of RAM is not enough for many modern service stacks; it is more than enough to hold three jails still while something else pokes at them.
• Test the test. Reverting the fix and watching the regression reappear took two minutes and is the only reason I can claim the bug is actually fixed.

The plugin fix shipped as 2.0.2. The jails are still running, costing nothing, waiting for the next bug report.