Automating FreeBSD Jails with cdist - Zero Dependencies Inside the Jail

Most configuration management tools still assume they own the target. Ansible ships Python modules over the wire and runs them in place. Salt wants a minion on every host. Chef wants a client, Puppet wants an agent.

Jails break that assumption in a satisfying way. A FreeBSD jail is supposed to be small - sometimes a single static binary, an rc.d script, and a few lines in rc.conf. Installing Python into every jail just so Ansible can run its setup module is, to borrow a phrase, the tail wagging the dog. I already wrote about a workaround for Ansible: the jailexec connection plugin, which SSHes to the jail host and uses jexec to tunnel commands into each jail. It works, and I still reach for it when I already have an Ansible setup. But many Ansible modules assume Python on the target, so in practice some of those jails still end up growing a Python interpreter.

Then I tried cdist, and everything got smaller.

Table of Contents

What cdist Actually Is

cdist is a configuration management system written in Python that runs entirely on the control machine. On the target, it expects a Bourne-style sh environment. That is it. No agent, no client, no target-side runtime of any kind. cdist assembles a shell script locally, ships it over the wire, and watches stdout come back. The target does not know it is being managed any more than it would know it was being SSHed into.

That minimalism is the selling point. cdist’s own tagline reads “usable configuration management”, but what I read between the lines is: the entire target-side runtime is whatever shell is already on the box. If you can ssh to it, you can cdist it. No bootstrap step, no package install, no “please add our apt key first”. For a FreeBSD jail that exists precisely to run one daemon and nothing else, that is perfect.

The model is also refreshingly simple conceptually. You write manifests (declarative entry points) that invoke types (reusable building blocks: __file, __package_pkgng, __service, __line, and so on). Every type is itself a small directory of shell scripts - an explorer that reports current state, a gencode-remote that emits the shell commands to reach the desired state, and optional helpers. cdist runs the explorers on the target, generates a shell script locally, sends it over ssh, and runs it with sh -e. That is the whole loop.

The Two Hooks That Make This Interesting

cdist’s default transport is OpenSSH: ssh user@host sh -c '…' for commands, scp for files. But the transport is swappable. Two command-line flags override it:

• --remote-exec PATH - a script that takes ssh-style options followed by <target> <command…> and runs the command on the target somehow.
• --remote-copy PATH - a script that takes scp-style arguments and copies files to the target somehow.

In this setup, the cdist target name is not a hostname at all; it is the jail name. JAIL_HOST tells the wrappers which real SSH endpoint to use.

The API is dumb in the best way. cdist gives you the target name and a command line; the script does whatever it needs to. If you want to tunnel through a bastion, go ahead. If you want to talk to a switch over a serial console, go ahead. And if you want to SSH to a FreeBSD host and drop into a jail with jexec - which is exactly what I want - that is two small Python scripts. The only persistent requirement is host-side access to ssh, scp, jexec, and jls; the jail itself remains untouched except for the configuration changes cdist applies.

jexec-ssh.py

Here is the remote-exec wrapper in full. It is 60 lines, most of them boilerplate:

#!/usr/bin/env python3
import os
import shlex
import sys

_SSH_OPTS_WITH_VALUE = frozenset(
    ("-o", "-F", "-i", "-l", "-p", "-c", "-m", "-L")
)


def strip_ssh_options(args):
    i = 0
    while i < len(args):
        arg = args[i]
        if arg in _SSH_OPTS_WITH_VALUE:
            i += 2
        elif arg.startswith("-"):
            i += 1
        else:
            return i
    return i


def main():
    jail_host = os.environ.get("JAIL_HOST")
    if not jail_host:
        sys.exit("error: JAIL_HOST is not set")

    jail_user = os.environ.get("JAIL_USER", "root")

    args = sys.argv[1:]
    start = strip_ssh_options(args)
    if start >= len(args):
        sys.exit("error: no target jail name on the command line")

    jailname = args[start]
    remote_payload = " ".join(args[start + 1:])

    jexec_cmd = (
        f"jexec -u {shlex.quote(jail_user)} {shlex.quote(jailname)} "
        f"/bin/sh -c {shlex.quote(remote_payload)}"
    )

    os.execvp("ssh", ["ssh", jail_host, jexec_cmd])


if __name__ == "__main__":
    main()

The logic is the whole story:

1. Read JAIL_HOST from the environment. That is the ssh endpoint for the host, not the jail.
2. Strip the ssh-style options cdist prepends - things like -o User=root - which are meaningful to a jail’s (nonexistent) direct ssh endpoint, not to ours. Real connection options belong in ~/.ssh/config for the host.
3. The first non-option argument is the jail name. Everything after that is the command cdist wants to execute.
4. Wrap the command in jexec -u <user> <jail> /bin/sh -c '…', shlex.quote‘d so pipelines, redirections and globs survive transit through two shells.
5. execvp into the real ssh, replacing the Python process. ssh does the heavy lifting, and its exit code flows straight back to cdist.

The command is crossing two shell boundaries: the host-side SSH command line and the in-jail sh -c. That is why the wrapper leans so hard on shlex.quote(). The /bin/sh -c wrapper itself is not cosmetic. Without it, jexec hands the argv directly to whatever binary sits at position one, and cdist-generated constructs like for f in /etc/*.conf; do …; done die the moment a glob shows up. Running the payload through a fresh sh inside the jail restores the shell environment cdist expects.

jexec-scp.py

File transfer is slightly more involved, but only slightly. scp does not know what a jail is - but it does know what a host path is, and a jail’s “inside” is always some path on the host (/zroot/jails/<name> is typical). We can translate.

#!/usr/bin/env python3
import os
import subprocess
import sys

_SCP_OPTS_WITH_VALUE = frozenset(("-o", "-F", "-i", "-S", "-P"))


def split_args(args):
    opts, paths = [], []
    i = 0
    while i < len(args):
        arg = args[i]
        if arg in _SCP_OPTS_WITH_VALUE and i + 1 < len(args):
            opts.extend((arg, args[i + 1]))
            i += 2
        elif arg.startswith("-"):
            opts.append(arg)
            i += 1
        else:
            paths.append(arg)
            i += 1
    return opts, paths


def jail_root(host, jailname):
    out = subprocess.check_output(
        ["ssh", host, "/usr/sbin/jls", "-j", jailname, "path"],
        text=True,
        stderr=subprocess.STDOUT,
    )
    return out.strip()


def rewrite(path, host):
    if ":" not in path:
        return path
    prefix, rpath = path.split(":", 1)
    if "/" in prefix or not prefix:
        return path
    root = jail_root(host, prefix)
    return f"{host}:{os.path.join(root, rpath.lstrip('/'))}"


def main():
    jail_host = os.environ.get("JAIL_HOST")
    if not jail_host:
        sys.exit("error: JAIL_HOST is not set")

    opts, paths = split_args(sys.argv[1:])
    if len(paths) < 2:
        sys.exit("error: scp needs a source and a destination")

    translated = [rewrite(p, jail_host) for p in paths]
    os.execvp("scp", ["scp", *opts, "-r", "-p", *translated])


if __name__ == "__main__":
    main()

Walking through it:

1. For each argument, check if it looks like <jail>:/some/path. If not - local paths, host-qualified paths - leave it alone.
2. If it does, ssh to the host and run jls -j <jail> path. jls is the canonical way to ask FreeBSD “where does jail X live on disk right now”.
3. Rewrite the argument to <JAIL_HOST>:<jailroot>/<path>, which is just a perfectly ordinary scp remote path.
4. execvp into the system scp with the rewritten arguments.

There is nothing magical here. scp was already perfectly capable of copying files to root@radon.example.com:/zroot/jails/testvnet/etc/motd. All the wrapper does is save the user the trouble of typing it and let cdist’s file types behave as if the jail were a first-class ssh target. The simple implementation asks jls once per jail-qualified path; caching jail roots would be an easy optimization if you start moving lots of files.

Actually Running It

On a host with many jails, SSH connection setup can dominate runtime; enabling multiplexing on the host connection makes a dramatic difference. More on that in the repo’s README.

A minimal setup. First, a manifest at ./conf/manifest/init - one line is enough to prove the whole pipeline works end to end:

__file /tmp/cdist-wrapper-success --state present

Then point cdist at the two wrappers:

~/tmp/cdist-test ❯ export JAIL_HOST=root@radon.edelga.se
~/tmp/cdist-test ❯ cdist config -c ./conf \
    --remote-exec ./jexec-ssh.py \
    --remote-copy ./jexec-scp.py \
    testvnet lg

In this example, testvnet and lg are jail names, not DNS names or SSH inventory entries. cdist is quiet on success: no output is good output. A sanity check on the host itself confirms it:

[root@radon ~]# jexec testvnet ls -l /tmp/cdist-wrapper-success
-rw-------  1 root wheel 0 Apr 11 21:02 /tmp/cdist-wrapper-success
[root@radon ~]# jexec lg ls -l /tmp/cdist-wrapper-success
-rw-------  1 root wheel 0 Apr 11 21:18 /tmp/cdist-wrapper-success

Two jails, one cdist invocation, zero daemons, zero extra interpreters, nothing mutated outside the jails themselves. Replace the one-liner manifest with real cdist types - __package_pkgng, __file, __line, __service, __user - and you get the same thing at whatever scale you need.

A More Practical Example: A Custom Maintenance Type

The smoke test above proves the transport works. Here is a pragmatic host-maintenance example I use in practice: a custom type that runs base and package updates inside a jail. This is more of a maintenance action than a pure convergent state type, but it is a good illustration of how little machinery cdist needs when you do want to run shell-driven lifecycle tasks.

First, create the type’s directory and its gencode-remote script. This is the shell that cdist will execute inside the jail:

mkdir -p conf/type/__freebsd_system_update

#!/bin/sh
# conf/type/__freebsd_system_update/gencode-remote

set -e

echo "Checking for and installing FreeBSD base updates..."
# PAGER=cat prevents freebsd-update from dropping into an
# interactive pager and waiting for 'q' on a headless target.
env PAGER=cat freebsd-update fetch install

echo "Updating package catalogue and installing upgrades..."
pkg update
pkg upgrade -y

chmod +x conf/type/__freebsd_system_update/gencode-remote

Then wire it into the manifest. cdist exposes $__target_os as an explorer, so the type only fires on FreeBSD targets:

#!/bin/sh
# conf/manifest/init

if [ "$__target_os" = "FreeBSD" ]; then
    __freebsd_system_update
else
    echo "Info: $__target_host is not FreeBSD, skipping updates."
fi

Run it the same way as before:

~/tmp/cdist-test ❯ cdist config -c ./conf \
    --remote-exec ./jexec-ssh.py \
    --remote-copy ./jexec-scp.py \
    testvnet lg

That is the whole workflow for writing a custom cdist type: a directory, a shell script, and a one-line invocation in the manifest. No YAML, no plugin protocol, no class hierarchy. The type is just shell, and the wrappers make sure that shell runs inside the jail.

Why This Is The Unix Way

I wrote these scripts one Monday afternoon. By dinnertime I had working state management across every jail on my FreeBSD hosts, without installing anything new on a single one of them. That is not a coincidence; it is what happens when every component in the stack knows exactly one job and does it well.

Count the actors:

• ssh - authenticated, encrypted transport
• scp - file copy over ssh
• jexec - process execution inside a running jail’s namespace
• jls - jail inspection
• sh - shell execution
• cdist - render desired state into a shell script

Each piece is narrow, composable, and independently useful. The two Python files in this repo do not replace any of them; they only connect them. Together that is roughly 120 lines of Python orchestrating several decades of battle-tested Unix primitives. None of those tools are tightly coupled to each other. None of them share a runtime. If I delete any one file from the pipeline, the rest still do their jobs. That is the thing people mean when they talk about “the Unix philosophy”, and it is the thing I keep coming back to every time I look at how modern tools solve the same problems.

Compare the dependency footprint inside the jail:

The same story plays out in the implementations themselves. My Ansible jailexec connection plugin is roughly 960 lines of Python, because Ansible’s connection plugin API asks for a fair amount of scaffolding: argument translation, a file-transfer state machine, subprocess management, the whole plugin protocol. These two cdist wrappers together come to about 120 lines. That difference is not about quality; it is about how much each tool asks of the transport layer. cdist’s hooks are deliberately narrow, so there is less to implement. And because cdist’s types are shell, the jail does not need a Python interpreter at all, whereas the Ansible path still typically does.

The smallest running jail on my radon.edelga.se host is 61 MB on disk - a thin-jail nullfs view over a shared base, running one static binary. It still configures beautifully with this pipeline, because the pipeline asks it for nothing it was not already providing.

When To Still Reach For Ansible

This is not the universal answer.

cdist is opinionated about shell, which means that if your configuration domain is “take this complicated JSON API, reason about its responses, compute a delta, push it back” - the kind of thing Ansible modules do in a page of Python each - you will end up writing that logic yourself in sh. That is fine for ten types; it is painful for a hundred. The moment you need rich remote introspection, structured data handling, or an ecosystem module that already solves a weird API, Ansible starts to earn its weight again.

If you already have a working Ansible setup across your infrastructure, the jailexec connection plugin is the path of least resistance for adding jails to it. You keep your playbooks, your roles, your inventory. I still use it where “use what’s already there” beats “be minimal”.

But for a fresh build of jails on a new host - or for the handful of single-purpose jails I run on radon - cdist plus these two wrappers is the right tool. It makes “managed state on a FreeBSD jail” exactly as simple as “I can ssh to the host and run jexec”, and it asks nothing more of the jail than FreeBSD already provides.

Getting The Scripts

Both scripts live on Codeberg as jexec-cdist, CC0 licensed, with a README that walks through the environment variables, the ssh multiplexing tip that turns a 15-second run into an instant one, the doas pattern for running as a non-root user on the host, and the handful of edge cases (nullfs unions, non-standard jail roots) where the path rewriting needs a small nudge.

Clone, chmod +x, point cdist at them, export JAIL_HOST. That is the entire onboarding. The jails stay as clean as the day you created them - and that is the thing I wanted all along.

References

• cdist - the configuration management system this article is about
• jexec-cdist on Codeberg - the two wrappers in this article
• cdist: writing your own remote-exec and remote-copy - the upstream documentation for the hooks these scripts plug into
• Managing FreeBSD Jails with Ansible: the jailexec connection plugin - the Ansible equivalent, for comparison
• FreeBSD Foundationals: Jails - the jail fundamentals this article assumes
• jexec(8), jls(8) - the FreeBSD primitives doing the actual work