bhyve Inside a Jail: The FreeBSD Feature Nobody Uses

Thu 27 August 2026
22 min read
FreeBSD
#freebsd, #bhyve, #virtualization, #jails, #vm-bhyve, #zfs, #vnet, #networking

A hypervisor running virtual machines inside a container

Table of Contents
What bhyve Is, and What It Is Not
- Loading the modules
Raw bhyve, Once
vm-bhyve as the Daily Interface
Storage: zvol or Raw Image
Networking, and the Diagram That Explains Everything
The Actual Subject: bhyve Inside a Jail
Failure Modes
What I Would Keep

I have written about jails on this blog often enough that the position is not in doubt. Jails are cheap, they are fast, they are the right default, and the vast majority of what people run in virtual machines would be happier in a jail. That is still what I believe.

It is also, occasionally, wrong. A jail shares the host kernel. That single sentence is the whole boundary of the technology. The moment your workload needs a kernel that is not the host’s kernel, no amount of enthusiasm for jails will help you, and you need a hypervisor.

I hit that wall four times in the last year, and every one of them has already shown up on this blog in passing:

Home Assistant OS ships as an appliance image. It is not going into a jail.
The DN42 border router runs MikroTik CHR, which is somebody else’s routing stack and definitely somebody else’s kernel.
Testing a 16-CURRENT kernel without volunteering the host as the test subject.
The occasional vendor tool that only exists for Linux or Windows.

In every one of those articles, bhyve appeared in a sentence or two and then the article moved on to its actual subject. It has never had a piece of its own here. This is that piece, and the part I actually want to write about is the part that gets skipped everywhere else: the hypervisor does not have to run on the host. It can run inside a jail.

What bhyve Is, and What It Is Not

bhyve is a hypervisor in the FreeBSD base system. There is no port to install for the core functionality, no subscription, and no daemon. You need hardware virtualization extensions (on amd64, which is what this article is about, that means VT-x with EPT on Intel or AMD-V with RVI on AMD; bhyve also runs on arm64 and RISC-V these days), the vmm kernel module, and that is essentially the list.

# kldstat | grep vmm
 7    1 0xffffffff82584000   380270 vmm.ko

Each running guest gets a device under /dev/vmm/<name>, and serial consoles are handled by nmdm, the null-modem driver, which gives you a pair of pseudo-terminals wired back to back.

What bhyve is not is a virtualization platform. bhyve itself provides no live migration, clustering, HA, web interface, or concept of a resource pool. If you want the thing where a VM moves between hosts because one of them got sad, that is what the Proxmox cluster article was about, and bhyve is not competing for that job. bhyve runs virtual machines on one box, extremely well, with almost no moving parts. Judge it on that.

One naming detail that will trip you up, because it tripped me up. On 15.1-RELEASE the userland control utility is still bhyvectl:

# ls -l /usr/sbin/bhyve*
-r-xr-xr-x  1 root wheel 505672 Jun 24 14:53 /usr/sbin/bhyve
-r-xr-xr-x  1 root wheel  55216 Jun 24 14:53 /usr/sbin/bhyvectl
-r-xr-xr-x  1 root wheel  18744 Jun 24 14:53 /usr/sbin/bhyveload

There is no /usr/sbin/vmmctl. But /dev/vmmctl very much exists, and it is the control device the kernel side uses to create and destroy VM instances. So when you see vmmctl in a devfs ruleset later in this article, that is a device node, not a command. The names invite exactly the wrong assumption.

Loading the modules

While we are in the basement, one thing worth doing correctly from the start. My hypervisor host’s /boot/loader.conf:

# --- Bhyve & Virtualization Modules ---
vmm_load="YES"              # The bhyve hypervisor kernel module
nmdm_load="YES"             # Null modem for VM serial consoles
if_bridge_load="YES"        # Network bridging
if_epair_load="YES"         # Virtual ethernet pairs (for VNET jails)
if_tap_load="YES"           # Virtual ethernet tap devices (for bhyve)

There are two separate concerns here. For ordinary virtio guests, vmm can be loaded later with kldload, and putting it in kld_list is valid. The vmm(4) manual requires boot-time loading only when PCI devices must be reserved for passthrough, before their normal host drivers claim them.

This setup deliberately does not use passthrough. I still keep the five modules together in loader.conf so they are present before the jail service starts; in particular, the allow.vmm jail parameter does not exist until vmm.ko is loaded. If you prefer kld_list, make sure vmm is loaded before the jail starts.

Raw bhyve, Once

I am going to show the manual invocation exactly once, because everything after this hides it behind a tool, and it is worth seeing what is being hidden.

A minimal FreeBSD guest, booted with bhyveload rather than UEFI:

bhyveload -m 1024 -d /dev/zvol/zroot/vm/example/disk0 example

bhyve -c 2 -m 1024 -H -P \
    -s 0,hostbridge \
    -s 3,virtio-blk,/dev/zvol/zroot/vm/example/disk0 \
    -s 4,virtio-net,tap0 \
    -s 31,lpc \
    -l com1,/dev/nmdm-example.1A \
    example

Two commands, and you have a virtual machine. The -s flags are PCI slot assignments: a host bridge, a virtio block device backed by a zvol, a virtio NIC backed by a tap interface, and an LPC bridge carrying the serial console out to an nmdm device.

When a guest dies badly, the VM instance survives it, and the next start fails because the name is taken:

bhyvectl --destroy --vm=example

That is the entire mental model. vm-bhyve, which is what I actually use, is a shell script that writes those command lines for you from a config file. Nothing more mysterious than that.

vm-bhyve as the Daily Interface

One thing to hold in mind for the next three sections: on my box every command here runs inside the jail, not on the host. The host has no vm-bhyve package, no vm_enable, and no datastore of its own. I am showing the setup without that complication first because it is the same either way, and coming back to the jail-specific half once the jail itself has been built.

pkg install vm-bhyve
sysrc vm_enable="YES"
sysrc vm_dir="zfs:zroot/vm"
vm init

The zfs: prefix on vm_dir is the part that matters. It tells vm-bhyve that the datastore is a ZFS dataset rather than a directory, and from that point on every guest is a dataset of its own. Which means every guest inherits your snapshot policy for free. Mine looks like this:

NAME                       USED  RECSIZE
zroot/vm                  28.3G     128K
zroot/vm/bsdmirror        19.5G     128K
zroot/vm/bsdmirror/disk0  19.5G        -
zroot/vm/bsdtest          7.45G     128K
zroot/vm/bsdtest/disk0    7.45G        -
zroot/vm/smallpox         68.9M     128K

zroot/vm is covered by the same sanoid policy as everything else on the box, so there are daily and weekly snapshots of every guest sitting there without my having configured anything VM-specific. Snapshot before a guest upgrade, roll back when the guest upgrade goes the way guest upgrades go. This is the FreeBSD Foundationals: ZFS argument applied to virtual machines, and it is most of the reason to put the datastore on ZFS in the first place.

Day to day, the interface is four commands: vm create, vm install, vm console, vm list.

NAME       DATASTORE  LOADER     CPU  MEMORY  VNC  AUTO     STATE
bsdmirror  default    bhyveload  2    1024    -    No       Running (4162)
bsdtest    default    bhyveload  1    256M    -    No       Stopped
smallpox   default    uefi       1    256m    -    Yes [1]  Running (9704)

Note the LOADER column: bhyveload for the FreeBSD guests, uefi for the one that needs firmware. Both paths coexist happily on the same host, which is worth knowing before you go looking for a global setting.

Storage: zvol or Raw Image

Both of my storage styles are visible in that output, so let me use the real thing rather than a hypothetical.

bsdmirror and bsdtest are backed by zvols, requested in the guest config with one line:

disk0_type="virtio-blk"
disk0_name="disk0"
disk0_dev="sparse-zvol"

Which produces:

NAME                      VOLSIZE  VOLBLOCK   USED  COMPRESS
zroot/vm/bsdmirror/disk0      50G       16K  19.5G  on
zroot/vm/bsdtest/disk0        20G       16K  7.45G  on

Sparse, so a 50G guest disk costs 19.5G of pool. 16K volblocksize, compression on.

smallpox uses the default instead, a raw image file on the dataset:

disk0_type="virtio-blk"
disk0_name="disk0.img"

The tradeoff is not dramatic and I would not lose sleep over either choice. zvols give you a block device with its own properties, snapshots, and tuning knobs, and they are the natural fit when the guest is going to be long-lived. Actual performance depends on the workload and the block-size choices at both layers. Raw images are a single file, which means that, with the guest stopped, you can copy one out with cp, hand it to somebody, or move it between datastores without thinking about ZFS at all.

The one trap worth naming: if the guest runs its own copy-on-write filesystem, and for FreeBSD guests it usually does, you now have ZFS on top of ZFS. Both layers will try to be clever about block sizes, and the interaction is rarely an improvement. Match volblocksize to what the guest actually writes rather than leaving it at whatever you first typed, and resist the urge to enable every feature on both layers.

Networking, and the Diagram That Explains Everything

This is where the jail story starts, so it is worth being precise.

On a conventional bhyve host, the topology is short: a bridge, a tap per guest NIC, done. On my setup the hypervisor lives inside a jail, called tardis, and the result is a two-tier arrangement that took me longer to hold in my head than I would like to admit. That jail is the subject of the rest of this article; for now all you need is its name, because it turns up in every interface name below. Here is one NIC, end to end:

host bridge0
   |
   +-- e0a_tardis        (epair, "a" side, stays on the host)
         ~~~~~~~~~~
       e0b_tardis        (epair, "b" side, handed into the jail,
                          then renamed to vnet0 by the jail's rc.conf)
                              |
                          vm-public   (vm-bhyve switch "public",
                                       a bridge inside the jail)
                              |
                            tap0  ->  guest virtio-net

Three of those, in parallel: bridge0 to vm-public, bridge1 to vm-wan, bridge2 to vm-lan. Management, WAN, and LAN, which is the same three-way separation I use everywhere else and the same one described in the AS201379 peering articles.

Two details in there cost me real time.

The interface is renamed explicitly. The jail config names the b side of each epair e0b_tardis, e1b_tardis, and e2b_tardis. The vnet.interface parameter moves them into the jail with those names intact. These lines in the jail’s /etc/rc.conf perform the rename when its network service starts:

ifconfig_e0b_tardis_name="vnet0"
ifconfig_e1b_tardis_name="vnet1"
ifconfig_e2b_tardis_name="vnet2"

That is why the interfaces are vnet0, vnet1, and vnet2 by the time vm-bhyve sees them. If you omit these lines, configure the vm-bhyve switches with the original e0b_tardis names instead. The rename is a policy choice, not something VNET does automatically.

The taps are invisible from the host. This is the payoff of the entire arrangement, and you can see it in two commands. From the host:

# ifconfig -l
bge0 bge1 lo0 pflog0 bridge0 bridge1 bridge2
e0a_tardis e1a_tardis e2a_tardis

Bridges, and the host side of some epairs. No taps, no VM plumbing, nothing hypervisor-shaped at all. From inside the jail:

# jexec tardis ifconfig -l
lo0 pflog0 vnet0 vnet1 vnet2 vm-public vm-wan vm-lan tap0 tap1 tap2 tap3 tap4

Everything. And vm-bhyve labels the taps helpfully, so you never have to guess which one belongs to what:

tap0: description: vmnet/smallpox/0/public
tap1: description: vmnet/smallpox/1/wan
tap2: description: vmnet/smallpox/2/lan
tap3: description: vmnet/bsdmirror/0/public
tap4: description: vmnet/bsdmirror/1/lan

Five taps for two running guests: three NICs on smallpox, two on bsdmirror. bsdtest is stopped and therefore adds no tap interfaces.

If you already run pf on the host, and after the pf guide I assume some of you do, be aware that bridged traffic and pf interact through net.link.bridge.pfil_bridge and its neighbours. The defaults have changed over the years and the failure mode is “traffic silently does or does not get filtered, differently from what you assumed”. Check the sysctls rather than trusting your memory of them.

The Actual Subject: bhyve Inside a Jail

Here is the thing that made me want to write this. allow.vmm is not new. It was committed as r337023 on 1 August 2018 and shipped in FreeBSD 12.0. It has been sitting in the base system for eight years, it is documented in jail(8), and in all that time I have seen it discussed approximately never.

Why bother

Because it turns the hypervisor into a normal service.

Everything the VM host needs, the zvols, the taps, the consoles, its own routing table and its own firewall, lives inside one jail. service jail restart tardis restarts the entire virtualization stack. zfs send of the jail plus its datastore moves it. The host underneath stays a boring machine whose only job is running jails, with no VM-shaped special case in its configuration.

If you have spent any time keeping a hypervisor host tidy, you will recognise the appeal. The host stops being a pet.

The four things that have to line up

One: allow.vmm = 1. This is the parameter that lets a process inside the jail open vmm at all. jail(8) puts it plainly:

allow.vmm - The jail may access vmm(4). This flag is only available when the vmm(4) kernel module is loaded.

Note the second sentence, because it produces a confusing failure: if vmm.ko is not loaded on the host, the parameter does not merely have no effect, it does not exist, and your jail fails to start with a complaint about an unknown parameter.

Two: a devfs ruleset that unhides the right nodes. This is the piece people get wrong, and it is why vm console works for some and not others. My ruleset 10:

[devfsrules_bhyve_jail=10]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path fuse unhide
add path zfs unhide
add path pf unhide
add path 'vmm*' unhide
add path 'vmm/*' unhide
add path 'vmm.io*' unhide
add path 'vmm.io/*' unhide
add path 'vmmctl' unhide
add path 'tap*' unhide
add path 'nmdm*' unhide
add path 'zvol*' unhide
add path 'zvol/*' unhide
add path 'zvol/*/*' unhide
add path 'zvol/*/*/*' unhide
add path 'zvol/*/*/*/*' unhide

Working through it:

It starts from devfsrules_hide_all and then unhides basic and login. Start from nothing and add, never the reverse.
vmm*, vmm/*, vmm.io*, vmm.io/*, vmmctl are the hypervisor devices, including the per-guest instances that appear under /dev/vmm/ and the control device discussed earlier.
tap* is guest NICs. Without it the guest starts and has no network.
nmdm* is guest serial consoles. Without it vm console is dead and you will assume the guest failed to boot when it is running perfectly and simply cannot talk to you.
zvol* and the nested globs are block-backed guests. The nesting matters: the path is zvol/<pool>/<dataset>/<disk>, so a single level of glob is not enough, and how deep you need to go depends on how deep your datasets are nested. Mine goes to five levels because I would rather not revisit this.
zfs and pf let the jail manage its own datasets and its own firewall.
fuse is, on my system, a leftover. fusefs is not loaded and nothing in the jail mounts anything through it. I am leaving it in the printed ruleset because it is what is actually deployed, but it is doing nothing, and if you are building this fresh you can skip it.

The result is a /dev that stays genuinely small:

# jexec tardis ls /dev
fd  nmdm-bsdmirror.1A  nmdm-bsdmirror.1B  nmdm-smallpox.1A
nmdm-smallpox.1B  null  pf  pts  random  stderr  stdin  stdout
tap0  tap1  tap2  tap3  tap4  urandom  vmm  vmm.io  vmmctl
zero  zfs  zvol

Three: ZFS delegation. vm-bhyve needs to create datasets and zvols, from inside the jail, in a dataset the host owns. That takes a matching pair. On the host side, the dataset is marked jailed and handed over as the jail is created:

zfs set jailed=on zroot/vm

exec.created += "zfs jail $name zroot/vm";

And in the jail config, permission to actually mount things:

allow.mount;
allow.mount.devfs;
allow.mount.zfs;
enforce_statfs = 1;

Miss jailed=on and vm create fails with a permissions error that points at the dataset and does not tell you which of the two halves you forgot. The result, once it works, is that the jail mounts its own datastore:

# jexec tardis mount
zroot/jails/tardis on / (zfs, local, noatime, nfsv4acls)
zroot/vm on /zroot/vm (zfs, local, noatime, nfsv4acls)
zroot/vm/smallpox on /zroot/vm/smallpox (zfs, local, noatime, nfsv4acls)
zroot/vm/bsdtest on /zroot/vm/bsdtest (zfs, local, noatime, nfsv4acls)
zroot/vm/bsdmirror on /zroot/vm/bsdmirror (zfs, local, noatime, nfsv4acls)
devfs on /dev (devfs)

Four: allow.mlock = 1. bhyve wires guest memory rather than letting it be paged. A jail cannot do that by default. Leave this out and the guest refuses to start, with an error that describes a memory allocation failure and does not obviously say “you forgot a jail parameter”.

The fifth thing, which is deliberately absent

Ask the running jail what it is allowed to do and one entry stands out:

# jls -j tardis -n
...
allow.vmm
allow.novmm_ppt
...

allow.vmm_ppt is off. PCI passthrough is a separate capability from access to vmm, and this is new: it was added in FreeBSD 15.1 as commit 94066f9a07ad, sponsored by the FreeBSD Foundation and Klara. The release notes describe it as a knob “to control PCI passthrough access in bhyve(8) jails”.

jail(8) is unusually direct about why you want it off:

This allows privileged users inside the jail to manipulate physical devices claimed by the ppt driver, and thus must not be configured in untrusted jails.

So a jailed hypervisor runs virtio guests all day and cannot hand a physical device to a guest. For me that is the correct trade, and it is also the honest answer to the question people will ask after the ErsatzTV article: no, you cannot do the GPU passthrough thing this way. If you need passthrough, you are running bhyve on the host, or you are turning the knob on and accepting what jail(8) just told you.

The full jail configuration

For reference, the whole thing, with the three-NIC vnet plumbing:

tardis {
    devfs_ruleset = 10;
    exec.clean;
    exec.consolelog = /var/log/jails/tardis_console.log;
    exec.start = '/bin/sh /etc/rc';
    exec.stop = '/bin/sh /etc/rc.shutdown';
    host.hostname = "tardis";
    mount.devfs;
    path = "/zroot/jails/$name";

    # --- Virtualization & ZFS Permissions ---
    allow.vmm = 1;
    # allow.vmm_ppt is deliberately omitted, and therefore off
    allow.mount;
    allow.mount.devfs;
    allow.mount.zfs;
    enforce_statfs = 1;
    allow.mlock = 1;
    allow.chflags;   # not bhyve-related, see below

    vnet = new;
    vnet.interface = "e0b_tardis", "e1b_tardis", "e2b_tardis";

    # NIC 0 (Management -> bridge0)
    exec.prestart += "epair0=\$(ifconfig epair create) && ifconfig \${epair0} up -txcsum -rxcsum -txcsum6 -rxcsum6 -tso -lro name e0a_tardis && ifconfig \${epair0%a}b up -txcsum -rxcsum -txcsum6 -rxcsum6 -tso -lro name e0b_tardis";
    exec.prestart += "ifconfig bridge0 addm e0a_tardis";
    exec.prestart += "ifconfig e0a_tardis description \"vnet0 mgmt for tardis\"";

    # NIC 1 (WAN/BGP -> bridge1)
    exec.prestart += "epair1=\$(ifconfig epair create) && ifconfig \${epair1} up -txcsum -rxcsum -txcsum6 -rxcsum6 -tso -lro name e1a_tardis && ifconfig \${epair1%a}b up -txcsum -rxcsum -txcsum6 -rxcsum6 -tso -lro name e1b_tardis";
    exec.prestart += "ifconfig bridge1 addm e1a_tardis";
    exec.prestart += "ifconfig e1a_tardis description \"vnet1 WAN/BGP for tardis\"";

    # NIC 2 (LAN -> bridge2)
    exec.prestart += "epair2=\$(ifconfig epair create) && ifconfig \${epair2} up -txcsum -rxcsum -txcsum6 -rxcsum6 -tso -lro name e2a_tardis && ifconfig \${epair2%a}b up -txcsum -rxcsum -txcsum6 -rxcsum6 -tso -lro name e2b_tardis";
    exec.prestart += "ifconfig bridge2 addm e2a_tardis";
    exec.prestart += "ifconfig e2a_tardis description \"vnet2 LAN for tardis\"";

    # Teardown
    exec.poststop += "ifconfig e0a_tardis destroy";
    exec.poststop += "ifconfig e1a_tardis destroy";
    exec.poststop += "ifconfig e2a_tardis destroy";

    exec.created += "zfs jail $name zroot/vm";
    exec.jail_user = root;
    exec.system_user = root;
    persist;
}

The exec.prestart lines each create an epair, disable every offload on both halves, rename both ends to something a human can read, and add the host side to a bridge. The offload flags (-txcsum -rxcsum -txcsum6 -rxcsum6 -tso -lro) are not superstition. Checksum offload and TSO across an epair, through a bridge, into a virtio NIC in a guest is a well-known way to get packets that are subtly wrong rather than absent, which is a far worse debugging experience than no connectivity at all. Turn them off and move on with your life.

allow.chflags is not part of the bhyve story. It lets privileged users in the jail clear system file flags such as schg, which matters for jails that maintain their own userland and not at all for running virtual machines. It is in my config because it is in my other jail configs, and I am printing what is deployed. Nothing here needs it.

exec.poststop destroys the host-side interfaces, so restarting the jail does not leave a trail of orphaned epairs behind. persist keeps the jail alive with no processes running in it, which matters because a vnet jail that exits takes its network configuration with it.

Bringing vm-bhyve up inside it

Now the half I deferred earlier. With the jail running, the setup from the vm-bhyve section happens inside it, and nowhere else:

jexec tardis pkg install vm-bhyve
jexec tardis sysrc vm_enable="YES"
jexec tardis sysrc vm_dir="zfs:zroot/vm"
jexec tardis vm init

vm_enable and vm_dir go in the jail’s /etc/rc.conf. The host’s rc.conf has neither, and that is the point of the whole exercise: the host does not know it is a hypervisor.

vm_dir is the delegated dataset by its ZFS name, zroot/vm, not by a path. That name is identical inside and outside the jail, which is convenient, and it is the same dataset that zfs jail handed over at exec.created. The jail mounts it itself at startup, which is what the mount output above shows, and vm init then creates the datastore layout inside it.

The vm-bhyve switches are created in the jail too, against the renamed vnet interfaces:

jexec tardis vm switch create public
jexec tardis vm switch add public vnet0

Repeat for wan on vnet1 and lan on vnet2, and that produces the vm-public, vm-wan, and vm-lan bridges from the diagram. From here vm create, vm install, and vm console behave exactly as they do on a bare host, which is the whole reason this arrangement is worth the setup.

What this does not buy you

I want to be clear about this, because “hypervisor in a jail” sounds like it should be a security improvement, and it is not.

allow.vmm hands the jail a large and complicated piece of kernel attack surface. The vmm code is not a small, carefully minimised interface, and a root process inside that jail is talking to it directly. Confining the hypervisor to a jail is an operational win: it packages the thing, it makes it movable and restartable, it keeps the host clean. It is not a containment upgrade, and a bhyve-capable jail is not somewhere to put a workload you do not trust. Turning allow.vmm_ppt on as well would make that considerably more true.

Treat the jail boundary here as an organisational tool, not a security boundary, and you will not be disappointed by it.

Failure Modes

The ones I actually hit, in roughly the order I hit them:

The console works from the host but not from the jail. Missing nmdm* in the devfs ruleset. The guest is fine. You just cannot talk to it.

vm create fails with a permissions error on the dataset. Half of the ZFS delegation is missing. jailed=on on the host side and zfs jail at exec.created are two separate steps and you need both.

The guest refuses to start with a memory error. allow.mlock.

The jail refuses to start, complaining about an unknown parameter. vmm.ko is not loaded on the host, so allow.vmm does not exist yet. Put vmm_load="YES" in loader.conf.

Networking that half works. Offloads on the epairs. Small packets fine, large transfers corrupt or stalled.

A guest will not restart after an unclean shutdown. Stale instance under /dev/vmm/. bhyvectl --destroy --vm=<name> and try again.

What I Would Keep

All of it, which is not something I say often after a year of running something.

Jails remain the answer for almost everything. bhyve is for the handful of workloads that genuinely need their own kernel, and there are fewer of those than people assume. But wrapping the hypervisor in a jail is the part I would do again without hesitating, because it moved virtual machines from “special thing the host does” into “one more jail on a host that only runs jails”. The host has no VM-specific configuration beyond five module loads in loader.conf. Everything else is inside tardis, snapshotted nightly along with everything else, and restartable with one command.

Eight years this has been in the base system. It deserves more use than it gets.