Larvitz Blog

Running Your Own AS: Direct Hetzner Peering, a Fourth Edge, and Bringing the Home LAN into the Fabric

2026-04-16T00:00:00+02:00

Part 1 set up a single FreeBSD BGP router with two upstream providers. Part 2 added a Vultr edge with native peering and tied both routers together with iBGP. Part 3 joined LocIX Düsseldorf with a dedicated third edge router. This is Part 4.

A few months of operating a multi-PoP BGP network produces a shopping list. I wanted direct peering with networks that move real traffic, a fourth edge in a new PoP, and my own IPv6 space on the home LAN instead of ISP-assigned addressing. This article covers the changes that made that happen.

The headline, if I had to pick one, is two mtr traces. First, from a nettest jail on my home LAN to Hetzner’s network:

root@nettest:~ # mtr -rwz hetzner.com
HOST: nettest                                Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. AS201379      2a06:9801:1c:6000::1       0.0%    10    0.2   0.2   0.1   0.2   0.0
  2. AS201379      2a06:9801:1c:fff0::1       0.0%    10    3.2   3.2   2.9   3.4   0.2
  3. AS201379      ixbgp.edge.hofstede.it     0.0%    10    6.9   7.0   6.8   7.3   0.2
  4. AS???         2001:7f8:ca:1:0:2:4940:1   0.0%    10    7.6   7.6   7.3   7.9   0.2
  5. AS24940       core11.nbg1.hetzner.com    0.0%    10   10.9  10.7  10.5  10.9   0.1

Hop 1 is the home LAN gateway - a MikroTik announcing a /64 from my /48 via iBGP. Hop 2 is the iBGP tunnel to the core router. Hop 3 is the new fourth edge (ixbgp) at the iFog datacenter in Zürich. Hop 4 is Hetzner’s peering-LAN address on FogIXP in Frankfurt. That’s a direct BGP session between AS201379 and AS24940. Hop 5 is Hetzner’s backbone. Five hops, 10 ms, and everything in the middle is either mine or Hetzner.

The return path, from a Hetzner machine back to blog.hofstede.it, completes the picture:

[root@krypton ~]# mtr -rwz blog.hofstede.it
HOST: krypton                              Loss%   Snt   Last   Avg  Best  Wrst StDev
  4. AS???    2001:7f8:ca:1:0:20:1379:1     0.0%    10    5.1   5.1   5.1   5.2   0.0
  5. AS201379 ifog-gw.core.hofstede.it      0.0%    10    9.2  14.3   8.9  61.1  16.4
  6. AS201379 radon.server.hofstede.it      0.0%    10    9.9  10.2   9.7  10.9   0.3
  7. AS201379 hofstede.it                   0.0%    10   10.0  10.3   9.9  11.1   0.4

Hop 4 is AS201379’s own peering-LAN address on FogIXP - Hetzner hands traffic for my prefix directly to my router. From there it’s three hops inside AS201379 to the jail serving the blog. No intermediate transit AS, no DE-CIX hop, no shared upstream - just a direct handoff at the exchange.

Note on addresses: AS201379’s prefix 2a06:9801:1c::/48 and all hofstede.it hostnames are shown as-is. Public peering-LAN addresses from PeeringDB (FogIXP’s 2001:7f8:ca:1::/64, LocIX’s 185.1.155.0/24 and 2a0c:b641:701::/64) are also shown as-is. Everything else - provider-assigned addresses, tunnel endpoints, trusted-management IPs - has been replaced with RFC 5737 / RFC 3849 documentation ranges. Upstream and peer AS numbers are visible in public routing tables.

At a Glance

Part 4 adds three things to AS201379:

a fourth FreeBSD edge router at iFog Zürich (ixbgp)
a direct BGP peering session with Hetzner on FogIXP
a MikroTik home router speaking iBGP, bringing the home LAN into the /48

The result is lower-latency paths to Hetzner, more control over outbound policy, and stable provider-independent IPv6 at home.

Architecture: Four Edges and a Home Router

The topology grew in two dimensions. A fourth FreeBSD edge router (ixbgp) joins the existing three at a new PoP. Below hobgp, a new downstream site appears: the home LAN, reachable via an iBGP-speaking MikroTik.

                    ┌──────────────────────────────────────────────────┐
                    │                Default-Free Zone                 │
                    └──┬──────────┬──────────────┬──────────────┬──────┘
                       │          │              │              │
                   AS209533   AS209735       AS212895         AS34927
                   (iFog free) (Lagrange)    (route64)       (iFog paid)
                       │          │              │              │
                    GRE          GRE            GRE            eBGP over vtnet0
                       │          │              │              │      +
                       │          │              │              │  FogIXP RS1/2/3 (AS47498)
                       │          │              │              │      +
                       │          │              │              │  AS24940 (Hetzner, direct)
                  ┌────┴──────────┴──────────────┴────┐   ┌─────┴────────────────────────┐
                  │             hobgp (Core)          │   │       ixbgp (Edge - iFog)    │
                  │       FreeBSD + FRR, AS201379     │◄──┤  FreeBSD + FRR, AS201379     │
                  │       2a06:9801:1c::/48           │iBGP  FogIXP peering LAN          │
                  └─┬───────────┬─────────────┬───────┘   └──────────────────────────────┘
                    │           │             │
               iBGP GIF   iBGP GIF       iBGP 6to4
                    │           │             │
            ┌───────┴───────┐ ┌─┴──────────┐ ┌┴────────────────────────────────┐
            │ vtbgp (Vultr) │ │lobgp(LocIX)│ │ MikroTik (Home LAN)             │
            │ Native AS64515│ │Servperso   │ │ RouterOS 7.20, AS201379         │
            │               │ │+ LocIX RS  │ │ Announces 2a06:9801:1c:6000::/64│
            └───────────────┘ └────────────┘ └─────────────────────────────────┘

Downstream tunnels from hobgp (provider-independent IPv6 for services):
   :1000::/64  radon      (blog, DNS, Gemini, fedi comments)
   :2000::/64  colo       (Hetzner colo OPNsense)
   :3000::/64  mail       (mail.linuxserver.pro)
   :4000::/64  bb         (burningboard.net)
   :5000::/64  road       (road-warrior network)
   :6000::/64  home       (home LAN via MikroTik)

hobgp at Hetzner Nuremberg remains the hub from Part 2 - the core router that every edge speaks iBGP to, and every downstream site tunnels through. ixbgp is the new announcement point at iFog Zürich with a direct link to FogIXP Frankfurt. The home network is a downstream site on the same tunnel pattern as radon, but runs iBGP instead of static routing.

Downstream /64s ride GIF or 6to4 tunnels rather than native transport because most hosting providers don’t let customers announce foreign prefixes on their infrastructure. A tunnel gives each site a direct layer-3 path back to a router that can originate and route the prefix.

The Fourth Edge: ixbgp at iFog and FogIXP

Why a Fourth Edge

The free BGPTunnel service I used in Parts 1-3 is a great on-ramp, but it’s operated as a best-effort community service. Pairing it with a paid iFog BGP port gives me a real transit contract and something the free tunnel does not: a VM with a second NIC directly on FogIXP’s peering LAN. FogIXP is a smaller IXP than LocIX but has a different participant mix - notably, a direct Hetzner presence on that fabric.

The architecture mirrors lobgp from Part 3: two physical interfaces (internet-facing and peering-LAN-facing), an iBGP GIF tunnel back to hobgp, route-maps per peer. The main addition is the direct Hetzner session.

Network Configuration

/etc/rc.conf on ixbgp:

hostname="ixbgp"
kern_securelevel_enable="YES"
kern_securelevel="2"
kld_list="if_gif"

# Internet side (iFog)
ifconfig_vtnet0="inet 198.51.100.176 netmask 255.255.255.0 -lro -tso"
defaultrouter="198.51.100.1"
ifconfig_vtnet0_ipv6="inet6 2001:db8:1030::1229 prefixlen 48 -rxcsum6 -tso6"
ipv6_defaultrouter="2001:db8:1030::1"

# FogIXP peering LAN (direct L2)
ifconfig_vtnet1="up -lro -tso"
ifconfig_vtnet1_ipv6="inet6 2001:7f8:ca:1::20:1379:1 prefixlen 64 -rxcsum6 -tso6"

# iBGP tunnel to core
cloned_interfaces="gif0"
ifconfig_gif0="tunnel 198.51.100.176 198.51.100.10 mtu 1480"
ifconfig_gif0_ipv6="inet6 2a06:9801:1c:fffc::2 2a06:9801:1c:fffc::1 prefixlen 128"
ifconfig_gif0_descr="Uplink-to-hobgp"

ipv6_gateway_enable="YES"
ipv6_static_routes="myblock"
ipv6_route_myblock="2a06:9801:1c::/48 -interface gif0"

pf_enable="YES"
frr_enable="YES"
frr_daemons="mgmtd zebra bgpd bfdd staticd"

The peering-LAN address 2001:7f8:ca:1::20:1379:1 encodes my ASN in the interface ID (20:1379 → AS201379), which is a common convention on IXP LANs. FogIXP assigns these statically; the /64 is the peering-LAN subnet shared by every participant.

ipv6_route_myblock="2a06:9801:1c::/48 -interface gif0" is the same trick as on lobgp: any traffic arriving at ixbgp for the /48 gets forwarded through the iBGP tunnel to hobgp. ixbgp originates the route, but forwarding still happens through hobgp.

FRR Configuration

Five BGP neighbors: three FogIXP route servers, the Hetzner direct peer, the paid iFog transit, and iBGP back to the core. The excerpt below omits the full bogon and hygiene policy for brevity (see Part 1 for the complete PL-BOGONS list); only the per-peer local-preference differences matter here.

frr version 10.5.1
hostname ixbgp
!
ipv6 prefix-list PL-MY-NET seq 5 permit 2a06:9801:1c::/48
!
! [PL-BOGONS: same as Part 1, trimmed for brevity]
!
route-map RM-IXP-IN permit 10
 match ipv6 address prefix-list PL-BOGONS
 set local-preference 300
exit
!
route-map RM-HETZNER-IN permit 10
 match ipv6 address prefix-list PL-BOGONS
 set local-preference 301
exit
!
route-map RM-IFOG-IN permit 10
 match ipv6 address prefix-list PL-BOGONS
 set local-preference 100
exit
!
route-map RM-IXP-OUT permit 10
 match ipv6 address prefix-list PL-MY-NET
exit
!
route-map RM-HETZNER-OUT permit 10
 match ipv6 address prefix-list PL-MY-NET
exit
!
route-map RM-IFOG-OUT permit 10
 match ipv6 address prefix-list PL-MY-NET
exit
!
route-map RM-IBGP-OUT permit 10
exit
!
router bgp 201379
 bgp router-id 198.51.100.176
 no bgp default ipv4-unicast
 no bgp enforce-first-as
 neighbor 2001:db8:1030::1 remote-as 34927
 neighbor 2001:db8:1030::1 description Transit-iFog-Paid
 neighbor 2a06:9801:1c:fffc::1 remote-as 201379
 neighbor 2a06:9801:1c:fffc::1 description Core-Hetzner
 neighbor 2a06:9801:1c:fffc::1 update-source 2a06:9801:1c:fffc::2
 neighbor 2001:7f8:ca:1::111 remote-as 47498
 neighbor 2001:7f8:ca:1::111 description FogIXP-RS1
 neighbor 2001:7f8:ca:1::222 remote-as 47498
 neighbor 2001:7f8:ca:1::222 description FogIXP-RS2
 neighbor 2001:7f8:ca:1::333 remote-as 47498
 neighbor 2001:7f8:ca:1::333 description FogIXP-RS3
 neighbor 2001:7f8:ca:1:0:2:4940:1 remote-as 24940
 neighbor 2001:7f8:ca:1:0:2:4940:1 description Hetzner-Peer
 !
 address-family ipv6 unicast
  neighbor 2001:db8:1030::1 activate
  neighbor 2001:db8:1030::1 soft-reconfiguration inbound
  neighbor 2001:db8:1030::1 route-map RM-IFOG-IN in
  neighbor 2001:db8:1030::1 route-map RM-IFOG-OUT out
  neighbor 2a06:9801:1c:fffc::1 activate
  neighbor 2a06:9801:1c:fffc::1 next-hop-self
  neighbor 2a06:9801:1c:fffc::1 route-map RM-IBGP-OUT out
  neighbor 2001:7f8:ca:1::111 activate
  neighbor 2001:7f8:ca:1::111 soft-reconfiguration inbound
  neighbor 2001:7f8:ca:1::111 route-map RM-IXP-IN in
  neighbor 2001:7f8:ca:1::111 route-map RM-IXP-OUT out
  neighbor 2001:7f8:ca:1::222 activate
  neighbor 2001:7f8:ca:1::222 soft-reconfiguration inbound
  neighbor 2001:7f8:ca:1::222 route-map RM-IXP-IN in
  neighbor 2001:7f8:ca:1::222 route-map RM-IXP-OUT out
  neighbor 2001:7f8:ca:1::333 activate
  neighbor 2001:7f8:ca:1::333 soft-reconfiguration inbound
  neighbor 2001:7f8:ca:1::333 route-map RM-IXP-IN in
  neighbor 2001:7f8:ca:1::333 route-map RM-IXP-OUT out
  neighbor 2001:7f8:ca:1:0:2:4940:1 activate
  neighbor 2001:7f8:ca:1:0:2:4940:1 soft-reconfiguration inbound
  neighbor 2001:7f8:ca:1:0:2:4940:1 route-map RM-HETZNER-IN in
  neighbor 2001:7f8:ca:1:0:2:4940:1 route-map RM-HETZNER-OUT out
 exit-address-family

The local-preference scheme is deliberate:

Hetzner direct (LP 301) - highest. A direct session with a network the size of Hetzner is preferable to transit.
FogIXP route servers (LP 300) - next. Routes learned via the exchange are free to use and generally short.
iFog paid transit (LP 100) - lowest. Transit is the fallback, so it gets the lowest local preference.

The one-point gap is deliberate: if I learn a Hetzner route both directly and via the route servers, the direct session wins. That keeps path selection predictable and avoids relying on route-server propagation behavior.

FogIXP’s route servers are “transparent” - they don’t prepend their own AS onto forwarded routes, so the first AS in the path is the actual origin, not the route server itself. FRR checks for this by default, so route-server sessions usually need no bgp enforce-first-as.

PF: Stateless Transit, Stateful Control Plane

The PF configuration on ixbgp follows the same pattern established in Parts 2 and 3. The tunnel encapsulation must be stateless (otherwise PF’s state table interferes with asymmetric paths), while control-plane TCP sessions must be stateful (TCP needs state to track three-way handshakes). The key bits:

ext_if    = "vtnet0"
ixp_if    = "vtnet1"
hobgp_tun = "gif0"

my_network_v6 = "2a06:9801:1c::/48"
my_router_ip  = "2a06:9801:1c:fffc::2"

# Outer tunnel encapsulation: stateless
pass in  quick on $ext_if proto { 41, ipencap } from 198.51.100.10 to ($ext_if) no state
pass out quick on $ext_if proto { 41, ipencap } from ($ext_if) to 198.51.100.10 no state

# eBGP sessions: stateful
pass in quick on $ext_if proto tcp from 2001:db8:1030::1 to ($ext_if) port 179 keep state
pass in quick on $ixp_if proto tcp from 2001:7f8:ca:1::/64 to ($ixp_if) port 179 keep state

# Transit data plane: stateless (essential for DSR and inner path asymmetry)
pass in  quick inet6 from any to $my_network_v6 no state
pass out quick inet6 from any to $my_network_v6 no state
pass in  quick inet6 from $my_network_v6 to any no state
pass out quick inet6 from $my_network_v6 to any no state

# IXP best practice: don't leak multicast onto the peering LAN
block out quick on $ixp_if to { 224.0.0.0/4, 255.255.255.255, ff02::/16 }
block out quick on $ixp_if proto { igmp, ospf, pim, vrrp, gre }

# Monitoring: allow-list from trusted scrapers only
pass  in quick inet6 proto tcp from { 2a06:9801:1c:2000::21, 2a06:9801:1c:2000::25 } \
    to $my_router_ip port { 9100, 9342 } keep state
block in quick proto tcp from any to any port { 9100, 9342 }

The routing-protocol block rule is IXP-specific: OSPF hellos and the like have no business leaking onto a shared peering LAN, and most IXPs explicitly require participants to block them. The multicast block rule is the same principle - anything that’s not unicast peering traffic gets dropped before it hits the fabric.

Direct Peering with Hetzner

The Path Before

Before the FogIXP session, traffic from my home network to a Hetzner server went:

home MikroTik → hobgp → iFog BGPTunnel → iFog's upstream → DE-CIX fabric
  → AS24940 Hetzner core → target

Four transit hops after leaving my network, round-trip time in the 30-40 ms range despite Frankfurt and Nuremberg being ~180 km apart.

The Path Now

Post-FogIXP:

home MikroTik → hobgp → ixbgp (Zürich) → FogIXP fabric → Hetzner backbone → target

One exchange point, one peering session, one handoff. The mtr at the top of this article shows ~10 ms end-to-end - not because the physical distance changed, but because the routing got shorter.

What Hetzner Sees

Hetzner’s routers now learn 2a06:9801:1c::/48 directly from AS201379, with no transit AS in between. From any Hetzner-hosted machine, traffic to my prefix exits through the Hetzner-side peering session at FogIXP and lands directly on ixbgp, which forwards over iBGP to hobgp.

bgp.tools shows the AS path diversity: the prefix is still visible via iFog free, Lagrange, route64, Vultr, and the LocIX route servers, and now via FogIXP including the Hetzner direct session. Traffic originating inside Hetzner generally prefers the direct path; everyone else picks based on their own policy.

What This Buys in Practice

The practical benefit is not just lower latency to one VPS. Because the /48 is mine and announced by AS201379, every service behind it inherits the direct peering automatically.

Bringing the Home LAN into AS201379

The Gap in Parts 1-3

Parts 1-3 built an AS that serves my public-facing infrastructure. Downstream servers like radon or the colo OPNsense get their own /64 routed natively via GIF. My home network, however, used Deutsche Telekom’s provider-assigned IPv6 - a /56 from DTAG that changes on every reconnect and is useless for inbound connectivity.

The fix is conceptually identical to what radon already does: allocate a /64 from my /48, tunnel it to a router at the endpoint, announce it from there. The difference is that the home router is a MikroTik, not FreeBSD, and MikroTik’s tunnel implementation is 6to4 (interface 6to4) rather than GIF. The iBGP session plugs the home router into the existing core router.

MikroTik Configuration

RouterOS 7.20 has a usable BGP implementation. The tunnel to hobgp is a 6to4 interface (IPv6-in-IPv4, protocol 41), and iBGP runs over it:

/interface 6to4
add local-address=203.0.113.86 mtu=1480 name=tun-hobgp \
    remote-address=198.51.100.10

/ipv6 address
add address=2a06:9801:1c:fff0::2/128 advertise=no interface=tun-hobgp
add address=2a06:9801:1c:6000::1 interface=lan

/ipv6 route
add dst-address=2a06:9801:1c:fff0::1/128 gateway=tun-hobgp
add blackhole dst-address=2a06:9801:1c:6000::/64

/routing bgp instance
add as=201379 name=default router-id=203.0.113.86

/routing bgp template
add as=201379 name=ibgp-hobgp

/routing bgp connection
add afi=ipv6 disabled=no instance=default \
    local.address=2a06:9801:1c:fff0::2 .role=ibgp \
    remote.address=2a06:9801:1c:fff0::1 .as=201379 \
    name=hobgp templates=ibgp-hobgp \
    output.filter-chain=bgp-out .redistribute=connected,static

/routing filter rule
add chain=bgp-out rule="if (dst == 2a06:9801:1c:6000::/64) { accept }"
add chain=bgp-out rule=reject

Three things are worth calling out:

The blackhole route for the /64. Same idea as the -reject route on hobgp for the /48: if the MikroTik gets traffic for an address in the /64 that isn’t assigned, it drops it locally rather than following the default route back out (which would cause a loop). On RouterOS this is expressed as a specific blackhole route.

The output filter. bgp-out only permits the /64 that belongs to this site. Even though redistribute=connected,static would otherwise announce every connected subnet (including LAN RFC 1918 ranges), the filter strips everything that isn’t 2a06:9801:1c:6000::/64. This is the MikroTik equivalent of the PL-MY-NET prefix lists on the FreeBSD edges.

advertise=no on the tunnel address. The tunnel link address 2a06:9801:1c:fff0::2 is a /128 point-to-point - it should not be announced to the LAN as something hosts can SLAAC from.

The Core-Side Configuration

On hobgp, the MikroTik is a downstream iBGP peer with a different routing policy than the other edges. The other three edges receive the full table via RM-IBGP-OUT permit 10 | match PL-MY-NET - they only get our prefix, since that’s the only thing they need to advertise. The home router gets a default route and nothing else:

ipv6 prefix-list PL-MIKROTIK-NET seq 5 permit 2a06:9801:1c:6000::/64
!
route-map RM-DOWNSTREAM-IN permit 10
 match ipv6 address prefix-list PL-MIKROTIK-NET
exit
route-map RM-DOWNSTREAM-IN deny 20
exit
!
route-map RM-DEFAULT-OUT deny 10
exit
!
router bgp 201379
 ...
 neighbor 2a06:9801:1c:fff0::2 remote-as 201379
 neighbor 2a06:9801:1c:fff0::2 description Downstream-MikroTik-Lab
 neighbor 2a06:9801:1c:fff0::2 update-source 2a06:9801:1c:fff0::1
 !
 address-family ipv6 unicast
  neighbor 2a06:9801:1c:fff0::2 activate
  neighbor 2a06:9801:1c:fff0::2 default-originate
  neighbor 2a06:9801:1c:fff0::2 soft-reconfiguration inbound
  neighbor 2a06:9801:1c:fff0::2 route-map RM-DOWNSTREAM-IN in
  neighbor 2a06:9801:1c:fff0::2 route-map RM-DEFAULT-OUT out

default-originate generates a synthetic ::/0 route and sends it to the MikroTik. RM-DEFAULT-OUT deny 10 prevents any other prefix from being advertised. The inbound filter (RM-DOWNSTREAM-IN) only accepts 2a06:9801:1c:6000::/64 - a safety measure against the home router accidentally announcing something it shouldn’t. Combined with the MikroTik’s own outbound filter, this means the worst-case scenario of a misconfigured home lab is a correctly-announced /64 or nothing - never route leakage.

The result is that a PC on the home LAN now SLAAC-configures from 2a06:9801:1c:6000::/64, keeps a stable globally routable address across ISP reconnects, and exits through the AS like any other downstream site. The home LAN is now just another site in AS201379, indistinguishable from radon’s /64 as far as the rest of the internet is concerned.

Traffic Engineering: Steering DTAG via Vultr

The Observation

Deutsche Telekom (AS3320) is the largest German eyeball network. A large share of blog readers reach my services over DTAG’s network, so the return path matters.

With four edge routers, I have four potential ways to reach DTAG. Watching the AS paths that show up on bgp.tools and in show ipv6 bgp on hobgp, the Vultr edge consistently learned shorter paths to DTAG than the others - Vultr’s upstream connectivity into DTAG is short and direct in ways that the transit providers aren’t.

The Implementation

The tool is a BGP AS-path access list combined with a route-map that matches on both the AS-path and the learning peer. On hobgp:

bgp as-path access-list AS-DTAG seq 5 permit _3320$

route-map RM-IBGP-IN permit 5
 match as-path AS-DTAG
 match peer 2a06:9801:1c:fffe::2
 set local-preference 260
exit
route-map RM-IBGP-IN permit 10
exit

Reading this top to bottom:

bgp as-path access-list AS-DTAG matches any AS path that ends in 3320 (the $ anchors to the end of the AS path). That means “routes originated by DTAG” - DTAG is the terminal AS, not a transit.
route-map RM-IBGP-IN permit 5 has two match conditions: the path must match AS-DTAG, and the update must have come from 2a06:9801:1c:fffe::2 - the Vultr edge router. Both must be true.
Matching routes get local-preference 260. Higher local-pref wins in BGP’s selection algorithm, so these routes are preferred over the same prefix learned via iFog, Lagrange, route64, LocIX, or FogIXP direct.
route-map RM-IBGP-IN permit 10 is the catch-all - any route that didn’t match clause 5 falls through and gets accepted with default attributes.

The match peer clause is essential. Without it, the rule would match routes to DTAG learned from any iBGP peer, which isn’t what I want. Vultr happens to have the short path; the other edges don’t. The policy is specifically “prefer Vultr’s version of DTAG routes, if it has one.”

The Effect

show ipv6 bgp X:X:X::/32 for a DTAG-customer prefix shows multiple paths - from iFog, LocIX, Vultr, and so on. Before the route-map, FRR’s tiebreaker (AS path length, then MED, then BGP origin) picked whichever was shortest at that moment. After the route-map, if Vultr has a path for a DTAG destination, Vultr wins - because 260 > 100 (default).

Inbound traffic is out of my control (that’s determined by the announcing side), but outbound traffic from my servers to DTAG customers now consistently exits through Vultr Frankfurt. Latency improvement varies by destination but averages around 5 ms lower than the transit paths, and jitter drops because the path no longer changes based on transient BGP convergence.

This kind of traffic engineering was technically possible before, but not especially useful - with one or two edges, there’s nothing to steer between. With four, “which path goes where” becomes a legitimate question, and AS-path regex gives a precise answer.

Hub Hygiene: One IP for Traceroutes

hobgp has many interfaces. When a packet transits the router, the source address used for any router-originated reply (ICMP time-exceeded from a traceroute, for instance) depends on which interface is closest in the routing table. The result was that hobgp appeared under different addresses in mtr output depending on the flow direction.

The fix is a PF match rule that rewrites router-originated IPv6 traffic to a stable loopback alias from the /48:

ifconfig_lo0_alias0="inet6 2a06:9801:1c::1 prefixlen 64"

match out on { gre0, gre1, gre2, gif0, gif1, gif2, gif3, gif4, gif5, gif6 } \
    inet6 from 2001:db8:1c19:3ee1::1 to any nat-to 2a06:9801:1c::1

match is a non-terminating PF action - it modifies packets without making a pass/block decision. Now traceroutes through hobgp always show 2a06:9801:1c::1 (which resolves to hofstede.it) regardless of which tunnel the forward packet took, and the Hetzner-assigned address stays off public traceroutes.

Downstream Sites: PI Addressing for Friends and Services

A side effect of having a /48 and a working BGP infrastructure is that you can offer stable, provider-independent IPv6 addressing to services you run for friends or that live in separate failure domains. The current allocation:

Prefix	Site	Purpose
`2a06:9801:1c::/64`	loopback on hobgp	router identity
`2a06:9801:1c:1000::/64`	radon	blog, DNS, Gemini, fediverse comments
`2a06:9801:1c:2000::/64`	colo OPNsense	Hetzner colo test network
`2a06:9801:1c:3000::/64`	mail.linuxserver.pro	mail server
`2a06:9801:1c:4000::/64`	burningboard.net	Mastodon instance
`2a06:9801:1c:5000::/64`	road-warrior	laptop on the move
`2a06:9801:1c:6000::/64`	home LAN	residential network
`2a06:9801:1c:fff*::/64`	infra	point-to-point tunnel link addresses

The high :fff* ranges are reserved for infrastructure and point-to-point links: :ffff::/64 for the radon GIF, :fffe::/64 for the Vultr iBGP link, :fffd::/64 for the LocIX edge, :fffc::/64 for the FogIXP edge, :fff0::/64 for the MikroTik. Keeping link addresses out of the customer-facing ranges means the useful space (:1000:: through :6000::) is contiguous and easy to reason about.

Every site runs its own services, has its own firewall, and is operationally independent. What they share is an address prefix that doesn’t change if their hoster does - and that, more than anything else, is why I did all of this.

Lessons Learned

Direct peering is worth the setup effort. A peering session with a large network replaces an entire transit path with a single handoff. The operational cost is one more BGP neighbor and a handful of route-map entries; the benefit is shorter, more predictable paths for a large chunk of real traffic.

RouterOS BGP is good enough for a downstream site. The MikroTik iBGP session has been stable since it was brought up. The configuration is verbose compared to FRR but not surprising - prefix-list equivalents, route-maps, the blackhole for the /64. If the home router is what you already have, BGP on it is a reasonable answer.

Two-condition route-maps are a precision instrument. match as-path plus match peer lets you say exactly what you mean: “prefer this specific path for this specific destination class.” It’s conceptually a small jump from single-condition filters, but it unlocks meaningful traffic engineering the moment you have more than two edges.

A consistent router identity helps more than it should. Rewriting hobgp‘s source address to the /48 loopback is five lines of PF, but having a single hostname appear for the hub in every traceroute - regardless of interface - makes debugging paths significantly easier. It also keeps provider addresses out of public traceroutes.

The “announce at the edge, forward at the core” pattern scales. Each new edge is the same recipe: two interfaces or one plus a tunnel, FRR with per-peer route-maps, PF with stateless transit rules, an iBGP tunnel to hobgp. The core router does the work. Adding a fourth edge was faster than adding the second one, because the pattern was already established.

Interested in peering with AS201379? More details about the network, including upstreams, IXPs, and policy, live at hofstede.it/as201379.html.

If you’d like to establish a peering session - either over a tunnel or on the shared fabric at LocIX Düsseldorf or FogIXP Frankfurt - send a mail to peering@hofstede.it. Hobbyists, small ASNs, and experimental setups are explicitly welcome.

Conclusion

Four months in, AS201379 has started to feel less like a project and more like a very small ISP: a core router, multiple edges in different PoPs, direct peering, traffic engineering, and downstream sites with stable addressing. It is still, by any real metric, tiny. But the difference between “tiny” and “zero” is categorical - Hetzner now learns my prefix directly over a peering session, and a PC on my home LAN now introduces itself to the internet with an address from my own prefix.

Whether there is a Part 5 depends on what breaks, what improves, or what new opportunity shows up next. So far, every version of this network has felt complete right up until the moment it wasn’t.

References

FogIXP - Frankfurt Open Exchange - the IXP used for this expansion
LocIX Düsseldorf - the IXP from Part 3
bgp.tools - looking glass and AS-path visibility
FRR Documentation: BGP - route-maps, AS-path access-lists, match peer
MikroTik RouterOS BGP - filter chains, connection templates
MANRS Routing Security - the filtering and peering-LAN hygiene principles applied on every edge
PeeringDB - where FogIXP and LocIX peering-LAN addresses come from

BGP is one of those protocols where you never really finish. There’s always one more peer to add, one more route-map clause to tune, one more site to bring into the fabric. The satisfying part is that each addition compounds: every new edge improves paths for every existing service, and every new downstream inherits the entire backbone’s routing decisions for free. Four edges and six downstream sites ago, this was one router and one VPS. The architecture was the hard part. Everything since has been variations on the same theme.

Automating FreeBSD Jails with cdist - Zero Dependencies Inside the Jail

2026-04-12T00:00:00+02:00

Most configuration management tools still assume they own the target. Ansible ships Python modules over the wire and runs them in place. Salt wants a minion on every host. Chef wants a client, Puppet wants an agent.

Jails break that assumption in a satisfying way. A FreeBSD jail is supposed to be small - sometimes a single static binary, an rc.d script, and a few lines in rc.conf. Installing Python into every jail just so Ansible can run its setup module is, to borrow a phrase, the tail wagging the dog. I already wrote about a workaround for Ansible: the jailexec connection plugin, which SSHes to the jail host and uses jexec to tunnel commands into each jail. It works, and I still reach for it when I already have an Ansible setup. But many Ansible modules assume Python on the target, so in practice some of those jails still end up growing a Python interpreter.

Then I tried cdist, and everything got smaller.

Table of Contents
What cdist Actually Is
The Two Hooks That Make This Interesting
jexec-ssh.py
jexec-scp.py
Actually Running It
- A More Practical Example: A Custom Maintenance Type
Why This Is The Unix Way
When To Still Reach For Ansible
Getting The Scripts
References

What cdist Actually Is

cdist is a configuration management system written in Python that runs entirely on the control machine. On the target, it expects a Bourne-style sh environment. That is it. No agent, no client, no target-side runtime of any kind. cdist assembles a shell script locally, ships it over the wire, and watches stdout come back. The target does not know it is being managed any more than it would know it was being SSHed into.

That minimalism is the selling point. cdist’s own tagline reads “usable configuration management”, but what I read between the lines is: the entire target-side runtime is whatever shell is already on the box. If you can ssh to it, you can cdist it. No bootstrap step, no package install, no “please add our apt key first”. For a FreeBSD jail that exists precisely to run one daemon and nothing else, that is perfect.

The model is also refreshingly simple conceptually. You write manifests (declarative entry points) that invoke types (reusable building blocks: __file, __package_pkgng, __service, __line, and so on). Every type is itself a small directory of shell scripts - an explorer that reports current state, a gencode-remote that emits the shell commands to reach the desired state, and optional helpers. cdist runs the explorers on the target, generates a shell script locally, sends it over ssh, and runs it with sh -e. That is the whole loop.

The Two Hooks That Make This Interesting

cdist’s default transport is OpenSSH: ssh user@host sh -c '…' for commands, scp for files. But the transport is swappable. Two command-line flags override it:

--remote-exec PATH - a script that takes ssh-style options followed by <target> <command…> and runs the command on the target somehow.
--remote-copy PATH - a script that takes scp-style arguments and copies files to the target somehow.

In this setup, the cdist target name is not a hostname at all; it is the jail name. JAIL_HOST tells the wrappers which real SSH endpoint to use.

The API is dumb in the best way. cdist gives you the target name and a command line; the script does whatever it needs to. If you want to tunnel through a bastion, go ahead. If you want to talk to a switch over a serial console, go ahead. And if you want to SSH to a FreeBSD host and drop into a jail with jexec - which is exactly what I want - that is two small Python scripts. The only persistent requirement is host-side access to ssh, scp, jexec, and jls; the jail itself remains untouched except for the configuration changes cdist applies.

jexec-ssh.py

Here is the remote-exec wrapper in full. It is 60 lines, most of them boilerplate:

#!/usr/bin/env python3
import os
import shlex
import sys

_SSH_OPTS_WITH_VALUE = frozenset(
    ("-o", "-F", "-i", "-l", "-p", "-c", "-m", "-L")
)


def strip_ssh_options(args):
    i = 0
    while i < len(args):
        arg = args[i]
        if arg in _SSH_OPTS_WITH_VALUE:
            i += 2
        elif arg.startswith("-"):
            i += 1
        else:
            return i
    return i


def main():
    jail_host = os.environ.get("JAIL_HOST")
    if not jail_host:
        sys.exit("error: JAIL_HOST is not set")

    jail_user = os.environ.get("JAIL_USER", "root")

    args = sys.argv[1:]
    start = strip_ssh_options(args)
    if start >= len(args):
        sys.exit("error: no target jail name on the command line")

    jailname = args[start]
    remote_payload = " ".join(args[start + 1:])

    jexec_cmd = (
        f"jexec -u {shlex.quote(jail_user)} {shlex.quote(jailname)} "
        f"/bin/sh -c {shlex.quote(remote_payload)}"
    )

    os.execvp("ssh", ["ssh", jail_host, jexec_cmd])


if __name__ == "__main__":
    main()

The logic is the whole story:

Read JAIL_HOST from the environment. That is the ssh endpoint for the host, not the jail.
Strip the ssh-style options cdist prepends - things like -o User=root - which are meaningful to a jail’s (nonexistent) direct ssh endpoint, not to ours. Real connection options belong in ~/.ssh/config for the host.
The first non-option argument is the jail name. Everything after that is the command cdist wants to execute.
Wrap the command in jexec -u <user> <jail> /bin/sh -c '…', shlex.quote‘d so pipelines, redirections and globs survive transit through two shells.
execvp into the real ssh, replacing the Python process. ssh does the heavy lifting, and its exit code flows straight back to cdist.

The command is crossing two shell boundaries: the host-side SSH command line and the in-jail sh -c. That is why the wrapper leans so hard on shlex.quote(). The /bin/sh -c wrapper itself is not cosmetic. Without it, jexec hands the argv directly to whatever binary sits at position one, and cdist-generated constructs like for f in /etc/*.conf; do …; done die the moment a glob shows up. Running the payload through a fresh sh inside the jail restores the shell environment cdist expects.

jexec-scp.py

File transfer is slightly more involved, but only slightly. scp does not know what a jail is - but it does know what a host path is, and a jail’s “inside” is always some path on the host (/zroot/jails/<name> is typical). We can translate.

#!/usr/bin/env python3
import os
import subprocess
import sys

_SCP_OPTS_WITH_VALUE = frozenset(("-o", "-F", "-i", "-S", "-P"))


def split_args(args):
    opts, paths = [], []
    i = 0
    while i < len(args):
        arg = args[i]
        if arg in _SCP_OPTS_WITH_VALUE and i + 1 < len(args):
            opts.extend((arg, args[i + 1]))
            i += 2
        elif arg.startswith("-"):
            opts.append(arg)
            i += 1
        else:
            paths.append(arg)
            i += 1
    return opts, paths


def jail_root(host, jailname):
    out = subprocess.check_output(
        ["ssh", host, "/usr/sbin/jls", "-j", jailname, "path"],
        text=True,
        stderr=subprocess.STDOUT,
    )
    return out.strip()


def rewrite(path, host):
    if ":" not in path:
        return path
    prefix, rpath = path.split(":", 1)
    if "/" in prefix or not prefix:
        return path
    root = jail_root(host, prefix)
    return f"{host}:{os.path.join(root, rpath.lstrip('/'))}"


def main():
    jail_host = os.environ.get("JAIL_HOST")
    if not jail_host:
        sys.exit("error: JAIL_HOST is not set")

    opts, paths = split_args(sys.argv[1:])
    if len(paths) < 2:
        sys.exit("error: scp needs a source and a destination")

    translated = [rewrite(p, jail_host) for p in paths]
    os.execvp("scp", ["scp", *opts, "-r", "-p", *translated])


if __name__ == "__main__":
    main()

Walking through it:

For each argument, check if it looks like <jail>:/some/path. If not - local paths, host-qualified paths - leave it alone.
If it does, ssh to the host and run jls -j <jail> path. jls is the canonical way to ask FreeBSD “where does jail X live on disk right now”.
Rewrite the argument to <JAIL_HOST>:<jailroot>/<path>, which is just a perfectly ordinary scp remote path.
execvp into the system scp with the rewritten arguments.

There is nothing magical here. scp was already perfectly capable of copying files to root@radon.example.com:/zroot/jails/testvnet/etc/motd. All the wrapper does is save the user the trouble of typing it and let cdist’s file types behave as if the jail were a first-class ssh target. The simple implementation asks jls once per jail-qualified path; caching jail roots would be an easy optimization if you start moving lots of files.

Actually Running It

On a host with many jails, SSH connection setup can dominate runtime; enabling multiplexing on the host connection makes a dramatic difference. More on that in the repo’s README.

A minimal setup. First, a manifest at ./conf/manifest/init - one line is enough to prove the whole pipeline works end to end:

__file /tmp/cdist-wrapper-success --state present

Then point cdist at the two wrappers:

~/tmp/cdist-test ❯ export JAIL_HOST=root@radon.edelga.se
~/tmp/cdist-test ❯ cdist config -c ./conf \
    --remote-exec ./jexec-ssh.py \
    --remote-copy ./jexec-scp.py \
    testvnet lg

In this example, testvnet and lg are jail names, not DNS names or SSH inventory entries. cdist is quiet on success: no output is good output. A sanity check on the host itself confirms it:

[root@radon ~]# jexec testvnet ls -l /tmp/cdist-wrapper-success
-rw-------  1 root wheel 0 Apr 11 21:02 /tmp/cdist-wrapper-success
[root@radon ~]# jexec lg ls -l /tmp/cdist-wrapper-success
-rw-------  1 root wheel 0 Apr 11 21:18 /tmp/cdist-wrapper-success

Two jails, one cdist invocation, zero daemons, zero extra interpreters, nothing mutated outside the jails themselves. Replace the one-liner manifest with real cdist types - __package_pkgng, __file, __line, __service, __user - and you get the same thing at whatever scale you need.

A More Practical Example: A Custom Maintenance Type

The smoke test above proves the transport works. Here is a pragmatic host-maintenance example I use in practice: a custom type that runs base and package updates inside a jail. This is more of a maintenance action than a pure convergent state type, but it is a good illustration of how little machinery cdist needs when you do want to run shell-driven lifecycle tasks.

First, create the type’s directory and its gencode-remote script. This is the shell that cdist will execute inside the jail:

mkdir -p conf/type/__freebsd_system_update

#!/bin/sh
# conf/type/__freebsd_system_update/gencode-remote

set -e

echo "Checking for and installing FreeBSD base updates..."
# PAGER=cat prevents freebsd-update from dropping into an
# interactive pager and waiting for 'q' on a headless target.
env PAGER=cat freebsd-update fetch install

echo "Updating package catalogue and installing upgrades..."
pkg update
pkg upgrade -y

chmod +x conf/type/__freebsd_system_update/gencode-remote

Then wire it into the manifest. cdist exposes $__target_os as an explorer, so the type only fires on FreeBSD targets:

#!/bin/sh
# conf/manifest/init

if [ "$__target_os" = "FreeBSD" ]; then
    __freebsd_system_update
else
    echo "Info: $__target_host is not FreeBSD, skipping updates."
fi

Run it the same way as before:

~/tmp/cdist-test ❯ cdist config -c ./conf \
    --remote-exec ./jexec-ssh.py \
    --remote-copy ./jexec-scp.py \
    testvnet lg

That is the whole workflow for writing a custom cdist type: a directory, a shell script, and a one-line invocation in the manifest. No YAML, no plugin protocol, no class hierarchy. The type is just shell, and the wrappers make sure that shell runs inside the jail.

Why This Is The Unix Way

I wrote these scripts one Monday afternoon. By dinnertime I had working state management across every jail on my FreeBSD hosts, without installing anything new on a single one of them. That is not a coincidence; it is what happens when every component in the stack knows exactly one job and does it well.

Count the actors:

ssh - authenticated, encrypted transport
scp - file copy over ssh
jexec - process execution inside a running jail’s namespace
jls - jail inspection
sh - shell execution
cdist - render desired state into a shell script

Each piece is narrow, composable, and independently useful. The two Python files in this repo do not replace any of them; they only connect them. Together that is roughly 120 lines of Python orchestrating several decades of battle-tested Unix primitives. None of those tools are tightly coupled to each other. None of them share a runtime. If I delete any one file from the pipeline, the rest still do their jobs. That is the thing people mean when they talk about “the Unix philosophy”, and it is the thing I keep coming back to every time I look at how modern tools solve the same problems.

Compare the dependency footprint inside the jail:

System	Typically required inside the jail
Ansible (default)	Python interpreter for many modules, plus a normal remote access path
Ansible (jailexec)	Python interpreter (for most modules)
Salt	`salt-minion`, Python, dependencies
Puppet	`puppet` agent, Ruby, dependencies
Chef	`chef-client`, Ruby, dependencies
cdist + these wrappers	POSIX `sh`, which is already in base

The same story plays out in the implementations themselves. My Ansible jailexec connection plugin is roughly 960 lines of Python, because Ansible’s connection plugin API asks for a fair amount of scaffolding: argument translation, a file-transfer state machine, subprocess management, the whole plugin protocol. These two cdist wrappers together come to about 120 lines. That difference is not about quality; it is about how much each tool asks of the transport layer. cdist’s hooks are deliberately narrow, so there is less to implement. And because cdist’s types are shell, the jail does not need a Python interpreter at all, whereas the Ansible path still typically does.

The smallest running jail on my radon.edelga.se host is 61 MB on disk - a thin-jail nullfs view over a shared base, running one static binary. It still configures beautifully with this pipeline, because the pipeline asks it for nothing it was not already providing.

When To Still Reach For Ansible

This is not the universal answer.

cdist is opinionated about shell, which means that if your configuration domain is “take this complicated JSON API, reason about its responses, compute a delta, push it back” - the kind of thing Ansible modules do in a page of Python each - you will end up writing that logic yourself in sh. That is fine for ten types; it is painful for a hundred. The moment you need rich remote introspection, structured data handling, or an ecosystem module that already solves a weird API, Ansible starts to earn its weight again.

If you already have a working Ansible setup across your infrastructure, the jailexec connection plugin is the path of least resistance for adding jails to it. You keep your playbooks, your roles, your inventory. I still use it where “use what’s already there” beats “be minimal”.

But for a fresh build of jails on a new host - or for the handful of single-purpose jails I run on radon - cdist plus these two wrappers is the right tool. It makes “managed state on a FreeBSD jail” exactly as simple as “I can ssh to the host and run jexec”, and it asks nothing more of the jail than FreeBSD already provides.

Getting The Scripts

Both scripts live on Codeberg as jexec-cdist, CC0 licensed, with a README that walks through the environment variables, the ssh multiplexing tip that turns a 15-second run into an instant one, the doas pattern for running as a non-root user on the host, and the handful of edge cases (nullfs unions, non-standard jail roots) where the path rewriting needs a small nudge.

Clone, chmod +x, point cdist at them, export JAIL_HOST. That is the entire onboarding. The jails stay as clean as the day you created them - and that is the thing I wanted all along.

References

cdist - the configuration management system this article is about
jexec-cdist on Codeberg - the two wrappers in this article
cdist: writing your own remote-exec and remote-copy - the upstream documentation for the hooks these scripts plug into
Managing FreeBSD Jails with Ansible: the jailexec connection plugin - the Ansible equivalent, for comparison
FreeBSD Foundationals: Jails - the jail fundamentals this article assumes
jexec(8), jls(8) - the FreeBSD primitives doing the actual work

Replacing Lenovo’s WWAN Unlock Blob with a 100-Line Bash Script

2026-04-11T00:00:00+02:00

My Lenovo ThinkPad T14s Gen 4 (AMD Ryzen) shipped with an Intel XMM7560 LTE Advanced Pro modem soldered to the mainboard. Useful little thing: real LTE on the go, no tethering dance, no MiFi puck in the bag. The catch: out of the box, the modem refuses to register on the network. ModemManager dutifully detects it, the SIM is recognised, but on my machine AT+CFUN=1 would come back OK while the radio quietly stayed dark.

The reason is something called FCC lock, and the official fix from Lenovo is a package of proprietary helpers and shared libraries. I replaced Lenovo’s proprietary helper with a bash script that performs the same handshake in clear, auditable shell. Here is the why, the how, and the script.

Table of Contents
What FCC Lock Actually Is
The ModemManager FCC Unlock Mechanism
Lenovo’s Solution: A Proprietary Helper Package
The Alternative: A Shell Script That Does the Same Thing
How the Unlock Actually Works
Installing It Yourself
A Disclaimer About Jurisdiction
Wrapping Up
References

What FCC Lock Actually Is

If you have never run into this before, the term “FCC lock” sounds like DRM. It is not - or at least, not exactly. It is a regulatory compliance mechanism.

When a laptop ships with an embedded WWAN modem, the combination of the laptop chassis, the antenna layout, and the modem radio is what gets certified by the regulator. In the US that regulator is the FCC, and the relevant rule is the well-known 47 CFR Part 15 / Part 22-27 framework: a radio module is certified for a specific host device, with a specific antenna gain, in a specific physical configuration. If you yank the modem out of one laptop and stick it into another, the certification no longer applies.

To enforce this, modem vendors ship the radio in a state where the RF subsystem is administratively disabled until the host system performs a vendor-defined unlock handshake. The handshake is meant to be a challenge-response proof that the modem is sitting in a host the OEM has already certified together with the modem. Until it succeeds, attempts to fully enable the modem fail - upstream documentation talks about the enable step erroring out, and on my XMM7560 specifically the symptom was that AT+CFUN=1 came back OK while the radio never actually came up on the network.

In practice, it behaves like an OEM lock justified in regulatory terms. The same XMM7560 family has shown up across multiple OEM laptops, with different vendors using different unlock material. The radio is identical across hosts; what changes is the OEM-controlled signature that gates its use.

On Windows this is invisible because the Lenovo / Dell / HP driver stack performs the handshake at boot. On Linux the responsibility falls to ModemManager, which provides the unlock mechanism and even ships some upstream helper scripts itself - but, since version 1.18.4, leaves activation to either the user or a vendor package rather than running anything by default.

The ModemManager FCC Unlock Mechanism

ModemManager’s design here is sensible, and it is worth understanding properly because it is more nuanced than a single search path. There are three directories involved, each with a distinct role:

${libdir}/ModemManager/fcc-unlock.d/ (e.g. /usr/lib64/ModemManager/fcc-unlock.d/ on Fedora) - this is where vendor-shipped, third-party unlock tools live. If Lenovo packages their helper for your distribution, this is where it goes.
/usr/share/ModemManager/fcc-unlock.available.d/ - this is where ModemManager itself ships a small library of unlock scripts upstream. They are present on the system but not enabled.
/etc/ModemManager/fcc-unlock.d/ - this is where you, the system administrator, opt in to one of the available scripts (or to your own) by symlinking it under the appropriate VID:PID name.

In all three locations the file is named after the modem’s USB or PCI VID:PID. For my XMM7560 the relevant name is 8086:7560 - Intel’s vendor ID, the modem’s product ID. ModemManager invokes the helper with the modem’s DBus path and the names of its control ports as arguments, and waits for it to exit successfully. If the helper returns 0, ModemManager continues bringing the modem up; if not, the modem stays disabled.

Two things are worth noting about this layout. First, the split between /etc and the upstream fcc-unlock.available.d directory exists precisely so that activation is an explicit decision, not something that happens by default - that change landed in ModemManager 1.18.4. Second, the helper can be any executable. A binary, a Python script, a shell script - ModemManager doesn’t care. That is the seam I used.

Lenovo’s Solution: A Proprietary Helper Package

Lenovo’s answer to this is lenovo-wwan-unlock, a GitHub-distributed package of proprietary helpers and shared libraries with setup scripts, SELinux policy and release packaging for several modules and systems. The README itself describes the project as “FCC and DPR unlock for Lenovo PCs”, which is worth pausing on: it is not just one unlock primitive. On supported systems Lenovo’s tooling also covers DPR (Dynamic Power Reduction) / SAR-related behaviour, the mechanism by which the laptop tells the modem to back off transmit power based on proximity sensors. So a vendor helper in this space can do more than flip a single lock bit.

You install the package, it drops the right files into the right places, and it works. It is also:

Closed-source, with no way to audit what the helpers actually do to your modem
Linked against specific runtime libraries, so on slightly newer or older distributions it can break in surprising ways
The only path the vendor offers, on a free operating system, to use a piece of hardware you already paid for

The whole point of running Fedora on a ThinkPad is that I get to look at, modify, and trust every component of the user-space stack. Even if the proprietary helper is benign, accepting it as the permanent answer to “how do I use my modem” felt like the wrong default. So I went looking for an alternative.

The Alternative: A Shell Script That Does the Same Thing

The alternative turned out to be sitting in a ModemManager merge request, posted by Floris Stoica-Marcu. It is a single bash script, about 100 lines, CC0-licensed, that performs the entire XMM7560 unlock handshake using only bash, grep, awk, printf, xxd and sha256sum. No compiled code, no hidden state, no vendor SDK.

I want to be very clear about credit: I did not write this script. Floris did. What I did was take their script, drop it into /opt/unlocker/8086, and symlink it from ModemManager’s helper directory:

~ ❯ ls -lah /usr/lib64/ModemManager/fcc-unlock.d/
Permissions Size User Date Modified Name
drwxr-xr-x@    - root 11 Apr 12:14  .
drwxr-xr-x@    - root 11 Apr 12:14  ..
lrwxrwxrwx@    - root 11 Apr 12:14  8086:7560 -> /opt/unlocker/8086

That is the entire installation. Restart ModemManager.service, plug in the SIM, and watch the journal:

Apr 11 12:15:13 neochristop ModemManager[9628]: [modem0] state changed (enabling -> enabled)
Apr 11 12:15:15 neochristop ModemManager[9628]: [modem0] 3GPP registration state changed (registering -> home)
Apr 11 12:15:15 neochristop ModemManager[9628]: [modem0] state changed (enabled -> registered)
Apr 11 12:15:16 neochristop ModemManager[9628]: [modem0] state changed (connecting -> connected)
Apr 11 12:15:16 neochristop ModemManager[9628]: [modem0] simple connect state (10/10): all done

And the interface itself, with a real public IPv6 from my carrier:

~ ❯ ifconfig wwan0
wwan0: flags=209<UP,POINTOPOINT,RUNNING,NOARP>  mtu 1500
        inet 10.127.157.102  netmask 255.255.255.0  destination 10.127.157.102
        inet6 fe80::b2ab:8cc1:6305:c055  prefixlen 64  scopeid 0x20<link>
        inet6 2a02:3037:48d:1b55:f07e:922f:8154:e65  prefixlen 64  scopeid 0x0<global>
        RX packets 199131  bytes 251330013 (239.6 MiB)
        TX packets 52690  bytes 4544146 (4.3 MiB)

That is 240 MiB of traffic over LTE through an entirely auditable user-space path.

How the Unlock Actually Works

Here is the part that matters for anyone who wants to understand what they are running. The Intel XMM7560 unlock is a challenge/response handshake using a small set of vendor AT commands. Walking through the script in order:

1. Find the AT control port. ModemManager passes the modem’s control port names as arguments. The modem exposes both MBIM and AT ports; the AT port is what we need. The script picks it by reading /sys/class/wwan/<port>/type (Linux 5.14+) or by name pattern (older kernels):

for PORT in "$@"; do
  grep -q AT "/sys/class/wwan/$PORT/type" 2>/dev/null && {
    AT_PORT=$PORT
    break
  }
done

The chosen port is something like wwan0at0, which is then opened as /dev/wwan0at0.

2. Talk AT over a file descriptor. Bash can open a character device on a numbered file descriptor with exec, write a command, and read the reply. There is no socat, no chat, no helper - just bash redirection. The script wraps that in a small at_command function:

at_command() {
    exec 99<>"$DEVICE"
    echo -e "$1\r" >&99
    read answer <&99
    read answer <&99
    echo "$answer"
    exec 99>&-
}

The double read is there because the modem echoes the command on the first line and replies on the second. Crude, but it works on this modem.

3. Request a challenge. The XMM7560 implements a vendor command, AT+GTFCCLOCKGEN, that returns a fresh 32-bit pseudo-random nonce. This is the “challenge” half of the handshake:

RAW_CHALLENGE=$(at_command "at+gtfcclockgen")
CHALLENGE=$(echo "$RAW_CHALLENGE" | grep -o '0x[0-9a-fA-F]\+' | awk '{print $1}')

4. Compute the response. This is the only “magic” step, and it is not really magic at all. The response is a SHA-256 hash of the challenge concatenated with a fixed per-vendor secret, with some byte-order shuffling on each end. The Lenovo per-vendor secret in this script is the constant bb23be7f:

VENDOR_ID_HASH="bb23be7f"

HEX_CHALLENGE=$(printf "%08x" "$CHALLENGE")
REVERSE_HEX_CHALLENGE=$(reverseWithLittleEndian "${HEX_CHALLENGE}")
COMBINED_CHALLENGE="${REVERSE_HEX_CHALLENGE}${VENDOR_ID_HASH}"
RESPONSE_HASH=$(printf "%s" "$COMBINED_CHALLENGE" | xxd -r -p | sha256sum | cut -d ' ' -f 1)
TRUNCATED_RESPONSE=$(printf "%.8s" "${RESPONSE_HASH}")
REVERSED_RESPONSE=$(reverseWithLittleEndian "$TRUNCATED_RESPONSE")
RESPONSE=$(printf "%d" "0x${REVERSED_RESPONSE}")

In plain English:

Take the challenge as a 32-bit value, write it as 8 hex chars in little-endian byte order.
Append the Lenovo vendor secret (bb23be7f).
Decode the resulting hex string into raw bytes (xxd -r -p).
Compute SHA-256 over those bytes.
Take the first 4 bytes of the digest, swap to little-endian, and convert to a decimal integer.

That decimal integer is the response.

This is the entire secret embedded in Lenovo’s helper: a four-byte constant and the byte-order conventions around it. There is no hardware-bound key, no TPM-sealed secret, no per-laptop derivation - just a baked-in constant that someone reverse-engineered out of the Lenovo helper years ago. The XMM7560 hardware does not actually verify that host certified it; it verifies that whoever is talking to it knows the constant.

That is why I am comfortable running this script. I am not defeating any hardware-bound secret here; I am reproducing the same host-side response the proprietary helper would generate, in shell instead of machine code.

5. Send the response and unlock the radio. Three more AT commands:

at_command "at+gtfcclockver=$RESPONSE"      # verify the response
at_command "at+gtfcclockmodeunlock"          # actually flip the lock bit
at_command "at+cfun=1"                       # bring the radio fully online
at_command "at+gtfcclockstate"               # read the current lock state

If the final at+gtfcclockstate reports OK or 1, the modem is unlocked and the script exits 0. ModemManager picks up from there, runs its normal enable sequence, and the modem registers on the network. If the response is wrong, the modem responds with ERROR and the script retries up to nine times with a 0.5 second backoff.

That is the entire handshake. About 100 lines of bash, no compiled dependencies, every step inspectable.

Installing It Yourself

If you have the same modem - check with lspci | grep XMM for XMM7560 LTE Advanced Pro - the steps to swap the proprietary helper for this script are short. One caveat first about paths. On Fedora, Lenovo’s package and some third-party tooling may use /usr/lib64/ModemManager/fcc-unlock.d/, but the upstream user-facing path for manually enabled scripts is /etc/ModemManager/fcc-unlock.d/. The example below uses the upstream user path.

sudo install -d /opt/unlocker
sudo install -m 0755 8086 /opt/unlocker/8086
sudo ln -s /opt/unlocker/8086 /etc/ModemManager/fcc-unlock.d/8086:7560
sudo systemctl restart ModemManager.service

Then watch journalctl -u ModemManager -f while you bring the connection up in NetworkManager. You should see the modem walk through enabling -> enabled -> registered -> connecting -> connected and end up with an interface like wwan0 carrying real traffic.

If you want to see exactly what the script does on your machine, set FCC_UNLOCK_DEBUG_LOG=1 in the environment ModemManager runs in, and the script will append every step (including the AT exchanges) to /var/log/mm-xmm7560-fcc.log. That is a great way to convince yourself the script is doing nothing but the handshake described above.

If you have a different modem - a Sierra Wireless EM7565, a Quectel EM160, a different Intel chip - the same overall pattern (drop a helper into fcc-unlock.d named after the VID:PID) applies, but the AT command set and the secret are different. Don’t blindly use this script on hardware it wasn’t written for.

A Disclaimer About Jurisdiction

I have to write this part carefully, because the legal picture here is genuinely fuzzy, varies by country, and I am not a lawyer.

The FCC unlock mechanism exists because regulators in many jurisdictions require radio equipment to be certified as part of a specific host device. In the US that is the FCC’s modular and limited modular approval rules. In the EU the RED directive (2014/53/EU) imposes broadly similar obligations on the manufacturer. Other countries have their own equivalents. The OEM - in this case Lenovo - is the entity that holds the certification, and the unlock procedure exists so that the OEM can attest to the modem that “yes, I am still the host you were certified with.”

When you replace the OEM’s unlock helper with your own, you are bypassing that attestation. The modem does not become illegal hardware, and the radio characteristics do not change one bit - the script flips exactly the same lock bit the proprietary helper would have - but you may, depending on where you live and how strictly the rule is read, be operating a radio in a configuration the regulator did not formally bless. In some jurisdictions that is a non-issue for an end user modifying their own equipment for personal use. In others it might not be.

To be specific about the risks I am personally aware of:

United States (FCC): US equipment authorization rules are not especially sympathetic to modified certified equipment, even though the direct compliance burden usually falls on manufacturers and grantees rather than on end users.
European Union (RED): the user is generally not the addressee of the directive, but a modification that departs from the certified configuration creates an obvious compliance grey area around the original CE-marked configuration.
United Kingdom, Switzerland, Canada, Australia: broadly similar to the EU model.
Anywhere else: check your own regulator before you assume you are fine.

I am running this on my own laptop, on a SIM I personally own, on bands and with output power that the modem itself enforces in firmware. I consider the risk to me, in my jurisdiction, acceptable. I cannot make that call for you. If you are not comfortable with the regulatory ambiguity, keep using Lenovo’s helper package, or do not use the modem at all.

I will also point out the obvious: this changes nothing about the radio’s actual emissions. The modem firmware still respects the band tables, the antenna gain limits, and the transmit power caps that Intel certified. You cannot “unlock more bands” or “transmit at higher power” through this handshake. You are flipping the bit that says “I am allowed to talk to the network at all,” nothing else.

Wrapping Up

This was not a heroic hack. I did not reverse-engineer the modem, and I did not write the script. Floris Stoica-Marcu did the real work years ago in a ModemManager merge request. What I did was notice that Lenovo’s proprietary helper was not the only option, install the auditable alternative, and write down what it actually does. That still feels worth documenting, because free software does not stop mattering just because the proprietary piece is small.

References

ModemManager merge request !1141 - the bash unlocker - the script and the discussion around it
lenovo-wwan-unlock - Lenovo’s official proprietary helper package, for comparison
ModemManager FCC Unlock Tools documentation - upstream description of the helper directory mechanism
Directive 2014/53/EU (RED) - EU radio equipment directive

Podman on FreeBSD: OCI Containers Without systemd

2026-04-06T00:00:00+02:00

My previous article covered Podman in depth on Linux - Quadlets, systemd integration, secrets management, auto-updates. That article was explicitly a Linux story. But Podman also runs on FreeBSD, and the experience is different enough to deserve its own treatment.

FreeBSD’s Podman support has been maturing steadily. The FreeBSD port is current and actively maintained, with a dedicated port maintainer who participates in upstream Podman development. That said, the port is still labeled experimental and intended for evaluation and testing - so treat it as a powerful option with some rough edges rather than a drop-in Linux-equivalent experience. The absence of systemd changes the operational model fundamentally. There are no Quadlets, no systemd timers, no journald integration. What you get instead is Podman’s core container runtime integrated with FreeBSD’s own service management, and the ability to run both native FreeBSD containers and Linux containers side by side.

Installing Podman on FreeBSD

Podman is available in the FreeBSD ports tree and as a binary package:

pkg install podman

There’s also a podman-suite meta package that pulls in additional tooling (Buildah, Skopeo) if you need a more complete container workflow.

This pulls in the Podman binary, its dependencies, and the necessary OCI runtime components. On FreeBSD, Podman relies on FreeBSD-specific runtime support, typically involving ocijail - a FreeBSD-native OCI runtime that leverages jails under the hood to provide the container isolation boundary, rather than runc or crun from the Linux world. Container data lives under /var/db/containers/ by default (not /var/lib/containers/ as on Linux).

Prerequisites

Before Podman will work, you need two things beyond the package itself: fdescfs and PF.

fdescfs - Podman’s container monitor (conmon) needs fdescfs(5) mounted on /dev/fd to properly support container restart policies. If it’s not already mounted:

mount -t fdescfs fdesc /dev/fd

Make it permanent in /etc/fstab:

fdesc   /dev/fd         fdescfs         rw      0       0

PF firewall - Container networking relies on NAT to route container traffic out to the host’s network. This requires PF. The Podman package ships a sample configuration:

cp /usr/local/etc/containers/pf.conf.sample /etc/pf.conf

Edit /etc/pf.conf and set the v4egress_if and v6egress_if variables to your network interface(s), then enable PF:

sysrc pf_enable=YES
service pf start

To support port redirections from the host to services inside containers (e.g., podman run -p 8080:80), you also need the pf kernel module loaded and local filtering enabled:

echo 'pf_load="YES"' >> /boot/loader.conf
kldload pf
sysctl net.pf.filter_local=1
echo 'net.pf.filter_local=1' >> /etc/sysctl.conf.local
service pf restart

The sample PF configuration includes the necessary nat-anchor "cni-rdr/*" rule for redirect support. If you’re integrating into an existing PF ruleset rather than using the sample, make sure that anchor is present.

Storage - If your system uses ZFS (and it should), create a dedicated dataset for Podman:

zfs create -o mountpoint=/var/db/containers zroot/containers

If ZFS is not available, change the storage driver to vfs in /usr/local/etc/containers/storage.conf:

sed -I .bak -e 's/driver = "zfs"/driver = "vfs"/' /usr/local/etc/containers/storage.conf

Enabling the Service

The FreeBSD ports package includes rc.d integration for Podman and its API/socket service:

sysrc podman_enable=YES
service podman start

The API/socket service is particularly useful when you need Docker-compatible socket access or API-driven tooling (like Traefik’s Docker provider or CI runners). The daemonless-vs-service story on FreeBSD is less clean-cut than the usual Linux Podman model - the FreeBSD port exposes distinct service knobs like podman_enable and podman_service_enable, and the exact role of the service in local CLI operations depends on your configuration. When in doubt, enable it.

Verification

With everything in place, verify with the Podman hello-world image:

podman run --rm quay.io/dougrabson/hello

If you see the Podman mascot, you’re good.

Native FreeBSD OCI Containers

This is the part that surprises people: Podman on FreeBSD can run containers built from native FreeBSD base images. These aren’t Linux containers running through a compatibility layer - they’re actual FreeBSD userland processes running in jail-based isolation.

The FreeBSD community maintains OCI images specifically for this purpose:

$ podman run --rm -it docker.io/freebsd/freebsd-runtime:15.0 freebsd-version -u
15.0-RELEASE

Inside that container, you’re in FreeBSD userland - freebsd-version reports a real FreeBSD release, and standard FreeBSD tooling works as you’d expect from the image contents. You can build and run FreeBSD software natively, with OCI image packaging and distribution.

Available base images include:

freebsd/freebsd-runtime - minimal FreeBSD userland, suitable for running pre-built binaries
freebsd/freebsd - fuller base with development tools, suitable for building software inside the container

You can build your own FreeBSD OCI images with a standard Containerfile:

FROM docker.io/freebsd/freebsd-runtime:15.0

RUN pkg install -y nginx
COPY nginx.conf /usr/local/etc/nginx/nginx.conf
EXPOSE 80

CMD ["nginx", "-g", "daemon off;"]

Build and run it like any OCI image:

podman build -t my-freebsd-nginx .
podman run -d -p 8080:80 my-freebsd-nginx

This is native performance - no emulation, no translation layer. The container process runs as a FreeBSD process inside a jail, with OCI-standard image packaging on top.

Linux Containers via the Linuxulator

FreeBSD’s Linux binary compatibility layer (the Linuxulator) extends to Podman. With linux64.ko loaded, Podman can run Linux OCI images - the same images you’d pull from Docker Hub for any Linux-based deployment.

Prerequisites

Load the Linux kernel module on the host:

# Persistent across reboots
sysrc linux_enable=YES
echo 'linux64_load="YES"' >> /boot/loader.conf

# Load immediately
kldload linux64

If you’ve followed my Factorio on FreeBSD article, this is the same Linuxulator setup - just applied to OCI containers instead of standalone binaries.

Running Linux Images

With the Linuxulator loaded, pulling and running Linux images works as you’d expect:

$ podman run --rm --os=linux docker.io/library/alpine cat /etc/os-release | head -1
NAME="Alpine Linux"

The --os=linux flag tells Podman to select the Linux variant of a multi-platform image. Inside the container, the process sees a Linux environment - glibc calls work, Linux-specific syscalls are translated by the Linuxulator.

This means many Linux-oriented OCI images can run on FreeBSD without modification, especially userland-heavy workloads that don’t depend on Linux-specific kernel facilities. Your existing application server images, many database images, and standard web tooling are good candidates.

Limitations

One important practical difference from Linux is that current FreeBSD Podman deployments are typically run as root. Rootless Podman is not part of the FreeBSD workflow today, so the threat model is different from Linux setups where rootless operation is a major security feature.

The Linuxulator itself is remarkably complete, but it’s not perfect:

Syscall coverage: The Linuxulator implements a large subset of Linux syscalls, but some less common ones may be missing. Most mainstream software works fine; highly Linux-specific tools (those that depend on cgroups v2 internals, eBPF, or kernel-specific /proc and /sys interfaces) may not.
Performance: Native FreeBSD containers have zero translation overhead. Linux containers go through the Linuxulator’s syscall translation, which adds some overhead - negligible for I/O-bound workloads, potentially noticeable for syscall-heavy applications.
Kernel features: Linux containers on FreeBSD don’t have access to Linux-specific kernel features like cgroups (FreeBSD uses its own resource limiting via rctl), seccomp, or kernel namespaces. Isolation is provided by FreeBSD jails instead.

Container Lifecycle Without systemd

Here’s where the operational model diverges most from Linux. On Linux, Quadlets turn containers into systemd services with dependency management, restart policies, logging integration, and boot startup - all through the init system. FreeBSD doesn’t have systemd, so none of that exists.

Instead, you have two approaches.

Podman’s Built-in Restart Policy

The simplest method. Podman’s --restart flag works on FreeBSD just as it does on Linux:

podman run -d \
    --name my-app \
    --restart always \
    -p 8080:80 \
    docker.io/library/nginx:latest

With podman_enable=YES in /etc/rc.conf, the Podman service starts at boot, and containers with --restart=always are automatically restarted. This gives you basic lifecycle management without writing any service scripts.

The restart policies are the same as Docker’s:

Policy	Behavior
`no`	Never restart (default)
`always`	Always restart, including at boot
`on-failure[:max]`	Restart on non-zero exit, optional retry limit
`unless-stopped`	Like `always`, but not if manually stopped

For simple deployments, this is often sufficient. The container starts at boot, restarts on crash, and Podman handles the lifecycle.

rc.d Service Scripts

For more control - dependency ordering, custom health checks, integration with FreeBSD’s service framework - write an rc.d script. This is FreeBSD’s native equivalent of a systemd unit file:

#!/bin/sh

# PROVIDE: myapp
# REQUIRE: DAEMON podman
# KEYWORD: shutdown

. /etc/rc.subr

name="myapp"
rcvar=myapp_enable

load_rc_config $name

: ${myapp_enable:="NO"}
: ${myapp_image:="docker.io/library/nginx:latest"}
: ${myapp_name:="myapp"}

start_cmd="${name}_start"
stop_cmd="${name}_stop"
status_cmd="${name}_status"

myapp_start()
{
    echo "Starting ${name}."
    # Clean up any orphaned container from a dirty shutdown
    podman rm -f ${myapp_name} 2>/dev/null || true

    podman run -d \
        --name ${myapp_name} \
        --restart on-failure:5 \
        -p 8080:80 \
        ${myapp_image}
}

myapp_stop()
{
    echo "Stopping ${name}."
    podman stop ${myapp_name}
    podman rm ${myapp_name}
}

myapp_status()
{
    podman inspect --format '{{.State.Status}}' ${myapp_name} 2>/dev/null || \
        echo "${name} is not running."
}

run_rc_command "$1"

Save this as /usr/local/etc/rc.d/myapp, make it executable, and enable it:

chmod +x /usr/local/etc/rc.d/myapp
sysrc myapp_enable=YES
service myapp start

The # REQUIRE: DAEMON podman line ensures the Podman service is running before your container starts. You can chain dependencies between container services the same way:

# In your database service script
# PROVIDE: myapp_db
# REQUIRE: DAEMON podman

# In your application service script
# PROVIDE: myapp_web
# REQUIRE: myapp_db

This gives you explicit startup ordering through FreeBSD’s rcorder system - the same mechanism that orders every other service on the system.

What You Lose Without Quadlets

Let’s be direct about the trade-offs. Compared to the Quadlet workflow on Linux:

Capability	Linux (Quadlets)	FreeBSD (rc.d / restart policy)
Declarative container definition	`.container` files	`podman run` flags in scripts
Dependency management	systemd `After=`, `Requires=`	rcorder `REQUIRE:`
Log integration	journald via `journalctl -u`	stdout/stderr to files, or syslog
Auto-updates	`podman auto-update` with systemd timer	Custom scripting (pull + recreate via cron)
Health checks driving restarts	`HealthOnFailure=stop` + systemd restart	Custom scripting
Resource limits	systemd cgroup directives	`rctl` rules on the jail
Secrets injection	`Secret=` directive	`podman secret` CLI (works the same)

The Quadlet workflow is genuinely more ergonomic for complex multi-container deployments. FreeBSD’s approach requires more manual wiring. But it’s the same tools FreeBSD administrators already use for everything else on the system - rc.d, cron, rctl - so the operational patterns are familiar even if they’re less integrated.

Logging

Without journald, container logs go to Podman’s log driver. By default, Podman stores logs per container under its local container storage. You access them the usual way:

podman logs -f my-app

For centralized logging, configure the syslog log driver:

podman run -d \
    --log-driver syslog \
    --log-opt syslog-facility=local0 \
    --name my-app \
    docker.io/library/nginx:latest

This sends container output to FreeBSD’s syslogd, where it integrates with your existing log infrastructure - rotation via newsyslog.conf, forwarding to a central syslog server, or whatever your setup uses.

Image Updates

Podman’s documented auto-update workflow is systemd-centric - it’s designed for containers running inside systemd units, and after pulling a newer image it restarts the systemd unit managing the container. The timer that triggers the check is only one piece; the restart/recreation mechanism itself depends on systemd. That entire chain doesn’t exist on FreeBSD.

What you can do is script the image refresh and container recreation yourself. A cron job can check for newer images and log the results:

# /etc/cron.d/podman-image-refresh
0 4 * * * root /usr/local/sbin/podman-refresh.sh 2>&1 | logger -t podman-refresh

But unlike the Linux auto-update flow, you need to handle the restart and recreation step yourself - pulling the new image, stopping the old container, and starting a fresh one with the updated image. A simple approach:

#!/bin/sh
# /usr/local/sbin/podman-refresh.sh
for name in traefik myapp; do
    image=$(podman inspect --format '{{.ImageName}}' "$name" 2>/dev/null) || continue
    if podman pull "$image" | grep -q "Writing manifest"; then
        podman stop "$name"
        podman rm "$name"
        # Re-create via your rc.d script or however the container is defined
        service "$name" start
    fi
done

This is more manual than the Linux Quadlet auto-update experience, and it requires that your rc.d scripts or startup commands can fully recreate the container from scratch. It’s operational overhead worth acknowledging: on Linux, Podman and systemd handle this loop for you; on FreeBSD, you own it.

Podman and Jails: Complementary, Not Competing

This is the important framing. FreeBSD already has the most mature OS-level container technology in existence: Jails. They’ve been around since FreeBSD 4.0, they provide kernel-enforced isolation, and they’re deeply integrated with the rest of the operating system. So why would you want Podman?

The answer isn’t “Podman is better than Jails.” It’s that they solve different problems:

Jails are FreeBSD’s native isolation primitive. A jail is essentially a lightweight FreeBSD instance with its own filesystem, network stack, process space, and user database. Jails are ideal for:

Long-lived services that you build and maintain yourself
Workloads that benefit from direct access to FreeBSD’s kernel features (ZFS, PF, rctl, DTrace)
Environments where you want full FreeBSD package management inside the isolated environment
Multi-tenant hosting where each tenant gets a complete FreeBSD environment

Podman brings OCI container compatibility. It’s ideal for:

Running third-party software distributed as OCI/Docker images without repackaging
Linux-only applications that have no FreeBSD port (via the Linuxulator)
Workflows that already use Dockerfiles and container registries
CI/CD pipelines that produce OCI images as artifacts
Development environments that need to match Linux production targets

The practical scenario: you run your core infrastructure - your reverse proxy, your databases, your custom applications - in Jails, managed by Bastille or your preferred jail manager, with ZFS snapshots, VNET networking, and PF firewall rules. Then a project requires a specific application that’s only distributed as a Docker image, or your CI pipeline produces OCI images that need to run somewhere. Instead of setting up a Linux VM or wrestling with ports and dependencies, you run that image with Podman. The Jail handles what Jails do best; Podman handles what the OCI ecosystem does best.

In theory, Podman-inside-a-Jail sounds attractive for defense in depth, but I would treat it as experimental at best rather than a recommended deployment pattern.

# Create a jail for container workloads
bastille create containers 14.3-RELEASE 10.254.252.50 bastille0

# Install Podman inside the jail
bastille pkg containers install podman

# Enable and start Podman inside the jail
bastille sysrc containers podman_enable=YES
bastille service containers podman start

# Run OCI containers inside the jail
bastille cmd containers podman run -d --name myapp docker.io/library/nginx:latest

This gives you defense-in-depth: the jail provides network isolation and resource limits at the FreeBSD kernel level, while Podman handles the OCI runtime inside.

Practical Tips

Networking

Podman networking on FreeBSD follows the same broad model as on Linux - user-defined networks, container name resolution, and host port publishing - but the backend may differ by port version and build configuration. Current FreeBSD ports have been observed using CNI rather than netavark. Check what your install is using with podman info:

podman info --format '{{.Host.NetworkBackend}}'

Regardless of backend, the operational patterns are the same:

# Create a network
podman network create mynet

# Run containers on that network
podman run -d --network mynet --name db postgres:17
podman run -d --network mynet --name app myapp:latest

DNS resolution between containers on the same network works by container name.

Remember that all of this depends on the PF setup from the installation section. Without PF doing NAT, container traffic has no route out of the container network. Port redirections (-p flag) flow through the cni-rdr PF anchors. If you’re already running PF with your own ruleset (and you probably are if you’ve read my PF guide), integrate the Podman anchors into your existing configuration rather than replacing it with the sample file.

Storage

With the ZFS dataset from the installation section in place, Podman automatically uses the ZFS storage driver. This is one area where Podman on FreeBSD genuinely outshines the Linux experience: instead of fighting with overlayfs quirks or fuse-overlayfs for rootless setups, Podman creates real ZFS datasets for container image layers. You get copy-on-write, compression, checksumming, and snapshot support for free, all backed by the same storage stack you’re already using for everything else on the system. You can snapshot the entire container storage tree, roll back after a bad update, or send it to another host with zfs send - all operations that feel natural on FreeBSD and awkward on Linux.

Secrets

Podman’s secret store works identically on FreeBSD:

pwgen -s 32 1 | tr -d '\n' | podman secret create db_password -
podman run -d --secret db_password,type=env,target=DB_PASS myapp:latest

No differences here - the secrets management story from the Linux article applies as-is.

When to Use What

Scenario	Recommended approach
Custom FreeBSD service, full OS access needed	Jail
Third-party app only available as Docker image	Podman
Linux-only software with no FreeBSD port	Podman + Linuxulator
Multi-tenant isolation with independent networks	Jail (VNET)
CI/CD pipeline producing OCI artifacts	Podman
Database with ZFS snapshots for backup	Jail (direct ZFS access)
Development parity with Linux production	Podman + Linux containers
Mix of both needs	Podman inside a Jail

Wrapping Up

Podman on FreeBSD isn’t trying to replace Jails. It’s filling a gap that Jails were never designed to fill: compatibility with the OCI container ecosystem. The ability to podman pull an image from Docker Hub and run it on FreeBSD - whether it’s a native FreeBSD image or a Linux image through the Linuxulator - means you don’t have to choose between FreeBSD’s operational model and the container ecosystem.

The operational workflow is less polished than on Linux. Without systemd, you lose Quadlets and the tight init-system integration that makes Podman on Linux so compelling. But what you get in return is Podman integrated with FreeBSD’s own tools: rc.d for service management, cron for scheduling, rctl for resource limits, PF for network policy, ZFS for storage. Different tools, same result.

If you’re running FreeBSD and you’ve been eyeing the OCI ecosystem from the outside, Podman is your bridge in.

References

Podman Installation: FreeBSD - official installation guide
Podman on FreeBSD - containers/podman Wiki
FreeBSD OCI Images
FreeBSD Handbook: Linux Binary Compatibility
ocijail - FreeBSD OCI Runtime
Podman in Production: Quadlets, Secrets, Auto-Updates, and Docker Compatibility - the Linux-focused companion article
FreeBSD Foundationals: Jails - deep dive into FreeBSD’s native isolation
Running Factorio on FreeBSD with the Linuxulator - another Linuxulator use case

Podman in Production: Quadlets, Secrets, Auto-Updates, and Docker Compatibility

2026-04-05T00:00:00+02:00

I’ve been running Podman in production for years now. Every container on my infrastructure - Forgejo, Traefik, PostgreSQL, Keycloak, CI runners - is managed by Podman on RHEL. Not a single Docker daemon in sight. Regular readers know that my first instinct for isolation is FreeBSD Jails - but when I’m on Linux and dealing with OCI containers, Podman is the tool I reach for. It follows the Unix model more closely than Docker, using smaller composable pieces instead of a single central daemon.

This isn’t a “getting started” tutorial. This is an opinionated production-ops perspective on Linux hosts - not a universal answer for every developer workstation or platform. It’s the article I wish someone had written when I was migrating away from Docker: the architectural reasons I prefer Podman, the practical patterns that make it work in production, and the real gotchas you’ll hit along the way. If you’ve read my earlier Quadlets guide or the Keycloak deployment article, this ties those threads together into a bigger picture.

Table of Contents
Why Podman Is Better Than Docker
Quadlets: Containers as systemd Services
- The Basics
- Why Quadlets Beat Compose
Real-World Deployment: A Complete Stack
Secrets Management
Automatic Updates
- Two Update Modes
- A Word of Caution
Docker Compatibility
Practical Tips
When Podman Isn’t the Right Choice
The Bigger Picture
References

Why Podman Is Better Than Docker

I’m not going to pretend this is a balanced comparison. On Linux servers, I think Podman is the cleaner architecture, and the reasons are structural rather than cosmetic.

No Daemon, No Single Point of Failure

Docker’s architecture has a trade-off I no longer think is worth accepting on Linux hosts: a privileged central daemon. Every docker run, every docker build, every container lifecycle event goes through dockerd - a process running as root with effectively unrestricted access to your system.

Docker does offer --live-restore to keep containers running during daemon restarts, but it comes with limitations around networking and interactive sessions, and it’s not the default. The fundamental coupling remains: every container’s control path runs through a single privileged process. That’s operational risk and attack surface that adds up.

Podman uses a fork/exec model. Each container is a direct child process of whatever started it - your shell, systemd, a CI runner. There is no central daemon. Containers are independent processes managed by the kernel, the way Unix was designed to work.

Docker:
  User → dockerd (root daemon) → containerd → runc → container
                ↑
        centralized control, shared fate by default

Podman:
  User → conmon → runc → container
  systemd → conmon → runc → container
         (no daemon, direct process tree)

I can update Podman on my server without touching running containers. Each container’s lifecycle is independent.

Rootless by Design, Not by Afterthought

Docker bolted on rootless support years after the fact, and it shows. Running Docker rootless still requires a separate dockerd-rootless process, has compatibility limitations with storage drivers, and adds limitations and extra setup around things like privileged ports.

Podman was designed rootless from the start. User namespaces, subordinate UID/GID mapping, and unprivileged container execution are core architecture, not a compatibility layer. A regular user can run containers without any elevated privileges:

# As a regular user, no sudo needed
podman run --rm -it docker.io/library/alpine:latest sh

The security implications are significant. A container escape in a rootless Podman setup lands you in an unprivileged user namespace. A compromise involving Docker’s privileged daemon has a much higher potential blast radius on the host than a compromise contained inside a rootless user namespace. That’s not a theoretical distinction - it’s a meaningful difference in your threat model.

SELinux Is a Feature, Not an Obstacle

Docker’s relationship with SELinux has historically been adversarial. Countless Docker tutorials start with setenforce 0 because the daemon and SELinux don’t always cooperate. Podman integrates with SELinux as a first-class security layer.

Every Podman container automatically gets an SELinux label (container_t). Volume mounts use :z and :Z flags to handle context relabeling. The container_runtime_t type exists specifically for containers that need elevated access patterns (like mounting the Podman socket). None of this requires disabling SELinux or writing custom policy modules.

If you’re running RHEL or Fedora with SELinux enforcing (as you should be - see my SELinux guide), Podman just works with it. Docker sometimes fights it.

The OCI Guarantee

Podman is fully OCI-compliant. For most real-world use cases, Docker images, registries, and Dockerfiles work with Podman without modification. You can podman pull from Docker Hub, podman build with a Dockerfile, and podman push to any registry. The images are identical because they follow the same OCI specification.

# These do the same thing
docker pull nginx:latest
podman pull docker.io/library/nginx:latest

# Buildah (Podman's build companion) reads Dockerfiles natively
podman build -t myapp -f Dockerfile .

The migration path from Docker to Podman is a find-and-replace of docker with podman in most cases. Your images, your registries, your CI pipelines - they all work.

Quadlets: Containers as systemd Services

If you’re still writing podman run commands in shell scripts or using Compose files, you’re missing the most compelling feature in Podman’s ecosystem: Quadlets.

Quadlets are systemd unit files that describe containers, networks, volumes, and images. Drop them in /etc/containers/systemd/ (system-wide) or ~/.config/containers/systemd/ (rootless), and systemd manages them like any other service. No daemon, no orchestrator, no YAML parser sitting between you and your containers.

The Basics

A Quadlet file looks like a systemd unit file with a [Container] section:

[Container]
ContainerName=my-app
Image=docker.io/library/nginx:latest
PublishPort=8080:80
Volume=/srv/www:/usr/share/nginx/html:z

[Service]
Restart=always

[Install]
WantedBy=multi-user.target

Save this as /etc/containers/systemd/my-app.container, run systemctl daemon-reload, and you have a container managed by systemd:

systemctl start my-app.service
systemctl status my-app.service
journalctl -u my-app.service -f

No compose file. No daemon. Just systemd doing what it already does well: managing service lifecycles.

Why Quadlets Beat Compose

Docker Compose requires a separate binary, parses YAML, and maintains its own state about which containers belong to which project. It’s a parallel service manager running alongside (or on top of) your actual service manager.

Quadlets eliminate that entire layer:

Concern	Docker Compose	Podman Quadlets
Service lifecycle	`docker compose up/down`	`systemctl start/stop`
Boot startup	Requires daemon + compose plugin	Native systemd `WantedBy=`
Dependencies	`depends_on:` (limited)	systemd `After=`, `Requires=` (battle-tested)
Logs	`docker compose logs`	`journalctl -u` (integrated with system logging)
Resource limits	Compose YAML `deploy:` section	systemd cgroup directives (kernel-enforced)
Update mechanism	Manual `docker compose pull && up`	`podman auto-update` with systemd timer
Restart policy	Compose-level	systemd’s proven restart logic

The dependency management alone is worth the switch. systemd’s dependency graph has been solving service ordering for over a decade. Compose’s depends_on is limited to basic ordering, and it can’t express dependency chains involving non-container services. systemd’s After= and Requires= handle ordering and hard dependencies natively, and when you need actual readiness (not just “the process started”), Quadlets support health check primitives:

[Container]
HealthCmd=pg_isready -U myuser
HealthInterval=10s
HealthTimeout=5s
HealthRetries=5
HealthStartPeriod=30s
HealthOnFailure=stop

HealthStartPeriod gives the application time to initialize before health checks count. HealthOnFailure=stop tells Podman to stop the container if it fails health checks - and systemd’s Restart=always brings it back. This gets you much closer to real readiness and recovery behavior than simple startup ordering alone.

Real-World Deployment: A Complete Stack

Let me show you what a real Podman deployment looks like. These are based on my production Quadlets, sanitized for opsec but architecturally identical.

Network Topology

First, the network isolation. Every multi-container deployment gets a dedicated backend network:

/etc/containers/systemd/forgejo-backend.network:

[Network]
Subnet=172.16.0.0/24
Gateway=172.16.0.1
IPRange=172.16.0.0/28

The /28 in IPRange limits DHCP-assigned addresses to 14 hosts. The database and application container need two. This prevents the network from becoming a dumping ground for unrelated containers and makes the architecture self-documenting.

The frontend network (where Traefik lives) is separate. Application containers join both; databases join only the backend. The database has no route to the internet, ever.

Database Container

/etc/containers/systemd/forgejo-db.container:

[Container]
ContainerName=forgejo-db
AutoUpdate=registry
Image=registry.redhat.io/rhel10/postgresql-16:latest
Network=forgejo-backend.network
Environment=POSTGRESQL_USER=forgejo
Environment=POSTGRESQL_DATABASE=forgejo

Secret=forgejo_db_password,type=env,target=POSTGRESQL_PASSWORD

Volume=/opt/forgejo/postgres:/var/lib/pgsql/data:z

[Service]
Restart=always

[Install]
WantedBy=default.target

Note what’s not here: no frontend network, no published ports, no Traefik labels. This container talks to exactly one network and serves exactly one purpose. The password comes from Podman’s secret store (more on that below), not from the unit file.

Application Container

/etc/containers/systemd/forgejo-server.container:

[Container]
ContainerName=forgejo-server
Image=codeberg.org/forgejo/forgejo:14
AutoUpdate=registry

# Internal network for database connection
Network=forgejo-backend.network
# External network with Traefik
Network=frontend.network

Environment=USER_UID=1000
Environment=USER_GID=1000
Environment=FORGEJO__database__DB_TYPE=postgres
Environment=FORGEJO__database__HOST=forgejo-db:5432
Environment=FORGEJO__database__NAME=forgejo
Environment=FORGEJO__database__USER=forgejo

Secret=forgejo_db_password,type=env,target=FORGEJO__database__PASSWD

Volume=/opt/forgejo/forgejo:/data:z
Volume=/etc/timezone:/etc/timezone:ro
Volume=/etc/localtime:/etc/localtime:ro

# Traefik routing labels
Label="traefik.enable=true"
Label="traefik.docker.network=frontend"
Label="traefik.http.routers.forgejo.rule=Host(`git.example.com`)"
Label="traefik.http.routers.forgejo.entrypoints=https"
Label="traefik.http.routers.forgejo.service=forgejo-http"
Label="traefik.http.routers.forgejo.tls.certresolver=traefiktls"
Label="traefik.http.routers.forgejo.middlewares=secure-headers@file"
Label="traefik.http.services.forgejo-http.loadbalancer.server.port=3000"
Label="traefik.tcp.routers.forgejo-ssh.rule=HostSNI(`*`)"
Label="traefik.tcp.routers.forgejo-ssh.entrypoints=ssh"
Label="traefik.tcp.routers.forgejo-ssh.service=forgejo-ssh"
Label="traefik.tcp.services.forgejo-ssh.loadbalancer.server.port=22"

[Service]
Restart=always

[Install]
WantedBy=default.target

[Unit]
After=forgejo-db.service
After=traefik.service

The dual network attachment is the key pattern: backend for the database, frontend for Traefik. Podman’s built-in DNS resolves forgejo-db on the backend network, so the app finds its database by hostname without managing IP addresses.

The After= directives ensure proper startup ordering - systemd won’t start Forgejo until the database and Traefik services have started. Note that After= is ordering, not readiness: it guarantees the services launch in sequence, but not that the database is actually accepting connections yet. For that, combine with the health check primitives shown earlier.

Reverse Proxy

/etc/containers/systemd/traefik.container:

[Container]
ContainerName=traefik
Image=docker.io/traefik:latest
AutoUpdate=registry

AddCapability=CAP_NET_BIND_SERVICE

Network=frontend.network
PublishPort=80:80
PublishPort=443:443
PublishPort=2222:2222

NoNewPrivileges=true
SecurityLabelType=container_runtime_t

Volume=/etc/localtime:/etc/localtime:ro
Volume=/run/podman/podman.sock:/var/run/docker.sock:ro
Volume=/opt/traefik/traefik.yml:/etc/traefik/traefik.yml:z,ro
Volume=/opt/traefik/config.yml:/etc/traefik/config.yml:z,ro
Volume=/opt/traefik/letsencrypt:/letsencrypt:z
Volume=/var/log/traefik:/var/log/traefik:z

Label=traefik.enable=true
Label=traefik.http.routers.dashboard.rule=Host(`traefik.example.com`)
Label=traefik.http.routers.dashboard.entrypoints=https
Label=traefik.http.routers.dashboard.service=api@internal
Label=traefik.http.routers.dashboard.tls=true
Label=traefik.http.routers.dashboard.tls.certresolver=traefiktls
Label=traefik.http.routers.dashboard.middlewares=dashboard-auth,secure-headers@file
Label=traefik.http.middlewares.dashboard-auth.basicauth.users=admin:$$2y$$05$$...

[Service]
Restart=always

[Install]
WantedBy=default.target

Notice SecurityLabelType=container_runtime_t. Traefik needs to read the Podman socket to discover containers, and the default container_t SELinux type doesn’t permit that. The container_runtime_t type grants the necessary access without disabling SELinux. This is precisely the kind of fine-grained security control that Docker makes difficult and Podman makes natural.

CI Runner

/etc/containers/systemd/forgejo-runner.container:

[Unit]
Description=Forgejo Runner
After=forgejo-server.service network-online.target

[Container]
ContainerName=forgejo-runner
Image=code.forgejo.org/forgejo/runner:12

User=root

NoNewPrivileges=true
Exec=forgejo-runner daemon

Network=forgejo-backend.network

SecurityLabelType=container_runtime_t

Volume=/opt/forgejo/runner:/data:z
Volume=/opt/forgejo/runner/config.yml:/data/config.yml:ro
Volume=/run/podman/podman.sock:/var/run/docker.sock:z

Environment=CONFIG_FILE=/data/config.yml
Environment=DOCKER_HOST=unix:///var/run/docker.sock

[Service]
Restart=always

[Install]
WantedBy=default.target

The CI runner is a container that spawns other containers - it needs the Podman socket. The socket is mounted as /var/run/docker.sock because the runner expects Docker’s socket path. Podman’s Docker-compatible API handles the rest transparently. The runner doesn’t know (or care) that it’s talking to Podman instead of Docker.

Secrets Management

Hardcoding passwords in unit files is a non-starter. Environment variables in Quadlet files are visible in process listings, in systemd’s journal, and in the unit files themselves on disk. Podman’s secret store solves this properly.

Creating Secrets

# Generate a strong password and store it
pwgen -s 32 1 | tr -d '\n' | podman secret create forgejo_db_password -

# Or from an existing file
podman secret create my_api_key /path/to/keyfile

# List stored secrets
podman secret ls

The tr -d '\n' matters. pwgen appends a trailing newline, and podman secret create stores bytes verbatim. Some applications strip trailing whitespace from passwords; others don’t. A PostgreSQL init script might strip it while a JDBC driver sends it as-is, resulting in a password mismatch between two containers reading the exact same secret. Always strip the newline.

Using Secrets in Quadlets

[Container]
# Inject as environment variable
Secret=forgejo_db_password,type=env,target=FORGEJO__database__PASSWD

# Or mount as file
Secret=my_tls_cert,type=mount,target=/etc/ssl/certs/app.crt

The type=env variant injects the secret as an environment variable at container creation time. The type=mount variant mounts it as a file inside the container, which is preferable for TLS certificates or configuration files that applications expect on disk.

Rotating Secrets

Because type=env secrets are injected at container creation time, a normal restart of the existing container won’t refresh a secret that was injected when the container was created. You need to destroy the old container and create a fresh one:

# Remove the old secret
podman secret rm forgejo_db_password

# Create new secret
pwgen -s 32 1 | tr -d '\n' | podman secret create forgejo_db_password -

# Stop and start (not restart) to force container recreation
systemctl stop forgejo-db.service forgejo-server.service
systemctl start forgejo-db.service forgejo-server.service

The stop/start cycle destroys the old container and creates a new one from the Quadlet definition, picking up the new secret value. Volumes persist across this cycle, so no data is lost.

Automatic Updates

One of Podman’s most underrated features. Enable the systemd timer:

systemctl enable --now podman-auto-update.timer

With AutoUpdate=registry in your Quadlet files, Podman checks daily for new image versions. When a newer image is available at the same tag, Podman pulls it and recreates the container, preserving volumes and configuration.

# See what would be updated without doing it
podman auto-update --dry-run

# Force an update check now
podman auto-update

Two Update Modes

AutoUpdate=registry: Checks the remote registry for a newer image at the same tag. Use this for images you pull from Docker Hub, Quay, or Red Hat’s registry.
AutoUpdate=local: Checks if the local image has been rebuilt. Use this for images you build locally with podman build.

A Word of Caution

Auto-updates are excellent for reverse proxies, utility containers, and applications where rolling forward is safe. For databases and stateful services, be more deliberate: pin to a specific version tag and bump manually after reviewing the changelog. A PostgreSQL major version bump via auto-update at 3 AM is not how you want to discover that a migration is required.

# Safe to auto-update (stateless, easily rolled back)
Image=docker.io/traefik:latest
AutoUpdate=registry

# Pin and update manually (stateful, needs migration planning)
Image=registry.redhat.io/rhel10/postgresql-16:1-30
AutoUpdate=registry

Even with a pinned version like 16:1-30, AutoUpdate=registry still checks for image rebuilds at that exact tag (security patches, base image updates). The tag acts as a ceiling for how far the update can go.

Docker Compatibility

Podman goes out of its way to be a drop-in replacement for Docker. This isn’t just marketing - the compatibility is real, practical, and covers the two areas people worry about most.

The Docker Socket

Podman provides a Docker-compatible API socket via a systemd-managed service:

# Enable the socket (root)
systemctl enable --now podman.socket

# Or rootless
systemctl --user enable --now podman.socket

This creates /run/podman/podman.sock (root) or /run/user/$UID/podman/podman.sock (rootless), exposing a REST API that speaks Docker’s protocol. Any tool expecting a Docker socket - Traefik, Portainer, CI runners, monitoring agents - can connect to this socket and work without modification.

In Quadlet files, mount it where the application expects Docker’s socket:

Volume=/run/podman/podman.sock:/var/run/docker.sock:ro

The application sees /var/run/docker.sock, sends Docker API calls, and Podman handles them. The application never knows the difference. This is how the Traefik and CI runner examples above work - Traefik’s Docker provider discovers containers through this socket, reading their labels for routing configuration, without any Podman-specific configuration.

Docker Compose Compatibility

Podman provides podman compose as a thin wrapper around an external Compose provider, wiring that provider up to the local Podman socket. The provider can be docker-compose (the standalone Python binary), the Go-based docker compose plugin, or podman-compose. Install a supported provider:

# podman-compose is the most common choice on Fedora/RHEL
dnf install podman-compose

Your existing docker-compose.yml files work as-is:

podman compose up -d
podman compose logs -f
podman compose down

The result is standard Podman containers - you can inspect them with podman ps, check their logs with podman logs, and manage them with all the usual Podman tooling.

That said, I don’t recommend using Compose on Podman in production. It works, and it’s useful for quick local development or migrating an existing Docker Compose setup. But you’re layering a Docker abstraction on top of Podman, bypassing the systemd integration that makes Podman compelling in the first place. Compose manages its own container lifecycle, its own dependency ordering, its own restart logic - duplicating what systemd already does better.

The migration path looks like this:

Start with podman compose to verify your existing setup works on Podman
Convert each service to a Quadlet file
Delete the Compose file

For step 2, podman generate systemd --new --files --name my-container can produce systemd unit files from running containers, but this command is deprecated in Podman 5.x in favor of writing Quadlet files directly. In practice, manually converting Compose services to Quadlets is straightforward - the mapping from YAML keys to Quadlet directives is mostly one-to-one, and you end up with cleaner, more maintainable configuration.

The `podman` CLI Alias

Many tools and scripts hardcode docker as the command. A simple alias handles this:

# In /etc/profile.d/docker-compat.sh or your shell rc
alias docker=podman

On Fedora and RHEL, the podman-docker package does this system-wide and also creates a Docker-compatible socket symlink:

dnf install podman-docker

This installs a docker wrapper that calls podman, provides /var/run/docker.sock as a symlink to the Podman socket, and makes Docker-expecting tools work without configuration changes. It even suppresses the “Emulate Docker CLI using podman” message that podman normally prints.

Practical Tips

Networking Patterns

Dedicated backend networks for every stack. Don’t share a single network across unrelated services. Each application stack (app + database) gets its own isolated backend. This is defense-in-depth: if one application is compromised, the attacker can’t reach another stack’s database.

# Each stack gets its own backend
/etc/containers/systemd/forgejo-backend.network
/etc/containers/systemd/keycloak-backend.network
/etc/containers/systemd/nextcloud-backend.network

One shared frontend network for the reverse proxy. Traefik (or whatever you use) joins this network, and each application container joins it as a second network. This is the only network with a route to the outside.

DNS resolution is automatic. Containers on the same Podman network resolve each other by container name. No need to manage IP addresses or /etc/hosts entries. forgejo-db:5432 just works from any container on forgejo-backend.network.

Volume and Permission Patterns

Always use :z on SELinux systems. The :z flag relabels the host directory with container_file_t, allowing container access. Without it, SELinux blocks the mount and you get cryptic permission errors.

# Shared volume (multiple containers can access)
Volume=/opt/app/data:/data:z

# Private volume (only this container)
Volume=/opt/app/secrets:/secrets:Z

# Read-only system files don't need relabeling
Volume=/etc/localtime:/etc/localtime:ro

Set filesystem ACLs for container UIDs. Containers often run as non-root UIDs internally. The RHEL PostgreSQL image uses UID 26, Forgejo uses UID 1000. Without filesystem-level access, the container can’t write to its mounted volumes:

mkdir -p /opt/forgejo/postgres
setfacl -m u:26:rwx /opt/forgejo/postgres

This is separate from SELinux labeling - :z handles the MAC context, setfacl handles the DAC permissions. You need both.

Security Hardening

NoNewPrivileges=true on every container that doesn’t explicitly need privilege escalation. This prevents setuid binaries inside the container from gaining elevated privileges:

[Container]
NoNewPrivileges=true

Drop capabilities by default. Podman already drops most capabilities, but you can explicitly add only what’s needed:

# Only grant what the container actually needs
AddCapability=CAP_NET_BIND_SERVICE

Use SecurityLabelType sparingly. Most containers run fine with the default container_t type. Only use container_runtime_t for containers that need to manage other containers (CI runners, monitoring tools that access the Podman socket).

Debugging

When things go wrong, systemd and Podman give you more integrated diagnostic tooling on Linux hosts:

# Service status with recent log lines
systemctl status forgejo-server.service

# Full journal with filtering
journalctl -u forgejo-server.service --since "10 minutes ago"

# Container events
podman events --filter container=forgejo-server

# Inspect networking
podman inspect forgejo-server --format '{{.NetworkSettings.Networks}}'

# Check systemd dependency tree
systemctl list-dependencies forgejo-server.service

# See what Quadlet generated
/usr/libexec/podman/quadlet --dryrun

The last command is especially useful. Quadlet is a generator that produces systemd unit files from your .container files. If a Quadlet isn’t behaving as expected, --dryrun shows you the actual systemd unit that was generated, so you can see exactly what systemd is working with.

Registry Authentication

For private registries (Red Hat’s registry.redhat.io, corporate registries), authenticate once and Podman stores the credentials:

# Interactive login
podman login registry.redhat.io

# Credentials are stored in
# /run/containers/0/auth.json (root)
# $XDG_RUNTIME_DIR/containers/auth.json (rootless)

For automated environments, you can provide credentials via a JSON file:

podman login --authfile /etc/containers/auth.json registry.redhat.io

Automation with Ansible

Once you’re managing more than a handful of hosts, writing Quadlet files by hand stops scaling. The containers.podman Ansible Collection can manage every aspect of Podman - containers, pods, networks, volumes, secrets, registry logins - and recent versions can generate Quadlet files directly. Instead of templating unit files yourself, you declare the desired state in a playbook and the collection handles the rest:

- name: Deploy Forgejo database
  containers.podman.podman_container:
    name: forgejo-db
    image: registry.redhat.io/rhel10/postgresql-16:latest
    state: quadlet
    quadlet_dir: /etc/containers/systemd
    network: forgejo-backend.network
    secrets:
      - forgejo_db_password,type=env,target=POSTGRESQL_PASSWORD
    volumes:
      - /opt/forgejo/postgres:/var/lib/pgsql/data:z
    env:
      POSTGRESQL_USER: forgejo
      POSTGRESQL_DATABASE: forgejo

The state: quadlet parameter is the key - it tells the module to generate a Quadlet file in quadlet_dir rather than starting a container directly. This gives you Ansible’s idempotency and inventory management on top of Podman’s systemd integration. If you’re running Podman across a fleet of RHEL hosts, this collection is the missing piece between “it works on one server” and “it works everywhere.”

When Podman Isn’t the Right Choice

I said this article was opinionated, not delusional. There are cases where Docker or Kubernetes makes more sense:

Docker Desktop for local development on macOS/Windows. Podman has podman machine for non-Linux platforms, and it works, but Docker Desktop’s integration with macOS and Windows is more polished. If your developers are on Macs and just need to run containers locally, Docker Desktop is a fine choice. The production host should still be Podman.

Multi-node orchestration. If you need containers spanning multiple hosts with service discovery, rolling updates, and horizontal scaling, that’s Kubernetes territory. Podman is a single-host container runtime. It does that job exceptionally well, but it doesn’t pretend to be an orchestrator.

Ecosystem lock-in. Some CI/CD platforms, development tools, and monitoring solutions have deep Docker-specific integrations that go beyond the API compatibility layer. If your entire toolchain assumes Docker and the compatibility layer doesn’t fully cover your use case, forcing Podman may create more friction than it solves.

For everything else - single-host deployments, homelab infrastructure, edge computing, production services that don’t need orchestration - Podman with Quadlets is the better tool.

The Bigger Picture

Containers are processes. systemd manages processes. Quadlets connect the two. Your containers start at boot, restart on failure, log to journald, respect cgroup limits, and integrate with SELinux - all through mechanisms that have been battle-tested in production Linux systems for over a decade.

Docker popularized containers and deserves credit for that. But the Linux ecosystem has matured - user namespaces, cgroups v2, and systemd’s service management have made the central daemon architecture unnecessary for most Linux server workloads. Podman builds on that maturity rather than working around it.

If you’re starting new container infrastructure on Linux, start with Podman. If you’re running Docker in production, the compatibility layer makes migration surprisingly low-friction, and the architectural improvements make it worthwhile.

References

Podman Documentation
Podman Quadlet Documentation
Podman Auto-Update Documentation
Podman Secrets Documentation
Red Hat: From Docker Compose to Podman Quadlets
containers.podman Ansible Collection - manage Podman and generate Quadlets via Ansible
Production-Grade Container Deployment with Podman Quadlets - my earlier Quadlet guide
Keycloak 26 on Podman with Quadlets - practical Quadlet deployment example
SELinux: A Practical Guide - understanding SELinux in container contexts

Speeding Up Forgejo CI with a Custom OCI Image

2026-04-01T00:00:00+02:00

Every push to this blog triggers a Forgejo Actions pipeline that builds the site with Pelican, then deploys it via rsync. If you want the full picture of how this blog’s infrastructure works - Bastille jails, Caddy, the whole deployment chain - I covered that in an earlier article. The pipeline worked fine. But every single run started with the same ritual: apt-get update, install four system packages, pip install five Python packages. The actual build took seconds - the dependency installation took longer than the rest of the pipeline combined.

This is a solved problem in the container world. Instead of installing dependencies at runtime, bake them into the image. Here’s how I did it with Forgejo’s built-in container registry, keeping everything self-contained on a single Forgejo instance.

The Problem

My original workflow used python:3.11-slim as the CI container and installed everything from scratch on every run:

jobs:
  deploy:
    runs-on: docker
    container:
      image: python:3.11-slim
    steps:
      - name: Install Runtime Dependencies
        run: |
          apt-get update
          apt-get install -y nodejs rsync openssh-client git
          pip install pelican pelican-sitemap markdown typogrify

Every push - even a single typo fix - paid this tax. The install step alone accounted for the majority of the pipeline’s wall clock time.

The Fix: A Custom CI Image

The solution is a Containerfile that does the installation once, at image build time:

FROM python:3.11-slim

RUN apt-get update \
    && apt-get install -y --no-install-recommends \
       nodejs rsync openssh-client git \
    && rm -rf /var/lib/apt/lists/*

RUN pip install --no-cache-dir \
    pelican pelican-sitemap markdown typogrify

Nothing fancy. Same base image, same packages - just shifted from “every CI run” to “once, when I build the image”. The --no-install-recommends and --no-cache-dir flags keep the image lean.

Forgejo’s Built-in OCI Registry

Here’s the part that makes this particularly clean: Forgejo ships with a built-in OCI-compliant container registry. No need for Docker Hub, no need for a separate Harbor instance, no external dependency. Your CI images live right next to your code.

Build and push with Podman:

podman build -t git.example.com/youruser/yourrepo-ci:latest .
podman login git.example.com
podman push git.example.com/youruser/yourrepo-ci:latest

The image shows up under Packages in your Forgejo instance. Same authentication, same access control, same backup strategy as the rest of your Forgejo data.

Updating the Pipeline

With the image pushed, the workflow drops the install step entirely and pulls the pre-built image instead:

jobs:
  deploy:
    runs-on: docker
    container:
      image: git.example.com/youruser/yourrepo-ci:latest
      credentials:
        username: ${{ secrets.REGISTRY_USER }}
        password: ${{ secrets.REGISTRY_PASSWORD }}
    steps:
      - name: Checkout Code
        uses: actions/checkout@v3
        # ... rest of pipeline unchanged

The credentials block authenticates against the Forgejo registry. Store your registry username and an application token as repository secrets.

That’s the entire change. The install step is gone. The pipeline goes straight from image pull to checkout to build to deploy.

The Result

The dependency installation step disappeared completely. What used to be the slowest part of the pipeline is now a single container pull that Forgejo’s runner likely has cached locally after the first run. The actual build-and-deploy cycle is all that remains.

When to Rebuild

The trade-off is that you now maintain a container image. In practice, this is minimal overhead:

Adding a Pelican plugin? Rebuild and push the image
Upgrading Python packages? Rebuild and push
System package update? Rebuild and push

For a blog pipeline, that’s maybe once a quarter. The ci-image/ directory lives in the same repository as the blog, so the Containerfile is versioned alongside everything else.

Takeaway

If your CI pipeline spends more time installing dependencies than doing actual work, build a custom image. If you’re already running Forgejo, you have a container registry sitting right there - use it. Two commits, one Containerfile, and the entire install step vanishes.

This is one of several small improvements I’ve been making to the blog over time. If you’re running a Pelican blog yourself, you might also be interested in how I added Fediverse-based comments without any server-side dependencies.

References

Keycloak 26 on Podman with Quadlets: Identity Management the systemd Way

2026-03-31T00:00:00+02:00

Running your own identity provider is one of those things that sounds straightforward until you’re three hours into debugging OIDC token flows at 2 AM. Keycloak has become the de facto open-source solution for identity and access management, but deploying it properly - with a real database backend, network isolation, and no credentials in plaintext - still takes some deliberate architecture.

This guide walks through deploying the Red Hat Build of Keycloak (RHBK) 26 on Podman using Quadlets: systemd-native unit files that give you declarative container management without a daemon or a compose file. If you’ve read my earlier Podman Quadlets guide, this follows the same architectural pattern - isolated backend network for the database, dual-attached application container, secrets managed through Podman’s secret store.

A note on support: Red Hat’s official documentation for the RHBK container image targets OpenShift as the supported deployment platform. Running RHBK on plain Podman is technically sound and works well, but it is not a Red Hat-supported configuration. If you need a fully supported deployment path, OpenShift is where Red Hat stands behind it. This guide is for those of us who want the RHBK image’s quality, security errata, and build lifecycle on a single-host Podman setup - and are comfortable with that trade-off.

Architecture Overview

The deployment consists of three components:

PostgreSQL 16 - Keycloak’s database backend, isolated on a private network
Keycloak 26 - The identity provider, connected to both backend and frontend networks
A dedicated backend network - Ensuring the database is never reachable from the outside

                    Internet
                       |
               Reverse Proxy (443)
                       |
          +------------+------------+
          |     frontend network    |
          +------------+------------+
                       |
                Keycloak Container
                   (Port 8080)
                       |
          +------------+------------+
          | keycloak-backend.network|
          |   (172.16.0.0/24)       |
          +------------+------------+
                       |
              PostgreSQL Container
                  (Port 5432)

Keycloak sits on both networks: it talks to PostgreSQL over the isolated backend, and your reverse proxy reaches it on the frontend. The database has no route to the outside world.

This guide assumes you already have a reverse proxy (Traefik, Caddy, nginx) handling TLS termination on your frontend network. If you need that piece, my earlier Quadlet article covers Traefik in detail.

Prerequisites

You’ll need:

RHEL 10, Fedora 43+, or any system with Podman 5.x+ and Quadlet support
An active Red Hat subscription for pulling from registry.redhat.io (this guide uses the Red Hat Build of Keycloak)
A reverse proxy already handling TLS on your frontend network
Root access for system-wide Quadlet deployment (or adapt for rootless with systemctl --user)

All Quadlet files go into /etc/containers/systemd/ for system-wide deployments.

Step 1: Create the Secrets

Never put passwords in unit files. Podman’s secret store keeps credentials encrypted and out of process listings:

# Generate and store the database password
pwgen -s 32 1 | tr -d '\n' | podman secret create keycloak_db_password -

# Generate and store the Keycloak admin password
pwgen -s 32 1 | tr -d '\n' | podman secret create keycloak_admin_password -

The tr -d '\n' is important. pwgen appends a trailing newline, and podman secret create stores the raw bytes verbatim. PostgreSQL’s init script strips the newline when setting the password, but Keycloak’s JDBC driver sends it as-is - resulting in a password mismatch that produces a confusing “authentication failed” error despite both containers reading the exact same secret.

Write down the admin password somewhere safe - you’ll need it for first login. After initial setup, you can create additional admin accounts through the Keycloak UI and remove the bootstrap admin.

Step 2: Backend Network

Create /etc/containers/systemd/keycloak-backend.network:

[Network]
NetworkName=keycloak-backend
Subnet=172.16.0.0/24
Gateway=172.16.0.1
IPRange=172.16.0.0/28

A dedicated subnet with an explicit IP range. The /28 range in IPRange limits DHCP assignments to 14 addresses - more than enough for a database and application container, and it prevents the network from being used as a dumping ground for unrelated containers.

Why not just use the default bridge? Network segmentation is the point. The database should only be reachable by containers that explicitly join this network. Defense-in-depth starts at the network layer.

Step 3: Database Container

Create /etc/containers/systemd/keycloak-db.container:

[Unit]
Description=Keycloak PostgreSQL database

[Container]
ContainerName=keycloak-db
Image=registry.redhat.io/rhel10/postgresql-16:latest
AutoUpdate=registry

# Isolated on backend network only - no frontend access
Network=keycloak-backend.network

# PostgreSQL configuration
Environment=POSTGRESQL_USER=keycloak
Environment=POSTGRESQL_DATABASE=keycloak

# Password injected at runtime from Podman secret store
Secret=keycloak_db_password,type=env,target=POSTGRESQL_PASSWORD

# Persistent storage with SELinux relabeling
Volume=/opt/keycloak/postgres:/var/lib/pgsql/data:z

# Health check - verify PostgreSQL is accepting connections
HealthCmd=/usr/libexec/check-container
HealthInterval=30s
HealthTimeout=5s
HealthRetries=3
HealthStartPeriod=30s

[Service]
Restart=always

[Install]
WantedBy=multi-user.target

Key points:

registry.redhat.io/rhel10/postgresql-16: The RHEL-based PostgreSQL image follows Red Hat’s support lifecycle and receives security errata. If you don’t have a Red Hat subscription, docker.io/postgres:16-alpine is a solid alternative - just adjust the environment variables (POSTGRES_USER, POSTGRES_DB, POSTGRES_PASSWORD), the internal data path (/var/lib/postgresql/data instead of /var/lib/pgsql/data), and the health check (pg_isready -U keycloak instead of /usr/libexec/check-container).
Single network attachment: The database lives exclusively on the backend network. It has no route to the frontend and cannot be reached from outside the backend subnet.
:z volume flag: Tells Podman to relabel the SELinux context for shared container access. Don’t skip this on SELinux-enforcing systems.
Health check: The RHEL PostgreSQL image ships with /usr/libexec/check-container. For the upstream image, use pg_isready -U keycloak instead.

Create the data directory and set permissions for the container’s PostgreSQL user (UID 26):

mkdir -p /opt/keycloak/postgres
setfacl -m u:26:rwx /opt/keycloak/postgres

The RHEL PostgreSQL container runs as UID 26 (postgres). Without this ACL, the container can’t initialize the data directory and will fail with permission errors. The :z SELinux flag handles labeling, but filesystem-level ownership is a separate concern - setfacl grants access without changing the directory’s ownership.

Password rotation: Red Hat’s PostgreSQL container resets the database password to match the POSTGRESQL_PASSWORD environment variable on each startup, so rotating the password is straightforward: update the Podman secret and recreate the container. Since Podman injects type=env secrets at container creation time, a simple systemctl restart isn’t enough - you need a fresh container creation to pick up the new secret value:

podman secret rm keycloak_db_password
pwgen -s 32 1 | tr -d '\n' | podman secret create keycloak_db_password -
systemctl stop keycloak-db.service keycloak-server.service
systemctl start keycloak-db.service keycloak-server.service

Step 4: Keycloak Application Container

Create /etc/containers/systemd/keycloak-server.container:

[Unit]
Description=Red Hat Build of Keycloak
After=keycloak-db.service
After=traefik.service
Requires=keycloak-db.service

[Container]
ContainerName=keycloak-server
Image=registry.redhat.io/rhbk/keycloak-rhel9:26.4-12
AutoUpdate=registry

# Dual network: backend for database, frontend for reverse proxy
Network=keycloak-backend.network
# Must match the network your reverse proxy (e.g. Traefik) is on
Network=frontend.network

# Database configuration
Environment=KC_DB=postgres
Environment=KC_DB_URL=jdbc:postgresql://keycloak-db:5432/keycloak
Environment=KC_DB_USERNAME=keycloak

# Proxy configuration - Keycloak runs behind a reverse proxy
Environment=KC_PROXY_HEADERS=xforwarded
Environment=KC_HTTP_ENABLED=true
Environment=KC_HOSTNAME_STRICT=false

# Hostname - set to your actual domain
Environment=KC_HOSTNAME=keycloak.example.com

# Enable health and metrics endpoints
Environment=KC_HEALTH_ENABLED=true
Environment=KC_METRICS_ENABLED=true

# Bootstrap admin user (used for first login only)
Environment=KC_BOOTSTRAP_ADMIN_USERNAME=admin

# Secrets - database and admin passwords
Secret=keycloak_db_password,type=env,target=KC_DB_PASSWORD
Secret=keycloak_admin_password,type=env,target=KC_BOOTSTRAP_ADMIN_PASSWORD

# Persistent storage for themes and providers
Volume=/opt/keycloak/providers:/opt/keycloak/providers:z
Volume=/opt/keycloak/themes:/opt/keycloak/themes:z
Volume=/etc/localtime:/etc/localtime:ro

# Run Keycloak in production mode
Exec=start

# Traefik labels for routing
Label="traefik.enable=true"
Label="traefik.docker.network=frontend"
Label="traefik.http.routers.rhbk.rule=Host(`keycloak.example.com`)"
Label="traefik.http.routers.rhbk.entrypoints=https"
Label="traefik.http.routers.rhbk.service=rhbk-svc"
Label="traefik.http.routers.rhbk.tls.certresolver=traefiktls"
Label="traefik.http.routers.rhbk.middlewares=secure-headers@file"
Label="traefik.http.services.rhbk-svc.loadbalancer.server.port=8080"

[Service]
Restart=always

[Install]
WantedBy=multi-user.target

There’s a lot happening here, so let’s break it down:

Network configuration: The container joins both keycloak-backend.network (to reach PostgreSQL by hostname keycloak-db) and the frontend network where your reverse proxy lives. Replace frontend.network with whatever network your Traefik (or other reverse proxy) container is on - the name must match, or Traefik won’t discover the container. The traefik.docker.network label must also match.

Database connection: Keycloak resolves keycloak-db via Podman’s built-in DNS on the backend network. The JDBC URL points directly to the container name - no IP addresses to manage.

Proxy settings: KC_PROXY_HEADERS=xforwarded tells Keycloak to trust X-Forwarded-* headers from your reverse proxy for correct URL generation, redirect URIs, and HTTPS detection. KC_HTTP_ENABLED=true allows plain HTTP on port 8080 since TLS terminates at the proxy.

KC_HOSTNAME_STRICT=false: This is set to false here to simplify initial setup, but the Keycloak documentation recommends true for production. With hostname-strict disabled, Keycloak will respond on any hostname forwarded by the proxy, which can create security issues if your reverse proxy doesn’t overwrite the Host header. Once you’ve verified that your reverse proxy correctly sets Host and X-Forwarded-* headers to your intended domain, switch this to true. If you’re using Traefik with the Host() rule shown above, Traefik already filters by hostname, so setting KC_HOSTNAME_STRICT=true from the start is safe.

Why start without --optimized? Keycloak 26 has a build phase and a runtime phase. The upstream community image ships pre-built, but the RHBK image does not - it needs to run the build step on first startup. Using start (without --optimized) lets Keycloak handle both phases automatically. This adds a few seconds to each startup, which is negligible for a service that starts once and runs continuously. If startup time matters to you (e.g., in a scaling scenario), you can build a custom image with kc.sh build baked in (see Step 5) and then switch to start --optimized.

Bootstrap admin: KC_BOOTSTRAP_ADMIN_USERNAME and KC_BOOTSTRAP_ADMIN_PASSWORD create an admin account on first startup. This account is meant for initial setup only. After logging in, create a proper admin user in the Keycloak UI and consider removing the bootstrap credentials.

No health check in this Quadlet: The RHBK image is based on UBI 9 Micro, which doesn’t include curl, wget, or any other HTTP client. A HealthCmd that relies on curl will always fail, marking the container as unhealthy - and Traefik ignores unhealthy containers by default. You can define a basic readiness check using Bash’s built-in /dev/tcp to probe the management port:

HealthCmd=bash -c 'echo > /dev/tcp/localhost/9000'
HealthInterval=30s
HealthTimeout=5s
HealthRetries=3
HealthStartPeriod=60s

This confirms the management port is accepting connections, but it’s a TCP check, not a full HTTP readiness probe. It won’t catch cases where Keycloak’s JVM is up but the application hasn’t finished initializing. Use it as a basic liveness signal, not as a gate for traffic routing. We omit it from the main Quadlet above to keep things simple and avoid Traefik interaction issues, but add it if you want systemd-level health visibility via systemctl status.

Traefik labels: The labels configure Traefik’s dynamic routing. traefik.docker.network must point to the shared frontend network so Traefik routes traffic over the right interface. The service label explicitly names the load balancer backend and maps it to port 8080. If you’re using a different reverse proxy, replace these labels with your proxy’s configuration (see the reverse proxy section below).

Systemd dependencies: Requires=keycloak-db.service ensures PostgreSQL is running before Keycloak attempts to connect - if the database service fails, Keycloak stops too. After=traefik.service ensures the reverse proxy is up before Keycloak starts, so Traefik can discover it immediately.

Create the data directories and set permissions. The RHBK container runs as UID 1000, so the bind-mounted directories need to be accessible to that user - just as we set up UID 26 for PostgreSQL:

mkdir -p /opt/keycloak/{providers,themes}
setfacl -m u:1000:rwx /opt/keycloak/providers /opt/keycloak/themes

Step 5: Build the Optimized Image (Optional)

If you need custom providers, custom themes baked into the image, or non-default database drivers, build a custom image:

FROM registry.redhat.io/rhbk/keycloak-rhel9:26.4-12 AS builder

# Set build-time configuration
ENV KC_DB=postgres
ENV KC_HEALTH_ENABLED=true
ENV KC_METRICS_ENABLED=true

# Add custom providers if needed
# COPY my-provider.jar /opt/keycloak/providers/

RUN /opt/keycloak/bin/kc.sh build

FROM registry.redhat.io/rhbk/keycloak-rhel9:26.4-12

COPY --from=builder /opt/keycloak/ /opt/keycloak/

Build with Podman:

podman build -t localhost/keycloak-custom:26.4 .

Then update the Quadlet to use Image=localhost/keycloak-custom:26.4, switch AutoUpdate=registry to AutoUpdate=local (Podman uses registry for remote refs and local for locally built images), and change Exec=start to Exec=start --optimized since the build step is already baked into the image.

For most deployments, the default RHBK image with start is sufficient. Use a custom image plus start --optimized when you need faster restarts or want providers and themes baked in rather than bind-mounted.

Step 6: Deployment

Reload systemd to pick up the new Quadlet files:

systemctl daemon-reload

Start the stack:

systemctl enable --now keycloak-db.service
systemctl enable --now keycloak-server.service

Verify everything is running:

# Check container status
systemctl status keycloak-db.service
systemctl status keycloak-server.service

# Watch Keycloak logs for successful startup
journalctl -u keycloak-server.service -f

You should see Keycloak log its startup sequence, database migration (on first run), and eventually:

Keycloak 26.4.10.redhat-00001 on JVM (powered by Quarkus 3.27.2.redhat-00001) started in 32.000s.
Listening on: http://0.0.0.0:8080. Management interface listening on http://0.0.0.0:9000.

Step 7: Reverse Proxy Notes

The Quadlet above includes Traefik labels that handle routing configuration declaratively - no separate Traefik config files needed. If you’re using Traefik, the Quadlet is self-contained.

If you’re using a different reverse proxy, remove the Traefik labels and configure routing manually. Keycloak listens on port 8080 for HTTP. Here are two common alternatives:

Caddy (Caddyfile snippet):

keycloak.example.com {
    reverse_proxy keycloak-server:8080
}

nginx (server block):

server {
    listen 443 ssl;
    server_name keycloak.example.com;

    location / {
        proxy_pass http://keycloak-server:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host $host;
        proxy_buffer_size 128k;
        proxy_buffers 4 256k;
        proxy_busy_buffers_size 256k;
    }
}

The larger proxy buffers for nginx may be necessary - Keycloak’s OIDC responses can include headers that exceed nginx’s conservative defaults. Without sufficient buffer space, you’ll see 502 Bad Gateway errors during authentication flows. The sizes above are generous; adjust downward if your setup works without them.

Both examples above assume the reverse proxy is containerized on the same frontend network, so Podman’s DNS resolves keycloak-server. If your reverse proxy runs directly on the host, you’ll need to publish Keycloak’s port (PublishPort=8080:8080 in the Quadlet) and use proxy_pass http://127.0.0.1:8080 instead.

Step 8: Automatic Updates

Enable Podman’s auto-update timer:

systemctl enable --now podman-auto-update.timer

With AutoUpdate=registry in both Quadlet files, Podman checks daily for new images and recreates containers when updates are available. Volumes persist across recreations, so your data is safe.

For Keycloak specifically, be deliberate about version updates. Pin to a specific tag like 26.4-12 (not latest) in the Quadlet and bump the version manually after reviewing the changelog. Keycloak upgrades can include database migrations that you want to run intentionally, not at 3 AM via an auto-update.

Navigate to https://keycloak.example.com and log in with the bootstrap admin credentials. From here:

Create a new realm for your applications (don’t use the master realm for end users)
Create a permanent admin user within the master realm
Set up your first client - this is the OIDC/SAML application registration
Configure identity providers if you want social login or federation with existing LDAP/AD

Once you have a proper admin account, clean up the bootstrap user:

Delete the bootstrap admin in the Keycloak UI: navigate to the master realm, go to Users, find the admin account, and delete it
Remove the environment variables from the Quadlet: delete the KC_BOOTSTRAP_ADMIN_USERNAME line, and the Secret=keycloak_admin_password line
Restart to apply:

systemctl daemon-reload
systemctl restart keycloak-server.service

Removing the environment variables alone doesn’t delete the user from the database - it only stops Keycloak from trying to create it on boot. Both steps are necessary.

Monitoring

Keycloak 26 exposes Prometheus-compatible metrics on the management port:

# From the host (or any container on the same network)
curl http://keycloak-server:9000/metrics

Point your Prometheus instance at this endpoint for dashboards covering authentication rates, token issuance, active sessions, and JVM health.

Health endpoints are equally useful for alerting:

# Liveness - is the process alive?
curl http://keycloak-server:9000/health/live

# Readiness - is it ready to handle requests?
curl http://keycloak-server:9000/health/ready

Operational Notes

Backups: The PostgreSQL data lives at /opt/keycloak/postgres. Use pg_dump inside the container for logical backups:

podman exec keycloak-db pg_dump -U keycloak keycloak > keycloak-backup.sql

Log management: All container output flows to journald. Standard journalctl filtering applies:

# Keycloak errors only
journalctl -u keycloak-server.service -p err

# Database logs since last boot
journalctl -u keycloak-db.service -b

Scaling considerations: This single-host setup handles a surprising number of users. Keycloak’s session cache is local by default. If you eventually need clustering, Keycloak supports Infinispan-based distributed caches - but that’s a different article.

A Note on the Upstream Keycloak Image

The upstream community project publishes images at quay.io/keycloak/keycloak. If you want to use the upstream image instead:

Image=quay.io/keycloak/keycloak:26.0

The configuration is identical - same environment variables, same startup flags. One difference: the upstream image ships pre-built, so you can use start --optimized directly without a custom build step.

The upstream images are community-maintained and don’t carry Red Hat’s security errata or support lifecycle. This guide uses RHBK for its build quality and predictable errata cadence, though as noted above, running RHBK on Podman (rather than OpenShift) is itself not a Red Hat-supported configuration. Choose whichever image fits your operational model.

References

Keycloak 26 Server Guide
Keycloak Container Image Documentation
Podman Quadlet Documentation
Red Hat Build of Keycloak Documentation
Production-Grade Container Deployment with Podman Quadlets - my earlier deep-dive into Quadlet-based deployments

My Multi-Stage Backup Strategy: ZFS, Proxmox, and Paranoia

2026-03-28T00:00:00+01:00

Everybody has a backup strategy. Some people’s backup strategy is “I should really set up backups.” Others involve a cronjob that worked once in 2019 and has been silently failing ever since. A select few involve a USB drive in a desk drawer labeled “IMPORTANT” in permanent marker.

Mine layers ZFS snapshots, off-site replication, a Proxmox Backup Server that’s really just a proxy in front of cheap object storage, a Podman container doing something that would make a packaging engineer cry, and a dead man’s switch that pages me if any of it stops working. It’s not elegant in every layer, but it covers roughly a dozen servers across FreeBSD and Linux, and it lets me sleep at night.

The classic 3-2-1 backup rule says: three copies of your data, on two different media types, with one copy off-site. What I’m about to describe exceeds it in some areas and only meets it in others - but the principle is sound, and if you take nothing else from this article, take 3-2-1.

Before diving into tools, it helps to name the threats each layer addresses:

Accidental deletion or bad deploys → local ZFS snapshots (instant rollback)
Disk or pool failure → off-site replication to rsync.net
Total host loss → remote backups via PBS + S3 and syncoid
Silent backup failure → dead man’s switch monitoring
Corrupt or incomplete backups → quarterly restore tests

Every layer below maps to at least one of these. If you’re designing your own strategy, start from your threat model and work outward - the tools are interchangeable, the thinking isn’t.

Table of Contents
Layer 1: ZFS Snapshots with Sanoid (FreeBSD)
Layer 2: Off-Site Replication with Syncoid
- The SSH Configuration
Layer 3: Proxmox Backup Server for Virtual Machines
Layer 4: Linux Servers
The Glue: Dead Man’s Switch Monitoring
Quarterly Restore Tests
- How Restores Actually Work
Putting It All Together
What This Costs
References

Layer 1: ZFS Snapshots with Sanoid (FreeBSD)

All my FreeBSD servers run ZFS. If you’re on FreeBSD and not using ZFS, I have questions. (If you want answers about ZFS first, I wrote an entire article about it.)

ZFS snapshots are the first line of defense. They’re instant, they’re effectively free at creation time (copy-on-write means a snapshot consumes zero additional space until data changes), and they’ve saved me from “oops” moments more times than I care to admit. But snapshots alone aren’t backups - they live on the same pool as your data. If the disk dies, the snapshots die with it.

Enter sanoid, a policy-driven snapshot management tool. On FreeBSD, it’s a pkg install sanoid away. You define retention policies per dataset, point sanoid at them, and it handles creation and pruning automatically.

Here’s a representative configuration from one of my servers:

# /usr/local/etc/sanoid/sanoid.conf

[template_production]
    autosnap = yes
    autoprune = yes

    # Retention Policy
    hourly = 0
    daily = 14
    weekly = 4
    monthly = 0
    yearly = 0

[template_database]
    autosnap = yes
    autoprune = yes

    # Retention Policy
    hourly = 0
    daily = 3
    weekly = 0
    monthly = 0
    yearly = 0

# Apply to datasets
[zroot]
    use_template = production
    recursive = yes
    process_children_only = no

[zroot/postgres_data]
    use_template = database
    recursive = yes
    process_children_only = no

# Exclude datasets that don't need snapshots
[zroot/tmp]
    autosnap = no

[zroot/bastille/logs]
    autosnap = no

[zroot/bastille/jails/nginx/cache]
    autosnap = no

The philosophy is straightforward: production data gets 14 daily snapshots and 4 weekly ones. That’s about six weeks of rollback capability. Database datasets get a shorter window (3 days) because database snapshots without coordinated dumps are crash-consistent at best - useful, but not a replacement for proper application-level backups. Temporary directories and caches get nothing, because snapshotting /tmp is a waste of everyone’s time.

Sanoid runs via cron:

0 0 * * * /usr/local/bin/sanoid --cron

This pattern is identical across all ~10 of my FreeBSD servers. Same templates, same retention policies, same cron entry. The only thing that changes is which datasets exist and which exclusions apply. Consistency is a feature - when you’re managing multiple hosts, you don’t want to remember that this server has different retention because you were feeling creative at 2 AM six months ago.

Layer 2: Off-Site Replication with Syncoid

Local snapshots protect against accidental deletion and bad updates. They do absolutely nothing against disk failure, ransomware, or the catastrophic scenario where the entire server goes away. For that, you need off-site copies.

Syncoid (part of the sanoid suite) handles this. It uses zfs send/recv over SSH to replicate datasets to a remote ZFS pool. The first run sends a full copy; subsequent runs send only incremental differences. For a server with hundreds of gigabytes of data, the nightly incremental transfer is typically measured in megabytes.

I use rsync.net as my off-site target. They offer ZFS-capable storage specifically designed for this use case - you get a remote ZFS pool that accepts zfs recv over SSH. It’s not the cheapest option - but “cheap” and “I trust this with my only off-site copy of everything” are different categories.

Here’s the backup script that runs nightly:

#!/bin/sh

# Configuration
SOURCE_POOL="zroot"
TARGET="backup@ab1234c.rsync.net:data1/apollo/zroot"
MONITOR_URL="https://uptime.example.com/api/push/xxxxxxxxxx?status=up&msg=OK&ping="

echo "Starting off-site backup to rsync.net..."

/usr/local/bin/syncoid --no-sync-snap --no-privilege-elevation --recursive \
    --sendoptions="w" \
    --exclude="zroot/tmp" \
    --exclude="zroot/bastille/logs" \
    --exclude="zroot/bastille/jails/nginx/cache" \
    --recvoptions="u o canmount=off o mountpoint=none" \
    "${SOURCE_POOL}" "${TARGET}"

if [ $? -eq 0 ]; then
    echo "Backup successful. Triggering monitor..."
    /usr/local/bin/curl -fsS "${MONITOR_URL}"
else
    echo "ERROR: Backup FAILED. Monitor NOT triggered."
    exit 1
fi

A few things worth noting:

--sendoptions="w" is critical. The w flag means “raw send” - encrypted datasets are sent in their encrypted form. Without this, zfs send would transmit the decrypted data stream, and the datasets would arrive unencrypted at rest on the remote end. With raw send, your encrypted datasets remain encrypted at rest on the backup target. rsync.net never needs your encryption keys, and if someone compromises the backup storage, they get ciphertext.

--no-privilege-elevation means syncoid won’t try to use sudo on the remote end. This works because the remote user (backup in this example) has been granted only the specific ZFS permissions needed to receive snapshots - nothing more.

--recvoptions="u o canmount=off o mountpoint=none" tells the receiving end not to mount the datasets. This is a backup target, not a live system. You don’t want zfs recv trying to mount zroot on the remote machine.

The same excludes from the sanoid config reappear here. Yes, the duplication is intentional - each layer should be independently correct, so if you change one without the other, neither silently breaks. There’s no point in replicating temp files and cache directories across the internet.

The cron schedule runs this at 4 AM, after sanoid has already created the nightly snapshots at midnight:

0 0 * * * /usr/local/bin/sanoid --cron
0 4 * * * /usr/local/bin/daily_backup.sh > /var/log/backup_last.log 2>&1

Retention of old snapshots on the rsync.net side is handled by a separate sanoid cron job running on the rsync.net host itself:

0 5 * * * /usr/local/bin/sanoid --verbose --prune-snapshots

This keeps the remote snapshot history tidy without me having to manage it from every source server.

The SSH Configuration

The backup user connects via a dedicated SSH key with limited permissions:

# ~/.ssh/config (on the source server)
Host rsync_backup
    HostName ab1234c.rsync.net
    User backup
    Port 22
    IdentityFile /root/.ssh/id_ed25519_backup
    Compression yes

The backup user on rsync.net has only the ZFS permissions required for receiving snapshots. If someone compromises a source server and steals the SSH key, the worst they can do is send snapshots to the backup pool and list existing ones. They can’t delete snapshots, destroy datasets, or access anything outside the designated backup path. This is the ransomware angle: even if an attacker has full root on a source server, they cannot destroy or encrypt the off-site backups.

Layer 3: Proxmox Backup Server for Virtual Machines

Not everything is FreeBSD. I run two Proxmox VE nodes that serve as lab environments for testing things quickly, plus a handful of utility VMs. These need a different backup strategy.

My approach: a Proxmox Backup Server (PBS) instance running on a minimal VM (1 vCPU, 1 GB RAM) at a cheap cloud provider, with Backblaze B2 as the storage backend.

The beauty of this setup is that PBS supports S3-compatible object storage as a datastore. The PBS VM itself is tiny and stores almost nothing locally - it’s essentially a proxy that handles deduplication, encryption, and the Proxmox backup protocol, then shoves the actual data into B2. You get Proxmox’s excellent incremental backup and deduplication without paying for a beefy VM with terabytes of local storage.

On the Proxmox VE side, this is completely transparent. You add the PBS as a storage target in the Proxmox web UI, configure a backup schedule, and the VMs back up nightly through the standard Proxmox backup mechanism. The PVE nodes don’t know or care that B2 is the actual storage behind it - they just see a PBS endpoint.

The economics work out nicely: a tiny VPS to run PBS costs a few euros per month, and B2 storage is $6/TB/month with free egress via the Cloudflare Bandwidth Alliance. For a lab environment with a few hundred gigabytes of VM data, the total cost is negligible.

Layer 4: Linux Servers

Linux servers are the messy middle child of this strategy. There’s no single approach, because the servers themselves vary too much.

Option A: Proxmox Backup Client

For Linux hosts where it’s available, proxmox-backup-client is the cleanest option. It supports file-level and image-level backups to PBS with deduplication and encryption. Install the client, point it at your PBS instance, schedule a cron job, done. These backups land in the same PBS + B2 infrastructure described above.

Option B: Plain rsync

Not every Linux server needs the full PBS treatment. Some are simple workhorses where the important data is just files - configuration, application data, maybe some database exports. For these, rsync over SSH to rsync.net does the job.

The key is what happens before rsync runs. You can’t just rsync a live PostgreSQL data directory and expect a consistent backup. The same goes for MySQL, Redis AOF files, or anything else that keeps state in memory. So the backup script follows a pattern: dump first, then sync.

#!/bin/bash
set -euo pipefail

BACKUP_DIR="/var/backups/pre-rsync"
RSYNC_TARGET="backup@ab1234c.rsync.net:data1/hostname/"
SSH_KEY="/root/.ssh/id_ed25519_backup"
MONITOR_URL="https://uptime.example.com/api/push/xxxxxxxxxx?status=up&msg=OK&ping="

mkdir -p "${BACKUP_DIR}"

# Dump databases before syncing
pg_dump -U postgres myapp > "${BACKUP_DIR}/myapp.sql"
# Or for MySQL:
# mysqldump --single-transaction --all-databases > "${BACKUP_DIR}/all-databases.sql"

# Export container volumes (Podman or Docker)
podman volume export myapp-data > "${BACKUP_DIR}/myapp-data.tar"

# Sync everything to rsync.net
rsync -avz --delete -e "ssh -i ${SSH_KEY}" \
    /etc/ "${RSYNC_TARGET}/etc/"

rsync -avz --delete -e "ssh -i ${SSH_KEY}" \
    /var/backups/pre-rsync/ "${RSYNC_TARGET}/dumps/"

rsync -avz --delete -e "ssh -i ${SSH_KEY}" \
    /opt/app-data/ "${RSYNC_TARGET}/app-data/"

# If we got here, everything succeeded (set -e exits on any failure)
curl -fsS "${MONITOR_URL}"

The database dump ensures you get a consistent, point-in-time export rather than a half-written data directory. podman volume export gives you a clean tarball of container volumes - useful when your application runs in containers and the actual state lives in named volumes rather than bind mounts.

For containers specifically, don’t forget that the compose file or quadlet unit and any environment files are just as important as the volume data. Losing your database is bad; losing your database and the configuration that tells you how it was deployed is worse.

The rsync.net destination is just a directory on a ZFS filesystem, so you get the benefit of rsync.net’s own ZFS snapshots on the remote side. Even if --delete removes a file during sync, the previous version survives in a snapshot. It’s belt and suspenders, and it costs nothing extra.

Boring? Yes. Reliable? Also yes. Sometimes the best backup tool is the one that’s been working since 1996.

Option C: The Podman Trick (When Packages Don’t Exist)

And then there’s the creative part.

I have a RHEL 9 server running Forgejo (a self-hosted Git forge) in Podman containers. This is a production system with SELinux in enforcing mode. I wanted to back it up to my existing PBS infrastructure, but here’s the problem: proxmox-backup-client doesn’t ship as an RPM for RHEL. There’s no package in EPEL, no Copr repo, nothing in the Proxmox repositories for Red Hat-based systems.

The solution? Run proxmox-backup-client inside a Podman container, and pass the entire root filesystem in read-only:

podman run --rm \
    --name pbs-backup-job \
    --security-opt label=type:spc_t \
    --hostname hermes \
    --entrypoint /usr/bin/proxmox-backup-client \
    --secret pbs_repo_password,type=env,target=PBS_PASSWORD \
    --secret pbs_key_passphrase,type=env,target=PBS_ENCRYPTION_PASSWORD \
    -v /:/mnt/root:ro \
    -v /root/backup.key:/etc/proxmox/backup.key:ro,Z \
    docker.io/ayufan/proxmox-backup-server:v3.4.1 \
    backup root.pxar:/mnt/root \
        --repository backup@pbs@backup-vps.example.com:data \
        --keyfile /etc/proxmox/backup.key

Let’s break down why this actually works well:

--security-opt label=type:spc_t is the key ingredient on a SELinux system. The spc_t (super privileged container) type allows the container to read host files that would normally be blocked by SELinux labels. Without this, the container would get a wall of “permission denied” errors trying to read the host filesystem, even though it’s mounted read-only. If you’re running SELinux in enforcing mode (and you should be), this flag is non-negotiable.

-v /:/mnt/root:ro mounts the entire host root filesystem into the container as read-only. The container can see everything but modify nothing. The backup client reads files from /mnt/root and creates a root.pxar archive (Proxmox’s archive format) that gets sent to PBS.

Podman secrets handle the credentials. The PBS repository password and encryption key passphrase are stored as Podman secrets and injected as environment variables - they never appear in the command line, process list, or container inspect output.

--hostname hermes sets the hostname inside the container to match the actual host. PBS uses the hostname to identify backup sources, so this ensures the backups show up correctly in the PBS UI.

The result: a full file-level backup of a RHEL server to Proxmox Backup Server, with encryption, deduplication, and incremental transfers - all without installing a single package on the host. It runs nightly via a systemd timer, and it has been rock-solid.

This is absolutely a workaround, not a vendor-supported path. But it’s contained, repeatable, and has proven stable in practice. Sometimes the ugliest solutions are the most reliable ones.

The Glue: Dead Man’s Switch Monitoring

The most dangerous failure mode for backups isn’t failure. It’s silent failure. The backup that stopped working three months ago and nobody noticed until the restore attempt. By the time you need it, you discover that your last good backup is from February and it’s now April.

This is why every backup script ends with a monitoring ping. I run an Uptime Kuma instance that monitors these pings via push-based checks (sometimes called dead man’s switches or heartbeat monitors). The logic is inverted from normal monitoring: instead of checking whether a service is up, it checks whether a backup has reported in.

Each backup job pings a unique URL on success:

curl -fsS "https://uptime.example.com/api/push/xxxxxxxxxx?status=up&msg=OK&ping="

If the backup fails, the curl never fires. If the backup script doesn’t run at all (cron misconfigured, server down, disk full, whatever), the curl never fires. Uptime Kuma is configured with a 28-hour timeout (24 hours for the daily schedule + 4 hours of tolerance). If 28 hours pass without a ping, I get an alert.

This catches every failure mode: - Backup script crashes → no ping → alert - Cron job stops running → no ping → alert - Server goes offline → no ping → alert - SSH key expires → backup fails → no ping → alert - Remote target is full → backup fails → no ping → alert

And yes - who monitors the monitor? Uptime Kuma itself is monitored by an external service (a simple HTTP check), so if it goes down, I know about that too.

The only thing push monitoring doesn’t catch is a backup that appears to succeed but produces corrupt or incomplete data. For that, you need test restores.

Quarterly Restore Tests

A mentor told me at one of my first jobs: “Untested backups are just expensive hopes and dreams.” That stuck with me, and I take it seriously.

Every quarter, I run full restore tests for every machine I consider production. These are scheduled in my calendar as recurring appointments, and I treat them with the same priority as any other production maintenance window. The process is straightforward: spin up a fresh VM, restore the system entirely from backups, and verify that it comes up in a functionally working state - services running, data intact, applications responding.

I record every restore session with asciinema and document the results in my DokuWiki, with the terminal recording attached. This gives me three things at once: confidence that the backups actually work, regular practice with the restore procedure so I’m not fumbling through it for the first time during an actual emergency, and a living library of documentation I can refer to when things go wrong at 3 AM. Under pressure is the worst possible time to be reading man pages and guessing at flags. Having a recording of yourself successfully restoring the exact same system last quarter is worth more than any runbook.

The quarterly cadence is a compromise. Monthly would be better but hard to justify the time. Yearly would be negligent. Quarterly is frequent enough that procedural drift doesn’t accumulate, and any backup issue gets caught within 90 days at most.

How Restores Actually Work

Since each layer uses different tools, the restore path differs per layer. A quick overview:

ZFS local snapshots → zfs rollback to revert a dataset in place, or zfs clone to mount a snapshot as a separate dataset for selective file recovery without affecting the live system
Syncoid / rsync.net → zfs send from the remote pool back to the local host via SSH. Same tool, reverse direction. For a full rebuild: install the OS, create the pool, zfs recv the datasets
PBS (Proxmox Backup Server) → restore via the PBS web UI or proxmox-backup-client restore on the CLI. For VMs on Proxmox VE, the restore is a one-click operation from the PVE interface
Plain rsync → copy the dumps and files back, re-import database dumps with psql or mysql, re-create container volumes from the tarballs. The most manual path, but also the most transparent

This is exactly what I walk through during the quarterly restore tests. Each layer’s restore procedure is documented and recorded, so I’m not figuring it out under pressure.

Putting It All Together

Here’s the full picture:

┌─────────────────────────────────────────────────────────────┐
│                     FreeBSD Servers (~10)                   │
│                                                             │
│  sanoid (local snapshots) ──► syncoid ──► rsync.net (ZFS)   │
│          daily/weekly              nightly    5 TB pool     │
│          retention                 encrypted  remote prune  │
└─────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────┐
│                  Proxmox VE Nodes (2)                       │
│                                                             │
│  PVE backup schedule ──► PBS VM ──► Backblaze B2 (S3)       │
│       nightly              1C/1G    deduplicated            │
│       incremental          proxy    encrypted               │
└─────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────┐
│                     Linux Servers                           │
│                                                             │
│  proxmox-backup-client ──► PBS ──► Backblaze B2             │
│  rsync over SSH ──► rsync.net                               │
│  podman + pbs-client ──► PBS  (RHEL/no native package)      │
└─────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────┐
│                      Monitoring                             │
│                                                             │
│  Every backup job ──► Uptime Kuma (push / dead man switch)  │
│                        28h timeout ──► alert on silence     │
└─────────────────────────────────────────────────────────────┘

What This Costs

One of the most common excuses for not having proper backups is cost. So let’s put actual numbers on this:

Component	What It Does	Cost
Scaleway STARDUST1-S (IPv6, 1 GB RAM, 1 vCPU, 10 GB disk)	Runs Proxmox Backup Server	€0.10/month
Backblaze B2 object storage	Stores VM backups from PBS	$6/TB/month
rsync.net ZFS account (5 TB)	Off-site ZFS replication target for all FreeBSD servers	$60/month

The Scaleway instance is almost comically cheap. It’s a STARDUST1-S - the smallest instance they offer - and it’s more than enough for PBS, because PBS is doing deduplication and proxying to S3, not storing data locally. Ten cents a month. That’s not a typo.

Backblaze B2 scales linearly with what you store. For a lab environment with a few hundred gigabytes of VM data after deduplication, you’re looking at low single digits per month. Egress is free if you route through Cloudflare’s Bandwidth Alliance, which matters a lot if you ever actually need to restore.

The rsync.net line item looks expensive at first glance - $60/month for 5 TB. But you’re not just buying storage. You get a fully managed FreeBSD virtual host with root access, sitting on top of a tolerant raidz3 pool with ZFS snapshots. rsync.net handles all hardware maintenance, OS patches, upgrades, and provides unlimited support. You can optionally add geo-redundancy. For a managed service with that feature set and reliability, it’s genuinely good value. Try pricing a comparable setup at any other provider where you get a dedicated ZFS pool that accepts zfs recv natively - the options are slim, and they aren’t cheaper.

All in, the entire backup infrastructure for about a dozen servers costs roughly $70/month. That’s less than one hour of emergency consulting when you lose data you can’t recover.

Is it perfect? No. There are things I’d improve: better alerting on partial failures, maybe a dashboard that shows backup age across all hosts at a glance, and more automation around the restore tests. But the fundamentals are solid: every server has local snapshots for fast rollback, every server has off-site copies for disaster recovery, every backup job is monitored, restores are tested quarterly, and the whole thing runs unattended.

The most important property of a backup strategy isn’t that it’s clever. It’s that it’s boring, consistent, and actually running. Fancy architectures that you never finish implementing protect exactly zero bytes of data. A simple cron job that runs every night and phones home when it’s done? That protects everything.

References

Shell Tricks That Actually Make Life Easier (And Save Your Sanity)

2026-03-26T00:00:00+01:00

There is a distinct, visceral kind of pain in watching an otherwise brilliant engineer hold down the Backspace key for six continuous seconds to fix a typo at the beginning of a line.

We’ve all been there. We learn ls, cd, and grep, and then we sort of… stop. The terminal becomes a place we live in-but we rarely bother to arrange the furniture. We accept that certain tasks take forty keystrokes, completely unaware that the shell authors solved our exact frustration sometime in 1989.

Here are some tricks that aren’t exactly secret, but aren’t always taught either. To keep the peace in our extended Unix family, I’ve split these into two camps: the universal tricks that work on almost any POSIX-ish shell (like sh on FreeBSD or ksh on OpenBSD), and the quality-of-life additions specific to interactive shells like Bash or Zsh.

The “Works (Almost) Everywhere” Club

These tricks rely on standard terminal line disciplines, generic Bourne shell behaviors, or POSIX features. If you SSH into an embedded router from 2009, a fresh OpenBSD box, or a minimal Alpine container, these will still have your back.

The Backspace Replacements

Why shuffle character-by-character when you can teleport? These are standard Emacs-style line-editing bindings (via Readline or similar), enabled by default in most modern shells.

CTRL + W: You’re typing /var/log/nginx/ but you actually meant /var/log/apache2/. You have two choices: hold down Backspace until your soul leaves your body, or hit CTRL + W to instantly delete the word before the cursor. Once you get used to this, holding Backspace feels like digging a hole with a spoon.

CTRL + U and CTRL + K: You typed out a beautifully crafted, 80-character rsync command, but suddenly realize you need to check if the destination directory actually exists first. You don’t want to delete it, but you don’t want to run it. Hit CTRL + U to cut everything from the cursor to the beginning of the line. Check your directory, and then hit CTRL + Y to paste (“yank”) your masterpiece right back into the prompt. (CTRL + K does the same thing, but cuts from the cursor to the end of the line.)

CTRL + A and CTRL + E: Jump instantly to the beginning (A) or end (E) of the line. Stop reaching for the Home and End keys; they are miles away from the home row anyway.

ALT + B and ALT + F: Move backward (B) or forward (F) one entire word at a time. It’s the arrow key’s much faster, much cooler sibling. (Mac users: you usually have to tweak your terminal settings to use Option as Meta for this to work).

The “Oh No, Binary Output” Fix

reset (or stty sane): While strictly more of a terminal recovery tip than an interactive shell trick, it belongs here. We’ve all done it: you meant to cat a text file, but you accidentally cat a compiled binary or a compressed tarball. Suddenly, your terminal is spitting out ancient runes and Wingdings, and your prompt is completely illegible. Instead of closing the terminal window in shame, type reset (even if you can’t see the letters you’re typing) and hit enter. Your terminal will heal itself.

The Emergency Exits

CTRL + C: Cancel the current command immediately. Your emergency exit when a command hangs, or you realize you’re tailing the wrong log file.

CTRL + D: Sends an EOF (End of File) signal. If you’re typing input to a command that expects it, this closes the stream. But if the command line is empty, it logs you out of the shell completely-be careful where you press it.

The Screen Cleaner

CTRL + L: Your terminal is cluttered with stack traces, compiler spaghetti, and pure digital noise. Running the clear command works, but what if you’re already halfway through typing a new command? CTRL + L wipes the slate clean, throwing your current prompt right up to the top without interrupting your train of thought.

The Previous Directory Ping-Pong

cd -: The classic channel-flipper. You’re deep down in /usr/local/etc/postfix and you need to check something in /var/log. You type cd /var/log, look at the logs, and now you want to go back. Instead of typing that long path again, type cd -. It switches you to your previous directory. Run it again, and you’re back in logs. Perfect for toggling back and forth.

pushd and popd: If cd - is a toggle switch, pushd is a stack. Need to juggle multiple directories? pushd /etc changes to /etc but saves your previous directory to a hidden stack. When you’re done, type popd to pop it off the stack and return exactly where you left off.

The Instant Truncate

> file.txt: This empties a file completely without deleting and recreating it. Why does this matter? It preserves file permissions, ownership, and doesn’t interrupt processes that already have the file open. It’s much cleaner than echo "" > file.txt (which actually leaves a newline character) or rm file && touch file.

The “Last Argument” Variable

$_: In most shells, $_ expands to the last argument of the previous command-especially useful interactively or in simple scripts when you need to operate on the same long path twice:

mkdir -p /some/ridiculously/long/path/newdir && cd "$_"

No more re-typing paths or declaring temporary variables to enter a directory you created a second ago.

Scripting Sanity Savers

If you are writing shell scripts, put these at the top immediately after your shebang. It will save you from deploying chaos to production.

set -e: Exit on error. Very useful, but notoriously weird with edge cases (especially inside conditionals like if statements, while loops, and pipelines). Don’t rely on it blindly as it can create false confidence. (Pro-tip: consider set -euo pipefail for a more robust safety net, but learn its caveats first.)
set -u: Treats referencing an unset variable as an error. This protects you from catastrophic disasters like rm -rf /usr/local/${MY_TYPO_VAR}/* accidentally expanding into rm -rf /usr/local/*.

The Bash & Zsh Comfort Zone

If you’re on a Linux box or using a modern interactive shell, these are the tools that make the CLI feel less like a rusty bicycle and more like something that actually responds when you steer.

The History Hunter

CTRL + R: Reverse incremental search. Stop pressing the up arrow forty times to find that one awk command you used last Tuesday. Press CTRL + R, start typing a keyword from the command, and it magically pulls it from your history. Press CTRL + R again to cycle backwards through matches.

The “Oops, Sudo” Move

!!: This expands to the entirety of your previous command. Its most famous use case is the “Permission denied” walk of shame. You confidently type systemctl restart nginx, hit enter, and the system laughs at your lack of privileges. Instead of retyping it, run:

sudo !!

It’s your way of telling the shell, “Do what I said, but this time with authority.”

The Ultimate Editor Escape Hatch

CTRL + X, then CTRL + E: You start typing a quick one-liner. Then you add a pipe. Then an awk statement. Soon, you’re editing a four-line monster inside your prompt and navigation is getting difficult. Hit CTRL + X followed by CTRL + E (in Bash; in Zsh, this needs configuring). This drops your current command into your default text editor (like Vim or Nano). You can edit it with all the power of a proper editor, save, and exit. The shell then executes the command instantly.

fc: The highly portable, traditional sibling to CTRL+X CTRL+E. Running fc opens your previous command in your $EDITOR. It works across most shells and is a fantastic hidden gem for fixing complex, multi-line commands that went wrong.

The Last Argument (Interactive Edition)

ESC + . (or ALT + .): Inserts the last argument of the previous command right at your cursor. Press it repeatedly to cycle further back through your history, dropping the exact filename or parameter you need right into your current command.

!$: The non-interactive sibling of ESC + .. Unlike ESC + . (which inserts the text live at your cursor for you to review or edit), !$ expands blindly at the exact moment you hit enter. (Pro-Tip: For scripting or standard sh, use the $_ variable mentioned earlier instead!)

The Renaming Trick & Brace Expansion

Brace expansion is pure magic for avoiding repetitive typing, especially when doing quick backups or renames.

The Backup Expansion: Need to edit a critical config file and want to make a quick backup first?

cp pf.conf{,.bak}

The shell expands this seamlessly into cp pf.conf pf.conf.bak.

The Rename Trick:

mv filename.{txt,md}

This expands to mv filename.txt filename.md. Fast, elegant, and makes you look like a wizard.

Need multiple directories? mkdir -p project/{src,tests,docs} creates all three at once.

Process Substitution

<(command): Treats the output of a command as if it were a file. Say you want to diff the sorted versions of two files. Traditionally, you’d sort them into temporary files, diff those, and clean up. Process substitution skips the middleman:

diff <(sort file1.txt) <(sort file2.txt)

The Ultimate Glob

** (Globstar): find is a great command, but sometimes it feels like overkill. If you run shopt -s globstar in Bash (it’s enabled by default in Zsh), ** matches files recursively in all subdirectories. Need to find all JavaScript files in your current directory and everything beneath it?

ls **/*.js

No find command required.

Backgrounding and Disowning

CTRL + Z, then bg, then disown: You started a massive, hour-long database import task, but you forgot to run it in tmux or screen. It’s tying up your terminal, and if your SSH connection drops, the process dies. Panic sets in.

Hit CTRL + Z to suspend (pause) the process.
Type bg to let it resume running in the background. Your prompt is free!
Type disown to detach it from your shell entirely. You can safely close your laptop, grab a coffee, and the process will survive.

The Everything-Logger

command |& tee file.log: Standard pipes (|) only catch standard output (stdout). If a script throws an error (stderr), it skips the pipe and bleeds directly onto your screen, missing the log file. |& pipes both stdout and stderr (it’s a helpful shorthand for 2>&1 |).

Throw in tee, and you get to watch the output on your screen while simultaneously saving it to a log file. It’s the equivalent of watching live TV while recording it to your DVR.

The shell is a toolbox, not an obstacle course. You don’t need to memorize all of these today. Pick just one trick, force it into your daily habits for a week, and then pick another. Stop letting the terminal push you around, and start rearranging the furniture. It’s your house now.

Dual-FIB Policy Routing on FreeBSD: Two Upstreams, One Server, Zero Confusion

2026-03-23T00:00:00+01:00

The previous articles in this series covered the BGP router side: obtaining an AS, peering with upstream providers, and building a tunnel overlay. This article covers the other end - the downstream server that consumes that tunnel and needs to serve traffic from two completely different IP ranges through two completely different paths, simultaneously, without either one interfering with the other.

The server in question is radon, a Netcup VPS running FreeBSD with Bastille jails. It has two internet uplinks: a physical connection to Netcup’s network and a GIF tunnel to my BGP router (hobgp). Jails on this server use three distinct routing paradigms - private NAT, natively routed BGP IPv6, and pure public routed BGP IPv4 - all on the same bridge interface.

The mechanism that makes this work is dual-FIB policy routing: two independent routing tables in the kernel, with PF deciding which table handles which traffic based on source address. It’s elegant once you understand it, and surprisingly simple to configure once you’ve seen it done.

The Problem: One Server, Two Gateways

A server with a single default gateway has simple routing: everything goes out the same door. But what happens when you want traffic from different source addresses to take different paths?

On radon, the physical interface (vtnet0) is assigned Netcup’s IP space: 152.53.147.5 for IPv4 and 2a0a:4cc0:c1:2f90::2 for IPv6. The default gateway points to Netcup’s router. Standard traffic - package updates, DNS queries, NATed jail traffic - flows through this path.

But radon also has addresses from my own AS201379: 194.28.98.217 (shared via loopback for port forwarding), 194.28.98.216 (directly assigned to a jail), and the entire 2a06:9801:1c:1000::/64 IPv6 subnet routed to jails. This address space is announced to the internet through my BGP router. Traffic arriving for these addresses traverses the BGP tunnel. If replies to that traffic exit through Netcup’s default gateway instead of the tunnel, the provider drops them as spoofed - the source IP doesn’t belong to Netcup’s network.

The kernel’s single routing table (FIB 0) knows one default route: Netcup. It has no concept of “this source address should use a different exit.” That’s where dual-FIB comes in.

Architecture Overview

                        ┌─────────────────────────────────────────────┐
                        │           Public Internet                   │
                        └────────┬──────────────────────┬─────────────┘
                                 │                      │
                         Netcup Network           BGP (AS201379)
                         152.53.144.0/22          via hobgp router
                                 │                      │
                                 │               ┌──────┴──────────┐
                                 │               │  hobgp (Core)   │
                                 │               │  128.140.64.181 │
                                 │               └──────┬──────────┘
                                 │                      │
                                 │              GIF tunnel (proto 41)
                                 │              IPv6-in-IPv4 encap
     ┌───────────────────────────┴──────────────────────┴─────────────────────┐
     │                        radon.edelga.se                                 │
     │                                                                        │
     │  vtnet0 ──── FIB 0 (Default)                                           │
     │  152.53.147.5/22        <- Standard internet, NAT, mgmt                │
     │  2a0a:4cc0:c1:2f90::2/64                                               │
     │                                                                        │
     │  gif0 ────── FIB 1 (Tunnel)                                            │
     │  194.28.98.216/29       <- BGP-addressed traffic                       │
     │  2a06:9801:1c:ffff::2/128                                              │
     │                                ┌─────────────────────────────────────┐ │
     │  bastille0 ════════════════════│        Jail Bridge                  │ │
     │  10.254.254.1/24               │                                     │ │
     │  2a06:9801:1c:1000::1/64       │  ┌────────┐  ┌─────────────────────┐│ │
     │                                │  │ Caddy  │  │ testvnet            ││ │
     │  lo0:                          │  │ .254.10│  │194.28.98.216        ││ │
     │  194.28.98.217/32 (shared)     │  │ (NAT)  │  │2a06:9801:1c:1000::99││ │
     │                                │  └────────┘  │   (Pure Public)     ││ │
     │                                │              └─────────────────────┘│ │
     │                                └─────────────────────────────────────┘ │
     └────────────────────────────────────────────────────────────────────────┘

The key insight: FIB 0 handles “normal” traffic (Netcup-addressed). FIB 1 handles BGP-addressed traffic (AS201379). PF is the glue that assigns packets to the correct FIB based on source or destination address.

The Physical Upstream: Netcup (FIB 0)

The physical interface lives in the default routing table. This is the server’s “normal” internet connection:

# Primary physical interface - Netcup VPS
ifconfig_vtnet0="inet 152.53.147.5 netmask 255.255.252.0 -lro -tso"
defaultrouter="152.53.144.1"

ifconfig_vtnet0_ipv6="inet6 2a0a:4cc0:c1:2f90::2 prefixlen 64"
ipv6_defaultrouter="fe80::1%vtnet0"

The -lro -tso flags deserve a callout. LRO (Large Receive Offload) and TSO (TCP Segmentation Offload) are hardware acceleration features on the virtio NIC. They work fine for traffic that terminates at the host. But when the host forwards packets - as a router does for jail traffic - these offloads break PF’s ability to correctly NAT and checksum forwarded packets. The symptom is maddening: connections from the host work, but NATed jail traffic silently fails with bad checksums. Disable both whenever a FreeBSD host acts as a gateway.

SLAAC is disabled globally to avoid conflicts with statically routed tunnel addresses:

ipv6_activate_all_interfaces="NO"

The Logical Upstream: BGP Tunnel (FIB 1)

The GIF tunnel connects to the BGP router at 128.140.64.181. This is where the multi-FIB magic lives:

cloned_interfaces="bridge0 gif0"

ifconfig_gif0="fib 1 tunnel 152.53.147.5 128.140.64.181 tunnelfib 0"
ifconfig_gif0_alias0="inet 10.255.255.6 10.255.255.5"
ifconfig_gif0_ipv6="inet6 2a06:9801:1c:ffff::2 2a06:9801:1c:ffff::1 prefixlen 128"

This single line - fib 1 tunnel 152.53.147.5 128.140.64.181 tunnelfib 0 - is the most important configuration directive on the entire server. It contains two directives that work in concert:

fib 1: The tunnel interface itself lives in routing table 1. Traffic arriving on gif0 and traffic routed out gif0 consults FIB 1 for routing decisions.
tunnelfib 0: The outer IPv4 encapsulation wrapper (the 152.53.147.5 → 128.140.64.181 packet) uses FIB 0 - the default routing table - to find the path to the BGP router.

Why both? Without tunnelfib 0, the encapsulated IPv4 packets would consult FIB 1 for their route to 128.140.64.181. But FIB 1’s default route points at gif0 itself (the tunnel). That creates a recursive loop: to send a packet out the tunnel, you’d need to use the tunnel. tunnelfib 0 breaks the recursion by telling the outer encapsulation to use the normal Netcup route.

The IPv4 point-to-point addresses (10.255.255.6/5) are used for the tunnel’s inner addressing. The IPv6 link addresses (2a06:9801:1c:ffff::2/1) come from our own /48 - they’re already routable within our BGP infrastructure.

FIB 1: Building a Complete Routing Table

FIB 1 starts empty. It knows nothing about the server’s local networks. Every route must be explicitly added - if FIB 1 doesn’t have a route to a destination, traffic in that routing table gets dropped.

static_routes="fib1_v6_default fib1_v6_jailnet fib1_v4_default fib1_v4_jailnet fib1_v4_host217 fib1_v4_jail216 v4_jail216"

# 1. Default Routes (How FIB 1 reaches the internet)
route_fib1_v6_default="-6 default -interface gif0 -fib 1"
route_fib1_v4_default="-inet default 10.255.255.5 -fib 1"

# 2. Subnet Routes (How FIB 1 finds the bridge)
route_fib1_v6_jailnet="-6 2a06:9801:1c:1000::/64 -interface bastille0 -fib 1"
route_fib1_v4_jailnet="-net 10.254.254.0/24 -interface bastille0 -fib 1"

# 3. Host Routes (Where FIB 1 finds specific public IPs)
route_fib1_v4_host217="-host 194.28.98.217 -interface lo0 -fib 1"
route_fib1_v4_jail216="-host 194.28.98.216 -interface bastille0 -fib 1"
route_v4_jail216="-host 194.28.98.216 -interface bastille0"       # FIB 0 also needs this — explained in the Pure Public section below

Each route category serves a specific purpose:

Default routes tell FIB 1 how to reach the internet. IPv6 exits through gif0 directly (the tunnel interface). IPv4 uses 10.255.255.5 as the next-hop - that’s the BGP router’s end of the tunnel’s inner IPv4 addressing.

Subnet routes are necessary because FIB 1 doesn’t automatically know that bastille0 has the jail network attached. Without these, return traffic arriving on the tunnel for jail addresses would try to exit via the default route (back out the tunnel) instead of being delivered locally to the bridge.

Host routes handle the public IPs. 194.28.98.217 is on lo0 (the host’s shared loopback IP for port forwarding). 194.28.98.216 is on bastille0 (directly assigned to the testvnet jail). Both FIBs need to know where these addresses live - note that v4_jail216 adds the same route to FIB 0 as well, so that the default routing table can also reach the jail.

Jail Networking and the Bridge

The bastille0 bridge is the single point where all jail routing paradigms converge:

ifconfig_bridge0_name="bastille0"
ifconfig_bastille0="inet 10.254.254.1/24"
ifconfig_bastille0_ipv6="inet6 2a06:9801:1c:1000::1 prefixlen 64"

Three types of traffic share this bridge:

Private NAT (10.254.254.0/24): Jails like Caddy (10.254.254.10) get private IPv4 addresses, NATed to the Netcup IP when exiting vtnet0. This is standard jail networking - nothing special here.
BGP IPv6 Routed (2a06:9801:1c:1000::/64): Jails also receive addresses from our BGP prefix. These are globally routable, announced by AS201379, and traffic for them arrives via the GIF tunnel. No NAT - pure end-to-end IPv6 as designed.
BGP IPv4 Routed (194.28.98.x): The testvnet jail has 194.28.98.216/32 - a real public IPv4 address, no NAT, directly routed through both FIBs.

The shared host IP (194.28.98.217) on lo0 is a different pattern. It’s not assigned to any jail; instead, PF uses RDR rules to forward ports from this IP to internal jails. This lets Caddy serve content from a BGP-routed IPv4 address while the jail itself only has a private 10.254.254.10 address.

PF: Where the Routing Decision Happens

The PF configuration is where dual-FIB stops being a kernel feature and becomes a routing policy. Let’s walk through each layer.

Normalization

scrub in all fragment reassemble
scrub all max-mss 1220

The MSS clamp at 1220 accounts for tunnel encapsulation overhead. MSS (Maximum Segment Size) dictates the maximum size of the TCP payload. To calculate it, subtract the inner IP header and the inner TCP header from the MTU. For IPv6 traffic inside the tunnel, clamped to the safe minimum MTU of 1280 bytes, you subtract the IPv6 header (40 bytes) and the TCP header (20 bytes): 1280 - 40 - 20 = 1220. Without this, large TCP segments get silently dropped or fragmented at the tunnel, causing mysterious stalls where small requests succeed but large transfers hang.

Translation (NAT and RDR)

Translation happens before filtering. This is critical to understand - filter rules for RDR traffic must match the translated internal IP, not the public IP.

# Outbound NAT: private jails → Netcup IP
nat on $ext_if inet from <jails_v4> to any -> ($ext_if)

# Port forwarding on the Netcup IP (vtnet0)
rdr pass on $ext_if inet proto {tcp, udp} to ($ext_if) port {80, 443} -> $frontend_v4
rdr pass on $ext_if inet proto tcp to ($ext_if) port 1965 -> $gemini_v4
rdr pass on $ext_if inet proto {tcp, udp} to ($ext_if) port 53 -> $powerdns_v4

# Port forwarding on the BGP host IP (arrives via tunnel)
rdr on $tun_if inet proto {tcp, udp} to $host_v4_routed port {80, 443} -> $frontend_v4

The last rule is different - traffic to 194.28.98.217:80/443 arrives on the tunnel interface and gets redirected to the same Caddy jail. This means Caddy serves web traffic regardless of whether it arrived via Netcup or the BGP tunnel, even though Caddy only has a private IP address.

Default Policies

block quick from <bruteforce>
block drop in log all
block drop out log all
antispoof quick for { $ext_if, bastille0 }

Default deny in both directions. Bruteforce entries get an instant drop before any evaluation. Antispoof prevents packets from entering interfaces they don’t belong to.

Egress: The `route-to` Safety Net

pass out quick on $ext_if route-to ($tun_if $tun_gw_v4) inet from $host_v4_routed to any keep state

This is a safety net rule that catches an edge case. When local processes on the host respond from the shared BGP IP (194.28.98.217), those replies naturally try to exit via vtnet0 - the host’s default route in FIB 0. But Netcup would drop traffic sourced from an IP that doesn’t belong to their network.

The route-to directive intercepts: “if you see a packet about to exit vtnet0 with source 194.28.98.217, redirect it to the tunnel interface instead.” It overrides the routing decision at the PF level, forcing the packet into gif0 toward 10.255.255.5 (the BGP router).

Egress: Jail Policy Routing via `rtable`

This is the core mechanism. When a jail sends traffic from a BGP address, PF must force it into FIB 1:

# IPv6 BGP-addressed jail traffic → routing table 1
pass in quick on bastille0 inet6 from $bgp_net_v6 to any rtable 1 keep state

# IPv4 pure public jail traffic → routing table 1
pass in quick on bastille0 inet  from $testvnet_v4 to any rtable 1 keep state

The rtable 1 directive is the key. When a packet from 2a06:9801:1c:1000::/64 or 194.28.98.216 arrives at the bridge, PF assigns it to routing table 1. The kernel then consults FIB 1 for the forwarding decision - and FIB 1’s default route points out gif0 to the BGP router. The packet gets encapsulated and sent through the tunnel.

Without rtable 1, these packets would use FIB 0’s default route and exit through Netcup - where they’d be dropped as spoofed.

There’s also a DNS64 translation rule for jails needing IPv4 destinations via IPv6:

pass in quick on bastille0 inet6 from $bgp_net_v6 to 64:ff9b::/96 af-to inet from ($ext_if) keep state

And a standard NAT egress rule for private jails:

pass in quick on bastille0 from <jails_v4> to ! $jail_net keep state

The ! $jail_net negation allows jails to reach the internet but prevents them from directly accessing other jails - a micro-segmentation pattern that documents and enforces the network architecture.

Ingress: Tunnel Encapsulation

The GIF tunnel uses protocol 41 (IPv6-in-IPv4). PF must allow these encapsulated packets to enter and exit the physical interface:

pass in  quick on $ext_if proto { 41, ipencap } from 128.140.64.181 to ($ext_if)
pass out quick on $ext_if proto { 41, ipencap } from ($ext_if) to 128.140.64.181

Strictly locked to the BGP router’s IP - no one else should be sending protocol 41 to this server.

Ingress: The `reply-to` Imperative

All inbound connections from the BGP tunnel must be tagged with reply-to:

# IPv4 tunnel ingress
pass in quick on $tun_if reply-to ($tun_if $tun_gw_v4) inet proto icmp to $host_v4_routed keep state
pass in quick on $tun_if reply-to ($tun_if $tun_gw_v4) inet proto tcp from <trusted_v4> to $host_v4_routed port 30822 flags S/SA keep state
pass in quick on $tun_if reply-to ($tun_if $tun_gw_v4) inet proto {tcp, udp} to $frontend_v4 port {80, 443} keep state
pass in quick on $tun_if reply-to ($tun_if $tun_gw_v4) inet to $testvnet_v4 keep state

# IPv6 tunnel ingress
pass in quick on $tun_if reply-to ($tun_if $bgp_hub_ip) inet6 proto ipv6-icmp from any to $bgp_net_v6 icmp6-type { echoreq, echorep, toobig, timex, paramprob } keep state
pass in quick on $tun_if reply-to ($tun_if $bgp_hub_ip) inet6 proto {tcp, udp} to $frontend_v6  port {80, 443} keep state
pass in quick on $tun_if reply-to ($tun_if $bgp_hub_ip) inet6 proto {tcp, udp} to $powerdns_v6  port 53 keep state
pass in quick on $tun_if reply-to ($tun_if $bgp_hub_ip) inet6 proto tcp        to $gemini_v6    port 1965 keep state

reply-to instructs PF’s state machine: “when you see a reply to this connection, send it back out $tun_if to the specified next-hop, regardless of what the routing table says.” Without it, the kernel would consult FIB 0 for return traffic (the jail doesn’t live in FIB 1), and replies would exit through vtnet0 - getting dropped by Netcup as spoofed.

The IPv4 and IPv6 reply-to targets differ because they use different tunnel endpoint addresses: - IPv4: $tun_gw_v4 = 10.255.255.5 (the inner IPv4 point-to-point address) - IPv6: $bgp_hub_ip = 2a06:9801:1c:ffff::1 (the inner IPv6 link address)

The Pure Public VNET Jail

The testvnet jail is the most interesting routing case. It has a real public IPv4 address (194.28.98.216/32) and a BGP IPv6 address (2a06:9801:1c:1000::99), both assigned directly to the jail’s vnet0 interface:

# Inside the jail
vnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP>
        inet 194.28.98.216 netmask 0xffffffff broadcast 194.28.98.216
        inet6 2a06:9801:1c:1000::99 prefixlen 64

This jail has no NAT. Outbound traffic from 194.28.98.216 hits the bridge, PF matches from $testvnet_v4 → rtable 1, and the packet exits via the tunnel. Inbound traffic arrives on the tunnel, PF matches to $testvnet_v4 with reply-to, and the packet is delivered to the jail on the bridge. Return traffic follows the reply-to state back through the tunnel.

Both FIBs need host routes for this address. FIB 1 needs it because tunnel traffic must find the jail:

route_fib1_v4_jail216="-host 194.28.98.216 -interface bastille0 -fib 1"

FIB 0 needs it because proto 41 decapsulation happens before PF routing decisions - the kernel must know where to deliver the inner packet:

route_v4_jail216="-host 194.28.98.216 -interface bastille0"

The Complete Flow

Inbound to a BGP-addressed service (IPv6)

 1. Client sends packet to 2a06:9801:1c:1000::10 (Caddy jail)
 2. Packet traverses the internet → AS201379 → hobgp BGP router
 3. hobgp forwards through GIF tunnel to radon (proto 41 encapsulated)
 4. radon receives proto 41 on vtnet0, decapsulates → gif0
 5. PF match: reply-to ($tun_if $bgp_hub_ip), creates state entry
 6. Forwarded to bastille0 → Caddy jail
 7. Caddy responds, packet exits on bastille0
 8. PF state table triggers reply-to: send via gif0 to bgp_hub_ip
 9. gif0 encapsulates (proto 41) using tunnelfib 0 → vtnet0 → hobgp
10. hobgp receives, forwards to upstream → internet → client

Outbound from a pure public jail (IPv4)

1. testvnet jail sends packet from 194.28.98.216
2. Packet arrives on bastille0
3. PF match: "from $testvnet_v4 → rtable 1"
4. Kernel routes via FIB 1 → default route → gif0
5. gif0 encapsulates using tunnelfib 0 → vtnet0 → hobgp
6. hobgp receives, forwards to internet (source: 194.28.98.216)

NAT’d jail egress (IPv4, standard path)

1. Caddy jail sends packet from 10.254.254.10
2. Packet arrives on bastille0
3. PF match: "from <jails_v4> to ! $jail_net" (standard egress)
4. Kernel routes via FIB 0 → default route → vtnet0
5. NAT translates source to 152.53.147.5
6. Packet exits to Netcup → internet

Verification

From the host: testing both FIBs

The setfib command selects which routing table a command uses:

# Via Netcup (FIB 0) - physical uplink
root@radon:~ # setfib 0 fetch -4 -o - -q https://ifconfig.co
152.53.147.5

root@radon:~ # setfib 0 fetch -6 -o - -q https://ifconfig.co
2a0a:4cc0:c1:2f90::2

# Via BGP tunnel (FIB 1) - bound to the routed IP
root@radon:~ # setfib 1 fetch --bind-address=194.28.98.217 -4 -o - -q https://ifconfig.co
194.28.98.217

root@radon:~ # setfib 1 fetch -6 -o - -q https://ifconfig.co
2a06:9801:1c:ffff::2

FIB 0 returns Netcup’s addresses. FIB 1 returns BGP addresses. Two completely independent paths to the internet from the same machine.

From inside the pure public jail

root@radon:~ # bastille cmd testvnet fetch -4 -o - -q https://ifconfig.co
194.28.98.216

root@radon:~ # bastille cmd testvnet fetch -6 -o - -q https://ifconfig.co
2a06:9801:1c:1000::99

The jail sees its own real public IP - not a NATed address. Both IPv4 and IPv6 traffic exits through the BGP tunnel and appears on the internet with the correct source addresses.

From external clients: both paths work

# Via Netcup (default DNS, resolves to 152.53.147.5)
~ ❯ curl -Iv4 http://radon.edelga.se
* IPv4: 152.53.147.5
* Connected to radon.edelga.se (152.53.147.5) port 80
< HTTP/1.1 308 Permanent Redirect
< Server: Caddy

# Via BGP tunnel (resolves to 2a06:9801:1c:1000::10)
~ ❯ curl -Iv6 http://blog.hofstede.it
* IPv6: 2a06:9801:1c:1000::10
* Connected to blog.hofstede.it (2a06:9801:1c:1000::10) port 80
< HTTP/1.1 308 Permanent Redirect
< Server: Caddy

# Via BGP tunnel (IPv4, routed through 194.28.98.217)
~ ❯ curl -Iv4 http://194.28.98.217
* Connected to 194.28.98.217 (194.28.98.217) port 80
< HTTP/1.1 308 Permanent Redirect
< Server: Caddy

Three different entry points, all reaching the same Caddy jail - which only has a 10.254.254.10 private address. The same service is accessible through Netcup’s network (physical) and through the BGP tunnel (logical), and Caddy doesn’t know or care which path the request took.

Security Considerations

Dual-FIB setups require extra security attention because more routing paths mean more attack surface:

kern_securelevel=2 prevents loading kernel modules and writing to raw disk devices post-boot. Even if root is compromised, the attacker can’t modify the running kernel or bypass the firewall.

SSH is restricted to known IPs on a non-standard port with aggressive rate limiting. The overload <bruteforce> flush global combination instantly blocks attackers and kills their existing connections:

pass in quick on $ext_if proto tcp from <trusted_v4> to any port 30822 flags S/SA keep state \
    (max-src-conn 5, max-src-conn-rate 3/30, overload <bruteforce> flush global)

SSH via the tunnel uses reply-to to ensure the management session stays within the tunnel path - an attacker on Netcup’s network can’t even see the tunnel-side SSH port.

Antispoof blocks packets with impossible source addresses for each interface, preventing attackers from injecting traffic that pretends to come from the internal network.

Monitoring is bound to the internal bridge only, keeping Prometheus exporters invisible from the internet:

node_exporter_listen_address="10.254.254.1:9100"
jail_exporter_listen_address="10.254.254.1:9452"

Lessons Learned

tunnelfib is the unsung hero. Without tunnelfib 0 on the GIF interface, the tunnel would try to route its own encapsulated packets through itself. The error mode is a silent hang - no error messages, no crash, just packets disappearing into recursive routing. If your GIF tunnel comes up but passes no traffic, check tunnelfib first.

FIB 1 needs manual completeness. Unlike FIB 0, which automatically gets interface routes when interfaces come up, FIB 1 starts empty. If you can ping the internet from FIB 1 but jails can’t reply, you’re missing a subnet route that tells FIB 1 where the bridge is. The symptom is asymmetric: inbound works (because reply-to handles replies), but jail-initiated connections fail.

reply-to and rtable solve different halves of the same problem. reply-to handles inbound connections: “replies to tunnel traffic must exit via the tunnel.” rtable handles outbound connections: “traffic from BGP addresses must route via FIB 1.” You need both - one without the other produces subtle failures.

Disable hardware offloading on forwarding hosts. LRO and TSO on virtio NICs cause PF to compute incorrect checksums for forwarded packets. The failure is invisible at the host level - everything looks fine locally - but downstream receivers silently drop the corrupted packets. The -lro -tso flags are non-negotiable when gateway_enable="YES".

MSS clamping is not optional with tunnels. Every encapsulation layer reduces the effective MTU. Without scrub all max-mss 1220, TCP connections that transfer more than ~1220 bytes of payload per segment will stall. The insidious part: small requests (DNS, HTTP redirects) work fine, making the problem seem intermittent until someone tries to download a file.

A single bridge can carry multiple routing paradigms. Private NATed traffic, natively routed IPv6, and pure public IPv4 all coexist on bastille0 without any VLAN tagging or interface separation. PF’s source-based rules handle the routing differentiation. This simplicity is a feature - fewer moving parts means fewer failure modes.

Conclusion

Dual-FIB policy routing on FreeBSD is the kind of feature that sounds complex until you see it in practice. Two routing tables, a handful of PF directives (rtable, reply-to, route-to), and some static routes. That’s the entire mechanism that lets a single server speak from two completely different address spaces through two completely different internet paths.

The configuration shown here is what runs in production. The same Caddy jail serves blog.hofstede.it via IPv6 through the BGP tunnel and radon.edelga.se via IPv4 through Netcup - simultaneously, transparently, without knowing the difference. The testvnet jail has a real public IPv4 address that’s globally routable through AS201379, with no NAT anywhere in the path. And the whole thing coexists peacefully with standard NATed jail traffic on the same bridge.

Is this setup necessary for running a blog? No. But it’s the same pattern that production multi-homed infrastructure uses at much larger scale. The tools are the same - FIBs, PF, tunnels - just running on a single virtual machine instead of a rack of routers. FreeBSD makes this accessible because these features are first-class kernel primitives, not bolted-on afterthoughts.

References

FreeBSD setfib(1) - Multiple routing table selection
FreeBSD gif(4) - Generic tunnel interface, including tunnelfib
FreeBSD pf.conf(5) - PF configuration reference
FreeBSD Handbook: Firewalls (PF)
Running Your Own AS: BGP on FreeBSD - The BGP router side of this setup
Going Multi-Homed with iBGP - The multi-PoP expansion
PF Firewall on FreeBSD: A Practical Guide - PF foundation concepts

The internet doesn’t care that all of this runs on one virtual machine. Packets arrive, get routed through the correct table, hit the correct jail, and the replies leave through the correct tunnel. Two FIBs, one bridge, zero confusion. That’s the beauty of policy routing done right - the complexity is in the configuration, not in the operation.

Running Your Own AS: Joining an IXP with a Third Edge Router

2026-03-21T00:00:00+01:00

Part 1 covered running a single BGP router with two upstream providers. Part 2 added a second Point of Presence with Vultr native peering and iBGP. This is Part 3: connecting to an Internet Exchange Point.

An IXP changes the game. Instead of sending traffic through transit providers and their upstreams, networks at an exchange peer directly over a shared switching fabric. The result is shorter paths, lower latency, and fewer intermediaries.

This article documents connecting my AS201379 to LocIX Düsseldorf - a community-run IXP with over 350 participants - using a dedicated FreeBSD edge router, and the iBGP plumbing that ties it back into the existing infrastructure. This article assumes familiarity with BGP, iBGP, and basic FreeBSD networking as covered in the previous parts.

Note on addresses: All provider-assigned and transit-provider IP addresses have been replaced with RFC 5737 / RFC 3849 documentation ranges. AS201379, its prefix 2a06:9801:1c::/48, and all IXP-facing addresses (which are publicly visible in peering databases) are shown as-is. LocIX route server addresses and the peering LAN prefix are equally public.

Why Join an IXP?

Transit providers create indirect paths: even if two networks are geographically close, their traffic may traverse multiple intermediary ASes. At an IXP, those intermediaries disappear - networks exchange traffic directly over a shared fabric. The practical impact is measurable:

One example is traffic between my AS201379 and meerfarbig.net:

Without IXP (via transit):
  hobgp → iFog → DE-CIX fabric → meerfarbig.net
  3+ hops, 20-30ms

With IXP (direct peering):
  hobgp → lobgp → LocIX fabric → meerfarbig.net
  2 hops, ~9ms

Beyond latency, IXP peering provides route diversity that transit alone cannot match. Route servers at the exchange announce routes from every participant. A single BGP session to the route server gives me access to routes from hundreds of networks - routes I won’t see through transit because those networks either peer-lock certain prefixes or prefer to exchange locally.

Architecture: The Three-Router Triangle

My third router (lobgp) connects to LocIX through a hosting provider with direct layer-2 access to the exchange fabric. It peers with LocIX’s route servers over the peering LAN, announces my /48, and sends everything it learns back to hobgp via iBGP over a GIF tunnel. The resulting topology looks like this:

                    ┌──────────────────────────────────────────────────┐
                    │                Default-Free Zone                 │
                    └──┬─────────────┬──────────────┬──────────────────┘
                       │             │              │
                   AS209533      AS209735       AS212895
                   (iFog)        (Lagrange)     (route64)
                       │             │              │
                  GRE tunnel    GRE tunnel     GRE tunnel
                       │             │              │
                  ┌────┴─────────────┴──────────────┴────────────┐
                  │                  hobgp (Core)                │
                  │          FreeBSD + FRR, AS201379             │
                  │          2a06:9801:1c::/48                   │
                  └────────────┬────────────────┬────────────────┘
                               │                │
                          GIF tunnel        GIF tunnel
                          (iBGP)            (iBGP)
                               │                │
  ┌────────────────────────────┴──┐   ┌─────────┴─────────────────────┐
  │         vtbgp (Edge)          │   │         lobgp (Edge)          │
  │  FreeBSD + FRR, AS201379      │   │  FreeBSD + FRR, AS201379      │
  │  Native BGP → AS64515 (Vultr) │   │  LocIX Peering LAN            │
  └───────────────────────────────┘   │  → RS1/RS2 (AS202409)         │
                                      │  → Transit AS34872 (Servperso)│
                                      └───────────────────────────────┘
                                                   │
                                          LocIX Düsseldorf
                                    ┌──────────────┴──────────────┐
                                    │  350+ participants          │
                                    │  Direct L2 peering fabric   │
                                    └─────────────────────────────┘

The design maintains the hub-and-spoke model from Part 2: hobgp remains the forwarding hub. Traffic arriving at lobgp from the IXP is forwarded through the iBGP tunnel to hobgp, which handles routing to downstream servers. lobgp announces my prefix to the exchange and passes learned routes to hobgp for path selection.

Getting IXP Access

LocIX is a community-run exchange, which makes it accessible to smaller networks. The connection itself requires a VM with a network interface on the LocIX peering LAN - a layer-2 VLAN that reaches the exchange fabric.

Several hosting providers offer VMs with LocIX connectivity. I use Servperso, which provides VMs in Düsseldorf with a dedicated interface connected to the LocIX peering VLAN. The VM gets two network interfaces: vtnet1 for regular internet connectivity (transit from Servperso’s own AS34872), and vtnet0 directly on the LocIX peering LAN.

After signing up with LocIX and receiving the peering LAN assignment, the setup is straightforward: configuring the peering LAN addresses, peering with the route servers, and announcing my prefix.

The IXP Edge Router: lobgp

Network Configuration

The /etc/rc.conf configures two physical interfaces and a GIF tunnel back to the core:

hostname="lobgp"
kern_securelevel_enable="YES"
kern_securelevel="2"
kld_list="if_gif"

# Internet connectivity (Servperso)
ifconfig_vtnet1="inet 198.51.100.50 netmask 255.255.255.224 -lro -tso"
defaultrouter="198.51.100.62"
ifconfig_vtnet1_ipv6="inet6 2001:db8:b640::50/112 -rxcsum6 -tso6"
ipv6_defaultrouter="2001:db8:b640::ffff"

# LocIX Peering LAN (direct L2)
ifconfig_vtnet0="inet 185.1.155.23/24 -lro -tso"
ifconfig_vtnet0_ipv6="inet6 2a0c:b641:701::a5:20:1379:1 prefixlen 64 -rxcsum6 -tso6"

# GIF tunnel to core (hobgp) - runs over IPv6
cloned_interfaces="gif0"
ifconfig_gif0="inet6 tunnel 2001:db8:b640::50 2001:db8:1c19::1 mtu 1460"
ifconfig_gif0_ipv6="inet6 2a06:9801:1c:fffd::2 2a06:9801:1c:fffd::1 prefixlen 128"
ifconfig_gif0_descr="Uplink-to-hobgp"

# Routing
ipv6_gateway_enable="YES"
ipv6_static_routes="myblock"
ipv6_route_myblock="2a06:9801:1c::/48 -interface gif0"

# Services
pf_enable="YES"
frr_enable="YES"
sshd_enable="YES"

Several things are different from vtbgp in Part 2:

Two physical interfaces instead of one. vtnet1 provides regular internet connectivity; vtnet0 connects directly to the LocIX peering LAN. This separation is important - peering LAN traffic should never traverse the internet-facing interface.

The GIF tunnel runs over IPv6, not IPv4. Both Servperso and Hetzner provide native IPv6, so the encapsulation uses inet6 tunnel to wrap IPv6-in-IPv6. The tunnel MTU is set to 1460 - the standard 1500-byte Ethernet MTU minus the 40-byte outer IPv6 header added by GIF. To avoid fragmentation, PF clamps TCP MSS to 1400 bytes on $hobgp_tun, ensuring segments fit cleanly inside the tunnel after the inner IPv6 header (40 bytes) and TCP header (20 bytes) are accounted for.

The peering LAN addresses are publicly registered. 185.1.155.23 (IPv4) and 2a0c:b641:701::a5:20:1379:1 (IPv6) are assigned by LocIX and visible in the PeeringDB entry. These aren’t documentation-range replacements - they’re the real addresses, because hiding them would be counterproductive for anyone wanting to peer.

Like vtbgp, the static route ipv6_route_myblock points the entire /48 at gif0, so any traffic arriving for my prefix gets forwarded through the tunnel to my core router hobgp.

FRR Configuration

The FRR config has four BGP sessions: two to LocIX route servers, one to Servperso for transit, and one iBGP back to hobgp:

frr version 10.5.1
frr defaults traditional
hostname lobgp
log syslog informational
service integrated-vtysh-config
!
ipv6 prefix-list PL-MY-NET seq 5 permit 2a06:9801:1c::/48
!
! [PL-BOGONS: denies reserved/invalid prefixes, permits ::/0 le 48 - see Part 1]
!
route-map RM-IXP-IN permit 10
 match ipv6 address prefix-list PL-BOGONS
 set local-preference 200
exit
!
route-map RM-IXP-OUT permit 10
 match ipv6 address prefix-list PL-MY-NET
exit
!
route-map RM-SERVPERSO-IN permit 10
 match ipv6 address prefix-list PL-BOGONS
 set local-preference 100
exit
!
route-map RM-SERVPERSO-OUT permit 10
 match ipv6 address prefix-list PL-MY-NET
exit
!
route-map RM-IBGP-OUT permit 10
exit
!
router bgp 201379
 bgp router-id 185.1.155.23
 no bgp enforce-first-as
 no bgp default ipv4-unicast
 neighbor 2a06:9801:1c:fffd::1 remote-as 201379
 neighbor 2a06:9801:1c:fffd::1 description Core-hobgp
 neighbor 2a06:9801:1c:fffd::1 update-source 2a06:9801:1c:fffd::2
 neighbor 2a0c:b641:701:0:a5:20:2409:1 remote-as 202409
 neighbor 2a0c:b641:701:0:a5:20:2409:1 description LocIX-RS1-v6
 neighbor 2a0c:b641:701:0:a5:20:2409:2 remote-as 202409
 neighbor 2a0c:b641:701:0:a5:20:2409:2 description LocIX-RS2-v6
 neighbor 2001:db8:b640::ffff remote-as 34872
 neighbor 2001:db8:b640::ffff description Transit-Servperso-v6
 !
 address-family ipv6 unicast
  neighbor 2a06:9801:1c:fffd::1 activate
  neighbor 2a06:9801:1c:fffd::1 next-hop-self
  neighbor 2a06:9801:1c:fffd::1 route-map RM-IBGP-OUT out
  neighbor 2a0c:b641:701:0:a5:20:2409:1 activate
  neighbor 2a0c:b641:701:0:a5:20:2409:1 soft-reconfiguration inbound
  neighbor 2a0c:b641:701:0:a5:20:2409:1 route-map RM-IXP-IN in
  neighbor 2a0c:b641:701:0:a5:20:2409:1 route-map RM-IXP-OUT out
  neighbor 2a0c:b641:701:0:a5:20:2409:2 activate
  neighbor 2a0c:b641:701:0:a5:20:2409:2 soft-reconfiguration inbound
  neighbor 2a0c:b641:701:0:a5:20:2409:2 route-map RM-IXP-IN in
  neighbor 2a0c:b641:701:0:a5:20:2409:2 route-map RM-IXP-OUT out
  neighbor 2001:db8:b640::ffff activate
  neighbor 2001:db8:b640::ffff soft-reconfiguration inbound
  neighbor 2001:db8:b640::ffff route-map RM-SERVPERSO-IN in
  neighbor 2001:db8:b640::ffff route-map RM-SERVPERSO-OUT out
 exit-address-family
exit

Route Servers: One Session, Hundreds of Routes

IXP route servers are the key simplification. Instead of establishing individual BGP sessions with every network at the exchange (which could mean hundreds of sessions), you peer with the exchange’s route servers. LocIX runs two route servers (AS202409) for redundancy. Each route server collects routes from all participants and re-announces them to everyone. You can think of the route server as a fan-out mechanism: one BGP session in, hundreds of routes out.

The two route server sessions deliver approximately 1,800 unique prefixes each - routes from the ~350 networks present at LocIX. These are routes you typically wouldn’t see through transit, or would see with a longer AS-path. At the exchange, they’re direct: one hop across the peering fabric.

`no bgp enforce-first-as`: Route Server Compatibility

Route servers are transparent BGP speakers. They don’t insert their own AS into the AS-path of re-announced routes. When RS1 (AS202409) sends you a route originally announced by, say, AS13335 (Cloudflare), the AS-path still starts with 13335 - not 202409 13335.

Standard BGP behavior rejects routes where the first AS in the path doesn’t match the neighbor’s configured remote-AS. Since the route server’s AS (202409) isn’t in the path, FRR would reject every route the server sends. no bgp enforce-first-as disables this check, which is required for route server peering.

This is safe in this context: route servers at established IXPs perform their own RPKI and IRR validation. In practice, you’re trusting their filtering - just as you trust a transit provider’s ingress filtering - only applied at a different point in the path.

Local Preference: IXP Routes Win

The route-map RM-IXP-IN sets local-preference 200 on routes learned from the route servers. On hobgp, the iBGP-learned routes from lobgp carry this LP value. The preference hierarchy across the entire network becomes:

Source	Local Preference	Path
LocIX route servers (via lobgp)	200	IXP direct peering
Vultr (via vtbgp)	200	Vultr native net
iFog BGPTunnel	150	GRE transit
Lagrange / route64 / Servperso	100	GRE transit (backup)

LocIX and Vultr share LP 200 intentionally. When a destination is reachable via both (LP tied), BGP’s tiebreakers apply. The most important one here is AS-path length: a route learned directly from a peer at LocIX carries that peer’s AS-path as-is, while the same route via Vultr traverses Vultr’s upstream chain, adding AS hops. Rather than artificially preferring one over the other, the tie lets BGP’s path-length comparison pick the objectively shorter route.

The Servperso transit (RM-SERVPERSO-IN, LP 100) serves as a fallback. If the tunnel to hobgp fails or hobgp is down, lobgp still has full internet connectivity through Servperso and can continue forwarding traffic for our /48.

`next-hop-self` on the iBGP Session

Same principle as vtbgp in Part 2. Routes learned from the LocIX route servers have next-hops on the peering LAN (e.g., 2a0c:b641:701::a5:20:13335:1 for Cloudflare). hobgp can’t reach those addresses - they’re on a VLAN in Düsseldorf that hobgp has no path to. next-hop-self rewrites all next-hops to 2a06:9801:1c:fffd::2 (lobgp’s tunnel address), making the routes actionable for hobgp: “send traffic down the iBGP tunnel to lobgp, which will forward it across the peering LAN.”

PF: Protecting the IXP Edge

The peering LAN is a shared medium. Dozens of networks on the same layer-2 segment means your edge router’s interfaces are exposed to ARP/NDP from every participant. The firewall must protect both the router itself and the peering infrastructure:

ext_if     = "vtnet1"
ixp_if     = "vtnet0"
hobgp_tun  = "gif0"

my_network_v6 = "2a06:9801:1c::/48"
my_router_ip  = "2a06:9801:1c:fffd::2"
gw_hobgp_v6   = "2001:db8:1c19::1"

rs1_v6 = "2a0c:b641:701::a5:20:2409:1"
rs2_v6 = "2a0c:b641:701::a5:20:2409:2"
peer_servperso = "2001:db8:b640::ffff"

table <bgp_peers> const { $rs1_v6, $rs2_v6, $peer_servperso }

set skip on lo0
set block-policy drop
scrub in all fragment reassemble
scrub on $hobgp_tun max-mss 1400

block log all
block in quick on $ext_if from { <bogons>, $my_network_v6 } to any
antispoof quick for { $ext_if }

# --- Control Plane ---

# ICMPv6 on external and IXP interfaces
pass in quick on { $ext_if, $ixp_if } inet6 proto ipv6-icmp \
    icmp6-type { neighbrsol, neighbradv, echoreq, 2 } keep state
pass out quick on { $ext_if, $ixp_if } inet6 proto ipv6-icmp keep state

# Outer tunnel: IPv6-in-IPv6 encapsulation, stateless
pass in quick on $ext_if proto ipv6 from $gw_hobgp_v6 to ($ext_if) no state
pass out quick on $ext_if proto ipv6 from ($ext_if) to $gw_hobgp_v6 no state

# iBGP session over tunnel
pass in quick on $hobgp_tun proto tcp from 2a06:9801:1c:fffd::1 to any port 179 keep state
pass out quick on $hobgp_tun proto tcp from any to 2a06:9801:1c:fffd::1 port 179 keep state

# eBGP sessions on peering LAN (route servers + Servperso transit)
pass in quick on $ixp_if proto tcp from <bgp_peers> to ($ixp_if) port 179 keep state
pass out quick on $ixp_if proto tcp from ($ixp_if) to <bgp_peers> port 179 keep state

# ARP/ICMP on peering LAN for neighbor discovery
pass in quick on $ixp_if inet proto icmp keep state
pass out quick on $ixp_if inet proto icmp keep state

# --- Data Plane ---

# Router's own outbound
pass out quick inet6 from $my_router_ip to any keep state

# Transit: stateless for asymmetric routing
pass in quick inet6 from any to $my_network_v6 no state
pass out quick inet6 from any to $my_network_v6 no state
pass in quick inet6 from $my_network_v6 to any no state
pass out quick inet6 from $my_network_v6 to any no state

# Default outbound fallback
pass out quick all keep state

Compared to vtbgp, the IXP edge firewall adds a few important constraints:

BGP is locked to the route server addresses. On the peering LAN, only the LocIX route servers and the Servperso transit peer are allowed to establish BGP sessions. Random participants can’t initiate unsolicited BGP sessions to our router.

ICMPv6 is permitted on the peering LAN. NDP (Neighbor Discovery Protocol) must work for the router to resolve MAC addresses of peers on the shared segment. Without this, the peering LAN interface is dead.

IPv4 ICMP is explicitly allowed on the peering LAN. Even though this is primarily an IPv6 setup, the peering LAN carries IPv4 ARP and ICMP for dual-stack participants. Allowing ICMP keeps diagnostics working.

Core Router Changes: Adding the Second iBGP Peer

On hobgp, adding lobgp means a new GIF tunnel and a new iBGP session. The additions to rc.conf:

# GIF tunnel to lobgp (LocIX via Servperso) - IPv6-in-IPv6
ifconfig_gif5="inet6 tunnel 2001:db8:1c19::1 2001:db8:b640::50 mtu 1460"
ifconfig_gif5_ipv6="inet6 2a06:9801:1c:fffd::1 2a06:9801:1c:fffd::2 prefixlen 128"
ifconfig_gif5_descr="Tunnel-to-lobgp"

And the FRR additions:

neighbor 2a06:9801:1c:fffd::2 remote-as 201379
neighbor 2a06:9801:1c:fffd::2 description Edge-Servperso-IXP
neighbor 2a06:9801:1c:fffd::2 update-source 2a06:9801:1c:fffd::1
!
address-family ipv6 unicast
 neighbor 2a06:9801:1c:fffd::2 activate
 neighbor 2a06:9801:1c:fffd::2 soft-reconfiguration inbound
 neighbor 2a06:9801:1c:fffd::2 route-map RM-IBGP-IN in
 neighbor 2a06:9801:1c:fffd::2 route-map RM-IBGP-OUT out

The RM-IBGP-IN and RM-IBGP-OUT route-maps are shared with the vtbgp iBGP session - no new policy needed. The core router’s firewall also needs the tunnel endpoint added to its permitted peers, and the gif5 tunnel added to MSS clamping rules.

With this, hobgp now has five BGP sessions: three eBGP upstreams (iFog, Lagrange, route64) and two iBGP peers (vtbgp for Vultr, lobgp for LocIX).

Verification

BGP Sessions

First, confirm that all BGP sessions are established. On lobgp, vtysh -c 'show bgp ipv6 summary' shows:

Neighbor                     V    AS   MsgRcvd   MsgSent  Up/Down   PfxRcd  Desc
2a06:9801:1c:fffd::1         4  201379    4034   1792956  2d19h10m       1  Core-hobgp
2001:db8:b640::ffff          4   34872 2512958      4034  2d19h10m  238115  Transit-Servperso
2a0c:b641:701:0:a5:20:2409:1 4  202409  123879      4035  2d19h10m    1859  LocIX-RS1-v6
2a0c:b641:701:0:a5:20:2409:2 4  202409  123882      4035  2d19h10m    1859  LocIX-RS2-v6

The route servers each contribute ~1,859 prefixes - the routes announced by other LocIX participants. Servperso provides a full DFZ view (~238K prefixes) as transit backup. The iBGP session to hobgp sends exactly 1 prefix outbound: our /48.

On hobgp, the five-peer summary shows the combined view:

Neighbor             V    AS   MsgRcvd   MsgSent  Up/Down   PfxRcd  Desc
2001:db8:300::1      4  209533 5096936     33805  10:02:36  239576  Upstream-iFog-FRA
fd00:ca::1           4  209735 4777022      5636  3d21h52m  237342  Upstream-Lagrange-UK
2001:db8:400::1      4  212895 1907099      5636  3d21h52m  243233  Upstream-Route64-FRA
2a06:9801:1c:fffd::2 4  201379 2762876      5622  2d19h09m  238120  Edge-Servperso-IXP
2a06:9801:1c:fffe::2 4  201379 2396813      5640  3d21h48m  233048  Edge-Vultr-Frankfurt

The vtbgp peer (2a06:9801:1c:fffe::2) is the Vultr edge router from Part 2. The new lobgp iBGP session (2a06:9801:1c:fffd::2) delivers ~238K prefixes to hobgp - the combination of LocIX routes and Servperso’s full table, merged and forwarded via iBGP. The IXP routes carry LP 200 and win path selection for destinations that are reachable via the exchange.

The IXP Difference: Real Traceroutes

The real test is measuring what the exchange connection actually changes. Here’s a traceroute to meerfarbig, a German ISP that peers with my AS201379 at LocIX:

HOST: hobgp                             Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- lobgp.edge.hofstede.it             0.0%    10    8.3   8.8   8.2  11.4   1.0
  2.|-- ae0.800.mx240.dus1.meerfarbig.net  0.0%    10    9.5  14.4   9.4  29.2   6.2
  3.|-- 2a00:f820:11::45                   0.0%    10    9.4   9.6   9.4  10.0   0.2

Three hops total. hobgp sends to lobgp through the iBGP tunnel (hop 1). lobgp forwards across the LocIX peering fabric to meerfarbig’s edge router ae0.800.mx240.dus1 in Düsseldorf (hop 2) - this is where the ICMP TTL-exceeded reply comes from, confirming the packet entered meerfarbig’s network directly from the exchange. Hop 3 (2a00:f820:11::45) is the final destination, one hop deeper inside meerfarbig’s infrastructure. Total latency: under 10ms. Without the IXP, this traffic would traverse iFog or route64, adding hops through transit infrastructure and roughly doubling the latency.

This is direct peering in practice. No transit AS in the path. The packet crosses one switching fabric and enters the peer’s network at their edge router - from there, it’s their internal routing to the destination.

Peering Policy

With IXP access comes a peering policy. My AS201379 operates an open peering policy - I’m happy to peer with anyone at LocIX or any other exchange point, with no traffic requirements or filtering restrictions beyond basic hygiene (valid RPKI, prefix limits).

The full peering policy, contact details, and current peering information are published at hofstede.it/as201379.html.

What’s Next: IPv4

The entire setup so far is IPv6-only. Every tunnel, every BGP session, every announced prefix is IPv6. This was deliberate: IPv6 address space is cheap and easy to obtain, making it realistic for a hobby AS.

The next step is adding IPv4. A sponsoring LIR will provide me a small IPv4 prefix, which opens up the setup to the ~40% of internet traffic that still runs on IPv4. Adding IPv4 means dual-stack on the existing routers, IPv4 tunnel overlays in parallel with the IPv6 ones, and PF rules for both address families. The three-router topology stays the same; it just gains a second address family.

Lessons Learned

Route servers are the IXP’s killer feature for small networks. Without them, joining an IXP would mean manually negotiating BGP sessions with individual participants - operationally intractable for a hobby network. Two BGP sessions give you access to every participant’s routes - the exchange handles policy enforcement and validation.

no bgp enforce-first-as is required for route server peering. This is a common gotcha. Route servers are transparent - they don’t insert their own AS into the path. Without disabling the first-AS check, FRR rejects every route the server sends.

The peering LAN is a hostile environment. Dozens of networks on the same L2 segment means your router is directly exposed. Lock BGP to known peers, allow only essential ICMPv6/NDP, and drop everything else. The peering LAN is not a trusted network.

IXP routes and transit routes complement each other. The ~1,800 prefixes from LocIX’s route servers are a subset of the ~240K full table. For those 1,800 destinations, the IXP path is dramatically shorter and faster. For everything else, transit providers fill in. Setting LP 200 on IXP routes ensures they always win when available.

IPv6-in-IPv6 tunnels work well for iBGP links. When both endpoints have native IPv6, there’s no reason to wrap IPv6 traffic in IPv4. The GIF tunnel between lobgp and hobgp uses inet6 tunnel and avoids the IPv4 dependency entirely.

Conclusion

Adding an IXP connection completes the final piece of a small but functional internet infrastructure: transit provides global reachability, native Vultr peering provides reach into Vultr-connected networks, and the IXP provides direct paths to every other participant at the exchange. Three routers, three distinct connectivity roles - each solving a problem the others can’t.

The three-router topology is surprisingly manageable. Each router has a clear role: hobgp is the forwarding hub with upstream transit, vtbgp announces into Vultr’s network, and lobgp peers at the exchange. iBGP ties them together, and local-preference ensures traffic takes the best available path automatically.

The practical impact is measurable in every traceroute. Destinations reachable via LocIX are one hop away instead of three or four. That’s not a theoretical improvement - it’s 10ms instead of 25ms, visible in every connection.

For anyone running a hobby AS and wondering whether IXP connectivity is worth the extra router: it is. The route diversity alone justifies the cost, and watching your traffic take a direct two-hop path to a network that used to be five hops away through transit is exactly the kind of thing that makes this hobby worth pursuing.

References

Three articles, three evolutionary steps, and the same fundamental toolset: FreeBSD, FRR, PF, and tunnels. The internet doesn’t care that AS201379 runs on three small virtual machines. It sees valid routes, clean filters, and responsive peering. From a single router with one upstream to a three-PoP network with direct exchange peering - the architecture scales because the protocols do.

Why I Prefer CentOS Stream Over Old CentOS

2026-03-15T00:00:00+01:00

When Red Hat announced in December 2020 that CentOS Linux would shift to CentOS Stream, the community response was swift and polarized. Many users who had built infrastructure around CentOS’s bug-for-bug RHEL compatibility were understandably concerned about what this change meant for their workflows.

Disclosure: I work for Red Hat, but the opinions here are my own, based on running my personal infrastructure.

Several years later, it’s worth looking at what CentOS Stream actually delivers - and why I think it’s a stronger model than what came before.

What Old CentOS Actually Was

CentOS Linux was a community-driven rebuild of Red Hat Enterprise Linux, stripped of branding and trademarks. Every binary, every library version, every kernel patch was matched as closely as possible to RHEL, bug for bug. It was a significant engineering effort, and the project served millions of users well.

And that’s exactly where its limitations lived.

But by design, old CentOS couldn’t diverge from RHEL. It couldn’t ship a bug fix before RHEL did, or adopt a newer library version independently, because any divergence would break the compatibility guarantee users depended on. The project waited for each RHEL release, rebuilt it, and shipped - with no mechanism to feed improvements back upstream.

That model delivered real value. But it also meant CentOS could only ever follow.

What CentOS Stream Actually Is

CentOS Stream sits between Fedora and RHEL in the development pipeline:

Fedora   ->     CentOS Stream  ->   RHEL
  (upstream)      (staging)     (stable release)

Fedora is where new packages, features, and technologies land first. It moves fast, breaks things occasionally, and serves as the proving ground for the broader Red Hat ecosystem. RHEL is the polished, commercially supported product - stable, certified, and conservative by design.

CentOS Stream occupies the middle ground. It’s the continuous delivery branch where the next minor release of RHEL is developed. When Red Hat prepares an upcoming release like RHEL 9.9 or RHEL 10.2, the changes flow through the corresponding CentOS Stream branch first. By the time those changes reach RHEL, they’ve been tested, validated, and refined in Stream.

This is fundamentally different from old CentOS, which sat downstream of RHEL - receiving changes only after they’d been released. Stream flips that relationship entirely.

Why This Is Better

You See What’s Coming

With old CentOS, updates arrived as a fait accompli. RHEL released a point release, CentOS rebuilt it, and you got whatever Red Hat decided you should have. No preview, no preparation, no input.

CentOS Stream lets you look slightly ahead into what’s coming to RHEL. The packages in Stream today are the packages that will ship in the next RHEL minor release. If you’re running RHEL in production alongside Stream in development or staging, you get advance visibility into every change heading your way. That’s not instability - that’s intelligence.

The Stability Is Real

This is the part that surprises people most. CentOS Stream is not Fedora. It’s not a bleeding-edge rolling release. It’s not experimental.

CentOS Stream contains well-tested software. Most components entering Stream have already matured through Fedora or internal RHEL development processes. Stream reflects the current development branch for the next RHEL minor release, staged for final validation.

In my personal experience running CentOS Stream since Stream 8 was released in September 2019, the stability has been remarkably good. For the types of workloads I run, I haven’t encountered stability issues attributable to Stream’s position in the pipeline. Stream stages already-proven changes, leaving the comprehensive hardware certification and extended QA sweeps for the official RHEL release.

The “development branch” label scares people, but the reality doesn’t match the fear. Stream isn’t where experiments happen - that’s Fedora’s job. Stream is where already-proven changes are staged for RHEL release. The delta between Stream and the next RHEL point release is intentionally small.

A Genuine Feedback Loop

Because Stream feeds into RHEL rather than copying from it, community contributions can influence what ships in RHEL. Bug fixes can be proposed and tested in Stream before they reach the enterprise release. The distribution is an active participant in its own development rather than a downstream consumer.

Stream is still closely aligned with RHEL - the packages are RHEL packages, the kernel is the RHEL kernel, the ecosystem is the RHEL ecosystem - but there’s room for the community to contribute meaningful improvements that benefit both Stream and RHEL users.

The Support Contract Argument

The most common objection to CentOS Stream is the lack of a commercial support contract. But let’s be clear: old CentOS didn’t have a support contract either. That was always the trade-off. You got RHEL-compatible binaries without paying for a subscription, but you also got them without commercial support, without certified hardware matrices, and without the ability to open a support case when something went sideways.

CentOS Stream has the same arrangement. If you need a support contract, you need RHEL - and that was true before Stream existed too. Nothing changed here.

What you do get with Stream is access to the CentOS community, Red Hat’s public issue trackers, and the ability to see and test changes before they hit your RHEL production systems. Arguably, that’s more than old CentOS ever offered.

Stream as a Server Platform

CentOS Stream is a valid choice for running production servers. Not a compromise, not a stopgap - a legitimate, well-engineered server operating system.

Consider what you’re actually getting:

RHEL-grade packages: The software in Stream is built from the same sources as RHEL, with the same compilation flags, the same security policies, and the same enterprise-focused defaults.
Continuous updates: Rather than waiting for the next point release, you receive a steady flow of tested updates. Many fixes and improvements appear earlier than they did on old CentOS, which had to wait for RHEL to release them and then rebuild. (Embargoed CVEs may reach RHEL subscribers first, but routine fixes flow through Stream without the old rebuild lag.)
SELinux, systemd, firewalld: The full enterprise Linux stack, configured and tuned the same way RHEL configures it.
Container and virtualization support: Podman, Buildah, and the container ecosystem work exactly as they do on RHEL.
~5-year lifecycle: CentOS Stream tracks the Full Support phase of its corresponding RHEL release - roughly five years. Stream 8 reached end of life in May 2024, and Stream 9 is expected to follow in 2027. That’s shorter than RHEL’s full 10-year lifecycle, but still excellent for a continuous delivery platform.

CentOS Stream is not a drop-in replacement for every workload that historically used CentOS Linux. Environments that require strict vendor certification, regulatory validation, or decade-long lifecycle guarantees will still be better served by RHEL itself. But for workloads that don’t need those specific assurances, Stream provides a compelling platform - and because it tracks ahead of RHEL rather than behind, you’re often running slightly newer software.

The Ecosystem Effect

CentOS Stream has become genuinely useful for the broader Linux ecosystem. ISVs can test against Stream to ensure compatibility with upcoming RHEL releases. Infrastructure teams can validate automation and deployment pipelines before RHEL updates land. Community members can contribute fixes that benefit both Stream and eventually RHEL users.

This collaborative model means that engineering effort goes toward improving the distribution rather than reproducing binaries. Contributions flow upstream into RHEL, creating a virtuous cycle that benefits the entire ecosystem.

The Bottom Line

Downstream RHEL rebuilds still exist for users who want the old model, and they fill the role old CentOS occupied. But CentOS Stream offers something those rebuilds structurally cannot: an upstream role with a genuine feedback loop into RHEL development.

Old CentOS served its users well, but its downstream model limited it to following RHEL. CentOS Stream replaces that with a seat at the table - stable in practice, with the ability to contribute back. After years of running it in production, I’m convinced it’s the better model.

References

Linux Firewalls: How to Actually Secure a Cloud Server (iptables, nftables, firewalld, ufw)

2026-03-14T00:00:00+01:00

You’ve just provisioned a fresh cloud server. It has a public IPv4 address, maybe an IPv6 range, and exactly zero protection between it and the internet’s endless stream of SSH brute-force bots, port scanners, and whatever that traffic on port 5900 is. The clock is ticking - within minutes, your auth log will start filling up with connection attempts from IP addresses you’ve never heard of in countries you’ve never visited.

Linux gives you several ways to build a firewall between your server and this chaos. The problem isn’t a lack of options - it’s too many. iptables, nftables, firewalld, ufw - they all filter packets, but they approach the job differently, target different audiences, and impose different mental models. This guide walks through all four with real configurations you can actually deploy on a freshly provisioned cloud server.

(And yes, I know - I’ve spent most of this blog praising PF on FreeBSD as the pinnacle of firewall design. PF’s syntax is cleaner, its stateful inspection is more elegant, and writing pf.conf genuinely brings me joy. But sometimes you’re handed a Linux box and told to make it safe. So here we are, slumming it on the other side of the fence. Let’s make the best of it.)

The Landscape: How We Got Here

Linux packet filtering has a history that mirrors Linux itself - organic, layered, and occasionally contradictory.

ipfwadm (1994) was the first Linux firewall, basic and limited. ipchains (1999, Linux 2.2) replaced it with chain-based filtering. iptables (2001, Linux 2.4) brought stateful inspection and became the dominant firewall for two decades. nftables (2014, Linux 3.13) was designed as iptables’ successor, with a unified framework and better performance. firewalld and ufw sit on top, providing higher-level abstractions.

The key thing to understand: all of these ultimately talk to Netfilter, the packet filtering framework in the Linux kernel. iptables and nftables are different interfaces to the same underlying machinery. firewalld and ufw are frontends that generate iptables or nftables rules for you. The packets don’t care which tool wrote the rules - they get filtered either way.

iptables: The Classic

iptables has been the standard Linux firewall since 2001. If you’ve administered a Linux server in the last twenty years, you’ve encountered it. Every tutorial, every Stack Overflow answer, every “how to set up a VPS” blog post assumes iptables.

The Mental Model

iptables organizes rules into tables and chains. The default table (filter) contains three built-in chains:

INPUT: packets destined for this machine
FORWARD: packets being routed through this machine
OUTPUT: packets originating from this machine

Each chain has a policy (default action) and a list of rules evaluated in order. The first matching rule wins.

Securing a Cloud Server with iptables

Here’s a complete iptables configuration for a cloud server running a web application and SSH:

#!/bin/bash
# Cloud server firewall - iptables
# Flush existing rules
iptables -F
iptables -X
ip6tables -F
ip6tables -X

# Default policies: drop everything, allow outbound
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT

ip6tables -P INPUT DROP
ip6tables -P FORWARD DROP
ip6tables -P OUTPUT ACCEPT

# Allow loopback
iptables -A INPUT -i lo -j ACCEPT
ip6tables -A INPUT -i lo -j ACCEPT

# Allow established and related connections
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
ip6tables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# Drop invalid packets
iptables -A INPUT -m conntrack --ctstate INVALID -j DROP
ip6tables -A INPUT -m conntrack --ctstate INVALID -j DROP

# SSH: rate-limited to slow down brute-force attempts
iptables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW \
    -m recent --set --name SSH
iptables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW \
    -m recent --update --seconds 60 --hitcount 4 --name SSH -j DROP
iptables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT

ip6tables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW \
    -m recent --set --name SSH
ip6tables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW \
    -m recent --update --seconds 60 --hitcount 4 --name SSH -j DROP
ip6tables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT

# HTTP and HTTPS
iptables -A INPUT -p tcp -m multiport --dports 80,443 -m conntrack \
    --ctstate NEW -j ACCEPT
ip6tables -A INPUT -p tcp -m multiport --dports 80,443 -m conntrack \
    --ctstate NEW -j ACCEPT

# ICMPv4: allow ping and necessary types
iptables -A INPUT -p icmp --icmp-type echo-request -j ACCEPT
iptables -A INPUT -p icmp --icmp-type destination-unreachable -j ACCEPT
iptables -A INPUT -p icmp --icmp-type time-exceeded -j ACCEPT

# ICMPv6: essential for IPv6 to function
ip6tables -A INPUT -p ipv6-icmp --icmpv6-type echo-request -j ACCEPT
ip6tables -A INPUT -p ipv6-icmp --icmpv6-type echo-reply -j ACCEPT
ip6tables -A INPUT -p ipv6-icmp --icmpv6-type destination-unreachable -j ACCEPT
ip6tables -A INPUT -p ipv6-icmp --icmpv6-type packet-too-big -j ACCEPT
ip6tables -A INPUT -p ipv6-icmp --icmpv6-type time-exceeded -j ACCEPT
ip6tables -A INPUT -p ipv6-icmp --icmpv6-type parameter-problem -j ACCEPT
ip6tables -A INPUT -p ipv6-icmp --icmpv6-type neighbour-solicitation -j ACCEPT
ip6tables -A INPUT -p ipv6-icmp --icmpv6-type neighbour-advertisement -j ACCEPT
ip6tables -A INPUT -p ipv6-icmp --icmpv6-type router-solicitation -j ACCEPT
ip6tables -A INPUT -p ipv6-icmp --icmpv6-type router-advertisement -j ACCEPT

# Log dropped packets (rate-limited to avoid log flooding)
iptables -A INPUT -m limit --limit 5/min -j LOG \
    --log-prefix "iptables-dropped: " --log-level 4
ip6tables -A INPUT -m limit --limit 5/min -j LOG \
    --log-prefix "ip6tables-dropped: " --log-level 4

Making Rules Persistent

iptables rules live in kernel memory - they vanish on reboot. Making them persistent depends on your distribution:

Debian/Ubuntu:

apt install iptables-persistent
netfilter-persistent save
netfilter-persistent reload

RHEL/Fedora (if not using firewalld):

dnf install iptables-services
systemctl enable iptables ip6tables
service iptables save
service ip6tables save

The rules are saved to /etc/iptables/rules.v4 and /etc/iptables/rules.v6 (Debian) or /etc/sysconfig/iptables and /etc/sysconfig/ip6tables (RHEL).

The Good and the Ugly

iptables works. It’s battle-tested, universally available, and documented to the point of exhaustion. But the syntax is verbose - especially with dual-stack configurations where you need to duplicate everything for IPv4 and IPv6. There’s no native concept of sets (you need ipset for that), and large rulesets become linear chains that the kernel evaluates rule by rule, which doesn’t scale elegantly.

Technically, iptables is considered deprecated in favor of nftables. Modern kernels include an iptables-nft compatibility layer that translates iptables commands into nftables rules behind the scenes. On many current distributions, when you run iptables, you’re actually running iptables-nft without knowing it. Check with iptables -V - if it says nf_tables, you’re already on the bridge.

nftables: The Modern Replacement

nftables is iptables’ designated successor, included in the Linux kernel since 3.13. It addresses iptables’ design limitations: a single tool handles both IPv4 and IPv6, the rule language is more expressive, and the kernel representation uses a virtual machine that evaluates rulesets more efficiently.

The Mental Model

nftables replaces the fixed table/chain structure with a flexible one you define yourself. You create your own tables, your own chains, and configure their hook points. This sounds more complex, but it means you can organize rules however makes sense for your infrastructure.

The Same Cloud Server, in nftables

#!/usr/sbin/nft -f

flush ruleset

table inet firewall {
    # Set for tracking SSH brute-force attempts
    set ssh_meter {
        type ipv4_addr
        flags dynamic
        timeout 60s
    }

    set ssh_meter6 {
        type ipv6_addr
        flags dynamic
        timeout 60s
    }

    chain input {
        type filter hook input priority 0; policy drop;

        # Loopback
        iif lo accept

        # Established/related connections
        ct state established,related accept

        # Drop invalid
        ct state invalid drop

        # SSH with rate limiting (max 4 new connections per 60 seconds)
        # The 'update' statement dynamically adds the source address to the set
        # and checks the rate limit in a single operation
        tcp dport 22 ct state new update @ssh_meter { ip saddr limit rate over 4/minute } drop
        tcp dport 22 ct state new update @ssh_meter6 { ip6 saddr limit rate over 4/minute } drop
        tcp dport 22 ct state new accept

        # HTTP/HTTPS
        tcp dport { 80, 443 } ct state new accept

        # ICMP and ICMPv6
        ip protocol icmp icmp type {
            echo-request,
            destination-unreachable,
            time-exceeded
        } accept

        ip6 nexthdr ipv6-icmp icmpv6 type {
            echo-request,
            echo-reply,
            destination-unreachable,
            packet-too-big,
            time-exceeded,
            parameter-problem,
            nd-neighbor-solicit,
            nd-neighbor-advert,
            nd-router-solicit,
            nd-router-advert
        } accept

        # Log dropped packets
        limit rate 5/minute log prefix "nft-dropped: " level warn
    }

    chain forward {
        type filter hook forward priority 0; policy drop;
    }

    chain output {
        type filter hook output priority 0; policy accept;
    }
}

Notice the inet family - a single table handles both IPv4 and IPv6. No more duplicating every rule. The set syntax for rate limiting is built into nftables natively, no external ipset needed.

Loading and Persisting

# Validate syntax
nft -c -f /etc/nftables.conf

# Load rules
nft -f /etc/nftables.conf

# Enable at boot
systemctl enable nftables

The entire ruleset lives in /etc/nftables.conf. One file, one syntax, both address families. Compared to iptables’ separate save files for v4 and v6, this is a breath of fresh air.

Named Sets: IP Allowlists and Blocklists

One of nftables’ strongest features is native support for sets - named collections of addresses, ports, or interfaces that can be referenced in rules and updated at runtime:

table inet firewall {
    # Trusted management IPs
    set trusted_admins {
        type ipv4_addr
        elements = { 198.51.100.22, 203.0.113.50 }
    }

    set trusted_admins6 {
        type ipv6_addr
        flags interval
        elements = { 2001:db8:ffff::/48 }
    }

    # Blocklist - populated dynamically
    set blocklist {
        type ipv4_addr
        flags timeout
        timeout 24h
    }

    chain input {
        type filter hook input priority 0; policy drop;

        # Block known bad actors immediately
        ip saddr @blocklist drop

        # SSH only from trusted sources
        tcp dport 22 ip saddr @trusted_admins accept
        tcp dport 22 ip6 saddr @trusted_admins6 accept
        tcp dport 22 drop

        # ... rest of rules
    }
}

Manage sets at runtime without reloading the ruleset:

# Add to blocklist (auto-expires after 24h due to timeout flag)
nft add element inet firewall blocklist { 192.0.2.100 }

# Add with custom timeout
nft add element inet firewall blocklist { 192.0.2.101 timeout 1h }

# Remove from set
nft delete element inet firewall blocklist { 192.0.2.100 }

# List set contents
nft list set inet firewall blocklist

Why nftables Over iptables

Unified IPv4/IPv6: One ruleset, one syntax, one mental model
Native sets: No external ipset dependency
Better performance: Binary rules compiled into a virtual machine, not linear chain matching
Atomic rule replacement: Load an entire new ruleset atomically - no window where rules are partially loaded
Programmable Netlink interface: nftables exposes a Netlink socket (NETLINK_NETFILTER) that lets applications manage rules directly without shelling out to nft. Libraries like Google’s google/nftables for Go make this practical - you can build fully programmable firewall tooling, including things like injecting BPF into nftables rules. ngrok’s team used this approach to build firewall_toolkit, a host-level DDoS mitigation library
Cleaner syntax: Subjective, but ask anyone who’s written a 200-line iptables script

The learning curve is real if you’re coming from iptables, but I’ve found the transition is worth the effort. Once the nftables mental model clicks, going back to iptables feels like writing assembly after learning a proper language.

firewalld: The Enterprise Linux Firewall

firewalld is the default firewall manager on Fedora, RHEL, CentOS Stream, and their derivatives - which means it’s also the firewall running on the majority of production Linux servers in enterprise environments. Rather than asking you to write packet filter rules, firewalld introduces zones - named security contexts that you assign to network interfaces.

What sets firewalld apart from the other tools in this article is its architecture. It runs as a system daemon (firewalld.service) with a D-Bus API, which means any authorized application - Cockpit, NetworkManager, libvirt, Ansible, or your own tooling - can query and modify firewall rules programmatically without parsing text files or shelling out to command-line tools. It also cleanly separates runtime state (what’s active right now) from permanent configuration (what survives a reboot), giving you a built-in safety net for testing changes on remote systems.

The Mental Model

Zones represent trust levels. An interface in the public zone gets restrictive rules. An interface in the trusted zone allows everything. Instead of thinking about chains and rules, you think about “this interface is in this zone, and this zone allows these services.”

The predefined zones, from most restrictive to most permissive:

Zone	Default Behavior
drop	Drop all incoming, no reply sent
block	Reject all incoming with ICMP response
public	Only selected services allowed (default zone)
external	Like public, with masquerading enabled
dmz	Selected services, limited incoming
work	Trust most networked machines
home	Trust most networked machines
internal	Trust most networked machines
trusted	Accept everything

Cloud Server with firewalld

On RHEL/Fedora, firewalld is likely already running. Here’s how to configure it for our cloud server scenario:

# Check current state
firewall-cmd --state
firewall-cmd --get-active-zones

# Set the default zone
firewall-cmd --set-default-zone=public

# Allow SSH (usually already enabled in public zone)
firewall-cmd --zone=public --add-service=ssh --permanent

# Allow HTTP and HTTPS
firewall-cmd --zone=public --add-service=http --permanent
firewall-cmd --zone=public --add-service=https --permanent

# Reload to apply permanent changes
firewall-cmd --reload

# Verify
firewall-cmd --zone=public --list-all

Pro tip: Instead of adding --permanent to every command and then reloading, you can test rules in runtime first. Runtime rules are active immediately but disappear on reboot (or --reload). If you accidentally lock yourself out, a reboot reverts the damage. Once you’ve confirmed everything works, save the running state in one shot:

# Test in runtime first (no --permanent flag)
firewall-cmd --zone=public --add-service=http
firewall-cmd --zone=public --add-service=https

# Everything working? Save the current runtime state as permanent
firewall-cmd --runtime-to-permanent

This workflow is especially useful on remote servers where a typo in a permanent rule could mean a trip to the out-of-band console.

Output:

public (active)
  target: default
  icmp-block-inversion: no
  interfaces: eth0
  sources:
  services: dhcpv6-client http https ssh
  ports:
  protocols:
  forward: yes
  masquerade: no
  forward-ports:
  source-ports:
  rich-rules:

Service Definitions

firewalld uses XML service definitions in /usr/lib/firewalld/services/. Each service defines which ports and protocols it needs. When you add a “service” rather than a raw port, you’re working at a higher abstraction level:

# These are equivalent:
firewall-cmd --zone=public --add-service=https --permanent
firewall-cmd --zone=public --add-port=443/tcp --permanent

# But the service approach is self-documenting and multi-port aware

You can create custom services for your own applications:

# Create a custom service definition
firewall-cmd --permanent --new-service=myapp
firewall-cmd --permanent --service=myapp --set-description="My Application"
firewall-cmd --permanent --service=myapp --add-port=8080/tcp
firewall-cmd --permanent --service=myapp --add-port=8443/tcp

# Use it
firewall-cmd --zone=public --add-service=myapp --permanent
firewall-cmd --reload

Rich Rules: When Zones Aren’t Enough

For more granular control, firewalld supports “rich rules” - a more expressive syntax that sits between simple zone/service management and raw nftables rules:

# Allow SSH only from a trusted subnet
firewall-cmd --zone=public --add-rich-rule='
    rule family="ipv4"
    source address="198.51.100.0/24"
    service name="ssh"
    accept' --permanent

# Rate-limit SSH connections
firewall-cmd --zone=public --add-rich-rule='
    rule service name="ssh"
    accept
    limit value="4/m"' --permanent

# Block a specific IP
firewall-cmd --zone=public --add-rich-rule='
    rule family="ipv4"
    source address="192.0.2.100"
    drop' --permanent

# Log dropped packets from a specific source
firewall-cmd --zone=public --add-rich-rule='
    rule family="ipv4"
    source address="192.0.2.0/24"
    log prefix="suspect-traffic: " level="warning"
    drop' --permanent

firewall-cmd --reload

Multi-Zone Network Segmentation

One of firewalld’s most powerful enterprise features is the ability to assign different zones to different interfaces, creating proper network segmentation without touching nftables directly:

# Public-facing interface: restrictive
firewall-cmd --zone=public --change-interface=eth0 --permanent
firewall-cmd --zone=public --add-service=http --permanent
firewall-cmd --zone=public --add-service=https --permanent

# Internal management network: more permissive
firewall-cmd --zone=internal --change-interface=eth1 --permanent
firewall-cmd --zone=internal --add-service=ssh --permanent
firewall-cmd --zone=internal --add-service=cockpit --permanent
firewall-cmd --zone=internal --add-service=dns --permanent

# Database network: only database traffic
firewall-cmd --zone=dmz --change-interface=eth2 --permanent
firewall-cmd --zone=dmz --add-port=5432/tcp --permanent

firewall-cmd --reload

This maps naturally to how enterprise networks are actually designed: distinct interfaces for distinct trust levels, each with its own policy. A single nftables ruleset can do the same thing, but you’d be hand-coding the interface dispatch logic that firewalld provides out of the box.

firewalld’s Backend: nftables

Since firewalld 0.6.0, the default backend is nftables. firewalld translates its zone/service model into nftables rules. You can see the generated rules with:

nft list ruleset

The output will be significantly more complex than hand-written nftables - firewalld generates a multi-table structure with dispatch chains for its zone model. This is the trade-off: you get a managed, auditable interface with a well-defined API, while the underlying nftables ruleset handles the heavy lifting.

When firewalld Shines

firewalld is at its best in environments where firewall management isn’t a one-person artisanal craft but a repeatable, auditable process across many machines:

Cockpit integration: The web-based Cockpit console (standard on RHEL) can manage firewalld zones, services, and ports through a graphical interface. For teams where not everyone is comfortable with CLI firewall management, this lowers the barrier without sacrificing security.
Ansible automation: The ansible.posix.firewalld module maps directly to firewalld’s zone/service model. Declaring service: https, zone: public, state: enabled, permanent: true in a playbook is about as clean as firewall configuration management gets.
D-Bus API: Any application can query the current firewall state or request changes through D-Bus. libvirt uses this to dynamically open ports for VM networking. NetworkManager uses it to assign interfaces to zones when connections come up. This kind of runtime integration simply isn’t possible with static ruleset files.
Consistent across the ecosystem: Whether you’re managing a single RHEL server, a fleet of Fedora CoreOS nodes, or a CentOS Stream build farm, the firewall interface is identical. The same commands, the same zones, the same service definitions.

Where firewalld is less ideal is when you need complete, low-level control over rule ordering or heavily customized chain structures. Rich rules and direct passthrough rules can handle most advanced cases, but if you’re building something truly bespoke, nftables gives you more direct control.

ufw: Uncomplicated Firewall

ufw does exactly what its name promises: it makes firewall configuration uncomplicated. It’s the default on Ubuntu and is popular on any Debian-based system where the administrator wants a firewall without studying Netfilter internals.

The Mental Model

ufw has essentially no mental model beyond “allow or deny things.” It’s a command-line interface that generates iptables (or nftables) rules from simple, human-readable commands. It trades flexibility for approachability.

Cloud Server with ufw

# Reset to clean state
ufw reset

# Default policies
ufw default deny incoming
ufw default allow outgoing

# Allow SSH (do this BEFORE enabling ufw, or you'll lock yourself out)
ufw allow ssh

# Allow HTTP and HTTPS
ufw allow http
ufw allow https

# Enable the firewall
ufw enable

# Check status
ufw status verbose

Output:

Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), disabled (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
22/tcp                     ALLOW IN    Anywhere
80/tcp                     ALLOW IN    Anywhere
443/tcp                    ALLOW IN    Anywhere
22/tcp (v6)                ALLOW IN    Anywhere (v6)
80/tcp (v6)                ALLOW IN    Anywhere (v6)
443/tcp (v6)               ALLOW IN    Anywhere (v6)

That’s it. Six commands and your server is firewalled. ufw automatically handles both IPv4 and IPv6.

Slightly More Advanced ufw

# Allow SSH only from a specific subnet
ufw allow from 198.51.100.0/24 to any port 22 proto tcp

# Allow a port range
ufw allow 6000:6007/tcp

# Deny a specific IP
ufw deny from 192.0.2.100

# Rate-limit SSH (allows 6 connections per 30 seconds)
ufw limit ssh

# Delete a rule
ufw delete allow http

# Insert a rule at a specific position
ufw insert 1 deny from 192.0.2.0/24

# Allow from a specific IP to a specific port
ufw allow from 198.51.100.22 to any port 5432 proto tcp comment 'PostgreSQL admin'

# Show numbered rules for deletion
ufw status numbered

Application Profiles

ufw supports application profiles - predefined port configurations stored in /etc/ufw/applications.d/. Many packages drop their own profile files into this directory automatically when installed via apt - install Nginx or Apache and the corresponding ufw profile appears without any extra work. This is part of why ufw feels so frictionless on Ubuntu: the ecosystem does the heavy lifting.

# List available profiles
ufw app list

# Get profile details
ufw app info 'Nginx Full'

Profile: Nginx Full
Title: Web Server (Nginx, HTTP + HTTPS)
Description: Small, but very powerful and efficient web server

Ports:
  80,443/tcp

# Use a profile
ufw allow 'Nginx Full'

Behind the Scenes

ufw stores its rules in /etc/ufw/user.rules and /etc/ufw/user6.rules. It also has before.rules and after.rules files for advanced configurations that go beyond ufw’s command-line interface - for example, NAT rules or custom FORWARD chains:

# /etc/ufw/before.rules - add NAT for containers/VMs
*nat
:POSTROUTING ACCEPT [0:0]
-A POSTROUTING -s 10.0.0.0/24 -o eth0 -j MASQUERADE
COMMIT

When to Use ufw

ufw is perfect for single-purpose servers, personal VPS instances, and situations where you just need basic allow/deny rules without ceremony. It’s genuinely the fastest path from “naked server” to “firewalled server,” and there’s no shame in using it. Not everything needs to be a hand-crafted artisanal nftables configuration.

Where it falls short: complex multi-zone setups, advanced NAT, detailed logging configurations, or anything where you need to understand and control the exact rule structure. ufw deliberately hides that complexity, which is a feature until it isn’t.

Choosing Your Firewall

Aspect	iptables	nftables	firewalld	ufw
Learning curve	Moderate	Moderate	Low	Very low
IPv4/IPv6 unified	No	Yes	Yes	Yes
Syntax	Verbose	Clean	CLI/XML/D-Bus	Minimal
Best for	Legacy systems	New deployments	Enterprise/fleet management	Ubuntu/quick setups
Native sets	No (needs ipset)	Yes	Via zones/ipsets	No
Atomic updates	No	Yes	Yes	No
Runtime/permanent split	No	No	Yes	No
Programmatic API	No	Netlink	D-Bus	No
Web UI (Cockpit)	No	No	Yes	No
Config management	Manual	Manual	Ansible module	Manual
RHEL default	No	Backend	Yes	No
Ubuntu default	No	Backend	No	Yes
CLI complexity	High	Medium	Low	Very low

My general recommendation for new Linux deployments:

Running RHEL, Fedora, or CentOS Stream? Use firewalld. It’s the native choice and the integration benefits are substantial - Cockpit, Ansible, NetworkManager, and libvirt all speak firewalld natively. If you’re managing more than a handful of servers, the zone/service abstraction and D-Bus API will save you significant time compared to hand-rolled nftables across every machine.
Running Ubuntu or Debian and just need basic rules? Use ufw. It gets the job done in under a minute.
Need precise control, custom rulesets, or complex NAT? Use nftables directly. It’s the future of Linux packet filtering.
Maintaining an existing iptables setup that works? Keep it. The iptables-nft compatibility layer means your rules are already running on nftables under the hood. Migrate when you have a reason, not because someone on Reddit told you to.

Common Patterns for Cloud Servers

Regardless of which tool you choose, certain patterns apply to every cloud server firewall:

1. Default Deny Incoming

Every firewall configuration should start with a default deny policy for incoming traffic. This is the single most important rule. If you forget to allow a service, it’s unreachable (you notice immediately). If you forget to block a service with a default-allow policy, it’s exposed (you might never notice).

2. Always Allow Established Connections

Stateful tracking is essential. A rule like ct state established,related accept (nftables) or -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT (iptables) ensures that return traffic for allowed outbound connections gets through without explicit rules for every possible response.

3. Rate-Limit SSH

Every public-facing SSH server gets brute-forced. Rate limiting won’t stop a determined attacker, but it dramatically slows down automated scanners. Combine it with key-only authentication (PasswordAuthentication no in sshd_config) for real security.

4. Never Block Essential ICMPv6

Blocking all ICMPv6 will break IPv6 connectivity. Neighbor Discovery (the IPv6 equivalent of ARP) uses ICMPv6. Path MTU Discovery uses ICMPv6. If you block neighbour-solicitation or packet-too-big, your IPv6 connectivity will fail in mysterious ways.

5. Log Selectively

Logging every dropped packet will fill your disk. Log with rate limiting, and log the things that matter - unexpected traffic on specific ports, connections from specific sources, or rules that should never trigger but do.

6. The Docker Trap

This one burns almost every Linux admin at least once: Docker bypasses your firewall rules.

When you run docker run -p 8080:80 nginx, Docker doesn’t politely ask ufw or firewalld to open port 8080. Instead, it manipulates the iptables PREROUTING and DOCKER chains directly, inserting rules that take effect before your INPUT chain is evaluated. The result: that container port is exposed to the entire internet, completely ignoring your carefully crafted ufw default deny incoming policy.

This happens because Docker needs to implement its port-forwarding via NAT, and it does so by writing directly to Netfilter - bypassing any frontend that manages the INPUT chain.

Mitigations:

# Option 1: Bind to localhost only (container accessible only from the host)
docker run -p 127.0.0.1:8080:80 nginx

# Option 2: Disable Docker's iptables manipulation entirely
# In /etc/docker/daemon.json:
{"iptables": false}
# WARNING: You are now responsible for all container networking rules.
# Containers will lose outbound internet access until you manually
# configure NAT masquerading for the docker0 bridge.

# Option 3: Use Docker's internal network and reverse-proxy through
# a host-level web server (Nginx, Caddy) that IS behind your firewall

The safest pattern for production: don’t publish ports directly with -p. Instead, put a reverse proxy on the host that listens on the public interface (behind your firewall), and have it forward traffic to containers on Docker’s internal bridge network. This way, your firewall rules actually apply to the traffic reaching your services.

Podman, notably, doesn’t have this problem in rootless mode - it uses slirp4netns or pasta for networking, which doesn’t touch the host’s iptables chains. One more reason to consider it for server workloads.

Conclusion

All four tools solve the same fundamental problem: controlling which packets get into your server and which don’t. The differences are in ergonomics, abstraction level, and ecosystem integration. iptables is the weathered veteran - it works everywhere but shows its age. nftables is the well-designed successor that hasn’t fully taken over yet because the old guard is entrenched. firewalld wraps everything in Red Hat’s zone model, which is either a helpful abstraction or an unnecessary layer depending on your perspective. ufw strips it all down to the bare essentials, which is either refreshingly simple or frustratingly limited.

Pick the tool that matches your distribution, your team, and your complexity requirements. Then configure it, test it, and - most importantly - actually turn it on. A perfectly designed ruleset sitting in a text file isn’t protecting anything.

Corrections

2026-03-14: An earlier version of this article stated that nftables has no programmatic API. This is incorrect - nftables exposes a Netlink interface (NETLINK_NETFILTER) that allows applications to manage rules directly without the nft command-line tool. The comparison table and nftables section have been updated to reflect this. Thanks to Joe Chilliams (@joew@hachyderm.io) for the correction and the excellent real-world examples.

References

nftables wiki - the canonical nftables documentation
iptables man page
Netfilter project - the umbrella project for Linux packet filtering
firewalld documentation
ufw community help (Ubuntu)
Red Hat - Using firewalld
Debian wiki - nftables

Every Linux firewall tool on this page gets the job done. But if you really want to see packet filtering done with elegance - a single, clean configuration file, last-match-wins semantics, tables that scale to hundreds of thousands of entries without breaking a sweat, and a syntax that reads like it was designed by someone who actually enjoys writing firewall rules - well, you know where I stand. PF on FreeBSD remains the gold standard. The rest of us are just approximating greatness.

FreeBSD Foundationals: ZFS - The Last Filesystem You’ll Ever Need

2026-03-13T00:00:00+01:00

ZFS was born at Sun Microsystems in 2004, open-sourced in 2005 as part of OpenSolaris, and has since become the default filesystem on FreeBSD. Not just the default in the installer - the default in production, the default in the Handbook, the default in the minds of people who have lost data exactly once and decided never again. (It’s also available on Linux, where it works beautifully - just don’t ask me how I know you can run RHEL on a ZFS root pool. That was a crime, not a tutorial.)

This is the second article in the FreeBSD Foundationals series. The first one covered Jails. We’re covering ZFS now because it’s the foundation everything else sits on: your jails, your databases, your mail spools, your backups. Understanding ZFS isn’t optional if you’re running FreeBSD seriously.

Why ZFS Exists

Traditional filesystems trust the hardware. They write data to disk, read it back later, and assume what comes back is what was written. This assumption is wrong more often than most people realize. Disks develop bad sectors. Controllers corrupt data in transit. RAM bit-flips go undetected. RAID controllers with battery-backed caches fail in ways that silently eat data. The industry term for this is silent data corruption, and every filesystem that doesn’t checksum its data is vulnerable to it.

ZFS was designed from the ground up with one overriding principle: your data should never be silently wrong. Every block of data and metadata is checksummed. Every read is verified against that checksum. If the checksum doesn’t match, ZFS knows the data is corrupt - and if redundancy exists (mirror or raidz), it automatically repairs the damage from a good copy without anyone having to notice or intervene.

But ZFS is not just a filesystem. It’s a filesystem, volume manager, and software RAID implementation rolled into one. Traditional Unix storage stacks look like this:

Traditional stack:              ZFS stack:

┌────────────────┐              ┌────────────────┐
│   Filesystem   │              │                │
│   (ext4/ufs)   │              │      ZFS       │
├────────────────┤              │                │
│ Volume Manager │              │  (filesystem + │
│   (LVM/gpart)  │              │  volume mgmt + │
├────────────────┤              │   RAID + ...   │
│  Software RAID │              │                │
│   (gmirror)    │              │                │
├────────────────┤              ├────────────────┤
│     Disks      │              │     Disks      │
└────────────────┘              └────────────────┘

With a traditional stack, you need to coordinate between three or four separate layers, each with its own tools, its own failure modes, and its own ideas about how data is organized. ZFS eliminates this entire class of problems by managing everything from raw disks to individual files in a single, coherent system.

Core Concepts: Pools and Datasets

Pools: The Storage Foundation

A ZFS pool (zpool) is the fundamental unit of storage. You give ZFS one or more disks (or partitions, or files - it doesn’t care), and it creates a pool. All storage in ZFS comes from a pool. There are no partitions to resize, no logical volumes to extend, no separate “allocate space, then format” steps.

# Create a simple pool from a single disk
zpool create tank da0

# Create a mirror (two-way redundancy)
zpool create tank mirror da0 da1

# Create a raidz1 pool (single-parity, like RAID-5)
zpool create tank raidz1 da0 da1 da2

# Create a raidz2 pool (double-parity, like RAID-6)
zpool create tank raidz2 da0 da1 da2 da3

# Check pool status
zpool status tank

Pool topology matters and cannot be changed after creation. You choose your redundancy level when you create the pool. A mirror can tolerate one disk failure per mirror pair. Raidz1 tolerates one disk failure per vdev. Raidz2 tolerates two. Choose raidz2 for anything that matters - modern drives are massive, and resilvering (rebuilding) a 16TB disk takes days of heavy I/O, which is exactly when a second aging drive is most likely to fail. The cost of one extra parity disk is nothing compared to a lost pool.

You can check the health of a pool at any time:

zpool status tank

  pool: tank
 state: ONLINE
  scan: scrub repaired 0B in 02:31:44 with 0 errors on Sun Mar  9 03:01:44 2026
config:

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            da0     ONLINE       0     0     0
            da1     ONLINE       0     0     0

errors: No known data errors

The CKSUM column is the one that matters most. A non-zero value there means ZFS detected a checksum mismatch - silent corruption that most other filesystems would have delivered to your application without a word.

Scrubs: Proactive Integrity Verification

ZFS doesn’t just verify data when you read it. You can (and should) run periodic scrubs that read every block in the pool and verify its checksum:

# Start a scrub
zpool scrub tank

# Check scrub progress
zpool status tank

On a redundant pool (mirror or raidz), a scrub will automatically repair any corruption it finds using the redundant copy. On a non-redundant pool, a scrub will detect the corruption and report it, but can’t repair it - there’s no good copy to repair from.

Schedule scrubs regularly. Weekly is common for servers. ZFS automatically pauses scrubs during heavy I/O to avoid degrading application performance, so don’t be afraid to schedule them during the work week if your maintenance windows are limited. FreeBSD enables periodic ZFS scrubs by default via a cron entry in /etc/periodic.conf:

# In /etc/periodic.conf (or check /etc/defaults/periodic.conf)
daily_scrub_zfs_enable="YES"

Datasets: Flexible Storage Without Partitions

A ZFS dataset is roughly analogous to a traditional filesystem, but without fixed size. Datasets share the pool’s storage dynamically. You don’t pre-allocate space to a dataset - it grows as needed, up to whatever the pool (or an optional quota) allows.

# Create datasets
zfs create tank/home
zfs create tank/home/chris
zfs create tank/jails
zfs create tank/jails/mailserver
zfs create tank/postgresql

Datasets are hierarchical. Properties set on a parent are inherited by children unless explicitly overridden. This inheritance model is one of ZFS’s most practical features:

# Set compression on the parent - all children inherit it
zfs set compression=zstd tank

# Override compression for a specific child dataset
zfs set compression=lz4 tank/postgresql

# Check inherited vs. local properties
zfs get compression tank/home

NAME        PROPERTY     VALUE           SOURCE
tank/home   compression  zstd            inherited from tank

Key Dataset Properties

These are the properties you’ll actually care about in practice:

compression

ZFS supports transparent compression at the dataset level. Every block is compressed before being written to disk and decompressed on read. This sounds like it should be slower, but on modern CPUs the compression is so fast that the reduced I/O usually makes things faster overall.

# Modern default - excellent compression ratio, very fast
zfs set compression=zstd tank

# Alternative - slightly faster, slightly worse ratio
zfs set compression=lz4 tank/fast-storage

# Check compression effectiveness
zfs get compressratio tank

There’s almost no reason to leave compression off. Even on data that doesn’t compress well (encrypted files, already-compressed media), ZFS detects incompressible blocks and stores them uncompressed with negligible overhead.

recordsize

The recordsize property controls the maximum block size for a dataset. The default is 128KB, which works well for general-purpose workloads. But specific workloads benefit from tuning this:

# PostgreSQL: match the database page size (8KB)
zfs set recordsize=8k tank/postgresql

# MySQL/InnoDB: match the InnoDB page size (16KB)
zfs set recordsize=16k tank/mysql

# Large sequential files (media, backups, VM images): use 1MB
zfs set recordsize=1M tank/backups

The principle: match the recordsize to the I/O pattern of the application. Databases do many small, random reads and writes at their page size - a 128KB recordsize means ZFS reads 128KB to satisfy an 8KB database page read. Setting recordsize=8k for PostgreSQL can dramatically improve random read performance.

For sequential workloads (backups, media storage, large file copies), a larger recordsize means fewer metadata operations and better compression ratios.

Important: recordsize only affects newly written data. Changing it on an existing dataset does not rewrite existing blocks. For databases, set the recordsize before importing data.

atime

Every time a file is read, Unix traditionally updates its “access time” (atime) metadata. On a busy system, this means every read operation also triggers a write operation. ZFS inherits this behavior by default, but you should almost always disable it:

# Disable access time updates entirely
zfs set atime=off tank

# Or use relative atime (only update when mtime/ctime is newer than atime)
zfs set relatime=on tank

Disabling atime eliminates a write I/O for every read I/O. For a mail server scanning thousands of files per second, or a web server serving static assets, this is a significant performance improvement with essentially no downside. Almost nothing on a modern system depends on accurate atime.

quota and reservation

Quotas limit how much space a dataset can consume. Reservations guarantee a minimum amount of space. These are useful for preventing runaway datasets from starving others:

# Limit a dataset to 50GB
zfs set quota=50G tank/home/chris

# Guarantee 10GB is always available for this dataset
zfs set reservation=10G tank/postgresql

refquota and refreservation are the variants that exclude snapshots from the accounting. Usually refquota is what you want for user quotas - you don’t want someone’s disk usage to silently increase because of snapshots they didn’t take.

Self-Healing: How It Actually Works

ZFS checksums every block of data and metadata using SHA-256 (by default; other algorithms are available). The checksum is stored separately from the data it protects - in the parent block’s metadata. This means corruption of a data block can’t also corrupt its own checksum, which is the fatal flaw of systems that store checksums inline (like a TCP checksum covering the TCP header).

The verification flow:

Write path:
  Application data → compress → checksum → write to disk
                                   ↓
                          store checksum in
                          parent metadata block

Read path:
  Read block from disk → verify checksum → decompress → deliver to application
       ↓                      ↓
  If checksum fails:    On redundant pool:
  "This block is bad"   → read from mirror/parity
                         → repair the bad copy
                         → deliver good data
                         → increment CKSUM error counter

This happens transparently. Your application never sees the corrupt data. It gets the correct data, repaired silently, and the error is logged. You’ll see it in zpool status output as a non-zero CKSUM count, which tells you a disk is developing problems before it fails catastrophically.

On a non-redundant pool, ZFS can detect corruption but can’t repair it. The read will fail with an I/O error, which is still better than silently returning corrupted data. At least you know the data is bad.

Snapshots: Instant Point-in-Time Copies

A ZFS snapshot is a read-only point-in-time copy of a dataset. Snapshots are essentially free to create - they take no additional space initially, because they share all their data with the live dataset. Space is only consumed as the live dataset changes and the snapshot needs to preserve the old blocks.

# Create a snapshot
zfs snapshot tank/jails/mailserver@before-upgrade

# Create a snapshot of all datasets under a hierarchy
zfs snapshot -r tank/jails@2026-03-13

# List snapshots
zfs list -t snapshot

# See how much space a snapshot is consuming
zfs list -t snapshot -o name,used,refer tank/jails/mailserver

Rolling Back

If something goes wrong, you can roll a dataset back to a snapshot instantly:

# Roll back to the snapshot (destroys all changes since the snapshot)
zfs rollback tank/jails/mailserver@before-upgrade

Note: zfs rollback can only roll back to the most recent snapshot. If you have multiple snapshots and want to go back further, you need to destroy the intermediate snapshots first (or use -r to do it automatically):

# Roll back, destroying intermediate snapshots
zfs rollback -r tank/jails/mailserver@before-upgrade

This makes snapshots invaluable for system upgrades. Snapshot before you upgrade, upgrade, test. If the upgrade broke something, roll back. The entire operation takes seconds, not hours of restoring from backup.

The Beauty of /.zfs: Accessing Snapshots Directly

Here’s something that surprises people who are new to ZFS: every dataset has a hidden .zfs directory at its mount point. Inside it, under .zfs/snapshot/, every snapshot is accessible as a normal read-only directory tree.

# The .zfs directory is hidden from normal ls output
ls -la /tank/home/chris/.zfs/

total 0
dr-xr-xr-x  2 root  wheel  2 Mar 13 19:00 .
drwxr-xr-x  5 chris chris 12 Mar 13 19:30 ..
drwxr-xr-x  5 chris chris 12 Mar 10 08:00 snapshot

# List all snapshots as directories
ls /tank/home/chris/.zfs/snapshot/

2026-03-10    2026-03-11    2026-03-12    2026-03-13

# Just... cd into a snapshot and find your file
cd /tank/home/chris/.zfs/snapshot/2026-03-12/
ls Documents/
# → there's the file you accidentally deleted yesterday
cp Documents/important-report.pdf /tank/home/chris/Documents/

No restore tools. No backup software. No waiting. You browse the snapshot like a normal directory, find the file you need, and copy it back. The .zfs directory doesn’t show up in normal ls output (it’s hidden by default), but you can access it directly by name. This means regular users can recover their own accidentally deleted files without root access or administrator intervention.

This is arguably ZFS’s most user-facing killer feature. With automated snapshots (see the sanoid section below), users have a continuously available time machine for their files.

The .zfs directory can be made visible in directory listings if you prefer:

zfs set snapdir=visible tank/home

ZFS Native Encryption

ZFS supports native dataset-level encryption, added in OpenZFS 0.8. Encryption is per-dataset, transparent to applications, and managed entirely through ZFS commands. No separate tools, no dm-crypt, no GELI - just ZFS.

Creating Encrypted Datasets

Encryption is set at dataset creation time and cannot be added to an existing unencrypted dataset (you’d need to create a new encrypted dataset and move the data):

# Create an encrypted dataset with a passphrase
zfs create -o encryption=aes-256-gcm -o keylocation=prompt -o keyformat=passphrase tank/private

# Create an encrypted dataset with a key file
dd if=/dev/random of=/root/keys/tank-private.key bs=32 count=1
zfs create -o encryption=aes-256-gcm \
    -o keylocation=file:///root/keys/tank-private.key \
    -o keyformat=raw tank/secure

Child datasets inherit encryption from their parent. If tank/private is encrypted, tank/private/documents and tank/private/photos are automatically encrypted with the same key:

# These inherit encryption from tank/private
zfs create tank/private/documents
zfs create tank/private/photos

Key Management

When a system boots, encrypted datasets are not automatically mounted because ZFS doesn’t have the decryption key yet. You need to load the key before the dataset becomes accessible:

# Load the key for a single dataset
zfs load-key tank/private

# Load keys for all encrypted datasets
zfs load-key -a

# Mount after loading key
zfs mount tank/private

# Check encryption status
zfs get keystatus tank/private

NAME           PROPERTY    VALUE        SOURCE
tank/private   keystatus   available    -

For servers that need encrypted datasets available at boot without manual intervention, you can store the key file on a separate device - for example, a USB stick that stays plugged into the server but could be removed and secured independently:

# Key file on a separate USB device mounted at boot
zfs set keylocation=file:///mnt/usbkey/tank-private.key tank/private

If you store the key file on the boot pool itself, be aware that this provides no protection against physical theft of the drive - the thief gets both the encrypted data and the key to decrypt it. The only scenario where a same-disk key file helps is disk disposal or RMA, where you wipe or destroy the boot partition but send the data disks out with encrypted blocks a third party can’t read.

For meaningful data-at-rest protection, either keep the key on a physically separate device, or use passphrase-based keys and accept the manual unlock step at boot.

Changing Keys

You can change the encryption key (or switch between passphrase and key file) without re-encrypting the data:

# Change from key file to passphrase
zfs change-key -o keyformat=passphrase -o keylocation=prompt tank/private

# Change from passphrase to key file
zfs change-key -o keyformat=raw -o keylocation=file:///root/keys/new.key tank/private

This is a key-wrapping operation - the actual data encryption key doesn’t change, only the wrapper protecting it. It’s instantaneous regardless of dataset size.

Locking Datasets

When you’re done working with encrypted data, you can unload the key, making the dataset inaccessible until the key is loaded again:

# Unmount and unload the key
zfs unmount tank/private
zfs unload-key tank/private

ZFS Send/Recv: The Killer Feature for Backups

zfs send serializes a snapshot (or the difference between two snapshots) into a stream of bytes. zfs recv takes that stream and reconstructs the dataset on the other end. This is ZFS’s native backup and replication mechanism, and it’s remarkably powerful.

Basic Send/Recv

# Send a full snapshot to a file
zfs send tank/jails/mailserver@2026-03-13 > /backup/mailserver-2026-03-13.zfs

# Receive it into a different pool
zfs recv backup/jails/mailserver < /backup/mailserver-2026-03-13.zfs

Incremental Sends: Only the Changes

The real power is incremental sends. After the initial full send, subsequent sends only transfer the blocks that changed between two snapshots:

# First: full send of the initial snapshot
zfs send tank/jails/mailserver@monday | \
    zfs recv backup/jails/mailserver

# Later: incremental send (only changes since Monday)
zfs send -i tank/jails/mailserver@monday tank/jails/mailserver@tuesday | \
    zfs recv backup/jails/mailserver

The -i flag tells ZFS to send only the delta between the two snapshots. On a 500GB dataset where 2GB changed since the last snapshot, the incremental send transfers ~2GB, not 500GB. This makes daily off-site backups of large datasets entirely practical, even over modest network links.

Send/Recv Over SSH

For remote backups, pipe through SSH:

# Send to a remote backup server
zfs send -i tank/jails/mailserver@monday tank/jails/mailserver@tuesday | \
    ssh backup-server zfs recv -o mountpoint=none -o canmount=off backup/jails/mailserver

# For initial full replication with all properties
zfs send -R tank/jails/mailserver@tuesday | \
    ssh backup-server zfs recv -F -o mountpoint=none -o canmount=off backup/jails/mailserver

The -F flag on the receiving end forces a rollback to match the incoming stream, discarding any local changes on the backup target. This ensures your backup remains an exact replica of the source - but be careful not to use -F against a dataset that contains data you care about independently, as any local modifications will be destroyed.

The -o mountpoint=none -o canmount=off flags are equally important: without them, the backup server will try to mount the incoming datasets at their original mount points. If you’re replicating a dataset with mountpoint=/etc or mountpoint=/var/mail, the backup server would mount it right over its own live directories. Setting these overrides ensures received datasets are stored but never auto-mounted.

Raw Sends for Encrypted Datasets

When sending encrypted datasets, you can choose whether to send the data encrypted or decrypted:

# Raw send: data stays encrypted. Receiver doesn't need the key.
zfs send --raw tank/private@2026-03-13 | \
    ssh backup-server zfs recv backup/private

# Normal send: data is decrypted, then re-encrypted (or stored plain) at the receiver.
zfs send tank/private@2026-03-13 | \
    ssh backup-server zfs recv backup/private

Raw sends (--raw) are essential for secure off-site backups. The backup server receives and stores encrypted data without ever having the decryption key. If the backup server is compromised, the attacker gets encrypted blocks they can’t read. An important practical benefit: the receiving pool doesn’t need to have encryption support enabled at all - it just stores opaque blocks. This makes raw sends ideal for replicating to older backup servers or cloud storage targets that may not run a current OpenZFS version.

Sanoid and Syncoid: Automated Snapshot Management

Manually creating snapshots and running zfs send works, but it doesn’t scale. Sanoid automates snapshot creation and retention, and its companion tool syncoid automates zfs send/recv replication.

Sanoid is available in FreeBSD’s package repository:

pkg install sanoid

Sanoid uses a configuration file at /usr/local/etc/sanoid/sanoid.conf to define snapshot policies:

[tank/jails]
    use_template = production
    recursive = yes

[tank/home]
    use_template = production
    recursive = yes

[tank/postgresql]
    use_template = production

[template_production]
    hourly = 24
    daily = 30
    monthly = 12
    yearly = 2
    autosnap = yes
    autoprune = yes

This keeps 24 hourly, 30 daily, 12 monthly, and 2 yearly snapshots - all managed automatically. Old snapshots are pruned when they exceed the retention count. Run sanoid --cron from a cron job (typically every 15 minutes), and snapshot management becomes a solved problem.

Syncoid handles the replication side:

# Replicate a dataset to a remote server
syncoid tank/jails/mailserver backup-server:backup/jails/mailserver

# Replicate recursively (all child datasets)
syncoid -r tank/jails backup-server:backup/jails

# Use raw mode for encrypted datasets
syncoid --sendoptions="w" tank/private backup-server:backup/private

Syncoid automatically determines whether a full or incremental send is needed, creates temporary snapshots for consistency, handles the zfs send | ssh | zfs recv pipeline, and cleans up after itself. A single cron entry running syncoid gives you continuous off-site replication with minimal bandwidth usage.

A typical backup cron setup (shown here as root’s personal crontab via crontab -e; if you add these to /etc/crontab instead, add root as the user field after the time specification):

# Sanoid: manage local snapshots every 15 minutes
*/15 * * * * /usr/local/bin/sanoid --cron

# Syncoid: replicate to backup server daily at 2 AM
0 2 * * * /usr/local/bin/syncoid -r tank backup-server:backup/tank

Performance Tuning: ARC and L2ARC

ZFS uses a sophisticated caching system called the ARC (Adaptive Replacement Cache). The ARC lives in RAM and caches recently and frequently accessed data. Unlike traditional filesystem caches, the ARC uses an algorithm that balances recency and frequency, keeping both “recently used” and “frequently used” data cached.

On FreeBSD, ZFS is greedy by default - it will automatically use most of your system’s RAM for the ARC, leaving only a small reserve for the kernel. On a dedicated file server, this is exactly what you want and no tuning is needed. But if you’re running memory-hungry jails on the same machine (PostgreSQL, application runtimes), you need to restrict the ARC so it doesn’t starve your applications:

# Check current ARC size
sysctl kstat.zfs.misc.arcstats.size

# Restrict ARC to 8GB on a jail host (in /boot/loader.conf)
# Leave the rest for jails and applications
vfs.zfs.arc_max="8589934592"

For systems with an available SSD, L2ARC extends the cache to SSD storage. It’s a second-level cache: data evicted from the RAM-based ARC gets written to the L2ARC device, giving you a much larger effective cache:

# Add an SSD as L2ARC
zpool add tank cache nvd1

L2ARC is most effective when your working set is larger than available RAM but fits on an SSD. For read-heavy workloads on spinning disks (databases, mail servers), an L2ARC can provide dramatic performance improvements.

Warning: L2ARC isn’t free. Every block cached in L2ARC requires an index pointer in your primary ARC (RAM). If you add a massive 1TB NVMe as L2ARC to a system with limited RAM, the pointer overhead will starve the primary ARC and actually degrade overall performance. Size your L2ARC proportionally to your available RAM - not to the size of the SSD you happen to have lying around.

Similarly, a SLOG (Separate Log) device accelerates synchronous writes by providing a fast storage target for the ZFS Intent Log (ZIL):

# Add an SSD as SLOG
zpool add tank log nvd2

This is primarily useful for NFS servers, databases with fsync-heavy workloads, and iSCSI targets where synchronous write performance matters.

ZFS and Jails: Better Together

If you read the first article in this series, you know that Jails and ZFS are a natural pairing. Each jail gets its own dataset, which means:

Independent snapshots: Snapshot a jail before upgrading its packages. Roll back if something breaks. No impact on other jails.
Instant cloning: Want a test copy of your production mail jail? zfs clone creates one in seconds, sharing data with the original until divergence.
Per-jail quotas: Prevent one jail from consuming all available storage.
Independent compression and recordsize: A jail running PostgreSQL gets recordsize=8k. A jail serving static files gets recordsize=1M. Each workload gets the optimal setting.

# Snapshot a jail before upgrade
zfs snapshot tank/jails/mailserver@pre-upgrade

# Clone a jail for testing
zfs clone tank/jails/mailserver@pre-upgrade tank/jails/mailserver-test

# Set per-jail storage quota
zfs set quota=20G tank/jails/webmail

# Tune recordsize per workload
zfs set recordsize=8k tank/jails/database

Cloning is particularly powerful. A clone is a writable snapshot - it starts sharing all data with the original and only consumes space as you make changes. Standing up a test environment from production data takes seconds, not hours.

Common Pitfalls

Not setting recordsize before importing data: Changing recordsize only affects new writes. If you import a PostgreSQL database and then set recordsize=8k, the existing data keeps the old block size. Set it first.

Forgetting to scrub: ZFS can only repair corruption it knows about. Without regular scrubs, corruption sits undetected until something tries to read the affected block - possibly months later when the redundant copy has also degraded. Scrub weekly.

Letting the pool get too full: ZFS performance degrades as the pool fills up, particularly above 80% capacity. This is because ZFS uses a copy-on-write allocation strategy that becomes increasingly constrained as free space fragments. Monitor pool usage and expand before you hit 80%.

Using dedup without understanding the cost: ZFS supports block-level deduplication, but it requires enormous amounts of RAM for the deduplication table (roughly 5GB of RAM per TB of deduplicated storage). Unless you have a specific, measured use case and the RAM to support it, don’t enable dedup. Use compression instead - it’s nearly free.

Not using raw send for encrypted backups: If you zfs send an encrypted dataset without --raw, the data is decrypted for the send and arrives unencrypted at the receiver. This might be what you want, or it might defeat the entire purpose of encrypting it. Be intentional about which mode you use.

Conclusion

ZFS replaces an entire stack of tools - filesystem, volume manager, RAID, backup - with a single system that checksums everything, heals itself, compresses transparently, encrypts natively, and replicates efficiently. It’s the foundation that makes FreeBSD’s jail-based architecture practical at scale: each jail gets its own dataset with its own properties, snapshots, and quotas.

The features that matter most in practice aren’t the flashy ones. It’s the scrub that catches a degrading disk before data loss. It’s the .zfs directory that lets a user recover a deleted file without filing a ticket. It’s the incremental send that replicates 500GB of jail data by transferring 2GB of changes. These are the things that let you sleep at night.

If you take nothing else away from this article: enable compression, schedule scrubs, automate snapshots with sanoid, and replicate off-site with syncoid. That’s a complete data management strategy in four decisions.

References

FreeBSD Handbook - The Z File System
zpool(8) man page
zfs(8) man page
OpenZFS Documentation
Sanoid/Syncoid - Policy-driven snapshot management
ZFS 101 - Understanding ZFS storage and performance (Ars Technica)
Jeff Bonwick - ZFS: The Last Word in Filesystems - the original Sun Microsystems presentation

FreeBSD Foundationals: Jails - From Chroot on Steroids to Full Virtual Networks

2026-03-02T00:00:00+01:00

FreeBSD Jails have been around since FreeBSD 4.0, released in the year 2000. That makes them older than Linux cgroups, older than LXC, older than Docker, and older than most people’s understanding of what “containers” even means. Yet they remain one of the most elegant and underappreciated isolation mechanisms available on any operating system.

This article is the first in a series called FreeBSD Foundationals - covering core FreeBSD concepts that deserve more than a man page skim. We start with Jails because they’re central to how FreeBSD is deployed in practice: from hosting providers to Netflix’s CDN to small mail servers on rented hardware.

If you’ve used Docker or LXC on Linux, some concepts will feel familiar. But Jails are not Linux containers with a different name. The design philosophy, the networking model, and the security boundaries are fundamentally different. Understanding those differences is the point.

What Problem Do Jails Solve?

Unix has always had chroot, which changes the apparent root directory for a process. A process inside a chroot sees / as whatever directory you pointed it at. It can’t traverse above that point in the filesystem. Useful, but limited: chroot only isolates the filesystem view. A chrooted process still shares the network stack, the process table, IPC, and - crucially - the ability to escape the chroot if it has root privileges.

Jails extend this concept into a proper isolation boundary. A jailed process gets:

Its own filesystem root (like chroot, but enforced at the kernel level)
Its own process space - processes inside a jail can only see other processes in the same jail
Its own hostname and network identity
Restricted system calls - no loading kernel modules, no mounting filesystems (unless explicitly allowed), no modifying the host’s network configuration

The key insight: Jails are a kernel-enforced boundary, not a userspace trick. A root user inside a jail is not equivalent to root on the host. The kernel itself refuses operations that would break the isolation. This is a fundamentally different security model from “root in a Docker container” on Linux, where escaping to the host has historically been a much shorter path.

Classic Jails vs VNET Jails

There are two distinct approaches to jail networking, and understanding the difference matters for every design decision that follows.

Classic (IP-Based) Jails

In a classic jail, you assign one or more IP addresses to the jail. The jail shares the host’s network stack - it uses the host’s interfaces, the host’s routing table, the host’s firewall. The kernel simply restricts which addresses the jailed processes can bind to and communicate from.

Classic Jail networking:

  ┌───────────────────────────────────────────────┐
  │                  Host Kernel                  │
  │                                               │
  │   Network Stack (shared)                      │
  │   ┌───────────┐                               │
  │   │  vtnet0   │  203.0.113.50                 │
  │   │           │  10.0.0.11 (bound to jail A)  │
  │   │           │  10.0.0.12 (bound to jail B)  │
  │   └───────────┘                               │
  │                                               │
  │   ┌─────────────┐    ┌─────────────┐          │
  │   │   Jail A    │    │   Jail B    │          │
  │   │ can only    │    │ can only    │          │
  │   │ bind to     │    │ bind to     │          │
  │   │ 10.0.0.11   │    │ 10.0.0.12   │          │
  │   └─────────────┘    └─────────────┘          │
  └───────────────────────────────────────────────┘

This is simple and lightweight. No extra interfaces, no bridges, no routing between host and jail. The jail just sees “its” IP on the host’s interface. Classic jails work well when you need basic process isolation and don’t need the jail to run its own firewall, its own routing, or anything that requires full control over a network stack.

The limitation: because the jail shares the host’s network stack, it cannot run services that need raw socket access, can’t configure its own routes, can’t run its own DHCP client, and can’t have its own firewall rules. The jail has an IP address, but it doesn’t have a network.

VNET (Virtual Network) Jails

VNET jails get their own complete network stack. Not just an IP - a full, independent network stack with its own interfaces, its own routing table, its own ARP cache, and the ability to run network services that would be impossible in a classic jail.

VNET Jail networking:

  ┌────────────────────────────────────────────────────────────────┐
  │                        Host Kernel                             │
  │                                                                │
  │  Host Network Stack            Jail A Network Stack            │
  │  ┌──────────┐                  ┌──────────┐                    │
  │  │ vtnet0   │                  │  vnet0   │                    │
  │  │ bridge0  │                  │  lo0     │                    │
  │  │ e0a_jail │                  │ (own routing table)           │
  │  └──────────┘                  └──────────┘                    │
  │       │                              │                         │
  │       └──────── epair ───────────────┘                         │
  │            (virtual ethernet cable)                            │
  └────────────────────────────────────────────────────────────────┘

This is what you want for production workloads. A VNET jail can run its own DNS resolver, its own firewall (yes, pf inside a jail), handle its own IPv6 router advertisements, and generally behave like a separate machine. From the network’s perspective, it is a separate machine - just one that happens to share a kernel with the host.

The trade-off is complexity. VNET jails need virtual interfaces, bridges, and explicit routing between the host and jail networks. That complexity is what the rest of this article explains.

When to Use Which

Requirement	Classic Jail	VNET Jail
Simple process isolation	Yes	Overkill
Service needs to bind to a specific IP	Yes	Yes
Service needs raw sockets	No	Yes
Jail needs its own routing table	No	Yes
Jail needs its own firewall	No	Yes
Jail runs DNS resolver (unbound, etc.)	Tricky	Yes
Jail needs DHCP client	No	Yes
Performance overhead	Negligible	Minimal

For anything resembling a real server workload - a mail server, a web server with its own TLS stack, a database - VNET is the right choice. Classic jails still make sense for batch jobs, build environments, or situations where the jail is purely a filesystem and process boundary.

The VNET Networking Model

This is where most people get lost, so let’s build the mental model piece by piece.

Epair Interfaces: Virtual Ethernet Cables

An epair is a pair of virtual Ethernet interfaces connected back-to-back. Think of it as a virtual Ethernet cable with a plug on each end. Creating one gives you two interfaces: epair0a and epair0b. Whatever goes into one end comes out the other.

# Create an epair - this gives you epair0a and epair0b
ifconfig epair create
# Output: epair0a

The a side stays on the host. The b side gets moved into the jail. They’re connected at the kernel level - traffic sent into epair0a appears on epair0b, and vice versa. No routing, no forwarding decision - it’s a direct Layer 2 link, like plugging an Ethernet cable between two physical machines.

  Host                          Jail
  ┌──────────┐                  ┌──────────┐
  │          │    epair "wire"  │          │
  │  e0a_web ├──────────────────┤ e0b_web  │
  │          │                  │ (→vnet0) │
  └──────────┘                  └──────────┘
     stays                        moves
     on host                      into jail

In practice, you rename both sides to something meaningful (like e0a_mailserver and e0b_mailserver) so you can tell which pair belongs to which jail when you have a dozen of them.

The Bridge: Connecting Jails Together

Each epair connects exactly one jail to the host. But jails usually need to talk to each other and to the outside world. That’s where bridge0 comes in.

A bridge is a virtual network switch. You add the host-side (a) end of each epair to the bridge, assign the bridge an IP address (which becomes the jails’ default gateway), and now all jails are on the same Layer 2 network segment:

                        ┌─────────────────────────────────────────┐
                        │              bridge0                    │
                        │          10.0.0.1/24                    │
                        │    2001:db8:1234:5678::1/64             │
  Internet              │                                         │
     │                  │  ┌──────────┐  ┌──────────┐  ┌────────┐ │
     │   NAT (pf)       │  │e0a_mail  │  │e0a_web   │  │e0a_mon │ │
  ┌──┴──────┐           │  └────┬─────┘  └────┬─────┘  └───┬────┘ │
  │ vtnet0  │           └───────┼─────────────┼────────────┼──────┘
  │ (host)  │                   │             │            │
  └─────────┘             ┌─────┴─────┐ ┌─────┴─────┐ ┌────┴──────┐
                          │ mailserver│ │  webmail  │ │ monitor   │
                          │ 10.0.0.11 │ │ 10.0.0.12 │ │ 10.0.0.13 │
                          └───────────┘ └───────────┘ └───────────┘

This topology is the same as plugging physical servers into a physical switch. The bridge does MAC learning, forwards frames between ports, and the jails communicate at Layer 2. The host is the gateway - it has an IP on the bridge and runs pf to NAT the jail traffic out to the internet (for IPv4) and to filter what comes in.

For IPv6, if you have a large enough allocation, you can route a subnet directly to the bridge and give each jail a public IPv6 address. No NAT needed. This is exactly how dual-stack jail hosting should work.

Putting It Together: The Lifecycle of a VNET Jail

When a VNET jail starts, several things happen in sequence. The exec.prestart commands in jail.conf run before the jail is created, on the host:

Create the epair: ifconfig epair create produces epair0a and epair0b
Rename and bring up both ends: ifconfig epair0a up name e0a_mailserver, same for the b side
Add the host side to the bridge: ifconfig bridge0 addm e0a_mailserver

Then the jail starts. The vnet.interface directive moves the b-side interface into the jail’s network stack:

Move e0b_mailserver into the jail - it disappears from the host’s interface list
Inside the jail’s rc.conf: rename e0b_mailserver to vnet0, assign IP addresses, set the default route to the bridge IP

When the jail stops, exec.poststop cleans up:

Destroy e0a_mailserver - this automatically destroys the paired e0b_mailserver too

An important detail about the exec.prestart lines: each line runs in its own shell invocation. The $epair0 variable from the ifconfig epair create call only exists within the same line, which is why the create, rename and bring-up are chained with && into a single line in the config below.

Here’s what this looks like in jail.conf:

mailserver {
    # Filesystem and process isolation
    path = "/jails/mailserver";
    host.hostname = "mailserver";
    enforce_statfs = 2;
    mount.devfs;
    devfs_ruleset = 4;
    exec.clean;              # reset environment before executing jail commands

    # Standard start/stop
    exec.start = '/bin/sh /etc/rc';
    exec.stop = '/bin/sh /etc/rc.shutdown';
    exec.consolelog = /var/log/jails/mailserver_console.log;

    # VNET networking
    vnet;
    vnet.interface = "e0b_mailserver";

    # Create epair, rename, bridge - runs on the host before jail starts
    exec.prestart += "epair0=\$(ifconfig epair create) && ifconfig \${epair0} up name e0a_mailserver && ifconfig \${epair0%a}b up name e0b_mailserver";
    exec.prestart += "ifconfig bridge0 addm e0a_mailserver";
    exec.prestart += "ifconfig e0a_mailserver description \"vnet0 host interface for Jail mailserver\"";

    # Clean up - runs on the host after jail stops
    exec.poststop += "ifconfig e0a_mailserver destroy";
}

And the corresponding rc.conf inside the jail at /jails/mailserver/etc/rc.conf:

# Rename the interface that was moved in by vnet.interface
ifconfig_e0b_mailserver_name="vnet0"

# IPv4
ifconfig_vnet0="inet 10.0.0.11 netmask 255.255.255.0"
defaultrouter="10.0.0.1"

# IPv6 (public address, no NAT needed)
ifconfig_vnet0_ipv6="inet6 2001:db8:1234:5678::11/64"
ipv6_defaultrouter="2001:db8:1234:5678::1"

# Description for ifconfig output
ifconfig_vnet0_descr="jail interface for bridge0"

# Standard jail housekeeping
syslogd_flags="-ss"
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"
cron_flags="-J 60"
moused_nondefault_enable="NO"
dumpdev="NO"

# Actual services
postfix_enable="YES"
dovecot_enable="YES"
rspamd_enable="YES"
local_unbound_enable="YES"

A few things to note about the rc.conf pattern:

syslogd_flags="-ss" disables syslogd’s network socket. Without this, syslogd inside the jail tries to open a socket it shouldn’t need.
sendmail_*="NO" disables all sendmail components. Even if you’re running Postfix, the base system’s sendmail cron jobs will try to start otherwise.
cron_flags="-J 60" tells cron to stagger job execution with a random delay of up to 60 seconds. Prevents all jails from firing cron jobs at the exact same second.

Host Configuration: Enabling the Infrastructure

The host needs a few things configured in /etc/rc.conf before jails can use VNET networking:

# Create the bridge at boot
cloned_interfaces="bridge0"

# Bridge gets the gateway IP for the jail network
ifconfig_bridge0="inet 10.0.0.1 netmask 255.255.255.0"
ifconfig_bridge0_ipv6="inet6 2001:db8:1234:5678::1/64"

# Enable IP forwarding (the host is the jails' router)
gateway_enable="YES"
ipv6_gateway_enable="YES"

# Enable the jail subsystem and packet filter
jail_enable="YES"
pf_enable="YES"

The gateway_enable lines are critical. Without them, the kernel won’t forward packets between the bridge (jail network) and the external interface. Your jails would have IP addresses but no connectivity beyond the bridge.

Firewalling with pf

With VNET jails, the host is the router. All traffic between jails and the internet passes through the host’s network stack, which means pf on the host controls everything.

NAT for IPv4

Unless you have a large IPv4 allocation (unlikely and expensive), your jails will use private addresses on the bridge and NAT through the host’s public IP:

ext_if = "vtnet0"
jail_net = "10.0.0.0/24"

# NAT jail traffic going to the internet
nat on $ext_if inet from $jail_net to any -> ($ext_if)

The parentheses around ($ext_if) tell pf to dynamically resolve the interface’s current IP. If the host’s IP changes (DHCP), the NAT rule adapts automatically.

Redirecting Inbound Traffic

For services that need to be reachable from the internet over IPv4, use rdr to forward incoming connections to the jail’s private address:

mail_ipv4 = "10.0.0.11"

# SMTP from anywhere
rdr pass on $ext_if inet proto tcp from any to ($ext_if) port 25 -> $mail_ipv4

# IMAP and submission
rdr pass on $ext_if inet proto tcp from any to ($ext_if) \
    port {143, 587, 4190} -> $mail_ipv4

IPv6: No NAT Needed

With IPv6, each jail has a globally routable address. No NAT, no rdr - just pass rules:

mail_ipv6 = "2001:db8:1234:5678::11"

pass in quick on $ext_if inet6 proto tcp from any to $mail_ipv6 \
    port 25 flags S/SA keep state
pass in quick on $ext_if inet6 proto tcp from any to $mail_ipv6 \
    port {143, 587, 4190} flags S/SA keep state

This is one of the practical advantages of deploying jails on a host with IPv6: the networking model becomes dramatically simpler. Each jail is a first-class citizen on the internet, directly addressable, with filtering handled by pass/block rules rather than NAT and redirection gymnastics.

Jail Egress

Jails also need to initiate outbound connections (DNS queries, package downloads, sending mail). Allow this with a broad egress rule on the bridge:

jail_net = "10.0.0.0/24"
jail_net6 = "2001:db8:1234:5678::/64"

# Allow jails to reach the internet, but not each other's private range
pass in quick on bridge0 from $jail_net to ! $jail_net keep state
pass in quick on bridge0 inet6 from $jail_net6 to ! $jail_net6 keep state

The ! $jail_net exclusion means this rule only matches traffic leaving the jail network towards the internet. Traffic between jails on the same bridge is switched at Layer 2 and never passes through pf in the default FreeBSD configuration (controlled by net.link.bridge.pfil_member and net.link.bridge.pfil_bridge sysctls, both off by default). If you need to restrict inter-jail communication, that requires either separate bridges per jail or enabling pfil_member and adding explicit block rules on the bridge member interfaces.

devfs Rules: Controlling Device Access

When a jail has mount.devfs enabled, it gets a /dev filesystem. But you don’t want a jail to see the host’s disk devices, USB devices, or hardware random number generators it shouldn’t touch. devfs rulesets control which device nodes appear inside the jail.

The relevant rulesets are defined in /etc/devfs.rules on the host. FreeBSD ships with a default set, and ruleset 4 (devfsrules_jail) is specifically designed for jails:

# View the built-in rulesets
devfs rule -s 4 show

Ruleset 4 typically hides everything and then unhides only what a jail needs:

/dev/null, /dev/zero, /dev/random, /dev/urandom - standard Unix devices
/dev/fd/* - file descriptors
/dev/stdin, /dev/stdout, /dev/stderr - standard streams

What it doesn’t include: /dev/ad*, /dev/da* (disk devices), /dev/mem (physical memory), /dev/kmem (kernel memory), /dev/io (I/O port access). A jailed process cannot read raw disk data or probe kernel memory, even as root.

Notably, ruleset 4 also does not unhide /dev/bpf* (Berkeley Packet Filter devices). This matters for VNET jails: without bpf, tools like tcpdump and dhclient inside the jail won’t work. If you need either, you’ll want a custom ruleset that adds bpf access - shown below.

You reference the ruleset in jail.conf:

devfs_ruleset = 4;

If you need to customize device access (for example, giving a jail access to /dev/pf for its own packet filter, or /dev/zvol/* for ZFS volume access), you create a custom ruleset in /etc/devfs.rules:

[devfsrules_jail_with_pf=100]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path 'bpf*' unhide
add path pf unhide

Then reference it as devfs_ruleset = 100 in the jail’s configuration.

Security: What Jails Actually Prevent

Jails enforce security at the kernel level. Here’s what a root user inside a jail cannot do, regardless of their privilege level:

Load or unload kernel modules - no kldload, no kldunload
Mount or unmount filesystems - unless enforce_statfs and allow.mount* are explicitly relaxed
Modify the host’s network - can’t touch the host’s interfaces, routes, or firewall
See processes outside the jail - ps aux shows only jail-local processes
Access raw disk devices - blocked by devfs rules
Modify the running kernel - no access to /dev/mem, /dev/kmem
Create device nodes - no mknod
Modify the system clock - no adjtime, no settimeofday

enforce_statfs

The enforce_statfs directive controls what the jail can see about mounted filesystems:

enforce_statfs = 0 - jail sees all mount points (don’t use this)
enforce_statfs = 1 - jail sees mount points beneath its root
enforce_statfs = 2 - jail sees only its own mount points (recommended)

At level 2, df inside the jail shows only the jail’s own filesystems. The jail can’t discover the host’s ZFS pool layout, other jails’ mount points, or the overall disk topology.

Managing Jails

Starting and Stopping

# Start all jails
service jail start

# Start a specific jail
service jail start mailserver

# Stop a specific jail
service jail stop webmail

# Restart (stop + start) a specific jail
service jail restart mailserver

Entering a Running Jail

# Get a shell inside a jail
jexec mailserver /bin/sh

# Run a specific command
jexec mailserver pkg update

# See running jails
jls

jls shows all active jails with their JID (Jail ID), IP addresses, hostname, and path. It’s the FreeBSD equivalent of docker ps.

Filesystem: ZFS Datasets

While you can use any directory as a jail root, ZFS datasets are the standard approach for production:

# Create a dataset for your jail infrastructure
zfs create zroot/jails

# Create a dataset for a specific jail
zfs create zroot/jails/mailserver

# Install a fresh userland into it
bsdinstall jail /jails/mailserver

ZFS gives you snapshots (zfs snapshot zroot/jails/mailserver@before-upgrade), cloning (create a new jail from an existing snapshot in seconds), quotas, compression, and send/receive for backup and migration. This is comparable to Docker’s layered images, but without the overlay filesystem complexity.

Sometimes a jail needs access to data that lives outside its filesystem root. The nullfs mount (a loopback mount, similar to mount --bind on Linux) makes a host directory visible inside the jail:

# In jail.conf
mount += "/var/vmail /jails/mailserver/var/vmail nullfs rw 0 0";

This mounts the host’s /var/vmail at /jails/mailserver/var/vmail. The mail jail sees it as a local directory. This is useful for shared mail spools, database directories on fast storage, or any data that should persist independently of the jail’s lifecycle.

A Complete Example: Three-Jail Architecture

Here’s a complete topology for a generic server running three VNET jails:

                                  Internet
                                     │
                            ┌────────┴────────┐
                            │     vtnet0      │
                            │   203.0.113.50  │
                            │  (public IPv4)  │
                            └────────┬────────┘
                                     │
                                 pf (NAT/RDR)
                                     │
     ┌───────────────────────────────┴──────────────────────────────┐
     │                          bridge0                             │
     │                      10.0.0.1/24                             │
     │               2001:db8:1234:5678::1/64                       │
     │                                                              │
     │  ┌───────────────┐  ┌──────────────┐  ┌───────────────────┐  │
     │  │e0a_mailserver │  │ e0a_webmail  │  │  e0a_monitor      │  │
     │  └───────┬───────┘  └──────┬───────┘  └─────────┬─────────┘  │
     └──────────┼─────────────────┼────────────────────┼────────────┘
                │ epair           │ epair              │ epair
     ┌──────────┴──────┐  ┌───────┴───────┐  ┌─────────┴──────────┐
     │   mailserver    │  │    webmail    │  │      monitor       │
     │   10.0.0.11     │  │   10.0.0.12   │  │     10.0.0.13      │
     │                 │  │               │  │                    │
     │   Postfix       │  │    nginx      │  │   Prometheus       │
     │   Dovecot       │  │    php-fpm    │  │   Grafana          │
     │   Rspamd        │  │               │  │                    │
     │   Unbound       │  │               │  │                    │
     └─────────────────┘  └───────────────┘  └────────────────────┘

Each jail gets its own epair, its own IP, and runs only the services it needs. The mailserver jail handles SMTP, IMAP, and spam filtering. The webmail jail runs a web interface. The monitor jail runs Prometheus and Grafana for metrics collection.

This separation means a vulnerability in the webmail application (PHP, web framework) doesn’t automatically grant access to the mail spool, the SMTP relay, or the host system. Each blast radius is contained.

Common Pitfalls

Forgetting gateway_enable: Your jails will get IPs, the bridge will be up, but nothing can reach the internet. The host kernel silently drops forwarded packets.

Not creating /var/log/jails/: If the exec.consolelog directory doesn’t exist, jail startup fails silently. Create it ahead of time:

mkdir -p /var/log/jails

Interface naming conflicts: If two jails try to create epairs simultaneously without unique names, you get collisions. The naming convention in jail.conf (e0a_<jailname> / e0b_<jailname>) avoids this.

DNS resolution inside jails: Jails don’t automatically inherit the host’s /etc/resolv.conf. Either copy it into the jail’s root, run a local resolver like unbound inside the jail, or point resolv.conf at the bridge IP if the host runs a resolver there.

Forgetting to enable jail services: After bsdinstall jail, the base system is minimal. Services need to be installed (pkg -j mailserver install postfix) and enabled in the jail’s rc.conf. Don’t forget sshd_enable="YES" if you want to SSH into the jail directly (though jexec from the host is often sufficient).

Conclusion

Jails are not a historical curiosity. They’re a production-grade isolation mechanism backed by over 25 years of kernel-level enforcement. VNET gives each jail a real network identity, epairs provide the plumbing, bridges create the fabric, and pf ties it all together with filtering and NAT.

The mental model to carry away: a VNET jail is a separate machine that shares a kernel with the host. It has its own network stack, its own filesystem view, and its own process space. What it doesn’t have is the ability to affect the host - and that’s enforced by the kernel, not by convention.

The next article in this series will cover ZFS - the filesystem that makes jail management, snapshots, and backups actually practical at scale.

References

FreeBSD Handbook - Jails
jail(8) man page
jail.conf(5) man page
devfs.rules(5) man page
epair(4) man page
if_bridge(4) man page
pf.conf(5) man page
Poul-Henning Kamp, Robert N. M. Watson - Jails: Confining the Omnipotent Root - the original 2000 paper introducing Jails

Neovim Crash Course for Sysadmins: The 20% That Solve 80% of the Pain

2026-02-27T00:00:00+01:00

This is not a beginner’s guide. If you’ve used Vim daily for a year and still occasionally fight with paste behavior, you’re the target audience. This article covers the things I got wrong (or never properly learned) after fifteen years of daily Vim usage - and the moment everything clicked.

The focus is practical: editing configuration files, YAML, shell scripts, and infrastructure code. No Vim philosophy lectures. No “Vim is a language” metaphors. Just the patterns that make the biggest difference for sysadmins and DevOps engineers who spend their days in terminals.

Motions That Actually Matter
Jumping, Not Crawling
The Power of Text Objects
Copy/Paste: The Thing You’ve Been Doing Wrong
Search and Replace: Scoped to What You Need
YAML: The Pain and the Antidote
Registers and Macros: Automation for the Lazy
Splits, Buffers, and Efficient File Navigation
The Neovim Advantage: LSP and Treesitter
Practical Combos: The Cheat Sheet
Wrapping Up

Motions That Actually Matter

Everyone knows hjkl. Most people know w and b. Here’s what separates fast editing from “I’ll just use sed”:

Jumping, Not Crawling

Motion	What it does
`f{char}` / `F{char}`	Jump to next/previous occurrence of `{char}` on the line
`t{char}` / `T{char}`	Jump to just before next/previous `{char}`
`;` and `,`	Repeat last `f`/`t` forward/backward
`%`	Jump to matching bracket/parenthesis
`{` / `}`	Jump to previous/next empty line (paragraph)
`Ctrl-d` / `Ctrl-u`	Half-page down/up
`*` / `#`	Search word under cursor forward/backward
`gd`	Go to local definition of word under cursor

The f/t family is criminally underused. Editing a firewall rule and need to change the port number? f: jumps to the colon, l moves one right, cw changes the word. Three keystrokes instead of mashing w twelve times.

The Power of Text Objects

This is where Vim stops being a text editor and starts being a scalpel. Text objects work with any operator (d, c, y, v):

Object	Meaning	Example
`iw` / `aw`	inner word / a word (includes trailing space)	`ciw` - change word under cursor
`i"` / `a"`	inside quotes / around quotes	`ci"` - change contents of quoted string
`i(` / `a(`	inside parentheses / around parens	`di(` - delete contents between `()`
`i{` / `a{`	inside braces / around braces	`ci{` - change block contents
`ip` / `ap`	inner paragraph / a paragraph	`yip` - yank entire paragraph
`it` / `at`	inside XML/HTML tag / around tag	`cit` - change tag contents

Visualizing Text Objects

Text objects are easier to grasp with a concrete example. Given this code:

function_name() {
    some_value = true
}

di{ deletes some_value = true - everything between the braces, leaving { } intact
da{ deletes { some_value = true } - the braces AND their contents

Same logic applies to quotes: di" deletes just the string contents, da" deletes the quotes too. For paragraphs, dip deletes the text, dap also removes the trailing blank line.

Real-world example - you have a Jinja2 template variable {{ old_value }} and need to change it:

ci{  →  deletes old_value, leaves you in insert mode between the braces

One combo instead of navigating, selecting, deleting, typing.

Copy/Paste: The Thing You’ve Been Doing Wrong

This was my fifteen-year blind spot.

The Problem

You visually select some lines with v, yank with y, move somewhere, hit p - and the text lands in the middle of a line instead of on a new line below. You undo, try P, and it ends up in the middle of the line above. Familiar?

Why It Happens

Vim tracks whether a yank was characterwise, linewise, or blockwise:

v (characterwise) → y creates a characterwise register → p inserts inline
V (linewise) → y creates a linewise register → p inserts below current line
Ctrl-v (blockwise) → y creates a blockwise register → p inserts as a column

The yank type determines the paste behavior. That’s it. That’s the whole mystery.

The Fix

Use V (Visual Line) when you want to copy/paste lines. This is the single biggest quality-of-life improvement:

V       → enter visual line mode
jjj     → select lines downward
y       → yank (linewise)
}       → jump to where you want it
p       → paste below current line. Clean. New line. No surprises.

Quick Reference

Want to…	Use
Yank current line	`yy`
Yank 5 lines	`5yy`
Yank a paragraph	`yip`
Yank lines visually	`V` + move + `y`
Paste below	`p` (with linewise yank)
Paste above	`P` (with linewise yank)
Force paste as new line	`:put` (below) / `:put!` (above)

The :put command is your safety net - it always pastes as a new line, regardless of how you yanked. Useful when you’ve already yanked characterwise and don’t want to redo it.

Named Registers: Your Clipboard Slots

Vim has 26 named registers ("a through "z). Use them:

"ayy    → yank current line into register a
"bV3jy  → yank 4 lines into register b
"ap     → paste from register a
"bp     → paste from register b

The system clipboard is register "+:

"+yy    → yank line to system clipboard
"+p     → paste from system clipboard

For Neovim, you can unify the clipboard by adding to your config:

vim.opt.clipboard = 'unnamedplus'

Now y and p use the system clipboard directly. Whether you want this is a matter of taste - I prefer keeping Vim’s registers separate and using "+ explicitly when I need the system clipboard.

Search and Replace: Scoped to What You Need

Global Replace

The classic everyone knows:

:%s/old/new/g

% means the entire file. g means all occurrences per line.

Replace Only in Visual Selection

Select a block with V (or v), then press :. Vim auto-fills the range:

:'<,'>s/old/new/g

This replaces only within the selected lines. But here’s the subtlety: with V (Visual Line), this replaces in the entire lines. If you selected with v (characterwise) and want to restrict the match to exactly the highlighted text:

:'<,'>s/\%Vold/new/g

The \%V atom constrains the match to the visual selection boundary, not just the lines it spans. Niche, but invaluable when you need surgical precision.

Useful Flags

Flag	Effect
`g`	All occurrences per line (not just first)
`c`	Confirm each replacement
`i`	Case insensitive
`I`	Case sensitive (overrides `ignorecase` setting)
`n`	Count matches without replacing

The c flag is underrated. :%s/foo/bar/gc lets you step through every match and decide with y/n. Much safer than blind replace on a production config.

Multi-File Replace with :argdo / :cfdo

Need to replace across multiple files? Open them and:

:args *.yaml
:argdo %s/old_value/new_value/g | update

Or use the quickfix list after a grep:

:vimgrep /pattern/ **/*.yaml
:cfdo %s/pattern/replacement/g | update

YAML: The Pain and the Antidote

YAML editing is where Vim either shines or makes you want to throw your keyboard. Here’s how to make it shine.

Indentation: The Basics

>>      → indent current line one shiftwidth
<<      → unindent current line
>ip     → indent entire paragraph
5>>     → indent 5 lines
V5j>    → visual select 6 lines, indent

In Visual mode, > indents and then exits Visual mode. Use gv to reselect the same area, or better - just use . to repeat:

V5j>    → select and indent
.       → indent the same lines again (still selected by `'<,'>`)

Set Up Your YAML Defaults

Essential in your Neovim config:

vim.api.nvim_create_autocmd('FileType', {
  pattern = 'yaml',
  callback = function()
    vim.opt_local.shiftwidth = 2
    vim.opt_local.tabstop = 2
    vim.opt_local.softtabstop = 2
    vim.opt_local.expandtab = true
    vim.opt_local.cursorcolumn = true  -- vertical line at cursor column
  end,
})

cursorcolumn draws a vertical highlight at your cursor’s column position - an instant visual guide for YAML indentation alignment.

Moving Blocks Up and Down

Rearranging YAML keys or Ansible tasks:

:m+1    → move current line down one
:m-2    → move current line up one

Or visually select a block and move it:

V3j     → select 4 lines
:m '>+1 → move block down one line
:m '<-2 → move block up one line

Bind these for convenience:

-- Move lines up/down in visual mode
vim.keymap.set('v', 'J', ":m '>+1<CR>gv=gv")
vim.keymap.set('v', 'K', ":m '<-2<CR>gv=gv")

Now J/K in Visual mode moves the selected block and re-indents. Invaluable for reordering Ansible tasks.

Folding YAML Sections

vim.opt.foldmethod = 'indent'   -- YAML's structure IS its indentation
vim.opt.foldlevelstart = 99     -- start with everything unfolded

Then:

Key	Action
`za`	Toggle fold under cursor
`zM`	Close all folds
`zR`	Open all folds
`zc` / `zo`	Close / open one fold

Folding by indent works perfectly for YAML because indentation is the structure. A folded Ansible playbook shows you just the task names - like a table of contents.

Registers and Macros: Automation for the Lazy

The Dot Command

The most powerful single key in Vim: . repeats the last change.

ciw"new_value"<Esc>   → change word to "new_value"
n                      → jump to next search match
.                      → apply the same change
n.n.n.                 → repeat across all matches

This pattern - search, change, repeat - handles 90% of “I need to change this in five places” without reaching for :%s.

Quick Macros

Record with q{register}, replay with @{register}:

qa              → start recording into register a
0f:w            → jump to start of line, find colon, move to next word
ciw"<C-r>""     → change word, insert quote, paste deleted word, close quote
Esc             → back to normal mode
j               → move down
q               → stop recording

5@a             → replay 5 times

Note: ciw"<C-r>"" is the native way - ciw deletes the word, " starts the quoted string, <C-r>" (Ctrl-r followed by double-quote) pastes the deleted word from the default register, and the final " closes it.

For YAML quoting, this is also a great use case for vim-surround or mini.surround. With mini.surround, ysiw" wraps the word under cursor in quotes in a single command. No macro needed.

Replaying the Last Macro

@@ repeats the last played macro. Combined with a count: 20@@ - fire and forget.

Sysadmins often edit multiple related files: the playbook and the inventory, the nginx config and the upstream block, pf.conf and the anchor file.

Splits

:sp filename    → horizontal split
:vsp filename   → vertical split
Ctrl-w h/j/k/l → navigate between splits
Ctrl-w =        → equalize split sizes
Ctrl-w _        → maximize current horizontal split
Ctrl-w |        → maximize current vertical split
Ctrl-w o        → close all other splits (:only)

Buffers

:e filename     → open file in current buffer
:ls             → list all open buffers
:bn / :bp       → next / previous buffer
:b <partial>    → switch to buffer by partial name match
:bd             → close buffer

:b with tab completion and partial matching is surprisingly fast: :b pf<Tab> jumps to pf.conf if it’s open.

Jumping Between Recent Files

Ctrl-o  → jump back through jump list
Ctrl-i  → jump forward
``      → jump to last position before latest jump
`"      → jump to position when last editing this file

Ctrl-o is the “undo for navigation.” Jumped somewhere with * or gd and want to go back? Ctrl-o. Multiple times if needed.

The Neovim Advantage: LSP and Treesitter

If you’re still on vanilla Vim, here’s what you’re missing. Neovim’s built-in LSP client and Treesitter integration transform YAML editing:

YAML Language Server

With yaml-language-server configured, you get:

Schema validation - red squiggles when your Kubernetes manifest has a wrong field
Auto-completion - Ctrl-x Ctrl-o suggests valid keys for your schema
Hover docs - K shows documentation for the key under cursor

A minimal LSP setup in init.lua:

-- Requires nvim-lspconfig plugin
require('lspconfig').yamlls.setup({
  settings = {
    yaml = {
      schemas = {
        ['https://json.schemastore.org/ansible-playbook'] = 'playbook*.yml',
        ['https://json.schemastore.org/github-workflow'] = '.github/workflows/*.yml',
      },
    },
  },
})

Treesitter

Treesitter gives you syntax-aware text objects and highlighting:

require('nvim-treesitter.configs').setup({
  ensure_installed = { 'yaml', 'bash', 'lua', 'json', 'toml', 'python' },
  highlight = { enable = true },
  indent = { enable = true },
})

With Treesitter, indentation and folding become structure-aware instead of relying on raw indent levels. It’s the difference between “this line has 4 spaces” and “this is a mapping value inside a sequence item.”

Practical Combos: The Cheat Sheet

A reference for the patterns that come up daily in sysadmin work:

Situation	Keystrokes	What happens
Change a value in quotes	`ci"`	Deletes contents between quotes, enters insert mode
Delete everything inside braces	`di{`	Clears the block
Yank a whole YAML section	`yap` or `yip`	Direct paragraph yank (no visual mode needed)
Comment 10 lines	`Ctrl-v 9j I# <Esc>`	Block insert `#` at start of 10 lines
Uncomment 10 lines	`Ctrl-v 9j ll x`	Block select `#` (two columns) and delete
Replace in selection only	`V` select, `:'<,'>s/a/b/g`	Scoped search-replace
Indent a block more	`V` select, `>`	Shift right by `shiftwidth`
Sort lines	`V` select, `:sort`	Alphabetical sort
Remove duplicate lines	`V` select, `:sort u`	Sort and deduplicate
Reformat a long line	`gq` + motion (`gqip` for paragraph)	Wrap to `textwidth`
Repeat last change everywhere	`n.n.n.`	Search next, apply same edit
Save file you opened without sudo	`:w !sudo tee %`	Writes via sudo without restarting the editor
Undo by time	`:earlier 5m` / `:later 5m`	Jump to file state 5 minutes ago/ahead

Wrapping Up

Vim rewards investment on a curve - the first week is brutal, the first year is productive, and the next decade is about discovering that you’ve been doing basic things inefficiently the entire time. The patterns in this article represent the subset that matters most for infrastructure work: efficient navigation, correct yank/paste semantics, scoped replacements, and YAML-specific workflows.

The best way to internalize these is not to memorize the table above but to deliberately use one new pattern each day until it becomes muscle memory. Start with V instead of v. Then graduate to ci". Then text objects. Each one compounds.

If you want a ready-made Neovim configuration optimized for Ansible, Python, and YAML work, check out nvim-ansible on Codeberg - it implements most of the patterns and plugins discussed in this article.

And if you’ve been using Vim for fifteen years and just learned that V solves your paste problems - you’re in good company.

Running Your Own AS: Going Multi-Homed with iBGP and three Transits

2026-02-26T00:00:00+01:00

The previous article covered the basics: obtain an AS number and IPv6 prefix, build a single FreeBSD BGP router with two upstream providers, and tunnel the prefix to downstream servers. That setup works. But it has a property that feels wrong once you’ve operated it for a while: all your traffic converges on one point. One machine, one location, two upstreams. Everything in, everything out, through the same box.

Multi-homing addresses this at multiple levels. More upstream providers give inbound path diversity. A second Point of Presence at a different network means traffic from different parts of the internet can enter through different doors, and you can engineer which traffic uses which path. This article documents the evolution from that single-router setup to a distributed network: two FreeBSD BGP routers connected by iBGP and three upstream GRE-based transit providers.

The routing concepts involved - iBGP, local-preference, next-hop-self, stateless transit filtering - are the same ones any real network operator deals with. The hardware is two virtual machines.

Note on addresses: All provider-assigned IP addresses, tunnel endpoints, and management IPs have been replaced with RFC 5737 / RFC 3849 documentation ranges. AS201379 and the prefix 2a06:9801:1c::/48 are real public BGP resources and shown as-is. Upstream AS numbers (AS209533, AS209735, AS212895, AS64515) are equally visible in public routing tables.

Why Add More?

The original two-upstream setup had one failure domain and one point of path selection. Three practical problems motivated the expansion:

Route diversity. Different parts of the internet have different peering arrangements. A network that transits heavily through Vultr’s infrastructure prefers paths that show up in Vultr’s BGP. A network at DE-CIX Frankfurt prefers the shorter path through a Frankfurt exchange point. With one upstream set, you can only be optimally reachable for the networks whose traffic happens to prefer your specific providers.

Single point of failure. One router means one failure domain. A reboot or misconfiguration takes down all reachability simultaneously.

Traffic engineering granularity. AS-path prepending lets you make one path look less preferred, but only when the receiving network actually has a choice of paths. Multiple announcement points give that choice.

Architecture

The setup has two tiers: a core router (hobgp) at a Hetzner facility in Nuremberg that handles three GRE upstreams and an edge router (vtbgp) on Vultr in Frankfurt that announces our prefix natively into Vultr’s network. The two routers are connected by a GIF tunnel running iBGP.

                    ┌──────────────────────────────────────────────────┐
                    │                Default-Free Zone                  │
                    └──┬─────────────┬──────────────┬──────────────────┘
                       │             │              │
                   AS209533      AS209735       AS212895
                   (iFog)        (Lagrange)     (route64)
                       │             │              │
                  GRE tunnel    GRE tunnel     GRE tunnel
                       │             │              │
                  ┌────┴─────────────┴──────────────┴────────────┐
                  │                  hobgp (Core)                 │
                  │          FreeBSD + FRR, AS201379              │
                  │          2a06:9801:1c::/48                    │
                  └──────────────────────┬────────────────────────┘
                                         │
                                    GIF tunnel (iBGP)
                                         │
                  ┌──────────────────────┴────────────────────────┐
                  │                  vtbgp (Edge)                  │
                  │          FreeBSD + FRR, AS201379              │
                  │          Native BGP → AS64515 (Vultr)          │
                  └───────────────────────────────────────────────┘
                                         │
                                     AS64515
                                      (Vultr)
                                         │
                              ┌──────────┴──────────┐
                              │   Default-Free Zone   │
                              └───────────────────────┘

hobgp also tunnels downstream (as in the previous article):
                           ┌────────────┐
                           │   radon    │
                           │  :1000::/64│
                           └────────────┘

The key routing properties of this design: hobgp is the actual forwarding hub - all traffic to and from downstream servers flows through it. vtbgp is an announcement-only node; traffic that arrives there is immediately sent through the iBGP tunnel to hobgp for forwarding. This asymmetry is intentional and efficient: Vultr’s infrastructure sees our prefix announced locally, which shortens the path for Vultr-connected networks, without requiring vtbgp to carry the full routing table or forward arbitrary internet traffic.

The Core Router: hobgp

Network Configuration

The rc.conf now manages three GRE upstreams and a GIF tunnel to the edge router, in addition to the downstream server tunnels from the original setup:

hostname="hobgp"
kern_securelevel_enable="YES"
kern_securelevel="2"
kld_list="if_gif if_gre"

# Physical interface (Hetzner)
ifconfig_vtnet0="inet 198.51.100.10/32 -lro -tso"
ifconfig_vtnet0_ipv6="inet6 2001:db8:1c19::1/64"
defaultrouter="198.51.100.1"
ipv6_defaultrouter="fe80::1%vtnet0"

# Loopback: router's address within our /48
ifconfig_lo0_alias0="inet6 2a06:9801:1c::1 prefixlen 64"

# Tunnel interfaces
cloned_interfaces="gif0 gif1 gre0 gre1 gre2"

# GRE to iFog (BGPTunnel.com free transit)
ifconfig_gre0="tunnel 198.51.100.10 198.51.100.44"
ifconfig_gre0_ipv6="inet6 2001:db8:300::2 prefixlen 126"
ifconfig_gre0_descr="Transit-iFog-Frankfurt BGPTunnel"

# GRE to Lagrange (UK transit, backup path)
ifconfig_gre1="tunnel 198.51.100.10 198.51.100.45"
ifconfig_gre1_ipv6="inet6 fd00:ca::2 prefixlen 64"
ifconfig_gre1_descr="Transit-Lagrange-UK"

# GRE to route64.org (Frankfurt transit)
ifconfig_gre2="tunnel 198.51.100.10 198.51.100.46"
ifconfig_gre2_ipv6="inet6 2001:db8:400::2 prefixlen 64"
ifconfig_gre2_descr="Transit-route64-Frankfurt"

# GIF to downstream VPS (radon)
ifconfig_gif0="tunnel 198.51.100.10 203.0.113.10 mtu 1480"
ifconfig_gif0_ipv6="inet6 2a06:9801:1c:ffff::1 2a06:9801:1c:ffff::2 prefixlen 128"
ifconfig_gif0_descr="Tunnel-to-Radon-Server"

# GIF to Vultr edge router (vtbgp) - iBGP link
ifconfig_gif1="tunnel 198.51.100.10 203.0.113.20 mtu 1480"
ifconfig_gif1_ipv6="inet6 2a06:9801:1c:fffe::1 2a06:9801:1c:fffe::2 prefixlen 128"
ifconfig_gif1_descr="Tunnel-to-Vultr-Edge"

# Routing
ipv6_static_routes="myblock cloud vtbgp"
ipv6_route_myblock="2a06:9801:1c::/48 -reject"
ipv6_route_cloud="2a06:9801:1c:1000::/64 2a06:9801:1c:ffff::2"
ipv6_route_vtbgp="2a06:9801:1c:fffe::2/128 -interface gif1"
ipv6_gateway_enable="YES"

The iBGP link (gif1) uses addresses from our own /48 (2a06:9801:1c:fffe::/64). This is convenient - those addresses are already routable within our infrastructure, so the iBGP session doesn’t depend on any provider-assigned addresses being reachable.

FRR Configuration

The FRR configuration grows to four BGP sessions. The substantive additions over the original are local-preference assignment on inbound route-maps, and the iBGP session to vtbgp:

frr version 10.5.1
frr defaults traditional
hostname hobgp
log syslog informational
service integrated-vtysh-config
!
ipv6 prefix-list PL-MY-NET seq 5 permit 2a06:9801:1c::/48
!
! [PL-BOGONS: same comprehensive bogon filter as original - see previous article]
!
route-map RM-IFOG-BGPTUNNEL-IN permit 10
 match ipv6 address prefix-list PL-BOGONS
 set local-preference 150
exit
!
route-map RM-IFOG-BGPTUNNEL-OUT permit 10
 match ipv6 address prefix-list PL-MY-NET
exit
!
route-map RM-LAGRANGE-IN permit 10
 match ipv6 address prefix-list PL-BOGONS
exit
!
route-map RM-LAGRANGE-OUT permit 10
 match ipv6 address prefix-list PL-MY-NET
 set as-path prepend 201379 201379
exit
!
route-map RM-ROUTE64-IN permit 10
 match ipv6 address prefix-list PL-BOGONS
exit
!
route-map RM-ROUTE64-OUT permit 10
 match ipv6 address prefix-list PL-MY-NET
exit
!
route-map RM-IBGP-IN permit 10
 set local-preference 200
exit
!
route-map RM-IBGP-OUT permit 10
 match ipv6 address prefix-list PL-MY-NET
exit
!
ipv6 route 2a06:9801:1c::/48 blackhole
ipv6 route 2a06:9801:1c:fffe::2/128 gif1
!
router bgp 201379
 bgp router-id 198.51.100.10
 no bgp default ipv4-unicast
 neighbor 2001:db8:300::1 remote-as 209533
 neighbor 2001:db8:300::1 description Upstream-iFog-Frankfurt
 neighbor 2001:db8:300::1 update-source 2001:db8:300::2
 neighbor fd00:ca::1 remote-as 209735
 neighbor fd00:ca::1 description Upstream-Lagrange-UK
 neighbor fd00:ca::1 update-source fd00:ca::2
 neighbor 2001:db8:400::1 remote-as 212895
 neighbor 2001:db8:400::1 description Upstream-Route64-FRA
 neighbor 2001:db8:400::1 update-source 2001:db8:400::2
 neighbor 2a06:9801:1c:fffe::2 remote-as 201379
 neighbor 2a06:9801:1c:fffe::2 description Edge-Vultr-Frankfurt
 neighbor 2a06:9801:1c:fffe::2 update-source 2a06:9801:1c:fffe::1
 !
 address-family ipv6 unicast
  network 2a06:9801:1c::/48
  neighbor 2001:db8:300::1 activate
  neighbor 2001:db8:300::1 soft-reconfiguration inbound
  neighbor 2001:db8:300::1 maximum-prefix 250000 90 restart 30
  neighbor 2001:db8:300::1 route-map RM-IFOG-BGPTUNNEL-IN in
  neighbor 2001:db8:300::1 route-map RM-IFOG-BGPTUNNEL-OUT out
  neighbor fd00:ca::1 activate
  neighbor fd00:ca::1 soft-reconfiguration inbound
  neighbor fd00:ca::1 maximum-prefix 250000 90 restart 30
  neighbor fd00:ca::1 route-map RM-LAGRANGE-IN in
  neighbor fd00:ca::1 route-map RM-LAGRANGE-OUT out
  neighbor 2001:db8:400::1 activate
  neighbor 2001:db8:400::1 soft-reconfiguration inbound
  neighbor 2001:db8:400::1 maximum-prefix 250000 90 restart 30
  neighbor 2001:db8:400::1 route-map RM-ROUTE64-IN in
  neighbor 2001:db8:400::1 route-map RM-ROUTE64-OUT out
  neighbor 2a06:9801:1c:fffe::2 activate
  neighbor 2a06:9801:1c:fffe::2 soft-reconfiguration inbound
  neighbor 2a06:9801:1c:fffe::2 route-map RM-IBGP-IN in
  neighbor 2a06:9801:1c:fffe::2 route-map RM-IBGP-OUT out
 exit-address-family
exit

Local Preference: The Outbound Traffic Hierarchy

Local-preference (LP) is the primary mechanism for outbound path selection within an AS. BGP evaluates LP before AS-path length, making it the dominant decision factor. The hierarchy here is:

Neighbor	Local Preference	Role
vtbgp (iBGP/Vultr)	200	Preferred exit
iFog BGPTunnel	150	Secondary exit
Lagrange	100 (default)	Tertiary
route64	100 (default)	Tertiary

With LP 200 on iBGP-learned routes, hobgp prefers to send outbound traffic through vtbgp and into Vultr’s network wherever vtbgp has a route. iFog serves as backup when vtbgp lacks a route or the tunnel is down. Lagrange and route64 act as last resort, carrying traffic to parts of the internet not well-served by the other two.

For inbound traffic - what the rest of the internet chooses - the tool is different: you control what you announce and how attractive you make it. RM-LAGRANGE-OUT adds two extra prepends (set as-path prepend 201379 201379), making our prefix appear three AS-hops away via Lagrange. Most networks see a shorter path via iFog or Vultr and prefer those, demoting Lagrange to a true backup.

A Note on Free Transit

Both iFog’s BGPTunnel.com service and route64.org offer free BGP transit aimed at hobbyists and researchers. BGPTunnel provides a GRE tunnel to iFog’s Frankfurt infrastructure and announces your prefix to their peers. route64 works similarly. Neither carries SLAs suitable for production, but for learning and redundancy in a hobby setup they’re genuinely useful. The community around these services is active and documentation is good.

The Edge Router: vtbgp

The second router runs on Vultr in Frankfurt. Its role is narrow: announce 2a06:9801:1c::/48 into Vultr’s network and pass routing information back to hobgp via iBGP. It does not forward arbitrary internet traffic.

Why Vultr?

Vultr offers a BGP service where customers announce their own prefixes through Vultr’s infrastructure. Vultr’s network (AS20473 for customer connectivity, AS64515 for BGP peering sessions) has good upstream diversity and significant peering at major exchanges. Networks that peer with or transit through Vultr have a direct path to a locally-announced prefix, resulting in measurably shorter routes than if they had to reach across to an independent transit provider.

Network Configuration

hostname="vtbgp"
kern_securelevel_enable="YES"
kern_securelevel="2"
kld_list="if_gif tcpmd5"

# Physical interface (Vultr-assigned)
ifconfig_vtnet0="inet 203.0.113.20/23 -rxcsum -tso"
ifconfig_vtnet0_ipv6="inet6 2001:db8:f480::20/64 -rxcsum6 -tso6 accept_rtadv"
defaultrouter="203.0.113.1"

# GIF tunnel back to core (hobgp)
cloned_interfaces="gif0"
ifconfig_gif0="tunnel 203.0.113.20 198.51.100.10 mtu 1480"
ifconfig_gif0_ipv6="inet6 2a06:9801:1c:fffe::2 2a06:9801:1c:fffe::1 prefixlen 128"
ifconfig_gif0_descr="Uplink-to-hobgp"

# Routing
ipv6_gateway_enable="YES"
ipv6_static_routes="myblock vultrbgp"
ipv6_route_myblock="2a06:9801:1c::/48 -interface gif0"
ipv6_route_vultrbgp="2001:19f0:ffff::1 fe80::1%vtnet0"

# Services
pf_enable="YES"
frr_enable="YES"
sshd_enable="YES"
ipsec_enable="YES"
rtsold_enable="YES"
rtsold_flags="-aF"

Three details stand out:

ipv6_route_myblock points the entire /48 at gif0. Any traffic for our prefix arriving at vtbgp via Vultr’s infrastructure gets forwarded through the tunnel to hobgp, which handles actual routing to downstream servers.

ipv6_route_vultrbgp is a host route to Vultr’s BGP peering address (2001:19f0:ffff::1) via the link-local next-hop. Vultr’s peering address isn’t directly connected - it’s reachable via Vultr’s own routing - so this static route tells the kernel where to send packets destined for it. FRR uses ebgp-multihop 2 to allow the BGP session to traverse that extra hop.

rtsold handles SLAAC for vtbgp’s provider-assigned IPv6 on vtnet0, receiving the gateway via router advertisements.

A reader might wonder how vtbgp reaches the internet for its own traffic - software updates, DNS, the BGP session setup itself - given that it deliberately doesn’t install the 240,000 BGP routes into the kernel. The answer is the IPv4 defaultrouter="203.0.113.1" in rc.conf. vtbgp’s own outbound traffic uses a plain static default route provided by Vultr, completely independent of the BGP routing table. The BGP table exists solely for FRR’s own decision-making and for propagation to hobgp via iBGP.

TCP-MD5 Authentication

Vultr requires TCP-MD5 authentication on BGP sessions. This is decades-old, MD5 is not modern, and it’s standard across the industry - practically every carrier requires it for BGP sessions. It prevents trivial session disruption from third parties who don’t know the shared key.

On FreeBSD, TCP-MD5 is implemented via the IPsec subsystem. The security associations live in /etc/ipsec.conf:

add 2001:db8:f480::20 2001:19f0:ffff::1 tcp 0x1000 -A tcp-md5 "your-shared-secret";
add 2001:19f0:ffff::1 2001:db8:f480::20 tcp 0x1000 -A tcp-md5 "your-shared-secret";

The SPI value (0x1000) is fixed and matches what FRR expects. With ipsec_enable="YES" in rc.conf and the IPsec SAs loaded, FRR picks up the authentication automatically when password is set on the neighbor. The shared secret comes from Vultr’s BGP configuration panel.

FRR Configuration

frr version 10.5.1
frr defaults traditional
hostname vtbgp
log syslog informational
service integrated-vtysh-config
!
ipv6 prefix-list PL-MY-NET seq 5 permit 2a06:9801:1c::/48
! [PL-BOGONS: same bogon filter as hobgp]
!
route-map RM-VULTR-OUT permit 10
 match ipv6 address prefix-list PL-MY-NET
exit
!
route-map RM-VULTR-IN permit 10
 match ipv6 address prefix-list PL-BOGONS
exit
!
route-map RM-IBGP-OUT permit 10
exit
!
route-map RM-KERNEL-DENY deny 10
exit
!
router bgp 201379
 bgp router-id 203.0.113.20
 no bgp default ipv4-unicast
 neighbor 2001:19f0:ffff::1 remote-as 64515
 neighbor 2001:19f0:ffff::1 description Vultr-Core
 neighbor 2001:19f0:ffff::1 password your-shared-secret
 neighbor 2001:19f0:ffff::1 ebgp-multihop 2
 neighbor 2a06:9801:1c:fffe::1 remote-as 201379
 neighbor 2a06:9801:1c:fffe::1 description Core-hobgp
 neighbor 2a06:9801:1c:fffe::1 update-source 2a06:9801:1c:fffe::2
 !
 address-family ipv6 unicast
  neighbor 2001:19f0:ffff::1 activate
  neighbor 2001:19f0:ffff::1 soft-reconfiguration inbound
  neighbor 2001:19f0:ffff::1 route-map RM-VULTR-IN in
  neighbor 2001:19f0:ffff::1 route-map RM-VULTR-OUT out
  neighbor 2a06:9801:1c:fffe::1 activate
  neighbor 2a06:9801:1c:fffe::1 next-hop-self
  neighbor 2a06:9801:1c:fffe::1 route-map RM-IBGP-OUT out
 exit-address-family
exit
!
ipv6 protocol bgp route-map RM-KERNEL-DENY

Two things here deserve careful explanation.

`RM-KERNEL-DENY`: Keep the Routing Table Clean

ipv6 protocol bgp route-map RM-KERNEL-DENY applies a deny-all route-map to every BGP route before FRR tries to install it into the kernel’s forwarding table. The result: FRR maintains its full BGP table internally (~240,000 routes), but none of those routes appear in the kernel.

This is intentional. vtbgp’s job is to announce our prefix, not to forward arbitrary internet traffic. The only forwarding it does is for traffic arriving destined for 2a06:9801:1c::/48, which is handled by the static route pointing at gif0. Pushing 240,000 routes into the kernel would be wasteful memory consumption with no benefit.

FRR still uses its internal table for its own purposes and still sends routes to hobgp via iBGP - they just never touch the kernel’s FIB on vtbgp.

A note for readers familiar with FRR internals: the standard layer for controlling kernel route installation is Zebra, FRR’s routing daemon that mediates between the protocol daemons (BGP, OSPF, etc.) and the OS. An alternative to RM-KERNEL-DENY is configuring Zebra directly to not install BGP routes into the kernel. Both approaches achieve the same result; RM-KERNEL-DENY intercepts routes at the BGP level before they reach Zebra at all. At this scale the distinction is academic, but Zebra is the canonical control point for RIB-to-FIB separation in production deployments.

`next-hop-self`: Making iBGP Routes Usable

When vtbgp receives a route from Vultr (next-hop: 2001:19f0:ffff::1) and re-advertises it to hobgp via iBGP, the standard iBGP behavior is to preserve the original next-hop. hobgp can’t reach 2001:19f0:ffff::1 directly - that’s inside Vultr’s network, reachable from vtbgp but not from hobgp.

next-hop-self rewrites the next-hop to vtbgp’s own address (2a06:9801:1c:fffe::2) before sending routes to hobgp. hobgp reaches that address via gif1, so the routes become actionable: “to get to this destination, send traffic down the iBGP tunnel to vtbgp.” Without next-hop-self, hobgp would receive routes with unreachable next-hops and ignore them.

iBGP: Connecting the Two Routers

iBGP (internal BGP) runs between routers in the same AS. It has different semantics from eBGP in two important ways:

First, the split-horizon rule: routes learned via iBGP are not re-advertised to other iBGP peers. For a two-router setup this doesn’t matter - each router hears from the other directly. For larger networks with three or more routers, you’d need route reflection or a full mesh; with two, a single session is a complete solution.

Second, local-preference propagation: LP is carried across iBGP sessions. Routes that vtbgp receives from Vultr carry LP 100 (default) until hobgp’s RM-IBGP-IN overrides them to 200. This means hobgp’s outbound preference for the Vultr path is configured entirely on hobgp - vtbgp doesn’t need to know anything about the preference hierarchy.

The session itself is mechanically simple. On hobgp, vtbgp is a neighbor with remote-as 201379 (our own AS number). FRR recognizes same-AS neighbors as iBGP peers automatically. The session runs over the gif1/gif0 tunnel pair, authenticated by virtue of the tunnel endpoints being within our own address space.

PF on a Transit Router: The Stateless Requirement

Operating a transit router exposes a fundamental tension in stateful packet filtering. On a server, stateful rules are ideal: track TCP connections, match replies to established state, drop unsolicited packets. On a transit router, stateful rules for through-traffic break things.

The reason is asymmetric routing. When hobgp receives a packet on gre0 (via iFog) destined for a host behind gif0, PF creates a state entry associating that flow with gre0. The destination host responds, the packet traverses gif0 back to hobgp, and the best exit path might be gre1 (Lagrange) - different from the arrival interface. PF sees the reply on gre1 with no matching state on that interface and drops it.

This isn’t an edge case. Path asymmetry is the norm on the public internet. A request and its reply routinely traverse different providers, different exchange points, different physical paths.

Request: Client ──> [iFog / gre0] ──> hobgp ──> [gif0] ──> Server
                                                               │
Reply:   Client <── [Lagrange / gre1] <── hobgp <── [gif0] <──┘
                    ^^^^^^^^^^^^^^^^^^^
                    PF drops it: state was created on gre0, reply arrives on gre1

The solution is to keep transit traffic stateless:

# Transit traffic: no state, no asymmetry problems
pass in quick inet6 from any to $my_network_v6 no state
pass out quick inet6 from any to $my_network_v6 no state
pass in quick inet6 from $my_network_v6 to any no state
pass out quick inet6 from $my_network_v6 to any no state

These four rules cover all IPv6 traffic to or from 2a06:9801:1c::/48. no state means PF allows the packets without creating any tracking entry. Each packet is evaluated independently on its own merits.

But not everything can be stateless. The router itself makes outbound connections - BGP keepalives, DNS queries, software updates - and those need state tracking to match replies. The control plane (SSH, BGP sessions) also needs state. The solution is a three-tier PF structure where ordering determines which rule wins:

# Tier 1: Control plane - stateful, locked to known peers
pass in quick proto tcp from <bgp_peers> to any port 179 keep state
pass out quick proto tcp from any to <bgp_peers> port 179 keep state

# Tier 2: Router's own traffic - stateful for outbound connectivity
pass in  quick inet6 from any to $my_router_ip keep state
pass out quick inet6 from $my_router_ip to any keep state

# Tier 3: Transit traffic - stateless for asymmetric routing
pass in quick inet6 from any to $my_network_v6 no state
pass out quick inet6 from any to $my_network_v6 no state
pass in quick inet6 from $my_network_v6 to any no state
pass out quick inet6 from $my_network_v6 to any no state

PF evaluates quick rules in order and stops at the first match. Traffic to $my_router_ip (2a06:9801:1c::1, the router’s address within our prefix) matches Tier 2 and gets state tracking. Everything else in the /48 falls through to Tier 3 and flows stateless.

NAT for Router-Originated Traffic

There’s a related problem: hobgp’s physical interface has a Hetzner-assigned IPv6 address (2001:db8:1c19::1). When the router sends traffic out a GRE tunnel using that address as the source, it’s sending from an address with no relation to our announced prefix. Transit providers may find this confusing; more practically, return traffic to that address may take a completely different path or simply fail.

The fix is NAT on the GRE tunnel interfaces, translating non-/48 source addresses to the router’s PI address:

nat on gre0 inet6 from ! $my_network_v6 to ! $peer_ifog  -> $my_router_ip
nat on gre1 inet6 from ! $my_network_v6 to ! $peer_lagrange -> $my_router_ip
nat on gre2 inet6 from ! $my_network_v6 to ! $peer_route64  -> $my_router_ip
nat on gif1 inet6 from ! $my_network_v6 to any             -> $my_router_ip

Reading the first rule: packets exiting gre0, with a source address not in our /48, destined for anywhere other than the iFog BGP peer itself, get their source rewritten to $my_router_ip (2a06:9801:1c::1). The peer exclusion is critical - BGP sessions must use the tunnel link address, not the NATed address, for the session to work. Everything else the router originates - traceroutes, pings, package fetches - appears on the internet as 2a06:9801:1c::1, an address inside our announced prefix.

Verification

vtysh -c 'show bgp ipv6 summary' on hobgp shows four sessions in established state:

Neighbor             V    AS   MsgRcvd   MsgSent  Up/Down   State/PfxRcd  Desc
2001:db8:300::1      4  209533  1873391    10601  16:21:35       237902    Upstream-iFog-FRA
fd00:ca::1           4  209735  1267858     1771  1d05h26m       239352    Upstream-Lagrange-UK
2001:db8:400::1      4  212895   609199     1771  1d05h26m       238542    Upstream-Route64-FRA
2a06:9801:1c:fffe::2 4  201379  1370565     1771  1d05h26m       229257    Edge-Vultr-Frankfurt

Four peers, three carrying the full DFZ table (~237-239K prefixes each), one iBGP peer with a partial Vultr view (~229K prefixes). All in established state.

The real test is whether different networks actually use different paths. MTR traces from three vantage points answer this directly.

From Deutsche Telekom (heavy Vultr peering):

HOST: client.example.com                                    Loss%  Snt  Last   Avg
  ...
  9.|-- constantcompany-ic-375791.ip.twelve99-cust.net       0.0%   10  25.9  26.7
 10.|-- ethernetae1-sr2.fkt3.constant.com                    0.0%   10  23.4  29.6
 11.|-- 2001:19f0:6c00:154::33                               0.0%   10  24.1  23.7
 12.|-- vtbgp.example.com                                    0.0%   10  26.8  26.3
 13.|-- hobgp.example.com                                    0.0%   10  27.1  26.6
 14.|-- radon.example.com                                    0.0%   10  27.9  31.2
 15.|-- 2a06:9801:1c:1000::10                                0.0%   10  27.1  30.4

Hops 9-11 are Constant Company infrastructure - that’s Vultr’s upstream. The traffic enters via vtbgp (hop 12), crosses the iBGP tunnel to hobgp (hop 13), and reaches the downstream server. The Vultr announcement is doing exactly what it should for traffic from DTAG-connected networks.

From Netcup (via AORTA/Vienna Internet Exchange):

HOST: server.example.com                                    Loss%  Snt  Last   Avg
  ...
  9.|-- 2001:7f8:15e::64                                     0.0%   10  16.7  16.7
 10.|-- 2a0d:5440:1::f                                       0.0%   10  15.8  15.9
 11.|-- hobgp.example.com                                    0.0%   10  19.9  19.6
 12.|-- radon.example.com                                    0.0%   10  20.7  21.4
 13.|-- 2a06:9801:1c:1000::10                                0.0%   10  20.8  22.0

Hop 9 (2001:7f8:15e::64) is an internet exchange peering LAN address (2001:7f8::/29 is IANA-allocated IXP space). Hop 10 (2a0d:5440:1::f) belongs to route64 (AS212895). The traffic entered via route64 at an exchange point and arrived directly at hobgp - no vtbgp involved.

From another Hetzner server (Falkenstein):

HOST: server3.example.com                                   Loss%  Snt  Last   Avg
  ...
  8.|-- 2001:7f8:11c:1:0:20:9533:1                          0.0%   10   5.5   5.6
  9.|-- hobgp.example.com                                    0.0%   10   9.5   9.6
 10.|-- radon.example.com                                    0.0%   10  10.3  16.4
 11.|-- 2a06:9801:1c:1000::10                                0.0%   10  10.5  11.3

Hop 8 is a DE-CIX Frankfurt switching address (the 2001:7f8:11c range belongs to DE-CIX). Traffic exits DE-CIX and hits hobgp in single-digit milliseconds - iFog’s Frankfurt presence at the same exchange. The prefix is reachable from Hetzner in under 10ms.

Three different paths, three different entry points. Each path is geometrically shorter for its respective traffic class than routing everything through a single upstream would be.

Meta: The destination address in every trace above - 2a06:9801:1c:1000::10 - is this blog. If you’re reading this article over IPv6, you just arrived here via one of those paths.

What This Costs

Running your own IPv6 AS is surprisingly inexpensive. Here are the actual numbers for this setup, with monthly equivalents in euros.

Item	Actual cost	Monthly (approx. €)
ASN registration (one-time)	£15	–
ASN annual fee	£54.99/year	€5.50
PA /48 IPv6 block	£7/year	€0.70
Hetzner CX23 (core router, hobgp)	€4.15/month	€4.15
Vultr vc2-1c-1gb (edge router, vtbgp)	$6.00/month	€5.50
Lagrange Cloud transit (100 Mbps)	£2/month	€2.40
iFog BGPTunnel.com (100 Mbps)	Free (non-commercial)	€0
route64.org (200 Mbps)	Free	€0
Vultr BGP	Included in VPS	€0
Total recurring		≈ €18/month

That is less than many people pay for a single mid-range VPS - and it buys a redundant multi-homed autonomous system with 1.4 Gbps of available transit capacity.

The reason IPv6-only is so accessible comes down to address space economics. An IPv4 PA block costs thousands of euros per year through a sponsoring LIR and requires documented justification. An IPv6 /48 costs €8. The transit situation is similarly skewed: iFog and route64 offer free GRE-based transit specifically because IPv6-only hobbyists place negligible load on their networks. The free tier is a real service, not a trial.

The Vultr edge router is the largest recurring cost after the ASN fees, and it’s paying for something specific: native BGP peering with Vultr’s infrastructure, which reaches networks that simply don’t have a short path to any other upstream in this setup. If path diversity from Vultr-connected networks isn’t a priority, the edge router is optional and the setup collapses back to a single core router at under €5/month in compute costs.

Lessons Learned

Stateless transit rules are not optional. Getting this wrong produces subtle failures: ICMP works but TCP stalls, small packets flow freely but large ones don’t, connections from one direction succeed while the other direction silently drops. When troubleshooting transit routing on PF, the first question is always whether state tracking is interfering with asymmetric paths.

iBGP between two routers is straightforward. The iBGP full-mesh requirement sounds complicated, but with exactly two routers it’s trivially satisfied by one session on each side. The main configuration concern is next-hop-self, which is required whenever the iBGP peers can’t directly reach the eBGP next-hops learned from upstream.

RM-KERNEL-DENY is useful for announcement-only nodes. A router whose job is to announce a prefix and forward traffic for that prefix back through a tunnel doesn’t need 240,000 kernel routes. Keeping the kernel’s RIB clean saves memory, avoids confusing interactions with static routes, and makes the router’s actual forwarding intent explicit.

Vultr’s BGP service reaches a meaningfully different part of the internet. The DTAG traceroute shows traffic entering via Vultr’s Constant Company upstream - a path that simply doesn’t exist via iFog or route64. The two sets of upstreams complement each other. For traffic where both would work, local-preference makes the decision; for traffic where only one path exists, the choice is made for you.

Free transit providers provide real diversity. BGPTunnel.com, route64.org, and similar services offer transit that’s genuinely usable for learning and redundancy. Three upstreams with different Frankfurt exchange presence means real path diversity without transit fees.

Conclusion

The gap between a single-router BGP setup and a distributed multi-homed network is mostly conceptual, not computational. The same FreeBSD tools - FRR, PF, GIF/GRE tunnels - handle the expanded setup. iBGP adds one session. The third upstream adds another GRE tunnel and two route-maps. The edge router requires TCP-MD5 and a careful decision to keep routes out of the kernel.

What changes is what the setup can do. Three entry points mean inbound traffic takes geometrically shorter paths for a larger portion of the internet. Local-preference makes outbound path selection explicit and tunable. The iBGP architecture means either router can be rebooted without taking down the other’s upstream sessions.

The next logical evolution would be geographic diversity - a PoP in a different region with genuinely independent upstream connectivity. But Frankfurt with three providers and two PoPs is already a substantial improvement over a single machine with one vendor’s connectivity, and the lessons from building it transfer directly to any larger scale.

Each of those traceroutes ends the same way: hop 11 or hop 13 or hop 15, an address in 2a06:9801:1c::/48. The path that gets it there varies by two thousand kilometres and half a dozen autonomous systems, chosen automatically by the routing protocol. That’s what multi-homing looks like when it’s working.

References

HTTP/3 on FreeBSD: Getting QUIC Working with nginx in a Bastille Jail

2026-02-21T00:00:00+01:00

What looks like a simple nginx config change turns out to involve SSL library compatibility, firewall rules for a new protocol, and a subtle multi-process routing problem that only surfaces under real traffic. This documents what it actually took to get HTTP/3 (QUIC) working on nginx 1.28 inside a FreeBSD 15.0 Bastille jail - serving the Mastodon instance at burningboard.net.

If you’re running nginx on FreeBSD and want HTTP/3, this should save you several hours of troubleshooting.

Prerequisites

FreeBSD 12+ (this guide uses FreeBSD 15.0-RELEASE)
nginx 1.25+ with HTTPV3=on and HTTPV3_BORING=on (built from ports)
A valid TLS certificate (Let’s Encrypt or similar)
UDP port 443 open in your firewall
DNS A and AAAA records pointing to your server

Step 1: The Listen Directives

The most common first mistake: replacing listen 443 ssl with listen 443 quic. QUIC doesn’t replace TCP - it runs alongside it. Browsers always connect via regular HTTPS (TCP) first, then upgrade to QUIC/HTTP3 only after seeing an Alt-Svc header announcing that HTTP/3 is available.

Wrong:

server {
    listen 443 quic;
    listen [::]:443 quic;
    http2 on;
}

This makes nginx listen only on UDP. No browser can connect at all because the initial connection is always TCP.

Correct:

server {
    listen 443 ssl;
    listen 443 quic;
    listen [::]:443 ssl;
    listen [::]:443 quic;
    http2 on;
}

Both TCP (ssl) and UDP (quic) listeners must be present in the same server block.

The reason TCP can never be skipped is that the HTTP/3 upgrade is self-announcing: the server tells the browser about QUIC support inside an HTTP response, which first requires a working TCP connection. Here is the full sequence:

 Client                              nginx (:443)
   │                                     │
   │   ① TCP + TLS handshake             │
   ├────────────────────────────────────>┤  \
   │<────────────────────────────────────┤   TCP (always required for first contact)
   │                                     │  /
   │   ② First request - still over TCP  │
   ├─── GET / ──────────────────────────>┤  \
   │<── 200 OK ──────────────────────────┤   TCP
   │    Alt-Svc: h3=":443"; ma=86400     │  /  ← browser learns HTTP/3 is available
   │                                     │
   │   (browser caches Alt-Svc)          │
   │                                     │
   │   ③ Subsequent requests via QUIC    │
   ├═══ QUIC handshake (UDP) ═══════════>┤  \
   │<════════════════════════════════════┤   UDP (HTTP/3)
   ├═══ GET /page ═══════════════════════>┤   lower latency, multiplexed
   │<═══ 200 OK ══════════════════════════┤  /

This is why both the ssl and quic listen directives are always required together: one to serve the first response that announces HTTP/3, the other to handle all subsequent QUIC connections.

Step 2: The Alt-Svc Header

Browsers discover HTTP/3 support through the Alt-Svc response header. Without it, no browser will ever attempt a QUIC connection:

add_header Alt-Svc 'h3=":443"; ma=86400' always;

The always keyword ensures the header is sent even on error responses and redirects, not just successful 200s.

The `add_header` Inheritance Trap

This is a subtle but critical nginx behavior: add_header directives inside a location block completely override all add_header directives from the parent server block. If you set Alt-Svc at the server level but have any location block with its own add_header (for Cache-Control, Access-Control-Allow-Origin, etc.), the Alt-Svc header silently disappears for those locations.

The fix: add Alt-Svc to every location block that already has any add_header directive.

location /assets {
    add_header Cache-Control public;
    add_header 'Access-Control-Allow-Origin' '*';
    add_header X-Cache-Status $upstream_cache_status;
    add_header Alt-Svc 'h3=":443"; ma=86400' always;  # Must be here too!
}

Verify coverage with curl:

# Check the root path
curl -sI https://yourdomain.com/ | grep -i alt-svc

# Check a subpath - if this is empty, you have the inheritance problem
curl -sI https://yourdomain.com/some/path | grep -i alt-svc

Step 3: The SSL Library Problem

With a correct config and firewall in place, browsers may still refuse HTTP/3. Capturing UDP traffic on port 443 inside the nginx jail reveals the problem:

tcpdump -i vnet0 udp port 443 -c 20

With stock OpenSSL, the server responds with tiny 51-byte packets to every QUIC connection attempt. These are QUIC version negotiation failures - the handshake is breaking at the HTTP/3 framing layer, not the TLS layer.

The Misleading `openssl s_client` Test

A common false positive when diagnosing this:

echo | openssl s_client -connect yourdomain.com:443 -quic -alpn h3
# Shows: CONNECTED

This only validates TLS-level QUIC support. The actual HTTP/3 protocol layer on top can still be broken - and with stock OpenSSL on FreeBSD, it is.

Why Stock OpenSSL Doesn’t Work

nginx’s HTTP/3 implementation was originally built against BoringSSL. While OpenSSL 3.2+ added QUIC API support, there are known compatibility issues at the HTTP/3 framing layer. The TLS handshake may succeed, but actual HTTP/3 data transfer fails.

Rebuilding nginx with BoringSSL

The FreeBSD nginx port offers several SSL backends for HTTP/3. Check the available options:

make -C /usr/ports/www/nginx showconfig | grep HTTPV3

In nginx 1.28.2 the relevant options are:

HTTPV3=on: Enable HTTP/3 protocol support
HTTPV3_BORING=on: Use security/boringssl
HTTPV3_LSSL=off: Use security/libressl-devel
HTTPV3_QTLS=off: Use security/openssl33-quictls

Enable HTTPV3 and HTTPV3_BORING, then rebuild:

cd /usr/ports/www/nginx
make config          # Enable HTTPV3 and HTTPV3_BORING
pkg unlock nginx     # If the package is locked
make clean
make deinstall
make && make install
pkg lock nginx       # Re-lock if desired
service nginx restart

Verify the new binary links against BoringSSL:

nginx -V 2>&1 | head -3

nginx version: nginx/1.28.2
built with OpenSSL 1.1.1 (compatible; BoringSSL) (running with BoringSSL)
TLS SNI support enabled

After rebuilding, tcpdump shows proper-sized QUIC handshake packets (1200, 892, 223 bytes) instead of the 51-byte rejections.

Step 4: The Worker Process Affinity Problem

With BoringSSL in place, the first HTTP/3 request succeeds - but subsequent requests on the same connection fail with ERR_QUIC_PROTOCOL_ERROR_QUIC_PUBLIC_RESET in Chrome.

Setting worker_processes 1 in nginx.conf makes HTTP/3 work perfectly. This confirms the diagnosis: QUIC packets from the same connection are being routed to different worker processes. A worker that didn’t establish the connection responds with a “public reset” packet, killing it.

The contrast between the broken and fixed behaviour:

WITHOUT reuseport - one shared UDP socket, OS picks any worker per packet:

  pkt 1 (conn A) ──┐
  pkt 2 (conn A) ──┼──> [ :443 UDP ] ──> OS ──> Worker 1  ✓ (owns conn A)
  pkt 3 (conn A) ──┘                       └──> Worker 2  ✗ (no context)
                                                     │
                                               sends RST → connection killed


WITH reuseport + quic_retry - one socket per worker, OS hashes by connection:

  pkt 1 (conn A) ──┐                    ┌──> Worker 1 socket ──> Worker 1  ✓
  pkt 2 (conn A) ──┼──> SO_REUSEPORT_LB ┤
  pkt 3 (conn A) ──┘  (hashes 4-tuple)  ├──> Worker 2 socket ──> Worker 2
                                         └──> Worker 3 socket ──> Worker 3

  All packets for the same connection always reach the same worker.
  quic_retry ensures the correct worker is pinned from the very first packet.

The Linux Solution (Not Available on FreeBSD)

On Linux, nginx solves this with quic_bpf on, which uses eBPF to route QUIC packets to the correct worker based on the connection ID. FreeBSD doesn’t have eBPF - this option doesn’t exist.

The FreeBSD Solution: `reuseport` + `quic_retry`

Two directives work together to solve this on FreeBSD:

reuseport on the QUIC listeners creates per-worker listening sockets. FreeBSD 12+ supports SO_REUSEPORT_LB, which load-balances incoming UDP packets across workers while keeping each connection pinned to one worker socket. Add it to one server block only - typically the primary vhost - to avoid duplicate binding errors:

# Primary server block only:
listen 443 ssl;
listen 443 quic reuseport;
listen [::]:443 ssl;
listen [::]:443 quic reuseport;

# All other server blocks:
listen 443 ssl;
listen 443 quic;
listen [::]:443 ssl;
listen [::]:443 quic;

quic_retry on forces a QUIC retry handshake (address validation) before establishing the connection. This extra round-trip ensures the correct worker process handles the entire QUIC session. Add it to the http block in nginx.conf:

http {
    quic_retry on;
    # ...
}

Verify reuseport is active by checking for multiple UDP sockets:

sockstat -4 -6 -l | grep nginx | grep 443

You should see multiple UDP entries on port 443 - one per worker process.

Step 5: Firewall Configuration

QUIC runs over UDP, so you need explicit UDP rules for port 443 alongside the existing TCP rules. The exact configuration depends on your network topology.

In this setup, nginx runs inside a Bastille jail. IPv6 is directly routed - the jail holds a public IPv6 address, so no RDR or NAT is needed for IPv6 QUIC. For IPv4, the host’s pf forwards traffic from the public IP to the jail’s private address using rdr.

The important detail: QUIC (UDP) needs its own rdr rule on the host, separate from the TCP one. A pass rule alone is not enough - without rdr, UDP packets arriving at the public IP are still addressed to the public IP when the filter rules run, so a pass ... to $frontend_v4 (jail IP) rule will never match them.

# --- RDR ---
# Redirect TCP 80/443 to the nginx jail
rdr on $ext_if inet proto tcp to ($ext_if) port {80,443} -> $frontend_v4

# Redirect UDP 443 (QUIC) to the nginx jail - separate rule required
rdr on $ext_if inet proto udp to ($ext_if) port 443 -> $frontend_v4

# --- Pass rules ---
# HTTP/HTTPS (TCP)
pass in quick on $ext_if inet  proto tcp from any to $frontend_v4 port {80,443} flags S/SA keep state
pass in quick on $ext_if inet6 proto tcp from any to $frontend_v6 port {80,443} flags S/SA keep state

# QUIC (UDP) - IPv4 via NAT/rdr above, IPv6 direct
pass in quick on $ext_if inet  proto udp from any to $frontend_v4 port 443 keep state
pass in quick on $ext_if inet6 proto udp from any to $frontend_v6 port 443 keep state

Don’t forget both IPv4 and IPv6 if your server is dual-stack. For a server without a Bastille jail - nginx running directly on the host - the rdr rules are not needed; only the pass rules for both TCP and UDP.

Step 6: Performance Tuning

TLS 1.3 Early Data (0-RTT)

ssl_early_data on;

Speeds up reconnections by allowing data in the first TLS flight. Early data is vulnerable to replay attacks, so pass the status to your backend:

proxy_set_header Early-Data $ssl_early_data;

Add this to all proxy_pass locations so your application can reject replayed requests on sensitive endpoints (POST, DELETE, etc.).

Session Tickets

ssl_session_tickets on;

Required for QUIC 0-RTT resumption. Unlike the common advice to disable session tickets for TLS 1.2 security, they’re important for QUIC performance.

Complete Example Configuration

nginx.conf

worker_processes auto;
error_log /var/log/nginx/error.log;

events {
    worker_connections 1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    sendfile        on;
    tcp_nopush      on;
    tcp_nodelay     on;
    keepalive_timeout 65;

    # QUIC settings
    quic_retry on;
    ssl_early_data on;

    # SSL global settings
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;
    ssl_session_cache shared:SSL:10m;
    ssl_session_tickets on;

    # gzip compression
    gzip on;
    gzip_vary on;
    gzip_proxied any;
    gzip_comp_level 6;
    gzip_types text/plain text/css application/json application/javascript
               text/xml application/xml application/xml+rss text/javascript
               image/svg+xml image/x-icon;

    include /usr/local/etc/nginx/vhosts/*.conf;
}

Primary vhost (with `reuseport`)

server {
    listen 443 ssl;
    listen 443 quic reuseport;
    listen [::]:443 ssl;
    listen [::]:443 quic reuseport;
    http2 on;

    server_name yourdomain.com;

    ssl_certificate     /path/to/fullchain.pem;
    ssl_certificate_key /path/to/privkey.pem;

    add_header Alt-Svc 'h3=":443"; ma=86400' always;

    location / {
        add_header Alt-Svc 'h3=":443"; ma=86400' always;
        # ... your config ...
    }

    location @proxy {
        add_header Alt-Svc 'h3=":443"; ma=86400' always;
        proxy_set_header Early-Data $ssl_early_data;
        # ... your proxy config ...
    }
}

Additional vhosts (without `reuseport`)

server {
    listen 443 ssl;
    listen 443 quic;
    listen [::]:443 ssl;
    listen [::]:443 quic;
    http2 on;

    server_name other.yourdomain.com;

    ssl_certificate     /path/to/fullchain.pem;
    ssl_certificate_key /path/to/privkey.pem;

    add_header Alt-Svc 'h3=":443"; ma=86400' always;

    location / {
        add_header Alt-Svc 'h3=":443"; ma=86400' always;
        # ... your config ...
    }
}

Testing and Verification

Server-side

# Verify QUIC TLS handshake (note: only tests TLS layer, not full HTTP/3 framing)
echo | openssl s_client -connect yourdomain.com:443 -quic -alpn h3

# Check Alt-Svc is present on all paths
curl -sI https://yourdomain.com/ | grep -i alt-svc
curl -sI https://yourdomain.com/some/path | grep -i alt-svc

# Monitor live QUIC traffic (inside the nginx jail)
tcpdump -i vnet0 udp port 443 -c 20

# Verify reuseport sockets - expect multiple UDP entries on 443
sockstat -4 -6 -l | grep nginx | grep 443

Client-side

Online checker: http3check.net - verifies both Alt-Svc and QUIC connectivity
Browser DevTools: Network tab → right-click column headers → enable “Protocol”. Look for h3 (may require a second page load after Alt-Svc is cached)
Chrome QUIC internals: chrome://net-export/ captures a netlog, analyzable at netlog-viewer.appspot.com

Debugging Tips

If Chrome marks QUIC as is_broken: true, clear cached data via Settings → Clear browsing data → Cached images and files
Use an incognito window for clean tests without cached Alt-Svc state
Check nginx error logs: grep -i quic /var/log/nginx/error.log
For deep debugging, temporarily set error_log /var/log/nginx/quic-debug.log debug; - remove it again promptly, it fills up fast

Summary of Pitfalls

Problem	Symptom	Solution
Missing `listen 443 ssl`	Site completely down	Add both `ssl` and `quic` listeners
Missing `Alt-Svc` header	Browsers never try HTTP/3	Add `Alt-Svc` with `always` flag
`add_header` inheritance	`Alt-Svc` missing on some paths	Repeat `Alt-Svc` in every `location` with `add_header`
Stock OpenSSL	51-byte QUIC rejections, `openssl s_client` lies	Rebuild nginx with `HTTPV3_BORING=on`
Worker affinity	`ERR_QUIC_PROTOCOL_ERROR_QUIC_PUBLIC_RESET`	Use `reuseport` + `quic_retry on`
Missing UDP `rdr` in pf	IPv4 QUIC silently unreachable despite pass rules	Add `rdr` for UDP 443 alongside the TCP one
Firewall blocks UDP 443	QUIC unreachable from clients	Add UDP 443 `pass` rules for IPv4 and IPv6

Conclusion

Getting HTTP/3 working on FreeBSD with nginx requires more legwork than on Linux, mostly because of the SSL library situation and the absence of quic_bpf. The four things that actually matter:

BoringSSL - rebuild from ports with HTTPV3=on and HTTPV3_BORING=on
quic_retry on - ensures correct worker routing during session establishment
reuseport on QUIC listeners - per-worker UDP sockets for reliable packet distribution
Alt-Svc in every location block - the silent add_header inheritance gotcha will bite you

In a jail setup with IPv4 NAT, add a dedicated rdr rule for UDP 443 - a pass rule alone isn’t enough.

Once all of this is in place, HTTP/3 works reliably and delivers noticeably lower latency, especially for connection-heavy workloads like a Mastodon instance.

References

nginx HTTP/3 module documentation
BoringSSL
http3check.net - online HTTP/3 availability checker
Chrome netlog viewer
FreeBSD Handbook – Jails

Integrating LaCrosse Sensors into Home Assistant via JeeLink on FreeBSD

2026-02-19T00:00:00+01:00

Home Assistant runs great inside a bhyve VM on FreeBSD, but USB passthrough into bhyve is not straightforward. When I needed to integrate a ConBee II Zigbee dongle, the solution was to run zigbee2mqtt inside a FreeBSD jail - where USB devices can be exposed via devfs rules - and forward everything over MQTT. This same pattern turns out to work perfectly for other USB receivers, like the JeeLink with its legacy LaCrosse temperature and humidity sensors.

The JeeLink is a small USB stick with an RFM12B radio module running the LaCrosseITPlusReader sketch. It picks up transmissions from LaCrosse TX-series sensors on 868 MHz, decodes them, and outputs the raw data over a serial port. All we need is a small bridge script that reads the serial data and publishes it to MQTT.

The Stack

[ LaCrosse TX sensors ]
        |  868 MHz
        v
[ JeeLink USB (FT232R) ] --> /dev/cuaU1
        |
        v
[ FreeBSD Jail ]
  lacrosse2mqtt.py
        |  MQTT
        v
[ Mosquitto Broker ]
        |
        v
[ Home Assistant (bhyve VM) ]

The jail already exists for zigbee2mqtt, so adding a second serial device and a Python script is minimal additional overhead.

Identifying the Device

FreeBSD recognizes the JeeLink’s FTDI chip out of the box:

usbconfig list

ugen0.4: <FT232 Serial (UART) IC Future Technology Devices International, Ltd> at usbus0

The kernel attaches it via the uftdi driver, which maps to a /dev/cuaU* serial device. If you already have another USB serial device (like a Zigbee dongle on /dev/cuaU0), the JeeLink will appear as /dev/cuaU1.

Passing the Device into the Jail

In /etc/devfs.rules, extend the jail’s ruleset to include the new serial port:

[devfsrules_jail_z2m=10]
add include $devfsrules_hide_all
add include $devfsrules_unhide_login
add path 'cuaU0' unhide
add path 'cuaU1' unhide
add path 'ugen*' unhide

The jail configuration in /etc/jail.conf references this ruleset:

z2m {
    # ... other settings ...
    devfs_ruleset = 10;
    enforce_statfs = 1;
    allow.mount.devfs;
}

Restart the jail for the new device to appear inside it.

Verifying the Serial Connection

Inside the jail, use cu to confirm the JeeLink is transmitting and the baud rate is correct. The LaCrosseITPlusReader sketch runs at 57600 baud:

cu -l /dev/cuaU1 -s 57600

If everything is working, you’ll see a header line followed by sensor readings:

[LaCrosseITPlusReader.10.1s (RFM12B f:868300 r:17241)]
OK 9 8 1 4 22 106
OK 9 44 129 4 182 52

Exit cu with ~. and move on. If you see garbage, double-check the baud rate - this was by far the most common cause of confusion during my setup.

Decoding the Data Format

Each OK 9 line from the JeeLink represents a sensor reading:

OK 9 <ID> <flags> <temp_high> <temp_low> <humidity>

Temperature is a 16-bit value divided by 10, offset by 100:

temp = (temp_high * 256 + temp_low) / 10.0 - 100.0

For example, bytes 4 182 decode to (4 * 256 + 182) / 10.0 - 100.0 = 20.6°C.

Humidity is the raw byte value in percent. A value of 52 means 52% relative humidity.

Flags encode battery status: bit 7 (0x80) indicates a new battery was just inserted, bit 0 (0x01) signals a weak battery.

The Bridge Script

Install the dependencies inside the jail:

pkg install py311-serial py311-paho-mqtt

Then create /usr/local/bin/lacrosse2mqtt.py:

#!/usr/bin/env python3
import serial
import paho.mqtt.client as mqtt
import time
import logging

logging.basicConfig(level=logging.INFO, format='%(asctime)s %(message)s')

SERIAL_PORT = "/dev/cuaU1"
BAUD        = 57600
MQTT_HOST   = "127.0.0.1"
MQTT_PORT   = 1883
MQTT_USER   = "your_mqtt_user"
MQTT_PASS   = "your_mqtt_password"
SENSOR_IDS  = [44]               # whitelist: only your own sensor IDs

client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2)
client.username_pw_set(MQTT_USER, MQTT_PASS)
client.connect(MQTT_HOST, MQTT_PORT)
client.loop_start()

ser = serial.Serial(SERIAL_PORT, BAUD, timeout=30)
logging.info("Connected to JeeLink on %s", SERIAL_PORT)

while True:
    try:
        line = ser.readline().decode("utf-8", errors="ignore").strip()

        if not line.startswith("OK 9"):
            continue

        parts = line.split()
        if len(parts) < 7:
            continue

        sensor_id = int(parts[2])

        if sensor_id not in SENSOR_IDS:
            continue

        flags     = int(parts[3])
        temp      = (int(parts[4]) * 256 + int(parts[5])) / 10.0 - 100.0
        humidity  = int(parts[6])
        new_batt  = bool(flags & 0x80)
        weak_batt = bool(flags & 0x01)

        logging.info("Sensor %d: %.1f°C, %d%% rH (new_batt=%s weak_batt=%s)",
                     sensor_id, temp, humidity, new_batt, weak_batt)

        topic = f"lacrosse/{sensor_id}"
        client.publish(f"{topic}/temperature", round(temp, 1), retain=True)
        client.publish(f"{topic}/humidity", humidity, retain=True)
        client.publish(f"{topic}/battery_low", int(weak_batt), retain=True)

    except Exception as e:
        logging.error("Error: %s", e)
        time.sleep(5)

The SENSOR_IDS whitelist is important. The 868 MHz band is shared, and you will pick up sensors from neighboring households. Temporarily remove the filter to discover your sensor’s ID - cross-reference it against the physical display or move the sensor around to identify which readings are yours.

The retain=True flag on every publish call is equally important. Without it, Home Assistant will show the entity as unavailable after a restart, because the broker has no stored value to replay to new subscribers.

Running as a Service

Create an rc script at /usr/local/etc/rc.d/lacrosse2mqtt:

#!/bin/sh
# PROVIDE: lacrosse2mqtt
# REQUIRE: NETWORKING
# KEYWORD: shutdown

. /etc/rc.subr

name="lacrosse2mqtt"
rcvar="lacrosse2mqtt_enable"
command="/usr/local/bin/python3"
command_args="/usr/local/bin/lacrosse2mqtt.py"
pidfile="/var/run/${name}.pid"
logfile="/var/log/${name}.log"

start_cmd="${name}_start"

lacrosse2mqtt_start()
{
    echo "Starting ${name}..."
    /usr/sbin/daemon -p ${pidfile} -o ${logfile} \
        ${command} ${command_args}
}

load_rc_config $name
: ${lacrosse2mqtt_enable:=NO}

run_rc_command "$1"

Enable and start it:

chmod +x /usr/local/etc/rc.d/lacrosse2mqtt
echo 'lacrosse2mqtt_enable="YES"' >> /etc/rc.conf
service lacrosse2mqtt start

Verify with service lacrosse2mqtt status and tail -f /var/log/lacrosse2mqtt.log.

Verifying MQTT

Before touching Home Assistant, confirm messages are arriving at the broker:

mosquitto_sub -h 127.0.0.1 -u your_mqtt_user -P your_mqtt_password -t "lacrosse/#" -v

lacrosse/44/temperature 20.7
lacrosse/44/humidity 51
lacrosse/44/battery_low 0

If nothing shows up, check the log file and make sure the MQTT credentials are correct.

Home Assistant Configuration

Add the sensors to configuration.yaml:

mqtt:
  sensor:
    - name: "Living Room Temperature"
      unique_id: "lacrosse_44_temperature"
      state_topic: "lacrosse/44/temperature"
      unit_of_measurement: "°C"
      device_class: temperature
      state_class: measurement

    - name: "Living Room Humidity"
      unique_id: "lacrosse_44_humidity"
      state_topic: "lacrosse/44/humidity"
      unit_of_measurement: "%"
      device_class: humidity
      state_class: measurement

    - name: "Living Room Sensor Battery"
      unique_id: "lacrosse_44_battery"
      state_topic: "lacrosse/44/battery_low"
      device_class: battery

Check the configuration under Developer Tools, restart Home Assistant, and the entities will appear under Settings > Devices & Services > Entities. You can also verify MQTT messages are arriving inside HA at Settings > Devices & Services > MQTT > Configure using the “Listen to a topic” field with lacrosse/#.

Adding More Sensors

When you add a new LaCrosse sensor:

Insert fresh batteries and let it transmit for a minute
Watch the log (tail -f /var/log/lacrosse2mqtt.log) for the new sensor ID
Add the ID to SENSOR_IDS in the Python script
Restart the service (service lacrosse2mqtt restart)
Add the corresponding MQTT sensors to configuration.yaml

Troubleshooting

Garbage output from the serial port: Wrong baud rate. The JeeLink with LaCrosseITPlusReader uses 57600. Verify interactively with cu before anything else.

Sensor IDs from neighbors: Expected on the shared 868 MHz band. Use the SENSOR_IDS whitelist.

new_batt=True after battery change: Normal. The sensor sets this flag temporarily after a battery insertion. It clears on its own.

Entity shows “—” in Home Assistant: MQTT messages are not being retained. Check that retain=True is set in all client.publish() calls.

MQTT connection refused: Your broker likely requires authentication. Add credentials via client.username_pw_set() in the script.

Conclusion

Component	Value
Serial device	`/dev/cuaU1`
Baud rate	57600
JeeLink sketch	LaCrosseITPlusReader 10.1s
MQTT topic structure	`lacrosse/<sensor_id>/temperature` etc.
Temperature formula	`(high * 256 + low) / 10.0 - 100.0`
Humidity	Direct byte value

The pattern here - USB device in a FreeBSD jail, data bridged over MQTT - generalizes well beyond LaCrosse sensors. Any USB-based RF receiver that can’t be passed directly into a bhyve VM can be handled this way. If you’re already running zigbee2mqtt in a jail, adding another serial device is just a few lines in your devfs rules and a small Python script away.

RHEL on ZFS Root: An Unholy Experiment

2026-02-14T00:00:00+01:00

Some experiments are born from necessity. Others from curiosity. This one was born from a Friday evening, a homelab VM with nothing to lose, and the thought: “I wonder if convert2rhel will just… let me do this.”

Let me be absolutely clear from the start: this is a terrible idea. Red Hat does not support ZFS on root. The supported solution in the RHEL ecosystem is Stratis. If you do this in production, you’re on your own. If you call Red Hat support with ZFS issues, they will politely suggest you reproduce the problem on a supported configuration before they even look at it.

This was purely an experiment for the lolz. Here’s what I did and why you shouldn’t.

The Proof It Works (For Now)

[root@testhost ~]# uname -a
Linux testhost 5.14.0-611.16.1.el9_7.x86_64 #1 SMP PREEMPT_DYNAMIC Sun Dec 7 05:52:24 EST 2025 x86_64 x86_64 x86_64 GNU/Linux

[root@testhost ~]# cat /etc/os-release 
NAME="Red Hat Enterprise Linux"
VERSION="9.7 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.7"
PRETTY_NAME="Red Hat Enterprise Linux 9.7 (Plow)"

[root@testhost ~]# zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
rpool  67.5G  4.09G  63.4G        -         -     3%     6%  1.00x    ONLINE  -

[root@testhost ~]# zfs list
NAME         USED  AVAIL  REFER  MOUNTPOINT
rpool       4.09G  61.3G    96K  none
rpool/home   176K  61.3G    96K  legacy
rpool/root  4.09G  61.3G  2.65G  /

[root@testhost ~]# lsmod | grep zfs
zfs                  7024640  8
spl                   217088  1 zfs

[root@testhost ~]# dmesg | grep zfs
[    1.466549] zfs: module license 'CDDL' taints kernel.
[    1.467131] zfs: module license taints kernel.

Yes, that’s genuine RHEL 9.7 running on a ZFS root pool. No, I don’t recommend it.

The How: A Tale of Two Bad Decisions

The path to this abomination was straightforward:

Step 1: Rocky Linux 9 on ZFS Root

Start with Rocky Linux 9 installed on ZFS root using the excellent OpenZFS documentation. This guide is well-maintained and produces a working system.

At this point, you have a perfectly functional Rocky Linux system with ZFS root. You could stop here. You should stop here. I did not stop here.

Step 2: Convert2RHEL (With Extreme Prejudice)

This is where things get cursed. Convert2RHEL is designed to convert CentOS, Rocky, or AlmaLinux systems to genuine RHEL. It performs extensive pre-flight checks to ensure the system is in a supported state before conversion.

ZFS root is very much not a supported state.

The tool correctly identified numerous problems:

Non-standard bootloader configuration (rEFInd instead of GRUB in the usual location)
Unexpected partition layout
Missing dracut modules for standard setups
Various “are you sure this is a real system?” sanity checks

The solution? Disable the checks. All of them. Every single one that stands between you and your bad decision.

convert2rhel --disable-submgr-checks --no-rpm-va \
    --disablerepo '*' --enablerepo rhel-9-for-x86_64-baseos-rpms \
    --enablerepo rhel-9-for-x86_64-appstream-rpms

That handles some of them. For the rest, I had to patch convert2rhel directly, commenting out validation functions that insisted on standard configurations. If you’ve ever edited a vendor tool’s source code to make it stop telling you “no,” you know the feeling. It’s the system administration equivalent of cutting the warning label off a mattress.

Step 3: Reboot and Pray

After the conversion completes, you need to manually clean up the remnants of the Rocky release packages. A few rpm -e invocations, some dnf distro-sync runs, and a quick check that the ZFS dracut modules are still intact in the initramfs. Nothing too dramatic.

Then comes the moment of truth. You type reboot, stare at the console, and wait.

The rEFInd menu appears. The kernel loads. ZFS imports the pool. Systemd starts doing its thing. And then - a login prompt. RHEL 9.7 on ZFS root, fully subscribed and receiving updates from Red Hat’s repositories. It actually worked.

I’m not going to pretend I wasn’t surprised.

Why This Is A Bad Idea

Let me count the ways:

No Support: Red Hat will not help you with ZFS issues. Period. The first thing support will ask is “can you reproduce this on a supported filesystem?”

Kernel Updates: ZFS is out-of-tree. Every kernel update triggers a DKMS rebuild of the ZFS modules. This usually works, but when it doesn’t, you’re debugging it yourself with no root filesystem.

Dracut Complexity: The initramfs needs ZFS modules and tools to mount the root pool. Any dracut configuration drift could leave you with an unbootable system.

Convert2RHEL Wasn’t Designed For This: By bypassing validation checks, you’re in uncharted territory. Future convert2rhel updates might break or refuse to run.

SELinux Contexts: ZFS supports xattrs (and xattr=sa is the default on Linux), so basic SELinux labeling works. But edge cases will bite you: dataset send/receive can lose security labels, full relabels are slow, and some ZFS operations don’t trigger the expected SELinux hooks. It mostly works, but “mostly” is doing a lot of heavy lifting in that sentence.

The Proper Solution: Stratis

If you want advanced storage features on RHEL, the supported path is Stratis. Stratis provides:

Pool-based storage management
Thin provisioning
Snapshots
Encryption support
Full Red Hat support

Stratis uses device-mapper for thin provisioning and XFS as the filesystem layer, providing a modern storage management interface while staying within the supported stack. It’s what Red Hat engineers will recommend when you ask about advanced filesystem features.

To use Stratis on RHEL 9:

sudo dnf install stratisd stratis-cli
sudo systemctl enable --now stratisd
sudo stratis pool create mypool /dev/sdb
sudo stratis filesystem create mypool myfs

The resulting filesystem can then be added to /etc/fstab with the x-systemd.requires=stratisd.service mount option.

What Actually Works

Surprisingly, most things just work. Package updates from RHEL repositories install without complaint. Systemd services start and stop like they should. Networking, user management, firewalld - all blissfully unaware that they’re sitting on a filesystem Oracle has lawyers for. SELinux mostly cooperates too, since ZFS defaults to xattr=sa on Linux and handles standard labeling fine.

What Might Not Work

Major version upgrades: Leapp (RHEL’s in-place upgrade tool) will absolutely not appreciate what you’ve done to this system. Going to RHEL 10 will require creative problem-solving, or more likely, a fresh install
Rescue mode: RHEL’s rescue environments assume your root is on ext4 or XFS. Booting into rescue mode on a ZFS root system gives you a very unhelpful shell
Automated diagnostics: Sosreport and friends expect / to be on a filesystem they recognize. Some checks will simply fail or produce nonsense
Support tickets: “Can you reproduce this on a supported filesystem?” will be the response to every case you open

The Honest Assessment

This works because the Linux kernel doesn’t particularly care what filesystem holds /. As long as the initramfs can mount it and userspace can find its libraries, the system boots. ZFS is mature, well-tested software. It handles the filesystem layer just fine.

But “works” and “supported” are very different things. This configuration works today, will probably work tomorrow, and could break spectacularly on the next kernel update. I wouldn’t run anything I care about on it.

Conclusion

Red Hat Enterprise Linux on ZFS root is technically possible. It’s also a bad idea. I did it because I wanted to see if I could, and because Friday evenings in the homelab are for experiments that would get you fired if you tried them at work.

If you’re reading this thinking “I should try this” - do it on a VM you’re prepared to destroy. And maybe don’t mention it in your next job interview.

For everyone else: Stratis exists, it’s supported, and Red Hat engineers won’t quietly judge you for using it.

References

My homelab VM didn’t deserve this treatment, but it got it anyway. The things we do for a blog post.

SELinux: A Practical Guide for Fedora and RHEL

2026-02-13T00:00:00+01:00

If you’ve spent time on Fedora or RHEL systems, you’ve encountered SELinux. And if you’re like many administrators, your first instinct when something doesn’t work is to check if SELinux is the culprit. The temptation to run setenforce 0 is strong - but it’s also the wrong approach.

SELinux (Security-Enhanced Linux) implements mandatory access control (MAC) at the kernel level. Unlike traditional Unix permissions (discretionary access control), SELinux policies are enforced regardless of file ownership or traditional permissions. A process running as root can still be blocked from accessing files if the SELinux policy doesn’t permit it.

This guide focuses on practical skills: understanding what SELinux is doing, troubleshooting denials, and configuring policies correctly. We’ll cover the concepts you need without getting lost in policy theory.

Most of the tools in this guide (semanage, audit2allow, restorecon) come from the policycoreutils-python-utils package, which isn’t always installed on minimal systems:

sudo dnf install policycoreutils-python-utils setroubleshoot-server

Why SELinux Matters

SELinux isn’t security theater. It provides genuine defense-in-depth:

Confinement: Even compromised root processes are limited by SELinux policy. A web server exploit that gains root can’t suddenly read SSH keys or modify system binaries.

Least privilege: Policies define exactly what each process type can access. Apache doesn’t need to read /etc/shadow, so it can’t.

Audit trail: Every denial is logged. Attempted intrusions leave fingerprints even when blocked.

The alternative - running in permissive mode or disabling SELinux entirely - removes this protection layer. The goal is to work with SELinux, not around it.

SELinux Modes

SELinux operates in three modes:

Mode	Behavior	Use Case
Enforcing	Policy is enforced, denials logged and blocked	Production systems
Permissive	Policy checked, denials logged but not blocked	Debugging, policy development
Disabled	SELinux entirely inactive	Not recommended

Check current status:

sestatus

Output shows the mode, policy name, and other details:

SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actual (secure)
Max kernel policy version:      33

Temporarily switch modes:

# Switch to permissive (survives until reboot)
sudo setenforce 0

# Return to enforcing
sudo setenforce 1

For permanent changes, edit /etc/selinux/config:

SELINUX=enforcing
SELINUXTYPE=targeted

The targeted policy is standard on Fedora and RHEL - it confines specific daemons while leaving user processes largely unrestricted. Other policy types exist (mls for multi-level security) but are niche.

Understanding SELinux Contexts

Everything in SELinux revolves around contexts (also called labels). Every process, file, socket, and network port has a security context that determines what can access what.

Viewing Contexts

Files and directories:

ls -Z /var/www/html/index.html

unconfined_u:object_r:httpd_sys_content_t:s0 /var/www/html/index.html

Processes:

ps -Z -C httpd

LABEL                              PID TTY          TIME CMD
system_u:system_r:httpd_t:s0      1234 ?        00:00:01 httpd

Ports:

semanage port -l | grep http

http_cache_port_t              tcp      8080, 8118, 8123, 10001-10010
http_port_t                    tcp      80, 81, 443, 488, 8008, 8009, 8443, 9000

Context Format

Contexts have four colon-separated fields:

user:role:type:level

For practical administration, the type field (httpd_sys_content_t, httpd_t) matters most. Types ending in _t are what policies match against.

User (system_u, unconfined_u): SELinux user, maps to Linux users
Role (system_r, object_r): Groups types for role-based access
Type (httpd_t, httpd_sys_content_t): The primary policy discriminator
Level (s0): MLS/MCS sensitivity level (usually s0 in targeted policy)

How Policy Works

The policy defines rules like:

“A process with type httpd_t can read files with type httpd_sys_content_t“

This is why placing a file in /var/www/html/ doesn’t automatically make it readable by Apache - the context must match. A file with user_home_t context won’t be accessible even if owned by apache:apache.

Common Administrative Tasks

Changing File Contexts

When you move or create files outside standard locations, they won’t have the correct context. This often catches people off guard: cp inherits the destination directory’s context, but mv preserves the original context. Moving a file from your home directory into /var/www/html/ leaves it with user_home_t - and Apache can’t read it despite the file being in the right place.

Two approaches exist to fix contexts:

Temporary (relabel gets reset):

chcon -R -t httpd_sys_content_t /srv/www/

Persistent (survives relabels):

# Define the context mapping
semanage fcontext -a -t httpd_sys_content_t "/srv/www(/.*)?"

# Apply the context
restorecon -Rv /srv/www/

The semanage fcontext command adds rules to the policy. restorecon applies those rules. This is the correct approach for production.

List custom context mappings:

semanage fcontext -l | grep srv

Managing Port Labels

Running a service on a non-standard port? The port needs the correct label:

# Allow Apache to listen on port 8081
semanage port -a -t http_port_t -p tcp 8081

Remove a custom port mapping:

semanage port -d -t http_port_t -p tcp 8081

SELinux Booleans

Booleans toggle specific policy behaviors without rewriting policy. They’re the preferred way to enable common functionality:

# List booleans related to HTTP
semanage boolean -l | grep httpd

# Or using getsebool
getsebool -a | grep httpd

Common HTTP booleans:

# Allow Apache to connect to network (for reverse proxy)
setsebool -P httpd_can_network_connect 1

# Allow Apache to send email
setsebool -P httpd_can_sendmail 1

# Allow home directory access
setsebool -P httpd_enable_homedirs 1

# Allow network relay (for mod_proxy)
setsebool -P httpd_can_network_relay 1

The -P flag makes changes persistent. Without it, changes reset on reboot.

Troubleshooting SELinux Denials

When something doesn’t work and you suspect SELinux, here’s the systematic approach:

Step 1: Confirm It’s SELinux

Temporarily switch to permissive:

sudo setenforce 0

If the problem goes away, SELinux is the cause. Switch back to enforcing while you debug:

sudo setenforce 1

Step 2: Find the Denial

SELinux denials are logged to /var/log/audit/audit.log. The format is dense, but tools help:

# Check recent denials
ausearch -m AVC -ts recent

# More readable format
ausearch -m AVC -ts recent | audit2allow -w

Example output:

Type: AVC
Summary:

SELinux is preventing /usr/sbin/httpd from write access on the directory /var/www/uploads.

*****  Plugin catchall_boolean (89.3 confidence) suggests   ******************

If you want to allow httpd to have unified access to all httpd content
Then you must tell SELinux about this by enabling the 'httpd_unified' boolean.

Do
setsebool -P httpd_unified 1

For systems without auditd, check /var/log/messages for setroubleshoot messages.

Step 3: Understand the Denial

Raw audit entry:

type=AVC msg=audit(1708032000.123:456): avc:  denied  { write } for  pid=1234 comm="httpd" name="uploads" dev="dm-0" ino=789 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:httpd_sys_content_t:s0 tclass=dir permissive=0

Key fields:

scontext: Source context (the process)
tcontext: Target context (what was accessed)
tclass: Object class (file, dir, socket)
{ write }: The denied permission

Step 4: Fix the Denial

Options, from best to worst:

1. Boolean (preferred for common cases):

audit2allow -w -a

Often suggests a boolean. Enable it.

2. Context fix (for mislabeled files):

# Check current context
ls -Z /var/www/uploads/

# Fix to appropriate type
semanage fcontext -a -t httpd_sys_rw_content_t "/var/www/uploads(/.*)?"
restorecon -Rv /var/www/uploads/

3. Custom policy module (for application-specific needs):

# Generate policy from audit log
ausearch -c 'myapp' --raw | audit2allow -M myapp

# Apply the module
semodule -i myapp.pp

4. Per-domain permissive (temporary workaround):

# Make httpd permissive while keeping system enforcing
semanage permissive -a httpd_t

Never disable SELinux globally for an application-specific issue.

SELinux and Containers

Podman and containers have specific SELinux considerations. The container runtime automatically sets up isolation, but volume mounts require attention.

Container File Labels

When mounting host directories into containers, SELinux contexts matter. Podman provides flags to handle this:

# The :z flag relabels the directory for container use
podman run -v /srv/data:/data:z nginx

# The :Z flag makes the volume private to this container
podman run -v /srv/data:/data:Z nginx

:z (lowercase): Relabels with container_file_t, shared among containers
:Z (uppercase): Private labeling, only this container can access

Warning: Both flags relabel the host directory. If another service on the host depends on the original label, :z or :Z will break it. Only use these on directories dedicated to container use.

Quadlet Example

In Podman Quadlet files:

[Container]
Image=docker.io/library/nginx:latest
Volume=/srv/web:/usr/share/nginx/html:z

The :z flag ensures SELinux doesn’t block the container from reading mounted files.

Checking Container Contexts

# Container processes run in container_t
ps -Z | grep container

# Files in container volumes should have container_file_t
ls -Z /srv/web

Relabeling the Filesystem

If contexts get completely mangled (or after restoring from backup), force a full relabel:

# Create relabel trigger
touch /.autorelabel

# Reboot
reboot

The system relabels all files on next boot. This takes time on large filesystems.

For targeted relabeling:

restorecon -Rv /path/to/directory

Common Scenarios

Web Server with Non-Standard Document Root

# Set context for new web root
semanage fcontext -a -t httpd_sys_content_t "/srv/www(/.*)?"
restorecon -Rv /srv/www

# For writable directories (uploads, cache)
semanage fcontext -a -t httpd_sys_rw_content_t "/srv/www/uploads(/.*)?"
restorecon -Rv /srv/www/uploads

Database with Custom Data Directory

# PostgreSQL
semanage fcontext -a -t postgresql_db_t "/srv/pgdata(/.*)?"
restorecon -Rv /srv/pgdata

# MariaDB/MySQL
semanage fcontext -a -t mysqld_db_t "/srv/mysqldata(/.*)?"
restorecon -Rv /srv/mysqldata

SSH on Non-Standard Port

# Add port to ssh_port_t
semanage port -a -t ssh_port_t -p tcp 2222

# Verify
semanage port -l | grep ssh

Home Directory Web Access

# Enable userdir module access
setsebool -P httpd_enable_homedirs 1

# For specific user home access (more restrictive)
setsebool -P httpd_read_user_content 1

Useful Commands Reference

Task	Command
Check status	`sestatus`
List file contexts	`ls -Z`
List process contexts	`ps -Z`
Change file context	`chcon -t type file`
Add persistent context	`semanage fcontext -a -t type "/path(/.*)?"`
Apply contexts	`restorecon -Rv /path`
List booleans	`getsebool -a` or `semanage boolean -l`
Set boolean	`setsebool -P boolean_name 1`
List port labels	`semanage port -l`
Add port label	`semanage port -a -t type -p tcp port`
Search denials	`ausearch -m AVC -ts recent`
Explain denials	`audit2allow -w -a`
Create policy module	`ausearch --raw \\| audit2allow -M name`
Install module	`semodule -i name.pp`
List modules	`semodule -l`

Best Practices

Never disable SELinux globally. If an application has issues, troubleshoot the specific denial.

Use booleans first. Many common use cases have pre-built toggles.

Prefer context changes over policy changes. Labeling files correctly is cleaner than adding policy rules.

Document customizations. Track semanage commands in configuration management (Ansible, Salt) so systems remain reproducible.

Test in permissive first. When developing policy, run in permissive to collect all denials before switching to enforcing.

Use audit2allow -w for explanations. Before creating a policy module, understand what you’re allowing.

Conclusion

SELinux is a powerful security layer that protects systems even when traditional permissions are bypassed. The learning curve is real, but the skills are essential for Fedora and RHEL administration.

The key insight: SELinux isn’t trying to break your applications. When denials occur, it’s usually because files or ports are mislabeled, or a boolean needs to be set. The tools exist to diagnose and fix these issues correctly - use them instead of disabling the protection.

Once you understand contexts, booleans, and the troubleshooting workflow, SELinux becomes a tool you work with rather than an obstacle to work around.

References

SELinux was born from the NSA’s research into mandatory access controls - military-grade security that somehow became a default on every Fedora workstation and RHEL server. Most of the time it’s invisible, doing its job without asking. When it does push back, it’s specific enough to tell you exactly what went wrong and how to fix it. In a world where privilege escalation makes regular headlines, a kernel-level layer that constrains even root isn’t an inconvenience - it’s quiet insurance.

Podman 5.8: Quadlet Multi-File Install, Automatic SQLite Migration, and the Road to 6.0

2026-02-12T00:00:00+01:00

Podman 5.8 dropped this week, and while it’s not a flashy major release, it lays important groundwork for the upcoming 6.0 transition. The headline features are a significantly improved Quadlet install workflow and an automatic database migration that moves users from BoltDB to SQLite - quietly preparing the ecosystem for the next generation.

If you’ve been running Quadlet-based deployments (I wrote about that approach in detail in my production-grade Podman Quadlets guide), several of these changes directly improve that workflow.

Quadlet Multi-File Install

The biggest usability improvement in 5.8 is multi-file support for podman quadlet install. Previously, installing a multi-container setup meant feeding each Quadlet unit file to the command individually. Now you can bundle multiple Quadlet definitions into a single file, separated by --- delimiters, with each section identified by a # FileName=<name> header.

# FileName=myapp-db.container
[Container]
Image=docker.io/library/postgres:17
...

---

# FileName=myapp-web.container
[Container]
Image=docker.io/library/nginx:latest
...

---

# FileName=myapp.network
[Network]
Subnet=10.89.1.0/24

This brings Quadlet closer to the “single manifest” experience people are used to from Docker Compose or Kubernetes YAML, while retaining the systemd-native integration that makes Quadlet compelling in the first place. For complex deployments with multiple containers, networks, and volumes, this cuts the friction significantly.

Automatic BoltDB to SQLite Migration

This is the change that matters most for the long term. Podman has been transitioning its internal database from BoltDB to SQLite, and 5.8 makes this migration automatic: legacy BoltDB databases will convert to SQLite upon system reboot.

# If automatic migration fails, you can trigger it manually
podman system migrate --migrate-db

The motivation is straightforward. SQLite offers better concurrent access, more reliable crash recovery, and a proven track record at scale. BoltDB served Podman well, but as container workloads grow more complex - particularly with pods and Quadlet-managed services - the limitations of a simple key-value store become apparent.

This is explicitly preparation for Podman 6.0, where SQLite will be the only supported backend. If you’re running 5.8, the migration happens transparently. If something goes wrong, the manual migration command is your fallback.

AppArmor Support in Quadlet

Container Quadlet files now accept an AppArmor key for configuring container AppArmor profiles directly in the unit definition:

[Container]
Image=docker.io/library/nginx:latest
AppArmor=my-custom-profile

For distributions that use AppArmor as their MAC system (Debian, Ubuntu, SUSE), this removes a gap in Quadlet’s security configuration. Previously, you’d need to pass AppArmor settings through the less ergonomic PodmanArgs escape hatch.

Performance and Usability Improvements

Several smaller changes round out the release:

podman exec --no-session disables session tracking for exec commands. If you’re running frequent exec calls against a container - health checks, monitoring scripts, batch operations - the session tracking overhead adds up. This flag trades tracking for raw speed.

podman update --ulimit lets you modify container ulimits on a running container without recreating it. Useful when you realize your database container needs more open file descriptors and you’d rather not cycle the whole service.

VM artifact path optimization improves podman artifact add when working with podman machine VMs. Files on shared paths are now loaded directly from the VM filesystem rather than streaming through the REST API - a notable speedup for large artifacts.

podman secret create - no longer requires pipe input. You can now type a secret directly at the terminal prompt, which is more intuitive for interactive use.

Bugfixes

The release addresses a range of issues across the stack:

Healthcheck start period timing now works correctly with Kubernetes YAML deployments
Environment variable precedence in kube play follows the expected order
Volume mount path handling is more robust across edge cases
Authentication handling improvements for private registries
Windows path processing fixes for cross-platform workflows

Updated Dependencies

Podman 5.8 ships with Buildah v1.43.0, containers/storage v1.62.0, containers/image v5.39.1, and containers/common v0.67.0.

Looking Ahead

The SQLite migration signals where Podman is heading. Version 6.0 will likely bring more significant architectural changes, and 5.8 is doing the responsible thing: migrating users incrementally rather than forcing a flag day. If you’re running Podman in production, updating to 5.8 sooner rather than later gives you a smooth transition path.

For Quadlet users in particular, the multi-file install support makes managing complex deployments meaningfully easier. Combined with the AppArmor integration and the ongoing maturation of the Quadlet system, Podman continues to build a compelling case as a production container runtime that doesn’t need a daemon or an orchestrator to be useful.

References

Podman 5.8.0 Release Notes
Linuxiac: Podman 5.8 Introduces Quadlet Multi-File Install and SQLite Migration
Podman Quadlet Documentation
Production-Grade Container Deployment with Podman Quadlets - my earlier deep-dive into Quadlet-based deployments

Adding Fediverse Comments to a Pelican Blog

2026-02-10T00:00:00+01:00

Every static site eventually faces the comment question. Disqus tracks your readers. Self-hosted solutions like Commento or Isso need a server-side component and a database. Most options either compromise privacy or add operational complexity that feels disproportionate for a personal blog.

Then I came across Jan Wildeboer’s approach for his Jekyll blog: use Mastodon as the comment backend. Every blog post gets a corresponding Mastodon post. Replies to that post become the article’s comments, fetched client-side through the public Mastodon API. No tracking, no database, no server-side logic. Just the Fediverse doing what it already does.

I liked the idea enough to port it to Pelican. This article explains how it works, how it’s integrated, and how you could do the same.

The Concept

The Mastodon API exposes a public endpoint that returns the full conversation context for any post:

GET https://instance.tld/api/v1/statuses/{id}/context

The response includes an ancestors array (the thread above the post) and a descendants array (all replies). We only care about descendants - those are the comments. No authentication is required for public posts.

The workflow is straightforward:

Publish your blog article
Post about it on Mastodon
Add the Mastodon post’s metadata to your article’s frontmatter
Rebuild the site

From that point on, anyone who replies to your Mastodon post will have their reply appear as a comment on the blog.

Integration into Pelican

Article Metadata

Pelican lets you define arbitrary metadata fields in article frontmatter. I added three fields to link an article to its Mastodon post:

Mastodon_Host: burningboard.net
Mastodon_User: Larvitz
Mastodon_Id: 116035059515573876

Mastodon_Host - the domain of the Mastodon instance
Mastodon_User - your handle on that instance
Mastodon_Id - the numeric status ID (grab it from the post URL)

These become available in templates as article.mastodon_host, article.mastodon_user, and article.mastodon_id. Articles without these fields simply don’t get a comment section.

The Comment Template

The comment system lives in a single Jinja2 include file at themes/pelican-alchemy-custom/templates/include/comments.html. The entire thing is wrapped in a conditional:

{% if article.mastodon_id %}
<section class="fediverse-comments">
  ...
</section>
{% endif %}

This gets included at the bottom of the article template:

{% include 'include/comments.html' %}

Since my blog uses the Alchemy theme, I override the default article.html in a custom template directory loaded via Pelican’s THEME_TEMPLATES_OVERRIDES setting. This keeps the base theme untouched as a git submodule while allowing targeted modifications.

The JavaScript

The client-side logic is compact. On page load, it fetches the conversation context and renders each reply:

fetch('https://{{ article.mastodon_host }}/api/v1/statuses/{{ article.mastodon_id }}/context')
    .then(function(response) {
      return response.json();
    })
    .then(function(data) {
      if (data['descendants'] &&
          Array.isArray(data['descendants']) &&
          data['descendants'].length > 0) {
        data['descendants'].forEach(function(reply) {
          // Build and render each comment
        });
      } else {
        // "No comments yet" message
      }
    });

Each reply from the API includes the author’s display name, avatar, profile URL, the reply content (as HTML), a timestamp, and engagement counts (replies, boosts, favourites). The script builds an HTML structure for each comment and appends it to the page.

One nice detail: Mastodon custom emojis in display names. The API returns an emojis array for each account, with shortcodes and image URLs. The script replaces :shortcode: patterns in display names with the actual emoji images, so usernames render correctly.

XSS Protection

Since Mastodon post content arrives as HTML, rendering it directly would be a textbook XSS vulnerability. The implementation handles this in two layers:

Manual escaping - all user-controlled strings (display names, URLs, avatar paths) are run through an escapeHtml() function before being placed into the comment markup
DOMPurify - the assembled comment HTML is sanitized through DOMPurify before being injected into the DOM

document.getElementById('mastodon-comments-list').appendChild(
    DOMPurify.sanitize(comment, {RETURN_DOM_FRAGMENT: true})
);

The DOMPurify library is served as a local static file rather than loaded from a CDN, avoiding external dependencies. It’s registered in pelicanconf.py as a static path:

STATIC_PATHS = ['images', 'extra/custom.css', 'extra/robots.txt', 'extra/purify.min.js']
EXTRA_PATH_METADATA = {
    'extra/purify.min.js': {'path': 'purify.min.js'},
}

Interaction Flow

Readers who want to comment need a Mastodon (or other ActivityPub) account. The template provides a “Copy post link” button that copies the Mastodon post URL to the clipboard. They can then search for that URL on their own instance, find the post, and reply. Their reply will show up on the blog the next time someone loads the page - no rebuild required, since everything is fetched live.

Styling

The comment section is styled to match the blog’s Solarized Dark theme. Each comment is a flexbox row with a circular avatar on the left and the content on the right. Author names link to their Mastodon profiles. Dates link to the original reply. Engagement metrics (replies, boosts, favourites) are displayed in a muted footer.

Trade-offs

This approach has real limitations worth acknowledging:

JavaScript required - the comments won’t load without it. A <noscript> fallback tells the reader.
Content warnings are ignored - Mastodon’s CW feature isn’t reflected in the rendered comments.
Media attachments are excluded - only text content is displayed.
Moderation is coarse - you moderate through Mastodon itself. Blocking an account or deleting the source post affects all comments. There’s no per-comment moderation from the blog side.
The source post must stay up - if you delete the Mastodon post, the comments disappear. Bookmark your posts.
CORS dependency - the Mastodon instance must allow cross-origin requests to the API, which is the default for most instances.

On the other hand, the advantages are compelling for a personal blog: zero infrastructure, full reader privacy, decentralized identity, and comments from people who already have Fediverse accounts - which self-selects for a community I actually want to hear from.

The Full Picture

The implementation touches exactly four things:

themes/pelican-alchemy-custom/templates/include/comments.html - the template with HTML structure and JavaScript logic
themes/pelican-alchemy-custom/templates/article.html - the article template, with one {% include %} line added
content/extra/purify.min.js - DOMPurify library as a static asset
content/extra/custom.css - styling for the comment section

Plus three metadata fields in any article that should have comments enabled. That’s it. No plugins, no external services, no build-time processing. The comments are entirely a client-side feature.

You can see it in action on my BGP article.

Attribution

This implementation is directly inspired by Jan Wildeboer (@jwildeboer@social.wildeboer.net), who published this approach for his Jekyll blog in February 2023. I took his logic and ported it to Pelican’s Jinja2 templating system. The core idea - using Mastodon’s public conversation API as a comment backend - is entirely his.

References

Running Your Own AS: BGP on FreeBSD with FRR, GRE Tunnels, and Policy Routing

2026-02-08T00:00:00+01:00

Running your own Autonomous System on the public internet sounds like something reserved for ISPs and large enterprises. It’s not. With sponsoring LIRs making AS numbers and IPv6 prefixes accessible to individuals, and FreeBSD providing the routing tools to make it work, you can announce your own address space to the Default-Free Zone from a single virtual machine.

This article walks through the complete setup: obtaining resources from RIPE via a sponsoring LIR, configuring a FreeBSD BGP router with FRR, building GRE/GIF tunnels to distribute prefixes to remote servers, and solving the routing challenge that arises when a server needs to speak from two different IPv6 address spaces simultaneously.

Note on addresses: All provider-assigned IP addresses, hostnames, and management IPs in this article have been replaced with RFC 5737 / RFC 3849 documentation ranges. My own AS number (AS201379) and prefix (2a06:9801:1c::/48) are public BGP resources and shown as-is. The upstream AS numbers (AS34927, AS209735) are equally visible in public routing tables.

Why Run Your Own AS?

Provider-assigned IPv6 addresses are tied to that provider. Move to a different hoster and your addresses change - along with DNS records, firewall rules, reputation, and every system that references them. With your own AS and prefix, your addresses follow you. Migrate a server, update a tunnel endpoint, and traffic flows again without touching a single service configuration.

There are also less practical reasons. Understanding BGP transforms how you think about internet routing. Watching your prefix propagate through the DFZ and appear on looking glasses worldwide is genuinely satisfying. And if you run services across multiple providers, having provider-independent addressing simplifies the architecture considerably.

Obtaining Resources

To announce prefixes on the internet, you need two things from a Regional Internet Registry (in Europe, that’s RIPE NCC):

An AS number - your identity in BGP. Mine is AS201379.
An IPv6 prefix - the address space you’ll announce. I received 2a06:9801:1c::/48.

As an individual, you don’t need to become a RIPE member (which involves fees and bureaucracy). Instead, you work with a sponsoring LIR - an existing RIPE member who sponsors your resource registration. Several LIRs cater to hobbyists and small operators. The process typically involves:

Filling out a request form with your intended use case
Creating the appropriate RIPE database objects (aut-num, inet6num, route6)
Setting up RPKI ROAs (Route Origin Authorizations) to cryptographically bind your prefix to your AS

Once the paperwork is done, you need upstream connectivity - someone willing to carry your BGP sessions and announce your routes to the rest of the internet.

Architecture Overview

The setup involves two tiers: a BGP router that peers with upstream providers, and downstream servers that receive tunneled subnets from the router’s /48.

                    ┌──────────────────────────────┐
                    │     Default-Free Zone         │
                    └──────┬──────────────┬─────────┘
                           │              │
                    AS34927 (iFog)   AS209735 (Lagrange)
                           │              │
                      GRE tunnel     Direct peering
                           │              │
                    ┌──────┴──────────────┴─────────┐
                    │    router01 (BGP Router)       │
                    │     FreeBSD + FRR              │
                    │     AS201379                   │
                    │     2a06:9801:1c::/48          │
                    └──────┬──────────────┬─────────┘
                           │              │
                      GIF tunnel     GIF tunnel
                      (proto 41)     (proto 41)
                           │              │
                    ┌──────┴───┐   ┌──────┴──────────┐
                    │  vps01   │   │  dcgw01          │
                    │  VPS     │   │  DC OPNsense     │
                    │  :1000:: │   │  :2000::/62      │
                    │  /64     │   │                   │
                    └──────────┘   └──────────────────┘

The BGP router (router01) announces 2a06:9801:1c::/48 to two upstream providers and maintains a blackhole route for the aggregate. Individual /64s (and a /62 for my Colocation datacenter) are tunneled to downstream servers via GIF tunnels (IPv6-in-IPv4 encapsulation). Each server receives real, globally routable addresses from my prefix while keeping its existing provider-assigned IPv6 fully operational.

The BGP Router

The router runs on a FreeBSD VM at a colocation facility with direct connectivity to two upstream networks. Let’s walk through each layer.

Network Configuration

The router’s /etc/rc.conf sets up the physical interface, tunnel interfaces, and static routes:

hostname="router01"

# Security
kern_securelevel_enable="YES"
kern_securelevel="2"

# Physical interface
ifconfig_vtnet0="inet 198.51.100.10/24 -rxcsum -txcsum -rxcsum6 -txcsum6 -lro -tso"
ifconfig_vtnet0_ipv6="inet6 2001:db8:100::96/64"

defaultrouter="198.51.100.1"
ipv6_defaultrouter="2001:db8:100::1"

# Loopback alias for originated prefix
ifconfig_lo0_alias0="inet6 2a06:9801:1c::1 prefixlen 64"

# Tunnel interfaces
cloned_interfaces="gif0 gif1 gre0"
kld_list="if_gif if_gre"

# GRE Tunnel to transit provider (iFog)
ifconfig_gre0="tunnel 198.51.100.10 198.51.100.44"
ifconfig_gre0_ipv6="inet6 2001:db8:300::2 2001:db8:300::1 prefixlen 128"
ifconfig_gre0_descr="Transit-iFog"

# GIF Tunnel to VPS (vps01)
ifconfig_gif0="tunnel 198.51.100.10 203.0.113.10"
ifconfig_gif0_ipv6="inet6 2a06:9801:1c:ffff::1 2a06:9801:1c:ffff::2 prefixlen 128"
ifconfig_gif0_descr="Tunnel-to-VPS"
ipv6_route_cloud="2a06:9801:1c:1000::/64 2a06:9801:1c:ffff::2"

# GIF Tunnel to datacenter firewall (dcgw01)
ifconfig_gif1="tunnel 198.51.100.10 192.0.2.50"
ifconfig_gif1_ipv6="inet6 2a06:9801:1c:ffff::3 2a06:9801:1c:ffff::4 prefixlen 128"
ifconfig_gif1_descr="Tunnel-to-Datacenter"
ipv6_route_dc="2a06:9801:1c:2000::/62 2a06:9801:1c:ffff::4"

# Blackhole route for the aggregate + downstream routes
ipv6_static_routes="myblock cloud dc"
ipv6_route_myblock="2a06:9801:1c::/48 -reject"
ipv6_gateway_enable="YES"

# Services
pf_enable="YES"
pflog_enable="YES"
frr_enable="YES"
zfs_enable="YES"
sshd_enable="YES"

A few things worth explaining:

The blackhole route (-reject for the /48) is essential. Without it, traffic for unassigned subnets within your prefix would follow the default route back to the upstream, creating a routing loop. The blackhole ensures unrouted traffic is dropped locally.
Point-to-point tunnel addresses use /128 prefixes on the 2a06:9801:1c:ffff::/64 link subnet. Each tunnel gets a pair of addresses from this range.
Downstream routes point specific subnets at the far end of each tunnel. The /64 for the VPS and /62 for the datacenter are routed to their respective tunnel endpoints.
GRE vs GIF: The iFog peering uses GRE because that’s what the provider requires. The downstream tunnels use GIF (protocol 41, IPv6-in-IPv4) which is simpler and has less overhead.

FRR Configuration

FRR (Free Range Routing) handles the BGP sessions. The configuration lives at /usr/local/etc/frr/frr.conf:

frr version 10.5.1
frr defaults traditional
hostname router01
log syslog informational
service integrated-vtysh-config
!
ipv6 prefix-list PL-MY-NET seq 5 permit 2a06:9801:1c::/48
!
ipv6 prefix-list PL-BOGONS seq 5 deny ::/0 le 7
ipv6 prefix-list PL-BOGONS seq 10 deny ::/8
ipv6 prefix-list PL-BOGONS seq 15 deny 100::/8
ipv6 prefix-list PL-BOGONS seq 20 deny 200::/7
ipv6 prefix-list PL-BOGONS seq 25 deny 400::/6
ipv6 prefix-list PL-BOGONS seq 30 deny 800::/5
ipv6 prefix-list PL-BOGONS seq 35 deny 1000::/4
ipv6 prefix-list PL-BOGONS seq 40 deny 4000::/3
ipv6 prefix-list PL-BOGONS seq 45 deny 6000::/3
ipv6 prefix-list PL-BOGONS seq 50 deny 8000::/3
ipv6 prefix-list PL-BOGONS seq 55 deny a000::/3
ipv6 prefix-list PL-BOGONS seq 60 deny c000::/3
ipv6 prefix-list PL-BOGONS seq 65 deny e000::/4
ipv6 prefix-list PL-BOGONS seq 70 deny f000::/5
ipv6 prefix-list PL-BOGONS seq 75 deny f800::/6
ipv6 prefix-list PL-BOGONS seq 80 deny fc00::/7
ipv6 prefix-list PL-BOGONS seq 85 deny fe80::/10
ipv6 prefix-list PL-BOGONS seq 90 deny fec0::/10
ipv6 prefix-list PL-BOGONS seq 95 deny ff00::/8
ipv6 prefix-list PL-BOGONS seq 100 deny 2a06:9801:1c::/48
ipv6 prefix-list PL-BOGONS seq 105 deny ::/0 ge 49
ipv6 prefix-list PL-BOGONS seq 110 permit ::/0 le 48
!
route-map RM-IFOG-OUT permit 10
 match ipv6 address prefix-list PL-MY-NET
 set community 34927:9501 34927:9301 additive
exit
!
route-map RM-LAGRANGE-OUT permit 10
 match ipv6 address prefix-list PL-MY-NET
 set as-path prepend 201379 201379
exit
!
route-map RM-IFOG-IN permit 10
 match ipv6 address prefix-list PL-BOGONS
exit
!
route-map RM-LAGRANGE-IN permit 10
 match ipv6 address prefix-list PL-BOGONS
exit
!
ipv6 route 2a06:9801:1c::/48 blackhole
!
router bgp 201379
 bgp router-id 198.51.100.10
 no bgp default ipv4-unicast
 neighbor 2001:db8:300::1 remote-as 34927
 neighbor 2001:db8:300::1 description Upstream-iFog
 neighbor 2001:db8:300::1 ttl-security hops 1
 neighbor 2001:db8:300::1 update-source gre0
 neighbor 2001:db8:100::ff remote-as 209735
 neighbor 2001:db8:100::ff description Upstream-Lagrange
 neighbor 2001:db8:100::ff ttl-security hops 1
 neighbor 2001:db8:100::ff update-source 2001:db8:100::96
 !
 address-family ipv6 unicast
  network 2a06:9801:1c::/48
  neighbor 2001:db8:300::1 activate
  neighbor 2001:db8:300::1 soft-reconfiguration inbound
  neighbor 2001:db8:300::1 maximum-prefix 250000 90 restart 30
  neighbor 2001:db8:300::1 route-map RM-IFOG-IN in
  neighbor 2001:db8:300::1 route-map RM-IFOG-OUT out
  neighbor 2001:db8:100::ff activate
  neighbor 2001:db8:100::ff soft-reconfiguration inbound
  neighbor 2001:db8:100::ff maximum-prefix 250000 90 restart 30
  neighbor 2001:db8:100::ff route-map RM-LAGRANGE-IN in
  neighbor 2001:db8:100::ff route-map RM-LAGRANGE-OUT out
 exit-address-family
exit

There’s a lot happening here. Let me break down the key design decisions.

Prefix Lists

Two prefix lists control what gets sent and received:

PL-MY-NET: Matches only our /48. Used in outbound route-maps to ensure we only ever announce our own prefix.
PL-BOGONS: A comprehensive bogon filter for inbound routes. This rejects non-routable address space (link-local, ULA, multicast, documentation ranges), our own prefix (to prevent loops), and anything more specific than a /48 or less specific than a /8. The final permit ::/0 le 48 at the end accepts everything that survived the deny rules.

The bogon filter deserves emphasis. Accepting bad routes from peers can cause anything from black-holed traffic to becoming an unwitting participant in route hijacks. Filter aggressively on inbound.

Route Maps

Each peer gets its own pair of inbound/outbound route maps:

Outbound to iFog (RM-IFOG-OUT): Announces our /48 with BGP communities 34927:9501 and 34927:9301. These are iFog-specific communities that control route propagation - in this case, requesting announcement to specific peering partners.
Outbound to Lagrange (RM-LAGRANGE-OUT): Announces our /48 with AS-path prepending (adds our ASN twice). This makes the Lagrange path appear longer to the rest of the internet, steering inbound traffic to prefer the iFog path. Useful for traffic engineering when one upstream has better connectivity.
Inbound from both: Apply the bogon filter to reject garbage routes.

BGP Session Details

no bgp default ipv4-unicast: We’re IPv6-only. Don’t activate IPv4 address family by default.
ttl-security hops 1: GTSM (Generalized TTL Security Mechanism) - reject BGP packets with TTL less than 254. This prevents remote attacks on the BGP session since only directly connected peers can send packets with TTL 255.
soft-reconfiguration inbound: Store received routes before applying filters. This lets you change inbound policy without resetting the BGP session.
maximum-prefix 250000 90 restart 30: Safety valve. If a peer sends more than 250,000 prefixes (or 90% of that as a warning), tear down the session and retry after 30 minutes. Protects against route leaks from upstream.

Firewall on the Router

The BGP router’s PF configuration protects the control plane while allowing data plane forwarding:

# --- Macros ---
ext_if = "vtnet0"
dc_tun = "gif1"
vps_tun = "gif0"

trusted_ipv4 = "{ 198.51.100.100, 198.51.100.101 }"
trusted_ipv6 = "{ 2001:db8:ffff:1::/64, 2001:db8:ffff:2::/64 }"

bgp_peers_v4 = "{ 198.51.100.20 }"
bgp_peers_v6 = "{ 2001:db8:100::ff }"
ifog_gre_endpoint = "198.51.100.44"
ifog_bgp_peer     = "2001:db8:300::1"

my_network_v6 = "2a06:9801:1c::/48"
vps_v4 = "203.0.113.10"

# --- Tables ---
table <bruteforce> persist
table <trusted_v4> const { $trusted_ipv4 }
table <trusted_v6> const { $trusted_ipv6 }
table <bgp_peers_v4> const { $bgp_peers_v4 }
table <bgp_peers_v6> const { $bgp_peers_v6 }
table <bogons> const { 0.0.0.0/8, 10.0.0.0/8, 172.16.0.0/12, \
    192.168.0.0/16, 169.254.0.0/16, ::/96, fc00::/7, \
    fec0::/10, ff00::/8 }

# --- Options ---
set skip on lo0
set block-policy drop
set loginterface $ext_if

# --- Scrub ---
scrub in all fragment reassemble
scrub on $vps_tun max-mss 1440
scrub on $dc_tun max-mss 1140
scrub on gre0 max-mss 1400

# --- Filtering ---
block log all
block in quick on $ext_if from { <bogons>, $my_network_v6 } to any
antispoof quick for { $ext_if }

# --- Control Plane ---

# SSH from trusted sources only
pass in quick on $ext_if proto tcp from <trusted_v4> to ($ext_if) port 22 \
    flags S/SA keep state \
    (max-src-conn 5, max-src-conn-rate 3/30, \
     overload <bruteforce> flush global)
pass in quick on $ext_if proto tcp from <trusted_v6> to ($ext_if) port 22 \
    flags S/SA keep state \
    (max-src-conn 5, max-src-conn-rate 3/30, \
     overload <bruteforce> flush global)

# BGP (TCP 179) - strictly limited to known peers
pass in quick on $ext_if proto tcp from <bgp_peers_v4> to ($ext_if) port 179 \
    flags S/SA keep state
pass in quick on $ext_if proto tcp from <bgp_peers_v6> to ($ext_if) port 179 \
    flags S/SA keep state

# GRE tunnel from iFog
pass in quick on $ext_if proto gre from $ifog_gre_endpoint to ($ext_if)
pass in quick on gre0 proto tcp from $ifog_bgp_peer to any port 179

# ICMPv6: essential for NDP, PMTUD, and diagnostics
pass in quick inet6 proto ipv6-icmp icmp6-type { \
    echoreq, echorep, neighbrsol, neighbradv, \
    toobig, timex, paramprob, routersol }
pass in quick inet proto icmp icmp-type { echoreq, unreach, timex }

# --- Data Plane ---

# Inbound traffic destined for our prefix
pass in quick on $ext_if inet6 from any to $my_network_v6 keep state
pass in quick on gre0 inet6 from any to $my_network_v6 keep state

# Return traffic from downstream tunnels
pass in quick on $vps_tun inet6 from $my_network_v6 to any keep state
pass in quick on $dc_tun inet6 from $my_network_v6 to any keep state

# GIF tunnel encapsulation (proto 41) from downstream endpoints
pass in quick on $ext_if proto 41 from $vps_v4 to ($ext_if)

# Outbound
pass out quick all keep state

The firewall cleanly separates control plane (SSH, BGP sessions) from data plane (forwarded traffic). The control plane rules are strict: BGP is locked to known peer addresses, SSH to trusted management IPs. The data plane rules are simpler since the router just needs to forward packets between upstreams and downstream tunnels.

The block in quick on $ext_if from { <bogons>, $my_network_v6 } rule is important - it drops packets claiming to come from our own prefix arriving on the external interface. If someone on the internet spoofs a source address from our range, this catches it before it enters the forwarding path.

Note the per-tunnel MSS clamping in the scrub section. Each tunnel has different overhead (GRE adds more headers than GIF), so the MSS values differ. Getting this wrong causes mysterious connection stalls with large packets.

The Downstream Server: Dual-Stack with Policy Routing

This is where things get interesting. The VPS (vps01) already has provider-assigned IPv6 from its hoster. Jails on this server use addresses from both address spaces:

Provider IPv6 (2001:db8:200:0:1000::/68) - the hoster’s addresses, NATed to the host
BGP IPv6 (2a06:9801:1c:1000::/64) - our own prefix, routed natively via the GIF tunnel
Private IPv4 (10.254.254.0/24) - NATed to the host’s public IPv4

The challenge: when a jail sends traffic from its BGP address (2a06:…), that traffic must exit through the GIF tunnel to the BGP router - not through the default route to the VPS provider, where it would be dropped as spoofed. But traffic from the provider address must continue using the normal default route.

The solution is dual-FIB policy routing - FreeBSD’s implementation of multiple routing tables.

How Dual-FIB Works

FreeBSD supports multiple routing tables called FIBs (Forwarding Information Bases). Each FIB is an independent routing table with its own default route and entries. Interfaces and PF rules can assign traffic to a specific FIB, and the kernel consults the appropriate table when forwarding.

FIB 0 (default):
  default --> vtnet0 --> VPS provider upstream
  Used by: host traffic, provider-addressed jail traffic

FIB 1:
  default --> gif0 --> BGP router (router01)
  Used by: BGP-addressed jail traffic (2a06:9801:1c::/48)

Network Configuration

Here’s the relevant portion of the server’s /etc/rc.conf:

hostname="vps01.example.com"

kern_securelevel_enable="YES"
kern_securelevel="2"

# Primary interface - provider IPv4 and IPv6
ifconfig_vtnet0="inet 203.0.113.10 netmask 255.255.252.0 -lro -tso"
ifconfig_vtnet0_ipv6="inet6 2001:db8:200::2 prefixlen 68"
defaultrouter="203.0.113.1"
ipv6_defaultrouter="fe80::1%vtnet0"

# Jail bridge - three address spaces
cloned_interfaces="bridge0 gif0"
ifconfig_bridge0_name="bastille0"
ifconfig_bastille0="inet 10.254.254.1/24"
ifconfig_bastille0_ipv6="inet6 2001:db8:200:0:1000::1 prefixlen 68"
ifconfig_bastille0_alias0="inet6 2a06:9801:1c:1000::1 prefixlen 64"

# GIF tunnel to BGP router - assigned to FIB 1
ifconfig_gif0="fib 1 tunnel 203.0.113.10 198.51.100.10 tunnelfib 0"
ifconfig_gif0_ipv6="inet6 2a06:9801:1c:ffff::2 2a06:9801:1c:ffff::1 prefixlen 128"

# Enable forwarding
gateway_enable="YES"
ipv6_gateway_enable="YES"

# FIB 1 routing table entries
static_routes="fib1default jailleak bgplink"
route_fib1default="-6 default -interface gif0 -fib 1"
route_jailleak="-6 2001:db8:200:0:1000::/68 -interface bastille0 -fib 1"
route_bgplink="-6 2a06:9801:1c:1000::/64 -interface bastille0 -fib 1"

The GIF tunnel configuration deserves a closer look:

ifconfig_gif0="fib 1 tunnel 203.0.113.10 198.51.100.10 tunnelfib 0"

This single line contains two critical directives:

fib 1: The tunnel interface itself lives in FIB 1. Traffic arriving on gif0 and traffic routed out gif0 consults routing table 1.
tunnelfib 0: But the outer IPv4 encapsulation (the 203.0.113.10 —> 198.51.100.10 wrapper) uses FIB 0. This is essential - the IPv4 path to the BGP router goes through the provider’s default route in FIB 0. Without tunnelfib 0, the encapsulated packets would try to use FIB 1’s default route (which points at gif0 itself), creating a recursive loop.

The three static routes in FIB 1 complete the picture:

fib1default: Default route in FIB 1 exits through gif0 to the BGP router
jailleak: Tells FIB 1 that the provider’s jail subnet is reachable via bastille0 (without this, return traffic in FIB 1 for jails’ provider addresses would try to exit through gif0)
bgplink: Same for the BGP jail subnet - FIB 1 needs to know these addresses are local on bastille0

PF: The Routing Glue

PF is where the address-based routing decision happens. When a jail sends a packet from a BGP address, PF assigns it to FIB 1:

# BGP-addressed jail traffic --> force into routing table 1 (exits via gif0)
pass in quick on bastille0 inet6 from $bgp_net to any rtable 1 keep state

The rtable 1 directive is the key. It tells PF to route matching packets using FIB 1 instead of the default FIB 0. Since FIB 1’s default route points out gif0 to the BGP router, these packets get encapsulated and sent to router01, which then forwards them to the internet with the correct source address.

For traffic arriving on the tunnel destined for jails, PF uses reply-to to ensure return traffic takes the same path:

# Inbound BGP traffic - reply-to ensures responses exit via gif0
pass in quick on $tun_if reply-to ($tun_if $bgp_hub_ip) inet6 \
    from any to $bgp_net keep state

# BGP ICMPv6 - also needs reply-to for correct return path
pass in quick on $tun_if reply-to ($tun_if $bgp_hub_ip) inet6 proto ipv6-icmp \
    from any to $bgp_net \
    icmp6-type { echoreq, echorep, toobig, timex, paramprob } \
    keep state

Without reply-to, the kernel would consult FIB 0 for return traffic (since the jail itself isn’t in FIB 1), and replies to BGP-addressed connections would exit through vtnet0 with the wrong source routing - getting dropped as spoofed by the provider. The reply-to directive forces PF to send reply packets back out the interface they arrived on, to the specified next-hop.

The Complete Picture

Here’s how a request to a BGP-addressed jail service flows:

 1. Client sends packet to 2a06:9801:1c:1000::10 (web jail)
 2. Packet traverses the internet, reaching AS201379 via iFog or Lagrange
 3. router01 forwards it through gif0 tunnel to vps01
 4. vps01 receives proto 41 on vtnet0, decapsulates --> gif0
 5. PF matches: reply-to ($tun_if $bgp_hub_ip), creates state
 6. Packet forwarded to bastille0 --> jail
 7. Jail responds, packet exits on bastille0
 8. PF's state table triggers reply-to: send via gif0 to bgp_hub_ip
 9. gif0 encapsulates (proto 41) using FIB 0 to reach router01
10. router01 receives, forwards to upstream --> internet --> client

And for outbound connections initiated by the jail using its BGP address:

1. Jail sends packet from 2a06:9801:1c:1000::10
2. Packet arrives on bastille0
3. PF matches: "from $bgp_net --> rtable 1"
4. Kernel routes via FIB 1 --> default route --> gif0
5. gif0 encapsulates using FIB 0 --> vtnet0 --> router01
6. router01 receives, forwards to internet (source: 2a06:9801:1c:1000::10)

Meanwhile, the exact same jail can communicate using its provider address through the normal default route in FIB 0, with NAT to the host’s address. Both address spaces coexist on the same interface, differentiated purely by PF rules and FIB selection.

Verification

Once everything is running, verification is straightforward. From inside a jail with both addresses:

# Traffic from the provider address - NATed through the hoster
root@caddy:~ # curl --interface 2001:db8:200:0:1000::10 https://ifconfig.co
2001:db8:200::2

# Traffic from the BGP address - routed natively through the tunnel
root@caddy:~ # curl --interface 2a06:9801:1c:1000::10 https://ifconfig.co
2a06:9801:1c:1000::10

The first request shows the host’s NATed provider address. The second shows the jail’s real BGP address - confirming the packet traversed the tunnel and reached the internet through AS201379.

A traceroute from an external host confirms the BGP path is working:

$ mtr -rw 2a06:9801:1c:1000::10
HOST:                              Loss%   Snt   Last   Avg  Best  Wrst StDev
 1.|-- [local-gateway]               0.0%    10    2.6   5.5   2.6  14.1   3.6
    ...
 9.|-- [transit-provider-edge]       0.0%    10   33.8  46.2  33.8  81.2  19.0
10.|-- [ifog-peering-fabric]         0.0%    10   33.5  46.7  33.5  87.0  18.6
11.|-- 2001:db8:300::2               0.0%    10   44.3  59.1  41.9 136.7  33.2
12.|-- 2a06:9801:1c:ffff::2         0.0%    10   72.7  98.9  68.8 198.8  42.4
13.|-- 2a06:9801:1c:1000::10        0.0%    10  164.1  83.1  63.5 164.1  33.4

Traffic enters via the transit provider (hops 9-10), traverses the GRE tunnel to router01 (hop 11), then the GIF tunnel to vps01 (hop 12, the 2a06:9801:1c:ffff::2 link address), and finally reaches the jail (hop 13). The prefix also shows up correctly on bgp.tools as active and originated by AS201379 with both upstreams visible.

Lessons Learned

MSS clamping is non-negotiable with tunnels. Every layer of encapsulation eats into the MTU. GIF adds 20 bytes (IPv4 header) to every packet. GRE adds more. If you don’t clamp the TCP MSS, large packets get fragmented or dropped, causing mysterious failures where small requests work but large transfers stall. Set max-mss in PF’s scrub rules for every tunnel interface, calculated as: MTU minus IPv6 header (40 bytes) minus TCP header (20 bytes).

FIB separation is cleaner than source-based routing hacks. FreeBSD’s multi-FIB support is a first-class feature. Using rtable in PF and fib/tunnelfib on interfaces gives you full control over which routing table handles which traffic. It’s conceptually cleaner and more debuggable than alternatives like ip6tables MARK targets on Linux.

Bogon filtering matters even for small networks. The internet is full of misconfigurations and occasional malice. Filtering inbound routes prevents your router from accepting nonsense that could black-hole traffic or worse. The cost is a few lines of configuration; the protection is real.

reply-to solves asymmetric routing. When traffic can arrive on multiple interfaces, the kernel’s default FIB selection for return traffic may choose the wrong path. PF’s reply-to directive forces replies back out the arrival interface, which is exactly what you need for tunnel overlay setups.

Start with two upstreams. A single upstream means zero redundancy and no ability to do traffic engineering. Two upstreams give you failover and the ability to prefer one path over the other using AS-path prepending or communities. The operational complexity increase is minimal.

Conclusion

Running your own AS on the internet is more accessible than most people assume. The barrier isn’t technical complexity - it’s knowing that the option exists. A FreeBSD VM, FRR, a couple of tunnels, and some careful PF rules give you provider-independent addressing, real BGP peering, and a deeper understanding of how the internet actually works.

The dual-FIB approach on the downstream server is the piece I’m most satisfied with. It elegantly solves the “two address spaces, one server” problem without hacks: BGP traffic takes the tunnel, provider traffic takes the default route, and PF’s rtable directive makes the decision based purely on source address. Both paths coexist transparently, and the jails don’t need to know anything about the routing underneath.

Is it overkill for a blog? Absolutely. But the same infrastructure carries every service I run, and having addresses that survive provider migrations has already paid for itself in operational simplicity. Besides, there’s something deeply satisfying about seeing your own AS number show up in a traceroute.

References

RIPE NCC - Requesting Resources
FRR Documentation
FreeBSD Handbook: Firewalls (PF)
FreeBSD setfib(1)
bgp.tools - BGP looking glass and analytics
RIPE RPKI Documentation
RFC 5082 - GTSM (TTL Security)

The internet is a network of networks, and now you’re one of them. There’s a certain elegance in participating in the same routing protocol that glues together every network on the planet - from your single /48 all the way up to the Tier 1 carriers. BGP doesn’t care about your size. It just cares that your routes are valid, your filters are clean, and your packets know where to go.

PF Firewall on FreeBSD: A Practical Guide

2026-02-06T00:00:00+01:00

PF (Packet Filter) is one of the most elegant firewall systems available on any operating system. Originally developed for OpenBSD and ported to FreeBSD, it combines a clean configuration syntax with powerful capabilities for filtering, NAT, traffic shaping, and logging. After running PF across multiple FreeBSD servers for years, I’ve developed a consistent configuration pattern that balances security with practicality.

This guide covers everything from basic concepts to production configurations, with tips and patterns I’ve refined through real-world deployment. Whether you’re protecting a single server or a complex jail infrastructure, the principles here should give you a solid foundation.

Core Concepts

PF processes rules in order, but with an important twist: the last matching rule wins (unless you use quick). This means you typically structure rules from general to specific, with explicit blocks at the top and specific allows below. The quick keyword short-circuits evaluation - when a quick rule matches, processing stops immediately.

The configuration file lives at /etc/pf.conf. After editing, validate syntax with:

pfctl -nf /etc/pf.conf

If validation passes, load the rules:

pfctl -f /etc/pf.conf

To enable PF at boot, add to /etc/rc.conf:

pf_enable="YES"
pflog_enable="YES"

Since this guide covers NAT and routing traffic between jails and the internet, you must also enable packet forwarding. Without this, the firewall rules will load but traffic will die at the interface:

gateway_enable="YES"
ipv6_gateway_enable="YES"

Anatomy of a PF Configuration

A well-organized pf.conf follows a predictable structure. Here’s the skeleton I use across all servers:

# --- Macros ---
# Named variables for interfaces and addresses

# --- Tables ---
# Dynamic lists of addresses

# --- Options ---
# Global behavior settings

# --- Scrub ---
# Packet normalization

# --- NAT ---
# Network address translation

# --- RDR ---
# Port redirections

# --- Filtering ---
# The actual firewall rules

Each section has a distinct purpose. Let’s walk through them with practical examples.

Macros: Naming Your Network

Macros make configurations readable and maintainable. Define them at the top:

# --- Macros ---
ext_if = "vtnet0"

# "bastille0" is the bridge interface created by the Bastille framework.
# If using iocage or raw jails, this might be "lo1" or "bridge0".
int_if = "bastille0"

jail_net = "10.100.0.0/24"
jail_net6 = "2001:db8:1000:8000::/65"

# For IPv6 NAT (if needed), use a single address, not a CIDR block
host_ipv6 = "2001:db8:1000::1"

web_jail_v4 = "10.100.0.10"
web_jail_v6 = "2001:db8:1000:8000::10"
db_jail_v4 = "10.100.0.20"

# Lists use braces
trusted_ipv4 = "{ 198.51.100.22, 203.0.113.50 }"
trusted_ipv6 = "{ 2001:db8:ffff::/48 }"

Using macros means changing an IP address requires editing one line, not hunting through the entire ruleset. The $macro syntax references them in rules.

Tables: Dynamic Address Lists

Tables are PF’s mechanism for large or changing address sets. Unlike macros, tables can be modified at runtime without reloading the entire ruleset:

# --- Tables ---
table <bruteforce> persist
table <jails_v4> { $jail_net }
table <jails_v6> { $jail_net6 }
table <trusted_v4> persist { $trusted_ipv4 }
table <trusted_v6> persist { $trusted_ipv6 }

The persist keyword keeps the table in memory even when empty. This is essential for tables populated dynamically (like <bruteforce>).

Manage tables at runtime:

# Show table contents
pfctl -t bruteforce -T show

# Add an address
pfctl -t bruteforce -T add 192.0.2.100

# Remove an address
pfctl -t bruteforce -T delete 192.0.2.100

# Flush entire table
pfctl -t bruteforce -T flush

# Load from file
pfctl -t trusted_v4 -T add -f /etc/pf.trusted.txt

Tables scale efficiently - PF uses radix trees internally, so even tables with hundreds of thousands of entries (like GeoIP databases) perform well.

Options: Global Behavior

The options section configures PF’s global behavior:

# --- Options ---
set skip on lo0
set block-policy drop
set loginterface $ext_if

Key options explained:

set skip on lo0: Never filter loopback traffic. Critical for local services.
set block-policy drop: Silently drop blocked packets (vs. return which sends RST/ICMP). Drop is better for security - it gives attackers no information.
set loginterface: Enable per-interface packet/byte counters. View with pfctl -si.

For large tables (GeoIP filtering, for example), increase the table entry limit:

set limit table-entries 1000000

Scrub: Packet Normalization

Scrubbing normalizes packets to prevent various attacks and fix fragmentation issues:

# --- Scrub ---
scrub in all fragment reassemble
scrub out all random-id max-mss 1500

The fragment reassemble directive reassembles fragmented packets before filtering - essential because fragments can be used to evade stateless filters. The random-id directive randomizes IP IDs on outbound packets, making traffic analysis harder. The max-mss clamps TCP MSS to prevent fragmentation issues, particularly important for tunneled traffic or VPN scenarios.

For specific interfaces with lower MTUs (tunnels, VSwitch connections), add targeted scrub rules:

scrub out on gif0 max-mss 1240
scrub out on $int_if max-mss 1410

NAT: Address Translation

NAT translates private addresses to public ones for outbound traffic. With FreeBSD jails on private networks, NAT is essential for IPv4:

# --- NAT ---
nat on $ext_if inet from <jails_v4> to any -> ($ext_if)

The parentheses around ($ext_if) make the NAT rule dynamic - if the external interface’s IP changes (DHCP), PF automatically uses the new address.

For IPv6, if you have a routed prefix, skip NAT entirely and route natively - this is the recommended approach. Jails get real global addresses and IPv6 works end-to-end as designed.

If you must NAT IPv6 (not recommended, but sometimes necessary), the syntax is similar. Ensure the target is a single address, not a CIDR block:

# host_ipv6 must be a single address assigned to the external interface
nat on $ext_if inet6 from <jails_v6> to any -> $host_ipv6

RDR: Port Forwarding

Redirections forward incoming traffic to internal hosts. For a web server jail:

# --- RDR ---
rdr pass on $ext_if inet proto tcp to ($ext_if) port {80, 443} -> $web_jail_v4
rdr pass on $ext_if inet6 proto tcp to $host_ipv6 port {80, 443} -> $web_jail_v6

The rdr pass syntax combines redirection with an implicit pass rule. Without pass, you’d need a separate filter rule for the translated traffic.

For port translation (external port differs from internal):

# External 2222 -> internal 22
rdr pass on $ext_if inet proto tcp from $trusted_runner to ($ext_if) port 2222 -> $deploy_jail port 22

Restricting the source in RDR rules is a powerful pattern - the service is invisible to everyone except specified sources.

Filtering: The Ruleset Core

The filtering section is where access control happens. Start with default deny:

# --- Filtering ---
# Block known bad actors immediately
block quick from <bruteforce>

# Default deny everything
block drop in log all
block drop out log all

# Allow all established connections out
pass out quick all keep state

# Anti-spoofing
antispoof quick for { $ext_if, bastille0 }

Understanding Quick and State

The quick keyword stops rule processing on match. Use it for definitive decisions:

# Immediately drop known attackers - no further processing
block quick from <bruteforce>

The keep state directive enables stateful filtering. When an outbound connection is allowed, PF creates a state entry that automatically permits return traffic. This is why a single pass out rule handles both the outbound packet and inbound replies.

Note: While newer OpenBSD versions of PF make keep state implicit, FreeBSD’s PF is based on an older codebase. Being explicit with keep state ensures connection tracking works exactly as intended across FreeBSD versions.

SSH Protection Pattern

Protecting SSH deserves special attention. This pattern restricts access to trusted sources, and automatically blocks brute-force attempts:

# SSH only from trusted sources
pass in quick on $ext_if proto tcp from <trusted_v4> to any port 22 \
    flags S/SA keep state \
    (max-src-conn 5, max-src-conn-rate 3/30, \
     overload <bruteforce> flush global)

pass in quick on $ext_if inet6 proto tcp from <trusted_v6> to any port 22 \
    flags S/SA keep state \
    (max-src-conn 5, max-src-conn-rate 3/30, \
     overload <bruteforce> flush global)

# Block and log all other SSH attempts (for monitoring)
block in log quick on $ext_if proto tcp from any to any port 22 label "ssh_blocked"

The state options deserve explanation:

max-src-conn 5: Maximum 5 simultaneous connections per source IP
max-src-conn-rate 3/30: Maximum 3 new connections per 30 seconds per source
overload <bruteforce>: Add violators to the bruteforce table
flush global: Kill all existing connections from the offender

The label on the block rule tags logged packets for easy filtering.

Essential ICMP

Never block all ICMP - it breaks essential network functionality:

# Essential ICMPv6 (required for IPv6 to function)
pass in quick inet6 proto ipv6-icmp icmp6-type { \
    echoreq, echorep, neighbrsol, neighbradv, \
    toobig, timex, paramprob }

# Useful ICMPv4
pass in inet proto icmp icmp-type { echoreq, unreach }

The IPv6 types are critical: - neighbrsol/neighbradv: Neighbor Discovery (IPv6’s ARP equivalent) - toobig: Path MTU Discovery (breaks connections if blocked) - echoreq/echorep: Ping (useful for diagnostics)

Jail Egress Rules

Allow jails to reach the internet while preventing them from directly accessing other internal networks:

# Jails can reach anywhere except internal networks
pass in quick on bastille0 from <jails_v4> to ! 10.100.0.0/24 keep state
pass in quick on bastille0 inet6 from <jails_v6> to ! 2001:db8:1000:8000::/65 keep state

The ! negation ensures jails can reach the internet but can’t directly contact other jails (unless explicitly permitted elsewhere).

Inter-Jail Communication

When jails need to communicate (database connections, for example), add explicit rules:

# Web jail -> Database jail (PostgreSQL + Redis)
pass in quick on bastille0 proto tcp from $web_jail_v4 to $db_jail_v4 port { 5432, 6379 } keep state

This explicit allowlisting is preferable to permitting all inter-jail traffic - it documents and enforces your service architecture.

Complete Example Configuration

Here’s a production-ready configuration for a server running multiple jails:

# --- Macros ---
ext_if = "vtnet0"

# Jail bridge interface (bastille0 for Bastille, bridge0 or lo1 for others)
int_if = "bastille0"

jail_net = "10.100.0.0/24"
jail_net6 = "2001:db8:1000:8000::/65"
host_ipv6 = "2001:db8:1000::1"

web_v4 = "10.100.0.10"
web_v6 = "2001:db8:1000:8000::10"
db_v4 = "10.100.0.20"
deploy_v4 = "10.100.0.25"

trusted_ipv4 = "{ 198.51.100.22, 203.0.113.50 }"
trusted_ipv6 = "{ 2001:db8:ffff::/48 }"
deploy_runner = "192.0.2.100"

# --- Tables ---
table <bruteforce> persist
table <jails_v4> { $jail_net }
table <jails_v6> { $jail_net6 }
table <trusted_v4> persist { $trusted_ipv4 }
table <trusted_v6> persist { $trusted_ipv6 }

# --- Options ---
set skip on lo0
set block-policy drop
set loginterface $ext_if

# --- Scrub ---
scrub in all fragment reassemble
scrub out all random-id max-mss 1500

# --- NAT ---
nat on $ext_if inet from <jails_v4> to any -> ($ext_if)

# --- RDR ---
rdr pass on $ext_if inet proto tcp to ($ext_if) port {80, 443} -> $web_v4
rdr pass on $ext_if inet6 proto tcp to $host_ipv6 port {80, 443} -> $web_v6

# Deployment SSH from CI runner only
rdr pass on $ext_if inet proto tcp from $deploy_runner to ($ext_if) port 2222 -> $deploy_v4 port 22

# --- Filtering ---
block quick from <bruteforce>
block drop in log all
block drop out log all

pass out quick all keep state
antispoof quick for { $ext_if, bastille0 }

# Management SSH
pass in quick on $ext_if proto tcp from <trusted_v4> to any port 22 \
    flags S/SA keep state \
    (max-src-conn 5, max-src-conn-rate 3/30, \
     overload <bruteforce> flush global)

pass in quick on $ext_if inet6 proto tcp from <trusted_v6> to any port 22 \
    flags S/SA keep state \
    (max-src-conn 5, max-src-conn-rate 3/30, \
     overload <bruteforce> flush global)

block in log quick on $ext_if proto tcp from any to any port 22 label "ssh_blocked"

# Essential ICMP
pass in quick inet6 proto ipv6-icmp icmp6-type { \
    echoreq, echorep, neighbrsol, neighbradv, toobig, timex, paramprob }
pass in inet proto icmp icmp-type { echoreq, unreach }

# Web jail public access
pass in quick on $ext_if inet proto tcp from any to $web_v4 port {80, 443} flags S/SA keep state
pass in quick on $ext_if inet6 proto tcp from any to $web_v6 port {80, 443} flags S/SA keep state

# Inter-jail: web -> database
pass in quick on bastille0 proto tcp from $web_v4 to $db_v4 port { 5432, 6379 } keep state

# Jail egress
pass in quick on bastille0 from <jails_v4> to ! 10.100.0.0/24 keep state
pass in quick on bastille0 inet6 from <jails_v6> to ! 2001:db8:1000:8000::/65 keep state

Tips and Tricks

Viewing Loaded Rules

# Show all rules with rule numbers
pfctl -sr

# Verbose output with statistics
pfctl -sr -v

# Show NAT rules
pfctl -sn

# Show state table
pfctl -ss

# Show interface statistics
pfctl -si

Logging and Debugging

PF logs to pflog0, viewable with tcpdump:

# Live log viewing
tcpdump -n -e -ttt -i pflog0

# Read saved log file
tcpdump -n -e -ttt -r /var/log/pflog

# Filter by label
tcpdump -n -e -ttt -i pflog0 'label ssh_blocked'

Enable pflog in rc.conf and optionally configure rotation in /etc/newsyslog.conf.

Clearing Brute-Force Entries

Sometimes legitimate users get caught in rate limiting:

# Show who's blocked
pfctl -t bruteforce -T show

# Unblock specific IP
pfctl -t bruteforce -T delete 198.51.100.50

# Flush all blocked IPs
pfctl -t bruteforce -T flush

Consider a cron job to expire old entries:

# Clear entries older than 24 hours
pfctl -t bruteforce -T expire 86400

Testing Changes Safely

Before applying potentially locking changes, use the classic sysadmin safety net:

# Validate syntax first
pfctl -nf /etc/pf.conf

# Apply with automatic rollback
# (If you get locked out, rules revert after timeout)
echo "pfctl -f /etc/pf.conf.backup" | at now + 5 minutes
pfctl -f /etc/pf.conf
# If everything works, cancel the at job
atq
atrm <job_number>

Note: The at command requires the atd daemon. On minimal installs, you may need to enable it first:

# In /etc/rc.conf
atd_enable="YES"

# Then start it
service atd start

Monitoring Specific Traffic

Add temporary rules with counters for debugging:

pass in log on $ext_if proto tcp to port 443 label "https_debug"

Then watch:

pfctl -sr -v | grep https_debug

For bastion hosts or jump servers, authpf provides per-user firewall rules that activate upon SSH login. Instead of static rules, the firewall dynamically permits access based on who’s connected.

authpf works by loading a user-specific ruleset (from /etc/authpf/users/$USER/ or /etc/authpf/authpf.rules) when the user logs in via SSH. When they log out, the rules are removed. This creates a “knock first” security model where services are invisible until authenticated.

Basic setup:

Set the user’s shell to /usr/sbin/authpf: bash pw usermod bastion_user -s /usr/sbin/authpf
Create /etc/authpf/authpf.rules with rules to load on login: pf pass in quick on $ext_if from $user_ip to $internal_net
The macro $user_ip is automatically set to the connecting client’s address.

This approach is powerful for restricting access to internal networks: users must SSH to the bastion first, then their IP gets temporary firewall access. When the SSH session ends, access is revoked. Combined with short-lived certificates or hardware tokens, authpf creates a robust zero-trust perimeter without VPN complexity.

For production deployments, consider authpf as an alternative to VPNs for administrative access - it’s simpler to audit, integrates with existing SSH infrastructure, and provides per-session access control.

Conclusion

PF rewards careful configuration with robust, maintainable security. The patterns here - default deny, explicit allowlists, brute-force protection, and clean macro organization - scale from single servers to complex multi-jail deployments.

The key principles worth remembering:

Default deny with explicit allows: Never assume traffic is safe
State tracking: Let PF manage connection state rather than permitting return traffic explicitly
Tables for dynamic data: Use tables for address lists that change or grow large
Log selectively: Log blocked traffic for analysis, but avoid logging high-volume allowed traffic
Test before applying: Always validate syntax and have a rollback plan

PF’s syntax takes time to internalize, but once it clicks, writing firewall rules becomes almost intuitive. The investment pays dividends in security, clarity, and the confidence that comes from truly understanding what your firewall permits.

References

PF - The OpenBSD Packet Filter (PF’s canonical documentation)
FreeBSD Handbook: Firewalls
FreeBSD pf.conf(5) man page
authpf(8) man page

A good firewall is invisible when everything works and invaluable when something goes wrong. PF sits quietly in the kernel, inspecting every packet, blocking the noise, and letting through exactly what you’ve specified. It’s one of those tools that makes you appreciate the Unix philosophy: do one thing well, make it composable, keep it simple.

Immutable Linux Desktops: Universal Blue, OSTree, and the Future of Desktop Linux

2026-01-26T00:00:00+01:00

Traditional Linux desktop installations accumulate state over time. Package upgrades modify system files in place, configuration changes drift from defaults, and after a few years, systems become unique snowflakes that are difficult to reproduce or repair. Atomic desktops take a fundamentally different approach: the operating system is treated as a single, versioned image that can be deployed, rolled back, or replaced entirely.

This model has proven itself in server infrastructure through CoreOS and similar projects. Now it’s coming to the desktop through Fedora’s atomic variants and, more ambitiously, through Universal Blue’s community-built images like Aurora and Bazzite.

The Foundation: OSTree

At the heart of atomic Linux systems lies OSTree (also known as libostree), a content-addressed filesystem designed to store and deploy complete bootable operating system trees. Think of it as “git for operating systems”. Though the comparison is imperfect, it captures the essential concept.

OSTree stores filesystem trees as immutable, deduplicated objects identified by their content hash. When you deploy a new system version, OSTree:

Downloads only the changed objects (delta updates)
Hardlinks unchanged files from the existing deployment
Creates a new boot entry pointing to the new tree
Keeps the previous deployment intact for rollback

The result is atomic updates - the system either fully transitions to the new state or remains unchanged. No more half-completed upgrades leaving your system in an inconsistent state. No more “let me try rebooting again” after a failed update.

$ sudo rpm-ostree status
State: idle
Deployments:
  fedora:fedora/43/x86_64/silverblue
                  Version: 43.20260126.0 (2026-01-26T00:26:25Z)
                   Commit: e36246b0837cfe0d1c09e3ee60205ed5be57d57260adc10a625f9353bbef3c40
             GPGSignature: Valid signature by C6E7F081CF80E13146676E88829B606631645531
                     Diff: 522 upgraded, 11 removed, 26 added

● fedora:fedora/43/x86_64/silverblue
                  Version: 43.1.6 (2025-10-23T03:11:18Z)
                   Commit: 4d40d281be93a88f3d559b5756df602f454f932f3c809a6a4250b91049ce40e8
             GPGSignature: Valid signature by C6E7F081CF80E13146676E88829B606631645531

The bullet point indicates your current deployment. The previous deployment remains available - a single rpm-ostree rollback and reboot returns you to the exact state you left.

rpm-ostree: Hybrid Package Management

While pure OSTree deployments are completely immutable, desktop users need flexibility. Enter rpm-ostree, which provides a hybrid model: you get the reliability of image-based deployments while retaining the ability to layer additional RPM packages.

The base system remains a verified, signed image from your distribution. On top of this, rpm-ostree allows you to:

Layer packages: Install additional RPMs that persist across updates
Override packages: Replace base system packages with different versions
Remove packages: Exclude unwanted components from the base image

# Layer a package on top of the base image
rpm-ostree install htop neovim

# Override a base package with a newer version
rpm-ostree override replace ./updated-package.rpm

# Remove a base package you don't need
rpm-ostree override remove firefox

Each modification creates a new deployment, leaving the previous state intact. Updates from upstream are rebased on top of your layered packages, maintaining a coherent system state.

The key insight: your customizations are explicit and tracked. Unlike traditional package managers where the system state is the accumulated result of thousands of operations over time, rpm-ostree maintains a clear separation between the base image and your modifications.

bootc: Container-Native Systems

The current standard in this space is bootc, which takes the logical next step: if containers are the standard way to build and distribute software, why not use them for entire operating systems?

bootc systems are built as standard OCI container images. The same tools used for application containers - Podman, Docker, Buildah - can build bootable system images. The same registries that host application images can host operating system images. The entire container ecosystem becomes available for OS development.

FROM quay.io/fedora/fedora-bootc:43

# Install additional packages
RUN dnf install -y htop neovim && dnf clean all

# Add custom configuration
COPY custom-config.conf /etc/myapp/

# Standard container build produces a bootable system

Building this Containerfile produces a complete, bootable operating system image. You can test it in a VM, push it to a registry, and deploy it to physical hardware. The same image works everywhere.

bootc uses OSTree under the hood for the actual deployment mechanics, but the authoring experience shifts entirely to container workflows. For organizations already invested in container tooling, this dramatically simplifies system management.

Current state: bootc is production-ready for servers and is making rapid progress on desktop support. Universal Blue’s images are already built using container workflows, positioning them well for the full bootc transition.

Fedora Atomic Desktops

Fedora provides official atomic desktop variants, each tailored to a specific desktop environment:

Variant	Desktop Environment	Target Users
Fedora Silverblue	GNOME	General desktop users
Fedora Kinoite	KDE Plasma	KDE enthusiasts
Fedora Sway Atomic	Sway	Tiling WM users
Fedora Budgie Atomic	Budgie	Budgie fans

These provide the foundation: a stable, tested base with Fedora’s regular release cadence. They’re excellent choices for users who want the atomic model with minimal deviation from upstream Fedora.

However, Fedora’s official images are intentionally conservative. They include only what’s necessary for a functional desktop, leaving additional customization to the user through package layering.

Universal Blue: Opinionated Images Done Right

Universal Blue takes the Fedora atomic base and builds upon it with thoughtful defaults, additional hardware support, and curated software selections. Rather than starting from scratch, they leverage Fedora’s extensive testing and add value through careful curation.

The project produces several image variants:

Aurora

Aurora is Universal Blue’s flagship desktop image, based on Fedora Kinoite (KDE Plasma). It targets users who want a polished, productive desktop without manual configuration:

Enhanced hardware support: Additional firmware, codecs, and drivers included (UBlue images come with Nvidia drivers pre-installed and signed)
Developer tooling: Distrobox pre-configured for containerized development environments
Quality-of-life improvements: Thoughtful defaults and missing pieces filled in
Automatic updates: System images update in the background and apply on reboot

Aurora represents what a modern Linux desktop can be when someone with taste makes the integration decisions. The KDE Plasma experience feels cohesive and complete.

Bazzite

Bazzite targets gaming and HTPC use cases, building on the same solid foundation with gaming-specific optimizations:

Steam and Lutris: Pre-installed and configured
Graphics drivers: Latest Mesa, Vulkan support, and vendor-specific drivers
Controller support: Out-of-box recognition for a wide range of game controllers
Performance tuning: Kernel parameters and system settings optimized for gaming
Steam Deck compatibility: Specific images for Steam Deck and similar handhelds
HTPC mode: Console-like experience for living room setups

Bazzite solves the perennial “gaming on Linux” setup problem. Instead of spending an afternoon configuring drivers, Proton, and various compatibility layers, you boot into a system where everything already works.

Other Variants

Universal Blue also produces:

Bluefin: GNOME-based developer workstation image
uCore: Server-focused images based on Fedora CoreOS
Framework-specific images: Optimized for Framework laptop hardware

The Advantages of Atomic Desktops

Reliability and Stability

The most significant advantage is predictability. Every user running Aurora 43.20260126 has exactly the same base system. Support becomes tractable - you’re not debugging the unique accumulated state of someone’s five-year-old installation.

Updates are atomic: they either complete successfully or don’t happen at all. The half-updated, inconsistent states that sometimes plague traditional distributions simply cannot occur.

Painless Rollback

Every update preserves the previous system state. If an update causes problems:

# Rollback to the previous deployment
rpm-ostree rollback
systemctl reboot

After reboot, you’re running the exact system you had before the update. This isn’t “uninstall the broken package and hope” - it’s a complete restoration of the previous state.

This dramatically lowers the risk of updates. You can confidently run the latest packages knowing that recovery is trivial.

Rebasing: Switch Distributions Instantly

Perhaps the most surprising capability: you can switch between compatible images without reinstalling. Running Aurora but want to try Bazzite for gaming?

# Rebase to Bazzite
rpm-ostree rebase ostree-unverified-registry:ghcr.io/ublue-os/bazzite:latest # (Select the specific tag for your hardware/GPU)
systemctl reboot

After reboot, you’re running Bazzite. Your home directory, user data, and Flatpak applications remain untouched - only the base system changed. Don’t like it? Rebase back to Aurora.

This works across any compatible OSTree-based images. You can move between Fedora’s official images and Universal Blue’s variants freely. The operating system becomes as swappable as a container image - because, fundamentally, that’s what it is.

Separation of Concerns

Atomic desktops enforce a clean separation:

System image: Managed by OSTree, updated atomically, reproducible
Applications: Installed via Flatpak, containerized, independent of system
User data: Lives in your home directory, persists across rebases
Development tools: Run in Distrobox containers, isolated from system

This separation means you can experiment freely with your development environment without risking system stability. Your tooling runs in containers; your OS remains pristine.

Reproducibility

Need to set up a new machine identically to your current one? The process is trivial:

Install the same base image
Apply the same layered packages (if any)
Sync your home directory

There’s no “reinstall all the packages I’ve accumulated over five years” step because your system state is declaratively defined by the image plus your explicit modifications.

Working with Atomic Desktops

Daily Workflow

Day-to-day usage feels identical to traditional Linux. You use your applications, edit documents, browse the web. The atomic nature is invisible until you need it.

Updates happen automatically in the background. When a new image is ready, you’ll see a notification. The update applies on your next reboot - no waiting for package downloads during your workday.

Installing Software

For graphical applications, Flatpak is the primary method:

flatpak install flathub org.mozilla.firefox
flatpak install flathub com.spotify.Client

Flatpak applications are containerized and independent of the base system. They update separately and work across all Linux distributions.

For command-line tools and development environments, Distrobox provides seamless container integration:

# Create a development container
distrobox create --name dev --image fedora:43

# Enter the container - it feels like native
distrobox enter dev

# Inside the container, install whatever you need
sudo dnf install gcc cmake python3-devel

Distrobox containers integrate with your desktop - applications appear in your menu, files are accessible, and the experience is nearly transparent.

Crucially, you can export binaries or GUI apps from the container to the host. This means you can launch tools like VS Code or PyCharm directly from your system dock or application menu, just as if they were installed natively.

When to Layer Packages

Some software legitimately needs system integration that Flatpak can’t provide:

Virtualization: libvirt, QEMU, virt-manager
VPNs: OpenVPN, WireGuard with system integration
System services: Custom daemons that run at boot
Kernel modules: Out-of-tree drivers (though check if the image already includes them)

For these cases, rpm-ostree layering is appropriate:

rpm-ostree install libvirt virt-manager
systemctl reboot

The general guidance: use Flatpak when possible, Distrobox for development, and rpm-ostree layering only when neither alternative works.

Considerations and Trade-offs

Atomic desktops aren’t without trade-offs:

Adjustment Period

Users accustomed to dnf install whatever need to adapt their mental model. The system isn’t broken - it’s intentionally designed to guide you toward better patterns. This adjustment takes a few days for most users.

Reboot Requirements

System changes require a reboot to take effect. While this ensures atomicity, it can feel inconvenient compared to traditional package managers. In practice, most users update infrequently enough that this isn’t burdensome.

Layered Package Limits

While you can layer packages with rpm-ostree, heavy layering defeats the purpose. If you’re layering fifty packages, you’re fighting the model. Consider whether those packages truly need system integration or whether Flatpak/Distrobox would work.

Some Niche Software

Certain applications expect traditional FHS layouts or assume they can modify system files. Most mainstream software works fine, but occasional edge cases exist.

Getting Started

Choosing an Image

For general desktop use with KDE Plasma: Aurora

For gaming or HTPC: Bazzite

For GNOME and developer workflows: Bluefin

For conservative, upstream-focused experience: Fedora Silverblue/Kinoite

Installation

Universal Blue images install like any other Linux distribution. Download the ISO, create a bootable USB, and run the installer. The Anaconda installer handles partitioning, user creation, and initial setup.

Post-installation, enable automatic updates if not already enabled:

sudo systemctl enable --now rpm-ostreed-automatic.timer

First Steps

Explore Flatpak: Visit Flathub and install your essential applications
Set up Distrobox: Create a development container for your programming environment
Configure your desktop: KDE Plasma or GNOME settings work exactly as expected
Verify automatic updates: Check rpm-ostree status to see your current deployment

The Future

The atomic desktop model aligns with where computing is heading. Containers have proven their value for applications; applying the same principles to the operating system itself is the logical evolution.

bootc represents the next major step, unifying system management with container tooling. Universal Blue is already building toward this future while maintaining stability today.

For users tired of nursing aging installations, wondering what broke after the last update, or simply wanting a system that stays out of their way, atomic desktops offer a compelling alternative. They’re not experimental technology - they’re the result of years of refinement in the server space, now polished for desktop use.

The best part: if you don’t like it, you can always rebase back.

References

Universal Blue - Project home
Aurora - KDE-based desktop image
Bazzite - Gaming-focused image
OSTree Documentation
rpm-ostree Documentation
bootc Project
Fedora Atomic Desktops
Flathub - Flatpak application repository
Distrobox - Container integration for development

Thanks to the Universal Blue community for building these excellent images and to the Fedora team for providing the solid foundation they’re built upon.

Integrating FreeBSD 15 with FreeIPA: Native Kerberos and LDAP Authentication

2026-01-25T00:00:00+01:00

Most FreeIPA documentation assumes you’re running Linux and will use ipa-client-install to join hosts to the realm. FreeBSD doesn’t have this luxury. There’s no official IPA client, and the enrollment scripts expect systemd and other Linux-specific components. But that’s not necessarily a disadvantage.

FreeBSD’s native Kerberos implementation and the lightweight nslcd daemon provide everything needed to integrate with FreeIPA cleanly. The result is arguably more elegant than the Linux approach: pure GSSAPI authentication via SSH, LDAP-backed identity lookups, and zero local user management. No SSSD, no realm daemon, no complexity.

This guide details integrating a FreeBSD 15.0 host into a Red Hat Identity Management (IdM) or FreeIPA realm using only native FreeBSD components.

Design Philosophy

The approach here follows what I call “Clean and Sane Engineering”:

Authentication: Pure Kerberos via GSSAPI. Users authenticate with their Kerberos tickets - no passwords transmitted over the network
Identity: LDAP via nslcd for user and group lookups. The NSS layer queries FreeIPA’s LDAP directory transparently
Management: Stateless. No local users created manually. The host is a pure consumer of central identity services

This separation of concerns mirrors how enterprise authentication should work: the KDC handles cryptographic verification, the directory provides attributes, and the local system just needs to know where to ask.

Why Not SSSD?

The Linux world has largely converged on SSSD (System Security Services Daemon) for this kind of integration. It’s a powerful tool, but it’s also a monolith with a sprawling dependency tree, D-Bus requirements, and complexity that feels foreign on a FreeBSD system.

Following Unix philosophy and KISS principles, we actively avoid such heavyweight solutions. Each component in this setup does one thing well: nslcd handles LDAP lookups, the native krb5 library handles Kerberos, PAM modules handle session setup. These are small, focused tools that compose together.

Yes, the initial setup requires understanding how the pieces fit together. But the result is a solution that’s lighter, more transparent, and equally robust. When something breaks, you can debug each component independently. There’s no opaque daemon trying to be clever on your behalf. The configuration files are human-readable, the logs are straightforward, and the failure modes are predictable.

This trade-off, slightly more upfront complexity for long-term simplicity and maintainability, is at the heart of the FreeBSD philosophy. We’re not fighting the system. We’re working with its native primitives.

Prerequisites and Packages

The package requirements are minimal. Install the LDAP connector and the PAM module for automatic home directory creation:

pkg install nss-pam-ldapd pam_mkhomedir sudo

The nss-pam-ldapd package provides the nslcd daemon, which bridges FreeBSD’s NSS (Name Service Switch) to LDAP. Unlike heavier solutions like SSSD, it does one thing well: translate NSS queries into LDAP lookups.

The pam_mkhomedir module handles a common pain point: Creating home directories on first login. Without it, users would authenticate successfully but fail to get a shell because /home/username doesn’t exist.

Host Identity and Keytab Provisioning

Since there’s no ipa-client-install, we provision the host identity manually. This is actually straightforward: Create the host entry on the IPA server and export a keytab.

On the IPA Server

First, create the host entry. If DNS is integrated with IPA, the --ip-address flag will create the A record automatically:

ipa host-add bsdhost.example.com --ip-address=192.168.1.25

Then generate and export the keytab. The ipa-getkeytab command randomizes the host’s password (just like machine account passwords in Active Directory) and saves the cryptographic key material:

ipa-getkeytab -s idm01.example.com \
    -p host/bsdhost.example.com@EXAMPLE.COM \
    -k /tmp/bsdhost.keytab

On the FreeBSD Host

Transfer the keytab securely (via scp) to /etc/krb5.keytab. The permissions here matter:

chown root:nslcd /etc/krb5.keytab
chmod 640 /etc/krb5.keytab

The nslcd daemon needs read access to this keytab to authenticate its LDAP queries via GSSAPI. Without this, directory lookups will fail when the host’s cached credentials expire.

You also need to ensure, sshd can read the file (or SSH authentication will fail). That can be done either with ACLs or group membership for sshd in the nslcd group.

Kerberos Configuration

The Kerberos configuration is minimal. FreeBSD’s native krb5 implementation needs to know where the KDC is and which realm to use.

/etc/krb5.conf

[libdefaults]
    default_realm = EXAMPLE.COM
    dns_lookup_kdc = false
    dns_lookup_realm = false

[realms]
    EXAMPLE.COM = {
        kdc = idm01.example.com
        admin_server = idm01.example.com
    }

[domain_realm]
    .example.com = EXAMPLE.COM
    example.com = EXAMPLE.COM

I explicitly disable DNS lookups for the KDC. While FreeIPA publishes SRV records, hardcoding the KDC hostname makes the configuration deterministic and easier to debug. In environments with multiple KDCs, you can list them all in the kdc directive.

Verify the configuration works:

klist -k

This should display the host principal from the keytab:

Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   1 host/bsdhost.example.com@EXAMPLE.COM

LDAP Configuration with nslcd

The nslcd daemon handles all LDAP communication. It runs as an unprivileged user, connects to FreeIPA’s LDAP server, and responds to NSS queries over a Unix socket.

The key insight is using SASL/GSSAPI authentication - nslcd authenticates to LDAP using the host’s Kerberos keytab rather than a bind password. This eliminates stored credentials and leverages the existing trust relationship.

/usr/local/etc/nslcd.conf

# Run as unprivileged user
uid nslcd
gid nslcd

# Connection details
uri ldap://idm01.example.com
base dc=example,dc=com

# Authentication: Use the system keytab (Host Principal)
sasl_mech GSSAPI
sasl_realm EXAMPLE.COM

# Mapping: Force all IPA users to use /bin/sh
# IPA typically defaults to /bin/bash, which lives in /usr/local/bin on FreeBSD
map passwd loginShell "/bin/sh"

The loginShell mapping deserves explanation. FreeIPA stores /bin/bash as the default shell, but on FreeBSD, bash (if installed) lives at /usr/local/bin/bash. Rather than requiring every user to update their shell in IPA, we override it locally to /bin/sh, which is guaranteed to exist. Users who need bash can set it explicitly in their profile.

Enable and start the service:

sysrc nslcd_enable="YES"
service nslcd start

NSS Configuration

Tell FreeBSD’s name service switch to query LDAP after local files. The order matters - local accounts (root, system users) take precedence, with LDAP providing everything else.

/etc/nsswitch.conf

group: files ldap
passwd: files ldap
hosts: files dns
networks: files
shells: files
services: compat
protocols: files
rpc: files

We avoid the legacy compat mode for passwd and group - the files ldap ordering is cleaner and more predictable.

Test the integration immediately:

id someuser

This should return UID and GID information from FreeIPA:

uid=1234(someuser) gid=1234(someuser) groups=1234(someuser),5000(admins)

If this fails, check that nslcd is running and can read the keytab. The logs in /var/log/messages usually reveal the problem.

SSH Configuration

Configure SSH to accept Kerberos tickets via GSSAPI. This enables true single sign-on - users with valid Kerberos tickets can SSH without any password prompt.

/etc/ssh/sshd_config

# Disable password authentication entirely
PasswordAuthentication no
KbdInteractiveAuthentication no

# Optional: Disable pubkey if relying purely on Kerberos
# PubkeyAuthentication no

# Kerberos / GSSAPI authentication
GSSAPIAuthentication yes
GSSAPICleanupCredentials yes
GSSAPIStrictAcceptorCheck no

The GSSAPIStrictAcceptorCheck no setting relaxes hostname verification. This is sometimes necessary when the SSH server’s hostname doesn’t exactly match the principal in the keytab (e.g., connecting via IP or a CNAME).

Restart SSH:

service sshd restart

PAM Configuration for Home Directories

When a user logs in for the first time, their home directory won’t exist. The pam_mkhomedir module creates it automatically with appropriate permissions.

Edit the SSH PAM configuration to add the module:

/etc/pam.d/sshd

auth            required        pam_unix.so             no_warn try_first_pass

# account
account         required        pam_nologin.so
account         required        pam_login_access.so
account         required        pam_unix.so

# session
session         required        pam_mkhomedir.so        mode=0700
session         required        pam_permit.so

# password
password        required        pam_unix.so             no_warn try_first_pass

The mode=0700 ensures home directories are created with restrictive permissions.

Before this works, verify that /home exists as a symlink to /usr/home (the FreeBSD convention):

ls -la /home

If /home doesn’t exist or isn’t a symlink:

ln -s /usr/home /home

Sudo Configuration

Instead of managing local group membership, leverage FreeIPA’s groups directly. Members of the IPA admins group get sudo access without any local configuration per user.

/usr/local/etc/sudoers (or use visudo)

# Allow members of the LDAP 'admins' group full sudo access
%admins ALL=(ALL:ALL) ALL

The % prefix tells sudo to treat admins as a group name. Since nslcd handles group lookups, this transparently queries FreeIPA.

For more granular control, you can define multiple group-based rules:

%admins ALL=(ALL:ALL) ALL
%developers ALL=(ALL) /usr/local/bin/docker, /usr/local/bin/podman
%dba ALL=(postgres) ALL

Verification

With everything configured, verify each component:

1. Identity Lookups

id admin

Expected output shows UIDs and groups from IPA:

uid=1000(admin) gid=1000(admin) groups=1000(admin),5000(admins)

2. Keytab Status

klist -k

Should list the host principal.

3. SSH Login Test

From another machine with a valid Kerberos ticket:

kinit someuser@EXAMPLE.COM
ssh bsdhost.example.com

The login should succeed without a password prompt. After login:

pwd

Should show /home/someuser (created automatically by pam_mkhomedir).

4. Sudo Test

If the user is in the admins group:

sudo whoami

Should return root without asking for a password (assuming NOPASSWD is configured) or after entering the user’s Kerberos password.

Troubleshooting

nslcd fails to start or lookups fail

Check the keytab permissions:

ls -la /etc/krb5.keytab

The nslcd user (or group) must have read access. Also verify the keytab contains valid keys:

klist -k /etc/krb5.keytab

SSH rejects GSSAPI authentication

Ensure the hostname matches the keytab principal. Check /var/log/auth.log for details. Common issues:

Connecting via IP when the keytab has only the FQDN
Clock skew between the FreeBSD host and KDC (Kerberos is sensitive to time differences)

Users can authenticate but have no home directory

Verify pam_mkhomedir is in /etc/pam.d/sshd and that /home is a symlink to /usr/home.

Group membership not visible

This usually indicates nslcd isn’t querying group information correctly. Verify the base DN in nslcd.conf includes the container where groups are stored. For IPA, this is typically cn=groups,cn=accounts,dc=example,dc=com, but using just dc=example,dc=com as the base should work with subtree searches.

Conclusion

Integrating FreeBSD with FreeIPA doesn’t require complex third-party tools or Linux-specific software. The combination of native Kerberos, nslcd for LDAP, and standard PAM modules provides a clean, maintainable solution.

The key advantages of this approach:

No local user management - users exist only in IPA
True SSO - Kerberos tickets authenticate SSH sessions
Minimal attack surface - no SSSD, no realm daemon, no Python dependencies
Stateless configuration - rebuild the host without losing identity integration

This pattern scales well across a fleet of FreeBSD hosts. Once you’ve configured one system, the configuration files can be templated and deployed with Ansible or similar tools. The only per-host customization is the keytab, which IPA generates automatically.

For environments already invested in FreeIPA for Linux hosts, adding FreeBSD becomes trivial - just another host entry in the directory, with the same users, groups, and policies applying uniformly across the infrastructure.

References

Reviving Life is Strange: Before the Storm on Modern Linux with a glibc Shim

2026-01-25T00:00:00+01:00

Life is Strange: Before the Storm shipped with native Linux support back in 2017. That was a different era - glibc 2.26 was current, and some developers made the unfortunate choice of linking against internal, undocumented glibc symbols. Fast forward to 2026, and the game refuses to start on modern distributions like Fedora 43. The symbols it depends on no longer exist.

The fix is a small shim library that brings back the missing functions. After implementing it, I played through the entire game without issues.

The Problem: Missing glibc Internals

When you try to launch the game on a modern system, it fails silently or crashes immediately. Running it from the terminal reveals the culprit:

./LifeIsStrange: symbol lookup error: ./LifeIsStrange: undefined symbol: __libc_dlopen_mode, version GLIBC_PRIVATE

The game’s binary was linked against __libc_dlopen_mode and __libc_dlsym - internal glibc functions that were never part of the public API. These symbols lived in a special version namespace called GLIBC_PRIVATE, which is exactly what it sounds like: private implementation details that upstream reserves the right to change at will.

In glibc 2.34, the dynamic loader was merged into libc proper, and these internal symbols were restructured. The functions still exist internally, but they’re no longer exported with the same symbol names and versions that older binaries expect.

The Solution: A Compatibility Shim

Since the internal functions are gone, we provide replacements. The game doesn’t actually need the internal implementation - it just needs something that loads shared libraries. The standard POSIX functions dlopen and dlsym do exactly that.

The trick is making our replacement functions appear with the exact symbol names and versions the game expects. GCC’s __symver__ attribute handles this:

#include <dlfcn.h>

__attribute__((__symver__("__libc_dlopen_mode@GLIBC_PRIVATE")))
void* libc_dlopen_mode(const char *filename, int flags)
{
    return dlopen(filename, flags);
}

__attribute__((__symver__("__libc_dlsym@GLIBC_PRIVATE")))
void *libc_dlsym(void *restrict handle, const char *restrict symbol)
{
    return dlsym(handle, symbol);
}

The __symver__ attribute tells the linker to export these functions under specific versioned symbol names. When the game’s dynamic linker searches for __libc_dlopen_mode@GLIBC_PRIVATE, it finds our shim instead of the missing glibc symbol.

Building the Shim Library

Create a working directory and add three files:

libc_dlopen_mode.c:

#include <dlfcn.h>

__attribute__((__symver__("__libc_dlopen_mode@GLIBC_PRIVATE")))
void* libc_dlopen_mode(const char *filename, int flags)
{
    return dlopen(filename, flags);
}

__attribute__((__symver__("__libc_dlsym@GLIBC_PRIVATE")))
void *libc_dlsym(void *restrict handle, const char *restrict symbol)
{
    return dlsym(handle, symbol);
}

version.map:

GLIBC_PRIVATE {
global:
    *;
};

The version script defines our custom GLIBC_PRIVATE version tag. Without it, the linker wouldn’t know how to apply the version specified in __symver__.

Makefile:

.PHONY: all clean

RES_LIB_NAME    =   liblibc_dlopen_mode
RES_LIB         =   $(RES_LIB_NAME).so
SRCS            =   \
    libc_dlopen_mode.c

CFLAGS  +=  -fpic -shared -flto -Wl,--version-script -Wl,version.map -Wl,--as-needed

all: $(RES_LIB)

$(RES_LIB): $(SRCS) version.map
    $(CC) $(CFLAGS) -o $@ $(SRCS)

clean:
    $(RM) $(RES_LIB)

Build and install:

make
sudo cp liblibc_dlopen_mode.so /usr/local/lib/

You can verify the symbols are correctly versioned:

$ nm -D /usr/local/lib/liblibc_dlopen_mode.so
                 w __cxa_finalize@GLIBC_2.2.5
                 U dlopen@GLIBC_2.34
                 U dlsym@GLIBC_2.34
0000000000001100 T libc_dlopen_mode@GLIBC_PRIVATE
0000000000001120 T libc_dlsym@GLIBC_PRIVATE

The T entries show our exported functions with the correct GLIBC_PRIVATE version tag.

Configuring Steam

Open Steam, right-click on Life is Strange: Before the Storm, and select Properties. In the Launch Options field, add:

LD_PRELOAD=/usr/local/lib/liblibc_dlopen_mode.so %command%

If you’re using GameMode for performance optimization:

LD_PRELOAD=/usr/local/lib/liblibc_dlopen_mode.so gamemoderun %command%

The LD_PRELOAD environment variable tells the dynamic linker to load our shim library before any others. When the game binary starts and requests __libc_dlopen_mode@GLIBC_PRIVATE, the linker finds it in our preloaded library instead of failing with an undefined symbol error.

Why This Works

The game uses these internal functions to dynamically load additional libraries at runtime. The actual implementation details don’t matter to the game - it just needs a function that accepts a filename and flags, calls the kernel to map a shared object, and returns a handle. The public dlopen and dlsym functions do exactly this.

Our shim acts as a translation layer: the game calls what it thinks are internal glibc functions, and we redirect those calls to the stable public API. The game can’t tell the difference.

Other Affected Games

Life is Strange: Before the Storm isn’t alone. Several games from the 2015-2018 era made the same mistake of depending on glibc internals. If you encounter similar “undefined symbol” errors mentioning GLIBC_PRIVATE, this same technique should work - just verify which specific symbols are missing and add corresponding wrapper functions.

Caveats

A few things to keep in mind:

This is a workaround, not a proper fix. The correct solution would be for the game developers to rebuild against the public API. Unfortunately, many older Linux ports are abandoned.
The shim intercepts calls globally within the game process. This is fine for single-player games but be mindful when using LD_PRELOAD with any software that handles sensitive data.
Future glibc changes could theoretically break even the public dlopen/dlsym API, though this is extremely unlikely given POSIX compatibility requirements.

Conclusion

A few lines of C and a Steam launch option are all it takes to revive this abandoned Linux port. The game runs flawlessly - I completed all three episodes and the bonus Farewell episode without a single crash.

It’s a shame that native Linux ports from this era are becoming unplayable due to dependency rot, but at least the fix is straightforward. Sometimes the best compatibility layer is just a thin wrapper that pretends to be something it isn’t.

References

glibc 2.34 Release Notes - Details on the dynamic loader merge
Life is Strange: Before the Storm on Steam
GCC Symbol Versioning - Documentation for __symver__ attribute
LD_PRELOAD Trick - Dynamic linker documentation

Self-Hosted CryptPad on FreeBSD with VNET Jails and Caddy

2026-01-24T00:00:00+01:00

CryptPad is an end-to-end encrypted collaboration suite. Think Google Docs, but where the server never sees your content. It’s a compelling option for privacy-conscious teams or anyone wanting to own their data. This post documents installing CryptPad in a FreeBSD VNET jail, served behind a Caddy reverse proxy, with network isolation enforced by PF.

This guide assumes familiarity with FreeBSD jails, PF, and BastilleBSD. It focuses on architecture and pitfalls rather than introductory jail setup.

Why CryptPad?

Most collaborative document platforms require trusting the provider with your data. CryptPad takes a different approach: all encryption happens client-side in the browser. The server stores only encrypted blobs and has no way to decrypt your documents, even if compelled to.

Self-hosting adds another layer: your data never leaves infrastructure you control. Combined with FreeBSD’s VNET jails for network isolation, this creates a robust privacy-respecting setup. Unlike tools such as Nextcloud or OnlyOffice, CryptPad’s threat model assumes the server itself is untrusted.

Architecture Overview

[ Internet ]
     |
     v
+-----------------------+
|    PF Firewall        |
|    NAT + RDR          |
+-----------+-----------+
            |
     +------+------+
     | bastille0   |  (bridge interface)
     | 10.254.254.1|
     +------+------+
            |
   +--------+--------+
   |                 |
   v                 v
+-------+      +-----------+
| Caddy |      | CryptPad  |
| Jail  |      |   Jail    |
| .10   |----->|   .36     |
+-------+      +-----------+

The key security principle: jails live on a private RFC1918 network (10.254.254.0/24) that is completely invisible to the internet. PF on the host performs NAT for outbound traffic and only redirects specific ports to specific jails. The CryptPad jail has no direct internet exposure - all traffic flows through the Caddy jail.

VNET Jail Networking

This setup uses VNET jails, which give each jail its own complete network stack including interfaces, routing tables, and firewall state. Unlike traditional IP-based jails that share the host’s network stack, VNET jails behave like independent networked machines.

The jails attach to a bridge interface on the host:

# Host /etc/rc.conf (relevant networking section)
cloned_interfaces="bridge0"
ifconfig_bridge0_name="bastille0"
ifconfig_bastille0="inet 10.254.254.1/24"
gateway_enable="YES"

The host acts as the default gateway for all jails. Each jail gets an IP from the 10.254.254.0/24 range:

$ bastille list
 JID  Name           State  IP Address
 2    caddy          Up     10.254.254.10
 10   cryptpad       Up     10.254.254.36
 ...

Creating the CryptPad jail with Bastille:

bastille create -B cryptpad 15.0-RELEASE 10.254.254.36 bastille0

The -B flag creates a VNET jail attached to the bastille0 bridge.

PF Firewall: NAT and Selective Exposure

The critical security layer is PF on the host. It provides:

NAT for outbound traffic - jails can reach the internet (for package updates, etc.) but appear to come from the host’s public IP
Port redirection - only ports 80/443 are exposed, and only to the Caddy jail
Default deny - everything else is silently dropped

# /etc/pf.conf
ext_if = "vtnet0"
jail_net = "10.254.254.0/24"
frontend_v4 = "10.254.254.10"  # Caddy jail

table <jails_v4> { $jail_net }

set skip on lo0
set block-policy drop

# NAT: Jails -> Internet
nat on $ext_if inet from <jails_v4> to any -> ($ext_if)

# RDR: Internet -> Caddy jail only
rdr pass on $ext_if inet proto tcp to ($ext_if) port {80,443} -> $frontend_v4

# Default deny
block drop in log all
block drop out log all

# Allow established connections out
pass out quick all keep state

# Anti-spoofing
antispoof quick for { $ext_if, bastille0 }

# ICMP
pass in inet proto icmp icmp-type { echoreq, unreach }

# Jail egress (but not to other jails)
pass in quick on bastille0 from <jails_v4> to ! 10.254.254.0/24 keep state

Notice what’s not here: there’s no rule allowing direct internet access to the CryptPad jail. Traffic to ports 80/443 is redirected to 10.254.254.10 (Caddy), which then proxies to CryptPad internally over the bridge network. The CryptPad jail is completely hidden from the internet.

This is defense in depth: even if CryptPad had a vulnerability, an attacker couldn’t establish a reverse shell or exfiltrate data directly - they’d have to go through Caddy first.

CryptPad Installation

Two domains are required: one for the main application and one for the sandbox (CryptPad’s XSS protection mechanism). Both domains resolve to the host’s public IP, where PF redirects them to Caddy.

Prerequisites

Inside the CryptPad jail, install the necessary packages. CryptPad requires Node.js 20+ and build tools for native modules:

pkg install git node24 npm-node24 gmake python3 bash

The full package list for a working installation looks like this:

brotli-1.2.0,1          git-tiny-2.52.0         node24-24.12.0
c-ares-1.34.6           gmake-4.4.1             npm-node24-11.7.0
corepack-0.34.5         icu-76.1,1              python311-3.11.14
libuv-1.51.0            llhttp-9.3.0            simdjson-4.2.4

User and Installation

Never run CryptPad as root. Create a dedicated user:

pw useradd -n cryptpad -c "CryptPad User" -d /usr/local/cryptpad -s /bin/sh -m
su - cryptpad
git clone https://github.com/cryptpad/cryptpad.git .

The Critical Installation Step

This is where many installations fail. Running only npm install will leave you with a site that hangs on “Loading…” and throws 404 errors for require.js. CryptPad needs three distinct build steps:

# 1. Install backend dependencies
npm install

# 2. Install frontend components (bower, bootstrap, require.js, etc.)
npm run install:components

# 3. Build static assets (pages, CSS, etc.)
npm run build

The second step is easily missed because it’s not obvious from the documentation. If you skip it, the application will appear to start but the browser will never finish loading.

Configuration

Copy the example configuration and edit it:

cp config/config.example.js config/config.js

The critical settings in config/config.js:

module.exports = {
    // Listen on the jail's interface (not localhost)
    httpAddress: '10.254.254.36',

    // Main domain - where users access CryptPad
    httpUnsafeOrigin: 'https://pad.example.com',

    // Sandbox domain - MUST be different from main domain
    httpSafeOrigin: 'https://sandbox.example.com',

    // Admin keys (add after registering your admin account)
    adminKeys: [
        // "[your-public-key-here]"
    ],
};

Understanding the Two Domains

CryptPad’s security model relies on browser same-origin policy. The main domain handles authentication and cryptographic operations. The sandbox domain loads the UI in an iframe with a different origin, preventing XSS attacks from accessing your encryption keys.

Both domains point to the same CryptPad instance, the application routes requests appropriately based on the Host header. This is why correctly passing the Host header through the reverse proxy is essential.

FreeBSD Service Setup

CryptPad includes an rc.d script template. Copy it and enable the service:

cp /usr/local/cryptpad/docs/rc.d-cryptpad /usr/local/etc/rc.d/cryptpad
chmod +x /usr/local/etc/rc.d/cryptpad
sysrc cryptpad_enable="YES"

Here’s the actual rc.d script adapted for FreeBSD:

#!/bin/sh
# PROVIDE: cryptpad
# REQUIRE: DAEMON
# KEYWORD: shutdown

. /etc/rc.subr

name="cryptpad"
start_cmd="start"
stop_cmd="stop"
rcvar=cryptpad_enable

pidfile="/var/run/cryptpad/${name}.pid"
desc="CryptPad Service"

load_rc_config ${name}

start() {
    /bin/mkdir -p /var/run/cryptpad
    /usr/sbin/chown cryptpad:cryptpad /var/run/cryptpad

    /usr/bin/su cryptpad -c "export PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:~/bin && \
        cd /usr/local/cryptpad && \
        /usr/sbin/daemon -T ${name} \
            -P /var/run/cryptpad/${name}_supervisor.pid \
            -p /var/run/cryptpad/${name}.pid \
            -f -S -r /usr/local/bin/node server"
}

stop() {
    /bin/kill -9 `cat /var/run/cryptpad/${name}_supervisor.pid`
    /bin/kill -15 `cat /var/run/cryptpad/${name}.pid`
}

run_rc_command "$1"

Start the service:

service cryptpad start

A running CryptPad instance spawns multiple worker processes:

USER       PID %CPU %MEM    VSZ   RSS TT  STAT COMMAND
cryptpad 89849  0.0  0.0  13068  2508  -  IsJ  daemon: /usr/local/bin/node[89850]
cryptpad 89850  0.0  1.1 839584 87964  -  SJ   /usr/local/bin/node server
cryptpad 89851  0.0  1.1 840864 88624  -  SJ   /usr/local/bin/node ./lib/http-worker.js
cryptpad 89852  0.0  1.1 842400 88000  -  SJ   /usr/local/bin/node ./lib/http-worker.js
...
cryptpad 89857  0.0  1.0 834088 79448  -  SJ   /usr/local/bin/node lib/workers/db-worker

Caddy Reverse Proxy

Caddy runs in its own jail (10.254.254.10) and is the only jail exposed to the internet via PF’s port redirection. Configure it to proxy both CryptPad domains to the application jail. The Host header passthrough is crucial! Without it, CryptPad’s WebSocket security checks will fail:

pad.example.com {
    reverse_proxy 10.254.254.36:3000 {
        header_up Host {host}
        header_up X-Real-IP {remote}
    }

    header {
        Strict-Transport-Security "max-age=31536000; includeSubdomains"
        X-XSS-Protection "1; mode=block"
        X-Content-Type-Options "nosniff"
        X-Frame-Options "SAMEORIGIN"
        Referrer-Policy "same-origin"
    }
}

sandbox.example.com {
    reverse_proxy 10.254.254.36:3000 {
        header_up Host {host}
        header_up X-Real-IP {remote}
    }

    header {
        Strict-Transport-Security "max-age=31536000; includeSubdomains"
    }
}

Note that both domains proxy to the same backend (10.254.254.36:3000). Caddy handles TLS automatically via Let’s Encrypt. Since the jails communicate over the private bridge network, this internal traffic doesn’t need encryption.

Troubleshooting

Site Hangs on “Loading…”

This is almost always caused by missing frontend components. Check if require.js exists:

ls -l /usr/local/cryptpad/www/components/requirejs/require.js

If missing, you skipped npm run install:components. Fix it:

su - cryptpad
cd /usr/local/cryptpad
npm run install:components
npm run build
exit
service cryptpad restart

WebSocket Connection Failures

If you see WebSocket errors in the browser console, verify:

The Host header is being passed through your reverse proxy
Both domains are correctly configured in config.js
The httpAddress in config.js matches the jail’s IP

White Screen

Usually a sandbox domain misconfiguration. The main and sandbox domains must be different but both must resolve to your CryptPad instance. Check browser console for CORS or CSP errors.

Adding an Admin Account

After the first successful login:

Register a normal account through the web interface
Go to Settings and find your public signing key
Add it to the adminKeys array in config/config.js
Restart CryptPad

adminKeys: [
    "[cryptpad-admin@pad.example.com/YZgXQxKR0Rcb6r6CmxHPdAGLVludrAF2lEnkbx1vVOo=]",
],

The admin panel then becomes available at https://pad.example.com/admin/.

Conclusion

CryptPad in a FreeBSD VNET jail provides a solid foundation for privacy-respecting collaboration. The layered security model gives defense in depth:

VNET jails provide complete network stack isolation
PF NAT hides all jails behind RFC1918 addresses
Selective port redirection exposes only the Caddy frontend
End-to-end encryption means even you as administrator cannot read documents

The main gotcha is the three-step build process - if your installation hangs on loading, that’s almost certainly the culprit. Get that right, ensure both domains are configured, and you’ll have a working instance that’s invisible to port scanners and attackers probing for Node.js vulnerabilities.

References

Self-Hosting Email on FreeBSD: A Secure, Jailed Setup with Postfix and Dovecot

2026-01-18T00:00:00+01:00

Self-hosting email is often described as a Sisyphean task. A constant battle against spam lists, deliverability issues, and security threats. While there’s truth to that reputation, running your own mail infrastructure remains one of the best ways to reclaim privacy and truly understand how the internet’s oldest federation protocol works.

In this article, I’ll walk through my current mail server setup running on FreeBSD 15.0. The design philosophy prioritizes security through isolation (VNET jails), data protection (ZFS encryption), and attack surface reduction (GeoIP filtering). This isn’t a beginner’s tutorial - I’ll assume familiarity with FreeBSD basics and focus on the architecture decisions and configuration details that make this setup work.

Architecture Overview

Before diving into configuration files, here’s how the pieces fit together:

                        Internet
                            |
                            v
                +-----------------------+
                |     PF Firewall       |
                |  GeoIP + NAT + RDR    |
                +-----------+-----------+
                            |
            +---------------+---------------+
            |                               |
     Port 25 (SMTP)              Ports 143/587/4190
     Open to all                 GeoIP restricted
            |                               |
            v                               v
    +-------+-------------------------------+-------+
    |              VNET Jail: mailstack             |
    |                                               |
    |  +----------+  +----------+  +-----------+    |
    |  | Postfix  |  | Dovecot  |  |  Rspamd   |    |
    |  |   MTA    |  |IMAP/LMTP |  |  Filter   |    |
    |  +----+-----+  +----+-----+  +-----+-----+    |
    |       |             |              |          |
    |       +------+------+------+-------+          |
    |              |                                |
    |              v                                |
    |     +------------------+                      |
    |     | /var/vmail       |  (nullfs mount)      |
    |     | ZFS encrypted    |                      |
    |     +------------------+                      |
    +-----------------------------------------------+
                            |
            +---------------+---------------+
            |                               |
    +-------+-------+               +-------+-------+
    | VNET Jail:    |               |    Host:      |
    |   webmail     |               | zroot/secure  |
    | (Roundcube)   |               | (encrypted)   |
    +---------------+               +---------------+

The key insight here is separation of concerns: the host handles networking and storage, while the jails run the actual services. Mail data lives on an encrypted ZFS dataset on the host and is mounted into the jail via nullfs. This means I can destroy and recreate the jail without touching any mail data, and backups of the encrypted dataset can be sent offsite without the backup server ever possessing the decryption key.

The Host System: FreeBSD 15.0

The foundation is a hardened FreeBSD 15.0 system. Security starts at the kernel level.

Security Hardening

I run with kern_securelevel="2", which is a significant step up from the default. At this level, even the root user cannot write to immutable files, load kernel modules, or modify the firewall rules without a reboot. This means even if an attacker gains root access, they can’t disable PF or load a rootkit.

Additional hardening via sysctl prevents users from seeing other processes and restricts access to kernel internals:

/etc/sysctl.conf

# Prevent users from seeing processes of other UIDs
security.bsd.see_other_uids=0
security.bsd.see_other_gids=0
security.bsd.see_jail_proc=0

# Disable unprivileged access to kernel internals
security.bsd.unprivileged_read_msgbuf=0
security.bsd.unprivileged_proc_debug=0

# Randomize PIDs to make race conditions harder to exploit
kern.randompid=1

# ZFS tuning
vfs.zfs.vdev.min_auto_ashift=12

# Allow large PF tables for GeoIP filtering
net.pf.request_maxcount=500000

The last setting is crucial for GeoIP filtering - the default PF table size is too small to hold the ~273,000 CIDR blocks we’ll be loading.

Kernel Module Loading

The necessary modules for jail networking are loaded at boot:

/boot/loader.conf

kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
zfs_load="YES"

# Required for VNET jails
nullfs_load="YES"
if_bridge_load="YES"
if_epair_load="YES"
kern.racct.enable=1

Network and ZFS Layout

The host handles the physical connection and bridges traffic to the jails. The network configuration uses a bridge interface that provides connectivity to all VNET jails:

/etc/rc.conf

hostname="mail"
keymap="us.kbd"

# Security hardening
kern_securelevel_enable="YES"
kern_securelevel="2"

# External Interface (VPS provider network)
ifconfig_vtnet0="inet 192.0.2.50/32 -lro -tso"
ifconfig_vtnet0_ipv6="inet6 2001:db8:1c1c:4d2::1/65"
defaultrouter="172.31.1.1"
ipv6_defaultrouter="fe80::1%vtnet0"

# Bridge for VNET Jails
cloned_interfaces="bridge0"
ifconfig_bridge0="inet 10.0.0.1 netmask 255.255.255.0"
ifconfig_bridge0_ipv6="inet6 2001:db8:1c1c:4d2:8000::1/65"

# Enable routing for jails
gateway_enable="YES"
ipv6_gateway_enable="YES"

# Services
dumpdev="NO"
pf_enable="YES"
zfs_enable="YES"
jail_enable="YES"
sshd_enable="YES"
pflog_enable="YES"
clear_tmp_enable="YES"
moused_nondefault_enable="NO"

The ZFS layout separates sensitive data onto an encrypted dataset. Here’s the relevant portion:

NAME                          USED  AVAIL  REFER  MOUNTPOINT
zroot/secure                 4.28G  29.2G   200K  /zroot/secure
zroot/secure/jails           4.04G  29.2G   264K  /jails
zroot/secure/jails/mailstack 1.40G  29.2G  1.00G  /jails/mailstack
zroot/secure/jails/webmail   1022M  29.2G   805M  /jails/webmail
zroot/secure/vmail            248M  29.2G   233M  /var/mail

The encryption uses AES-256-GCM:

$ zfs get encryption zroot/secure
NAME          PROPERTY    VALUE        SOURCE
zroot/secure  encryption  aes-256-gcm  -

This means all child datasets (jails and mail storage) inherit the encryption. After a reboot, the dataset must be manually unlocked before the jails can start - a deliberate trade-off between availability and security.

The Firewall: PF with GeoIP Filtering

I use PF (Packet Filter) to ruthlessly cut down on attack surface. The most effective measure has been GeoIP filtering. Since my users are all in Central Europe, I can block authentication attempts from everywhere else while keeping SMTP open for mail delivery.

/etc/pf.conf

# --- Macros ---
ext_if = "vtnet0"
jail_net = "10.0.0.0/24"
jail_net6 = "2001:db8:1c1c:4d2:8000::/65"
host_ipv6 = "2001:db8:1c1c:4d2::1"

mail_ipv4 = "10.0.0.3"
mail_ipv6 = "2001:db8:1c1c:4d2:8000::3"
webmail_ipv6 = "2001:db8:1c1c:4d2:8000::4"

# --- Tables ---
table <bruteforce> persist
table <jails_v4> { $jail_net }
table <jails_v6> { $jail_net6 }
table <geoip_users> persist

# --- Options ---
set limit table-entries 1000000
set skip on lo0
set block-policy drop
set loginterface $ext_if

# --- Scrub ---
scrub in all fragment reassemble
scrub out on $ext_if random-id

# --- NAT (IPv4 only for legacy support) ---
nat on $ext_if inet from <jails_v4> to any -> ($ext_if)

# --- RDR for services ---
# Client ports: GeoIP restricted
rdr pass on $ext_if inet proto tcp from <geoip_users> \
    to ($ext_if) port {143, 587, 4190} -> $mail_ipv4

# SMTP: Open to all (server-to-server)
rdr pass on $ext_if inet proto tcp from any \
    to ($ext_if) port 25 -> $mail_ipv4

# --- Filtering ---
# Block known bad actors immediately
block quick from <bruteforce>

# Default deny
block drop in log all
block drop out log all

# Allow all outbound (stateful)
pass out quick all keep state

# Anti-spoofing
antispoof quick for { $ext_if, bridge0 }

# SSH only from GeoIP restricted IP ranges
pass in quick on $ext_if proto tcp from <geoip_users> to any port 22 \
    flags S/SA keep state \
    (max-src-conn 5, max-src-conn-rate 3/30, \
     overload <bruteforce> flush global)

# Block and log all other SSH attempts
block in log quick on $ext_if proto tcp from any to any port 22 \
    label "ssh_not_trusted"

# IPv6: Mail client ports with GeoIP restriction
pass in quick on $ext_if inet6 proto tcp from <geoip_users> \
    to $mail_ipv6 port {143, 587, 4190} flags S/SA keep state

# IPv6: SMTP open to all
pass in quick on $ext_if inet6 proto tcp from any \
    to $mail_ipv6 port 25 flags S/SA keep state

# IPv6: Webmail with GeoIP restriction
pass in quick on $ext_if inet6 proto tcp from <geoip_users> \
    to $webmail_ipv6 port {80, 443} flags S/SA keep state

# Essential ICMPv6
pass in quick inet6 proto ipv6-icmp icmp6-type \
    { echoreq, echorep, neighbrsol, neighbradv, toobig, timex, paramprob }

# ICMP for IPv4 (ping, MTU discovery)
pass in inet proto icmp icmp-type { echoreq, unreach }

# Jail egress
pass in quick on { bridge0 } from <jails_v4> to ! 10.0.0.0/24 keep state
pass in quick on { bridge0 } inet6 from <jails_v6> \
    to ! 2001:db8:1c1c:4d2:8000::/65 keep state

The dual approach to traffic handling is visible in the rules:

SMTP (port 25): from any - must be globally accessible for mail delivery
Client ports (143, 587, 4190): from <geoip_users> - restricted to allowed countries

I covered the GeoIP update script in detail in my previous article on GeoIP-aware firewalling. The short version: I parse MaxMind’s GeoLite2 CSV data for my target countries (DE, AT, CH, NL, LU, FR) and load the resulting ~273,000 CIDR blocks into PF in chunks to avoid memory exhaustion.

Jail Architecture: VNET for True Isolation

I use VNET jails rather than traditional IP alias jails. Unlike IP aliases, VNET gives each jail its own fully virtualized network stack with dedicated interfaces. This allows firewall rules inside the jail if needed and keeps the host’s networking clean.

/etc/jail.conf

mailstack {
    enforce_statfs = 2;
    devfs_ruleset = 4;
    exec.clean;
    exec.consolelog = /var/log/jails/mailstack_console.log;
    exec.start = '/bin/sh /etc/rc';
    exec.stop = '/bin/sh /etc/rc.shutdown';
    host.hostname = "mailstack";
    mount.devfs;
    path = "/jails/mailstack";

    # Mount the mail storage from the host
    mount += "/var/mail /jails/mailstack/var/vmail nullfs rw 0 0";

    # VNET configuration
    vnet;
    vnet.interface = "e0b_mailstack";
    exec.prestart += "epair0=$(ifconfig epair create) && \
        ifconfig ${epair0} up name e0a_mailstack && \
        ifconfig ${epair0%a}b up name e0b_mailstack";
    exec.prestart += "ifconfig bridge0 addm e0a_mailstack";
    exec.prestart += "ifconfig e0a_mailstack description \
        \"vnet0 host interface for Jail mailstack\"";
    exec.poststop += "ifconfig e0a_mailstack destroy";
}

webmail {
    enforce_statfs = 2;
    devfs_ruleset = 4;
    exec.clean;
    exec.consolelog = /var/log/jails/webmail_console.log;
    exec.start = '/bin/sh /etc/rc';
    exec.stop = '/bin/sh /etc/rc.shutdown';
    host.hostname = "webmail";
    mount.devfs;
    path = "/jails/webmail";

    vnet;
    vnet.interface = "e0b_webmail";
    exec.prestart += "epair0=$(ifconfig epair create) && \
        ifconfig ${epair0} up name e0a_webmail && \
        ifconfig ${epair0%a}b up name e0b_webmail";
    exec.prestart += "ifconfig bridge0 addm e0a_webmail";
    exec.prestart += "ifconfig e0a_webmail description \
        \"vnet0 host interface for Jail webmail\"";
    exec.poststop += "ifconfig e0a_webmail destroy";
}

To ensure the jails have only access to the /dev device nodes that they need, we need to have a devfs rule with the same id in /etc/devfs.rules and re-start the devfs servive with service devfs restart. This is good practice for isolation of the jails.

/etc/devfs.rules

[jail_vnet=4]
add include \$devfsrules_hide_all
add include \$devfsrules_unhide_basic
add include \$devfsrules_unhide_login
add include \$devfsrules_jail
add include \$devfsrules_jail_vnet
add path 'bpf*' unhide

The epair interfaces create a virtual cable - one end attached to the host’s bridge, the other inside the jail. When the jail stops, the interface is automatically destroyed.

Inside the mailstack jail, the network is configured like any FreeBSD system:

/jails/mailstack/etc/rc.conf

ifconfig_e0b_mailstack_name="vnet0"
ifconfig_vnet0="inet 10.0.0.3 netmask 255.255.255.0"
ifconfig_vnet0_ipv6="inet6 2001:db8:1c1c:4d2:8000::3/65"
ifconfig_vnet0_descr="jail interface for bridge0"
defaultrouter="10.0.0.1"
ipv6_defaultrouter="2001:db8:1c1c:4d2:8000::1"

# Disable sendmail (we use Postfix)
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"

# Services
syslogd_flags="-ss"
cron_flags="-J 60"
postfix_enable="YES"
dovecot_enable="YES"
rspamd_enable="YES"
local_unbound_enable="YES"

Note that I run a local Unbound resolver inside the jail. This ensures DNS queries for DKIM verification and SPF checks don’t leak to external resolvers and provides caching for the many lookups mail processing requires.

The Mail Stack Configuration

Inside the mailstack jail, three components work together: Postfix handles SMTP, Dovecot manages storage and IMAP access, and Rspamd filters spam and signs outgoing mail with DKIM.

Postfix (The MTA)

Postfix handles both receiving mail from the internet and accepting submissions from authenticated users. The configuration enforces strict restrictions to reject spam early in the SMTP conversation.

/usr/local/etc/postfix/main.cf (key settings)

compatibility_level = 3.10

# Identity
myhostname = mail.example.org
myorigin = $myhostname
mydestination = localhost.$mydomain, localhost

# Network
inet_interfaces = all
inet_protocols = all
mynetworks_style = host

# TLS (Incoming & Outgoing)
smtpd_tls_cert_file = /etc/mail/certs/cert.pem
smtpd_tls_key_file = /etc/mail/certs/key.pem
smtpd_tls_security_level = may
smtpd_tls_auth_only = yes
smtpd_tls_loglevel = 1
smtp_tls_security_level = may
smtp_tls_loglevel = 1
smtp_tls_CApath = /etc/ssl/certs

# SASL (Auth via Dovecot)
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth
smtpd_sasl_auth_enable = yes
smtpd_sasl_security_options = noanonymous

# Virtual Domains
virtual_mailbox_domains = example.org example.net
virtual_mailbox_maps = hash:/usr/local/etc/postfix/vmailbox
virtual_alias_maps = hash:/usr/local/etc/postfix/virtual
virtual_uid_maps = static:5000
virtual_gid_maps = static:5000
virtual_minimum_uid = 5000

# Hand off to Dovecot for local delivery
virtual_transport = lmtp:unix:private/dovecot-lmtp

# Rspamd integration via milter protocol
smtpd_milters = inet:localhost:11332
non_smtpd_milters = inet:localhost:11332
milter_protocol = 6
milter_default_action = accept

# Strict HELO requirements
smtpd_helo_required = yes
smtpd_helo_restrictions =
    permit_mynetworks,
    permit_sasl_authenticated,
    reject_invalid_helo_hostname,
    reject_non_fqdn_helo_hostname,
    permit

# Sender restrictions
smtpd_sender_restrictions =
    permit_mynetworks,
    permit_sasl_authenticated,
    reject_non_fqdn_sender,
    reject_unknown_sender_domain,
    permit

# Recipient restrictions
smtpd_recipient_restrictions =
    permit_mynetworks,
    permit_sasl_authenticated,
    reject_unauth_destination,
    reject_non_fqdn_recipient,
    reject_unknown_reverse_client_hostname,

The submission port (587) for authenticated users is configured in master.cf:

/usr/local/etc/postfix/master.cf (submission entry)

submission inet n       -       n       -       -       smtpd
  -o syslog_name=postfix/submission
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_tls_auth_only=yes
  -o smtpd_milters=inet:localhost:11332

The virtual mailbox and alias maps define which addresses are valid and where they’re delivered:

/usr/local/etc/postfix/vmailbox

# example.org
admin@example.org        OK
backups@example.org      OK

# example.net
user@example.net         OK
accounts@example.net     OK

/usr/local/etc/postfix/virtual

# Aliases for example.org
postmaster@example.org   admin@example.org
abuse@example.org        admin@example.org
webmaster@example.org    admin@example.org

# Catch-all for specific services -> accounts
github@example.net       accounts@example.net
hetzner@example.net      accounts@example.net

After modifying these files, rebuild the hash databases:

postmap /usr/local/etc/postfix/vmailbox
postmap /usr/local/etc/postfix/virtual
postfix reload

Dovecot (Storage & Authentication)

Dovecot is the workhorse. It serves IMAP to clients, handles authentication for Postfix via SASL, receives mail from Postfix via LMTP, and processes Sieve filter rules. The special_use attributes implement RFC 6154. This signals to modern clients (iOS, Outlook) which folders are for Sent, Trash, and Drafts, preventing the chaos of mixed ‘Sent’ and ‘Sent Messages’ folders.

/usr/local/etc/dovecot/dovecot.conf

protocols = imap lmtp sieve
auth_mechanisms = plain login

# Security - require TLS
ssl = required
ssl_cert = </etc/mail/certs/cert.pem
ssl_key = </etc/mail/certs/key.pem
ssl_min_protocol = TLSv1.2

# Mail storage location
mail_location = maildir:/var/vmail/%d/%n/Maildir
mail_uid = 5000
mail_gid = 5000
mmap_disable = yes

# Sieve for server-side filtering
plugin {
  sieve = /var/vmail/%d/%n/.dovecot.sieve
  sieve_dir = /var/vmail/%d/%n/sieve
}

# Authentication (simple passwd-file for easy management)
passdb {
  driver = passwd-file
  args = scheme=BLF-CRYPT username_format=%u /usr/local/etc/dovecot/users
}

userdb {
  driver = static
  args = uid=5000 gid=5000 home=/var/vmail/%d/%n
}

# Socket for Postfix SASL authentication
service auth {
  unix_listener /var/spool/postfix/private/auth {
    mode = 0660
    user = postfix
    group = postfix
  }
}

# Socket for Postfix LMTP delivery
service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
    mode = 0660
    user = postfix
    group = postfix
  }
}

protocol lmtp {
  mail_plugins = $mail_plugins sieve
}

# Standard IMAP folders (RFC 6154)
namespace inbox {
  inbox = yes
  separator = /

  mailbox Drafts {
    special_use = \Drafts
    auto = subscribe
  }
  mailbox Junk {
    special_use = \Junk
    auto = subscribe
  }
  mailbox Trash {
    special_use = \Trash
    auto = subscribe
  }
  mailbox Sent {
    special_use = \Sent
    auto = subscribe
  }
  mailbox "Sent Messages" {
    special_use = \Sent
  }
  mailbox Archive {
    special_use = \Archive
    auto = subscribe
  }
}

The users file contains password hashes (generated with doveadm pw -s BLF-CRYPT):

/usr/local/etc/dovecot/users

user@example.net:{BLF-CRYPT}$2y$05$...hash...
admin@example.org:{BLF-CRYPT}$2y$05$...hash...

Rspamd (The Gatekeeper)

Rspamd analyzes incoming messages and handles DKIM signing for outgoing mail. Without valid DKIM signatures, your mail will likely land in Gmail’s spam folder.

/usr/local/etc/rspamd/local.d/dkim_signing.conf

path = "/var/lib/rspamd/dkim/$domain.key";
selector = "mail";
allow_username_mismatch = true;

Generate DKIM keys for each domain:

rspamadm dkim_keygen -s mail -d example.org \
    -k /var/lib/rspamd/dkim/example.org.key > /var/lib/rspamd/dkim/example.org.pub
chmod 640 /var/lib/rspamd/dkim/example.org.key
chown rspamd:rspamd /var/lib/rspamd/dkim/example.org.key

The public key goes into your DNS as a TXT record at mail._domainkey.example.org.

Boot and Recovery Procedure

Because the mail data lives on an encrypted ZFS dataset, the system requires manual intervention after a reboot. This is a deliberate security trade-off.

After the system boots:

# Unlock the encrypted dataset
zfs load-key -r zroot/secure

# Mount all ZFS filesystems
zfs mount -a

# Start the jails
service jail start

This sequence ensures that even if someone gains physical access to the server (or its disks), they cannot read the mail data without the encryption passphrase.

Backups: Encrypted at Rest

One of ZFS’s killer features for mail hosting is the ability to send encrypted raw streams. The backup server never sees unencrypted data.

# Create a recursive snapshot
zfs snapshot -r zroot@backup_20260118

# Send encrypted stream to offsite backup
# The -w flag sends the raw encrypted blocks
zfs send -R -w -i @backup_20260117 zroot@backup_20260118 | \
    ssh backup-user@backup.example.net zfs recv -F -o mountpoint=none \
    -o canmount=off data1/mailserver

The backup server stores the data in encrypted form. Even if it’s compromised, the attacker only gets encrypted blocks without the key.

Verification

After everything is configured, verify the running services inside the jail:

root@mailstack:/ # ps auxw | grep -E 'postfix|dovecot|rspamd'
root     16095  0.0  1.3 130672 53916  -  SsJ  13:21   0:00.10 rspamd: main process
rspamd   16324  0.0  1.3 131056 54692  -  SJ   13:21   0:00.12 rspamd: rspamd_proxy process
rspamd   16521  0.0  1.4 131964 56244  -  SJ   13:21   0:01.09 rspamd: controller process
rspamd   16619  0.0  1.4 132592 58544  -  SJ   13:21   0:00.58 rspamd: normal process
root     35467  0.0  0.1  15984  4104  -  IsJ  13:21   0:00.15 /usr/local/sbin/dovecot
dovecot  36239  0.0  0.1  15916  3832  -  IJ   13:21   0:00.03 dovecot/anvil
root     69385  0.0  0.4  61164 14668  -  IsJ  13:21   0:00.11 /usr/local/libexec/postfix/master
postfix  69918  0.0  0.4  61188 14676  -  IJ   13:21   0:00.03 pickup -l -t unix -u
postfix  70681  0.0  0.4  61244 14744  -  IJ   13:21   0:00.03 qmgr -l -t unix -u

Test SMTP connectivity:

$ telnet mail.example.org 25
220 mail.example.org ESMTP Postfix
EHLO test.example.com
250-mail.example.org
250-STARTTLS
250-AUTH PLAIN LOGIN
...

Test email deliverability using tools like mail-tester.com or MXToolbox. A properly configured server with SPF, DKIM, and DMARC should score 10/10.

Conclusion

Running a mail server in 2026 on FreeBSD 15.0 feels robust. The combination of PF’s fine-grained control, VNET jails for network isolation, ZFS’s native encryption, and GeoIP filtering creates a platform that is secure by design.

While it requires initial effort to configure correctly - especially getting DNS records (SPF, DKIM, DMARC) aligned - the result is a private, secure communication hub that you truly own. You aren’t mining your own data for ads, and you aren’t subject to the whims of a provider who might lock your account without recourse.

The architecture also makes maintenance straightforward: update the host and jails independently, restore from encrypted backups without exposing plaintext data, and adjust GeoIP rules as your travel patterns change.

Is it more work than using a hosted provider? Absolutely. Is it worth it? For those who value understanding their infrastructure and maintaining control over their communications, unquestionably yes.

References

The mail protocols we use today were designed in an era of mutual trust between operators. That trust has eroded, but the protocols remain remarkably resilient. With careful configuration and modern security layers, self-hosted email is not just viable - it’s a statement of digital autonomy.

Card Wars: Hiding Smartcard Readers from Eager Rust Agents with LD_PRELOAD

2026-01-17T00:00:00+01:00

Running multiple hardware security tokens simultaneously sounds straightforward until you try it. I recently built what I affectionately call a “Workstation from Hell”. A Fedora setup with two distinct smartcard ecosystems that absolutely refuse to get along:

Enterprise Layer: An Aventra MyEID card (PKCS#15) in the internal laptop reader for Kerberos/PKINIT authentication
GPG Layer: A Nitrokey 3 (OpenPGP) for SSH and Git signing, managed by the modern Rust-based openpgp-card-ssh-agent

The MyEID handles enterprise authentication. The Nitrokey handles developer workflows. In theory, they should coexist without issue. In practice, one of them throws a tantrum every time it sees the other.

The Problem: Aggressive Slot Scanning

The Rust OpenPGP stack is modern, fast, and memory-safe. It’s also a bit eager. When the SSH agent starts, it scans all available PC/SC slots looking for OpenPGP cards - a reasonable approach when you’re the only smartcard on the system.

When it encounters my internal Alcor reader containing the MyEID card, things go sideways. The MyEID speaks strict PKCS#15, not OpenPGP. The Rust stack receives unexpected response codes, interprets them as a fatal protocol failure, and crashes:

Jan 17 19:42:43 neochristop openpgp-card-ssh-agent[44822]: [2026-01-17T18:42:43Z ERROR ssh_agent_lib::agent] Error handling message: Proto(UnsupportedCommand { command: 27 })
Jan 17 19:42:43 neochristop openpgp-card-ssh-agent[44822]: [2026-01-17T18:42:43Z ERROR ssh_agent_lib::agent] Error handling message: Failure

The agent doesn’t offer a configuration option to whitelist specific readers. I needed a way to make the internal reader invisible to the Rust agent without affecting the rest of the system - the Kerberos stack still needs full access to the MyEID card.

The Solution: LD_PRELOAD Interception

If the application won’t filter readers, we filter them at the system API level. The PC/SC API uses SCardListReaders to enumerate available smartcard slots. By intercepting this function, we can surgically remove specific readers from the list before the application sees them.

The approach is straightforward:

Create a shared library that intercepts SCardListReaders
Call the real function to get the complete reader list
Remove any entries matching our filter criteria
Return the filtered list to the application

This keeps the Kerberos stack happy (it sees both readers) while the OpenPGP agent only sees the reader it can actually work with.

The Shim Library

Here’s the C code that makes this work:

#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <dlfcn.h>
#include <winscard.h>

// Function pointer matching the PC/SC prototype
typedef LONG (*SCardListReaders_t)(SCARDCONTEXT, LPCSTR, LPSTR, LPDWORD);

LONG SCardListReaders(SCARDCONTEXT hContext, LPCSTR mszGroups,
                      LPSTR mszReaders, LPDWORD pcchReaders) {
    // Load and call the original function
    SCardListReaders_t original_fn =
        (SCardListReaders_t)dlsym(RTLD_NEXT, "SCardListReaders");
    LONG rv = original_fn(hContext, mszGroups, mszReaders, pcchReaders);

    // If the call failed or we're just querying buffer size, return immediately
    if (rv != SCARD_S_SUCCESS || mszReaders == NULL || pcchReaders == NULL) {
        return rv;
    }

    // Filter the multi-string list
    // PC/SC returns readers as "Reader A\0Reader B\0\0"
    char *p = mszReaders;
    char *writer = mszReaders;
    DWORD new_len = 0;
    DWORD total_len = *pcchReaders;

    while (p < mszReaders + total_len) {
        size_t len = strlen(p);
        if (len == 0) break;  // Double null indicates end of list

        // FILTER LOGIC: If "Alcor" is in the name, skip it
        if (strstr(p, "Alcor") != NULL) {
            // Skip this reader (hide it from the application)
        } else {
            // Keep this reader: shift bytes if necessary
            if (writer != p) {
                memmove(writer, p, len + 1);
            }
            writer += len + 1;
            new_len += len + 1;
        }
        p += len + 1;
    }

    // Terminate the new list properly
    *writer = '\0';
    new_len++;

    // Update the length returned to the caller
    *pcchReaders = new_len;

    return rv;
}

The key insight is that PC/SC returns reader names as a multi-string buffer, a sequence of null-terminated strings followed by an extra null byte. The filter walks this structure, copying entries that don’t match the filter pattern while skipping those that do.

Compilation and Installation

Building the shim requires gcc and the PC/SC headers. On Fedora:

sudo dnf install pcsc-lite-devel gcc
mkdir -p ~/.local/lib

gcc -shared -fPIC -o ~/.local/lib/libhackyreaderfix.so \
    hacky_reader_fix.c -ldl -I/usr/include/PCSC

The -shared -fPIC flags create a position-independent shared library suitable for LD_PRELOAD injection. The -ldl links against the dynamic loader library for the dlsym call.

Injecting the Shim

I use systemd --user to manage my SSH agent. To inject the library, create a drop-in override:

systemctl --user edit openpgp-card-ssh-agent.service

Add the following content:

[Service]
Environment="LD_PRELOAD=%h/.local/lib/libhackyreaderfix.so"

The %h specifier expands to the user’s home directory, keeping the configuration portable.

After saving, restart the service:

systemctl --user daemon-reload
systemctl --user restart openpgp-card-ssh-agent

The Result

With the shim in place, the two security stacks are perfectly isolated:

System/Kerberos: Sees both readers, including the internal Alcor with the MyEID card. PKINIT authentication works normally.
SSH Agent: Completely unaware that the Alcor reader exists. Connects directly to the Nitrokey in the external Identiv reader. No more crashes, no more protocol errors.

You can verify the filtering is working using pcsc_scan (from the pcsc-tools package) to see exactly what the shim exposes:

# Without LD_PRELOAD - shows both readers
pcsc_scan
# Reader 0: Identiv uTrust 3512 SAM slot Token [CCID Interface] (55512030603915) 00 00
# Reader 1: Alcor Link AK9563 01 00
# Reader 2: Nitrokey Nitrokey 3 [CCID/ICCD Interface] 02 00

# With LD_PRELOAD - only shows the allowed reader
LD_PRELOAD=~/.local/lib/libhackyreaderfix.so pcsc_scan
# Reader 0: Identiv uTrust 3512 SAM slot Token [CCID Interface] (55512030603915) 00 00
# Reader 1: Nitrokey Nitrokey 3 [CCID/ICCD Interface] 02 00

Extending the Filter

The current implementation hardcodes “Alcor” as the filter pattern. For more flexibility, you could read the pattern from an environment variable:

const char *filter = getenv("PCSC_HIDE_READER");
if (filter != NULL && strstr(p, filter) != NULL) {
    // Skip this reader
}

Then configure it via systemd:

[Service]
Environment="LD_PRELOAD=%h/.local/lib/libhackyreaderfix.so"
Environment="PCSC_HIDE_READER=Alcor"

This makes the shim reusable across different reader combinations without recompilation.

Why Not Upstream a Fix?

The right solution would be for the Rust OpenPGP stack to handle non-OpenPGP cards gracefully. Either by ignoring readers that return unexpected responses or by offering a configuration option to whitelist readers. Both would be reasonable feature requests.

However, the LD_PRELOAD approach solves the immediate problem without waiting for upstream changes, and it works for any application that misbehaves when confronted with unexpected smartcard readers. It’s a general-purpose tool for a specific class of problems.

Caveats

A few things to keep in mind:

Security implications: LD_PRELOAD modifies application behavior at runtime. Only use libraries you control and trust.
Fragility: If the PC/SC API changes significantly, the shim may need updates. In practice, this API has been stable for decades.
Debugging: If something goes wrong, remember the shim is in the path. Temporarily removing the LD_PRELOAD can help isolate issues.

Conclusion

Sometimes the cleanest solution is a dirty hack. LD_PRELOAD interception isn’t elegant, but it’s effective. When two pieces of software refuse to cooperate and neither offers the configuration knobs you need, a small shim library can bridge the gap.

The MyEID handles enterprise authentication. The Nitrokey handles my development workflow. And a few dozen lines of C keep them from stepping on each other’s toes.

References

openpgp-card-ssh-agent - Rust-based OpenPGP SSH agent
PC/SC Lite - Middleware for smartcard access on Unix systems
Aventra MyEID - PKCS#15 compliant smartcards
Nitrokey 3 - Open-source hardware security keys

The best kind of interoperability hack is one you can forget about. Set it up once, and your hardware security layers peacefully ignore each other forever.

GeoIP-Aware Firewalling with PF on FreeBSD

2026-01-13T00:00:00+01:00

Running a mail server on the public internet means dealing with a constant stream of brute-force attempts. Credential stuffers, password sprayers, and opportunistic bots hammer away at IMAP, submission ports, and webmail interfaces around the clock. While fail2ban-style rate limiting helps, the sheer volume of attempts still clutters logs and wastes resources on connection handling.

The solution I’ve implemented takes a different approach: geographic restriction. My users are all located in Central Europe - Germany, Austria, Switzerland, and neighboring countries. There’s no legitimate reason for someone in a botnet-heavy region to connect to my IMAP server. By restricting client-facing ports to IP ranges allocated to specific countries, I’ve dramatically reduced both attack surface and log noise.

The Strategy

The key insight is distinguishing between server-to-server and client-to-server traffic:

SMTP (port 25) must remain globally accessible. Mail servers from anywhere in the world need to deliver messages to my server. Restricting this would break email.
Client ports - IMAP (143), Submission (587), Sieve (4190), and webmail (80/443) - only need to be accessible from where my users actually are.

This creates a two-tier firewall policy: SMTP open to all, everything else restricted by geography.

Architecture Overview

                    Internet
                        |
                        v
            +-----------------------+
            |    PF Firewall        |
            |                       |
            |  <geoip_users> table  |
            |  ~273,000 CIDR blocks |
            +-----------+-----------+
                        |
        +---------------+---------------+
        |                               |
        v                               v
   +----------+                   +-----------+
   | SMTP:25  |                   | Client    |
   | Open to  |                   | Ports     |
   | everyone |                   | GeoIP     |
   +----------+                   | filtered  |
                                  +-----------+

The <geoip_users> table contains approximately 273,000 CIDR blocks covering the allowed countries. PF evaluates incoming connections against this table before allowing access to restricted services.

MaxMind GeoLite2 Database

The geographic data comes from MaxMind’s GeoLite2 database, available free with registration. The CSV format works well for our purposes:

$ ls -lah /usr/local/share/GeoIP/GeoLite2-Country-CSV_20260106
total 11 MB
-rw-r--r--  1 root wheel   22M Jan  6 07:42 GeoLite2-Country-Blocks-IPv4.csv
-rw-r--r--  1 root wheel   27M Jan  6 07:42 GeoLite2-Country-Blocks-IPv6.csv
-rw-r--r--  1 root wheel  9.6K Jan  6 07:42 GeoLite2-Country-Locations-en.csv
...

The database structure is straightforward:

Locations files map geoname IDs to country codes and names
Blocks files map CIDR ranges to geoname IDs

To find all IP ranges for Germany, you first look up Germany’s geoname ID in the locations file, then find all CIDR blocks with that ID in the blocks files.

The Update Script

Processing the GeoLite2 data and loading it into PF requires careful handling. With nearly 300,000 entries, you can’t just dump everything at once - PF will reject the operation with a memory error. The solution is chunked loading:

#!/bin/sh

# Configuration
GEOIP_DIR="/usr/local/share/GeoIP/GeoLite2-Country-CSV_20260106"
PF_FILE="/var/db/pf/geoip_allowed"
TEMP_OUT="/tmp/geoip_allowed.tmp"
TEMP_IDS="/tmp/target_ids.txt"
CHUNK_DIR="/tmp/geoip_chunks"

# Allowed countries (ISO Codes)
COUNTRIES="DE|AT|CH|NL|LU|FR"

# Cleanup
: > "$TEMP_OUT"
rm -rf "$CHUNK_DIR"

echo "Processing GeoIP data for: $COUNTRIES ..."

# Step 1: Extract geoname IDs for target countries
grep -E ",($COUNTRIES)," "$GEOIP_DIR/"*Locations-en.csv | cut -d, -f1 > "$TEMP_IDS"

# Step 2: Extract CIDRs from block files
# IPv4
grep -F -f "$TEMP_IDS" "$GEOIP_DIR/"*Blocks-IPv4.csv | cut -d, -f1 >> "$TEMP_OUT"

# IPv6
grep -F -f "$TEMP_IDS" "$GEOIP_DIR/"*Blocks-IPv6.csv | cut -d, -f1 >> "$TEMP_OUT"

if [ -s "$TEMP_OUT" ]; then
    mkdir -p "$(dirname "$PF_FILE")"
    mv "$TEMP_OUT" "$PF_FILE"

    echo "Loading firewall table in 50k chunks..."

    # Flush existing entries
    pfctl -t geoip_users -T flush

    # Split into manageable chunks
    mkdir -p "$CHUNK_DIR"
    split -l 50000 "$PF_FILE" "$CHUNK_DIR/chunk_"

    # Load each chunk
    for chunk in "$CHUNK_DIR"/chunk_*; do
        pfctl -t geoip_users -T add -f "$chunk"
    done

    # Verify
    COUNT=$(pfctl -t geoip_users -T show | wc -l | tr -d ' ')
    echo "Success: $COUNT networks now active in memory."

    # Cleanup
    rm -rf "$CHUNK_DIR"
    rm -f "$TEMP_IDS"
else
    echo "Error: No data extracted."
    rm -f "$TEMP_OUT"
    exit 1
fi

The script processes both IPv4 and IPv6 ranges, creating a unified list that PF can use for dual-stack filtering.

Kernel Tuning

By default, PF has a relatively low limit on table entries. Loading 273,000 CIDR blocks requires increasing this limit via sysctl.conf:

# Allow large PF tables for GeoIP filtering
net.pf.request_maxcount=500000

This setting must be applied before loading the GeoIP data. After adding it to /etc/sysctl.conf, either reboot or apply it live:

sysctl net.pf.request_maxcount=500000

PF Configuration

Here’s the relevant portion of /etc/pf.conf showing how the GeoIP table integrates with the firewall rules:

# --- Macros ---
ext_if = "vtnet0"
jail_net = "10.0.0.0/24"
jail_net6 = "2001:db8:1c1c:4d2:8000::/65"

mail_ipv4 = "10.0.0.3"
mail_ipv6 = "2001:db8:1c1c:4d2:8000::3"
webmail_ipv6 = "2001:db8:1c1c:4d2:8000::4"

# --- Tables ---
table <bruteforce> persist
table <jails_v4> { $jail_net }
table <jails_v6> { $jail_net6 }
table <geoip_users> persist

# --- Options ---
set limit table-entries 1000000
set skip on lo0
set block-policy drop
set loginterface $ext_if

# --- Scrub ---
scrub in all fragment reassemble
scrub out on $ext_if random-id
scrub out all random-id max-mss 1500

# --- NAT for jails (IPv4 only) ---
nat on $ext_if inet from <jails_v4> to any -> ($ext_if)

# --- RDR for services ---
# Client ports: GeoIP restricted
rdr pass on $ext_if inet proto tcp from <geoip_users> \
    to ($ext_if) port {143, 587, 4190} -> $mail_ipv4

# SMTP: Open to all (server-to-server)
rdr pass on $ext_if inet proto tcp from any \
    to ($ext_if) port 25 -> $mail_ipv4

# --- Filtering ---
block quick from <bruteforce>
block drop in log all
block drop out log all

pass out quick all keep state
antispoof quick for { $ext_if, bridge0 }

# IPv6: Client ports with GeoIP restriction
pass in quick on $ext_if inet6 proto tcp from <geoip_users> \
    to $mail_ipv6 port {143, 587, 4190} flags S/SA keep state

# IPv6: SMTP open to all
pass in quick on $ext_if inet6 proto tcp from any \
    to $mail_ipv6 port 25 flags S/SA keep state

# IPv6: Webmail with GeoIP restriction
pass in quick on $ext_if inet6 proto tcp from <geoip_users> \
    to $webmail_ipv6 port {80, 443} flags S/SA keep state

# Essential ICMPv6
pass in quick inet6 proto ipv6-icmp icmp6-type \
    { echoreq, echorep, neighbrsol, neighbradv, toobig, timex, paramprob }

# ICMP for IPv4
pass in inet proto icmp icmp-type { echoreq, unreach }

# Jail egress
pass in quick on { bridge0 } from <jails_v4> to ! 10.0.0.0/24 keep state
pass in quick on { bridge0 } inet6 from <jails_v6> \
    to ! 2001:db8:1c1c:4d2:8000::/65 keep state

Note the set limit table-entries 1000000 directive - this is the PF-level configuration that complements the sysctl tuning.

The dual approach to traffic handling is visible in the rules:

SMTP (port 25): from any - no source restriction
Client ports (143, 587, 4190, 80, 443): from <geoip_users> - restricted to allowed countries

Verifying the Setup

After loading the GeoIP data, verify the table is populated:

$ pfctl -t geoip_users -T show | wc -l
  273460

$ pfctl -t geoip_users -T show | tail -n 5
   2a1b:3e0:500::/64
   2a1d:3e0:500::/64
   2a1f:3e0:500::/64
   2c0f:2c40:a310::/48
   2c0f:fc04:2000::/64

The loaded rules can be verified with pfctl -s r:

pass in quick on vtnet0 inet6 proto tcp from <geoip_users> \
    to 2001:db8:1c1c:4d2:8000::3 port = imap flags S/SA keep state
pass in quick on vtnet0 inet6 proto tcp from <geoip_users> \
    to 2001:db8:1c1c:4d2:8000::3 port = submission flags S/SA keep state
pass in quick on vtnet0 inet6 proto tcp from any \
    to 2001:db8:1c1c:4d2:8000::3 port = smtp flags S/SA keep state

The rules clearly show the differentiation: IMAP and submission require membership in <geoip_users>, while SMTP allows any source.

Automating Updates

MaxMind updates the GeoLite2 database weekly. A cron job keeps the firewall table current:

# Update GeoIP data weekly (Mondays at 3 AM)
0 3 * * 1 /usr/local/bin/update_geoip_maxmind.sh >> /var/log/geoip_update.log 2>&1

For the database download itself, MaxMind provides a geoipupdate tool that handles authentication and retrieval. The update script then processes the fresh data.

Results

The impact has been significant:

Log volume: Brute-force attempt logs dropped by roughly 90%
Connection overhead: Fewer TCP handshakes to reject means lower resource usage
Signal-to-noise ratio: When something does appear in the logs, it’s more likely to be worth investigating

The remaining attempts typically come from compromised hosts within allowed countries or VPN exit nodes. These still hit rate limiting, but at a manageable volume.

Caveats

This approach isn’t without trade-offs:

Traveling users: Anyone traveling outside the allowed countries will be blocked. Communicate this clearly to users, or consider adding VPN access as a bypass.
VPN services: Users connecting through VPN providers with exit nodes outside allowed countries will be blocked.
Database accuracy: GeoIP data isn’t perfect. Some IP ranges are misattributed, and mobile carriers sometimes route traffic through unexpected locations.
Table size: 273,000 entries consume kernel memory. On memory-constrained systems, consider restricting to fewer countries.

For my use case - a small mail server with users who rarely travel - these trade-offs are acceptable.

Conclusion

Geographic filtering is a powerful layer in a defense-in-depth strategy. It doesn’t replace proper authentication, TLS, or rate limiting, but it dramatically reduces the attack surface by eliminating entire classes of attackers before they even reach the application layer.

PF’s table mechanism handles large GeoIP datasets efficiently, and the chunked loading approach works around memory limitations during updates. Combined with MaxMind’s regularly updated databases, this provides a low-maintenance way to keep your servers visible only where they need to be.

The bots scanning from halfway around the world? They now see nothing but a silent drop. Your logs stay clean, your resources stay available for legitimate users, and your threat surface shrinks to a fraction of what it was.

References

MaxMind GeoLite2 Free Geolocation Data
PF - The OpenBSD Packet Filter (FreeBSD’s PF derives from OpenBSD)
FreeBSD Handbook: Firewalls
FreeBSD pf.conf(5) man page

Sometimes the best security is simply not being there. If an attacker can’t reach your service, they can’t attack it. Geographic filtering makes your server invisible to most of the internet while remaining fully accessible to the people who actually need it.

Managing FreeBSD Jails with Ansible: The jailexec Connection Plugin

2025-12-31T00:00:00+01:00

FreeBSD jails are elegant isolation containers, but managing them with Ansible has traditionally required either running SSH daemons inside each jail or using awkward workarounds. The jailexec connection plugin solves this by connecting to the jail host via SSH and using jexec to execute commands inside jails - just like you would manually.

The Problem with Jail Management

When automating FreeBSD jail infrastructure, you face a choice:

Run SSH in every jail - Works, but defeats much of the security benefit of jails and adds management overhead
Use the jail connection plugin - Requires Ansible to run directly on the jail host, limiting remote management
Manual jexec wrapper scripts - Brittle, hard to maintain, and doesn’t integrate cleanly with Ansible’s ecosystem

What I wanted was simple: SSH to my jail host, then use jexec to run commands inside any jail. That’s exactly what the jailexec plugin does.

How It Works

The plugin extends Ansible’s SSH connection to create a two-hop execution model:

[ Control Machine ] --SSH--> [ Jail Host ] --jexec--> [ Jail ]

When you run an Ansible task against a jail:

The plugin establishes an SSH connection to the jail host
Commands are wrapped with privilege escalation (doas or sudo) and jexec
Output is captured and returned to Ansible as if running directly in the jail
File transfers use a two-stage process: upload to the host, then move into the jail’s filesystem

This means your jails don’t need SSH or any additional services - just a working FreeBSD environment.

Installation

The plugin is a single Python file. Drop it into your Ansible plugins directory:

# User-specific installation
mkdir -p ~/.ansible/plugins/connection/
curl -o jailexec.py https://raw.githubusercontent.com/chofstede/ansible_jailexec/main/jailexec.py
mv jailexec.py ~/.ansible/plugins/connection/

# Or project-specific
mkdir -p connection_plugins/
cp jailexec.py connection_plugins/

For project-specific installation, add to your ansible.cfg:

[defaults]
connection_plugins = ./connection_plugins

Inventory Configuration

The key difference from standard Ansible inventory is specifying the jail host separately from the jail itself:

[freebsd_hosts]
jail-host.example.com ansible_connection=ssh ansible_user=ansible ansible_port=22

[freebsd_jails]
web-jail    ansible_connection=jailexec  ansible_jail_host=jail-host.example.com ansible_user=ansible
db-jail     ansible_connection=jailexec  ansible_jail_host=jail-host.example.com ansible_user=ansible
app-jail    ansible_connection=jailexec  ansible_jail_host=jail-host.example.com ansible_user=ansible

The inventory hostname becomes the jail name by default. SSH authentication settings are inherited from your SSH configuration or can be specified per-host.

Configuration Variables

Variable	Default	Description
`ansible_jail_host`	(required)	FreeBSD host running the jails
`ansible_jail_name`	inventory_hostname	Override jail name
`ansible_jail_user`	`root`	User for command execution in jail
`ansible_jail_privilege_escalation`	`doas`	Privilege escalation method
`ansible_jail_remote_tmp`	`/tmp/.ansible/tmp`	Temporary directory path

Jail Host Setup

The SSH user on your jail host needs privilege escalation configured for jail management commands. I recommend doas for its simplicity:

# Install doas
pkg install doas

# Configure /usr/local/etc/doas.conf
permit nopass ansible as root cmd jls
permit nopass ansible as root cmd jexec
permit nopass ansible as root cmd mkdir
permit nopass ansible as root cmd mv
permit nopass ansible as root cmd rm

For sudo, the equivalent in /usr/local/etc/sudoers:

ansible ALL=(root) NOPASSWD: /usr/sbin/jls, /usr/sbin/jexec, /bin/mkdir, /bin/mv, /bin/rm

Basic Usage

Test connectivity:

ansible -i hosts.ini freebsd_jails -m ping

Expected output:

web-jail | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

Run commands inside jails:

# Check FreeBSD version in all jails
ansible -i hosts.ini freebsd_jails -m command -a "freebsd-version"

# Install packages
ansible -i hosts.ini freebsd_jails -m community.general.pkgng -a "name=nginx state=present"

Example Playbook

A typical playbook for configuring a web server jail:

---
- name: Configure web server jail
  hosts: web-jail
  connection: jailexec
  become: true

  tasks:
    - name: Install nginx
      community.general.pkgng:
        name: nginx
        state: present

    - name: Copy nginx configuration
      ansible.builtin.copy:
        src: files/nginx.conf
        dest: /usr/local/etc/nginx/nginx.conf
        owner: root
        group: wheel
        mode: '0644'
      notify: restart nginx

    - name: Enable and start nginx
      ansible.builtin.service:
        name: nginx
        state: started
        enabled: true

  handlers:
    - name: restart nginx
      ansible.builtin.service:
        name: nginx
        state: restarted

Multi-Environment Inventory

For more complex setups with multiple jail hosts:

[production_jails]
prod-web-01   ansible_connection=jailexec  ansible_jail_host=prod-host-01.example.com
prod-web-02   ansible_connection=jailexec  ansible_jail_host=prod-host-02.example.com
prod-db-01    ansible_connection=jailexec  ansible_jail_host=prod-host-01.example.com  ansible_jail_user=postgres

[staging_jails]
stage-web     ansible_connection=jailexec  ansible_jail_host=stage-host.example.com
stage-db      ansible_connection=jailexec  ansible_jail_host=stage-host.example.com

[all_jails:children]
production_jails
staging_jails

Security Design

The plugin implements several security measures:

Input validation: Jail names are validated against FreeBSD naming conventions. Path traversal attempts (..) and shell injection patterns are blocked.

Two-stage file transfers: Files are first uploaded to a temporary location the SSH user can access, then moved into the jail using privilege escalation. This prevents direct writes to privileged paths.

Secure temporary files: Temporary files use mode 600 and are cleaned up on failure.

Connection reuse: SSH connections are pooled, reducing authentication overhead and exposure.

Privilege escalation control: You choose between doas or sudo, and can lock down exactly which commands are permitted.

Troubleshooting

Enable verbose output for debugging:

# Maximum verbosity
ansible -vvv -i hosts.ini freebsd_jails -m ping

# With Ansible debugging
ANSIBLE_DEBUG=1 ansible -vvv -i hosts.ini freebsd_jails -m ping

Common Issues

“No jail host specified” Add ansible_jail_host=your-host.example.com to your inventory.

“Permission denied accessing jail” Verify doas/sudo configuration. Test manually on the jail host:

doas jls
doas jexec web-jail /bin/sh -c "id"

“Jail ‘name’ not found” Check that the jail is running:

jls
service jail onestart web-jail

File transfer failures Verify the SSH user can write to /tmp on the jail host and that disk space is available.

Future Development

The plugin is actively maintained. Potential future additions:

Ansible Collection packaging for easier distribution
Support for jail templates and cloning operations
Parallel execution optimizations

Contributions are welcome on GitHub or Codeberg.

Conclusion

The jailexec plugin fills a gap in FreeBSD automation tooling. It brings proper Ansible support to jail management without compromising the security benefits of running minimal jail environments.

If you’re managing FreeBSD jails at any scale, this plugin lets you treat them as first-class Ansible targets - the same playbooks, roles, and patterns you use everywhere else, now working seamlessly with FreeBSD’s native containerization.

References

FreeBSD 15 Cloud-Init on Proxmox: Working Around nuageinit’s Network-Config Gap

2025-12-28T00:00:00+01:00

FreeBSD 15.0-RELEASE shipped earlier this month with VM images that are “cloud-init compatible” with a catch. Instead of using the Python-based cloud-init that Linux distributions rely on, FreeBSD implemented their own solution: nuageinit(7). It’s lighter, written in Lua, and fits FreeBSD’s philosophy of minimal dependencies. The only problem? It doesn’t understand network-config version 1.

This matters because Proxmox VE-even in version 9.1 still generates network-config v1 format when you configure cloud-init through the web UI. The result: your carefully configured static IP settings are silently ignored, and the VM falls back to DHCP. For production environments where predictable addressing matters, this is a deal-breaker.

The Version Mismatch

Cloud-init’s network configuration has evolved through several versions:

Version 1: The original format, with nested dictionaries and explicit type fields
Version 2 (Netplan): A cleaner YAML structure that became the de facto standard

Here’s what Proxmox generates (v1):

version: 1
config:
  - type: physical
    name: vtnet0
    subnets:
      - type: static
        address: 10.254.254.42/24
        gateway: 10.254.254.1

And what nuageinit expects (v2):

version: 2
ethernets:
  vtnet0:
    addresses:
      - 10.254.254.42/24
    gateway4: 10.254.254.1

The structures are fundamentally different. nuageinit simply doesn’t parse v1, leaving your VM with DHCP-assigned addresses instead of the static configuration you specified.

The Workaround

Until Proxmox adds v2 support or nuageinit gains v1 compatibility, the solution is to generate your own cloud-init ISO. The script below runs on a Proxmox host and creates a properly formatted ISO that FreeBSD will actually understand.

Installation

Save the script as /usr/local/bin/freebsd-cloudinit-iso and make it executable:

chmod +x /usr/local/bin/freebsd-cloudinit-iso

Dependencies are minimal-just genisoimage or mkisofs, which are typically already present on Proxmox hosts.

Basic Usage

freebsd-cloudinit-iso \
    --vmid 203 \
    --hostname myfreebsd \
    --domain example.com \
    --user admin \
    --password 'secretpassword' \
    --ip4 10.254.254.42/24 \
    --gw4 10.254.254.1 \
    --dns 1.1.1.1 \
    --dns 1.0.0.1 \
    --storage local

This creates an ISO with:

Hostname set to myfreebsd.example.com
A user admin with sudo privileges and the specified password
Static IPv4 configuration
Cloudflare DNS servers

The ISO is automatically attached to VM 203 as a CD-ROM drive.

Dual-Stack Configuration

For IPv6-first environments:

freebsd-cloudinit-iso \
    --vmid 204 \
    --hostname webserver \
    --domain prod.example.com \
    --user deploy \
    --ssh-key-file ~/.ssh/id_ed25519.pub \
    --ip6 2001:db8::50/64 \
    --gw6 2001:db8::1 \
    --ip4 10.0.0.50/24 \
    --gw4 10.0.0.1 \
    --dns 2001:db8::1 \
    --dns 10.0.0.1 \
    --storage ceph-nvme

Note that IPv6 addresses are listed first in the generated config-nuageinit applies them in order, and IPv6-first makes sense for modern networks.

SSH Key Authentication

For production deployments, prefer SSH keys over passwords:

freebsd-cloudinit-iso \
    --vmid 205 \
    --hostname secure \
    --domain internal.lan \
    --user ansible \
    --ssh-key-file ~/.ssh/automation.pub \
    --ip4 10.100.0.10/24 \
    --gw4 10.100.0.1 \
    --dns 10.100.0.1 \
    --storage local-zfs

You can also pass the key directly with --ssh-key "ssh-ed25519 AAAA...".

What the Script Generates

The script creates three files in the ISO:

meta-data

instance-id: 203-1735400000
local-hostname: myfreebsd

The instance-id uses the VM ID and timestamp, ensuring cloud-init recognizes each boot as potentially needing reconfiguration.

user-data

#cloud-config
hostname: myfreebsd
fqdn: myfreebsd.example.com
manage_etc_hosts: true

users:
  - name: admin
    groups: wheel
    shell: /bin/sh
    sudo: ALL=(ALL) NOPASSWD:ALL
    lock_passwd: false
    passwd: $6$rounds=5000$...
    ssh_authorized_keys:
      - ssh-ed25519 AAAA...

ssh_pwauth: true
chpasswd:
  expire: false

Passwords are hashed using SHA-512 before being written-plaintext credentials never touch the ISO.

network-config

version: 2
ethernets:
  vtnet0:
    addresses:
      - 2001:db8::50/64
      - 10.0.0.50/24
    gateway6: 2001:db8::1
    gateway4: 10.0.0.1
    nameservers:
      addresses:
        - 2001:db8::1
        - 10.0.0.1
      search:
        - prod.example.com

This is the crucial part-version 2 format that nuageinit actually parses.

Full Option Reference

Required:
  --vmid <id>           VM ID in Proxmox
  --hostname <name>     Hostname (without domain)
  --user <username>     Username to create

Network (at least one required):
  --ip4 <addr/prefix>   IPv4 address with prefix (e.g., 10.0.0.50/24)
  --gw4 <addr>          IPv4 gateway
  --ip6 <addr/prefix>   IPv6 address with prefix
  --gw6 <addr>          IPv6 gateway

Authentication (at least one required):
  --password <pass>       Password for user (will be hashed)
  --root-password <pass>  Password for root (will be hashed)
  --ssh-key <key>         SSH public key string
  --ssh-key-file <file>   Path to SSH public key file

DNS:
  --dns <addr>          DNS server (repeatable)
  --search <domain>     Search domain (repeatable)
  --domain <domain>     Domain (also added to search)

Storage:
  --storage <name>      Proxmox storage for ISO (default: auto-detect)

Options:
  --iface <name>        Network interface (default: vtnet0)
  --groups <groups>     Comma-separated groups (default: wheel)
  --output-dir <dir>    Output directory instead of storage
  --no-attach           Create ISO but don't attach to VM

Workflow Integration

With Terraform/OpenTofu

If you’re provisioning FreeBSD VMs with Terraform, call the script as a local-exec provisioner after creating the VM but before first boot:

resource "proxmox_vm_qemu" "freebsd" {
  name        = "freebsd-web"
  vmid        = 210
  target_node = "pve1"
  # ... other config ...
}

resource "null_resource" "cloudinit" {
  depends_on = [proxmox_vm_qemu.freebsd]

  provisioner "local-exec" {
    command = <<-EOT
      ssh root@pve1 'freebsd-cloudinit-iso \
        --vmid 210 \
        --hostname freebsd-web \
        --domain example.com \
        --user terraform \
        --ssh-key-file /root/.ssh/terraform.pub \
        --ip4 10.0.0.210/24 \
        --gw4 10.0.0.1 \
        --dns 10.0.0.1 \
        --storage local'
    EOT
  }
}

With Ansible

For Ansible-driven provisioning:

- name: Generate FreeBSD cloud-init ISO
  delegate_to: "{{ proxmox_host }}"
  command: >
    freebsd-cloudinit-iso
    --vmid {{ vm_id }}
    --hostname {{ inventory_hostname_short }}
    --domain {{ domain }}
    --user {{ ansible_user }}
    --ssh-key-file /root/.ssh/ansible.pub
    --ip4 {{ ansible_host }}/24
    --gw4 {{ gateway }}
    --dns {{ dns_server }}
    --storage {{ proxmox_storage }}

Limitations and Caveats

Single interface only: The script currently supports one network interface. Multiple NICs would require extending the network-config generation.

No DHCP option: This is intentionally for static configurations. If you want DHCP, the default Proxmox cloud-init works fine (nuageinit falls back to DHCP when it can’t parse the network config).

VirtIO interface name: The default vtnet0 assumes VirtIO network devices. If you’re using emulated e1000 or another driver, specify --iface accordingly.

Storage detection: Auto-detection looks for storage with iso content type. If your Proxmox setup is non-standard, specify --storage explicitly.

Why Not Just Fix Proxmox?

Fair question. There’s an open feature request for network-config v2 support in Proxmox, but it’s not trivial-the entire cloud-init subsystem assumes v1 internally. FreeBSD’s nuageinit could also add v1 parsing, but the project explicitly chose v2 as the modern format.

In the meantime, this script fills the gap. It’s not elegant, but it works reliably and integrates cleanly with existing Proxmox workflows.

The Script

The full script is available below. It’s a single Bash file with no external dependencies beyond standard Proxmox tools:

#!/bin/bash
#
# freebsd-cloudinit-iso - Generate cloud-init ISO for FreeBSD VMs on Proxmox
#
# This works around FreeBSD 15's nuageinit not supporting network-config v1
# by generating v2 format that nuageinit actually understands.

set -e

# Defaults
VMID=""
HOSTNAME=""
DOMAIN=""
USER=""
PASSWORD=""
ROOT_PASSWORD=""
SSH_KEY=""
SSH_KEY_FILE=""
IP4=""
GW4=""
IP6=""
GW6=""
DNS=()
SEARCH=()
STORAGE=""
IFACE="vtnet0"
OUTPUT_DIR=""
ATTACH=true
USER_GROUPS="wheel"

usage() {
    cat <<EOF
Usage: $(basename "$0") [options]

Required:
  --vmid <id>           VM ID
  --hostname <name>     Hostname (without domain)
  --user <username>     Username to create

Network (at least one of --ip4 or --ip6 required):
  --ip4 <addr/prefix>   IPv4 address with prefix (e.g., 10.0.0.50/24)
  --gw4 <addr>          IPv4 gateway
  --ip6 <addr/prefix>   IPv6 address with prefix (e.g., 2001:db8::50/64)
  --gw6 <addr>          IPv6 gateway

Authentication (at least one required):
  --password <pass>     Password for user (plaintext, will be hashed)
  --root-password <pass> Password for root (plaintext, will be hashed)
  --ssh-key <key>       SSH public key string
  --ssh-key-file <file> Path to SSH public key file

DNS:
  --dns <addr>          DNS server (can be specified multiple times)
  --search <domain>     Search domain (can be specified multiple times)
  --domain <domain>     Domain name (also added to search if not present)

Storage:
  --storage <name>      Proxmox storage name (default: auto-detect)

Options:
  --iface <name>        Network interface name (default: vtnet0)
  --groups <groups>     Comma-separated groups (default: wheel)
  --output-dir <dir>    Output directory for ISO (default: storage path)
  --no-attach           Don't attach ISO to VM, just create it
  --help                Show this help
EOF
    exit 1
}

error() {
    echo "Error: $1" >&2
    exit 1
}

warn() {
    echo "Warning: $1" >&2
}

# Parse arguments
while [[ $# -gt 0 ]]; do
    case $1 in
        --vmid)       VMID="$2"; shift 2 ;;
        --hostname)   HOSTNAME="$2"; shift 2 ;;
        --domain)     DOMAIN="$2"; shift 2 ;;
        --user)       USER="$2"; shift 2 ;;
        --password)   PASSWORD="$2"; shift 2 ;;
        --root-password) ROOT_PASSWORD="$2"; shift 2 ;;
        --ssh-key)    SSH_KEY="$2"; shift 2 ;;
        --ssh-key-file) SSH_KEY_FILE="$2"; shift 2 ;;
        --ip4)        IP4="$2"; shift 2 ;;
        --gw4)        GW4="$2"; shift 2 ;;
        --ip6)        IP6="$2"; shift 2 ;;
        --gw6)        GW6="$2"; shift 2 ;;
        --dns)        DNS+=("$2"); shift 2 ;;
        --search)     SEARCH+=("$2"); shift 2 ;;
        --storage)    STORAGE="$2"; shift 2 ;;
        --iface)      IFACE="$2"; shift 2 ;;
        --groups)     USER_GROUPS="$2"; shift 2 ;;
        --output-dir) OUTPUT_DIR="$2"; shift 2 ;;
        --no-attach)  ATTACH=false; shift ;;
        --help)       usage ;;
        *)            error "Unknown option: $1" ;;
    esac
done

# Validation
[[ -z "$VMID" ]] && error "--vmid is required"
[[ -z "$HOSTNAME" ]] && error "--hostname is required"
[[ -z "$USER" ]] && error "--user is required"
[[ -z "$IP4" && -z "$IP6" ]] && error "At least one of --ip4 or --ip6 is required"
[[ -z "$PASSWORD" && -z "$SSH_KEY" && -z "$SSH_KEY_FILE" ]] && \
    error "At least one of --password, --ssh-key, or --ssh-key-file is required"

# Check for required tools
command -v genisoimage >/dev/null 2>&1 || \
    command -v mkisofs >/dev/null 2>&1 || \
    error "genisoimage or mkisofs required"
command -v qm >/dev/null 2>&1 || warn "qm not found - not running on Proxmox?"

# Load SSH key from file if specified
if [[ -n "$SSH_KEY_FILE" ]]; then
    [[ -f "$SSH_KEY_FILE" ]] || error "SSH key file not found: $SSH_KEY_FILE"
    SSH_KEY=$(cat "$SSH_KEY_FILE")
fi

# Find storage path
find_storage_path() {
    local storage="$1"

    if [[ -n "$OUTPUT_DIR" ]]; then
        echo "$OUTPUT_DIR"
        return
    fi

    if [[ -z "$storage" ]]; then
        storage=$(pvesm status --content iso 2>/dev/null | awk 'NR>1 {print $1; exit}')
        [[ -z "$storage" ]] && \
            error "No storage with 'iso' content type found. Specify --storage or --output-dir"
    fi

    local path=$(pvesm path "${storage}:iso/dummy.iso" 2>/dev/null | sed 's|/dummy\.iso$||')
    [[ -z "$path" ]] && \
        error "Could not determine path for storage: $storage"

    STORAGE="$storage"
    echo "$path"
}

SNIPPETS_PATH=$(find_storage_path "$STORAGE")
mkdir -p "$SNIPPETS_PATH"

# Create temp directory
TMPDIR=$(mktemp -d)
trap "rm -rf '$TMPDIR'" EXIT

# Generate password hashes
PASSWORD_HASH=""
if [[ -n "$PASSWORD" ]]; then
    PASSWORD_HASH=$(openssl passwd -6 "$PASSWORD" 2>/dev/null) || \
    PASSWORD_HASH=$(python3 -c "import crypt; print(crypt.crypt('$PASSWORD', crypt.mksalt(crypt.METHOD_SHA512)))" 2>/dev/null) || \
    error "Could not hash password"
fi

ROOT_PASSWORD_HASH=""
if [[ -n "$ROOT_PASSWORD" ]]; then
    ROOT_PASSWORD_HASH=$(openssl passwd -6 "$ROOT_PASSWORD" 2>/dev/null) || \
    ROOT_PASSWORD_HASH=$(python3 -c "import crypt; print(crypt.crypt('$ROOT_PASSWORD', crypt.mksalt(crypt.METHOD_SHA512)))" 2>/dev/null) || \
    error "Could not hash root password"
fi

# Build FQDN
FQDN="$HOSTNAME"
[[ -n "$DOMAIN" ]] && FQDN="${HOSTNAME}.${DOMAIN}"

# Add domain to search if not present
if [[ -n "$DOMAIN" ]]; then
    domain_in_search=false
    for s in "${SEARCH[@]}"; do
        [[ "$s" == "$DOMAIN" ]] && domain_in_search=true
    done
    $domain_in_search || SEARCH=("$DOMAIN" "${SEARCH[@]}")
fi

# Create meta-data
cat > "$TMPDIR/meta-data" <<EOF
instance-id: $(uuidgen 2>/dev/null || echo "${VMID}-$(date +%s)")
local-hostname: ${HOSTNAME}
EOF

# Create user-data
cat > "$TMPDIR/user-data" <<EOF
#cloud-config
hostname: ${HOSTNAME}
fqdn: ${FQDN}
manage_etc_hosts: true

users:
  - name: ${USER}
    groups: ${USER_GROUPS}
    shell: /bin/sh
    sudo: ALL=(ALL) NOPASSWD:ALL
EOF

if [[ -n "$PASSWORD_HASH" ]]; then
    cat >> "$TMPDIR/user-data" <<EOF
    lock_passwd: false
    passwd: ${PASSWORD_HASH}
EOF
fi

if [[ -n "$SSH_KEY" ]]; then
    cat >> "$TMPDIR/user-data" <<EOF
    ssh_authorized_keys:
      - ${SSH_KEY}
EOF
fi

if [[ -n "$PASSWORD_HASH" ]]; then
    cat >> "$TMPDIR/user-data" <<EOF

ssh_pwauth: true
chpasswd:
  expire: false
EOF
fi

if [[ -n "$ROOT_PASSWORD_HASH" ]]; then
    cat >> "$TMPDIR/user-data" <<EOF

chpasswd:
  expire: false
  users:
    - name: root
      password: ${ROOT_PASSWORD_HASH}
      type: HASH
EOF
fi

# Create network-config (v2 format)
cat > "$TMPDIR/network-config" <<EOF
version: 2
ethernets:
  ${IFACE}:
EOF

echo "    addresses:" >> "$TMPDIR/network-config"
[[ -n "$IP6" ]] && echo "      - ${IP6}" >> "$TMPDIR/network-config"
[[ -n "$IP4" ]] && echo "      - ${IP4}" >> "$TMPDIR/network-config"

[[ -n "$GW6" ]] && echo "    gateway6: ${GW6}" >> "$TMPDIR/network-config"
[[ -n "$GW4" ]] && echo "    gateway4: ${GW4}" >> "$TMPDIR/network-config"

if [[ ${#DNS[@]} -gt 0 ]] || [[ ${#SEARCH[@]} -gt 0 ]]; then
    echo "    nameservers:" >> "$TMPDIR/network-config"

    if [[ ${#DNS[@]} -gt 0 ]]; then
        echo "      addresses:" >> "$TMPDIR/network-config"
        for dns in "${DNS[@]}"; do
            echo "        - ${dns}" >> "$TMPDIR/network-config"
        done
    fi

    if [[ ${#SEARCH[@]} -gt 0 ]]; then
        echo "      search:" >> "$TMPDIR/network-config"
        for search in "${SEARCH[@]}"; do
            echo "        - ${search}" >> "$TMPDIR/network-config"
        done
    fi
fi

# Generate ISO
ISO_NAME="freebsd-ci-${VMID}.iso"
ISO_PATH="${SNIPPETS_PATH}/${ISO_NAME}"

if command -v genisoimage >/dev/null 2>&1; then
    MKISO="genisoimage"
else
    MKISO="mkisofs"
fi

$MKISO -output "$ISO_PATH" -volid cidata -joliet -rock "$TMPDIR" 2>/dev/null

echo "Created: ${ISO_PATH}"

# Attach to VM
if $ATTACH && command -v qm >/dev/null 2>&1; then
    if qm status "$VMID" >/dev/null 2>&1; then
        qm set "$VMID" --delete ide2 2>/dev/null || true
        qm set "$VMID" --ide2 "${STORAGE}:iso/${ISO_NAME},media=cdrom"
        echo "Attached to VM ${VMID} as ide2"
    else
        warn "VM ${VMID} not found, ISO created but not attached"
    fi
fi

Conclusion

FreeBSD 15’s nuageinit and Proxmox’s cloud-init generator speak different dialects of the same configuration language. Until upstream fixes arrive, this script provides a reliable workaround for anyone running FreeBSD VMs on Proxmox who needs static network configuration.

The approach is straightforward: generate the ISO yourself with the correct format. It integrates with existing automation tools, requires no modifications to either Proxmox or FreeBSD, and produces predictable results.

References

Interactive System Troubleshooting with AI: The Linux MCP Server

2025-12-25T00:00:00+01:00

Troubleshooting Linux systems traditionally involves a familiar dance: run a command, copy the output, paste it into a chat with an AI assistant, wait for suggestions, repeat. The linux-mcp-server changes this paradigm entirely. Instead of being a passive advisor, your AI assistant becomes an active participant that can directly query your systems, correlate information across multiple data sources, and provide contextual analysis in real-time.

What is MCP?

The Model Context Protocol (MCP) is an open standard introduced by Anthropic in November 2024. It defines how AI models can interact with external tools and data sources in a structured, secure way. Think of it as a standardized API that lets AI assistants reach beyond their training data to access live information.

MCP servers expose specific capabilities - tools that the AI can invoke when needed. The AI decides when and how to use these tools based on the conversation context, making interactions feel natural rather than scripted.

Enter linux-mcp-server

The linux-mcp-server is a Red Hat project that implements MCP for Linux system administration. It provides read-only diagnostic tools covering:

System information: OS details, CPU, memory, disk usage, hardware specs
Service management: systemd service status, logs, and configuration
Process monitoring: Running processes with resource metrics
Log analysis: Journal queries, audit logs, application logs
Network diagnostics: Interfaces, connections, listening ports
Storage analysis: Block devices, mount points, directory listings

The key design principle is safety: all operations are read-only. The server can inspect your system but cannot modify it, making it safe to use even on production systems.

Installation

The server requires Python 3.10+ and can be installed several ways:

# Via pip
pip install linux-mcp-server

# Via uv (recommended for isolation)
uv tool install linux-mcp-server

# Or run directly without installing
uvx linux-mcp-server

For Claude Desktop or Claude Code integration, add it to your MCP configuration:

{
  "mcpServers": {
    "linux-mcp-server": {
      "command": "uvx",
      "args": ["linux-mcp-server"]
    }
  }
}

Local and Remote Systems

One of the most powerful features is transparent SSH support. Every tool accepts an optional host parameter. Without it, commands run locally. With it, they execute on the remote system via SSH:

# Query local system
get_system_info()

# Query remote system
get_system_info(host="webserver.example.com")

SSH authentication uses key-based auth exclusively (no passwords), reading keys from ~/.ssh/ or a path specified via LINUX_MCP_SSH_KEY_PATH. Your existing ~/.ssh/config is respected, so per-host usernames, ports, and jump hosts work automatically.

Real-World Example: Diagnosing a Remote Server

Here’s where things get interesting. Instead of explaining what the tools do, let me demonstrate by querying a real system. The following analysis was performed live on a server named “janeway”:

System Overview

When I ask the AI to check the system, it queries the server directly:

“What can you tell me about the janeway server?”

Hostname: janeway.home.hofstede.it
Operating System: Red Hat Enterprise Linux 10.1 (Coughlan)
Kernel Version: 6.12.0-124.21.1.el10_1.x86_64
Architecture: x86_64
Uptime: up 5 days, 3 hours, 59 minutes
Boot Time: 2025-12-20 12:54:29

CPU Model: Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz
CPU Cores: 2 logical (1 physical)
Load Average: 0.37, 0.22, 0.13

This is a RHEL 10.1 VM running on a Proxmox host, with modest resources allocated for its workload. The AI can immediately see this is a stable system - low load averages and over 5 days of uptime.

Service Health Check

“Is the Omada controller running properly?”

The server runs a TP-Link Omada network controller as a Podman container managed through systemd Quadlets:

● omada-controller.service - TP-Link Omada Controller (Podman Quadlet)
     Loaded: loaded (/etc/containers/systemd/omada-controller.container; generated)
     Active: active (running) since Sat 2025-12-20 12:54:56 CET; 5 days ago
   Main PID: 1398 (conmon)
      Tasks: 317 (limit: 22891)
     Memory: 2.1G (peak: 2.1G)
        CPU: 8h 6min 41.871s
     CGroup: /system.slice/omada-controller.service
             ├─libpod-payload-6cbbd984675b...
             │ ├─1413 java -server -Xms128m -Xmx1024m ...
             │ ├─1811 mongod --port 27217 --dbpath ../data/db ...

The AI correlates multiple data points: the service is active, has been running since boot (matching the system uptime), and is using about 2.1GB of memory for its Java process and embedded MongoDB. It can also see recent log entries showing normal operation - firmware checks running on schedule, topology updates proceeding without errors.

Resource Analysis

“How are resources looking? Any concerns?”

=== RAM Information ===
Total: 3.6GB
Available: 1.4GB
Used: 2.2GB (61.7%)
Free: 550.8MB

=== Swap Information ===
Total: 3.9GB
Used: 0.0B (0.0%)
Free: 3.9GB

=== Filesystem Usage ===
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/rhel-root   35G  4.0G   31G  12% /
/dev/vda2              960M  448M  513M  47% /boot

The AI provides contextual analysis: “Memory usage at 62% is healthy for a server running a Java application with MongoDB. The fact that swap is completely unused indicates no memory pressure - the system has headroom. Disk usage at 12% on root is excellent, though /boot at 47% might warrant attention during future kernel updates.”

This kind of interpretation is where the conversational approach shines. The raw numbers are available, but the AI adds the “so what” that helps you understand whether action is needed.

Network Visibility

“What services are exposed on this machine?”

=== Listening Ports ===
tcp  LISTEN  0.0.0.0:22     SSH
tcp  LISTEN  0.0.0.0:111    rpcbind
tcp  LISTEN  0.0.0.0:8043   Omada Controller (HTTPS management)
tcp  LISTEN  0.0.0.0:8088   Omada Controller (HTTP redirect)
tcp  LISTEN  0.0.0.0:8843   Omada Controller (HTTPS portal)
tcp  LISTEN  0.0.0.0:29810-29814  Omada device communication
udp  UNCONN  0.0.0.0:29810  Omada device discovery

The AI maps the ports to their purposes: “This is a dedicated Omada controller. SSH is available for administration. The Omada-specific ports (8043, 8088, 8843, 29810-29814) are for the controller’s web interface and communication with managed network devices. I also notice port 111 (rpcbind) is listening - this is typically for NFS support. If NFS isn’t actively used, consider whether this service needs to be running.”

The Conversational Advantage

The real power isn’t in individual commands - it’s in the conversation. Consider this exchange:

User: “My web server seems slow. Can you check what’s going on?”

The AI assistant can now:

Query system load and memory pressure
Check if the web server service is running properly
Examine recent logs for errors
Look at network connections and listening ports
Correlate all this information and provide a coherent diagnosis

All without the user needing to know which specific commands to run or how to interpret the raw output. The assistant contextualizes numbers that might be meaningless to a casual user: “Memory usage is at 89%, which is high but not critical. However, I notice significant swap activity which could explain the perceived slowness.”

Security Considerations

The linux-mcp-server takes a defense-in-depth approach:

Read-only operations: No tool can modify system state
Predefined tools only: Arbitrary command execution is not possible
Log path allowlisting: Only explicitly permitted log files can be read (controlled via LINUX_MCP_ALLOWED_LOG_PATHS)
Parameter validation: Inputs are sanitized to prevent injection attacks
Audit logging: All operations are logged for accountability

For remote access, standard SSH security applies. The server uses your existing SSH configuration and key-based authentication, inheriting whatever access controls you’ve already established.

Configuration Options

Environment variables control server behavior:

Variable	Purpose
`LINUX_MCP_ALLOWED_LOG_PATHS`	Comma-separated list of accessible log file paths
`LINUX_MCP_LOG_LEVEL`	Logging verbosity (DEBUG, INFO, WARNING, ERROR)
`LINUX_MCP_SSH_KEY_PATH`	Path to SSH private key for remote connections
`LINUX_MCP_USER`	Default SSH username for remote connections

Available Tools

The server exposes these diagnostic capabilities:

Tool	Description
`get_system_information`	OS version, kernel, hostname, uptime
`get_cpu_information`	CPU model, cores, current load
`get_memory_information`	RAM usage, swap, available memory
`get_disk_usage`	Filesystem usage, mount points
`get_hardware_information`	PCI/USB devices, DMI data
`list_services`	All systemd services and their states
`get_service_status`	Detailed status of a specific service
`get_service_logs`	Journal entries for a service
`list_processes`	Running processes with resource usage
`get_process_info`	Detailed info about a specific PID
`get_network_interfaces`	Network interfaces and addresses
`get_network_connections`	Active network connections
`get_listening_ports`	Services listening on ports
`get_journal_logs`	Systemd journal queries
`get_audit_logs`	Security audit log entries
`read_log_file`	Read from allowlisted log files
`list_block_devices`	Disk and partition information
`list_files` / `list_directories`	Directory contents with metadata

Use Cases

Beyond ad-hoc troubleshooting, the linux-mcp-server enables:

Pre-upgrade assessment: “Check if this server is ready for a major OS upgrade” - the AI can examine repositories, disk space, running services, and potential compatibility issues.

Capacity planning: “Analyze resource usage trends” - correlate CPU, memory, and disk metrics to identify bottlenecks before they become problems.

Security auditing: “Review what services are exposed to the network” - examine listening ports, running services, and their configurations.

Documentation: “Summarize this server’s configuration” - generate human-readable descriptions of system setup for documentation purposes.

Limitations

The current implementation has intentional constraints:

No write operations: You can’t restart services, edit files, or make changes through the MCP server
RHEL/systemd focus: Optimized for Red Hat-based distributions; some tools may not work on non-systemd systems
Smaller models need verification: While Claude and other large models provide reliable guidance, smaller open-source models may produce suggestions that require verification

Getting Started

Install the server: pip install linux-mcp-server or uv tool install linux-mcp-server
Configure your AI client (Claude Desktop, Claude Code, or another MCP-compatible client)
Start asking questions about your systems

The barrier to entry is remarkably low. If you can SSH to a machine, the linux-mcp-server can query it.

Conclusion

The linux-mcp-server represents a meaningful shift in how we interact with systems. Instead of the AI being a sophisticated search engine that helps you find the right commands, it becomes a collaborative partner that can actually see your system’s state.

This isn’t about replacing sysadmin knowledge - you still need to understand what the AI is telling you and make informed decisions. But it dramatically lowers the friction of system diagnostics, especially for:

Developers who occasionally need to debug production issues
Junior administrators learning the ropes
Anyone managing systems outside their primary expertise
Experienced admins who want faster initial triage

The project is open source and actively developed. Contributions for additional diagnostic tools, support for other distributions, and integration with documentation via RAG are welcome.

References

Thanks to the Red Hat and Fedora teams for developing linux-mcp-server and making AI-assisted system administration a reality. The future of troubleshooting is conversational.

Running a Factorio Headless Server on FreeBSD with the Linuxulator

2025-12-20T00:00:00+01:00

Factorio doesn’t have a native FreeBSD build, but that doesn’t mean you can’t run it on FreeBSD. The Linuxulator, FreeBSD’s Linux binary compatibility layer, handles Linux ELF binaries seamlessly. This article walks through setting up a Factorio headless server inside a Bastille jail, complete with firewall rules for public access.

Prerequisites

This guide assumes you already have:

A FreeBSD host (14.x or later) with Bastille installed
A working bridge network for jails (see my previous article on FreeBSD jail infrastructure)
A Factorio account to download the headless server

Enabling the 64-bit Linuxulator

The Factorio server is a 64-bit Linux binary. FreeBSD’s Linuxulator needs to be loaded on the host system before jails can use it.

Add to /boot/loader.conf on the host:

linux64_load="YES"

Reboot, or load it immediately:

kldload linux64

You can verify it’s loaded:

$ kldstat | grep linux
 9    1 0xffffffff827fa000    619f8 linux64.ko
10    2 0xffffffff8285c000    20970 linux_common.ko

Creating the Jail

Create a standard VNET jail attached to your existing bridge. In this example, I’m using the bastille0 bridge with a private IPv4/IPv6 address pair:

bastille create -B factorio 14.3-RELEASE 10.254.252.98 bastille0

Configure the jail’s network in its /etc/rc.conf:

ifconfig_e0b_factorio_name="vnet0"
ifconfig_vnet0="inet 10.254.252.98 netmask 255.255.255.0"
ifconfig_vnet0_ipv6="inet6 2001:db8:8000::98/64"
defaultrouter="10.254.252.1" # That's the hosts IP address on the bridge bastille0
ipv6_defaultrouter="2001:db8:8000::1" # That's the hosts IPv6 address on the bridge bastille0

syslogd_flags="-ss"
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"
cron_flags="-J 60"

linux_enable="YES"

The key line is linux_enable="YES" - this enables the Linux compatibility layer inside the jail.

Start the jail:

bastille start factorio

Installing Linux Userland

Inside the jail, install the Rocky Linux 9 base package. This provides the necessary Linux shared libraries:

bastille cmd factorio pkg install -y linux_base-rl9

This pulls in a minimal Linux userland including glibc, which the Factorio binary needs. The installed packages are lightweight:

linux_base-rl9-9.6_1           Base set of packages needed in Linux mode (Rocky Linux 9.6)

Creating the Factorio User

Create a dedicated user to run the server:

bastille cmd factorio adduser

Follow the prompts - the defaults are fine. I used factorio as the username with /home/factorio as the home directory. When prompted for the shell, /bin/sh is a safe choice.

Downloading and Extracting Factorio

Enter the jail and switch to the factorio user:

bastille console factorio
su - factorio

Download and extract the server. You can use FreeBSD’s built-in fetch or install wget:

fetch -o factorio-headless.tar.xz "https://factorio.com/get-download/stable/headless/linux64"
tar xf factorio-headless.tar.xz
rm factorio-headless.tar.xz

This creates a factorio/ directory with the server files. You can verify the binary is a Linux ELF:

$ file factorio/bin/x64/factorio
factorio/bin/x64/factorio: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV),
dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0,
with debug_info, not stripped

Configuring the Server

Factorio stores its configuration in factorio/data/server-settings.json. Copy the example and customize it:

cd factorio
cp data/server-settings.example.json data/server-settings.json

Edit at minimum:

name: Your server’s public name
description: What players will see in the server browser
visibility: Set to {"public": true, "lan": true} if you want it listed
username and password: Your Factorio account credentials (required for public servers)
game_password: Optional password for players to join

Running the Server

When running under the Linuxulator, Factorio can’t automatically detect its installation directory and defaults to looking in /usr/share/factorio. The --executable-path flag tells it where to find the game data.

Quick Start with tmux

The simplest approach is running the server in a tmux session. This keeps the console accessible and survives disconnections. Note that ~/factorio expands to /home/factorio/factorio since the home directory contains the extracted factorio/ folder:

tmux new-session -d -s factorio
tmux send-keys -t factorio 'cd ~/factorio && bin/x64/factorio --executable-path ~/factorio/bin/x64/ --start-server-load-latest --server-settings ./data/server-settings.json' Enter

To attach to the console later:

tmux attach -t factorio

To create a new map (run from within ~/factorio):

bin/x64/factorio --executable-path ~/factorio/bin/x64/ --create ./saves/myworld.zip

Then start the server with that save:

bin/x64/factorio --executable-path ~/factorio/bin/x64/ --start-server ./saves/myworld.zip --server-settings ./data/server-settings.json

Proper rc.d Service Script

For production use, an rc.d script integrates with FreeBSD’s service management. Create /usr/local/etc/rc.d/factorio inside the jail:

#!/bin/sh

# PROVIDE: factorio
# REQUIRE: LOGIN
# KEYWORD: shutdown

. /etc/rc.subr

name="factorio"
rcvar=factorio_enable

load_rc_config $name

: ${factorio_enable:="NO"}
: ${factorio_user:="factorio"}
: ${factorio_dir:="/home/factorio/factorio"}
: ${factorio_savefile:="--start-server-load-latest"}
: ${factorio_flags:=""}

pidfile="/var/run/factorio/${name}.pid"

start_cmd="${name}_start"
stop_cmd="${name}_stop"
status_cmd="${name}_status"

factorio_start()
{
    install -o ${factorio_user} -g ${factorio_user} -m 755 -d /var/run/factorio
    echo "Starting ${name}."
    # -S: don't send SIGHUP to child on exit, -T: set process title for ps
    /usr/sbin/daemon -u ${factorio_user} -p ${pidfile} -S -T ${name} \
        ${factorio_dir}/bin/x64/factorio \
        --executable-path ${factorio_dir}/bin/x64/ \
        ${factorio_savefile} \
        --server-settings ${factorio_dir}/data/server-settings.json \
        ${factorio_flags}
}

factorio_status()
{
    if [ -f "${pidfile}" ] && kill -0 $(cat "${pidfile}") 2>/dev/null; then
        echo "${name} is running as pid $(cat ${pidfile})."
        return 0
    fi
    echo "${name} is not running."
    return 1
}

factorio_stop()
{
    if [ -f "${pidfile}" ]; then
        echo "Stopping ${name}."
        kill -TERM $(cat "${pidfile}") 2>/dev/null
        sleep 1
        rm -f "${pidfile}"
    else
        echo "${name} is not running."
        return 1
    fi
}

run_rc_command "$1"

Make it executable and enable it:

chmod +x /usr/local/etc/rc.d/factorio
sysrc factorio_enable=YES

Now you can manage the server with standard commands:

service factorio start
service factorio stop
service factorio status

To specify a particular save file instead of loading the latest:

sysrc factorio_savefile="--start-server /home/factorio/factorio/saves/myworld.zip"

Firewall Configuration

Factorio uses UDP port 34197 by default. Add redirect and pass rules to your host’s /etc/pf.conf:

# Macros
ext_if = "vtnet0"
host_ipv6 = "2001:db8::f3d1"
factorio_v4 = "10.254.252.98"
factorio_v6 = "2001:db8:8000::98"

# Redirect incoming Factorio traffic to the jail
rdr on $ext_if inet proto udp to ($ext_if) port 34197 -> $factorio_v4
rdr on $ext_if inet6 proto udp to $host_ipv6 port 34197 -> $factorio_v6

# Allow the traffic through
pass in quick on $ext_if inet proto udp from any to $factorio_v4 port 34197 keep state
pass in quick on $ext_if inet6 proto udp from any to $factorio_v6 port 34197 keep state

Test and reload PF:

pfctl -nf /etc/pf.conf && pfctl -f /etc/pf.conf

Verifying the Setup

If using the rc.d script, start the server and check the logs (inside the jail):

service factorio start
tail -f /home/factorio/factorio/factorio-current.log

A successful startup looks like this:

   0.000 2025-12-20 12:02:46; Factorio 2.0.72 (build 84292, linux64, headless)
   0.000 Operating system: Linux
   0.000 Config path: /usr/home/factorio/factorio/config/config.ini
   0.000 Read data path: /usr/home/factorio/factorio/data
   0.000 Write data path: /usr/home/factorio/factorio [108461/109184MB]
   0.009 System info: [CPU: AMD EPYC-Rome Processor, 8 cores, RAM: 15957 MB]
   0.010 Running in headless mode
   ...
   2.404 Hosting game at IP ADDR:({0.0.0.0:34197})
   2.739 Info ServerMultiplayerManager.cpp:808: changing state to(InGame)

The “Operating system: Linux” line is the key indicator - it proves the Linuxulator translation layer is active and the binary genuinely believes it’s running on Linux. Players can now connect via your server’s public IP on port 34197.

Troubleshooting

“error while loading shared libraries”: The Linux userland isn’t installed or linux_enable isn’t set. Install linux_base-rl9 and ensure /etc/rc.conf has linux_enable="YES".

Server starts but players can’t connect: Check your PF rules. Ensure both the rdr and pass rules are in place for UDP 34197. Use tcpdump -i vtnet0 udp port 34197 on the host to verify traffic is arriving.

“Failed to find the system certificate authority file”: This warning is harmless. Factorio falls back to its bundled certificate file for HTTPS requests to the auth server.

“Error configuring paths: There is no package core in /usr/share/factorio”: The --executable-path flag is missing. Under the Linuxulator, Factorio can’t auto-detect its installation directory. Always specify --executable-path /path/to/factorio/bin/x64/ when starting the server.

Summary

Running Factorio on FreeBSD is straightforward thanks to the Linuxulator:

Load linux64.ko on the host
Create a jail with linux_enable="YES"
Install linux_base-rl9 for the Linux shared libraries
Download the headless server and run it

The Linuxulator handles the translation transparently - the Factorio binary thinks it’s running on Linux. Combined with FreeBSD jails for isolation and PF for traffic control, you get a clean, manageable game server setup.

The factory must grow. Even on FreeBSD.

References

Thanks to the FreeBSD team for maintaining the Linuxulator, making it possible to run Linux-only software without virtualization overhead.

Hosting a Static Blog on FreeBSD with Bastille Jails and Automated Deployment

2025-12-14T00:00:00+01:00

Self-hosting a blog might seem like overkill in the age of managed platforms, but there’s something deeply satisfying about controlling your entire stack. This article walks through how this very blog is hosted: a FreeBSD 15.0 server running multiple Bastille jails, with automated deployments triggered by git pushes to a self-hosted Forgejo instance. The setup prioritizes security through isolation, simplicity through static site generation, and reliability through automation.

Architecture Overview

Before diving into the details, here’s how the pieces fit together:

                    Internet
                        |
                        v
            +-----------------------+
            |    PF Firewall        |
            |    NAT + RDR          |
            +-----------+-----------+
                        |
           +------------+------------+
           |                         |
           v                         v
    +-------------+           +-------------+
    | Caddy Jail  |           | Transporter |
    | TLS + Proxy |           |    Jail     |
    +------+------+           | rsync only  |
           |                  +------+------+
           v                         ^
    +-------------+                  |
    |  Blog Jail  |    nullfs mount  |
    |   Nginx     +------------------+
    +-------------+                  |
                              (rsync over SSH)
                                     |
                            +--------+--------+
                            | Forgejo Runner  |
                            | (remote server) |
                            +-----------------+

Traffic flows from the internet through PF’s packet filter, gets redirected to the Caddy jail for TLS termination, then proxied to the blog jail where Nginx serves static files. Deployments come in separately: a Forgejo runner builds the site and rsyncs it to the transporter jail, which has the blog’s webroot mounted via nullfs.

The Forgejo Git-Forge installation is not covered in this article. This might be a topic for a future post :-)

The Host System

The server runs FreeBSD 15.0-RELEASE on a VPS with dual-stack networking. Here’s the relevant portion of /etc/rc.conf:

hostname="server.example.com"

# Security hardening
kern_securelevel_enable="YES"
kern_securelevel="2"

# Network setup - dual stack
ifconfig_vtnet0="inet 192.0.2.10 netmask 255.255.252.0 -lro -tso"
ifconfig_vtnet0_ipv6="inet6 2001:db8::2 prefixlen 68"
defaultrouter="192.0.2.1"
ipv6_defaultrouter="fe80::1%vtnet0"

# Bastille jail network bridge
cloned_interfaces="bridge0"
ifconfig_bridge0_name="bastille0"
ifconfig_bastille0="inet 10.254.254.1/24"
ifconfig_bastille0_ipv6="inet6 2001:db8:1000::1 prefixlen 68"
gateway_enable="YES"
ipv6_gateway_enable="YES"

# Services
pf_enable="YES"
bastille_enable="YES"
zfs_enable="YES"

Setting kern_securelevel="2" is a simple but effective hardening measure. At securelevel 2, the system prevents loading kernel modules, writing to mounted filesystems marked immutable, and modifying the firewall rules without a reboot. This means even if an attacker gains root access, they can’t disable PF or load a rootkit module.

The Bastille bridge network (bastille0) provides a private network for all jails. Each jail gets an IP from the 10.254.254.0/24 range for IPv4 and a corresponding address in the 2001:db8:1000::/68 range for IPv6.

Jail Architecture

Bastille manages three jails relevant to this setup:

$ bastille list all
 JID  State  IP Address       Hostname       Release
 1    Up     10.254.254.20    blog           15.0-RELEASE
             2001:db8:1000::20
 2    Up     10.254.254.10    caddy          15.0-RELEASE
             2001:db8:1000::10
 3    Up     10.254.254.25    transporter    15.0-RELEASE
             2001:db8:1000::25

The Caddy Jail

This jail handles all incoming HTTPS traffic. Caddy automatically obtains and renews Let’s Encrypt certificates, terminates TLS, and proxies requests to the appropriate backend. It’s the only jail that needs to be reachable from the internet on ports 80 and 443.

The Blog Jail

A minimal jail running Nginx to serve static files. The webroot at /usr/local/www/ contains the generated Pelican output. The Nginx configuration is intentionally vanilla - just a basic server block pointing at the webroot with no special tuning required for static content. This jail has no public network exposure - all traffic comes through Caddy’s reverse proxy.

The Transporter Jail

This is the clever part. The transporter jail exists solely to receive deployments. It runs SSH but with heavily restricted access: only rsync is allowed, only from the Forgejo runner’s IP, and only to a specific directory. The magic happens through nullfs mounts:

# /usr/local/bastille/jails/transporter/fstab
/usr/local/bastille/jails/blog/root/usr/local/www/blog.example.com \
    /usr/local/bastille/jails/transporter/root/usr/local/deploy/blog nullfs 0 0
/usr/local/bastille/jails/blog/root/usr/local/www/staging.example.com \
    /usr/local/bastille/jails/transporter/root/usr/local/deploy/stageblog nullfs 0 0

When the CI runner rsyncs files to /usr/local/deploy/blog/ inside the transporter jail, they’re actually being written directly to the blog jail’s webroot. The transporter never stores anything - it’s just a secure gateway.

PF Firewall Configuration

PF ties everything together with NAT for outbound jail traffic and redirects for inbound services:

# --- Macros ---
ext_if = "vtnet0"
jail_net = "10.254.254.0/24"
jail_net6 = "2001:db8:1000::/68"
host_ipv6 = "2001:db8::2"

frontend_v4 = "10.254.254.10"
frontend_v6 = "2001:db8:1000::10"
transport_v4 = "10.254.254.25"
transport_v6 = "2001:db8:1000::25"
forgejo_runner_v4 = "198.51.100.50"

# --- Tables ---
table <bruteforce> persist
table <jails_v4> { $jail_net }
table <jails_v6> { $jail_net6 }

# --- Options ---
set skip on lo0
set block-policy drop

# --- NAT for jail egress ---
nat on $ext_if inet from <jails_v4> to any -> ($ext_if)
nat on $ext_if inet6 from <jails_v6> to any -> $host_ipv6

# --- RDR for services ---
rdr pass on $ext_if inet proto tcp to ($ext_if) port {80,443} -> $frontend_v4
rdr pass on $ext_if inet6 proto tcp to $host_ipv6 port {80,443} -> $frontend_v6

# Deployment SSH - only from Forgejo runner
rdr pass on $ext_if inet proto tcp from $forgejo_runner_v4 \
    to ($ext_if) port 2225 -> $transport_v4 port 22

# --- Filtering ---
block quick from <bruteforce>
block drop in log all
block drop out log all

pass out quick all keep state
antispoof quick for { $ext_if, bastille0 }

# Essential ICMP
pass in inet proto icmp icmp-type { echoreq, unreach }
pass in quick inet6 proto ipv6-icmp icmp6-type { echoreq, echorep, neighbrsol, neighbradv }

# Jail egress
pass in quick on bastille0 from <jails_v4> to ! 10.254.254.0/24 keep state
pass in quick on bastille0 inet6 from <jails_v6> to ! 2001:db8:1000::/68 keep state

A few things worth noting:

Default deny: Everything is blocked unless explicitly allowed
Deployment SSH on port 2225: The transporter jail’s SSH is only reachable from the Forgejo runner’s IP address. Anyone else hitting that port gets silently dropped.
Brute-force protection: The <bruteforce> table can be populated by rules elsewhere (e.g., for the host’s SSH) to automatically block repeat offenders
NAT for jails: Jails can reach the internet for package updates and certificate validation, but they appear to come from the host’s public IP

The Deployment Pipeline

The deployment relies on an SSH keypair generated beforehand: the public key lives in the transporter jail’s authorized_keys (with the rrsync restriction shown later), while the private key is stored as a secret in Forgejo. When I push to the blog’s git repository, a Forgejo Actions workflow kicks off:

on:
  push:
    branches: [ '**' ]

jobs:
  deploy:
    runs-on: docker
    container: python:3.11-slim
    steps:
      - name: Install Dependencies
        run: |
          apt-get update
          apt-get install -y rsync openssh-client git
          pip install pelican pelican-sitemap markdown typogrify

      - name: Checkout Code
        uses: actions/checkout@v3
        with:
          submodules: recursive

      - name: Configure Target
        id: meta
        run: |
          SHORT_SHA=$(git rev-parse --short HEAD)
          if [[ "${{ github.ref }}" == "refs/heads/main" ]]; then
            echo "url=https://blog.example.com" >> $GITHUB_OUTPUT
            echo "dest=blog/" >> $GITHUB_OUTPUT
          else
            echo "url=https://staging.example.com/$SHORT_SHA" >> $GITHUB_OUTPUT
            echo "dest=stageblog/$SHORT_SHA/" >> $GITHUB_OUTPUT
          fi

      - name: Build Site
        env:
          SITEURL: ${{ steps.meta.outputs.url }}
        run: pelican content -o output -s publishconf.py

      - name: Deploy via Rsync
        run: |
          rsync -avz --delete -e "ssh -p 2225" \
            output/ deploy@server.example.com:${{ steps.meta.outputs.dest }}

The branch strategy is simple: pushes to main deploy to production, pushes to any other branch deploy to a staging URL with the commit hash in the path. This lets me preview changes before merging.

The transporter jail’s authorized_keys file locks down what the deploy user can do:

command="/usr/local/sbin/rrsync -wo /usr/local/deploy/",no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding ssh-ed25519 AAAA... deploy@runner

The rrsync wrapper (restricted rsync) ensures the SSH key can only write to /usr/local/deploy/ and nothing else. Combined with the PF rules limiting source IPs, this creates a minimal attack surface for deployments.

Caddy Configuration

Caddy’s configuration is refreshingly simple. Here’s the production site block:

blog.example.com {
    reverse_proxy 10.254.254.20:80  # blog jail

    header {
        Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
        X-Content-Type-Options "nosniff"
        X-Frame-Options "DENY"
        Referrer-Policy "strict-origin-when-cross-origin"
    }

    log {
        output file /var/log/caddy/blog.access.log {
            roll_size 100mb
            roll_keep 10
        }
        format json
    }
}

staging.example.com {
    reverse_proxy 10.254.254.20:80

    header {
        Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
        X-Content-Type-Options "nosniff"
        X-Frame-Options "DENY"
        Referrer-Policy "strict-origin-when-cross-origin"
    }
}

Caddy handles certificate management automatically via ACME. The security headers provide defense-in-depth: HSTS ensures browsers always use HTTPS, X-Frame-Options prevents clickjacking, and the other headers mitigate various web attacks. For a static blog these might seem excessive, but they cost nothing and establish good habits.

Conclusion

This setup might look complex at first glance, but each piece serves a purpose:

Bastille jails provide lightweight isolation without the overhead of full VMs
PF gives fine-grained control over traffic flow and enforces the principle of least privilege
The transporter pattern allows secure deployments without exposing the blog jail directly
Forgejo Actions automates the entire build-and-deploy cycle
Caddy simplifies TLS and reverse proxying to a few lines of config

The result is a blog that deploys automatically on every git push, runs in isolated jails with minimal attack surface, and costs just a few euros per month on a small VPS. FreeBSD’s jails remain one of the most elegant isolation mechanisms available - simpler than containers, more lightweight than VMs, and battle-tested over two decades.

If you’re considering self-hosting, I’d encourage giving FreeBSD and Bastille a try. The learning curve is worth it.

Migrating burningboard.net Mastodon instance to a Multi-Jail FreeBSD Setup

2025-12-07T00:00:00+01:00

Over the last few weeks, I’ve been working on migrating our Mastodon instance burningboard.net from its current Linux host to a modular FreeBSD jail-based setup powered by BastilleBSD.

This post walks through the architecture and design rationale of my new multi-jail Mastodon system, with aggressive separation of concerns, centralized firewalling, and a fully dual-stack network design.

Acknowledgements

This work is based on the excellent post by Stefano Marinelli:

“Installing Mastodon on a FreeBSD jail”

Stefano’s article inspired me to try Mastodon on FreeBSD. My implementation takes that foundation and extends it for a more maintainable, production-ready architecture.

Design Goals

The motivation behind this move:

Central PF firewall – all filtering, NAT, and routing are handled by the host only. Jails see a clean, local L2 view - no PF inside jails, no double NAT.
Separation of concerns – every jail runs exactly one functional service:
- nginx — reverse proxy + TLS termination
- mastodonweb — Puma / Rails web backend
- mastodonsidekiq — background jobs
- database — PostgreSQL and Valkey (Redis fork)
Host‑managed source – Mastodon source tree shared via nullfs between web and sidekiq jails. Common .env.production, shared dependencies, single codebase to maintain.
Clean dual‑stack (IPv4 + IPv6) – every component visible under both protocols; no NAT66 or translation hacks.
Predictable networking – each functional group lives on its own bridge with private address space.

Jail and Network Overview

Example address plan (using RFC 5737 and 3849 documentation spaces):

Jail	Purpose	IPv4	IPv6
nginx	Reverse proxy	192.0.2.13	2001:db8:8000::13
mastodonweb	Rails backend	198.51.100.9	2001:db8:9000::9
mastodonsidekiq	Workers	198.51.100.8	2001:db8:b000::8
database	PostgreSQL + Valkey	198.51.100.6	2001:db8:a000::6
Host	“burningboard.example.net”	203.0.113.1	2001:db8::f3d1

Each functional bucket gets its own bridge(4) interface on the host (bastille0..bastille3) and its own /24 and /64 subnet.
Jails are created and attached to the corresponding bridge.

Schematic diagram

[ Internet ]
     |
     v
 [ PF Host ]
     ├── bridge0 — nginx (192.0.2.13 / 2001:db8:8000::13)
     ├── bridge1 — mastodonweb (198.51.100.9 / 2001:db8:9000::9)
     ├── bridge2 — database (198.51.100.6 / 2001:db8:a000::6)
     └── bridge3 — sidekiq (198.51.100.8 / 2001:db8:b000::8)

With the address plan established, the next step is creating the individual jails and assigning virtual network interfaces.

Jail Creation and Per‑Jail Configuration

Each jail was created directly through Bastille using VNET support, attaching it to its respective bridge.
For example, creating the nginx frontend jail on the bastille0 bridge:

bastille create -B nginx 14.3-RELEASE 192.0.2.13 bastille0

Bastille automatically provisions a VNET interface inside the jail (vnet0) and associates it with the corresponding bridge on the host.
Inside each jail, the /etc/rc.conf defines its own network interface, IPv4/IPv6 addresses, default routes, and any service daemons enabled for that jail.

Example configuration for the database jail (substituted with documentation addresses):

ifconfig_e0b_database_name="vnet0"
ifconfig_vnet0="inet 198.51.100.6 netmask 255.255.255.0"
ifconfig_vnet0_ipv6="inet6 2001:db8:a000::6/64"
ifconfig_vnet0_descr="database jail interface on bastille2"
defaultrouter="198.51.100.1"
ipv6_defaultrouter="2001:db8:a000::1"
syslogd_flags="-ss"
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"
cron_flags="-J 60"
valkey_enable="YES"
postgresql_enable="YES"

Each jail is therefore a fully self‑contained FreeBSD environment with native rc(8) configuration, its own routing table, and service definition. Bastille’s role ends at boot‑time network attachment - the rest is standard FreeBSD administration.

Host `/etc/rc.conf`

Below is a simplified version of the host configuration that ties everything together.
Each jail bridge subnet is assigned both IPv4 and IPv6 space; the host acts as gateway.

# Basic host config
hostname="burningboard.example.net"
keymap="us.kbd"

# Networking
ifconfig_vtnet0="inet 203.0.113.1 netmask 255.255.255.255"
ifconfig_vtnet0_ipv6="inet6 2001:db8::f3d1/64"
defaultrouter=="203.0.113.254"
ipv6_defaultrouter="2001:db8::1"

# Bridges for jails
cloned_interfaces="bridge0 bridge1 bridge2 bridge3"
ifconfig_bridge0_name="bastille0"
ifconfig_bridge1_name="bastille1"
ifconfig_bridge2_name="bastille2"
ifconfig_bridge3_name="bastille3"

# Bridge interfaces for individual networks

# Frontend (nginx)
ifconfig_bastille0="inet 192.0.2.1/24"
ifconfig_bastille0_ipv6="inet6 2001:db8:8000::1/64"

# Mastodon Web (Rails / Puma)
ifconfig_bastille1="inet 198.51.100.1/24"
ifconfig_bastille1_ipv6="inet6 2001:db8:9000::1/64"

# Database
ifconfig_bastille2="inet 198.51.100.2/24"
ifconfig_bastille2_ipv6="inet6 2001:db8:a000::1/64"

# Sidekiq (workers)
ifconfig_bastille3="inet 198.51.100.3/24"
ifconfig_bastille3_ipv6="inet6 2001:db8:b000::1/64"

gateway_enable="YES"
ipv6_gateway_enable="YES"

# Services
pf_enable="YES"
pflog_enable="YES"
bastille_enable="YES"
zfs_enable="YES"
sshd_enable="YES"
ntpd_enable="YES"
ntpd_sync_on_start="YES"

This provides proper L3 separation for each functional group.
In this layout, bastille0 → frontend, bastille1 → app, bastille2 → DB, bastille3 → worker pool.

`/etc/pf.conf`

The host firewall serves the dual purpose of NAT gateway and service ingress controller.

Below is an anonymized but structurally identical configuration.

# --- Macros ---
ext_if = "vtnet0"
jail_net = "198.51.100.0/20"
jail_net6 = "2001:db8:8000::/64"

host_ipv6 = "2001:db8::f3d1"
frontend_v4 = "192.0.2.13"
frontend_v6 = "2001:db8:8000::13"

# Trusted management networks (example)
trusted_v4 = "{ 203.0.113.42, 192.0.2.222 }"
trusted_v6 = "{ 2001:db8:beef::/64 }"

table <bruteforce> persist

set skip on lo0
set block-policy drop
set loginterface $ext_if

scrub in all fragment reassemble
scrub out all random-id max-mss 1500

# --- NAT ---
# Jails -> egress internet (IPv4)
nat on $ext_if inet from $jail_net to any -> ($ext_if)

# --- Port redirection ---
# Incoming HTTP/HTTPS -> nginx jail
rdr on $ext_if inet  proto tcp to ($ext_if) port {80,443} -> $frontend_v4
rdr on $ext_if inet6 proto tcp to $host_ipv6 port {80,443} -> $frontend_v6

# --- Filtering policy ---

# Default deny (log for audit)
block in log all
block out log all

# Allow existing stateful flows out
pass out all keep state

# Allow management SSH (example port 30822) only from trusted subnets
pass in quick on $ext_if proto tcp from $trusted_v4 to ($ext_if) port 30822 \
    flags S/SA keep state (max-src-conn 5, max-src-conn-rate 3/30, overload <bruteforce> flush global)

pass in quick on $ext_if inet6 proto tcp from $trusted_v6 to $host_ipv6 port 30822 \
    flags S/SA keep state (max-src-conn 5, max-src-conn-rate 3/30, overload <bruteforce> flush global)

# Block all other SSH
block in quick on $ext_if proto tcp to any port 30822 label "ssh_blocked"

# ICMP/ICMPv6 essentials
pass in inet proto icmp icmp-type { echoreq, unreach }
pass in inet6 proto ipv6-icmp icmp6-type { echoreq, echorep, neighbrsol, neighbradv, toobig, timex, paramprob }

# Inter-jail traffic
# nginx -> mastodonweb
pass in quick on bastille0 proto tcp from 192.0.2.13 to 198.51.100.9 port {3000,4000} keep state
pass in quick on bastille0 proto tcp from 2001:db8:8000::13 to 2001:db8:9000::9 port {3000,4000} keep state

# mastodonweb -> database (Postgres + Valkey)
pass in quick on bastille1 proto tcp from 198.51.100.9 to 198.51.100.6 port {5432,6379} keep state
pass in quick on bastille1 proto tcp from 2001:db8:9000::9 to 2001:db8:a000::6 port {5432,6379} keep state

# sidekiq -> database
pass in quick on bastille3 proto tcp from 198.51.100.8 to 198.51.100.6 port {5432,6379} keep state
pass in quick on bastille3 proto tcp from 2001:db8:b000::8 to 2001:db8:a000::6 port {5432,6379} keep state

# Optional: temporary egress blocking during testing
block in quick on { bastille0, bastille1, bastille2, bastille3 } from $jail_net to any
block in quick on { bastille0, bastille1, bastille2, bastille3 } inet6 from $jail_net6 to any

# External access
pass in quick on $ext_if inet  proto tcp to $frontend_v4 port {80,443} keep state
pass in quick on $ext_if inet6 proto tcp to $frontend_v6 port {80,443} keep state

This PF configuration centralizes control at the host. The jails have no firewall logic - just clean IP connectivity.

Shared Source Design

Both mastodonweb and mastodonsidekiq jails mount /usr/local/mastodon from the host:

/usr/local/mastodon -> /usr/local/bastille/jails/mastodonweb/root/usr/home/mastodon

/usr/local/mastodon -> /usr/local/bastille/jails/mastodonsidekiq/root/usr/home/mastodon

Example fstab entry:

/usr/local/mastodon /usr/local/bastille/jails/mastodonweb/root/usr/home/mastodon nullfs rw 0 0

That way, only one source tree needs updates after a git pull or bundle/yarn operation. The jails simply see the current state of that directory.

Logs and tmp directories are symlinked to /var/log/mastodon and /var/tmp/mastodon inside each jail for persistence and cleanup.

Service Boot Integration

Each Mastodon jail defines lightweight /usr/local/etc/rc.d scripts:

#!/bin/sh
# PROVIDE: mastodon_web
# KEYWORD: shutdown

. /etc/rc.subr

name="mastodon_web"
rcvar=mastodon_web_enable
pidfile="/var/run/mastodon/${name}.pid"

start_cmd="mastodon_web_start"
stop_cmd="mastodon_web_stop"

mastodon_web_start() {
    mkdir -p /var/run/mastodon
    chown mastodon:mastodon /var/run/mastodon
    su mastodon -c "export PATH=/usr/local/bin:/usr/bin:/bin; \
        export RAILS_ENV=production; export PORT=3000; \
        cd /home/mastodon/live && \
        /usr/sbin/daemon -T ${name} -P /var/run/mastodon/${name}_supervisor.pid \
        -p /var/run/mastodon/${name}.pid -f -S -r \
        /usr/local/bin/bundle exec puma -C config/puma.rb"
}

mastodon_web_stop() {
    kill -9 `cat /var/run/mastodon/${name}_supervisor.pid` 2>/dev/null
    kill -15 `cat /var/run/mastodon/${name}.pid` 2>/dev/null
}

load_rc_config $name
run_rc_command "$1"

Equivalent scripts exist for mastodon_streaming and the Sidekiq worker.

Everything integrates seamlessly with FreeBSD’s native service management:

service mastodon_web start
service mastodon_streaming restart
service mastodonsidekiq status

No Docker, no systemd, no exotic process supervisors.

Why It Matters

The resulting system is simple, observable, and robust:

Firewall rules are centralized and auditable.
Each jail is a clean service container (pure FreeBSD primitives, no overlay complexity).
IPv4/IPv6 connectivity is symmetrical and clear.
Source and configs are under full administrator control, not hidden in containers.

It’s also easy to snapshot with ZFS or promote new releases jail-by-jail using Bastille’s clone/deploy model.

Summary

In short:

Host does PF, routing, NAT, bridges
Each jail has exactly one purpose
Source code lives once on the host
Dual-stack networking, no translation
Everything FreeBSD-native

This structure makes it easy to reason about - each moving part has one job.

That’s how I like my infrastructure: boringly reliable.

References

Production-Grade Container Deployment with Podman Quadlets

2025-11-16T00:00:00+01:00

Introduction

Containers have become the de facto standard for application deployment, but the conversation often jumps straight to Kubernetes when discussing production workloads. While K8s excels at large-scale orchestration, many production services don’t require that level of complexity. For single-host or small-scale deployments, a well-architected Podman setup with systemd integration can provide robust, secure, and maintainable infrastructure.

This article demonstrates a production-grade container deployment using Red Hat Enterprise Linux 10, Podman Quadlets, and Traefik as a reverse proxy. We’ll walk through deploying Forgejo (a self-hosted Git service) as a practical example, covering the technical implementation and the architectural decisions behind it.

Why This Approach?

Before diving into the implementation, let’s address the fundamental design decisions:

Podman over Docker

Red Hat has made Podman the standard container runtime in RHEL for compelling technical and security reasons. This isn’t just vendor preference - it represents fundamental architectural improvements:

Daemonless architecture: No privileged daemon running as root, reducing the attack surface significantly. Each container runs as a direct child of systemd or the user session.
Rootless containers: Native support for running containers as unprivileged users - a first-class feature, not a bolt-on
systemd integration: First-class integration with the init system that already manages your services. This is particularly powerful in RHEL environments where systemd’s maturity and tooling are well-established.
OCI compliance: Full compatibility with Docker images and registries - your existing container images work without modification
Pod support: Kubernetes-style pod concepts for grouping containers, making the transition to K8s smoother if needed
Fork/exec model: Unlike Docker’s client-server architecture, Podman uses traditional Unix fork/exec, making it more auditable and debuggable with standard tools

From an enterprise perspective, Podman aligns with RHEL’s security-first philosophy. SELinux, user namespaces, and cgroups v2 integration are not afterthoughts but core design elements.

Quadlets over Compose

Traditional Docker Compose files are imperative and require a separate daemon. Podman Quadlets leverage systemd’s unit file format, providing:

Declarative configuration: Define the desired state, let systemd handle the lifecycle
Native service management: Use familiar systemctl commands - the same tooling administrators already know
Dependency management: Leverage systemd’s robust dependency graph (After=, Requires=, etc.)
Automatic updates: Built-in support for container image updates via podman-auto-update.timer
Resource control: Direct access to systemd’s cgroup integration for CPU, memory, and I/O limits
Journal integration: All container logs automatically flow to journald, integrating with existing log infrastructure

Network Segmentation by Design

Proper network isolation is crucial for security:

Frontend network: IPv6-enabled network for Traefik and application containers
Backend networks: Isolated networks for database communication
No unnecessary exposure: Database containers never touch the frontend network

The Architecture

Our example deployment consists of three components:

Forgejo - The Git service application
PostgreSQL - The database backend
Traefik - The reverse proxy handling TLS and routing

                    Internet
                       │
                       ▼
                   Traefik (Port 443)
                       │
          ┌────────────┴────────────┐
          │    frontend network     │
          │    (10.89.0.0/24)       │
          └────────────┬────────────┘
                       │
                  Forgejo Container
                       │
          ┌────────────┴────────────┐
          │ forgejo-backend.network │
          │   (isolated)            │
          └────────────┬────────────┘
                       │
                PostgreSQL Container

Practical Implementation: Deploying Forgejo

Let’s walk through deploying a complete Git hosting service with database backend and TLS termination.

Step 1: Enable Podman Socket for Traefik

Traefik’s Docker provider expects a Docker-compatible API socket. Podman provides this through a systemd-managed socket:

# Enable and start the Podman socket
systemctl enable --now podman.socket

# Verify the socket is active
systemctl status podman.socket

# Check socket location
ls -la /run/podman/podman.sock

Technical detail: The Podman socket (/run/podman/podman.sock) provides a Docker-compatible REST API. Traefik connects to this socket to discover containers and read their labels for dynamic configuration. This is mounted into the Traefik container as /var/run/docker.sock, maintaining compatibility with Traefik’s Docker provider configuration.

Without this socket enabled, Traefik cannot discover containers or process their routing labels - the declarative configuration simply won’t work.

Step 2: Network Configuration

First, create the networks. The frontend network enables IPv6 and connects to Traefik:

# Create the IPv6-enabled frontend network
podman network create \
  --ipv6 \
  --subnet 10.89.0.0/24 \
  --gateway 10.89.0.1 \
  ipv6

# Create the isolated backend network for database communication
podman network create forgejo-backend.network

Design Decision: Why two networks? The frontend network (despite its name “ipv6”) handles all external-facing traffic. The backend network ensures PostgreSQL is never directly accessible from the frontend, implementing defense-in-depth.

Step 3: Secrets Management

Never hardcode passwords in configuration files. Podman secrets integrate with systemd and provide secure credential storage:

# Generate a strong database password
pwgen -s 32 1 | podman secret create forgejo_db_password -

Design Decision: Using podman secret over environment variables in files provides: - Encrypted storage - Proper access control - No secrets in process lists or logs - Easy rotation without rebuilding containers

Step 4: Database Container (Quadlet)

Create /etc/containers/systemd/forgejo-db.container:

[Container]
ContainerName=forgejo-db
AutoUpdate=registry
Image=docker.io/postgres:16-alpine

# Network isolation - only on backend network
Network=forgejo-backend.network

# PostgreSQL configuration
Environment=POSTGRES_USER=forgejo
Environment=POSTGRES_DB=forgejo

# Secret injection as environment variable
Secret=forgejo_db_password,type=env,target=POSTGRES_PASSWORD

# Persistent storage with SELinux context
Volume=/opt/forgejo/postgres:/var/lib/postgresql/data:z

[Service]
Restart=always

[Install]
WantedBy=default.target

Key technical points:

AutoUpdate=registry: Enables automatic image updates via podman auto-update
Volume flag :z: Automatically relabels SELinux contexts for container access
Secret directive: Injects the secret as an environment variable at runtime
No frontend network: Database is completely isolated from external access

Step 5: Application Container (Quadlet)

Create /etc/containers/systemd/forgejo-server.container:

[Container]
ContainerName=forgejo-server
Image=codeberg.org/forgejo/forgejo:13
AutoUpdate=registry

# Dual network attachment
Network=forgejo-backend.network
Network=ipv6

# Application configuration
Environment=USER_UID=1000
Environment=USER_GID=1000
Environment=FORGEJO__database__DB_TYPE=postgres
Environment=FORGEJO__database__HOST=forgejo-db:5432
Environment=FORGEJO__database__NAME=forgejo
Environment=FORGEJO__database__USER=forgejo

# Database password from secret
Secret=forgejo_db_password,type=env,target=FORGEJO__database__PASSWD

# Persistent storage
Volume=/opt/forgejo/forgejo:/data:z
Volume=/etc/timezone:/etc/timezone:ro
Volume=/etc/localtime:/etc/localtime:ro

# Traefik labels for routing
Label="traefik.enable=true"
Label="traefik.docker.network=ipv6"
Label="traefik.http.routers.forgejo.rule=Host(`git.example.com`)"
Label="traefik.http.routers.forgejo.entrypoints=https"
Label="traefik.http.routers.forgejo.service=forgejo-http"
Label="traefik.http.routers.forgejo.tls.certresolver=traefiktls"
Label="traefik.http.routers.forgejo.middlewares=secure-headers@file"
Label="traefik.http.services.forgejo-http.loadbalancer.server.port=3000"

# SSH Git access via Traefik TCP routing
Label="traefik.tcp.routers.forgejo-ssh.rule=HostSNI(`*`)"
Label="traefik.tcp.routers.forgejo-ssh.entrypoints=ssh"
Label="traefik.tcp.routers.forgejo-ssh.service=forgejo-ssh"
Label="traefik.tcp.services.forgejo-ssh.loadbalancer.server.port=22"

[Service]
Restart=always

[Install]
WantedBy=default.target

[Unit]
After=forgejo-db.service

Design decisions explained:

Dual network attachment: The container needs backend network for PostgreSQL and frontend for Traefik
Traefik labels: Declarative routing configuration - no manual Traefik config files needed
SSH routing: Traefik handles both HTTP and TCP (Git SSH) on different ports
Systemd dependency: After=forgejo-db.service ensures database starts first

Step 6: Reverse Proxy Configuration

Create /etc/containers/systemd/traefik.container:

[Container]
ContainerName=traefik
Image=docker.io/traefik:latest
AutoUpdate=registry

# Required for binding to privileged ports
AddCapability=CAP_NET_BIND_SERVICE

# Frontend network only
Network=ipv6

# Port exposure
PublishPort=80:80
PublishPort=443:443
PublishPort=2222:2222

# Security hardening
NoNewPrivileges=true
SecurityLabelType=container_runtime_t

# Configuration and state
Volume=/etc/localtime:/etc/localtime:ro
Volume=/run/podman/podman.sock:/var/run/docker.sock:ro
Volume=/opt/traefik/traefik.yml:/etc/traefik/traefik.yml:z,ro
Volume=/opt/traefik/config.yml:/etc/traefik/config.yml:z,ro
Volume=/opt/traefik/letsencrypt:/letsencrypt:z

# Self-configuration for dashboard
Label=traefik.enable=true
Label=traefik.http.routers.dashboard.rule=Host(`traefik.example.com`)
Label=traefik.http.routers.dashboard.entrypoints=https
Label=traefik.http.routers.dashboard.service=api@internal
Label=traefik.http.routers.dashboard.tls=true
Label=traefik.http.routers.dashboard.tls.certresolver=traefiktls
Label=traefik.http.routers.dashboard.middlewares=dashboard-auth,secure-headers@file
Label=traefik.http.middlewares.dashboard-auth.basicauth.users=admin:$$2y$$05$$...

[Service]
Restart=always

[Install]
WantedBy=default.target

Create /opt/traefik/traefik.yml:

global:
  checkNewVersion: true
  sendAnonymousUsage: false

api:
  dashboard: true
  insecure: false

entryPoints:
  http:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: https
          scheme: https
  https:
    address: ":443"
  ssh:
    address: ":2222"

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
    network: ipv6
  file:
    filename: /etc/traefik/config.yml
    watch: true

certificatesResolvers:
  traefiktls:
    acme:
      email: admin@example.com
      storage: /letsencrypt/acme.json
      httpChallenge:
        entryPoint: http

log:
  level: INFO

Create /opt/traefik/config.yml for shared middlewares:

http:
  middlewares:
    secure-headers:
      headers:
        stsSeconds: 31536000
        stsIncludeSubdomains: true
        stsPreload: true
        forceSTSHeader: true
        customFrameOptionsValue: "SAMEORIGIN"
        contentTypeNosniff: true
        browserXssFilter: true
        referrerPolicy: "strict-origin-when-cross-origin"
        permissionsPolicy: "geolocation=(), microphone=(), camera=()"

Security highlights:

NoNewPrivileges: Prevents privilege escalation
SecurityLabelType: SELinux type enforcement
Automatic HTTP to HTTPS redirect
HSTS headers for HTTPS enforcement
Let’s Encrypt automation for TLS certificates

Step 7: Deployment

Reload systemd to recognize the new units:

systemctl daemon-reload

Enable and start the services:

# Enable automatic startup
systemctl enable forgejo-db.service
systemctl enable forgejo-server.service
systemctl enable traefik.service

# Start the stack
systemctl start forgejo-db.service
systemctl start forgejo-server.service
systemctl start traefik.service

The beauty of systemd integration: You can now manage containers like any other service:

# Check status
systemctl status forgejo-server.service

# View logs
journalctl -u forgejo-server.service -f

# Restart
systemctl restart forgejo-server.service

# View dependency tree
systemctl list-dependencies forgejo-server.service

Step 8: Automated Updates

Enable automatic container image updates:

# Enable the timer
systemctl enable --now podman-auto-update.timer

# Check update status
podman auto-update --dry-run

With AutoUpdate=registry in the Quadlet files, Podman will:

Check for new images daily
Pull updates if available
Recreate containers with new images
Preserve all volumes and configuration

Advanced Topics

Working with Red Hat Registries

While this example uses public container registries, production RHEL deployments often leverage Red Hat’s container catalog:

# Authenticate to Red Hat registry (requires active subscription)
podman login registry.redhat.io

# Pull RHEL-based images
podman pull registry.redhat.io/rhel9/postgresql-15

# Use in Quadlet
Image=registry.redhat.io/rhel9/postgresql-15

Red Hat certified container images include: - Support lifecycle matching RHEL versions - Security errata and CVE fixes - Compliance with enterprise requirements - Verified compatibility with RHEL container hosts

SELinux Integration

RHEL’s mandatory access control is a feature, not a bug. While many Docker tutorials suggest disabling SELinux, Podman embraces it as a critical security layer. The :z flag in volume mounts automatically handles SELinux labeling:

Volume=/opt/forgejo/postgres:/var/lib/postgresql/data:z

This relabels the host directory with the correct SELinux context (container_file_t) for container access. For read-only mounts, use :ro without :z:

Volume=/etc/localtime:/etc/localtime:ro

RHEL best practice: Never disable SELinux. If you encounter permission issues, investigate the context (ls -Z) rather than setting permissive mode. Podman’s integration makes this dramatically easier than it was with Docker.

For advanced scenarios, you can use uppercase :Z to make the volume private to that specific container, or manually manage contexts:

# Check SELinux context
ls -Z /opt/forgejo/

# Manually set context if needed
semanage fcontext -a -t container_file_t "/opt/forgejo(/.*)?"
restorecon -Rv /opt/forgejo/

Resource Limits

Systemd provides granular resource control through cgroups:

[Service]
MemoryMax=2G
CPUQuota=200%
TasksMax=1024

These limits are enforced by the kernel and prevent resource exhaustion.

Health Checks

Podman supports container health checks:

[Container]
HealthCmd=/usr/bin/curl -f http://localhost:3000/ || exit 1
HealthInterval=30s
HealthTimeout=5s
HealthRetries=3

Systemd can react to failed health checks and restart containers automatically.

Monitoring and Observability

Viewing Logs

All container output goes to journald:

# Real-time logs
journalctl -u forgejo-server.service -f

# Last 100 lines
journalctl -u forgejo-server.service -n 100

# Logs from specific time
journalctl -u forgejo-server.service --since "2024-01-01 12:00:00"

Container Inspection

# Container details
podman inspect forgejo-server

# Resource usage
podman stats forgejo-server

# Network information
podman network inspect ipv6

Comparison with Kubernetes

This approach is not meant to replace Kubernetes for large-scale deployments, but it offers distinct advantages for single-host or small-scale scenarios:

Advantages:

Significantly lower resource overhead
Simpler mental model and troubleshooting
Direct integration with OS-level tools
No additional control plane components
Easier to audit and secure

When to choose K8s instead:

Multi-host orchestration requirements
Advanced scheduling needs
Built-in service mesh requirements
Teams already invested in K8s ecosystem

Security Considerations

This setup implements several security layers:

Network segmentation: Databases isolated from frontend
Rootless option: All containers can run as unprivileged users
SELinux enforcement: Mandatory access control
Secret management: No credentials in configuration files
Automatic updates: Regular security patches
TLS termination: Encrypted transport with Let’s Encrypt
Security headers: HSTS, CSP, and other protections

For even stricter security, run containers in rootless mode:

# As regular user (not root)
systemctl --user enable --now forgejo-server.service

Conclusion

Modern container deployment doesn’t require Kubernetes for every use case. With RHEL Quadlets, Podman, and proper architectural patterns, you can build production-grade container infrastructure that is:

Secure: Multiple layers of isolation and access control, leveraging RHEL’s security-first design
Maintainable: Declarative configuration with systemd integration
Observable: Native integration with journald and systemd tools
Automated: Built-in update mechanisms via systemd timers
Resilient: Systemd’s proven service management
Enterprise-ready: Backed by Red Hat’s support lifecycle and security practices

This approach proves particularly valuable for:

Self-hosted services and edge deployments
Development environments matching production
Organizations preferring simpler operational models
Hybrid scenarios where some services don’t warrant K8s overhead

Next Steps for Red Hat Practitioners

This foundation scales naturally into the broader Red Hat container ecosystem:

Fedora CoreOS: Apply these Quadlet patterns to immutable, auto-updating infrastructure
OpenShift: Recognize how systemd-managed containers relate to K8s pods
Ansible automation: Codify Quadlet deployment with containers.podman collection
Image building: Explore Buildah and Skopeo for OCI image workflows

The combination of Podman’s security-first design and systemd’s battle-tested service management creates a robust foundation for containerized applications without the operational overhead of full orchestration platforms - and provides essential knowledge for working across Red Hat’s entire container portfolio.

FreeBSD Dual-Stack Jails on Hetzner Cloud

2025-11-12T00:00:00+01:00

Hetzner Cloud networking has a few quirks that complicate otherwise standard FreeBSD setups:

IPv4 arrives as a /32 with a pseudo on-link next-hop at 172.31.1.1.
You typically get exactly one routed IPv6 /64 per server.
There’s no shared L2 domain; your VM doesn’t ARP/NDP for other tenants.
NIC offload features can misbehave with VNET + bridges on FreeBSD.

This guide shows a reproducible dual-stack configuration for a FreeBSD 14 host with VNET jails managed by Bastille. The pattern:

IPv4: RFC1918 on the jail bridge; NAT to host’s public IPv4.
IPv6: Split the single routed /64 into two /65s: one for the host, one routed to the jails (no ULA, no NAT66).
PF for IPv4 NAT and port forwards; pure routing for IPv6.

All addresses below use documentation ranges:

Host IPv4: 203.0.113.10/32
Hetzner pseudo-gateway (IPv4): 172.31.1.1
Assigned IPv6 /64: 2001:db8:abcd:1000::/64
Host half (/65): 2001:db8:abcd:1000::/65
Jails half (/65): 2001:db8:abcd:1000:8000::/65
Bastille bridge IPv4: 10.100.0.1/24
Jail example IPv4: 10.100.0.100/24
Host IPv6 (example): 2001:db8:abcd:1000::10/65
Bridge IPv6 (gateway for jails): 2001:db8:abcd:1000:8000::1/65
Jail example IPv6: 2001:db8:abcd:1000:8000::100/65

Why split the /64 into two /65s?

Hetzner Cloud gives you a single routed /64. To put global IPv6 addresses in jails without NAT66/ULA, split that /64: - Keep the lower /65 on the host interface. - Route the upper /65 to your jail bridge and use it for jails. This mirrors the “host + routed downstream” model and keeps IPv6 fully end-to-end.

0) Design Philosophy

This guide follows modern networking principles:

IPv6 is IP. IPv4 is the legacy compatibility layer.
No NAT66. NAT was an IPv4 workaround for address exhaustion. IPv6 has 340 undecillion addresses - use them.
End-to-end connectivity. Jails get real, globally routable IPv6 addresses. Firewalling provides security, not address translation.
IPv4 best-effort. NAT is acceptable for IPv4 because we have no choice. It’s temporary infrastructure.

If this offends you, you’re reading the wrong guide. 🙂

1) Host /etc/rc.conf

Configure the /32 IPv4 with the on-link host route to the pseudo gateway, enable forwarding, define the Bastille bridge, and assign both v4 and v6 (using the /65 split).

hostname="freebsd-hcloud"
keymap="us.kbd"

# Forwarding for jails
gateway_enable="YES"
ipv6_gateway_enable="YES"

# Hetzner Cloud NIC: /32 IPv4 and routed IPv6
# Disable LRO/TSO to avoid VNET/bridge issues
ifconfig_vtnet0="inet 203.0.113.10 netmask 255.255.255.255 -tso -lro"
ifconfig_vtnet0_ipv6="inet6 2001:db8:abcd:1000::10/65"

# IPv4 pseudo-gateway and default route
static_routes="hcloud default"
route_hcloud="-host 172.31.1.1 -interface vtnet0"
route_default="default 172.31.1.1"

# IPv6 default router (Hetzner uses link-local on your NIC)
ipv6_defaultrouter="fe80::1%vtnet0"

# Bastille VNET bridge
cloned_interfaces="bridge0"
ifconfig_bridge0_name="bastille0"
ifconfig_bastille0="inet 10.100.0.1/24"
# Routed upper /65 goes to the bridge (acts as gateway for jails)
ifconfig_bastille0_ipv6="inet6 2001:db8:abcd:1000:8000::1/65"

# Services
pf_enable="YES"
pflog_enable="YES"
sshd_enable="YES"
zfs_enable="YES"
ntpd_enable="YES"
ntpd_sync_on_start="YES"

Notes: - -tso -lro on vtnet0 avoids checksum/segmentation oddities with bridged epairs. - The host’s IPv6 address lives in the lower /65; the bridge gets a gateway address in the upper /65.

2) /etc/sysctl.conf

Ensure forwarding is on and (optionally) disable redirects.

net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1
net.inet.ip.redirect=0
net.inet6.ip6.redirect=0

3) PF: /etc/pf.conf

NAT IPv4 from the jail network to the host’s public IPv4. IPv6 is routed (no NAT). Forward HTTP/HTTPS to a specific jail as an example. Lock down SSH to trusted sources.

# --- Macros ---
ext_if = "vtnet0"

# Jail networks
jail_net_v4 = "10.100.0.0/24"
jail_net_v6 = "2001:db8:abcd:1000:8000::/65"

# Host IPv6 on the external interface (lower /65)
host_ipv6 = "2001:db8:abcd:1000::10"

# Example trusted admin sources
table <trusted_v4> persist { 198.51.100.22, 203.0.113.50 }
table <trusted_v6> persist { 2001:db8:ffff:1::/64 }

# Brute-force tracking
table <bruteforce> persist

# --- Options ---
set skip on lo0
set block-policy drop
set loginterface $ext_if

# --- Scrub ---
scrub in all fragment reassemble
scrub out all random-id max-mss 1500

# --- NAT (IPv4 only) ---
nat on $ext_if inet from $jail_net_v4 to any -> ($ext_if)

# --- RDR for public services to a "web" jail ---
# v4: to host IPv4, forward 80/443 to the jail v4
rdr pass on $ext_if inet  proto tcp to ($ext_if) port {80, 443} -> 10.100.0.100
# v6: to the host's IPv6, forward 80/443 to the jail v6
rdr pass on $ext_if inet6 proto tcp to $host_ipv6 port {80, 443} -> 2001:db8:abcd:1000:8000::100

# --- Policy ---
block in log all
block out log all

# Allow established and all outbound by default
pass out quick all keep state

# Anti-spoof
antispoof quick for { $ext_if, bastille0 }

# SSH allowlist (example nonstandard port 22222)
pass in quick on $ext_if inet  proto tcp from <trusted_v4> to ($ext_if) port 22222 \
    flags S/SA keep state (max-src-conn 5, max-src-conn-rate 3/30, overload <bruteforce> flush global)
pass in quick on $ext_if inet6 proto tcp from <trusted_v6> to ($ext_if) port 22222 \
    flags S/SA keep state (max-src-conn 5, max-src-conn-rate 3/30, overload <bruteforce> flush global)

# Log and drop other SSH attempts
block in log quick on $ext_if proto tcp to ($ext_if) port 22222 label "ssh_blocked"

# Essential ICMPv6 (ND, pMTU, echo)
pass in quick inet6 proto ipv6-icmp icmp6-type { echoreq, echorep, neighbrsol, neighbradv, toobig, timex, paramprob }

# Useful ICMPv4 (echo, unreach for pMTU)
pass in inet proto icmp icmp-type { echoreq, unreach }

# Allow from jails bridge (host-facing side)
pass in quick on bastille0 all keep state

# Public HTTP/HTTPS to forwarded jail
pass in quick on $ext_if proto tcp to ($ext_if) port {80, 443} keep state

4) Bastille/Jail networking

VNET jails get an epair; the host end attaches to bastille0. Inside the jail, rename the epair to vnet0 and assign addresses. The bridge addresses act as the default gateways.

Example jail rc.conf (inside the jail):

ifconfig_e0b_webjail_name="vnet0"
ifconfig_vnet0="inet 10.100.0.100 netmask 255.255.255.0"
ifconfig_vnet0_ipv6="inet6 2001:db8:abcd:1000:8000::100/65"
defaultrouter="10.100.0.1"
ipv6_defaultrouter="2001:db8:abcd:1000:8000::1"

# Optional: quiet base daemons
syslogd_flags="-ss"
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"

If templating with Bastille, a minimal jail.conf fragment:

vnet="on";
vnet.interface="e0a_${name}";

exec.prestart="ifconfig e0a_${name} create up";
exec.prestart="ifconfig bastille0 addm e0a_${name}";

exec.start="/bin/sh /etc/rc";
exec.stop="/bin/sh /etc/rc.shutdown";

mount.devfs;

Bastille will hand the jail end as e0b_${name}, which you rename to vnet0 in the jail’s rc.conf.

5) Bring-up and verification

Reload PF and confirm rules:
service pf restart
pfctl -sr -v
Verify routes on the host:
IPv4 default via 172.31.1.1 on vtnet0
IPv6 default via fe80::1%vtnet0
Local routes to 10.100.0.0/24 and 2001:db8:abcd:1000:8000::/65 on bastille0
Start Bastille and your jail:
service bastille start
bastille start webjail
Inside the jail:
IPv4: ping -c 3 1.1.1.1
IPv6: ping -c 3 2001:4860:4860::8888

From the internet: - HTTP/HTTPS should land on the jail via rdr. - SSH should only accept from your trusted sources.

6) Hetzner Cloud quirks and fixes

IPv4 /32 with pseudo next-hop:
You must add a host route to 172.31.1.1 on vtnet0 and set it as default.
Offload features:
-tso -lro on vtnet0 prevents oddities with bridged epairs and IPv6 checksums.
IPv6 routing model:
IPv6 is routed to your VM. Don’t expect SLAAC for jails. Assign static v6 from your routed half (/65) and route via the host.
If IPv6 works on the host but not in jails:
Ensure net.inet6.ip6.forwarding=1
Make sure jails use 2001:db8:abcd:1000:8000::1 as their default router
Confirm PF isn’t blocking NDP or pMTU (ipv6-icmp types are allowed above)
If IPv4 NAT seems flaky:
Check that no later PF rules override pass out
Verify MTU/MSS (scrub max-mss 1500 is set)
Inspect states: pfctl -ss | grep 10.100.0.

7) Security notes

Restrict SSH with allowlists and non-default ports; consider WireGuard for admin.
Enable pflog and inspect with tcpdump:
tcpdump -n -e -ttt -r /var/log/pflog
Rate-limit public services at your reverse proxy jail if needed.

8) Quick reference: full files

/etc/rc.conf

hostname="freebsd-hcloud"
keymap="us.kbd"

gateway_enable="YES"
ipv6_gateway_enable="YES"

ifconfig_vtnet0="inet 203.0.113.10 netmask 255.255.255.255 -tso -lro"
ifconfig_vtnet0_ipv6="inet6 2001:db8:abcd:1000::10/65"

static_routes="hcloud default"
route_hcloud="-host 172.31.1.1 -interface vtnet0"
route_default="default 172.31.1.1"
ipv6_defaultrouter="fe80::1%vtnet0"

cloned_interfaces="bridge0"
ifconfig_bridge0_name="bastille0"
ifconfig_bastille0="inet 10.100.0.1/24"
ifconfig_bastille0_ipv6="inet6 2001:db8:abcd:1000:8000::1/65"

pf_enable="YES"
pflog_enable="YES"
sshd_enable="YES"
zfs_enable="YES"
ntpd_enable="YES"
ntpd_sync_on_start="YES"

/etc/sysctl.conf

net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1
net.inet.ip.redirect=0
net.inet6.ip6.redirect=0

/etc/pf.conf

ext_if = "vtnet0"

jail_net_v4 = "10.100.0.0/24"
jail_net_v6 = "2001:db8:abcd:1000:8000::/65"

host_ipv6 = "2001:db8:abcd:1000::10"

table <trusted_v4> persist { 198.51.100.22, 203.0.113.50 }
table <trusted_v6> persist { 2001:db8:ffff:1::/64 }
table <bruteforce> persist

set skip on lo0
set block-policy drop
set loginterface $ext_if

scrub in all fragment reassemble
scrub out all random-id max-mss 1500

nat on $ext_if inet from $jail_net_v4 to any -> ($ext_if)

rdr pass on $ext_if inet  proto tcp to ($ext_if) port {80, 443} -> 10.100.0.100
rdr pass on $ext_if inet6 proto tcp to $host_ipv6 port {80, 443} -> 2001:db8:abcd:1000:8000::100

block in log all
block out log all

pass out quick all keep state

antispoof quick for { $ext_if, bastille0 }

pass in quick on $ext_if inet  proto tcp from <trusted_v4> to ($ext_if) port 22222 \
    flags S/SA keep state (max-src-conn 5, max-src-conn-rate 3/30, overload <bruteforce> flush global)
pass in quick on $ext_if inet6 proto tcp from <trusted_v6> to ($ext_if) port 22222 \
    flags S/SA keep state (max-src-conn 5, max-src-conn-rate 3/30, overload <bruteforce> flush global)

block in log quick on $ext_if proto tcp to ($ext_if) port 22222 label "ssh_blocked"

pass in quick inet6 proto ipv6-icmp icmp6-type { echoreq, echorep, neighbrsol, neighbradv, toobig, timex, paramprob }
pass in inet proto icmp icmp-type { echoreq, unreach }

pass in quick on bastille0 all keep state

pass in quick on $ext_if proto tcp to ($ext_if) port { 80, 443 } keep state

Jail rc.conf (inside the jail)

ifconfig_e0b_webjail_name="vnet0"
ifconfig_vnet0="inet 10.100.0.100 netmask 255.255.255.0"
ifconfig_vnet0_ipv6="inet6 2001:db8:abcd:1000:8000::100/65"
defaultrouter="10.100.0.1"
ipv6_defaultrouter="2001:db8:abcd:1000:8000::1"

syslogd_flags="-ss"
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"

9) Troubleshooting checklist

Host routing:
netstat -rn shows default v4 via 172.31.1.1; default v6 via fe80::1%vtnet0
Bridge membership:
ifconfig bastille0 lists e0a_ interfaces for running jails
IPv6 neighbors and pMTU:
ndp -an on host shows upstream neighbors; ensure ipv6-icmp is allowed
NAT state:
pfctl -ss | grep 10.100.0. confirms IPv4 NAT states
Throughput/stalls:
Verify -tso -lro on vtnet0; scrub out max-mss 1500 is present

Wrap-up

This pattern—IPv4 NATed jails, IPv6 routed using a /64 split into two /65s—fits Hetzner Cloud’s model and avoids NAT66/ULA while keeping end-to-end IPv6. It’s simple, robust, and friendly to PF.

FreeBSD 15.0 on the ThinkPad T480 — Efficient, Stable, and 8 Hours on Battery

2025-11-09T00:00:00+01:00

Introduction

The ThinkPad T480 has long had a reputation for being one of the most practical and robust laptops for engineers and developers. It’s built like a tank, easy to maintain, and works brilliantly with open-source operating systems.

Over the last few months, I’ve been tuning and optimizing FreeBSD on this machine with a very clear goal in mind:
reach the best possible power efficiency without sacrificing performance or usability.

After a good amount of trial and error, firmware testing, and sysctl tuning, I finally reached a sweet spot. A setup that reliably gives me 6–8 hours of real battery life under normal workloads.

This post documents that configuration for anyone in the BSD community who wants to make their ThinkPad a genuinely mobile FreeBSD workstation.

Hardware Overview

Model: Lenovo ThinkPad T480
CPU: Intel Core i7-8550U (4C/8T, Kaby Lake-R)
GPU: Intel UHD Graphics 620 (integrated)
Display: 1920×1080 (IPS, 14”)
Wireless: Intel AX210 (iwlwifi)
Storage: NVMe SSD (ZFS root)
Memory: 32 GiB DDR4
Battery: 72 Wh extended

The combination of efficient hardware and modern drivers from the FreeBSD 15 branch provides an excellent base to work with.

Drivers and Firmware

I’m using the latest driver sets available from the ports tree:

drm-latest-kmod-6.9.1500068
gpu-firmware-intel-kmod-kabylake-20230625.1500501
wifi-firmware-iwlwifi-kmod-ax210-20241017.1500501_2

These make the Intel GPU and Wi-Fi adapters fully functional with modern capabilities: RC6, PSR, FBC, and full 802.11ac/ax performance.

System Configuration

FreeBSD’s configuration model is simple and transparent, which makes power tuning straightforward once you understand the fundamentals.

/etc/rc.conf

hostname="christop-bsd"

# Wireless configuration
wlans_iwlwifi0="wlan0"
ifconfig_wlan0="WPA DHCP"
create_args_wlan0="wlanmode sta regdomain ETSI country DE mode 11na"
ifconfig_wlan0_ipv6="inet6 accept_rtadv"

# Networking bridge (for bhyve)
cloned_interfaces="bridge0"
ifconfig_bridge0="inet 192.168.87.1 netmask 255.255.255.0"

# Power management
powerd_enable="YES"
powerd_flags="-a hiadaptive -b adaptive -m 400 -M 4000"
performance_cx_lowest="Cmax"
economy_cx_lowest="Cmax"

# Driver autoloads
kld_list="i915kms"

# Services
sshd_enable="YES"
zfs_enable="YES"
seatd_enable="YES"
dbus_enable="YES"
sddm_enable="YES"
webcamd_enable="YES"
pf_enable="YES"
pflog_enable="YES"
dnsmasq_enable="YES"
pcscd_enable="YES"
gateway_enable="YES"
clear_tmp_enable="YES"

# bhyve
vm_enable="YES"
vm_dir="zfs:zroot/bhyve"

# Misc
moused_nondefault_enable="NO"
dumpdev="NO"

The key piece here is powerd, with different adaptive modes for AC and battery, letting the CPU drop down to deep C-states efficiently.
XFCE and lightweight services complete the power-slim system profile.

/boot/loader.conf

# Kernel features
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"

# Essential modules
zfs_load="YES"
cryptodev_load="YES"
cuse_load="YES"
cpuctl_load="YES"
coretemp_load="YES"
nvme_load="YES"

# Graphics and power tuning
i915kms_load="YES"
drm.i915.enable_rc6=7
drm.i915.enable_fbc=1
drm.i915.lvds_downclock=1
drm.i915.enable_psr=1
compat.linuxkpi.i915_disable_power_well="0"

# NVMe and PCI power tweaks
hw.nvme.use_nvd="0"
hw.pci.do_power_nodriver=3
hw.pci.do_power_suspend=1

These settings allow the GPU and PCI devices to enter the lowest power states reliably.
Together with powerd, they make a significant difference in idle and light load power draw.

Desktop Environment

Desktop: XFCE 4.20
Window Manager: Xfwm4 (X11)
Theme: Greybird-Dark
Icons: elementary-xfce
Fonts: Noto Sans (11 pt)
Terminal font: UbuntuMono Nerd Font (11 pt)

XFCE is lightweight, fast, and power-friendly - no unnecessary daemons, no heavy compositing. It feels modern and stays responsive even under load.

Real-World Results

Scenario	Power Draw	Notes
Idle (Wi-Fi on, 40 % brightness)	4–5 W	ultra-low power idle
Video playback (720p)	6–7 W	stable playback via Intel GPU
Moderate development workload	10–12 W	perfect balance
`make buildkernel`	18–22 W	all cores fully utilized

With the 72 Wh extended battery, real-world battery time ranges from 6 to 8 hours depending on activity, brightness, and Wi-Fi intensity.

Suspend and resume are fully functional and reliable, one of the last major milestones that used to be tricky on FreeBSD laptops.

Expert Notes & Additional Tweaks

Enable ZFS compression (lz4) and disable atime to cut down write I/O.
Keep firmware packages current using pkg upgrade.
Consider running powerdxx if you want even finer CPU scaling — though base powerd does a great job here.

Conclusion

FreeBSD 15.0 marks a huge step forward for laptop support.
On the ThinkPad T480, everything from suspend to Wi-Fi to graphics just works, and with the right configuration, you can enjoy Linux-level power efficiency combined with FreeBSD’s rock-solid stability.

It’s a joy to use this machine daily now: clean, simple, and efficient. True to FreeBSD’s spirit.

Thanks to everyone in the BSD community pushing kernel and driver work - this level of polish on a laptop would’ve been unthinkable just a few releases ago.

Comments and feedback welcome via Mastodon: @larvitz@bsd.cafe

Simple Temperature Monitoring on FreeBSD

2025-10-09T00:00:00+02:00

If you’re running FreeBSD 14.x and want to keep an eye on your system temperatures without installing third-party packages, you’re in luck. FreeBSD’s sysctl interface provides everything you need.

The Built-in Approach

Modern FreeBSD systems expose temperature data through the sysctl tree. You can query all temperature sensors with a simple command:

sysctl -a | grep temperature

This will show CPU core temperatures, ACPI thermal zones, and PCH (Platform Controller Hub) temperatures if available.

A Practical Monitoring Script

Rather than repeatedly typing sysctl commands, here’s a lightweight monitoring script that refreshes every two seconds:

#!/bin/sh
kldload coretemp
while true; do
clear
echo "======================================"
echo "   Temperature Monitor - $(date '+%H:%M:%S')"
echo "======================================"
echo ""
echo "CPU Cores:"
sysctl dev.cpu | grep temperature | sort -V
echo ""
echo "System:"
sysctl hw.acpi.thermal.tz0.temperature hw.acpi.thermal.tz1.temperature dev.pchtherm.0.temperature 2>/dev/null
echo ""
echo "Press Ctrl+C to exit"
sleep 2
done

Save this as tempmon.sh, make it executable with chmod +x tempmon.sh, and run it whenever you need real-time temperature monitoring.

Quick Access with an Alias

For instant temperature checks, add this alias to your shell configuration:

# For sh/bash (~/.shrc or ~/.bashrc)
alias temps='sysctl dev.cpu | grep temperature | sort -V; echo ""; sysctl hw.acpi.thermal.tz0.temperature dev.pchtherm.0.temperature'
# For csh/tcsh (~/.cshrc)
alias temps 'sysctl dev.cpu | grep temperature | sort -V; echo ""; sysctl hw.acpi.thermal.tz0.temperature dev.pchtherm.0.temperature'

Now temps gives you an instant snapshot of your system’s thermal state.

Understanding the Output

The dev.cpu.X.temperature values represent individual CPU core temperatures, while hw.acpi.thermal.tzX.temperature shows ACPI thermal zone readings (typically overall system/motherboard temperatures). The dev.pchtherm.0.temperature reports the Platform Controller Hub temperature when available. For most systems under normal load, CPU temperatures between 40-60°C are typical. Sustained temperatures above 80°C warrant investigation.

FreeBSD Cheat Sheet for Linux Admins

2025-08-29T00:00:00+02:00

Hardware Information

Task	FreeBSD	Linux
List PCI devices	`pciconf -lv`	`lspci -v`
List USB devices	`usbconfig`	`lsusb`
Show CPU info	`sysctl hw.model` or `dmesg \\| grep CPU`	`cat /proc/cpuinfo` or `lscpu`
Show memory info	`sysctl hw.physmem` or `top`	`free -h` or `cat /proc/meminfo`
Show kernel modules	`kldstat`	`lsmod`
Load kernel module	`kldload module_name`	`modprobe module_name`
Unload kernel module	`kldunload module_name`	`modprobe -r module_name`

Disk and Storage Management

Task	FreeBSD	Linux
List all disks	`geom disk list`	`lsblk` or `fdisk -l`
Show disk partitions	`gpart show`	`fdisk -l` or `parted -l`
Show disk usage	`df -h`	`df -h`
Show mounted filesystems	`mount`	`mount` or `findmnt`
Check filesystem	`fsck`	`fsck`
Show SMART data	`smartctl -a /dev/ada0`	`smartctl -a /dev/sda`
List ZFS pools	`zpool list`	`zpool list` (if ZFS installed)
Show ZFS datasets	`zfs list`	`zfs list` (if ZFS installed)

Network Commands

Task	FreeBSD	Linux
Show network interfaces	`ifconfig`	`ip addr` or `ifconfig`
Configure interface	`ifconfig em0 inet 192.168.1.10`	`ip addr add 192.168.1.10/24 dev eth0`
Show routing table	`netstat -rn`	`ip route` or `route -n`
Add static route	`route add default 192.168.1.1`	`ip route add default via 192.168.1.1`
Show ARP table	`arp -a`	`arp -a` or `ip neigh`
Clear ARP entry	`arp -d hostname`	`arp -d hostname` or `ip neigh del`
Show network statistics	`netstat -s`	`netstat -s` or `ss -s`
Show listening ports	`sockstat -l` or `netstat -an`	`ss -tuln` or `netstat -tuln`
Show all open ports/connections	`sockstat`	`ss -tuan` or `netstat -tuan`
Packet capture	`tcpdump -i em0`	`tcpdump -i eth0`

Process Management

Task	FreeBSD	Linux
List all processes	`ps aux`	`ps aux`
Process tree	`pstree`	`pstree` or `ps axjf`
Real-time process view	`top`	`top` or `htop`
Kill process	`kill PID`	`kill PID`
Show open files by process	`fstat`	`lsof`
Show process using port	`sockstat -p 80`	`lsof -i :80` or `ss -tulpn \\| grep :80`

Package Management

Task	FreeBSD	Linux (varies by distro)
Install package	`pkg install package_name`	`apt install` / `yum install` / `dnf install`
Remove package	`pkg delete package_name`	`apt remove` / `yum remove` / `dnf remove`
Update package list	`pkg update`	`apt update` / `yum check-update`
Upgrade packages	`pkg upgrade`	`apt upgrade` / `yum upgrade` / `dnf upgrade`
Search packages	`pkg search keyword`	`apt search` / `yum search` / `dnf search`
Show package info	`pkg info package_name`	`apt show` / `yum info` / `dnf info`
List installed packages	`pkg info`	`dpkg -l` / `rpm -qa`

System Services

Task	FreeBSD	Linux (systemd)
Start service	`service servicename start`	`systemctl start servicename`
Stop service	`service servicename stop`	`systemctl stop servicename`
Restart service	`service servicename restart`	`systemctl restart servicename`
Service status	`service servicename status`	`systemctl status servicename`
Enable at boot	`sysrc servicename_enable="YES"`	`systemctl enable servicename`
Disable at boot	`sysrc servicename_enable="NO"`	`systemctl disable servicename`
List all services	`service -e`	`systemctl list-units --type=service`

Firewall

Task	FreeBSD (pf/ipfw)	Linux (iptables/nftables)
Show rules (pf)	`pfctl -sr`	`iptables -L -n -v`
Show rules (ipfw)	`ipfw list`	`nft list ruleset`
Enable firewall (pf)	`pfctl -e`	`systemctl start firewalld`
Disable firewall (pf)	`pfctl -d`	`systemctl stop firewalld`
Reload rules (pf)	`pfctl -f /etc/pf.conf`	`iptables-restore < /etc/iptables/rules.v4`

System Information

Task	FreeBSD	Linux
System uptime	`uptime`	`uptime`
Kernel version	`uname -a` or `freebsd-version`	`uname -a`
Show all sysctl variables	`sysctl -a`	`sysctl -a`
Show system messages	`dmesg`	`dmesg` or `journalctl -k`
Show system logs	`tail /var/log/messages`	`journalctl` or `tail /var/log/syslog`

User Management

Task	FreeBSD	Linux
Add user	`adduser` or `pw useradd username`	`useradd username` or `adduser username`
Delete user	`rmuser` or `pw userdel username`	`userdel username`
Modify user	`pw usermod username`	`usermod username`
Change password	`passwd username`	`passwd username`
Show logged-in users	`who` or `w`	`who` or `w`

File Systems

Task	FreeBSD	Linux
Mount filesystem	`mount /dev/ada0p2 /mnt`	`mount /dev/sda2 /mnt`
Unmount filesystem	`umount /mnt`	`umount /mnt`
Create UFS filesystem	`newfs /dev/ada0p2`	N/A (UFS not common)
Create ext4 filesystem	N/A	`mkfs.ext4 /dev/sda2`
Check disk space	`du -sh /path`	`du -sh /path`

ZFS Commands (Same on FreeBSD and Linux with ZFS)

Pool Operations

Task	Command	Example
List all pools	`zpool list`	`zpool list`
Show pool status	`zpool status`	`zpool status tank`
Show pool history	`zpool history`	`zpool history tank`
Show pool I/O statistics	`zpool iostat`	`zpool iostat -v 2` (every 2 seconds)
Create simple pool	`zpool create`	`zpool create tank /dev/ada1`
Create mirror pool	`zpool create`	`zpool create tank mirror /dev/ada1 /dev/ada2`
Create RAIDZ pool	`zpool create`	`zpool create tank raidz /dev/ada1 /dev/ada2 /dev/ada3`
Add disk to pool	`zpool add`	`zpool add tank /dev/ada4`
Replace disk in pool	`zpool replace`	`zpool replace tank /dev/ada1 /dev/ada4`
Remove device from pool	`zpool remove`	`zpool remove tank /dev/ada4`
Scrub pool (check integrity)	`zpool scrub`	`zpool scrub tank`
Stop scrub	`zpool scrub -s`	`zpool scrub -s tank`
Clear pool errors	`zpool clear`	`zpool clear tank`
Export pool	`zpool export`	`zpool export tank`
Import pool	`zpool import`	`zpool import tank`
List importable pools	`zpool import`	`zpool import` (without pool name)
Upgrade pool	`zpool upgrade`	`zpool upgrade tank`
Set pool property	`zpool set`	`zpool set autoreplace=on tank`
Get pool properties	`zpool get`	`zpool get all tank`

Dataset (Filesystem) Operations

Task	Command	Example
List all datasets	`zfs list`	`zfs list`
List with specific properties	`zfs list`	`zfs list -o name,used,avail,mountpoint`
List snapshots	`zfs list -t snapshot`	`zfs list -t snapshot`
Create dataset	`zfs create`	`zfs create tank/home/user`
Destroy dataset	`zfs destroy`	`zfs destroy tank/old_data`
Destroy dataset and children	`zfs destroy -r`	`zfs destroy -r tank/old_data`
Set dataset property	`zfs set`	`zfs set compression=lz4 tank/home`
Get dataset properties	`zfs get`	`zfs get all tank/home`
Set quota	`zfs set quota=`	`zfs set quota=10G tank/home/user`
Set reservation	`zfs set reservation=`	`zfs set reservation=5G tank/database`
Mount dataset	`zfs mount`	`zfs mount tank/home`
Unmount dataset	`zfs unmount`	`zfs unmount tank/home`
Show mounted ZFS filesystems	`zfs mount`	`zfs mount` (without arguments)

Snapshot Operations

Task	Command	Example
Create snapshot	`zfs snapshot`	`zfs snapshot tank/home@backup-2024`
Create recursive snapshot	`zfs snapshot -r`	`zfs snapshot -r tank/home@daily-2024`
List snapshots	`zfs list -t snapshot`	`zfs list -t snapshot -r tank/home`
Rollback to snapshot	`zfs rollback`	`zfs rollback tank/home@backup-2024`
Destroy snapshot	`zfs destroy`	`zfs destroy tank/home@old-backup`
Rename snapshot	`zfs rename`	`zfs rename tank/home@old tank/home@archived`
Clone snapshot	`zfs clone`	`zfs clone tank/home@backup tank/home_clone`
Show snapshot disk usage	`zfs list -o space`	`zfs list -r -o space tank`

Send/Receive (Replication)

Task	Command	Example
Send snapshot	`zfs send`	`zfs send tank/home@backup > /backup/home.zfs`
Send incremental	`zfs send -i`	`zfs send -i @snap1 tank/home@snap2 > incremental.zfs`
Receive snapshot	`zfs receive`	`zfs receive tank/home_restore < /backup/home.zfs`
Send over SSH	`zfs send \\| ssh`	`zfs send tank/home@backup \\| ssh user@host zfs receive tank/backup`
Dry-run receive	`zfs receive -n`	`zfs receive -n tank/test < backup.zfs`
Send with progress	`zfs send -v`	`zfs send -v tank/home@backup > backup.zfs`

Common Property Settings

Task	Command	Example
Enable compression	`zfs set compression=`	`zfs set compression=lz4 tank/data`
Enable deduplication	`zfs set dedup=`	`zfs set dedup=on tank/backup`
Set record size	`zfs set recordsize=`	`zfs set recordsize=1M tank/media`
Enable encryption	`zfs create -o encryption=`	`zfs create -o encryption=on -o keyformat=passphrase tank/secure`
Set access time	`zfs set atime=`	`zfs set atime=off tank/database`
Set sync behavior	`zfs set sync=`	`zfs set sync=disabled tank/temp`
Enable case insensitivity	`zfs set casesensitivity=`	`zfs create -o casesensitivity=insensitive tank/windows`

Monitoring and Maintenance

Task	Command	Example
Show pool I/O stats	`zpool iostat`	`zpool iostat -v tank 2`
Show ARC stats (FreeBSD)	`sysctl kstat.zfs.misc.arcstats`	`sysctl kstat.zfs.misc.arcstats.size`
Show ARC stats (Linux)	`arc_summary`	`arc_summary` or `cat /proc/spl/kstat/zfs/arcstats`
Check pool health	`zpool status -x`	`zpool status -x` (only shows problems)
Show pool events	`zpool events`	`zpool events -v`
Estimated scrub time	`zpool status`	`zpool status` (during scrub)
Show compression ratio	`zfs get compressratio`	`zfs get compressratio tank/data`
Show space usage by type	`zfs list -o space`	`zfs list -r -o space tank`

Useful One-Liners

# Find largest datasets
zfs list -o name,used -s used

# Show all snapshots sorted by creation
zfs list -t snapshot -o name,creation -s creation

# Calculate total snapshot space
zfs list -t snapshot -o used -p | awk '{sum+=$1} END {print sum/1024/1024/1024 " GB"}'

# Show datasets with compression disabled
zfs get -r compression tank | grep -v "lz4\|gzip\|zle"

# Monitor pool I/O in real-time
zpool iostat -v 1

# Show properties that differ from defaults
zfs get all tank | grep -v default

# Quick backup to remote system
zfs snapshot tank/important@$(date +%Y%m%d) && \
zfs send tank/important@$(date +%Y%m%d) | ssh backup-server zfs receive tank/backup-2024

Quick Tips

Device naming: FreeBSD uses different naming conventions (ada0 for SATA, da0 for SCSI/USB) vs Linux (sda, sdb, etc.)
Network interfaces: FreeBSD names interfaces by driver (em0, re0, bge0) while Linux traditionally used eth0, eth1 (now often uses predictable names like enp0s3)
Configuration files: FreeBSD centralizes many configs in /etc/rc.conf while Linux distributes them across various files
Manual pages: Both use man command, but FreeBSD’s man pages are often more comprehensive
Ports vs Packages: FreeBSD has both ports (source) and packages (binary), while Linux typically uses one package manager per distribution
ZFS: While native to FreeBSD, ZFS on Linux (ZoL) has reached feature parity. Commands are identical, but kernel module loading differs (kldload zfs on FreeBSD vs modprobe zfs on Linux)

Larvitz Blog

Running Your Own AS: Direct Hetzner Peering, a Fourth Edge, and Bringing the Home LAN into the Fabric

Table of Contents

At a Glance

Architecture: Four Edges and a Home Router

The Fourth Edge: ixbgp at iFog and FogIXP

Why a Fourth Edge

Network Configuration

FRR Configuration

PF: Stateless Transit, Stateful Control Plane

Direct Peering with Hetzner

The Path Before

The Path Now

What Hetzner Sees

What This Buys in Practice

Bringing the Home LAN into AS201379

The Gap in Parts 1-3

MikroTik Configuration

The Core-Side Configuration

Traffic Engineering: Steering DTAG via Vultr

The Observation

The Implementation

The Effect

Hub Hygiene: One IP for Traceroutes

Downstream Sites: PI Addressing for Friends and Services

Lessons Learned

Conclusion

References

Automating FreeBSD Jails with cdist - Zero Dependencies Inside the Jail

Table of Contents

What cdist Actually Is

The Two Hooks That Make This Interesting

jexec-ssh.py

jexec-scp.py

Actually Running It

A More Practical Example: A Custom Maintenance Type

Why This Is The Unix Way

When To Still Reach For Ansible

Getting The Scripts

References

Replacing Lenovo’s WWAN Unlock Blob with a 100-Line Bash Script

Table of Contents

What FCC Lock Actually Is

The ModemManager FCC Unlock Mechanism

Lenovo’s Solution: A Proprietary Helper Package

The Alternative: A Shell Script That Does the Same Thing

How the Unlock Actually Works

Installing It Yourself

A Disclaimer About Jurisdiction

Wrapping Up

References

Podman on FreeBSD: OCI Containers Without systemd

Table of Contents

Installing Podman on FreeBSD

Prerequisites

Enabling the Service

Verification

Native FreeBSD OCI Containers

Linux Containers via the Linuxulator

Prerequisites

Running Linux Images

Limitations

Container Lifecycle Without systemd

Podman’s Built-in Restart Policy

rc.d Service Scripts

What You Lose Without Quadlets

Logging

Image Updates

Podman and Jails: Complementary, Not Competing

Practical Tips

Networking

Storage

Secrets

When to Use What

Wrapping Up

References

Podman in Production: Quadlets, Secrets, Auto-Updates, and Docker Compatibility

Table of Contents

Why Podman Is Better Than Docker

No Daemon, No Single Point of Failure

The `podman` CLI Alias