Dual-FIB Policy Routing on FreeBSD: Two Upstreams, One Server, Zero Confusion

Mon 23 March 2026
FreeBSD
#freebsd, #networking, #pf, #bgp, #jails, #tunneling, #policy-routing

Logo

The previous articles in this series covered the BGP router side: obtaining an AS, peering with upstream providers, and building a tunnel overlay. This article covers the other end - the downstream server that consumes that tunnel and needs to serve traffic from two completely different IP ranges through two completely different paths, simultaneously, without either one interfering with the other.

The server in question is radon, a Netcup VPS running FreeBSD with Bastille jails. It has two internet uplinks: a physical connection to Netcup’s network and a GIF tunnel to my BGP router (hobgp). Jails on this server use three distinct routing paradigms - private NAT, natively routed BGP IPv6, and pure public routed BGP IPv4 - all on the same bridge interface.

The mechanism that makes this work is dual-FIB policy routing: two independent routing tables in the kernel, with PF deciding which table handles which traffic based on source address. It’s elegant once you understand it, and surprisingly simple to configure once you’ve seen it done.

The Problem: One Server, Two Gateways

A server with a single default gateway has simple routing: everything goes out the same door. But what happens when you want traffic from different source addresses to take different paths?

On radon, the physical interface (vtnet0) is assigned Netcup’s IP space: 152.53.147.5 for IPv4 and 2a0a:4cc0:c1:2f90::2 for IPv6. The default gateway points to Netcup’s router. Standard traffic - package updates, DNS queries, NATed jail traffic - flows through this path.

But radon also has addresses from my own AS201379: 194.28.98.217 (shared via loopback for port forwarding), 194.28.98.216 (directly assigned to a jail), and the entire 2a06:9801:1c:1000::/64 IPv6 subnet routed to jails. This address space is announced to the internet through my BGP router. Traffic arriving for these addresses traverses the BGP tunnel. If replies to that traffic exit through Netcup’s default gateway instead of the tunnel, the provider drops them as spoofed - the source IP doesn’t belong to Netcup’s network.

The kernel’s single routing table (FIB 0) knows one default route: Netcup. It has no concept of “this source address should use a different exit.” That’s where dual-FIB comes in.

Architecture Overview

                        ┌─────────────────────────────────────────────┐
                        │           Public Internet                   │
                        └────────┬──────────────────────┬─────────────┘
                                 │                      │
                         Netcup Network           BGP (AS201379)
                         152.53.144.0/22          via hobgp router
                                 │                      │
                                 │               ┌──────┴──────────┐
                                 │               │  hobgp (Core)   │
                                 │               │  128.140.64.181 │
                                 │               └──────┬──────────┘
                                 │                      │
                                 │              GIF tunnel (proto 41)
                                 │              IPv6-in-IPv4 encap
     ┌───────────────────────────┴──────────────────────┴─────────────────────┐
     │                        radon.edelga.se                                 │
     │                                                                        │
     │  vtnet0 ──── FIB 0 (Default)                                           │
     │  152.53.147.5/22        <- Standard internet, NAT, mgmt                │
     │  2a0a:4cc0:c1:2f90::2/64                                               │
     │                                                                        │
     │  gif0 ────── FIB 1 (Tunnel)                                            │
     │  194.28.98.216/29       <- BGP-addressed traffic                       │
     │  2a06:9801:1c:ffff::2/128                                              │
     │                                ┌─────────────────────────────────────┐ │
     │  bastille0 ════════════════════│        Jail Bridge                  │ │
     │  10.254.254.1/24               │                                     │ │
     │  2a06:9801:1c:1000::1/64       │  ┌────────┐  ┌─────────────────────┐│ │
     │                                │  │ Caddy  │  │ testvnet            ││ │
     │  lo0:                          │  │ .254.10│  │194.28.98.216        ││ │
     │  194.28.98.217/32 (shared)     │  │ (NAT)  │  │2a06:9801:1c:1000::99││ │
     │                                │  └────────┘  │   (Pure Public)     ││ │
     │                                │              └─────────────────────┘│ │
     │                                └─────────────────────────────────────┘ │
     └────────────────────────────────────────────────────────────────────────┘

The key insight: FIB 0 handles “normal” traffic (Netcup-addressed). FIB 1 handles BGP-addressed traffic (AS201379). PF is the glue that assigns packets to the correct FIB based on source or destination address.

The Physical Upstream: Netcup (FIB 0)

The physical interface lives in the default routing table. This is the server’s “normal” internet connection:

# Primary physical interface - Netcup VPS
ifconfig_vtnet0="inet 152.53.147.5 netmask 255.255.252.0 -lro -tso"
defaultrouter="152.53.144.1"

ifconfig_vtnet0_ipv6="inet6 2a0a:4cc0:c1:2f90::2 prefixlen 64"
ipv6_defaultrouter="fe80::1%vtnet0"

The -lro -tso flags deserve a callout. LRO (Large Receive Offload) and TSO (TCP Segmentation Offload) are hardware acceleration features on the virtio NIC. They work fine for traffic that terminates at the host. But when the host forwards packets - as a router does for jail traffic - these offloads break PF’s ability to correctly NAT and checksum forwarded packets. The symptom is maddening: connections from the host work, but NATed jail traffic silently fails with bad checksums. Disable both whenever a FreeBSD host acts as a gateway.

SLAAC is disabled globally to avoid conflicts with statically routed tunnel addresses:

ipv6_activate_all_interfaces="NO"

The Logical Upstream: BGP Tunnel (FIB 1)

The GIF tunnel connects to the BGP router at 128.140.64.181. This is where the multi-FIB magic lives:

cloned_interfaces="bridge0 gif0"

ifconfig_gif0="fib 1 tunnel 152.53.147.5 128.140.64.181 tunnelfib 0"
ifconfig_gif0_alias0="inet 10.255.255.6 10.255.255.5"
ifconfig_gif0_ipv6="inet6 2a06:9801:1c:ffff::2 2a06:9801:1c:ffff::1 prefixlen 128"

This single line - fib 1 tunnel 152.53.147.5 128.140.64.181 tunnelfib 0 - is the most important configuration directive on the entire server. It contains two directives that work in concert:

fib 1: The tunnel interface itself lives in routing table 1. Traffic arriving on gif0 and traffic routed out gif0 consults FIB 1 for routing decisions.
tunnelfib 0: The outer IPv4 encapsulation wrapper (the 152.53.147.5 → 128.140.64.181 packet) uses FIB 0 - the default routing table - to find the path to the BGP router.

Why both? Without tunnelfib 0, the encapsulated IPv4 packets would consult FIB 1 for their route to 128.140.64.181. But FIB 1’s default route points at gif0 itself (the tunnel). That creates a recursive loop: to send a packet out the tunnel, you’d need to use the tunnel. tunnelfib 0 breaks the recursion by telling the outer encapsulation to use the normal Netcup route.

The IPv4 point-to-point addresses (10.255.255.6/5) are used for the tunnel’s inner addressing. The IPv6 link addresses (2a06:9801:1c:ffff::2/1) come from our own /48 - they’re already routable within our BGP infrastructure.

FIB 1: Building a Complete Routing Table

FIB 1 starts empty. It knows nothing about the server’s local networks. Every route must be explicitly added - if FIB 1 doesn’t have a route to a destination, traffic in that routing table gets dropped.

static_routes="fib1_v6_default fib1_v6_jailnet fib1_v4_default fib1_v4_jailnet fib1_v4_host217 fib1_v4_jail216 v4_jail216"

# 1. Default Routes (How FIB 1 reaches the internet)
route_fib1_v6_default="-6 default -interface gif0 -fib 1"
route_fib1_v4_default="-inet default 10.255.255.5 -fib 1"

# 2. Subnet Routes (How FIB 1 finds the bridge)
route_fib1_v6_jailnet="-6 2a06:9801:1c:1000::/64 -interface bastille0 -fib 1"
route_fib1_v4_jailnet="-net 10.254.254.0/24 -interface bastille0 -fib 1"

# 3. Host Routes (Where FIB 1 finds specific public IPs)
route_fib1_v4_host217="-host 194.28.98.217 -interface lo0 -fib 1"
route_fib1_v4_jail216="-host 194.28.98.216 -interface bastille0 -fib 1"
route_v4_jail216="-host 194.28.98.216 -interface bastille0"       # FIB 0 also needs this — explained in the Pure Public section below

Each route category serves a specific purpose:

Default routes tell FIB 1 how to reach the internet. IPv6 exits through gif0 directly (the tunnel interface). IPv4 uses 10.255.255.5 as the next-hop - that’s the BGP router’s end of the tunnel’s inner IPv4 addressing.

Subnet routes are necessary because FIB 1 doesn’t automatically know that bastille0 has the jail network attached. Without these, return traffic arriving on the tunnel for jail addresses would try to exit via the default route (back out the tunnel) instead of being delivered locally to the bridge.

Host routes handle the public IPs. 194.28.98.217 is on lo0 (the host’s shared loopback IP for port forwarding). 194.28.98.216 is on bastille0 (directly assigned to the testvnet jail). Both FIBs need to know where these addresses live - note that v4_jail216 adds the same route to FIB 0 as well, so that the default routing table can also reach the jail.

Jail Networking and the Bridge

The bastille0 bridge is the single point where all jail routing paradigms converge:

ifconfig_bridge0_name="bastille0"
ifconfig_bastille0="inet 10.254.254.1/24"
ifconfig_bastille0_ipv6="inet6 2a06:9801:1c:1000::1 prefixlen 64"

Three types of traffic share this bridge:

Private NAT (10.254.254.0/24): Jails like Caddy (10.254.254.10) get private IPv4 addresses, NATed to the Netcup IP when exiting vtnet0. This is standard jail networking - nothing special here.
BGP IPv6 Routed (2a06:9801:1c:1000::/64): Jails also receive addresses from our BGP prefix. These are globally routable, announced by AS201379, and traffic for them arrives via the GIF tunnel. No NAT - pure end-to-end IPv6 as designed.
BGP IPv4 Routed (194.28.98.x): The testvnet jail has 194.28.98.216/32 - a real public IPv4 address, no NAT, directly routed through both FIBs.

The shared host IP (194.28.98.217) on lo0 is a different pattern. It’s not assigned to any jail; instead, PF uses RDR rules to forward ports from this IP to internal jails. This lets Caddy serve content from a BGP-routed IPv4 address while the jail itself only has a private 10.254.254.10 address.

PF: Where the Routing Decision Happens

The PF configuration is where dual-FIB stops being a kernel feature and becomes a routing policy. Let’s walk through each layer.

Normalization

scrub in all fragment reassemble
scrub all max-mss 1220

The MSS clamp at 1220 accounts for tunnel encapsulation overhead. MSS (Maximum Segment Size) dictates the maximum size of the TCP payload. To calculate it, subtract the inner IP header and the inner TCP header from the MTU. For IPv6 traffic inside the tunnel, clamped to the safe minimum MTU of 1280 bytes, you subtract the IPv6 header (40 bytes) and the TCP header (20 bytes): 1280 - 40 - 20 = 1220. Without this, large TCP segments get silently dropped or fragmented at the tunnel, causing mysterious stalls where small requests succeed but large transfers hang.

Translation (NAT and RDR)

Translation happens before filtering. This is critical to understand - filter rules for RDR traffic must match the translated internal IP, not the public IP.

# Outbound NAT: private jails → Netcup IP
nat on $ext_if inet from <jails_v4> to any -> ($ext_if)

# Port forwarding on the Netcup IP (vtnet0)
rdr pass on $ext_if inet proto {tcp, udp} to ($ext_if) port {80, 443} -> $frontend_v4
rdr pass on $ext_if inet proto tcp to ($ext_if) port 1965 -> $gemini_v4
rdr pass on $ext_if inet proto {tcp, udp} to ($ext_if) port 53 -> $powerdns_v4

# Port forwarding on the BGP host IP (arrives via tunnel)
rdr on $tun_if inet proto {tcp, udp} to $host_v4_routed port {80, 443} -> $frontend_v4

The last rule is different - traffic to 194.28.98.217:80/443 arrives on the tunnel interface and gets redirected to the same Caddy jail. This means Caddy serves web traffic regardless of whether it arrived via Netcup or the BGP tunnel, even though Caddy only has a private IP address.

Default Policies

block quick from <bruteforce>
block drop in log all
block drop out log all
antispoof quick for { $ext_if, bastille0 }

Default deny in both directions. Bruteforce entries get an instant drop before any evaluation. Antispoof prevents packets from entering interfaces they don’t belong to.

Egress: The `route-to` Safety Net

pass out quick on $ext_if route-to ($tun_if $tun_gw_v4) inet from $host_v4_routed to any keep state

This is a safety net rule that catches an edge case. When local processes on the host respond from the shared BGP IP (194.28.98.217), those replies naturally try to exit via vtnet0 - the host’s default route in FIB 0. But Netcup would drop traffic sourced from an IP that doesn’t belong to their network.

The route-to directive intercepts: “if you see a packet about to exit vtnet0 with source 194.28.98.217, redirect it to the tunnel interface instead.” It overrides the routing decision at the PF level, forcing the packet into gif0 toward 10.255.255.5 (the BGP router).

Egress: Jail Policy Routing via `rtable`

This is the core mechanism. When a jail sends traffic from a BGP address, PF must force it into FIB 1:

# IPv6 BGP-addressed jail traffic → routing table 1
pass in quick on bastille0 inet6 from $bgp_net_v6 to any rtable 1 keep state

# IPv4 pure public jail traffic → routing table 1
pass in quick on bastille0 inet  from $testvnet_v4 to any rtable 1 keep state

The rtable 1 directive is the key. When a packet from 2a06:9801:1c:1000::/64 or 194.28.98.216 arrives at the bridge, PF assigns it to routing table 1. The kernel then consults FIB 1 for the forwarding decision - and FIB 1’s default route points out gif0 to the BGP router. The packet gets encapsulated and sent through the tunnel.

Without rtable 1, these packets would use FIB 0’s default route and exit through Netcup - where they’d be dropped as spoofed.

There’s also a DNS64 translation rule for jails needing IPv4 destinations via IPv6:

pass in quick on bastille0 inet6 from $bgp_net_v6 to 64:ff9b::/96 af-to inet from ($ext_if) keep state

And a standard NAT egress rule for private jails:

pass in quick on bastille0 from <jails_v4> to ! $jail_net keep state

The ! $jail_net negation allows jails to reach the internet but prevents them from directly accessing other jails - a micro-segmentation pattern that documents and enforces the network architecture.

Ingress: Tunnel Encapsulation

The GIF tunnel uses protocol 41 (IPv6-in-IPv4). PF must allow these encapsulated packets to enter and exit the physical interface:

pass in  quick on $ext_if proto { 41, ipencap } from 128.140.64.181 to ($ext_if)
pass out quick on $ext_if proto { 41, ipencap } from ($ext_if) to 128.140.64.181

Strictly locked to the BGP router’s IP - no one else should be sending protocol 41 to this server.

Ingress: The `reply-to` Imperative

All inbound connections from the BGP tunnel must be tagged with reply-to:

# IPv4 tunnel ingress
pass in quick on $tun_if reply-to ($tun_if $tun_gw_v4) inet proto icmp to $host_v4_routed keep state
pass in quick on $tun_if reply-to ($tun_if $tun_gw_v4) inet proto tcp from <trusted_v4> to $host_v4_routed port 30822 flags S/SA keep state
pass in quick on $tun_if reply-to ($tun_if $tun_gw_v4) inet proto {tcp, udp} to $frontend_v4 port {80, 443} keep state
pass in quick on $tun_if reply-to ($tun_if $tun_gw_v4) inet to $testvnet_v4 keep state

# IPv6 tunnel ingress
pass in quick on $tun_if reply-to ($tun_if $bgp_hub_ip) inet6 proto ipv6-icmp from any to $bgp_net_v6 icmp6-type { echoreq, echorep, toobig, timex, paramprob } keep state
pass in quick on $tun_if reply-to ($tun_if $bgp_hub_ip) inet6 proto {tcp, udp} to $frontend_v6  port {80, 443} keep state
pass in quick on $tun_if reply-to ($tun_if $bgp_hub_ip) inet6 proto {tcp, udp} to $powerdns_v6  port 53 keep state
pass in quick on $tun_if reply-to ($tun_if $bgp_hub_ip) inet6 proto tcp        to $gemini_v6    port 1965 keep state

reply-to instructs PF’s state machine: “when you see a reply to this connection, send it back out $tun_if to the specified next-hop, regardless of what the routing table says.” Without it, the kernel would consult FIB 0 for return traffic (the jail doesn’t live in FIB 1), and replies would exit through vtnet0 - getting dropped by Netcup as spoofed.

The IPv4 and IPv6 reply-to targets differ because they use different tunnel endpoint addresses: - IPv4: $tun_gw_v4 = 10.255.255.5 (the inner IPv4 point-to-point address) - IPv6: $bgp_hub_ip = 2a06:9801:1c:ffff::1 (the inner IPv6 link address)

The Pure Public VNET Jail

The testvnet jail is the most interesting routing case. It has a real public IPv4 address (194.28.98.216/32) and a BGP IPv6 address (2a06:9801:1c:1000::99), both assigned directly to the jail’s vnet0 interface:

# Inside the jail
vnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP>
        inet 194.28.98.216 netmask 0xffffffff broadcast 194.28.98.216
        inet6 2a06:9801:1c:1000::99 prefixlen 64

This jail has no NAT. Outbound traffic from 194.28.98.216 hits the bridge, PF matches from $testvnet_v4 → rtable 1, and the packet exits via the tunnel. Inbound traffic arrives on the tunnel, PF matches to $testvnet_v4 with reply-to, and the packet is delivered to the jail on the bridge. Return traffic follows the reply-to state back through the tunnel.

Both FIBs need host routes for this address. FIB 1 needs it because tunnel traffic must find the jail:

route_fib1_v4_jail216="-host 194.28.98.216 -interface bastille0 -fib 1"

FIB 0 needs it because proto 41 decapsulation happens before PF routing decisions - the kernel must know where to deliver the inner packet:

route_v4_jail216="-host 194.28.98.216 -interface bastille0"

The Complete Flow

Inbound to a BGP-addressed service (IPv6)

 1. Client sends packet to 2a06:9801:1c:1000::10 (Caddy jail)
 2. Packet traverses the internet → AS201379 → hobgp BGP router
 3. hobgp forwards through GIF tunnel to radon (proto 41 encapsulated)
 4. radon receives proto 41 on vtnet0, decapsulates → gif0
 5. PF match: reply-to ($tun_if $bgp_hub_ip), creates state entry
 6. Forwarded to bastille0 → Caddy jail
 7. Caddy responds, packet exits on bastille0
 8. PF state table triggers reply-to: send via gif0 to bgp_hub_ip
 9. gif0 encapsulates (proto 41) using tunnelfib 0 → vtnet0 → hobgp
10. hobgp receives, forwards to upstream → internet → client

Outbound from a pure public jail (IPv4)

1. testvnet jail sends packet from 194.28.98.216
2. Packet arrives on bastille0
3. PF match: "from $testvnet_v4 → rtable 1"
4. Kernel routes via FIB 1 → default route → gif0
5. gif0 encapsulates using tunnelfib 0 → vtnet0 → hobgp
6. hobgp receives, forwards to internet (source: 194.28.98.216)

NAT’d jail egress (IPv4, standard path)

1. Caddy jail sends packet from 10.254.254.10
2. Packet arrives on bastille0
3. PF match: "from <jails_v4> to ! $jail_net" (standard egress)
4. Kernel routes via FIB 0 → default route → vtnet0
5. NAT translates source to 152.53.147.5
6. Packet exits to Netcup → internet

Verification

From the host: testing both FIBs

The setfib command selects which routing table a command uses:

# Via Netcup (FIB 0) - physical uplink
root@radon:~ # setfib 0 fetch -4 -o - -q https://ifconfig.co
152.53.147.5

root@radon:~ # setfib 0 fetch -6 -o - -q https://ifconfig.co
2a0a:4cc0:c1:2f90::2

# Via BGP tunnel (FIB 1) - bound to the routed IP
root@radon:~ # setfib 1 fetch --bind-address=194.28.98.217 -4 -o - -q https://ifconfig.co
194.28.98.217

root@radon:~ # setfib 1 fetch -6 -o - -q https://ifconfig.co
2a06:9801:1c:ffff::2

FIB 0 returns Netcup’s addresses. FIB 1 returns BGP addresses. Two completely independent paths to the internet from the same machine.

From inside the pure public jail

root@radon:~ # bastille cmd testvnet fetch -4 -o - -q https://ifconfig.co
194.28.98.216

root@radon:~ # bastille cmd testvnet fetch -6 -o - -q https://ifconfig.co
2a06:9801:1c:1000::99

The jail sees its own real public IP - not a NATed address. Both IPv4 and IPv6 traffic exits through the BGP tunnel and appears on the internet with the correct source addresses.

From external clients: both paths work

# Via Netcup (default DNS, resolves to 152.53.147.5)
~ ❯ curl -Iv4 http://radon.edelga.se
* IPv4: 152.53.147.5
* Connected to radon.edelga.se (152.53.147.5) port 80
< HTTP/1.1 308 Permanent Redirect
< Server: Caddy

# Via BGP tunnel (resolves to 2a06:9801:1c:1000::10)
~ ❯ curl -Iv6 http://blog.hofstede.it
* IPv6: 2a06:9801:1c:1000::10
* Connected to blog.hofstede.it (2a06:9801:1c:1000::10) port 80
< HTTP/1.1 308 Permanent Redirect
< Server: Caddy

# Via BGP tunnel (IPv4, routed through 194.28.98.217)
~ ❯ curl -Iv4 http://194.28.98.217
* Connected to 194.28.98.217 (194.28.98.217) port 80
< HTTP/1.1 308 Permanent Redirect
< Server: Caddy

Three different entry points, all reaching the same Caddy jail - which only has a 10.254.254.10 private address. The same service is accessible through Netcup’s network (physical) and through the BGP tunnel (logical), and Caddy doesn’t know or care which path the request took.

Security Considerations

Dual-FIB setups require extra security attention because more routing paths mean more attack surface:

kern_securelevel=2 prevents loading kernel modules and writing to raw disk devices post-boot. Even if root is compromised, the attacker can’t modify the running kernel or bypass the firewall.

SSH is restricted to known IPs on a non-standard port with aggressive rate limiting. The overload <bruteforce> flush global combination instantly blocks attackers and kills their existing connections:

pass in quick on $ext_if proto tcp from <trusted_v4> to any port 30822 flags S/SA keep state \
    (max-src-conn 5, max-src-conn-rate 3/30, overload <bruteforce> flush global)

SSH via the tunnel uses reply-to to ensure the management session stays within the tunnel path - an attacker on Netcup’s network can’t even see the tunnel-side SSH port.

Antispoof blocks packets with impossible source addresses for each interface, preventing attackers from injecting traffic that pretends to come from the internal network.

Monitoring is bound to the internal bridge only, keeping Prometheus exporters invisible from the internet:

node_exporter_listen_address="10.254.254.1:9100"
jail_exporter_listen_address="10.254.254.1:9452"

Lessons Learned

tunnelfib is the unsung hero. Without tunnelfib 0 on the GIF interface, the tunnel would try to route its own encapsulated packets through itself. The error mode is a silent hang - no error messages, no crash, just packets disappearing into recursive routing. If your GIF tunnel comes up but passes no traffic, check tunnelfib first.

FIB 1 needs manual completeness. Unlike FIB 0, which automatically gets interface routes when interfaces come up, FIB 1 starts empty. If you can ping the internet from FIB 1 but jails can’t reply, you’re missing a subnet route that tells FIB 1 where the bridge is. The symptom is asymmetric: inbound works (because reply-to handles replies), but jail-initiated connections fail.

reply-to and rtable solve different halves of the same problem. reply-to handles inbound connections: “replies to tunnel traffic must exit via the tunnel.” rtable handles outbound connections: “traffic from BGP addresses must route via FIB 1.” You need both - one without the other produces subtle failures.

Disable hardware offloading on forwarding hosts. LRO and TSO on virtio NICs cause PF to compute incorrect checksums for forwarded packets. The failure is invisible at the host level - everything looks fine locally - but downstream receivers silently drop the corrupted packets. The -lro -tso flags are non-negotiable when gateway_enable="YES".

MSS clamping is not optional with tunnels. Every encapsulation layer reduces the effective MTU. Without scrub all max-mss 1220, TCP connections that transfer more than ~1220 bytes of payload per segment will stall. The insidious part: small requests (DNS, HTTP redirects) work fine, making the problem seem intermittent until someone tries to download a file.

A single bridge can carry multiple routing paradigms. Private NATed traffic, natively routed IPv6, and pure public IPv4 all coexist on bastille0 without any VLAN tagging or interface separation. PF’s source-based rules handle the routing differentiation. This simplicity is a feature - fewer moving parts means fewer failure modes.

Conclusion

Dual-FIB policy routing on FreeBSD is the kind of feature that sounds complex until you see it in practice. Two routing tables, a handful of PF directives (rtable, reply-to, route-to), and some static routes. That’s the entire mechanism that lets a single server speak from two completely different address spaces through two completely different internet paths.

The configuration shown here is what runs in production. The same Caddy jail serves blog.hofstede.it via IPv6 through the BGP tunnel and radon.edelga.se via IPv4 through Netcup - simultaneously, transparently, without knowing the difference. The testvnet jail has a real public IPv4 address that’s globally routable through AS201379, with no NAT anywhere in the path. And the whole thing coexists peacefully with standard NATed jail traffic on the same bridge.

Is this setup necessary for running a blog? No. But it’s the same pattern that production multi-homed infrastructure uses at much larger scale. The tools are the same - FIBs, PF, tunnels - just running on a single virtual machine instead of a rack of routers. FreeBSD makes this accessible because these features are first-class kernel primitives, not bolted-on afterthoughts.

References

FreeBSD setfib(1) - Multiple routing table selection
FreeBSD gif(4) - Generic tunnel interface, including tunnelfib
FreeBSD pf.conf(5) - PF configuration reference
FreeBSD Handbook: Firewalls (PF)
Running Your Own AS: BGP on FreeBSD - The BGP router side of this setup
Going Multi-Homed with iBGP - The multi-PoP expansion
PF Firewall on FreeBSD: A Practical Guide - PF foundation concepts

The internet doesn’t care that all of this runs on one virtual machine. Packets arrive, get routed through the correct table, hit the correct jail, and the replies leave through the correct tunnel. Two FIBs, one bridge, zero confusion. That’s the beauty of policy routing done right - the complexity is in the configuration, not in the operation.