Running Your Own AS: An IXP over VXLAN, IPv4 on the Overlay, and a Config Spring Clean

Table of Contents

Part 1 set up a single FreeBSD BGP router with two upstreams. Part 2 added a Vultr edge and iBGP. Part 3 joined LocIX Düsseldorf with a dedicated third edge. Part 4 added a fourth edge at iFog, direct peering with Hetzner on FogIXP, and brought the home LAN into the /48. This is Part 5.

Three months later, the network has fourteen BGP sessions and has stopped growing outward. What changed instead is the kind of thing that got added. The headline feature this time is an exchange point that has no switch port for me to plug into: EVIX, the Experimental Virtual Internet Exchange, delivers its peering LAN as a VXLAN tunnel. My core router terminates that tunnel directly and peers with two route servers on two continents over it, with no additional edge router involved.

The second change is less glamorous and more useful: AS201379 now carries IPv4. Not announced via BGP, but a routed /29 from one of my transit providers, delivered across the same IPv6 overlay the rest of the network runs on, using PF policy routing to keep an intentionally asymmetric path working. This site is served out of that /29.

The third change is the one that took the most hours and produces no screenshots at all: four routers’ worth of configuration got cleaned up. And because the IPv4 block has to actually land somewhere, this part also documents the most complicated firewall in the entire network, which is not on a router at all.

Note on addresses: AS201379’s prefix 2a06:9801:1c::/48 and all hofstede.it hostnames are shown as-is. So is the routed IPv4 block 194.28.98.216/29 - the site you are reading this on lives in it, so there is nothing to hide. Public peering-LAN addresses from PeeringDB (EVIX’s 2602:fed2:fff:ffff::/64, FogIXP’s 2001:7f8:ca:1::/64, LocIX’s 2a0c:b641:701::/64) are also shown as-is. Everything else - provider-assigned addresses, tunnel endpoints, trusted-management IPs - has been replaced with RFC 5737 / RFC 3849 documentation ranges. RFC 1918 addresses on private transport are shown unchanged. Upstream and peer AS numbers are visible in public routing tables.

At a Glance

Part 5 adds five things to AS201379:

• EVIX, joined over a VXLAN tunnel terminated on the core router, with route servers in Fremont and Amsterdam
• a routed IPv4 /29, carried across the IPv6 overlay and policy-routed back out the way it came
• dual-FIB routing on radon, the host at the far end of that tunnel, which has to be a normal VPS and a routed AS site at the same time
• WireGuard replacing 6to4 for the downstream MikroTik
• a configuration cleanup across all four routers: consolidated PF macros, a single peer table, deprecated provider addresses, trimmed FRR daemons

Nothing in this list makes the network bigger. All of it makes the network easier to reason about.

Architecture: Where Everything Sits Now

The four-router shape from Part 4 is unchanged. What grew is what hangs off hobgp, the core.

                    ┌──────────────────────────────────────────────────┐
                    │                Default-Free Zone                 │
                    └──┬───────────┬─────────────┬──────────────┬──────┘
                       │           │             │              │
                   AS209735    AS212895       AS41051         AS137933 / AS209762
                   (Lagrange)  (route64)   (freetransit)      (EVIX RS FMT / AMS)
                       │           │             │              │
                      GRE         GRE           GRE          VXLAN vni 107
                       │           │             │              │
                  ┌────┴───────────┴─────────────┴──────────────┴────┐
                  │                  hobgp (Core)                    │
                  │            FreeBSD + FRR, AS201379               │
                  │            2a06:9801:1c::/48 (origin)            │
                  └─┬──────────┬──────────┬────────────┬─────────────┘
                    │          │          │            │
               iBGP GIF   iBGP GIF   iBGP GIF     WireGuard
                    │          │          │            │
        ┌───────────┴──┐ ┌─────┴──────┐ ┌─┴────────┐ ┌─┴──────────────┐
        │ vtbgp        │ │ lobgp      │ │ ixbgp    │ │ MikroTik       │
        │ Vultr FRA    │ │ Servperso  │ │ iFog ZRH │ │ RouterOS 7.20  │
        │ AS64515      │ │ + LocIX RS │ │ + FogIXP │ │ :6000, :6001   │
        │              │ │ + v4 /29   │ │ + AS24940│ │                │
        └──────────────┘ └────────────┘ └──────────┘ └────────────────┘

Downstream sites tunnelled from hobgp:
   gif0  :1000::/64   radon      FreeBSD hypervisor host (blog, DNS, website)
   gif2  :2000::/64   colo       OPNsense at the Hetzner colo (management)
   gif2  :2001::/64   colo       OPNsense main network, fronting the Proxmox hosts
   gif2  :5000::/64   road       road-warrior network
   gif2  :7000::/64   dn42       DN42 node
   gif3  :3000::/64   mail       mail.linuxserver.pro
   gif4  :4000::/64   bb         burningboard.net (Mastodon)
   wg    :6000::/64   home       home LAN via MikroTik

Two things are worth reading off that diagram.

First, hobgp is now the router with the most eBGP sessions, not the fewest. In Part 2 it was a hub that happened to hold two GRE transits. It now holds three GRE transits plus an IXP fabric, while still terminating every iBGP session and every downstream tunnel.

Second, the OPNsense at the Hetzner colo has become a router in its own right. gif2 carries four separate /64s: management, the main colo network that fronts my Proxmox hosts, the road-warrior network, and a DN42 node. The actual workloads live behind it. gif0 to radon is the other end of that spectrum: a single FreeBSD hypervisor host that runs the blog, DNS and my website inside jails and bhyve guests.

The core router does not know or care about any of this. It has a static /64 route per site and a blackhole for the /48. Every site is a routing entry, not a special case.

EVIX: An Exchange Point With No Port

What EVIX Actually Is

Every IXP in Parts 3 and 4 required a machine physically (or at least virtually) attached to the exchange fabric. LocIX meant a VM at Servperso with a second NIC on the peering LAN. FogIXP meant a VM at iFog with the same arrangement. That is what “joining an IXP” normally costs: a presence at a facility that has a port.

EVIX inverts that. It is an experimental, community-run exchange whose fabric is an overlay: members receive a VXLAN tunnel to one of the EVIX VTEPs and terminate it wherever their router happens to be. The peering LAN, 2602:fed2:fff:ffff::/64, is a real shared layer-2 segment. It just happens to be built out of tunnels rather than fibre.

For a hobby AS this is a meaningful difference. There is no new VM, no new PoP, no monthly port fee, and no second NIC to argue with. My VTEP is in Zürich; my router is in Nuremberg; the exchange does not care.

The honest caveat is in the name. EVIX is experimental. The route servers carry a few hundred prefixes rather than a few thousand, and participants are overwhelmingly other hobbyist and research networks. This is not a path to eyeball networks. It is a path to a specific, interesting subset of the internet, and it costs almost nothing to have.

VXLAN on FreeBSD

FreeBSD has had a native vxlan(4) interface for years. It is configured entirely through rc.conf with a create_args_ line:

kld_list="if_gif if_gre if_wg if_vxlan"

cloned_interfaces="gif0 gif1 gif2 gif3 gif4 gif5 gif6 gre1 gre2 gre3 vxlan0"

# EVIX Zürich VXLAN
# - local endpoint (vtep): 198.51.100.10
# - remote endpoint:       203.0.113.34
# - vni: 107, dstport: 500
create_args_vxlan0="vxlanid 107 vxlanlocal 198.51.100.10 vxlanremote 203.0.113.34 \
    vxlanlocalport 500 vxlanremoteport 500 mtu 1400"
ifconfig_vxlan0_ipv6="inet6 2602:fed2:fff:ffff:6::14 prefixlen 64 -tentative"
ifconfig_vxlan0_descr="IXP-EVIX-Zurich"

A few details matter here.

Point-to-point VXLAN, not multicast. The usual VXLAN deployment uses a multicast group for BUM traffic. vxlanremote instead pins a single remote VTEP, which is exactly right for this topology: I have one tunnel, to one EVIX VTEP, and everything else on the peering LAN is reached through it.

Port 500. EVIX uses UDP port 500 rather than the IANA-assigned 4789. Port 500 is normally ISAKMP, and the practical reason to pick it is that a lot of consumer and hosting NAT gateways treat it as “VPN traffic” and pass it unmolested. Both vxlanlocalport and vxlanremoteport have to be set; FreeBSD keeps them independent.

-tentative. VXLAN is an Ethernet interface, so IPv6 duplicate address detection applies. On a peering LAN with dozens of members and a tunnel that takes a moment to come up, DAD can leave the address in tentative state long enough for FRR to fail its first bind attempt. Disabling it for this address makes boot deterministic.

MTU 1400. The chain is IPv6 payload inside Ethernet inside VXLAN inside UDP inside IPv4. The overhead is 50 bytes of VXLAN and outer Ethernet on top of 28 bytes of outer IPv4/UDP, and 1400 leaves comfortable room inside Hetzner’s 1500-byte path without relying on PMTUD working across every member’s tunnel.

The result looks like an ordinary Ethernet interface:

vxlan0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1400
        description: IXP-EVIX-Zurich
        ether 58:9c:fc:10:b0:08
        inet6 2602:fed2:fff:ffff:6::14 prefixlen 64
        vxlan vni 107 local 198.51.100.10:500 remote 203.0.113.34:500
        media: Ethernet autoselect (autoselect <full-duplex>)
        status: active

PF for a Tunnelled Peering LAN

Two rules bring the fabric up, and they sit in different halves of the ruleset.

The outer encapsulation is UDP to a single remote VTEP, and unlike the GRE and proto-41 tunnels elsewhere in this network it is fine to keep state on it, because there is exactly one flow:

evix_if = "vxlan0"

# Outer UDP encapsulation for VXLAN (port 500)
pass in  quick on $ext_if proto udp from 203.0.113.34 to ($ext_if) port 500 keep state
pass out quick on $ext_if proto udp from ($ext_if) to 203.0.113.34 port 500 keep state

Inside the tunnel, the peering LAN behaves like any other IXP interface, so it gets the same MSS clamp treatment as the rest:

scrub on { $evix_if } max-mss 1340

1340 is deliberately conservative: 1400 minus 40 bytes of IPv6 header minus 20 for TCP options headroom. Peering LANs are where MTU mismatches go to hide, and clamping the MSS on ingress is far cheaper than debugging why exactly one member’s web server hangs after the TLS handshake.

The route-server sessions themselves ride the generic BGP rules, because both EVIX route servers are members of the bgp_peers table (more on that below).

FRR: Two Route Servers, Two ASNs

EVIX runs route servers in multiple locations, each with its own AS number. I peer with two:

route-map RM-EVIX-IN permit 10
 match ipv6 address prefix-list PL-BOGONS
 set local-preference 250
exit
!
route-map RM-EVIX-OUT permit 10
 match ipv6 address prefix-list PL-MY-NET
exit
!
router bgp 201379
 ! EVIX FMT Route Server (Fremont)
 neighbor 2602:fed2:fff:ffff::1 remote-as 137933
 neighbor 2602:fed2:fff:ffff::1 description IXP-EVIX-RS-FMT
 neighbor 2602:fed2:fff:ffff::1 update-source 2602:fed2:fff:ffff:6::14
 no neighbor 2602:fed2:fff:ffff::1 enforce-first-as
 !
 ! EVIX AMS Route Server (Amsterdam)
 neighbor 2602:fed2:fff:ffff::253 remote-as 209762
 neighbor 2602:fed2:fff:ffff::253 description IXP-EVIX-RS-AMS
 neighbor 2602:fed2:fff:ffff::253 update-source 2602:fed2:fff:ffff:6::14
 no neighbor 2602:fed2:fff:ffff::253 enforce-first-as
 !
 address-family ipv6 unicast
  neighbor 2602:fed2:fff:ffff::1 activate
  neighbor 2602:fed2:fff:ffff::1 soft-reconfiguration inbound
  neighbor 2602:fed2:fff:ffff::1 maximum-prefix 2500 80 restart 30
  neighbor 2602:fed2:fff:ffff::1 route-map RM-EVIX-IN in
  neighbor 2602:fed2:fff:ffff::1 route-map RM-EVIX-OUT out
  neighbor 2602:fed2:fff:ffff::253 activate
  neighbor 2602:fed2:fff:ffff::253 soft-reconfiguration inbound
  neighbor 2602:fed2:fff:ffff::253 maximum-prefix 2500 80 restart 30
  neighbor 2602:fed2:fff:ffff::253 route-map RM-EVIX-IN in
  neighbor 2602:fed2:fff:ffff::253 route-map RM-EVIX-OUT out
 exit-address-family

Three things differ from the transit sessions.

update-source on an eBGP session. The peering-LAN address is on vxlan0, but it is not the address FRR would pick by default for a session whose next hop is reachable over the same interface as several other things. Pinning it removes the ambiguity, and it is the same pattern already used for every iBGP tunnel in this network.

no ... enforce-first-as. Same reason as FogIXP in Part 4: a transparent route server does not prepend its own ASN, so the first AS in the path is the origin, not the route server. FRR’s default sanity check would reject every update.

maximum-prefix 2500, not 350000. The transit sessions carry a full table and get a 350k ceiling. An experimental exchange should never send me more than a few hundred prefixes. Setting the ceiling near the expected value rather than near the theoretical maximum is what turns maximum-prefix from a formality into an actual safety mechanism. The 80 is the warning threshold in percent, and restart 30 brings the session back automatically half an hour after a trip rather than leaving it administratively down.

In practice the two route servers currently hand over 476 and 347 prefixes respectively - about a fifth of what LocIX sends and a tenth of FogIXP.

Where EVIX Sits in the Local-Preference Hierarchy

Local preference is transitive across iBGP, so the values set at the edges in Parts 3 and 4 arrive at the core intact. Adding EVIX at 250 slots it into an existing ladder:

EVIX sits below the two established exchanges and above transit. The reasoning is that an EVIX path is still a direct peering path and worth preferring over transit, but the fabric is an overlay of tunnels operated on a best-effort basis, so it should lose to a real IXP whenever both have a route.

Lagrange at 50 is the other end of the same idea, and the reason is purely geographic. Every router in AS201379 sits in Germany or Switzerland, and Lagrange’s PoP is on LonAP in London. For traffic to and from central Europe, sending a packet across the Channel and back is the path with the most avoidable latency, so it should never win against a Frankfurt or Zürich exit when one exists. That is not a judgement about the provider; it is a judgement about the map.

The same geography is the reason the session exists at all. A London PoP is a materially different position on the internet than Frankfurt, and a network with a presence in London has short paths to places a Frankfurt-only network does not - most obviously across the Atlantic. Nothing in the current configuration exploits that yet. When it does, it will look like the DTAG rule from Part 4: an AS-path match plus a match peer, raising local preference for a specific destination class on this specific session, rather than lifting the baseline for everything.

Its outbound route-map prepends my own ASN twice, for the same reason the local preference is low:

route-map RM-LAGRANGE-OUT permit 10
 match ipv6 address prefix-list PL-MY-NET
 set as-path prepend 201379 201379
exit

Local preference steers what leaves my network; prepending is the only real lever for what comes back. Both point the same direction here.

IPv4 Arrives, Sideways

The Situation

AS201379 is an IPv6-only AS. The /48 is mine, the AS number is mine, and there is no IPv4 allocation behind it. That is fine for the network itself and increasingly annoying for the services on it, because a meaningful share of the internet still cannot reach an IPv6-only host.

Servperso, the provider hosting the lobgp edge from Part 3, routes a small IPv4 block to my VM. It is not my address space and it is not announced by AS201379. It is a static route from their infrastructure to my edge router. Everything after that point is my problem.

The interesting part is that the block needs to be usable by services that do not live at that edge. radon, which runs the blog, is at a completely different provider, three tunnel hops away.

Carrying the Block Across the Overlay

The path is the same overlay the IPv6 network uses, with IPv4 addresses added to the point-to-point tunnels that need to carry it. On lobgp:

# Tunnel to core, now dual-stack
ifconfig_gif0="tunnel 198.51.100.50 198.51.100.10 mtu 1480"
ifconfig_gif0_alias0="inet 10.255.255.1 10.255.255.2"
ifconfig_gif0_ipv6="inet6 2a06:9801:1c:fffd::2 2a06:9801:1c:fffd::1 prefixlen 128"

gateway_enable="YES"
static_routes="hobgp_v4"
route_hobgp_v4="-net 194.28.98.216/29 10.255.255.2"

And on hobgp, the block is split up host by host and pointed at the tunnel to radon:

ifconfig_gif0="tunnel 198.51.100.10 203.0.113.5 mtu 1480"
ifconfig_gif0_alias0="inet 10.255.255.5 10.255.255.6"

static_routes="radon_v4 testvnet_v4 blog_v4"
route_radon_v4="-host 194.28.98.217 10.255.255.6"
route_testvnet_v4="-host 194.28.98.216 10.255.255.6"
route_blog_v4="-host 194.28.98.219 10.255.255.6"

The 10.255.255.0/24 addresses are RFC 1918 link addresses on the GIF tunnels. They carry no traffic of their own and exist purely so the kernel has an IPv4 next hop to point a route at. Routing a public /29 over private point-to-point links is entirely legal and slightly unnerving the first time you do it.

The Asymmetry Problem

Here is where it stops being a routing exercise.

Inbound packets for 194.28.98.219 arrive at Servperso, get routed to lobgp, cross gif0 to hobgp, cross gif0 again to radon, and are delivered. Four hops, all under my control, all fine.

The reply is the problem. radon sends it back to hobgp over the tunnel, and hobgp looks up the destination - some arbitrary internet address - in its IPv4 routing table. Its IPv4 default route points at Hetzner, its own hosting provider. So the reply leaves through Hetzner, with a source address of 194.28.98.219, from a network that has no business originating that address.

Every reasonable network drops that packet. Hetzner’s antispoofing filters see a source address that does not belong to their customer, and the connection dies silently.

The fix is a PF route-to rule that overrides the routing table for exactly this traffic:

# Inbound: edge (gif5) to app (gif0)
pass in  quick on gif5 inet from any to 194.28.98.216/29 no state
pass out quick on gif0 inet from any to 194.28.98.216/29 no state
pass out quick on $smallpox_tun inet from any to 194.28.98.220 no state

# Outbound: catch app replies and force them back to the edge
pass in  quick on gif0 route-to (gif5 10.255.255.1) \
    inet from 194.28.98.216/29 to any no state
pass in  quick on $smallpox_tun route-to (gif5 10.255.255.1) \
    inet from 194.28.98.216/29 to any no state
pass out quick on gif5 inet from 194.28.98.216/29 to any no state

route-to (gif5 10.255.255.1) says: for packets matching this rule, ignore the routing table and send them out gif5 towards 10.255.255.1, which is lobgp‘s end of the tunnel. The match condition is the source address, not the destination, which is what makes it work for arbitrary destinations.

Note that every one of these rules is no state. Keeping state on a path this asymmetric is how you produce a firewall that works for thirty seconds and then does not. The forward and return packets traverse different interfaces on this router in a way PF’s state tracking cannot follow, so state is simply not used. This is the same principle the IPv6 transit rules have followed since Part 2, applied to a case where the asymmetry is deliberate rather than incidental.

What This Buys

The site you are reading this on is served from 194.28.98.219, a globally routed IPv4 address delivered to a jail three tunnel hops away from the network that owns it. No proxy, no CDN, no NAT in front of it. The AS carries the /29 the same way it carries the /48.

Structurally it is also a demonstration that the overlay works. The /29 is transported by exactly the same tunnels, in exactly the same topology, as the IPv6 network. Adding a second address family took new routes and one policy-routing rule, not a new network.

The catch is that the complexity does not disappear, it moves. On the core router it is one route-to rule. At the far end of the tunnel, it is the most involved firewall configuration in the entire AS.

The Far End: Dual-FIB Routing on radon

radon is the FreeBSD hypervisor host that runs the blog, DNS, and the rest of the public-facing services in jails. It is also where the IPv4 story gets genuinely awkward, because it has two uplinks that have nothing to do with each other:

• vtnet0, its physical interface at Netcup, with a provider-assigned IPv4 address and a provider default route
• gif0, the tunnel to hobgp, which is how AS201379 reaches it

Both are real internet uplinks. Neither can be turned off: the tunnel needs the physical interface to exist at all, and some services deliberately want to answer on the Netcup address.

Part 1 introduced the mechanism that makes this survivable: FreeBSD’s multiple forwarding information bases. Three parts later, it has grown into something worth documenting properly.

Two Routing Tables, One Interface Each

FreeBSD can run several independent routing tables and assign an interface to one of them. radon uses two:

# FIB 0: the physical uplink (Netcup)
ifconfig_vtnet0="inet 203.0.113.5 netmask 255.255.252.0 -lro -tso"
defaultrouter="203.0.113.1"
ifconfig_vtnet0_ipv6="inet6 2001:db8:4cc0:2f90::2 prefixlen 64"
ipv6_defaultrouter="fe80::1%vtnet0"

# FIB 1: the BGP tunnel
cloned_interfaces="bridge0 gif0"
ifconfig_gif0="fib 1 tunnel 203.0.113.5 198.51.100.10 tunnelfib 0"
ifconfig_gif0_alias0="inet 10.255.255.6 10.255.255.5"
ifconfig_gif0_ipv6="inet6 2a06:9801:1c:ffff::2 2a06:9801:1c:ffff::1 prefixlen 128"

The single most important line in this file is the gif0 one, and specifically the combination of fib 1 and tunnelfib 0:

• fib 1 puts the interface, and therefore the inner payload, in routing table 1.
• tunnelfib 0 tells the kernel to resolve the outer IPv4 encapsulation - the proto-41 packets addressed to hobgp - using routing table 0.

Without the split, this is a chicken-and-egg problem. The tunnel’s own packets need to reach hobgp over the physical uplink, but if the tunnel interface lives in the same routing table as the physical one, and that table’s default route points into the tunnel, the encapsulated packets try to route themselves into the tunnel they are supposed to be building. Two FIBs break the loop cleanly: the outer wrapper uses Netcup, the inner payload uses AS201379.

The second-most-important line is -lro -tso. Large receive offload and TCP segmentation offload on a virtio NIC hand PF packets that do not look like what actually goes on the wire, and the result is that NAT and checksum rewriting silently produce corrupt output. Every FreeBSD router in this series disables both. It is the kind of thing that costs an evening exactly once.

FIB 1 Needs a Complete Map

A second routing table is not a filter, it is a whole separate routing table, and it starts empty. Anything FIB 1 is expected to reach has to be in it:

static_routes="fib1_default_v4 fib1_jailnet_v4 fib1_host217 fib1_jail219 fib0_jail219"
ipv6_static_routes="fib1_default fib1_jailnet"

# 1. How FIB 1 reaches the internet
ipv6_route_fib1_default="-6 default -interface gif0 -fib 1"
route_fib1_default_v4="-inet default 10.255.255.5 -fib 1"

# 2. How FIB 1 finds the jail bridge
ipv6_route_fib1_jailnet="-6 2a06:9801:1c:1000::/64 -interface bastille0 -fib 1"
route_fib1_jailnet_v4="-net 10.254.254.0/24 -interface bastille0 -fib 1"

# 3. Where FIB 1 finds specific public IPv4 addresses
route_fib1_host217="-host 194.28.98.217 -interface lo0 -fib 1"
route_fib1_jail219="-host 194.28.98.219 -interface bastille0 -fib 1"
route_fib0_jail219="-host 194.28.98.219 -interface bastille0"

Group 2 is the part people forget. If FIB 1 has a default route and nothing else, a packet arriving from the tunnel for a jail on the local bridge follows the default route straight back out the tunnel. The local subnets have to be present in both tables, or locally destined traffic loops.

The last two lines are the same host route in both tables, and that is deliberate: 194.28.98.219 must be reachable whether the packet arrived over the tunnel (FIB 1) or is being delivered from something in FIB 0.

Three Kinds of Jail

The bastille0 bridge carries three different networking paradigms simultaneously, and the difference between them is entirely a matter of policy routing:

The PF rules that implement this are three lines, one per paradigm:

# IPv6: forced into FIB 1. FIB 1 knows the local bridge route, so jail-to-jail
# traffic loops back locally while internet traffic goes out the tunnel.
pass in quick on bastille0 inet6 from $bgp_net_v6 to any rtable 1 keep state

# IPv4 (NAT'd): deliberately left in FIB 0 so it hits the nat rule on $ext_if
pass in quick on bastille0 inet from <jails_v4> to ! $jail_net keep state

# IPv4 (pure routed): forced straight into FIB 1
pass in quick on bastille0 inet from $frontend_v4 to any rtable 1 keep state

rtable 1 is PF’s instruction to perform the route lookup in a different table. It is the counterpart to route-to: route-to names an interface and a next hop explicitly, while rtable says “look it up, but over there”. For jail egress rtable is the better fit, because FIB 1 already contains the correct answer for every destination, including the local ones.

The NAT’d path is the odd one out and the comment in the ruleset earns its place. That traffic is left in FIB 0 on purpose, so it reaches the NAT rule on the physical interface:

nat on $ext_if inet from <jails_v4> to any -> $host_v4_routed

# The safety net: catch the NAT'd packets on their way out the physical
# interface and forcefully bounce them into the tunnel.
pass out quick on $ext_if route-to ($tun_if $tun_gw_v4) \
    inet from $host_v4_routed to any keep state

NAT is applied on $ext_if because that is where PF tracks the state accurately, and the packet is then hijacked into the tunnel by a route-to rule at the last possible moment. It reads like a hack because it is one, but the alternative - NAT directly on the tunnel interface - produces state entries that do not survive the FIB boundary.

reply-to on Every Tunnel Ingress Rule

Inbound traffic has the mirror-image problem to the core router’s. A packet arriving on gif0 for a jail will, by default, be answered according to whichever routing table the reply lands in, which may well send it out Netcup with a source address from the /29. Every ingress rule on the tunnel therefore carries an explicit reply-to:

# IPv4 tunnel ingress
pass in quick on $tun_if reply-to ($tun_if $tun_gw_v4) \
    inet proto { tcp, udp } to $frontend_v4 port { 80, 443 } keep state

# IPv6 tunnel ingress
pass in quick on $tun_if reply-to ($tun_if $bgp_hub_ip) \
    inet6 proto { tcp, udp } to $frontend_v6 port { 80, 443 } keep state
pass in quick on $tun_if reply-to ($tun_if $bgp_hub_ip) \
    inet6 proto { tcp, udp } to $powerdns_v6 port 53 keep state

reply-to pins the return path of a stateful connection to a specific interface and next hop. This is the opposite choice from the core router, which uses no state throughout, and both are correct for their position. The core forwards transit traffic whose two directions genuinely traverse different interfaces, so state is impossible. radon terminates connections, so state exists and is useful; what it needs is for the reply to ignore the routing table and go back the way the request came.

The rule of thumb that falls out of this: forwarders use no state, endpoints use reply-to. Getting that backwards produces a host that answers the first packet of every connection and then goes quiet.

The Two Extras

Two more things on this host are worth mentioning, both of which exist because it has two uplinks rather than one.

DNS answers on the Netcup address, on purpose. The authoritative nameserver is redirected from the physical interface, not the tunnel:

rdr pass on $ext_if inet proto { tcp, udp } to ($ext_if) port 53 -> $powerdns_v4
nat on $ext_if inet from $powerdns_v4 to any -> $host_v4_netcup

The nameserver for a domain should not depend on the availability of the network whose addresses it is serving. Keeping DNS on the provider’s address and its own NAT rule means a broken tunnel takes the services down but leaves the zone resolvable, which is a considerably better failure mode than the alternative.

NAT64 for IPv6-only jails. Jails that have no IPv4 address at all still occasionally need to reach an IPv4-only service. PF handles that in one rule:

pass in quick on bastille0 inet6 from $bgp_net_v6 to 64:ff9b::/96 \
    af-to inet from ($ext_if) keep state

af-to is PF’s address-family translation. Traffic to the well-known NAT64 prefix 64:ff9b::/96 is rewritten into IPv4 and sent out the physical interface. No Jool, no Tayga, no separate daemon: one rule, and IPv6-only jails get IPv4 egress.

Between the two FIBs, rtable, route-to, reply-to and af-to, this single host uses more of PF’s routing machinery than the four BGP routers combined. That is not an accident of taste. It is what happens when a machine has to be simultaneously a normal VPS at its hosting provider and a routed site inside somebody else’s autonomous system.

Downstream: WireGuard for the Home Router

Part 4 connected the home MikroTik over a 6to4 tunnel (protocol 41) at 2a06:9801:1c:fff0::/64. That worked, with one structural weakness: protocol 41 is unencrypted, unauthenticated, and gets dropped or mangled by a nontrivial number of consumer ISPs and hotel networks.

The downstream link now runs over WireGuard instead:

kld_list="if_gif if_gre if_wg if_vxlan"
wireguard_enable="YES"
wireguard_interfaces="wg-smallpox"

wg-smallpox: flags=10080c1<UP,RUNNING,NOARP,MULTICAST,LOWER_UP> metric 0 mtu 1420
        inet6 2a06:9801:1c:fff4:: prefixlen 127
        groups: wg

The /127 on the link is the RFC 6164 recommendation for point-to-point IPv6 links, and it is a cleaner fit than the /128-pair convention the GIF tunnels use. The iBGP session and the routing policy are otherwise identical to Part 4:

ipv6 prefix-list PL-MIKROTIK-NET seq 5 permit 2a06:9801:1c:6000::/64
ipv6 prefix-list PL-MIKROTIK-NET seq 10 permit 2a06:9801:1c:6001::/64
!
route-map RM-DOWNSTREAM-IN permit 10
 match ipv6 address prefix-list PL-MIKROTIK-NET
exit
!
route-map RM-DEFAULT-OUT deny 10
exit
!
router bgp 201379
 neighbor 2a06:9801:1c:fff4::1 remote-as 201379
 neighbor 2a06:9801:1c:fff4::1 description Downstream-MikroTik
 !
 address-family ipv6 unicast
  neighbor 2a06:9801:1c:fff4::1 activate
  neighbor 2a06:9801:1c:fff4::1 default-originate
  neighbor 2a06:9801:1c:fff4::1 route-map RM-DOWNSTREAM-IN in
  neighbor 2a06:9801:1c:fff4::1 route-map RM-DEFAULT-OUT out

The downstream gets a synthetic default route and nothing else; it is allowed to announce two specific /64s and nothing else. The second prefix (:6001::/64) is the reason for the change: a second segment behind the same router, which is much easier to justify adding when the transport is a protocol that survives arbitrary networks.

PF gets one macro and one clamp:

smallpox_tun = "wg-smallpox"
wg_peers_v4  = "{ 203.0.113.69, 203.0.113.118 }"

scrub on { $smallpox_tun } max-mss 1360

pass in quick on $ext_if proto udp from $wg_peers_v4 to ($ext_if) port { 51820, 51821 } keep state

WireGuard’s own overhead puts the usable MTU at 1420, so the MSS clamp lands at 1360. Unlike GRE and proto-41, the outer flow here is UDP with a real handshake, so keeping state on it is correct.

The Spring Clean

Four routers accumulated four routers’ worth of drift. None of the following changed how a single packet is forwarded. All of it made the configuration possible to read.

One Table for Every BGP Peer

The core router’s PF ruleset used to carry one pass rule per BGP session. Peer addresses now live in macros, and the macros feed a single table:

peer_lagrange    = "fd:0fca::1"
peer_route64     = "2a11:6c7:f00:119::1"
peer_freetransit = "2a01:20e:1001:27::1"
peer_vultr       = "2a06:9801:1c:fffe::2"
peer_lobgp       = "2a06:9801:1c:fffd::2"
peer_ixbgp       = "2a06:9801:1c:fffc::2"
peer_smallpox    = "2a06:9801:1c:fff4::1"
peer_evix_fmt    = "2602:fed2:fff:ffff::1"
peer_evix_ams    = "2602:fed2:fff:ffff::253"

table <bgp_peers> const {
    $peer_lagrange, $peer_route64, $peer_freetransit,
    $peer_vultr, $peer_lobgp, $peer_ixbgp, $peer_smallpox,
    $peer_evix_fmt, $peer_evix_ams }

pass in  quick proto tcp from <bgp_peers> to any port 179 keep state
pass out quick proto tcp from any to <bgp_peers> port 179 keep state

Adding a peer is now one macro and one table entry rather than a pair of rules copy-pasted from the peer above it. The const keyword makes the table immutable at runtime, so nothing can add an address to it via pfctl later.

Deprecating the Transit Addresses

The three GRE transit tunnels carry provider-assigned IPv6 addresses. Those addresses exist to bring up a BGP session and for nothing else, so the interfaces are configured with them marked deprecated:

ifconfig_gre1_ipv6="inet6 fd:0fca::2 prefixlen 64 deprecated"
ifconfig_gre2_ipv6="inet6 2a11:6c7:f00:119::2 prefixlen 64 deprecated"
ifconfig_gre3_ipv6="inet6 2a01:20e:1001:27::2/64 prefixlen 64 deprecated"

deprecated is an RFC 4862 address state that means “still valid for existing connections, never select me as a source address for a new one”. The effect is that anything the router originates picks the /48 loopback instead, without needing a policy rule to make it happen. It complements the nat-to identity rule from Part 4 rather than replacing it, and that rule now covers the VXLAN interface as well:

match out on { gre1, gre2, gre3, gif0, gif1, gif2, gif3, gif4, gif5, gif6, vxlan0 } \
    inet6 from 2001:db8:1c19:3ee1::1 to any nat-to 2a06:9801:1c::1

Prefix Lists, Interface Routes, and Daemons

Three smaller cleanups worth mentioning:

The bogon list is identical on all four routers. PL-BOGONS ends with deny ::/0 ge 49 followed by permit ::/0 le 48, which rejects anything longer than a /48 and accepts everything else that survived the preceding twenty-odd lines. Every router runs the same list, byte for byte. Divergence in a filter that is supposed to be universal is the kind of bug that only shows up in one direction of one path.

Interface routes for the iBGP next hops. hobgp carries explicit /128 routes to each edge’s tunnel address:

ipv6 route 2a06:9801:1c:fffc::2/128 gif6
ipv6 route 2a06:9801:1c:fffd::2/128 gif5
ipv6 route 2a06:9801:1c:fffe::2/128 gif1
ipv6 route 2a06:9801:1c:fff4::1/128 wg-smallpox

These are technically redundant with the connected routes the tunnel interfaces install, and they cost nothing. What they buy is that the iBGP session endpoints are named explicitly in the configuration, so show ipv6 route answers “which tunnel is this peer on” without cross-referencing ifconfig.

Trimmed FRR daemons. Every router runs the same short list:

frr_daemons="mgmtd zebra bgpd bfdd staticd"

No OSPF, no RIP, no PIM, no BABEL. The FRR exporter is scoped to match:

frr_exporter_args="--collector.bgp6 --no-collector.ospf --no-collector.bfd \
    --web.listen-address=[2a06:9801:1c::1]:9342"

Running a routing daemon you do not use is a listening socket and an attack surface for no benefit.

Monitoring: Fourteen Sessions on One Screen

Every router runs node_exporter and frr_exporter, bound to an address inside the /48 and reachable only over the overlay:

node_exporter_listen_address="[2a06:9801:1c::1]:9100"
frr_exporter_args="--collector.bgp6 ... --web.listen-address=[2a06:9801:1c::1]:9342"

The PF rules around them follow an allow-then-block pattern that is worth copying:

# Allow Prometheus -> exporters from two specific scrapers
pass quick inet6 proto tcp from { 2a06:9801:1c:2000::21, 2a06:9801:1c:2000::25 } \
    to $my_network_v6 port { 9100, 9342, 9187 } keep state

# Block those ports for the entire network, on every interface
block in quick proto tcp from any to any port { 9100, 9342 }

Order matters and quick matters. The allow rule matches first for the two scrapers and stops evaluation; everything else hits the block. Crucially the block rule sits above the stateless transit rules, so the exporters are protected not just on the router itself but for every downstream site whose traffic passes through it. A jail three hops away that carelessly binds node_exporter to a public address is covered by a rule on the core router.

The dashboard is deliberately boring. The top-left panel is fourteen stat tiles, one per eBGP session across all four routers, each of which is either UP in green or very obviously not. The three graphs underneath are the ones that actually diagnose things:

Full Table Received plots received-prefix count per session. The full-table sessions sit flat at roughly 250,000 each; the IXP route servers sit far below. The useful signal is not the absolute number, it is the shape. Every notch in that line is a session that reset and reconverged, and a session that resets repeatedly at the same time of day is a tunnel problem, not a BGP problem.

BGP Update Rate is normally near zero with occasional spikes into the hundreds of updates per second. Those spikes are somebody else’s network having a bad afternoon, propagated to me. What matters is that they end.

Interface Packet Drops is flat at zero, which is what a tunnel-based network wants to see. Drops on a GIF or VXLAN interface usually mean MTU, and MTU problems on an overlay are painful to find after the fact.

Fourteen sessions is about the point at which “check whether the sessions are up” stops being something you can do by hand. vtysh -c 'show bgp ipv6 unicast summary' on four routers is four SSH sessions and a lot of squinting; one dashboard is a glance.

Lessons Learned

An overlay IXP is a legitimate way to join an exchange. EVIX gives a small network a real peering LAN, real route servers, and real direct paths without a port, a facility, or a second machine. The prefix counts are modest and the participants are hobbyists, but the mechanics are identical to a physical exchange. If the interesting part of joining an IXP is learning to operate one, the tunnel version teaches exactly the same lesson.

FreeBSD’s vxlan(4) is a three-line rc.conf entry. No userspace daemon, no kernel module beyond if_vxlan, no configuration file. Between gif, gre, wg and vxlan, FreeBSD covers every tunnel type this network needs natively, which is a large part of why the network is built on it.

maximum-prefix is only useful if the number is realistic. A 350,000-prefix ceiling on a session that should carry 400 prefixes is not a safety mechanism, it is a comment. Setting it near the expected value is what makes it trip before a misconfigured peer fills the RIB.

Asymmetric routing needs stateless filtering and explicit route-to. The IPv4 /29 works because PF is told, in one rule, to ignore the routing table for traffic sourced from that block. Every attempt to solve this with NAT or with stateful rules produces something that appears to work and then fails on the third packet of a long-lived connection.

Forwarders use no state, endpoints use reply-to. The core router and radon face the same problem from opposite ends and need opposite answers. A router forwarding transit traffic cannot keep state, because the two directions legitimately traverse different interfaces. A host terminating connections must keep state and pin the return path instead. Applying either rule in the other’s position produces a firewall that works just long enough to look correct.

tunnelfib is the line that makes dual-uplink hosts possible. A tunnel interface whose inner payload lives in FIB 1 and whose outer encapsulation resolves in FIB 0 is the entire trick behind attaching a VPS at one provider to an autonomous system at another. It is one keyword in rc.conf, and without it the tunnel tries to route itself through itself.

deprecated is an underused address flag. One keyword in rc.conf stops a provider-assigned address from ever being chosen as a source, with no policy routing and no NAT involved. On a router with eleven interfaces and addresses from six different providers, that is worth knowing.

Cleanup is a feature. Consolidating peer addresses into a table and a set of macros did not change a single forwarding decision. It did mean that adding EVIX afterwards took two lines instead of six, and that the ruleset can be read top to bottom without cross-referencing. Every configuration cleanup pays for itself at the next change, which is precisely why it never feels urgent at the time.

Conclusion

Six months in, the shape of AS201379 has stopped changing. Four routers, a core and three edges, a fabric of tunnels, and a set of downstream sites that each get a /64 and their own opinions about firewalls. What changes now is what gets attached to that shape.

EVIX attached an entire internet exchange to it using a UDP socket. The IPv4 /29 attached a second address family using three static routes and one policy rule. The MikroTik moved from one tunnel protocol to another without the routing policy noticing. In each case the network absorbed the change rather than being redesigned around it, which is the actual return on the architecture work from Parts 1 through 4.

The remaining honest assessment: this is still a network with no IPv4 allocation of its own, whose largest peer is a hosting company, and whose total traffic would not register as noise at a real IXP. But it now peers on three exchanges across two continents, carries both address families, and tells me within a minute when any of it breaks. For a project that started as one VM and one GRE tunnel, that is a reasonable place to be.

References

• EVIX - Experimental Virtual Internet Exchange - the overlay IXP joined in this article
• FreeBSD vxlan(4) manual page - interface parameters and create_args_ syntax
• FreeBSD pf.conf(5) - route-to, scrub max-mss, tables and macros
• RFC 4862 - IPv6 stateless address autoconfiguration, including the deprecated state
• RFC 6164 - using /127 prefixes on inter-router links
• RFC 7348 - VXLAN
• RFC 6052 - the well-known NAT64 prefix 64:ff9b::/96
• FreeBSD setfib(1) - multiple forwarding information bases
• FreeBSD gif(4) - tunnelfib and the inner/outer FIB split
• FRR Documentation: BGP - maximum-prefix, enforce-first-as, route-maps
• frr_exporter - the Prometheus exporter behind the dashboard
• PeeringDB - where the EVIX, FogIXP and LocIX peering-LAN addresses come from

The thing nobody tells you about running an AS is how much of it is bookkeeping. The BGP part was solved decades ago and FRR implements it well; what actually takes the time is knowing which of eleven interfaces a prefix should leave through, why a route-map clause was added in March, and whether the tunnel that broke last night is the one carrying the thing that matters. Six months of notes, four rc.conf files, and one dashboard later, most of that is written down. The network is the easy part. Remembering why it is shaped this way is the work.