Larvitz Blog
Mastodon

A Caching FreeBSD Mirror for DN42: nginx proxy_store, pf, and a Dual-Homed VM

Table of Contents

Since joining DN42, the number of FreeBSD machines living inside the mesh has quietly grown. They sit on fdce:73f7:a2dc::/48, they speak IPv6 to the rest of the hobbyist internet, and by design they have no route to the clearnet. That is exactly how I want them - right up until the moment a FreeBSD security advisory lands and every one of them needs to talk to update.freebsd.org.

The usual answers are all unsatisfying. Punching clearnet holes into isolated hosts defeats the point of isolating them. NAT-ing the whole DN42 segment out through the home router reintroduces the coupling I was trying to avoid. And running a full FreeBSD mirror means rsyncing terabytes of distfiles I will never touch.

So I built the fourth option: a small dual-homed FreeBSD VM called bsdmirror that faces the clearnet on one interface and DN42 on the other, and lazily caches exactly the files my machines actually request - packages from pkg.freebsd.org, base system patches from update.freebsd.org, and release tarballs from ftp.freebsd.org for bootstrapping Bastille jails. The first client to ask for a file pays the upstream round trip; everyone after that is served from local ZFS. And since the thing exists anyway, it is now a public DN42 service: any DN42 participant can point their FreeBSD hosts at bsdmirror.chofstede.dn42.

This article walks through the whole thing: the dual-homed network setup, the strict pf policy, the three nginx server blocks (including the fun part - chasing CDN redirects server-side so the cache stays transparent), the client configuration, and a real 15.0 to 15.1 upgrade plus a Bastille jail bootstrap running entirely over the mesh.

At a Glance

  • A FreeBSD VM with two interfaces: vtnet0 (clearnet, IPv4, egress only) and vtnet1 (DN42, IPv6, ingress only)
  • pf with default deny in both directions - the box can only fetch from ports 80/443/53/123 outbound and only serves the mirror ports plus SSH inbound on the DN42 side
  • nginx proxy_store as a lazy, on-demand mirror: files land on disk in the same hierarchy as the upstream URL space
  • Three services: pkg repositories on :8080, freebsd-update patches on :80, and release tarballs (for bastille bootstrap and friends) on :8081
  • A named-location trick to follow pkg.freebsd.org and ftp.freebsd.org CDN redirects inside nginx, so clients never see a 302 pointing at a clearnet host they cannot reach
  • Signature files (latest.ssl, pub.ssl) deliberately passed through uncached, so freebsd-update clients always see the current patch level
  • Clients keep verifying signatures against the official FreeBSD keys and fingerprints - the mirror never becomes a trusted party

Architecture

                         Clearnet
                            │
                    pkg.freebsd.org
                  update.freebsd.org
                    ftp.freebsd.org
                            │
                   80/443/53/123 out only
                            │
              ┌─────────────┴──────────────┐
              │  bsdmirror (FreeBSD VM)    │
              │                            │
              │  vtnet0  10.254.240.30     │  ← clearnet side (egress)
              │                            │
              │  nginx + proxy_store       │
              │  /var/cache/nginx (ZFS)    │
              │                            │
              │  vtnet1  fdce:...:d::30    │  ← DN42 side (ingress)
              └─────────────┬──────────────┘
                            │
   :8080 pkg / :80 freebsd-update / :8081 releases
                            │
              ┌─────────────┴──────────────┐
              │         DN42 Mesh          │
              │  fd00::/8 via fdce:...:d::1│
              │                            │
              │  my jails and VMs, plus    │
              │  anyone else on DN42       │
              └────────────────────────────┘

The separation is physical, not just a firewall policy: the clearnet interface carries only outbound fetches, and the DN42 interface carries only inbound client traffic. There is no forwarding between the two - bsdmirror is not a router, it is a store-and-serve middlebox. A packet from DN42 never transits to the clearnet; the only thing that crosses the boundary is an HTTP request that nginx chooses to make on its own behalf.

The Host

The rc.conf is short. Two interfaces, a default route per world, and a static route that sends all of fd00::/8 back at the DN42 gateway:

hostname="bsdmirror"

# Network Configurations
ifconfig_vtnet0="inet 10.254.240.30 netmask 0xffffff00"
ifconfig_vtnet0_ipv6="inet6 fdbd:a6b7:c88f::30 prefixlen 64"
defaultrouter="10.254.240.1"
ipv6_defaultrouter="fdbd:a6b7:c88f::1"

# DN42 interface
ifconfig_vtnet1_ipv6="inet6 fdce:73f7:a2dc:d::30 prefixlen 64"

# Static routes
ipv6_static_routes="dn42_static"
ipv6_route_dn42_static="-6 fd00::/8 fdce:73f7:a2dc:d::1"

# Security Enhancements
clear_tmp_enable="YES"
syslogd_flags="-ss"

# Core Services
zfs_enable="YES"
nginx_enable="YES"
pf_enable="YES"
pflog_enable="YES"

sshd_enable="YES"

ntpd_enable="YES"
ntpd_sync_on_start="YES"
ntpd_flags="-g -N"

The fd00::/8 static route is what makes the box reachable from the entire mesh, not just my own /48: replies to any DN42 ULA address go back through the DN42-side gateway, while everything else (the upstream fetches) follows the clearnet default route. Routing does the traffic separation before pf even gets involved.

syslogd_flags="-ss" keeps syslogd from opening network sockets, and clear_tmp_enable is one of those defaults I set on every machine and forget about.

pf: Default Deny, Both Directions

Most firewall policies are strict inbound and permissive outbound. For a caching proxy the outbound side deserves the same treatment - this machine has exactly four reasons to open a connection to the internet, so those four are the whole egress policy:

# --- Macros & Interfaces ---
ext_v4_if = "vtnet0"
cache_v6_if = "vtnet1"

# --- Options ---
set skip on lo0
set block-policy drop
set loginterface $cache_v6_if

# --- Scrubbing ---
scrub in all fragment reassemble

# --- Filtering Rules ---
# Strict Default Deny for all directions
block drop in log all
block drop out log all

# 1. Allow Egress Traffic safely (States allow replies back in)
pass out quick on $ext_v4_if proto tcp to any port { 80, 443 } keep state
pass out quick on $ext_v4_if proto udp to any port 53 keep state   # DNS
pass out quick on $ext_v4_if proto udp to any port 123 keep state  # NTP

# 2. Strict IPv6 Nginx Cache Ingress Rules on vtnet1
pass in quick on $cache_v6_if inet6 proto tcp to fdce:73f7:a2dc:d::30 port 8080 flags S/SA keep state # PKG
pass in quick on $cache_v6_if inet6 proto tcp to fdce:73f7:a2dc:d::30 port 80 flags S/SA keep state   # Updates
pass in quick on $cache_v6_if inet6 proto tcp to fdce:73f7:a2dc:d::30 port 8081 flags S/SA keep state # Releases

# 3. Essential Management
pass in quick on $cache_v6_if proto tcp from any to fdce:73f7:a2dc:d::30 port 22 flags S/SA keep state

# 4. Mandatory ICMPv6 Rules for network health/IPv6 Neighbor Discovery
pass in quick inet6 proto ipv6-icmp icmp6-type { echoreq, neighbrsol, neighbradv, toobig, timex }
pass out quick inet6 proto ipv6-icmp all

A few deliberate choices:

block drop out log all as the baseline. If this box is ever compromised through the service it exposes to a hobbyist mesh, the attacker gets a machine that can talk HTTP, HTTPS, DNS and NTP to the clearnet and nothing else. No reverse shells on high ports, no outbound SSH, no exfiltration channels beyond what the box legitimately needs. Egress filtering is cheap on a single-purpose machine and expensive to retrofit everywhere else - single-purpose boxes are where you get to actually do it.

The service rules bind to the DN42 address, not the interface alone. pass in ... to fdce:73f7:a2dc:d::30 port 8080 will not accidentally start matching if the interface ever picks up an additional address.

The ICMPv6 rules are not optional. neighbrsol/neighbradv are neighbor discovery - without them IPv6 on the segment simply dies. And toobig (Packet Too Big) matters more in DN42 than almost anywhere else: the mesh is built from WireGuard tunnels with MTUs like 1360, so path MTU discovery is doing real work on nearly every transfer. Drop toobig and large package downloads will stall in exactly the way that takes an evening of tcpdump to diagnose.

set loginterface $cache_v6_if gives me per-interface counters on the DN42 side via pfctl -s info - a cheap way to watch how much the public service is actually being used.

nginx: A Lazy Mirror with proxy_store

The interesting design decision is proxy_store instead of proxy_cache. They sound similar and behave very differently:

  • proxy_cache is a real cache: hashed object names in a private directory layout, TTLs, revalidation, eviction. Great for accelerating a website, opaque on disk.
  • proxy_store is a mirror-maker: it saves the fetched response as a plain file whose path matches the request URI, permanently. GET /FreeBSD:15:amd64/quarterly/data.pkg lands on disk at /var/cache/nginx/pkg/FreeBSD:15:amd64/quarterly/data.pkg.

For a package mirror, proxy_store is the right shape. The on-disk tree is the URL space, which means try_files can serve hits directly with zero proxy involvement, I can inspect the cache with ls, prune it with find, and the whole thing degrades gracefully into “just a directory of files that nginx serves”. After a while the cache tree looks like a partial, demand-driven copy of the upstream:

/var/cache/nginx
├── freebsd-releases
│   └── pub/FreeBSD/releases/amd64/15.1-RELEASE/...  # base.txz & friends
├── freebsd-update
│   ├── 15.0-RELEASE/amd64/...
│   ├── 15.1-RELEASE/amd64/...
│   └── to-15.1-RELEASE/amd64/bp/...   # binary patches, fetched during upgrades
└── pkg
    ├── FreeBSD:13:amd64/
    ├── FreeBSD:14:powerpc64/
    ├── FreeBSD:15:amd64/
    │   ├── base_release_0/
    │   ├── kmods_quarterly_0/
    │   └── quarterly/
    └── FreeBSD:16:aarch64/

The FreeBSD:14:powerpc64 directory is not mine - that is the public-service part of the story. Someone on DN42 patches a powerpc64 machine through this box, and the cache grows a branch I will never use myself. That is the lazy mirror working exactly as intended.

The pkg Mirror (Port 8080)

server {
    listen [fdce:73f7:a2dc:d::30]:8080 ipv6only=on;
    server_name bsdmirror.chofstede.dn42;

    # A resolver is required for dynamic upstream DNS queries
    resolver     8.8.8.8;

    # Define the local storage root
    root /var/cache/nginx/pkg;

    location / {
        # Try serving the local file from disk first, fallback to upstream
        try_files $uri @pkg_upstream;
    }

    location @pkg_upstream {
        proxy_store          on;
        proxy_store_access   user:rw group:rw all:r;

        proxy_pass           https://pkg.freebsd.org;

        proxy_set_header     Host pkg.freebsd.org;
        proxy_ssl_name       pkg.freebsd.org;
        proxy_ssl_server_name on;

        proxy_intercept_errors on;
        recursive_error_pages  on;
        error_page 301 302 307 = @pkg_redirect;
    }

    location @pkg_redirect {
        set $redir $upstream_http_location;     # the Location header
        proxy_pass            $redir;
        proxy_ssl_server_name on;

        proxy_store        on;
        proxy_store_access user:rw group:rw all:r;
        proxy_temp_path    /var/cache/nginx/pkg_tmp;
    }
}

The flow for a cache miss: try_files fails to find the file on disk, control falls through to @pkg_upstream, nginx fetches from pkg.freebsd.org over TLS, streams the response to the client, and proxy_store writes a copy into the tree. The next request for the same URI never leaves the box.

The part that took actual thought is the redirect handling. pkg.freebsd.org is not a file server - it frequently answers with an HTTP redirect to a CDN mirror. A naive proxy would forward that 302 to the client, and the client - a DN42 host with no clearnet route - would then try to follow a Location: header pointing at a host it can never reach. Dead end.

The fix is the error_page 301 302 307 = @pkg_redirect construction. With proxy_intercept_errors on, nginx treats the upstream’s redirect as an internal event instead of a response to relay. The named location @pkg_redirect grabs the Location header via $upstream_http_location, issues a second upstream request to wherever the redirect pointed, and stores that response under the original URI. The client sees exactly one request and one 200 with file contents. The redirect chase happens entirely on the clearnet side of the box.

Two supporting details make this work: the resolver directive, because a proxy_pass with a variable target forces nginx to resolve hostnames at request time rather than config-load time; and proxy_ssl_server_name on in the redirect location, because the CDN endpoint needs a proper SNI handshake and its hostname is only known at runtime.

The freebsd-update Mirror (Port 80)

server {
    listen [fdce:73f7:a2dc:d::30]:80;
    server_name bsdmirror.chofstede.dn42;

    access_log /var/log/nginx/update.access.log;
    error_log  /var/log/nginx/update.error.log;

    # Local storage for base system patches
    root /var/cache/nginx/freebsd-update;

    location ~ ^/[\w.-]+/[\w.-]+/(latest|pub)\.ssl$ {
        proxy_pass http://update.freebsd.org;
        proxy_set_header Host update.freebsd.org;
    }

    location / {
        try_files $uri @update_upstream;
    }

    location @update_upstream {
        proxy_store          on;
        proxy_store_access   user:rw group:rw all:r;

        proxy_pass           http://update.freebsd.org;
        proxy_set_header     Host update.freebsd.org;

        proxy_redirect       off;
    }
}

This one is simpler - update.freebsd.org serves plain HTTP without redirect games - but it has one rule that carries the whole design: the regex location for latest.ssl and pub.ssl.

Those files are the signed pointers that tell freebsd-update what the current patch level is. Everything else in the update tree is content-addressed - metadata and patch files are named by their SHA256 hashes, so a cached copy is either bit-identical to upstream or useless, and caching them forever is perfectly safe. But latest.ssl changes every time a new patch level is released. Cache it once with proxy_store (which has no concept of expiry) and every client behind the mirror is frozen at whatever patch level existed the day the cache was primed - silently, with valid signatures, forever. So the signature pointers are passed through uncached on every request, and only the immutable hash-named content gets stored.

That is also the answer to the obvious “why port 8080 for pkg and port 80 for updates” question, by the way: freebsd-update.conf‘s ServerName directive accepts a hostname and nothing else - no port syntax. The update mirror must live on port 80. pkg’s repository URLs are full URLs, so the pkg mirror can sit anywhere; 8080 keeps the services separated.

The Release Mirror (Port 8081)

The third server block came later, when I wanted to bootstrap Bastille jails on a DN42-only host. bastille bootstrap fetches release distribution sets (base.txz, MANIFEST) from ftp.freebsd.org, which is yet another clearnet host the machine cannot reach - and yet another upstream that redirects to geolocated CDN mirrors:

server {
    listen [fdce:73f7:a2dc:d::30]:8081 ipv6only=on;
    server_name bsdmirror.chofstede.dn42;

    resolver     8.8.8.8;

    # Local storage for base installation tarballs
    root /var/cache/nginx/freebsd-releases;

    location / {
        try_files $uri @releases_upstream;
    }

    location @releases_upstream {
        proxy_store          on;
        proxy_store_access   user:rw group:rw all:r;

        proxy_pass           https://ftp.freebsd.org;

        proxy_set_header     Host ftp.freebsd.org;
        proxy_ssl_name       ftp.freebsd.org;
        proxy_ssl_server_name on;

        # Handle mirrors that redirect to geolocated CDN mirrors smoothly
        proxy_intercept_errors on;
        recursive_error_pages   on;
        error_page 301 302 307 = @releases_redirect;
    }

    location @releases_redirect {
        set $redir $upstream_http_location;
        proxy_pass            $redir;
        proxy_ssl_name        ftp.freebsd.org;
        proxy_ssl_server_name on;

        proxy_store           on;
        proxy_store_access    user:rw group:rw all:r;
    }
}

Structurally this is the pkg mirror again: try_files for hits, proxy_store for misses, and the same error_page/$upstream_http_location pair to absorb redirects server-side. Release tarballs are the best possible content for this pattern - a 15.1-RELEASE/base.txz will never change for the rest of its existence, so a stored copy is valid forever, and at 156 MB apiece the bandwidth savings per repeat bootstrap are the largest of the three services.

What About Trust?

A caching proxy in the middle of your update path sounds like a place where supply-chain nightmares are born, so it is worth being precise about what the mirror can and cannot do.

It cannot tamper. freebsd-update verifies every metadata file against signatures made with FreeBSD’s release keys, and the content files are verified against SHA256 hashes from that signed metadata. pkg is configured with signature_type: "fingerprints" against the fingerprints shipped in the FreeBSD base system (/usr/share/keys/pkg). Bastille validates each downloaded distribution set against the SHA256 in the release MANIFEST. In every case the trust anchor lives on the client, and the mirror is just an untrusted byte transport. If it corrupts or manipulates a file, verification fails and the client refuses it.

What the mirror can do is serve stale data - which is exactly why latest.ssl and pub.ssl bypass the cache, and why pkg’s mutable repository metadata needs the housekeeping described below.

Client Configuration

On a DN42-connected FreeBSD machine, three files change. For the base system, /etc/freebsd-update.conf:

ServerName bsdmirror.chofstede.dn42

For packages, a repository override. I use /etc/pkg/FreeBSD.conf variants pointing at the mirror, keeping the fingerprint verification exactly as stock:

FreeBSD-ports: {
  url: "http://bsdmirror.chofstede.dn42:8080/${ABI}/quarterly",
  mirror_type: "none",
  signature_type: "fingerprints",
  fingerprints: "/usr/share/keys/pkg",
  enabled: yes
}
FreeBSD-ports-kmods: {
  url: "http://bsdmirror.chofstede.dn42:8080/${ABI}/kmods_quarterly_${VERSION_MINOR}",
  mirror_type: "none",
  signature_type: "fingerprints",
  fingerprints: "/usr/share/keys/pkg",
  enabled: yes
}
FreeBSD-base: {
  url: "http://bsdmirror.chofstede.dn42:8080/${ABI}/base_release_${VERSION_MINOR}",
  mirror_type: "none",
  signature_type: "fingerprints",
  fingerprints: "/usr/share/keys/pkgbase-${VERSION_MAJOR}",
  enabled: no
}

mirror_type: "none" matters: the stock configuration uses mirror_type: "srv", which makes pkg look up SRV records to pick a mirror. Pointing at a single explicit host, that lookup is pointless at best and harmful at worst - "none" tells pkg to use the URL exactly as written.

The ${ABI} and ${VERSION_MINOR} expansions are what make the same config file portable across every FreeBSD machine in the mesh regardless of version and architecture - and what naturally grows the cache tree per-ABI on the mirror side, like that FreeBSD:14:powerpc64 branch.

And for jail hosts, one line in /usr/local/etc/bastille/bastille.conf points release bootstrapping at the mirror:

bastille_url_freebsd="http://bsdmirror.chofstede.dn42:8081/pub/FreeBSD/releases/"

The path mirrors ftp.freebsd.org‘s layout exactly, because the cache tree is that layout - proxy_store again.

The Proof: 15.0 to 15.1 Over the Mesh

The real test was the 15.1-RELEASE upgrade on a DN42-only host, running entirely through the mirror:

root@nethost:~ # freebsd-update -r 15.1-RELEASE upgrade
src component not installed, skipped
Looking up bsdmirror.chofstede.dn42 mirrors... none found.
Fetching metadata signature for 15.0-RELEASE from bsdmirror.chofstede.dn42... done.
Fetching metadata index... done.
Inspecting system... done.

The following components of FreeBSD seem to be installed:
kernel/generic world/base

Does this look reasonable (y/n)? y

Fetching metadata signature for 15.1-RELEASE from bsdmirror.chofstede.dn42... done.
Fetching 1 metadata patches. done.
Applying metadata patches... done.
Preparing to download files... done.
Fetching 3194 patches.....10....20....30 [...] ....3190.. done.
Applying patches... done.
Fetching 661 files... ....10....20 [...] ....660 done.

To install the downloaded upgrades, run 'freebsd-update [options] install'.
root@nethost:~ # freebsd-update install
Creating snapshot of existing boot environment... done.
Installing updates...
Kernel updates have been installed.  Please reboot and run
'freebsd-update [options] install' again to finish installing updates.

3194 binary patches and 661 full files, fetched over WireGuard tunnels through the DN42 mesh, cached on the way through - the second machine I upgraded pulled almost everything from local disk. Those patches are the to-15.1-RELEASE/amd64/bp/ directory in the cache tree above.

One line worth demystifying: Looking up bsdmirror.chofstede.dn42 mirrors... none found. is not an error. freebsd-update queries SRV records (_http._tcp.<ServerName>) to discover mirror pools; a plain hostname with no SRV records yields “none found” and a clean fallback to the hostname itself. Everything after that line is the answer that matters.

And the same host, bootstrapping a Bastille jail from the release mirror:

root@nethost:~ # bastille bootstrap 15.1-RELEASE

Bootstrapping FreeBSD release: 15.1-RELEASE

Fetching MANIFEST...
/usr/local/bastille/cache/15.1-RELEASE/MANIFES        1044  B   11 MBps    00s

Fetching distfile: base.txz
/usr/local/bastille/cache/15.1-RELEASE/base.tx         156 MB 3530 kBps    45s

Validating checksum for archive: base.txz
MANIFEST: 3768988b151c20f965679062b065c63a977d6bbb9f47fd83695ec2c40790c18f
DOWNLOAD: 3768988b151c20f965679062b065c63a977d6bbb9f47fd83695ec2c40790c18f

Extracting archive: base.txz

Bootstrap successful!

156 MB of base.txz at 3.5 MB/s through the mesh, checksum-verified against the MANIFEST on arrival - and now sitting in the cache, so the next jail host that bootstraps 15.1 gets it at local-network speed.

The Full Jail Lifecycle

What makes this satisfying is that the entire Bastille jail lifecycle now runs through the mirror, because every mechanism Bastille wraps happens to be one of the three services. bastille etcupdate bootstrap pulls src.txz (another 241 MB) through the release mirror on :8081:

root@nethost:~ # bastille etcupdate bootstrap 15.1-RELEASE

Attempting to bootstrap etcupdate release: 15.1-RELEASE...

Fetching distfile: src.txz
/usr/local/bastille/cache/15.1-RELEASE/src.txz         241 MB 3557 kBps 01m09s

Validating checksum for archive: src.txz
MANIFEST: cf5762da53fd52e1eaf0f9ceee9bf58cbe314c821031d0d9ffa76823185a89e1
DOWNLOAD: cf5762da53fd52e1eaf0f9ceee9bf58cbe314c821031d0d9ffa76823185a89e1

Etcupdate bootstrap complete: 15.1-RELEASE

bastille update patches the bootstrapped release with freebsd-update under the hood, so it lands on the :80 mirror via the host’s ServerName setting:

root@nethost:~ # bastille update 15.1-RELEASE

Attempting to update release: 15.1-RELEASE
Looking up bsdmirror.chofstede.dn42 mirrors... none found.
Fetching public key from bsdmirror.chofstede.dn42... done.
Fetching metadata signature for 15.1-RELEASE from bsdmirror.chofstede.dn42... done.
Fetching 105 patches.....10....20 [...] ....100.. done.
Applying patches... done.
Installing updates... done.

And packages inside the jails come from the :8080 pkg mirror, since the jails inherit the same repository configuration:

root@nethost:~ # bastille pkg ALL upgrade

[nethack]:
Installed packages to be UPGRADED:
        ldns: 1.9.0 -> 1.9.2 [FreeBSD-ports]
        nginx: 1.28.3,3 -> 1.30.3,3 [FreeBSD-ports]
        php85: 8.5.4 -> 8.5.6 [FreeBSD-ports]
        php85-sqlite3: 8.5.4 -> 8.5.6 [FreeBSD-ports]
        python311: 3.11.15_2 -> 3.11.15_4 [FreeBSD-ports]

[nethack] [3/5] Fetching php85-8.5.6: 100%    13 MiB   3.3 MB/s    00:04
[nethack] [5/5] Fetching python311-3.11.15_4: 100%    27 MiB   3.5 MB/s    00:08

Bootstrap a release, patch it, populate its jails with packages - a jail host inside DN42 goes from empty to fully patched and provisioned without ever holding a route to the clearnet.

Operational Notes

proxy_store never expires anything, and pkg metadata is mutable. This is the one sharp edge of the design. The hash-named freebsd-update content is immutable and safe to keep forever, but pkg repository metadata (meta.conf, data.pkg, packagesite.pkg) changes in place upstream whenever the quarterly branch gets updates. A stored copy pins clients to that snapshot of the repository. The pragmatic fix is a periodic cleanup job that drops the metadata files and lets the next request re-fetch them:

# /etc/cron.d/bsdmirror - refresh pkg repo metadata daily
0 4 * * * root find /var/cache/nginx/pkg -type f \
    \( -name meta.conf -o -name 'data.pkg' -o -name 'packagesite.*' \) \
    -mtime +1 -delete

Individual package files are versioned in their filenames, so they never need invalidation - old versions just stop being requested and can be pruned by age whenever disk pressure suggests it. Release tarballs are the same story: 15.1-RELEASE/base.txz is immutable for its lifetime, and an entire release directory can be dropped wholesale once that release is EOL.

Disk usage stays civil. A lazy mirror only holds what was actually requested. After the 15.1 upgrade wave plus normal package traffic across several ABIs, the cache sits comfortably in the low tens of gigabytes - compared to the multi-terabyte footprint of a full mirror.

Watch the temp path. proxy_temp_path /var/cache/nginx/pkg_tmp keeps in-flight downloads on the same filesystem as the store, so the final placement is an atomic rename rather than a cross-device copy. If interrupted transfers ever litter that directory, they are safe to delete.

Public DN42 Service

Running FreeBSD on DN42?

bsdmirror.chofstede.dn42 is open to the whole mesh. Point ServerName in /etc/freebsd-update.conf at it, use http://bsdmirror.chofstede.dn42:8080/${ABI}/quarterly as a pkg repository URL, or fetch release sets from http://bsdmirror.chofstede.dn42:8081/pub/FreeBSD/releases/ for bastille bootstrap and friends - signature and checksum verification against the official FreeBSD keys keeps working unchanged, so you are not extending any trust to me. If your architecture or release is not cached yet, the first fetch warms it for everyone after you. Feedback and peering requests: peering@hofstede.it.

Lessons Learned

proxy_store is the forgotten middle ground between a cache and a mirror. Everyone reaches for proxy_cache or a full rsync mirror; proxy_store gives you a demand-driven partial mirror with a human-readable on-disk layout for about fifteen lines of config. For content that is immutable and hash-named - which describes most package infrastructure - it is close to ideal.

Know which files in a repository are mutable, and treat them differently. The whole correctness story of this mirror rests on two observations: freebsd-update’s content is hash-addressed and safe to store forever, while latest.ssl/pub.ssl and pkg’s repository metadata are pointers that must stay fresh. Get that split wrong and you build a mirror that silently freezes clients in the past, with valid signatures on stale data.

Server-side redirect chasing keeps isolated clients isolated. The error_page 301 302 307 = @named_location plus $upstream_http_location pattern is obscure but powerful: the proxy absorbs the upstream’s CDN topology so clients on a network that cannot reach that CDN never have to know it exists.

Egress filtering is finally practical on single-purpose machines. “Default deny outbound” is aspirational on a general-purpose server and trivial on a box whose entire job is fetching files from two well-known hostnames. The pf policy here is short enough to audit in one screen.

freebsd-update’s constraints shape the design. No port syntax in ServerName, SRV-based mirror discovery, signature files that rotate per patch level - none of this is documented as “how to build a mirror”, but all of it dictates one. Reading the client’s behavior is the specification.

FreeBSD’s entire update ecosystem funnels through three hostnames. pkg, freebsd-update, and every Bastille subcommand - bootstrap, etcupdate, update, jail packages - ultimately fetch from pkg.freebsd.org, update.freebsd.org, or ftp.freebsd.org. Mirror those three and there is no fourth thing to chase; a jail host can go from empty to fully provisioned without a clearnet route. That coherence is a quiet advantage of a base-system operating system, and it is what made this project three nginx server blocks instead of a zoo of special cases.

A private itch makes a decent public service. The marginal cost of opening this to all of DN42 was one pf rule scoped to the mesh and a line in the wiki. In exchange, the cache warms itself with architectures I do not run, and other people’s isolated FreeBSD boxes get a patch path. That trade is the DN42 ethos in miniature.

Conclusion

There is a small FreeBSD VM in my network with a foot in two worlds: one interface that may only fetch from the FreeBSD project’s servers, and one that may only serve the hobbyist mesh. Between them sit nginx, a directory tree that grows to mirror exactly the slice of FreeBSD’s infrastructure my machines (and now other people’s machines) actually use, and a pf policy short enough to memorize.

The DN42-side hosts got what they needed - a patch path without a clearnet route - and the design stays honest about trust: the mirror moves bytes, the clients verify them, and the only real failure mode (staleness) is engineered around by refusing to cache the few files that change. If you run FreeBSD inside DN42, the mirror is yours to use.

References

Comments

You can use your Mastodon or other ActivityPub account to comment on this article by replying to the associated post.

Search for the copied link on your Mastodon instance to reply.

Loading comments...