An IdM Replica at Home: Kerberos That Survives a Dead Tunnel

Table of Contents

My identity setup has a single point of failure, and it is not a server. It is a tunnel.

The realm IPA.EXAMPLE.COM is served by two Red Hat Identity Management (IdM) servers, idm1 and idm2. Both run on different nodes of a Proxmox VE cluster in a data center. My home network reaches them through a site-to-site WireGuard tunnel between two OPNsense firewalls, one at home and one in the CoLo.

A quick note on naming: Red Hat Identity Management (IdM) is Red Hat’s supported, RHEL-integrated distribution of the upstream FreeIPA project. The ipa commands, the installers, the replication topology, and the DNS locations in this post work the same way in FreeIPA. I say IdM because that is what runs here, on RHEL 10.

All hostnames, addresses, and network names in this post are examples. The structure is real, the names are not.

At home, SSH goes through a single choke point: a jail called jump01 on my FreeBSD hypervisor hv01. It accepts GSSAPI and nothing else. No keys, no passwords, no keyboard-interactive. If I want a shell on hv01, I need a Kerberos service ticket for jump01, and to get one, I need a KDC.

As long as the tunnel is up, that works well. When it is down, my existing tickets keep working until they expire, and after that I am locked out of my own hypervisor from my own sofa. I have a PiKVM and a second way in through the firewall, so this is an inconvenience rather than a disaster. But it is a silly dependency, and it bothered me.

So the silly idea: put a third IdM replica at home.

Should You Even Do This?

I was not sure at first, and the answer is not an automatic yes.

What a local replica gives you:

• A KDC and LDAP server that do not care about the tunnel. Tickets and user lookups keep working at home. When the tunnel comes back, the replica catches up with whatever changed.
• Lower latency. A ticket request no longer makes a round trip through the tunnel.

What it does not give you:

• Anything else behind the tunnel. Satellite, other internal services, and the IdM web UI in the data center stay unreachable. A locally issued ticket does not fix routing.
• Anything for a laptop that already copes offline. My laptop logs in with a smartcard and works without a KDC (that is a story for another post). It just has no fresh ticket afterwards.

What it costs:

• A full copy of the realm. Every IdM replica holds every Kerberos key and every password hash. A replica at home means the most valuable system I run now also sits on a home hypervisor, in its backups, and in its snapshots.
• Another server to patch and monitor. It also needs reliable time and open ports in both directions through the tunnel. That last part turned out to be the interesting one.

If you only want Kerberos to work without the tunnel, IdM’s KDC proxy is the leaner option. It carries Kerberos over HTTPS, needs one reverse proxy entry, and works on the road as well, without copying the realm anywhere.

What tipped the decision for me was the SSH setup. With GSSAPI as the only way into jump01, a KDC at home is not a nice-to-have. It is the difference between “the tunnel is down, annoying” and “the tunnel is down, where is the PiKVM password”.

Two decisions came out of that weighing:

1. The replica gets its own VM. I already had a RHEL 10 VM at home, app01, but it runs a rootful container with a network-facing Java application. If that container is ever compromised, the attacker is root on the host. I do not want that host to also hold the realm.
2. No CA role. The CA private key stays where it already is, on the two servers in the data center. More on what that means later.

The Starting Point

  Home (hv01, FreeBSD 15, vm-bhyve)               Data center (Proxmox VE)
 ┌────────────────────────────────────┐        ┌─────────────────────────┐
 │  LAN     192.168.10.0/24           │        │  idm1                   │
 │  IOT     192.168.20.0/24           │        │  idm2                   │
 │  WORK    192.168.30.0/24           │        │  satellite              │
 │                                    │        │  ...                    │
 │  fw-home (OPNsense VM) ── wg0 ─────┼────────┼── fw-dc (OPNsense)      │
 │                                    │  WG    │  172.16.0.0/16          │
 │  jump01 (jail, SSH choke point)    │        │                         │
 └────────────────────────────────────┘        └─────────────────────────┘

Home DNS is Unbound on the home OPNsense (fw-home). It answers for home.example.com, iot.example.com, and work.example.com itself, and forwards ipa.example.com through the tunnel to the IdM servers, which run the zone with integrated DNS.

Clients at home have to resolve that zone when the tunnel is down, so the replica needs the DNS role as well. Otherwise the KDC at home is perfectly healthy and nobody can find it.

The Real Blocker: A Tunnel That Only Worked One Way

The first serious obstacle had nothing to do with IdM.

Home to data center worked. Data center to home did not. Years ago, I had set up the tunnel so that the home LAN was masqueraded behind a single tunnel address. The roughly ten networks in the data center had no idea my private home ranges existed, and I preferred to keep it that way.

That is fatal for a replica. IdM replication is not a client pulling from a server. Every server opens its own connections to its replication partners to push changes, and the installer checks connectivity in both directions before it changes anything. Behind masquerading, nothing from the data center would ever reach the replica.

The fix does not require announcing all home networks to the data center. It only requires one address, the replica’s own, to be routable in both directions without NAT. A port forward on the masqueraded address would technically work too, but then the replica would have one address at home and another one in the data center, and IdM finds its peers by hostname. One name, one set of addresses, everywhere. Anything else is a debugging session waiting to happen.

When I looked at the NAT rules on the home OPNsense, the situation turned out better than I remembered:

• IPv4: the masquerading rule only matched LAN net towards 172.16.0.0/16. A new network would simply not be covered by it. On the CoLo side, the home peer’s allowed IPs only contained the tunnel address 10.99.0.2/32, so return traffic for anything else had nowhere to go.
• IPv6: the CoLo peer already allowed my entire 2001:db8:aa00::/56. IPv6 had been crossing the tunnel unmasqueraded and routable in both directions all along. I had just never needed it from that side.

Which sets the priorities nicely. The replica speaks IPv6 first, and IPv4 gets a single /32 added to the CoLo peer, so that clients and tools that pick the A record do not run into a timeout. Legacy IP is a fallback, not the plan.

A DMZ That Only Exists Inside the Hypervisor

The replica gets its own network, the DMZ: 192.168.40.0/24 and 2001:db8:aa00:40::/64. Clients from the home network reach it through the firewall. The replica itself is not allowed to open anything towards my other home networks. If it is ever compromised, it does not get a free path into the rest of the house.

It also needs no SSID and no switch port. Nothing ever plugs into this network directly. fw-home is a VM on the same host, so the network only has to exist inside hv01.

My existing layout already follows that pattern. Each network is a host bridge, vm-bhyve attaches to it as a manual switch, and the OPNsense VM has one NIC per network:

root@hv01:~ # vm switch list
NAME     TYPE    IFACE    ADDRESS  PRIVATE  MTU  VLAN  PORTS
vm-lan   manual  bridge0  n/a      no       n/a  n/a   n/a
vm-iot   manual  bridge1  n/a      no       n/a  n/a   n/a
vm-mgmt  manual  bridge2  n/a      no       n/a  n/a   n/a
vm-work  manual  bridge3  n/a      no       n/a  n/a   n/a

Only the trunk towards the access point carries tags. So the DMZ becomes one more bridge, with no physical member at all:

sysrc cloned_interfaces+=" bridge4"
sysrc ifconfig_bridge4="up"
ifconfig bridge4 create up
vm switch create -t manual -b bridge4 vm-dmz

Two details matter here:

• No addm. bridge4 has no physical port and no VLAN interface. The network never appears on the trunk, and nobody can plug a cable into it.
• No address on the bridge. hv01 itself has no IP in the DMZ, so the hypervisor is not reachable from it.

fw-home gets a fifth NIC, appended at the end so that the existing interfaces in OPNsense keep their order:

network0_type="virtio-net"
network0_switch="vm-lan"  # Home
network1_type="virtio-net"
network1_switch="vm-iot"  # IoT
network2_type="virtio-net"
network2_switch="vm-mgmt"  # MGMT
network3_type="virtio-net"
network3_switch="vm-work"  # WORK
network4_type="virtio-net"
network4_switch="vm-dmz"  # DMZ

vm-bhyve generates the MAC address on first start and writes it into the file itself. My first attempt had vm-work in the last line, copy and paste being what it is. That would have put the DMZ interface straight into the network for my work devices. Read your diffs.

bhyve cannot hot-add NICs, so the new device only appears when bhyve starts the VM again. I did not rely on vm restart or a guest reboot and did a full stop and start of fw-home. Restarting the firewall VM takes down internet, Wi-Fi routing, and the tunnel, and in my case also the KDC path for any new SSH login. I made sure I had an open session on hv01 and the PiKVM as a fallback before I touched it. It came back fine, as it usually does when you prepare for it not to.

In OPNsense, the new NIC shows up as one more vtnet interface. I assigned it as DMZ with static addresses in both families.

Addressing

The replica got:

• idm-home.ipa.example.com
• 2001:db8:aa00:40::10/64
• 192.168.40.10/24

Using a global IPv6 address for an IdM server is only sane if the prefix never changes. Mine is static, part of a business line, and has not changed in years. With a dynamic prefix, I would have put a ULA prefix on the DMZ and registered only that in IdM, because the replica is supposed to work precisely when the uplink is broken, and a delegated prefix can disappear along with the uplink.

The VM is configured statically, with no SLAAC and no DHCPv6. IdM writes the server’s addresses into DNS during installation, and an extra privacy address that appears on the interface has no business ending up there.

The name is final. An IdM server cannot be renamed after installation. The VM name in bhyve does not matter to IdM. What matters is the hostname inside the guest.

Firewall Rules

With aliases, the rule set stays readable. They are identical on both firewalls:

The port list matches what the RHEL 10 documentation lists for IdM, and what the freeipa-4 and dns firewalld services open on the VM itself. Port 53 comes from dns, the rest from freeipa-4.

Home OPNsense:

CoLo OPNsense:

My LAN rule allows everything towards the DMZ, because that is how I treat my trusted LAN. The minimum the clients need is Kerberos, LDAP, HTTPS, and DNS, so IDM_TCP and IDM_UDP would work there as well.

Plus 192.168.40.10/32 in the allowed IPs of the home peer on the CoLo side. OPNsense creates the route from the allowed IPs on its own.

The DMZ rules sit above my standard “everything except INTERNAL_NETS” rule that every VLAN has. That rule lets the network reach the internet but none of my own networks, which includes the data center. Without the explicit rules above it, the replica could reach the internet but not its own replication partners.

The work network does not appear anywhere in these tables, and that is deliberate. The work network is for work devices, including customer laptops. Nothing from there gets near my private infrastructure, and certainly not a server holding every key of the realm.

Three things I tripped over on the way:

• “No internet” in the installer was not the firewall. Unbound on OPNsense simply was not listening on the new interface. DNS timeouts look exactly like a routing problem until you ping an IP address.
• A missing UDP rule. On the CoLo WireGuard interface, I had cloned the TCP rule and never created its UDP twin. TCP checks would have passed, and Kerberos and DNS over UDP would have hung. Reviewing all four rule sets side by side caught it.
• ICMP. I allow ICMP everywhere, via floating rules that apply to every internal interface. Twenty years of debugging path MTU problems through tunnels will do that to you. On a new wg0 inbound rule set, though, ICMP was not covered, and my first ping test from the data center failed for reasons that said nothing about TCP or UDP.

Test Both Directions Before Installing Anything

The direction that never existed before is data center to home. Before running a single IdM command, I wanted proof that it worked on a real IdM port, and that the source addresses arrived unchanged.

Nothing listens on 389 on a fresh VM, so I started a throwaway listener on it. The VM’s own firewall has to allow the port first, otherwise the test fails at the last hop. Opening the IdM services now saves a step later anyway:

firewall-cmd --permanent --add-service={freeipa-4,dns}
firewall-cmd --reload
cd $(mktemp -d) && python3 -m http.server 389 --bind ::

The --bind :: is not optional. My first attempt without it reported “Serving HTTP on 0.0.0.0”, listened on IPv4 only, and the IPv6 test from the data center failed. I spent a minute suspecting the firewall before reading the listener’s own output.

From idm1:

[admin@idm1 ~]$ curl -s -o /dev/null -w '%{http_code} %{remote_ip}\n' --connect-timeout 5 'http://[2001:db8:aa00:40::10]:389/'
200 2001:db8:aa00:40::10
[admin@idm1 ~]$ curl -s -o /dev/null -w '%{http_code} %{remote_ip}\n' --connect-timeout 5 http://192.168.40.10:389/
200 192.168.40.10

The listener’s log on the VM shows the source of each request. It has to be the real address of idm1, not a tunnel address. For the other direction, the Satellite’s request log did the job for free once the VM registered there:

Started POST "/register" for 2001:db8:aa00:40::10

Both families, both directions, real addresses. Only now does IdM get involved.

Detour: Satellite Says 422

The VM gets its content from my Satellite, like every other RHEL system I run. Registration failed with an HTTP 422 from the host initialization step, after the subscription-manager part had already succeeded. The Satellite log was clear about why:

Not queueing Host::Managed: ["Ip does not match selected subnet"]

The host group I registered into has a fixed subnet from the data center. A host in 192.168.40.0/24 cannot be in it. The fix was a nested host group, “Home-DMZ”, that inherits everything from the parent and only overrides the IPv4 and IPv6 subnets with two new ones for the DMZ.

The next attempt failed again, with a different message:

"Subnet is not defined for host's location", "Subnet is not defined for host's organization"

I had created the new subnets in the “Any Organization / Any Location” context, and subnets created there are not assigned to any location or organization. With a specific context selected, Satellite assigns them automatically. Once they were assigned, registration went through. Insights then failed with a 401, which is expected in my setup and harmless. Everything that mattered had already happened.

Installing the Replica

On idm1, with an admin ticket, the DNS records go in first. The installer would try to add them itself, but the host is not allowed to, and having them in place early lets you check resolution before the installer does:

ipa dnsrecord-add ipa.example.com idm-home \
    --aaaa-rec=2001:db8:aa00:40::10 --a-rec=192.168.40.10

Then the host entry with a one-time password, and membership in ipaservers, which allows the host to promote itself to a replica:

ipa host-add idm-home.ipa.example.com --random
ipa hostgroup-add-member ipaservers --hosts idm-home.ipa.example.com

The one-time password means I never type an admin password on the new machine. It fits a setup where admins authenticate with smartcards, and it is the cleaner path anyway.

On the VM, with the firewall already open from the listener test:

dnf -y install ipa-server ipa-server-dns
ipa-client-install --domain ipa.example.com --realm IPA.EXAMPLE.COM \
    --mkhomedir --password 'ONE-TIME-PASSWORD'

The client installer complained that the host had no A or AAAA record, although I had just created both. Most likely, Unbound at home had cached a negative answer from an earlier lookup. A few minutes later, four queries against three resolvers all returned the right addresses. The replica installer checks its own name early and aborts if it does not resolve, so it is worth checking before you start it:

dig +short idm-home.ipa.example.com AAAA
dig +short @idm1.ipa.example.com idm-home.ipa.example.com AAAA

Then the promotion, in tmux:

ipa-replica-install --setup-dns --forwarder 2001:db8:aa00:40::1 --no-reverse \
    --ip-address 2001:db8:aa00:40::10 --ip-address 192.168.40.10

• --setup-dns makes the replica a DNS server for ipa.example.com. Without it, the KDC at home works and nobody finds it when the tunnel is down.
• --no-reverse keeps IdM away from reverse zones. The home ranges belong to the OPNsense.
• --ip-address twice, explicitly, so that only these two addresses end up in DNS.
• No --setup-ca. The replica gets its certificates from the existing CA.

The installer opened with a question:

Unable to resolve the IP address 172.16.1.12 to a host name, check /etc/hosts and DNS name resolution
Lookup failed: Preferred host idm-home.ipa.example.com does not provide DNS.
Reverse DNS resolution of address ... (idm2.ipa.example.com) failed. Clients may not function properly. Please check your DNS setup.
Continue? [no]:

My data center servers have never had PTR records. IdM sets rdns = false in its Kerberos configuration, and the two existing servers have replicated happily for years without them. The second line is informational: the VM is not a DNS server yet, that is what this run is for. So: yes.

After that, Connection check OK, 40 steps of Directory Server setup, an initial replication that took 26 seconds, KDC, httpd, and DNS. Three lines in the output looked alarming and were not:

• “Replica DNS records could not be added on master: Insufficient access”. The host may not create its own records. They were already there.
• “Could not add certificate to service … no modifications to be performed”. The certificate was already in the service entry.
• “Global forwarders: 198.51.100.53, Forward policy: only”. This one made me look twice. My IdM DNS has global forwarders configured, and the installer points out that they override the forwarders in the local named.conf. That does not mean they win over --forwarder. The installer stores the option as a per-server setting in LDAP, and the per-server setting takes precedence over the global one:

$ ipa dnsserver-show idm-home.ipa.example.com
  Server name: idm-home.ipa.example.com
  SOA mname override: idm-home.ipa.example.com.
  Forwarders: 2001:db8:aa00:40::1
  Forward policy: only

So the replica sends foreign names to Unbound on the home OPNsense, as intended, while the data center servers keep using their own forwarders. If the internet connection is down, it still answers for ipa.example.com, which is the only thing I need from it in that situation.

And then: The ipa-replica-install command was successful.

Topology and Locations

The installer creates one replication segment, to the server it installed against. Discovery had picked idm2. A replica connected to a single partner loses replication when that partner goes away, so the second segment is manual. Segments are bidirectional, so one call per pair is enough:

ipa topologysegment-add domain idm-home-to-idm1 \
    --leftnode idm-home.ipa.example.com --rightnode idm1.ipa.example.com

Then a DNS location for home, and the replica in it:

ipa location-add home --description "Home network DMZ"
ipa server-mod idm-home.ipa.example.com --location home

named on the replica needs a restart to pick up the change. After that, the replica answers SRV queries with a location-specific view. It points to home._locations and lists itself with priority 0, ahead of the data center servers with 50:

$ dig +short @idm-home.ipa.example.com _kerberos._udp.ipa.example.com SRV
_kerberos._udp.home._locations.ipa.example.com.
50 100 88 idm2.ipa.example.com.
50 100 88 idm1.ipa.example.com.
0 100 88 idm-home.ipa.example.com.

The resulting domain topology is a triangle, while CA replication stays a single line between the two data center servers:

$ ipa topologysegment-find domain
  Segment name: idm1.ipa.example.com-to-idm2.ipa.example.com
  Segment name: idm-home-to-idm1
  Segment name: idm-home.ipa.example.com-to-idm2.ipa.example.com

$ ipa topologysegment-find ca
  Segment name: idm1.ipa.example.com-to-idm2.ipa.example.com

Why There Is No CA at Home

The topology graph in the web UI shows the missing CA very clearly: dashed CA lines between the two data center servers, nothing to idm-home. That is the point.

• It is not needed for what I want. Kerberos, LDAP, DNS, and smartcard logins all work without a local CA. The KDC validates my card certificate against the CA certificate it already has, and it received its own PKINIT certificate during installation.
• A CA at home means a copy of the CA private key at home. That is the single most valuable secret in the environment, and it already exists redundantly on two servers.
• What does not work with the tunnel down: issuing new certificates, and renewing the replica’s own. The latter is due every two years, and certmonger keeps retrying until the tunnel is back.

DNS at Home, and Where Locations Stop Helping

On the home OPNsense, Unbound now forwards ipa.example.com to three servers instead of one: the local replica over IPv6, and both data center servers. Previously, there was only idm1, which also means home DNS for the zone had been one server away from failing all along.

I assumed Unbound would settle on the fastest server, the local one, and every client at home would get the location-aware answer. A dig from my laptop seemed to confirm it.

It does not work that way. The location only shapes answers that come from idm-home. The data center servers have no location assigned and answer the same question differently:

== idm-home
_kerberos._udp.home._locations.ipa.example.com.
50 100 88 idm2.ipa.example.com.
50 100 88 idm1.ipa.example.com.
0 100 88 idm-home.ipa.example.com.
== idm1
0 100 88 idm2.ipa.example.com.
0 100 88 idm-home.ipa.example.com.
0 100 88 idm1.ipa.example.com.

All three servers, all priority 0. And Unbound does not stick to the fastest upstream. It picks randomly among all servers whose round-trip time is within a band of the best one. At 25 ms through the tunnel versus 1 ms locally, all three are well inside that band. So roughly two out of three answers for my home clients came without any preference for the local replica. My laptop’s dig had simply been lucky.

The obvious alternative, forwarding the zone only to idm-home, would make the location work every time and make all name resolution for the zone at home depend on a single VM. I would rather keep three forwarders for resilience and pin the preference on the clients that care. There are exactly three of them.

The Clients

The Laptop

My Fedora laptop is not an IdM client. It uses a local SSSD setup for smartcard logins with PKINIT, and has the KDC configured explicitly. The replica goes in front:

[realms]
    IPA.EXAMPLE.COM = {
        kdc = idm-home.ipa.example.com:88
        kdc = 172.16.1.12:88
        ...
    }

And in /etc/sssd/sssd.conf:

krb5_server = idm-home.ipa.example.com, 172.16.1.12

On the road, the first name either does not resolve or does not answer, and the client moves on to the second entry after a short timeout. Password changes still go to the data center, which is where they belong.

jump01, the FreeBSD Jump Host

jump01 is integrated the way I described in Integrating FreeBSD 15 with FreeIPA: native Kerberos, nslcd for LDAP, no SSSD. Both had idm1 hardcoded. Both get the replica first, and the second data center server as a third option:

/etc/krb5.conf

[libdefaults]
    default_realm = IPA.EXAMPLE.COM
    dns_lookup_kdc = false
    dns_lookup_realm = false

[realms]
    IPA.EXAMPLE.COM = {
        kdc = idm-home.ipa.example.com
        kdc = idm1.ipa.example.com
        kdc = idm2.ipa.example.com
        admin_server = idm1.ipa.example.com
    }

/usr/local/etc/nslcd.conf

uri ldap://idm-home.ipa.example.com
uri ldap://idm1.ipa.example.com
uri ldap://idm2.ipa.example.com

base dc=ipa,dc=example,dc=com
sasl_mech GSSAPI
sasl_realm IPA.EXAMPLE.COM
map passwd loginShell "/bin/sh"

nslcd works through the list from the top and fails over to the next one. The ldap/idm-home.ipa.example.com service principal was created during installation, so GSSAPI binds against the replica work without further steps.

My first check was getent passwd alice, which promptly returned an entry, and I promptly misread it. UID 1001 and a password hash: that is the local account from master.passwd, not LDAP. My own login on jump01 is a local account with a .k5login, which only needs the host keytab and my ticket. LDAP is not involved at all. The check that actually proves nslcd works is an account that exists only in IdM:

alice@jump01:~ $ getent passwd alice-adm
alice-adm:*:1734400003:1734400003:Alice Admin:/home/alice-adm:/bin/sh

A UID from the IdM range. That one came from the directory.

app01, a Regular IdM Client

app01 was enrolled with ipa-client-install and DNS discovery. Its SSSD config read ipa_server = _srv_, idm1.ipa.example.com, which looks like it should pick up the location automatically. After a restart, sssctl showed it had discovered all three servers and was still talking to idm1. That is the Unbound effect from above. The fix is the same as for the other two clients, with DNS discovery and the old server kept as fallbacks:

sed -i 's|^ipa_server = _srv_, idm1.ipa.example.com|ipa_server = idm-home.ipa.example.com, _srv_, idm1.ipa.example.com|' /etc/sssd/sssd.conf
systemctl restart sssd

[root@app01 ~]# sssctl domain-status ipa.example.com
Online status: Online

Active servers:
IPA: idm-home.ipa.example.com

On an IdM client, SSSD writes the server it is connected to into a kdcinfo file, and the Kerberos library asks SSSD’s locator plugin before it looks at krb5.conf. The hardcoded kdc line in app01‘s krb5.conf only matters when SSSD is not running.

Autostart

The last detail on hv01: the replica has to come up after the firewall.

sysrc vm_list="fw-home app01 idm-home"

A replica that does not start after a power cut is a replica that is missing precisely when the tunnel is likely to be down as well.

The Test

All of this is theory until the tunnel is actually gone. So I switched it off:

~ ❯ # tunnel off

~ ❯ ping 172.16.1.12
PING 172.16.1.12 (172.16.1.12) 56(84) bytes of data.
^C
--- 172.16.1.12 ping statistics ---
10 packets transmitted, 0 received, 100% packet loss, time 9239ms

~ took 9s ❯ kdestroy -A

~ ❯ kinit -X X509_user_identity="PKCS11:token=MyEID (Basic PIN)" alice@IPA.EXAMPLE.COM
MyEID (Basic PIN)                PIN:

~ took 4s ❯ rm ~/.ssh/controlmasters/*

~ took 2s ❯ ssh -J jump01 hv01
alice@hv01:~ $ id
uid=1001(alice) gid=1001(alice) groups=0(wheel),1001(alice)
alice@hv01:~ $ ^D
Shared connection to 192.168.99.1 closed.

The data center is unreachable. The smartcard login produces a fresh TGT in four seconds. Removing the SSH control sockets makes sure no existing connection carries the login. Then a jump through jump01 to hv01, authenticated purely with Kerberos.

The KDC log on the replica shows where the ticket came from. A PKINIT exchange, first the expected request for pre-authentication, then the ticket:

krb5kdc[18070](info): AS_REQ (...) 2001:db8:aa00:10::20: NEEDED_PREAUTH: alice@IPA.EXAMPLE.COM for krbtgt/IPA.EXAMPLE.COM@IPA.EXAMPLE.COM, Additional pre-authentication required
krb5kdc[18070](info): AS_REQ (...) 2001:db8:aa00:10::20: ISSUE: authtime 1794680169, etypes {rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha384-192(20), ses=aes256-cts-hmac-sha1-96(18)}, alice@IPA.EXAMPLE.COM for krbtgt/IPA.EXAMPLE.COM@IPA.EXAMPLE.COM

After switching the tunnel back on, the topology check on the data center side:

[admin@idm1 ~]$ ipa topologysuffix-verify domain
====================================================
Replication topology of suffix "domain" is in order.
====================================================

The silly idea from lunchtime worked by the evening.

Caveats

• The realm now lives at home. That is the trade, and I made it knowingly. The VM sits in a DMZ that cannot open connections into my other networks, holds no CA key, and its host is reachable only through the same choke point as everything else. It is still a full copy of every key in the realm, on a hypervisor at home, in its ZFS snapshots, and in its backups. Mine are encrypted, and that is not optional anymore.
• bhyve is not a supported hypervisor for RHEL. In a support case, Red Hat could ask you to reproduce the problem on a certified platform. For a home replica, I can live with that. Satellite shows the hardware model as “BHYVE” and gets a few facts wrong in its overview card, which is cosmetic.
• Locations are not magic. They only shape answers from servers in that location. If your clients reach the zone through a forwarder that spreads queries across several IdM servers, the preference applies to a fraction of answers. Pin it on the clients that matter, or forward only to the local server and accept that dependency.
• No CA at home means no certificate issuance while the tunnel is down. For me, that is a feature.
• Keep an eye on replication. ipa topologysuffix-verify domain checks the topology. For actual replication health over the tunnel, I will be watching the replica for a while before I stop thinking about it.

Conclusion

The IdM part of this project was the easy part. ipa-replica-install did exactly what the documentation says, the installer’s own connection check caught nothing because there was nothing left to catch, and the initial replication took 26 seconds.

Almost all of the work was networking. A tunnel designed years ago for one direction. A NAT rule I had to look up to find out what it actually matched. A DMZ that exists only as a bridge without members. A missing UDP rule. An Unbound that was not listening. And one assumption about DNS locations that turned out to be true for the replica and not for the forwarder in front of it.

What I would keep from this is the order of operations. Prove both directions on a real port, with real source addresses, before any identity software touches the machine. Every network problem I found was found there, where it was cheap. The replica installation itself was an anticlimax, and an anticlimax is exactly what you want from the system that holds every key you have.

References

• Installing Identity Management (RHEL 10)
• Managing replication topology (RHEL 10)
• Working with DNS in Identity Management (RHEL 10)
• FreeIPA: DNS Locations
• vm-bhyve
• Integrating FreeBSD 15 with FreeIPA: Native Kerberos and LDAP Authentication