Tornedo.tetrain.com Runbook
Reproducible, command-by-command runbook. Companion to tornedo.tetrain.com_upgrade_plan.md (the decision log — why each choice was made, dead ends, root-cause analysis). Read that for context; use this to execute.
What this actually is: not an OS upgrade of the original box. An in-place leapp chain was attempted first and was blocked by IPA's own tooling (ipa-server does not support in-place upgrade — a hard inhibitor, not a config problem). The real migration is build a brand-new IPA server, replicate data onto it, promote it to primary, retire the original — a replica-and-promote pattern, executed through an *intermediate* bridge server because a direct replica jump from IPA 4.6 (2017) straight to IPA 4.13 failed outright (CA-cloning 403 Forbidden — a real protocol incompatibility across ~7 major versions).
Provenance note on exactness: the commands below are the literal ones recorded during execution — most of the destructive/privileged steps in this migration required Directory Manager or admin Kerberos credentials, which were deliberately never handled directly (the client ran those specific commands themselves, each time, in their own session — noted inline below wherever that applies). Everything else was run and captured directly. Re-verified live on 2026-07-25: ipa.tetrain.com is confirmed AlmaLinux 9.7 / IPA 4.13.1, with the exact 4GB /swapfile entry in /etc/fstab described in Phase 3 still present and in use.
Phase 0 — Pre-upgrade diagnostics on the original server (tornedo.tetrain.com)[edit]
Do this regardless of which upgrade strategy you end up using — both found real, independently worth-fixing problems.
0.1 Cert auto-renewal was broken (found, not caused, by this work)[edit]
Symptom: three certmonger-tracked certs (LDAP/dirsrv, httpd, KDC) all show status: MONITORING / auto-renew: yes but every renewal attempt fails with:
Error setting up ccache for "host" service on client using default keytab: Keytab contains no suitable keys for host/ip-172-26-27-51.ap-south-1.compute.internal@.
Root cause: this is an AWS EC2 instance; its live system hostname had drifted to the AWS internal DNS name instead of the enrolled FQDN (/etc/krb5.keytab only has keys for host/tornedo.tetrain.com) — almost certainly cloud-init resetting the hostname on a prior reboot.
hostnamectl set-hostname tornedo.tetrain.com # prevent recurrence — stop cloud-init from resetting it again on the next reboot cat >> /etc/cloud/cloud.cfg <<'EOF' preserve_hostname: true EOF # resubmit the three near-expiry cert requests (get the request IDs from `getcert list` first) getcert resubmit -i <ldap-request-id> getcert resubmit -i <httpd-request-id> getcert resubmit -i <kdc-request-id> # confirm getcert list # all should show status: MONITORING, no ca-error, new far-future expiry
Verified: all three cleared, new expiry 2028-07-23 on all three.
0.2 Duplicate RPM versions + point-release patch[edit]
package-cleanup --dupes # will show duplicate-version entries for ipa-server,
# 389-ds-base, krb5-server, pki-ca etc. — sign of a prior
# interrupted/incomplete update
yum check
Client-run patch update to latest available CentOS 7.9 (via Vault — 7's regular mirrors are gone):
# repoint at Vault first if not already — the box's mirrorlist is dead for a pruned EOL release sed -i 's|mirrorlist=|#mirrorlist=|; s|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|' \ /etc/yum.repos.d/CentOS-Base.repo yum update -y needs-restarting -r # confirms glibc/systemd/openssl-libs/kernel installed but not yet active reboot
After reboot:
uname -r # confirm new kernel is actually running ipactl status # every service should report RUNNING getcert list # all certs still MONITORING, no errors
Then the actual cleanup (client ran this step directly — review the transaction list before confirming, watch for grub2/dracut/kernel entries):
package-cleanup --cleandupes package-cleanup --dupes # re-verify clean yum check ipactl status getcert list
Full consistent backup, taken after the above (needed regardless of upgrade strategy):
ipa-backup --data --logs # -> /var/lib/ipa/backup/ipa-full-<timestamp>, briefly stops/restarts IPA services for # consistency (seconds, not a real outage)
Phase 1 — In-place leapp attempt (abandoned — kept for the record, don't repeat)[edit]
leapp preupgrade
This is the point where the strategy changed: leapp preupgrade reported 5 inhibitors, one of which was ipa-server does not support in-place upgrade — Red Hat's own tooling refuses to leapp-upgrade a FreeIPA/IdM server outright and points to a separate IdM migration guide. Do not attempt leapp upgrade on an IPA server — go straight to Phase 2's replica-and-promote pattern instead.
Phase 2 — Build the intermediate bridge replica (ipa8.tetrain.com, Rocky Linux 8.10)[edit]
Needed because a direct IPA-4.6-to-IPA-4.13 replica failed at CA-cloning with 403 Forbidden — too many major versions apart. This bridge server is temporary — built, used once to carry data forward, then decommissioned (Phase 4).
2.1 Base OS + firewall[edit]
# this box has firewalld active (the eventual final-target box, ipa.tetrain.com, does not — # check per-box, don't assume) firewall-cmd --permanent --add-service=freeipa-ldap firewall-cmd --permanent --add-service=freeipa-ldaps firewall-cmd --permanent --add-service=dns firewall-cmd --permanent --add-service=http firewall-cmd --permanent --add-service=https firewall-cmd --permanent --add-service=kerberos firewall-cmd --permanent --add-port=464/tcp --add-port=464/udp firewall-cmd --permanent --add-port=8080/tcp --add-port=8443/tcp firewall-cmd --reload
2.2 Fix /etc/hosts before attempting install[edit]
Cloud-init's default template mapped this box's own hostname to both 127.0.0.1 and ::1 — IPA's installer explicitly refuses a hostname that resolves to loopback.
# edit /etc/hosts: remove the loopback mapping for this box's own hostname, add a real line: # <real-ip> ipa8.tetrain.com ipa8 # stop cloud-init from reverting it on next boot: echo 'manage_etc_hosts: false' >> /etc/cloud/cloud.cfg
Also add an entry for the old server here too — it has the same AWS-default-PTR reverse-DNS mismatch as Phase 0.1, and this box needs to resolve it correctly to talk to it:
echo '<tornedo-real-ip> tornedo.tetrain.com' >> /etc/hosts
2.3 Install IPA (EL8 packages it via a DNF module, not a plain package)[edit]
dnf module enable idm:DL1 -y dnf install -y @idm:DL1/server @idm:DL1/dns # dns component per your actual topology needs
2.4 Run the replica install[edit]
Needs real admin Kerberos credentials for check_creds, not just the host keytab — run kinit and the install command together in one session that never leaves your hands (this migration's rule throughout: admin/Directory Manager passwords are never typed into or passed through the assisting session):
kinit admin ipa-replica-install --setup-ca --unattended
If a prior attempt failed and needs retrying:
ipa-server-install --uninstall -U # this also wipes client enrollment — get a fresh OTP before retrying: # on the SOURCE server: ipa host-disable ipa8.tetrain.com ; ipa host-mod --random ipa8.tetrain.com
Two non-fatal warnings are expected and fine (acmeIPAServerCert profile import failure, dnaHostname lookup timeout — both known transient issues during replica install). Immediately after a successful install, run:
ipa-server-upgrade
(re-migrates certificate profiles, clears the two warnings above).
2.5 Validate before moving on[edit]
ipactl status # all 7 services RUNNING getcert list # all certs MONITORING, no errors ldapsearch -Y EXTERNAL -H ldapi:/// -b "" -s base # LDAPI search succeeds kinit -kt /etc/krb5.keytab host/ipa8.tetrain.com # ticket issuance works
Phase 3 — Build the final target (ipa.tetrain.com, AlmaLinux 9) from the bridge[edit]
Confirmed live 2026-07-25: this box is AlmaLinux 9.7, ipa-server-4.13.1-3.el9_8.2, 389-ds-base-2.8.0-8.el9_8, krb5-server-1.21.1-10.el9_8.
Networking between ipa8 and this box was already fully open both directions on ports 389/636/88/464/80/443/8080/8443 — check this per-pair, don't assume it's always already open.
3.1 Same /etc/hosts loopback fix as Phase 2.2[edit]
Same class of bug, same fix, on this box too.
3.2 Critical: add swap before attempting --setup-ca[edit]
This box has 1.6G RAM and (originally) zero swap. The CA database init step spawns a JVM (ca-db-init) that OOMs *silently* — no debug log, no catalina output, exit code 255, no kernel OOM-killer entry (the JVM fails to reserve heap before its own logging initializes). Add swap before attempting the CA setup, not after failing once:
fallocate -l 4G /swapfile chmod 600 /swapfile mkswap /swapfile swapon /swapfile echo '/swapfile swap swap defaults 0 0' >> /etc/fstab free -h # confirm swap is present
(Confirmed live 2026-07-25: this exact /swapfile + /etc/fstab entry is still in place and in active use — 295MB in use at time of check.)
3.3 Run the replica install[edit]
kinit admin # client-run, same rule as Phase 2.4 ipa-replica-install --setup-ca --skip-mem-check --unattended
--skip-mem-check is required here specifically because of the low-RAM box above — without it the installer refuses to proceed even with swap present.
If retrying after a failed attempt, clean up in this order (all recurring issues on this hop):
# 1. stale topology/master entry left by any attempt that got past "adding master entry" # before failing later — host-disable/host-mod CANNOT touch a master entry, must force-delete: ipa server-del ipa.tetrain.com --force # 2. kill orphaned processes from a failed uninstall (ipa-server-install --uninstall often # reports success without actually stopping these) — prefer stop+pkill over `disable` for # ad-hoc cleanup, since `disable` can remove the systemd .wants/ symlink directory as a side # effect and break the NEXT install attempt's symlink creation: systemctl stop pki-tomcatd@pki-tomcat.service pkill -9 ns-slapd pkill -9 -f pki-tomcatd ps aux | grep -E 'ns-slapd|pki-tomcatd' # confirm actually gone before retrying mkdir -p /etc/systemd/system/pki-tomcatd.target.wants/ # recreate if a `disable` removed it ipa-server-install --uninstall -U
Immediately after a successful install:
ipa-server-upgrade
3.4 Validate[edit]
ipactl status kinit -kt /etc/krb5.keytab host/ipa.tetrain.com # issues a valid krbtgt/TORNEDO.TETRAIN.COM ticket ipa user-show admin # full LDAP+API roundtrip curl -sI https://ipa.tetrain.com/ipa/ui/ # expect HTTP/1.1 200 free -h # confirm swap actually got used during install
Also worth checking from a real existing production client (not the box itself) — proves actual external reachability without touching that client's persistent config:
# from any already-enrolled client: nc -zv ipa.tetrain.com 389 636 88 464 80 443 ldapsearch -x -H ldap://ipa.tetrain.com -b '' -s base namingcontexts # expect: dc=tornedo,dc=tetrain,dc=com and o=ipaca
Phase 4 — Decommission the intermediate bridge (ipa8.tetrain.com)[edit]
Do not server-del the bridge directly — it's the only link connecting the original and the final target; deleting it first splits the topology into two disconnected islands with no sync path between them (IPA's own topology check catches and blocks this correctly). Add a direct segment between the two *remaining* servers first, for both suffixes:
ipa topologysegment-add domain --leftnode=tornedo.tetrain.com --rightnode=ipa.tetrain.com ipa topologysegment-add ca --leftnode=tornedo.tetrain.com --rightnode=ipa.tetrain.com
Verify both new agreements are actually replicating before proceeding:
ipa topologysegment-find domain ipa topologysegment-find ca
Now the bridge is safe to remove:
ipa server-del ipa8.tetrain.com --force
Verify:
ipactl status # all 7 services RUNNING on ipa.tetrain.com
getcert list # all 9 certs MONITORING (ipa-custodia's renewal role
# shouldn't be disrupted even though ipa8 was its
# install-time "master peer")
ipa user-show admin # LDAP+API roundtrip still works
ipa topologysegment-find domain # only the direct tornedo<->ipa.tetrain.com link remains,
# no references to ipa8 anywhere
ipa8.tetrain.com's VM is now safe to power off.
Phase 5 — SRV-based cutover (not an IP/identity handoff)[edit]
Checked first: confirm no DNS SRV records already exist for the domain (dig — if every client is relying purely on the literal hostname, this is your situation and this approach applies). This is simpler than making ipa.tetrain.com impersonate tornedo.tetrain.com's identity (fragile, not well-supported) — instead, publish real SRV records under the *old* hostname's zone, pointing at the new server's *own* name:
_ldap._tcp.tornedo.tetrain.com. SRV 0 100 389 ipa.tetrain.com. _kerberos._tcp.tornedo.tetrain.com. SRV 0 100 88 ipa.tetrain.com. _kerberos._udp.tornedo.tetrain.com. SRV 0 100 88 ipa.tetrain.com. _kerberos-master._tcp.tornedo.tetrain.com. SRV 0 100 88 ipa.tetrain.com. _kerberos-master._udp.tornedo.tetrain.com. SRV 0 100 88 ipa.tetrain.com. _kpasswd._tcp.tornedo.tetrain.com. SRV 0 100 464 ipa.tetrain.com. _kpasswd._udp.tornedo.tetrain.com. SRV 0 100 464 ipa.tetrain.com.
(Plain client-discovery SRV records — not the _msdcs.-style records ipa-adtrust-install prints, which are for actual Windows AD trust interop and weren't needed here.)
Every sssd.conf in this fleet already has ipa_server = _srv_, tornedo.tetrain.com — SRV lookup first, literal-name fallback second — so once these records exist, enrolled clients discover and switch to ipa.tetrain.com with zero client-side config changes, and tornedo.tetrain.com's A record can stay up as a fallback the whole time.
Client published all 7 records. Verify:
dig SRV _ldap._tcp.tornedo.tetrain.com +short dig SRV _kerberos._tcp.tornedo.tetrain.com +short # ... (repeat for all 7) — all should resolve, correct priority/weight/port, target ipa.tetrain.com.
End-to-end proof it's actually working, from a real enrolled client:
systemctl restart sssd cat /var/lib/sss/pubconf/kdcinfo.TORNEDO.TETRAIN.COM # should now show ipa.tetrain.com's IP, not tornedo.tetrain.com's — this is the literal # artifact SSSD writes with its discovery result, the most reliable way to confirm SRV # discovery actually worked (vs. just "dig succeeded")
Phase 6 — Critical gotcha: missing SIDs break login for real (non-admin) users[edit]
Symptom: admin logs into ipa.tetrain.com's web UI fine; every other real user fails with the same password, even though it works on tornedo.tetrain.com. Confirmed via krb5kdc.log: a normal user's AS-REQ fails at HANDLE_AUTHDATA: ... No such file or directory during PAC generation — not PREAUTH_FAILED (not a password problem).
Root cause: ipa-replica-install --unattended silently skips SID generation for existing users when it can't prompt interactively — this is the exact warning unattended mode swallows: *"N existing users or groups do not have a SID identifier assigned... installer will NOT run ipa-sidgen task."* IPA 4.13 enforces PAC/SID generation during ticket issuance more strictly than IPA 4.6 does, so this only breaks on the *new* server despite identical replicated LDAP data.
Fix — run this after every unattended replica install that involves a version jump, don't wait for a real user to hit it:
# ipa-adtrust-install isn't installed by the base @idm:DL1/server module — separate package: dnf install -y ipa-server-trust-ad ipa-adtrust-install -U --add-sids
Verify:
ldapsearch -Y EXTERNAL -H ldapi:/// -b "dc=tornedo,dc=tetrain,dc=com" \ "(&(objectClass=posixAccount)(!(ipaNTSecurityIdentifier=*)))" uid # should return zero users ldapsearch -Y EXTERNAL -H ldapi:/// -b "uid=<real-user>,cn=users,cn=accounts,dc=tornedo,dc=tetrain,dc=com" ipaNTSecurityIdentifier # should now show S-1-5-21-...-<rid>
smb and winbind services will now also be running (side effect of ipa-adtrust-install, expected). Have the real user re-test their actual login as final confirmation.
Phase 7 — Promote the CA renewal master[edit]
The renewal master defaults to whichever server is currently the only one — needs to move explicitly once there's more than one CA replica:
ipa-csreplica-manage set-renewal-master ipa.tetrain.com
Needs the Directory Manager password specifically (more privileged than admin) — client-run, same rule as every other admin-credential step. Verify:
ldapsearch -Y EXTERNAL -H ldapi:/// -b "cn=masters,cn=ipa,cn=etc,dc=tornedo,dc=tetrain,dc=com" \ "(cn=CA)" ipaConfigString # should show: ipaConfigString: caRenewalMaster on the ipa.tetrain.com entry
Phase 8 — Client remediation reference (recurring bug classes found during fleet sweep)[edit]
Not a linear sequence — this is a diagnostic reference for any client that doesn't switch to the new server after the Phase 5 SRV records go live, encountered across ~50 hosts during the actual sweep. Check in this order:
8.1 — Simple caching lag (try first, cheapest check):
systemctl restart sssd # or: /etc/init.d/sssd restart (no-systemd boxes)
Resolves the majority of "stuck on old IP" cases by itself.
8.2 — dns_lookup_kdc = false in /etc/krb5.conf: SRV discovery is disabled at the Kerberos level entirely — no amount of waiting fixes this.
sed -i 's/dns_lookup_kdc = false/dns_lookup_kdc = true/' /etc/krb5.conf systemctl restart sssd
8.3 — Wrong dns_discovery_domain in sssd.conf: If it's set to a *shorter* suffix than the real IPA domain (e.g. tetrain.com instead of tornedo.tetrain.com), SSSD constructs and queries the wrong SRV record names entirely (_ldap._tcp.tetrain.com, which doesn't exist) — silent failure, dig with the *correct* full name still works manually, which masks this during casual diagnosis. Confirmed on 3+ hosts.
# in sssd.conf, correct only dns_discovery_domain — leave ipa_domain alone if other config # depends on it (some long-established clients have ipa_domain = tetrain.com tied to other # working config; don't touch it, just fix dns_discovery_domain): sed -i 's/dns_discovery_domain = tetrain.com/dns_discovery_domain = tornedo.tetrain.com/' /etc/sssd/sssd.conf systemctl restart sssd
A host stuck on the old IP even after an aggressive sss_cache -E + full stop/start needs both 8.2 and 8.3 checked — they're independent settings causing an identical-looking symptom.
8.4 — Wrong ipa_domain (distinct from 8.3, found during a later live-outage remediation):
# correct ALL of: [domain/...] section header, domains=, ipa_domain, ipa_server static # fallback in sssd.conf, and [realms] kdc/admin_server in krb5.conf — all need to agree on # the real domain, tornedo.tetrain.com, not a shortened alias
8.5 — Host firewall allowlist (packets never leave the host): Some hosts run a default-DROP iptables OUTPUT policy with an explicit per-destination-IP allowlist that has the old server's IP hardcoded but never got the new one added — correct DNS/krb5 config doesn't help if the packets are dropped before leaving the box.
iptables -A OUTPUT -d <ipa.tetrain.com-IP> -j ACCEPT # mirror whatever rule pattern the
# old server's IP already has
iptables-save > /etc/sysconfig/iptables # or your distro's persistence mechanism
8.6 — Local DNS resolver lying (one host, but worth checking anywhere self-hosting DNS): A host whose own /etc/resolv.conf lists itself as the *first* nameserver can return a valid empty NOERROR for the SRV query — glibc's resolver treats that as final and never falls through to the other working resolvers listed after it, even though they'd succeed. Config looks completely correct; DNS is just silently lying.
# reorder /etc/resolv.conf: put a working recursive resolver FIRST, keep the local one but # deprioritized (don't remove it if other local config depends on it)
8.7 — Debian/Ubuntu-specific: ipa-client-install doesn't wire up PAM the way RHEL does. RHEL-family systems use authselect, which ipa-client-install configures automatically. Debian/Ubuntu use pam-auth-update, a separate mechanism ipa-client-install doesn't touch — enrollment can report success while login still fails completely with zero SSSD log activity (looks identical to "wrong password").
# after ipa-client-install completes on any Debian-family box, always:
pam-auth-update --force --enable sss
pam-auth-update --force --enable mkhomedir # if --mkhomedir was passed to ipa-client-install
# and doesn't seem to be creating home dirs
# --force is needed if the image ships a "local modification" in common-auth (e.g. a custom
# nullok tweak) that would otherwise block the non-forced update
grep pam_sss /etc/pam.d/common-auth /etc/pam.d/common-account # verify before calling it done
8.8 — /etc/hosts with swapped hostname labels + loopback (recurring on new-onboarding, not just the migration):
# e.g. "sysnet-zurt" written where the real hostname is "zurt-sysnet", AND pointed at 127.0.1.1 # — ipa-client-install fails outright with a Python socket.gaierror before even reaching the # usual loopback-resolution failure. Fix: correct hostname + real routable IP, same as Phase # 2.2/3.1, plus manage_etc_hosts: false in cloud-init config.
8.9 — Ubuntu 25.10+: sudo-rs silently ignores IPA sudo rules. Symptom: a user with a confirmed-valid IPA sudo rule (ipa sudorule-show <rule>) gets denied, with zero SSSD sudo-responder log activity even though libsss-sudo is installed and the SSSD sudo socket exists and is correctly permissioned. Root cause: Ubuntu 25.10 ships sudo-rs (Rust reimplementation) as the *default* /usr/bin/sudo, and it doesn't support SSSD/IPA-driven sudo rules — falls back to local-file-only sudoers evaluation with a plausible-looking denial and no diagnostic trail. Classic GNU sudo is usually already installed alongside it, just not the active alternative (registered at /usr/bin/sudo.ws on this Ubuntu convention, no dpkg-divert involved):
update-alternatives --install /usr/bin/sudo sudo /usr/bin/sudo.ws 50 update-alternatives --set sudo /usr/bin/sudo.ws sudo --version # confirm it no longer reports "sudo-rs" sudo -l -U <user> # confirm it now shows the real IPA-sourced rule
8.10 — Missing chrony on minimal/stripped images: ipa-client-install tries to configure chrony for time sync but doesn't install it if missing — fails outright at that step.
apt-get install -y chrony # or dnf/yum equivalent — before retrying ipa-client-install
Phase 9 — Security incident found + remediated during this work (fail2ban)[edit]
Not part of the migration itself, but found while investigating why admin's Kerberos account kept hitting lockout thresholds (krbPwdMaxFailure) during cleanup work — real, active exposure.
Diagnosis: krb5kdc.log showed repeated auth failures against kadmin/changepw for a long-disabled account (nsAccountLock: TRUE, expired since 2022) from relay.isgec.com, plus admin itself getting hit from two other boxes. On relay.isgec.com: ss -tnp showed multiple simultaneous SSH connections from wildly different, geographically scattered public IPs — signature of a distributed credential-stuffing botnet. That box had no fail2ban and PasswordAuthentication yes — fully exposed.
# CentOS 7, post-EOL mirrors pruned — fail2ban-server needs systemd-python, which needs a
# systemd bump unavailable on the regular (dead) mirrors:
sed -i 's|mirrorlist=|#mirrorlist=|; s|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org/7.9.2009|' \
/etc/yum.repos.d/CentOS-Base.repo
yum install -y fail2ban
systemctl is-system-running # confirm "running" after the systemd bump, no reboot needed on EL7
# (live daemon-reexec handles it)
/etc/fail2ban/jail.local:
[sshd] enabled = true maxretry = 5 bantime = 3600 findtime = 600 ignoreip = 127.0.0.1/8 <your-own-VPN-egress-IP>
systemctl enable --now fail2ban fail2ban-client status sshd # confirm jail active fail2ban-client get sshd ignoreip # confirm your own IP is whitelisted BEFORE relying on this
This picked up the already-in-progress attack immediately (214 failed attempts recorded, 5 attacking IPs banned on the spot).
This is very likely fleet-wide, not just this one host — was not swept beyond this box. Before considering this closed elsewhere: check fail2ban presence + PasswordAuthentication setting on every reachable box in the fleet, apply the same fix pattern, and consider disabling SSH password auth fleet-wide (key-only) as the actual fix rather than just rate-limiting.
Current state / what's still open[edit]
tornedo.tetrain.com(original box) — client-shut-down, not yet formally decommissioned (Phase 3-5 of the separate decommission checklist in the decision log). All hosts found still live-dependent on it at the last readiness check have since been individually remediated (Phase 8 patterns above) and verified with a livekinit -k.- Fleet-wide fail2ban sweep (Phase 9) — not done beyond
relay.isgec.com. - A handful of active clients were found hitting the old server that weren't in the original 50-host inventory at all (unidentified IPs with regular live traffic) — need identifying and properly enrolling against
ipa.tetrain.combefore the original box can be considered fully safe to power off for good.