Admin: Publish runbook documentation

2026-07-25T16:41:30Z

Publish runbook documentation

New page

Reproducible, command-by-command runbook. Companion to [[Tornedo.tetrain.com Upgrade Plan|<code><nowiki>tornedo.tetrain.com_upgrade_plan.md</nowiki></code>]] (the decision log — why each choice was made, dead ends, root-cause analysis). Read that for context; use this to execute.

'''What this actually is''': not an OS upgrade of the original box. An in-place <code><nowiki>leapp</nowiki></code> chain was attempted first and was blocked by IPA's own tooling (<code><nowiki>ipa-server does not support in-place upgrade</nowiki></code> — a hard inhibitor, not a config problem). The real migration is '''build a brand-new IPA server, replicate data onto it, promote it to primary, retire the original''' — a replica-and-promote pattern, executed through an *intermediate* bridge server because a direct replica jump from IPA 4.6 (2017) straight to IPA 4.13 failed outright (CA-cloning <code><nowiki>403 Forbidden</nowiki></code> — a real protocol incompatibility across ~7 major versions).

'''Provenance note on exactness''': the commands below are the literal ones recorded during execution — most of the destructive/privileged steps in this migration required Directory Manager or <code><nowiki>admin</nowiki></code> Kerberos credentials, which were deliberately never handled directly (the client ran those specific commands themselves, each time, in their own session — noted inline below wherever that applies). Everything else was run and captured directly. Re-verified live on 2026-07-25: <code><nowiki>ipa.tetrain.com</nowiki></code> is confirmed AlmaLinux 9.7 / IPA 4.13.1, with the exact 4GB <code><nowiki>/swapfile</nowiki></code> entry in <code><nowiki>/etc/fstab</nowiki></code> described in Phase 3 still present and in use.

----

== Phase 0 — Pre-upgrade diagnostics on the original server (<code><nowiki>tornedo.tetrain.com</nowiki></code>) ==

Do this regardless of which upgrade strategy you end up using — both found real, independently worth-fixing problems.

=== 0.1 Cert auto-renewal was broken (found, not caused, by this work) ===

Symptom: three certmonger-tracked certs (LDAP/<code><nowiki>dirsrv</nowiki></code>, <code><nowiki>httpd</nowiki></code>, KDC) all show <code><nowiki>status: MONITORING</nowiki></code> / <code><nowiki>auto-renew: yes</nowiki></code> but every renewal attempt fails with:

<pre>
<nowiki>
Error setting up ccache for "host" service on client using default keytab:
Keytab contains no suitable keys for host/ip-172-26-27-51.ap-south-1.compute.internal@.
</nowiki>
</pre>

Root cause: this is an AWS EC2 instance; its live system hostname had drifted to the AWS internal DNS name instead of the enrolled FQDN (<code><nowiki>/etc/krb5.keytab</nowiki></code> only has keys for <code><nowiki>host/tornedo.tetrain.com</nowiki></code>) — almost certainly cloud-init resetting the hostname on a prior reboot.

<pre>
<nowiki>
hostnamectl set-hostname tornedo.tetrain.com

# prevent recurrence — stop cloud-init from resetting it again on the next reboot
cat >> /etc/cloud/cloud.cfg <<'EOF'
preserve_hostname: true
EOF

# resubmit the three near-expiry cert requests (get the request IDs from `getcert list` first)
getcert resubmit -i <ldap-request-id>
getcert resubmit -i <httpd-request-id>
getcert resubmit -i <kdc-request-id>

# confirm
getcert list # all should show status: MONITORING, no ca-error, new far-future expiry
</nowiki>
</pre>

Verified: all three cleared, new expiry <code><nowiki>2028-07-23</nowiki></code> on all three.

=== 0.2 Duplicate RPM versions + point-release patch ===

<pre>
<nowiki>
package-cleanup --dupes # will show duplicate-version entries for ipa-server,
# 389-ds-base, krb5-server, pki-ca etc. — sign of a prior
# interrupted/incomplete update
yum check
</nowiki>
</pre>

Client-run patch update to latest available CentOS 7.9 (via Vault — 7's regular mirrors are gone):

<pre>
<nowiki>
# repoint at Vault first if not already — the box's mirrorlist is dead for a pruned EOL release
sed -i 's|mirrorlist=|#mirrorlist=|; s|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|' \
/etc/yum.repos.d/CentOS-Base.repo
yum update -y
needs-restarting -r # confirms glibc/systemd/openssl-libs/kernel installed but not yet active
reboot
</nowiki>
</pre>

After reboot:

<pre>
<nowiki>
uname -r # confirm new kernel is actually running
ipactl status # every service should report RUNNING
getcert list # all certs still MONITORING, no errors
</nowiki>
</pre>

Then the actual cleanup (client ran this step directly — review the transaction list before confirming, watch for grub2/dracut/kernel entries):

<pre>
<nowiki>
package-cleanup --cleandupes
package-cleanup --dupes # re-verify clean
yum check
ipactl status
getcert list
</nowiki>
</pre>

Full consistent backup, taken after the above (needed regardless of upgrade strategy):

<pre>
<nowiki>
ipa-backup --data --logs
# -> /var/lib/ipa/backup/ipa-full-<timestamp>, briefly stops/restarts IPA services for
# consistency (seconds, not a real outage)
</nowiki>
</pre>

----

== Phase 1 — In-place <code><nowiki>leapp</nowiki></code> attempt (abandoned — kept for the record, don't repeat) ==

<pre>
<nowiki>
leapp preupgrade
</nowiki>
</pre>

This is the point where the strategy changed: <code><nowiki>leapp preupgrade</nowiki></code> reported 5 inhibitors, one of which was <code><nowiki>ipa-server does not support in-place upgrade</nowiki></code> — Red Hat's own tooling refuses to leapp-upgrade a FreeIPA/IdM server outright and points to a separate IdM migration guide. '''Do not attempt <code><nowiki>leapp upgrade</nowiki></code> on an IPA server''' — go straight to Phase 2's replica-and-promote pattern instead.

----

== Phase 2 — Build the intermediate bridge replica (<code><nowiki>ipa8.tetrain.com</nowiki></code>, Rocky Linux 8.10) ==

Needed because a direct IPA-4.6-to-IPA-4.13 replica failed at CA-cloning with <code><nowiki>403 Forbidden</nowiki></code> — too many major versions apart. This bridge server is temporary — built, used once to carry data forward, then decommissioned (Phase 4).

=== 2.1 Base OS + firewall ===

<pre>
<nowiki>
# this box has firewalld active (the eventual final-target box, ipa.tetrain.com, does not —
# check per-box, don't assume)
firewall-cmd --permanent --add-service=freeipa-ldap
firewall-cmd --permanent --add-service=freeipa-ldaps
firewall-cmd --permanent --add-service=dns
firewall-cmd --permanent --add-service=http
firewall-cmd --permanent --add-service=https
firewall-cmd --permanent --add-service=kerberos
firewall-cmd --permanent --add-port=464/tcp --add-port=464/udp
firewall-cmd --permanent --add-port=8080/tcp --add-port=8443/tcp
firewall-cmd --reload
</nowiki>
</pre>

=== 2.2 Fix <code><nowiki>/etc/hosts</nowiki></code> before attempting install ===

Cloud-init's default template mapped this box's own hostname to '''both''' <code><nowiki>127.0.0.1</nowiki></code> and <code><nowiki>::1</nowiki></code> — IPA's installer explicitly refuses a hostname that resolves to loopback.

<pre>
<nowiki>
# edit /etc/hosts: remove the loopback mapping for this box's own hostname, add a real line:
# <real-ip> ipa8.tetrain.com ipa8

# stop cloud-init from reverting it on next boot:
echo 'manage_etc_hosts: false' >> /etc/cloud/cloud.cfg
</nowiki>
</pre>

Also add an entry for the '''old''' server here too — it has the same AWS-default-PTR reverse-DNS mismatch as Phase 0.1, and this box needs to resolve it correctly to talk to it:

<pre>
<nowiki>
echo '<tornedo-real-ip> tornedo.tetrain.com' >> /etc/hosts
</nowiki>
</pre>

=== 2.3 Install IPA (EL8 packages it via a DNF module, not a plain package) ===

<pre>
<nowiki>
dnf module enable idm:DL1 -y
dnf install -y @idm:DL1/server @idm:DL1/dns # dns component per your actual topology needs
</nowiki>
</pre>

=== 2.4 Run the replica install ===

Needs real <code><nowiki>admin</nowiki></code> Kerberos credentials for <code><nowiki>check_creds</nowiki></code>, not just the host keytab — '''run <code><nowiki>kinit</nowiki></code> and the install command together in one session that never leaves your hands''' (this migration's rule throughout: admin/Directory Manager passwords are never typed into or passed through the assisting session):

<pre>
<nowiki>
kinit admin
ipa-replica-install --setup-ca --unattended
</nowiki>
</pre>

If a prior attempt failed and needs retrying:

<pre>
<nowiki>
ipa-server-install --uninstall -U
# this also wipes client enrollment — get a fresh OTP before retrying:
# on the SOURCE server: ipa host-disable ipa8.tetrain.com ; ipa host-mod --random ipa8.tetrain.com
</nowiki>
</pre>

Two non-fatal warnings are expected and fine (<code><nowiki>acmeIPAServerCert</nowiki></code> profile import failure, <code><nowiki>dnaHostname</nowiki></code> lookup timeout — both known transient issues during replica install). Immediately after a successful install, run:

<pre>
<nowiki>
ipa-server-upgrade
</nowiki>
</pre>

(re-migrates certificate profiles, clears the two warnings above).

=== 2.5 Validate before moving on ===

<pre>
<nowiki>
ipactl status # all 7 services RUNNING
getcert list # all certs MONITORING, no errors
ldapsearch -Y EXTERNAL -H ldapi:/// -b "" -s base # LDAPI search succeeds
kinit -kt /etc/krb5.keytab host/ipa8.tetrain.com # ticket issuance works
</nowiki>
</pre>

----

== Phase 3 — Build the final target (<code><nowiki>ipa.tetrain.com</nowiki></code>, AlmaLinux 9) from the bridge ==

Confirmed live 2026-07-25: this box is AlmaLinux 9.7, <code><nowiki>ipa-server-4.13.1-3.el9_8.2</nowiki></code>, <code><nowiki>389-ds-base-2.8.0-8.el9_8</nowiki></code>, <code><nowiki>krb5-server-1.21.1-10.el9_8</nowiki></code>.

Networking between <code><nowiki>ipa8</nowiki></code> and this box was already fully open both directions on ports 389/636/88/464/80/443/8080/8443 — check this per-pair, don't assume it's always already open.

=== 3.1 Same <code><nowiki>/etc/hosts</nowiki></code> loopback fix as Phase 2.2 ===

Same class of bug, same fix, on this box too.

=== 3.2 Critical: add swap before attempting <code><nowiki>--setup-ca</nowiki></code> ===

This box has 1.6G RAM and (originally) zero swap. The CA database init step spawns a JVM (<code><nowiki>ca-db-init</nowiki></code>) that OOMs *silently* — no debug log, no catalina output, exit code 255, no kernel OOM-killer entry (the JVM fails to reserve heap before its own logging initializes). '''Add swap before attempting the CA setup, not after failing once''':

<pre>
<nowiki>
fallocate -l 4G /swapfile
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
echo '/swapfile swap swap defaults 0 0' >> /etc/fstab
free -h # confirm swap is present
</nowiki>
</pre>

(Confirmed live 2026-07-25: this exact <code><nowiki>/swapfile</nowiki></code> + <code><nowiki>/etc/fstab</nowiki></code> entry is still in place and in active use — 295MB in use at time of check.)

=== 3.3 Run the replica install ===

<pre>
<nowiki>
kinit admin # client-run, same rule as Phase 2.4
ipa-replica-install --setup-ca --skip-mem-check --unattended
</nowiki>
</pre>

<code><nowiki>--skip-mem-check</nowiki></code> is required here specifically because of the low-RAM box above — without it the installer refuses to proceed even with swap present.

If retrying after a failed attempt, clean up in this order (all recurring issues on this hop):

<pre>
<nowiki>
# 1. stale topology/master entry left by any attempt that got past "adding master entry"
# before failing later — host-disable/host-mod CANNOT touch a master entry, must force-delete:
ipa server-del ipa.tetrain.com --force

# 2. kill orphaned processes from a failed uninstall (ipa-server-install --uninstall often
# reports success without actually stopping these) — prefer stop+pkill over `disable` for
# ad-hoc cleanup, since `disable` can remove the systemd .wants/ symlink directory as a side
# effect and break the NEXT install attempt's symlink creation:
systemctl stop pki-tomcatd@pki-tomcat.service
pkill -9 ns-slapd
pkill -9 -f pki-tomcatd
ps aux | grep -E 'ns-slapd|pki-tomcatd' # confirm actually gone before retrying
mkdir -p /etc/systemd/system/pki-tomcatd.target.wants/ # recreate if a `disable` removed it

ipa-server-install --uninstall -U
</nowiki>
</pre>

Immediately after a successful install:

<pre>
<nowiki>
ipa-server-upgrade
</nowiki>
</pre>

=== 3.4 Validate ===

<pre>
<nowiki>
ipactl status
kinit -kt /etc/krb5.keytab host/ipa.tetrain.com # issues a valid krbtgt/TORNEDO.TETRAIN.COM ticket
ipa user-show admin # full LDAP+API roundtrip
curl -sI https://ipa.tetrain.com/ipa/ui/ # expect HTTP/1.1 200
free -h # confirm swap actually got used during install
</nowiki>
</pre>

Also worth checking from a real existing production client (not the box itself) — proves actual external reachability without touching that client's persistent config:

<pre>
<nowiki>
# from any already-enrolled client:
nc -zv ipa.tetrain.com 389 636 88 464 80 443
ldapsearch -x -H ldap://ipa.tetrain.com -b '' -s base namingcontexts
# expect: dc=tornedo,dc=tetrain,dc=com and o=ipaca
</nowiki>
</pre>

----

== Phase 4 — Decommission the intermediate bridge (<code><nowiki>ipa8.tetrain.com</nowiki></code>) ==

'''Do not <code><nowiki>server-del</nowiki></code> the bridge directly''' — it's the only link connecting the original and the final target; deleting it first splits the topology into two disconnected islands with no sync path between them (IPA's own topology check catches and blocks this correctly). Add a direct segment between the two *remaining* servers first, for '''both''' suffixes:

<pre>
<nowiki>
ipa topologysegment-add domain --leftnode=tornedo.tetrain.com --rightnode=ipa.tetrain.com
ipa topologysegment-add ca --leftnode=tornedo.tetrain.com --rightnode=ipa.tetrain.com
</nowiki>
</pre>

Verify both new agreements are actually replicating before proceeding:

<pre>
<nowiki>
ipa topologysegment-find domain
ipa topologysegment-find ca
</nowiki>
</pre>

Now the bridge is safe to remove:

<pre>
<nowiki>
ipa server-del ipa8.tetrain.com --force
</nowiki>
</pre>

Verify:

<pre>
<nowiki>
ipactl status # all 7 services RUNNING on ipa.tetrain.com
getcert list # all 9 certs MONITORING (ipa-custodia's renewal role
# shouldn't be disrupted even though ipa8 was its
# install-time "master peer")
ipa user-show admin # LDAP+API roundtrip still works
ipa topologysegment-find domain # only the direct tornedo<->ipa.tetrain.com link remains,
# no references to ipa8 anywhere
</nowiki>
</pre>

<code><nowiki>ipa8.tetrain.com</nowiki></code>'s VM is now safe to power off.

----

== Phase 5 — SRV-based cutover (not an IP/identity handoff) ==

Checked first: '''confirm no DNS SRV records already exist''' for the domain (<code><nowiki>dig</nowiki></code> — if every client is relying purely on the literal hostname, this is your situation and this approach applies). This is simpler than making <code><nowiki>ipa.tetrain.com</nowiki></code> impersonate <code><nowiki>tornedo.tetrain.com</nowiki></code>'s identity (fragile, not well-supported) — instead, publish real SRV records under the *old* hostname's zone, pointing at the new server's *own* name:

<pre>
<nowiki>
_ldap._tcp.tornedo.tetrain.com. SRV 0 100 389 ipa.tetrain.com.
_kerberos._tcp.tornedo.tetrain.com. SRV 0 100 88 ipa.tetrain.com.
_kerberos._udp.tornedo.tetrain.com. SRV 0 100 88 ipa.tetrain.com.
_kerberos-master._tcp.tornedo.tetrain.com. SRV 0 100 88 ipa.tetrain.com.
_kerberos-master._udp.tornedo.tetrain.com. SRV 0 100 88 ipa.tetrain.com.
_kpasswd._tcp.tornedo.tetrain.com. SRV 0 100 464 ipa.tetrain.com.
_kpasswd._udp.tornedo.tetrain.com. SRV 0 100 464 ipa.tetrain.com.
</nowiki>
</pre>

(Plain client-discovery SRV records — '''not''' the <code><nowiki>_msdcs.</nowiki></code>-style records <code><nowiki>ipa-adtrust-install</nowiki></code> prints, which are for actual Windows AD trust interop and weren't needed here.)

Every <code><nowiki>sssd.conf</nowiki></code> in this fleet already has <code><nowiki>ipa_server = _srv_, tornedo.tetrain.com</nowiki></code> — SRV lookup first, literal-name fallback second — so once these records exist, enrolled clients discover and switch to <code><nowiki>ipa.tetrain.com</nowiki></code> with '''zero client-side config changes''', and <code><nowiki>tornedo.tetrain.com</nowiki></code>'s A record can stay up as a fallback the whole time.

Client published all 7 records. Verify:

<pre>
<nowiki>
dig SRV _ldap._tcp.tornedo.tetrain.com +short
dig SRV _kerberos._tcp.tornedo.tetrain.com +short
# ... (repeat for all 7) — all should resolve, correct priority/weight/port, target ipa.tetrain.com.
</nowiki>
</pre>

End-to-end proof it's actually working, from a real enrolled client:

<pre>
<nowiki>
systemctl restart sssd
cat /var/lib/sss/pubconf/kdcinfo.TORNEDO.TETRAIN.COM
# should now show ipa.tetrain.com's IP, not tornedo.tetrain.com's — this is the literal
# artifact SSSD writes with its discovery result, the most reliable way to confirm SRV
# discovery actually worked (vs. just "dig succeeded")
</nowiki>
</pre>

----

== Phase 6 — Critical gotcha: missing SIDs break login for real (non-admin) users ==

'''Symptom''': <code><nowiki>admin</nowiki></code> logs into <code><nowiki>ipa.tetrain.com</nowiki></code>'s web UI fine; every other real user fails with the same password, even though it works on <code><nowiki>tornedo.tetrain.com</nowiki></code>. Confirmed via <code><nowiki>krb5kdc.log</nowiki></code>: a normal user's AS-REQ fails at <code><nowiki>HANDLE_AUTHDATA: ... No such file or directory</nowiki></code> during PAC generation — '''not''' <code><nowiki>PREAUTH_FAILED</nowiki></code> (not a password problem).

'''Root cause''': <code><nowiki>ipa-replica-install --unattended</nowiki></code> silently skips SID generation for existing users when it can't prompt interactively — this is the exact warning unattended mode swallows: *"N existing users or groups do not have a SID identifier assigned... installer will NOT run ipa-sidgen task."* IPA 4.13 enforces PAC/SID generation during ticket issuance more strictly than IPA 4.6 does, so this only breaks on the *new* server despite identical replicated LDAP data.

'''Fix — run this after every unattended replica install that involves a version jump''', don't wait for a real user to hit it:

<pre>
<nowiki>
# ipa-adtrust-install isn't installed by the base @idm:DL1/server module — separate package:
dnf install -y ipa-server-trust-ad
ipa-adtrust-install -U --add-sids
</nowiki>
</pre>

Verify:

<pre>
<nowiki>
ldapsearch -Y EXTERNAL -H ldapi:/// -b "dc=tornedo,dc=tetrain,dc=com" \
"(&(objectClass=posixAccount)(!(ipaNTSecurityIdentifier=*)))" uid
# should return zero users
ldapsearch -Y EXTERNAL -H ldapi:/// -b "uid=<real-user>,cn=users,cn=accounts,dc=tornedo,dc=tetrain,dc=com" ipaNTSecurityIdentifier
# should now show S-1-5-21-...-<rid>
</nowiki>
</pre>

<code><nowiki>smb</nowiki></code> and <code><nowiki>winbind</nowiki></code> services will now also be running (side effect of <code><nowiki>ipa-adtrust-install</nowiki></code>, expected). Have the real user re-test their actual login as final confirmation.

----

== Phase 7 — Promote the CA renewal master ==

The renewal master defaults to whichever server is currently the only one — needs to move explicitly once there's more than one CA replica:

<pre>
<nowiki>
ipa-csreplica-manage set-renewal-master ipa.tetrain.com
</nowiki>
</pre>

Needs the '''Directory Manager''' password specifically (more privileged than <code><nowiki>admin</nowiki></code>) — client-run, same rule as every other admin-credential step. Verify:

<pre>
<nowiki>
ldapsearch -Y EXTERNAL -H ldapi:/// -b "cn=masters,cn=ipa,cn=etc,dc=tornedo,dc=tetrain,dc=com" \
"(cn=CA)" ipaConfigString
# should show: ipaConfigString: caRenewalMaster on the ipa.tetrain.com entry
</nowiki>
</pre>

----

== Phase 8 — Client remediation reference (recurring bug classes found during fleet sweep) ==

Not a linear sequence — this is a diagnostic reference for '''any client that doesn't switch to the new server after the Phase 5 SRV records go live''', encountered across ~50 hosts during the actual sweep. Check in this order:

'''8.1 — Simple caching lag (try first, cheapest check):'''

<pre>
<nowiki>
systemctl restart sssd # or: /etc/init.d/sssd restart (no-systemd boxes)
</nowiki>
</pre>

Resolves the majority of "stuck on old IP" cases by itself.

'''8.2 — <code><nowiki>dns_lookup_kdc = false</nowiki></code> in <code><nowiki>/etc/krb5.conf</nowiki></code>:''' SRV discovery is disabled at the Kerberos level entirely — no amount of waiting fixes this.

<pre>
<nowiki>
sed -i 's/dns_lookup_kdc = false/dns_lookup_kdc = true/' /etc/krb5.conf
systemctl restart sssd
</nowiki>
</pre>

'''8.3 — Wrong <code><nowiki>dns_discovery_domain</nowiki></code> in <code><nowiki>sssd.conf</nowiki></code>:''' If it's set to a *shorter* suffix than the real IPA domain (e.g. <code><nowiki>tetrain.com</nowiki></code> instead of <code><nowiki>tornedo.tetrain.com</nowiki></code>), SSSD constructs and queries the wrong SRV record names entirely (<code><nowiki>_ldap._tcp.tetrain.com</nowiki></code>, which doesn't exist) — silent failure, <code><nowiki>dig</nowiki></code> with the *correct* full name still works manually, which masks this during casual diagnosis. Confirmed on 3+ hosts.

<pre>
<nowiki>
# in sssd.conf, correct only dns_discovery_domain — leave ipa_domain alone if other config
# depends on it (some long-established clients have ipa_domain = tetrain.com tied to other
# working config; don't touch it, just fix dns_discovery_domain):
sed -i 's/dns_discovery_domain = tetrain.com/dns_discovery_domain = tornedo.tetrain.com/' /etc/sssd/sssd.conf
systemctl restart sssd
</nowiki>
</pre>

A host stuck on the old IP even after an aggressive <code><nowiki>sss_cache -E</nowiki></code> + full stop/start needs '''both''' 8.2 and 8.3 checked — they're independent settings causing an identical-looking symptom.

'''8.4 — Wrong <code><nowiki>ipa_domain</nowiki></code> (distinct from 8.3, found during a later live-outage remediation):'''

<pre>
<nowiki>
# correct ALL of: [domain/...] section header, domains=, ipa_domain, ipa_server static
# fallback in sssd.conf, and [realms] kdc/admin_server in krb5.conf — all need to agree on
# the real domain, tornedo.tetrain.com, not a shortened alias
</nowiki>
</pre>

'''8.5 — Host firewall allowlist (packets never leave the host):''' Some hosts run a default-DROP <code><nowiki>iptables OUTPUT</nowiki></code> policy with an explicit per-destination-IP allowlist that has the old server's IP hardcoded but never got the new one added — correct DNS/krb5 config doesn't help if the packets are dropped before leaving the box.

<pre>
<nowiki>
iptables -A OUTPUT -d <ipa.tetrain.com-IP> -j ACCEPT # mirror whatever rule pattern the
# old server's IP already has
iptables-save > /etc/sysconfig/iptables # or your distro's persistence mechanism
</nowiki>
</pre>

'''8.6 — Local DNS resolver lying (one host, but worth checking anywhere self-hosting DNS):''' A host whose own <code><nowiki>/etc/resolv.conf</nowiki></code> lists itself as the *first* nameserver can return a valid empty <code><nowiki>NOERROR</nowiki></code> for the SRV query — glibc's resolver treats that as final and never falls through to the other working resolvers listed after it, even though they'd succeed. Config looks completely correct; DNS is just silently lying.

<pre>
<nowiki>
# reorder /etc/resolv.conf: put a working recursive resolver FIRST, keep the local one but
# deprioritized (don't remove it if other local config depends on it)
</nowiki>
</pre>

'''8.7 — Debian/Ubuntu-specific: <code><nowiki>ipa-client-install</nowiki></code> doesn't wire up PAM the way RHEL does.''' RHEL-family systems use <code><nowiki>authselect</nowiki></code>, which <code><nowiki>ipa-client-install</nowiki></code> configures automatically. Debian/Ubuntu use <code><nowiki>pam-auth-update</nowiki></code>, a '''separate''' mechanism <code><nowiki>ipa-client-install</nowiki></code> doesn't touch — enrollment can report success while login still fails completely with zero SSSD log activity (looks identical to "wrong password").

<pre>
<nowiki>
# after ipa-client-install completes on any Debian-family box, always:
pam-auth-update --force --enable sss
pam-auth-update --force --enable mkhomedir # if --mkhomedir was passed to ipa-client-install
# and doesn't seem to be creating home dirs
# --force is needed if the image ships a "local modification" in common-auth (e.g. a custom
# nullok tweak) that would otherwise block the non-forced update
grep pam_sss /etc/pam.d/common-auth /etc/pam.d/common-account # verify before calling it done
</nowiki>
</pre>

'''8.8 — <code><nowiki>/etc/hosts</nowiki></code> with swapped hostname labels + loopback (recurring on new-onboarding, not just the migration):'''

<pre>
<nowiki>
# e.g. "sysnet-zurt" written where the real hostname is "zurt-sysnet", AND pointed at 127.0.1.1
# — ipa-client-install fails outright with a Python socket.gaierror before even reaching the
# usual loopback-resolution failure. Fix: correct hostname + real routable IP, same as Phase
# 2.2/3.1, plus manage_etc_hosts: false in cloud-init config.
</nowiki>
</pre>

'''8.9 — Ubuntu 25.10+: <code><nowiki>sudo-rs</nowiki></code> silently ignores IPA sudo rules.''' Symptom: a user with a confirmed-valid IPA sudo rule (<code><nowiki>ipa sudorule-show <rule></nowiki></code>) gets denied, with '''zero''' SSSD sudo-responder log activity even though <code><nowiki>libsss-sudo</nowiki></code> is installed and the SSSD sudo socket exists and is correctly permissioned. Root cause: Ubuntu 25.10 ships <code><nowiki>sudo-rs</nowiki></code> (Rust reimplementation) as the *default* <code><nowiki>/usr/bin/sudo</nowiki></code>, and it doesn't support SSSD/IPA-driven sudo rules — falls back to local-file-only sudoers evaluation with a plausible-looking denial and no diagnostic trail. Classic GNU sudo is usually already installed alongside it, just not the active alternative (registered at <code><nowiki>/usr/bin/sudo.ws</nowiki></code> on this Ubuntu convention, no <code><nowiki>dpkg-divert</nowiki></code> involved):

<pre>
<nowiki>
update-alternatives --install /usr/bin/sudo sudo /usr/bin/sudo.ws 50
update-alternatives --set sudo /usr/bin/sudo.ws
sudo --version # confirm it no longer reports "sudo-rs"
sudo -l -U <user> # confirm it now shows the real IPA-sourced rule
</nowiki>
</pre>

'''8.10 — Missing <code><nowiki>chrony</nowiki></code> on minimal/stripped images:''' <code><nowiki>ipa-client-install</nowiki></code> tries to configure <code><nowiki>chrony</nowiki></code> for time sync but doesn't install it if missing — fails outright at that step.

<pre>
<nowiki>
apt-get install -y chrony # or dnf/yum equivalent — before retrying ipa-client-install
</nowiki>
</pre>

----

== Phase 9 — Security incident found + remediated during this work (fail2ban) ==

Not part of the migration itself, but found while investigating why <code><nowiki>admin</nowiki></code>'s Kerberos account kept hitting lockout thresholds (<code><nowiki>krbPwdMaxFailure</nowiki></code>) during cleanup work — real, active exposure.

'''Diagnosis''': <code><nowiki>krb5kdc.log</nowiki></code> showed repeated auth failures against <code><nowiki>kadmin/changepw</nowiki></code> for a long-disabled account (<code><nowiki>nsAccountLock: TRUE</nowiki></code>, expired since 2022) from <code><nowiki>relay.isgec.com</nowiki></code>, plus <code><nowiki>admin</nowiki></code> itself getting hit from two other boxes. On <code><nowiki>relay.isgec.com</nowiki></code>: <code><nowiki>ss -tnp</nowiki></code> showed multiple simultaneous SSH connections from wildly different, geographically scattered public IPs — signature of a distributed credential-stuffing botnet. That box had '''no fail2ban''' and '''<code><nowiki>PasswordAuthentication yes</nowiki></code>''' — fully exposed.

<pre>
<nowiki>
# CentOS 7, post-EOL mirrors pruned — fail2ban-server needs systemd-python, which needs a
# systemd bump unavailable on the regular (dead) mirrors:
sed -i 's|mirrorlist=|#mirrorlist=|; s|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org/7.9.2009|' \
/etc/yum.repos.d/CentOS-Base.repo
yum install -y fail2ban
systemctl is-system-running # confirm "running" after the systemd bump, no reboot needed on EL7
# (live daemon-reexec handles it)
</nowiki>
</pre>

<code><nowiki>/etc/fail2ban/jail.local</nowiki></code>:

<pre>
<nowiki>
[sshd]
enabled = true
maxretry = 5
bantime = 3600
findtime = 600
ignoreip = 127.0.0.1/8 <your-own-VPN-egress-IP>
</nowiki>
</pre>

<pre>
<nowiki>
systemctl enable --now fail2ban
fail2ban-client status sshd # confirm jail active
fail2ban-client get sshd ignoreip # confirm your own IP is whitelisted BEFORE relying on this
</nowiki>
</pre>

This picked up the already-in-progress attack immediately (214 failed attempts recorded, 5 attacking IPs banned on the spot).

'''This is very likely fleet-wide, not just this one host''' — was not swept beyond this box. Before considering this closed elsewhere: check <code><nowiki>fail2ban</nowiki></code> presence + <code><nowiki>PasswordAuthentication</nowiki></code> setting on every reachable box in the fleet, apply the same fix pattern, and consider disabling SSH password auth fleet-wide (key-only) as the actual fix rather than just rate-limiting.

----

== Current state / what's still open ==

* <code><nowiki>tornedo.tetrain.com</nowiki></code> (original box) — client-shut-down, not yet formally decommissioned (Phase 3-5 of the separate decommission checklist in the decision log). All hosts found still live-dependent on it at the last readiness check have since been individually remediated (Phase 8 patterns above) and verified with a live <code><nowiki>kinit -k</nowiki></code>.
* Fleet-wide fail2ban sweep (Phase 9) — not done beyond <code><nowiki>relay.isgec.com</nowiki></code>.
* A handful of active clients were found hitting the old server that weren't in the original 50-host inventory at all (unidentified IPs with regular live traffic) — need identifying and properly enrolling against <code><nowiki>ipa.tetrain.com</nowiki></code> before the original box can be considered fully safe to power off for good.

[[Category:Infrastructure Documentation]]

Tornedo.tetrain.com Runbook - Revision history

Admin: Publish runbook documentation