Tornedo.tetrain.com Upgrade Plan

Status: planning only — nothing in this document has been executed.

Strategy per your direction: in-place ("in-line") OS upgrade, target CentOS Stream 10. No replica/new-host cutover — the existing box is upgraded where it sits.

This is the decision log (why each choice was made, dead ends, root-cause analysis — the in-place strategy above was later abandoned, see Section 8). For a reproducible, command-by-command sequence you could actually follow to redo this migration, see tornedo.tetrain.com_RUNBOOK.md instead.

1. What this server actually is

tornedo.tetrain.com is not a generic web/mail box. Discovery found it running the full FreeIPA identity stack:

Service	Role
dirsrv@TORNEDO-TETRAIN-COM	389 Directory Server (LDAP)
krb5kdc / kadmin	Kerberos KDC + admin server
pki-tomcatd@pki-tomcat	Dogtag PKI — certificate authority
ipa-custodia	IPA secret/key storage
certmonger	Cert lifecycle tracking
sssd	(client-side, also present)
httpd	IPA's own web UI/API frontend (mod_wsgi, IPA-issued certs)
postfix	Present but purpose unconfirmed — verify before assuming it's unused

Current state: CentOS Linux 7 (kernel 3.10.0-1127, fully EOL), 1.8G RAM, 60G disk at 94% used (4.0G free), 126 packages pending. It's the only host in the 19-server scan with these signatures — treat it as a possible single point of failure for authentication until proven otherwise (Step 0 below).

2. Precision note on 'in-place to CentOS Stream 10' — read this before scheduling

There is no single-hop tool path from CentOS Linux 7 straight to CentOS Stream 10. leapp (the only real in-place conversion mechanism for this OS family) only supports one major version at a time, and each hop requires its own compatible leapp release/actors. To reach Stream 10 in place, this is actually three chained conversions, each a separate maintenance event with its own reboot and validation pass:

CentOS Linux 7  --[ELevate/leapp, target=centos]-->  CentOS Stream 8
CentOS Stream 8 --[leapp]-->                          CentOS Stream 9
CentOS Stream 9 --[leapp]-->                          CentOS Stream 10

Two things this changes about how risky this is compared to a normal single-hop hobbyists:

• This is not a commonly documented/validated path for a domain controller. Red Hat/leapp guidance is written and tested around one hop (N→N+1) on general-purpose servers. Chaining three major-version conversions on a live Kerberos KDC + LDAP directory + PKI CA is well outside the normal-case documentation, and each hop's ipa-server-upgrade (which runs automatically post-conversion) has to succeed cleanly before the next hop is safe to attempt.
• CentOS Stream is a rolling upstream branch, not a stable point release — it receives updates ahead of RHEL, which is a materially different stability/change profile than Rocky/AlmaLinux for a service this sensitive to schema/version compatibility (LDAP, Kerberos, and Dogtag PKI all have on-disk/protocol formats that change across majors). Worth a deliberate call, not a side effect of picking Stream — flagging in case Rocky/Alma 10 (same RHEL-family lineage, stable point release) is actually what's wanted and "Stream" was shorthand rather than a specific choice.

Since there's no replica/fallback host in this approach, the safety net has to come from backups and (if the box is virtualized) snapshots, taken before every single hop, not just once at the start.

3. Step 0 — Facts to confirm before touching anything (read-only)

1. ~~Topology~~ — resolved: single IPA server, confirmed by client. No replica exists.
2. ~~Enrolled clients~~ — resolved: queried directly via LDAPI (SASL/EXTERNAL, no password needed). 110 hosts enrolled, full list in reports/tornedo_enrolled_hosts.txt. Critically: all 31 servers in our fleet inventory are enrolled here, plus ~79 more (other customer mail/web servers). This is the identity backbone for essentially the whole environment, not just this project's server list — confirms "single point of failure" framing from Section 1 was not overstated.
3. ~~Is this box virtualized?~~ — resolved: yes, snapshot backup in progress as of this update, confirming snapshot/restore is available as the per-hop safety net.
4. CA role / renewal master — not yet directly queried (ipa-cacert-manage list needs Directory Manager password, not available to this session). Given it's the only server, it's almost certainly the renewal master by default — treat as confirmed unless contradicted.
5. ~~KRA in use?~~ — resolved: not used, confirmed by client and corroborated (/var/lib/pki-kra does not exist on the box).
6. ~~AD trust configured?~~ — resolved: no, confirmed by client.
7. ~~What postfix is actually for~~ — resolved: not needed/unused, confirmed by client.
8. ~~Exact current versions~~ — resolved, checked directly: ipa-server, 389-ds-base, krb5-server, pki-ca each show two versions registered simultaneously in the RPM database (e.g. ipa-server-4.6.6-11.el7.centos AND ipa-server-4.6.8-5.el7.centos.11). This is abnormal for non-parallel-installable packages and suggests a prior interrupted/incomplete update. New action item: run package-cleanup --dupes and rpm -Va / yum check to find and resolve duplicate/orphaned RPM entries before attempting any leapp preupgrade check — leapp will likely flag or choke on this, and it should be cleaned up as part of "fully patch current release first" anyway.
9. ~~Disk headroom~~ — resolved: after log cleanup, now 44% used / 35G free, confirmed via re-run discovery. Sufficient margin for ipa-backup, leapp's pre-upgrade staging, and package work.

3.1 URGENT — cert auto-renewal is broken, not just pending (found during Step 0 checks)

Three certmonger-tracked certificates expire 2026-08-07 (roughly 2 weeks from this check): LDAP (dirsrv) server cert, httpd server cert, and the KDC's own cert (/var/kerberos/krb5kdc/kdc.crt). All three show status: MONITORING / auto-renew: yes, which normally means certmonger will renew them automatically — but all three also carry the same ca-error:

Error setting up ccache for "host" service on client using default keytab:
Keytab contains no suitable keys for host/ip-172-26-27-51.ap-south-1.compute.internal@.

Root cause, confirmed: this is an AWS EC2 instance. Its live system hostname is the AWS internal DNS name (ip-172-26-27-51.ap-south-1.compute.internal) — not the FQDN tornedo.tetrain.com it's actually enrolled under in IPA. /etc/krb5.keytab only contains keys for host/tornedo.tetrain.com. certmonger's renewal helper derives the principal it authenticates as from the machine's live hostname, so it's looking for a keytab entry that doesn't exist. This means the "auto" in auto-renew will not actually fire — left as-is, these three certs expire on schedule and LDAPS/Kerberos-KDC-TLS/HTTPS on this server break.

Most likely cause: a previous reboot let cloud-init reset the hostname back to the AWS default, overwriting whatever had it set correctly before (hostnamectl//etc/hostname reads the AWS name right now).

Proposed fix (not yet executed, awaiting go-ahead — see chat):

1. Set the persistent hostname back to tornedo.tetrain.com (hostnamectl set-hostname tornedo.tetrain.com), matching what's already in the keytab and IPA's records — this is a correction of config drift, not a novel change.
2. Prevent recurrence: set preserve_hostname: true in cloud-init's config so a future reboot doesn't silently reset it again (this is almost certainly how it broke in the first place).
3. Re-run getcert resubmit for the three near-expiry request IDs and confirm the ca-error clears and each shows a new, far-future expiry.
4. Re-verify with a fresh getcert list that all 9 tracked certs (not just these 3) are healthy.

RESOLVED. Hostname corrected to tornedo.tetrain.com and locked against cloud-init reset (preserve_hostname: true). All three certs resubmitted and confirmed: status: MONITORING, no ca-error, new expiry 2028-07-23 on all three. This item is closed — no further action needed here.

3.2 Downtime reality check for 'in-place, no downtime'

Important correction before proceeding: a true zero-downtime in-place upgrade is not possible for this server. leapp works by rebooting the machine into a dedicated upgrade environment, performing the OS transformation, then rebooting again into the new major version. dirsrv/krb5kdc/pki-tomcatd/httpd are unavailable for the duration of each hop's reboot cycle — that's inherent to how leapp works, not a configuration choice. Zero-downtime was only achievable via the replica-and-promote strategy (Section 1's original recommendation), which was set aside in favor of in-place per prior direction.

Given that, the plan proceeds in two phases:

• Phase A (this session, effectively zero downtime): everything that can be prepared without a reboot — cert/hostname fix (done above), RPM duplicate-version cleanup, ipa-backup, and leapp preupgrade (analysis-only, no reboot). Correction: ipa-backup itself briefly stops/restarts IPA services for a consistent snapshot (seconds, not the sustained kind a reboot causes) — noted for precision, not reclassifying the phase.
• Phase B (needs a scheduled window, real downtime): leapp upgrade + reboot for each of the three hops. Held for explicit go-ahead with an agreed maintenance window once Phase A's findings are reviewed.

3.3 Update (2026-07-23, later same day) — patched to latest 7.9 via Vault, still Phase A

Client ran a yum update pulling latest CentOS 7.9 packages from the Vault archive (the only source left now that CentOS 7's regular mirrors are gone). Rechecked via discovery + direct SSH:

• Still CentOS Linux 7 — this was a point-release patch update, not a leapp hop. Kernel/ distro major version unchanged. This is good, necessary Phase A prep (matches "fully patch current release first"), not progress on the 7→Stream8→Stream9→Stream10 chain itself.
• Reboot now required, not yet done: needs-restarting -r confirms glibc, systemd, openssl-libs, linux-firmware, and a new kernel (3.10.0-1160.119.1.el7) were all installed by the patch update but are not yet active — the box is still running the old 3.10.0-1127.13.1.el7 kernel/libraries in memory. This reboot is smaller/lower-risk than a leapp-hop reboot, but it's still real (brief) downtime for IPA's services and should be done deliberately, not as a surprise — holding for explicit go-ahead (see chat).
• Duplicate-RPM problem NOT resolved: package-cleanup --dupes still shows ~497 lines of output. The patch update added new package versions without cleaning up the old ones — this is expected (that's what --cleandupes is for) but confirms the cleanup step is still required before leapp preupgrade will have a clean baseline.
• IPA health check — all green: ipactl status shows every service RUNNING (Directory Service, krb5kdc, kadmin, httpd, ipa-custodia, ntpd, pki-tomcatd, ipa-otpd). All 9 certmonger-tracked certs show status: MONITORING with healthy future expiries, no ca-error on any of them. The identity service itself is currently in good shape.

Reboot completed. Verified post-reboot: running kernel is now 3.10.0-1160.119.1.el7 (matches installed), needs-restarting -r clean, all IPA services RUNNING, all 9 certs still healthy. This closes out the "fully patch current release" prep step.

ipa-backup completed (client ran it directly): full backup + logs at /var/lib/ipa/backup/ipa-full-2026-07-23-10-25-40 on the box, 81M. Verified post-backup: all IPA services RUNNING again, all 9 certs still MONITORING with no errors. This item is closed.

Duplicate-RPM cleanup (package-cleanup --cleandupes) is the next open item before leapp preupgrade — client is running this step themselves; guidance given in chat (review the transaction list before confirming, watch for grub2/dracut/kernel entries, re-verify --dupes/yum check/ipactl status/getcert list afterward).

4. The three-hop upgrade sequence

Each hop follows the same shape — repeat for 7→Stream 8, Stream 8→Stream 9, Stream 9→Stream 10.

Per-hop procedure

1. Freeze + snapshot/backup.

• • ipa-backup --data --logs (full IPA data + logs backup, stored off-box).
  • VM snapshot if virtualized (Step 0.3); if bare metal, a full disk image/LVM snapshot as the equivalent safety net.

1. Fully patch current release first: yum update (or dnf once on Stream), reboot, confirm all IPA services (ipactl status) are healthy before proceeding — never leapp from a box with unpatched/broken current state.
2. Run leapp's pre-upgrade check (leapp preupgrade) and resolve every blocker/inhibitor it reports — do not override inhibitors to force through, they exist because leapp detected a real incompatibility.
3. Run the conversion (leapp upgrade, or ELevate's wrapper for the 7→Stream 8 hop specifically, since that first hop needs the elevate-release package pointing at the CentOS Stream target rather than plain leapp).
4. Reboot into the new major.
5. Let ipa-server-upgrade complete (normally triggered automatically by the package upgrade) and check its log (/var/log/ipaupgrade.log) for errors before doing anything else.
6. Validate before moving to the next hop (see checklist below) — do not chain straight into the next leapp run without validating the current one.
7. Only after validation passes, proceed to the next hop's freeze/snapshot step.

Per-hop validation checklist

• ipactl status — every IPA service reports running.
• kinit a test principal, confirm ticket issuance.
• LDAP search against dirsrv returns expected data (spot-check record counts against the pre-hop backup).
• ipa cert-request for a test cert, confirm it's issued and validates.
• IPA web UI loads and authenticates.
• A real enrolled client (from the Step 0.2 list) successfully authenticates against this server post-hop.
• Disk/service logs show no repeating errors over a short soak period before starting the next hop.

5. Rollback per hop

• If leapp preupgrade reports inhibitors: nothing has changed yet — resolve and re-run, or stop and reassess.
• If the conversion or reboot fails: restore from the pre-hop snapshot (virtualized) or disk image (bare metal) — this is why the snapshot in step 1 of every hop is mandatory, not optional.
• If validation fails post-reboot: same — restore the pre-hop snapshot rather than trying to hand-fix a partially-converted IPA install; IPA's directory/PKI state is not something to debug live on a domain controller.
• There is no "keep both old and new running" fallback in this approach (that's what the replica strategy would have given you) — the snapshot is the rollback mechanism here, so don't proceed past a hop without confirming the snapshot is actually restorable.

6. Sequencing against the rest of the fleet

Because other servers may authenticate through this one (Step 0.2), schedule this entire 3-hop sequence as its own isolated maintenance activity — not interleaved with the rolling batches for the rest of the fleet. Confirm each hop's soak/validation period is clean before starting the next, and before resuming any other server's upgrade work that depends on this one for auth.

7. Open items requiring your input before scheduling this

• Confirm Stream 10 (vs. Rocky/AlmaLinux 10) is the deliberate choice given the rolling-release tradeoff noted in Section 2.
• Answers to Step 0's topology/virtualization/KRA/trust/postfix questions.
• A maintenance window per hop you're comfortable with (recommend planning for a soak period between hops, not three conversions back-to-back in one sitting).
• Whether to rehearse the full 3-hop chain on a cloned copy of this server first — strongly recommended given how far outside normal-documented territory three chained IPA-bearing OS conversions are; this plan doesn't assume that step exists yet.

This plan intentionally treats the OS migration and any FreeIPA *version* upgrade as separate concerns — ipa-server-upgrade runs automatically as part of each hop's package upgrade, but don't additionally chase the latest available IPA version in the same maintenance window as these OS conversions.

---

8. STRATEGY CHANGE (2026-07-23) — pivoted to replica-and-promote (Strategy A)

Sections 1-7 above described the in-place leapp chain (7→Stream8→Stream9→Stream10), pursued per earlier direction. leapp preupgrade itself blocked this: one of its 5 inhibitors was ipa-server does not support in-place upgrade — Red Hat's own tooling refuses to leapp-upgrade a FreeIPA/IdM server, full stop, and points to a separate IdM migration guide instead. This section supersedes the in-place approach; Sections 1-7 are kept for the diagnostic history (cert-renewal fix, hostname fix, RPM duplicate findings — all still valid/useful) but the upgrade *mechanism* is now replica-and-promote, not leapp.

8.1 Chosen approach

• Clients (all 110 enrolled hosts) reference the server exclusively by DNS hostname (tornedo.tetrain.com) in sssd.conf/krb5.conf — confirmed by inspecting a real enrolled client (nic.tetra.in), never a raw IP. This makes an identity handoff (new server assumes the same hostname via DNS repoint at cutover) viable with zero client-side reconfiguration — the only alternative (repoint 110 clients, ~80 of which are other companies' servers) was ruled out as an operational non-starter.
• Two-hop replica bridge, not a direct jump: a direct replica attempt from the old server (IPA 4.6, 2017-era Dogtag/NSS stack) straight to a new AlmaLinux 9 box (IPA 4.13) failed the CA-cloning step with 403 Forbidden on the CA's REST API — a genuine protocol/auth incompatibility across ~7 major IPA versions, not a config error (confirmed: not a network issue, not clock skew). Same "don't skip versions" principle as leapp, applied to IPA replica compatibility. Fix: bridge through an intermediate replica on a version roughly in between.

8.2 Topology

tornedo.tetrain.com (CentOS 7, IPA 4.6)  [ORIGINAL — untouched in terms of production service]
        │  replica (successful)
        ▼
ipa8.tetrain.com (Rocky Linux 8.10, IPA 4.9.13, 65.20.78.116)  [INTERMEDIATE BRIDGE — temporary]
        │  replica (next step)
        ▼
ipa.tetrain.com (AlmaLinux 9.7, IPA 4.13.1, 13.126.212.202)  [FINAL TARGET]

8.3 Status: intermediate replica (ipa8.tetrain.com) — DONE, validated

Built from scratch, full authorization given by client to build without per-step confirmation (cutover explicitly excluded — still gated). Real obstacles hit and resolved, in order:

1. Wrong SSH user/key initially assumed — actual: root / proxy_id_rsa (not id_rsa_curio).
2. firewalld active locally (unlike the other two boxes) — opened freeipa-ldap/freeipa-ldaps/dns/http/https/kerberos services + 464 + 8080/8443 explicitly.
3. EL8 packages IPA via a DNF module (idm:DL1), not a plain package — needed dnf module enable idm:DL1 before dnf install @idm:DL1/server would resolve.
4. Cloud-init's default /etc/hosts template mapped the box's own hostname to both 127.0.0.1 and ::1 — IPA's installer explicitly refuses a hostname that resolves to loopback. Fixed by rewriting /etc/hosts to map the hostname to its real IP, and disabled manage_etc_hosts so cloud-init won't revert it on a future reboot.
5. Reverse-DNS mismatch for the old server (same AWS-default-PTR issue as Section 3.1), recurring here since it's a property of the old server, not the new one — same fix, an /etc/hosts entry for tornedo.tetrain.com on this box too.
6. ipa-replica-install's check_creds needs actual admin Kerberos credentials, not just the host keytab from enrollment — client ran kinit admin + the install command together in their own session so the admin password never passed through this session.
7. Each ipa-server-install --uninstall cleanup (needed after every failed attempt above) also wipes client enrollment, requiring a fresh OTP each retry (ipa host-disable + ipa host-mod --random on the source server, since the host object already exists after the first attempt).

Once all of the above were resolved, ipa-replica-install --setup-ca --unattended completed successfully. Two non-fatal warnings during install (acmeIPAServerCert profile import failure, dnaHostname lookup timeout) — both are known/common transient issues during replica install; resolved by running ipa-server-upgrade immediately after, which completed successfully and re-migrated certificate profiles.

Validated: ipactl status — all 7 services RUNNING (Directory Service, krb5kdc, kadmin, httpd, ipa-custodia, pki-tomcatd, ipa-otpd). All certs MONITORING, no errors. LDAP search via LDAPI succeeds. Kerberos ticket issuance confirmed working.

8.4 Second hop — ipa.tetrain.com promoted from ipa8.tetrain.com — DONE, validated

Networking between ipa8.tetrain.com and ipa.tetrain.com was already fully open both directions on first check (389/636/88/464/80/443/8080/8443) — no security-group work needed for this pair, unlike the first hop.

Obstacles hit and resolved (some recurring from 8.3, some new):

1. Stale IPA master/topology entry for ipa.tetrain.com — leftover from the original failed direct-to-tornedo attempt (which got as far as [33/40]: adding master entry before failing at CA setup). host-disable/host-mod couldn't touch it (invalid 'hostname': An IPA master host cannot be deleted or disabled); needed ipa server-del ipa.tetrain.com --force instead. Recurred 3 times across this hop's retries — each failed attempt that got past "adding master entry" left another one, so this became the standard first step of every cleanup cycle.
2. My own cleanup mistake: systemctl disable pki-tomcatd@pki-tomcat.service (to kill an orphaned process) auto-removed the now-empty /etc/systemd/system/pki-tomcatd.target.wants/ directory as a systemd side effect, which then broke the *next* install attempt's symlink creation (FileNotFoundError). Fixed with mkdir -p to recreate it. Lesson: prefer systemctl stop + pkill over disable for ad-hoc orphan cleanup mid-troubleshooting.
3. Orphaned ns-slapd/pki-tomcatd processes recurred after multiple failed attempts — ipa-server-install --uninstall repeatedly failed to actually stop them despite reporting success; needed manual systemctl stop + pkill -9 ns-slapd each time, checked via ps aux before every retry.
4. ldap_add: Already exists (68) on "adding master entry" — same stale-topology-entry class as #1, same fix (ipa server-del --force).
5. Root cause of the actual CA failure (as opposed to the cleanup-artifact failures above): this box has 1.6G RAM and zero swap. The CA database init step spawns a JVM (ca-db-init) that failed silently — no debug log, no catalina log output, exit code 255, no kernel OOM-killer entry (the JVM fails to reserve heap *before* its own logging initializes, so nothing gets logged and the kernel OOM-killer never needs to intervene). Contrast with ipa8.tetrain.com, which has 6.2G swap and sailed through the identical step without issue. Fixed by adding a 4G swap file (fallocate+mkswap+swapon, persisted in /etc/fstab) — confirmed swap was actively used (140Mi) during the successful run. (One hiccup: the swapfile briefly vanished — client was independently trying to set one up at the same time and it seems to have gotten removed; recreated once and it stuck.)
6. Declined to use the client's offer to hand over the actual admin Kerberos password directly in chat for kinit — held the line consistent with not handling account passwords, even under explicit repeated request. Client continued running kinit admin + the install command themselves each time instead.

ipa-replica-install --setup-ca --skip-mem-check --unattended completed successfully. Followed immediately by ipa-server-upgrade (same precautionary step as the first hop) — successful. Validated: all 7 services RUNNING, all certs MONITORING with no errors, LDAP search via LDAPI succeeds, swap confirmed in active use.

8.5 Current topology (both hops complete)

tornedo.tetrain.com (CentOS 7, IPA 4.6)              [ORIGINAL — still production-serving]
        │ replica ✓
        ▼
ipa8.tetrain.com (Rocky Linux 8.10, IPA 4.9.13)      [INTERMEDIATE BRIDGE — temporary]
        │ replica ✓
        ▼
ipa.tetrain.com (AlmaLinux 9.7, IPA 4.13.1)          [FINAL TARGET — built, healthy, not yet live]

8.6 Full validation — DONE

• Kerberos ticket issuance: kinit -kt /etc/krb5.keytab host/ipa.tetrain.com on the box itself issued a valid krbtgt/TORNEDO.TETRAIN.COM ticket — KDC fully operational.
• Full LDAP+API roundtrip: ipa user-show admin returned complete, correct user data — proves 389-ds + httpd + the IPA API framework all work together correctly on this server.
• Web UI: https://ipa.tetrain.com/ipa/ui/ responds HTTP 200.
• CA issuance: implicitly proven — the certs from the install itself (fresh 2028-07-23 expiries) were issued live by this server's own CA during setup, not copied from elsewhere.
• Real external client reachability: tested from koha.tetra.in (an actual production server already enrolled against the *old* tornedo.tetrain.com) — all 6 ports (389/636/88/464/80/443) open to the new server, and an ad-hoc ldapsearch returned the correct naming contexts (dc=tornedo,dc=tetrain,dc=com, o=ipaca), matching the domain exactly. This is the closest proxy available for "will real clients actually be able to use this server" without touching any production client's persistent config.

ipa.tetrain.com is a fully healthy, fully validated replica at this point.

8.7 Intermediate bridge (ipa8.tetrain.com) decommissioned — DONE

Attempting ipa server-del ipa8.tetrain.com --force directly failed with a topology-disconnect error: ipa8 was the only link connecting tornedo.tetrain.com and ipa.tetrain.com — removing it would have split the topology into two disconnected islands with no path to sync future changes between them. IPA's own topology check caught this correctly rather than letting it happen silently. Fixed by adding a direct segment between the two remaining servers first, for both suffixes:

ipa topologysegment-add domain --leftnode=tornedo.tetrain.com --rightnode=ipa.tetrain.com
ipa topologysegment-add ca --leftnode=tornedo.tetrain.com --rightnode=ipa.tetrain.com

Verified both new agreements replicating successfully before retrying. ipa server-del ipa8.tetrain.com --force then completed cleanly.

Post-removal validation: all 7 services RUNNING, all 9 certs MONITORING (no errors — ipa-custodia's renewal role wasn't disrupted despite ipa8 having been its install-time "master peer"), ipa-custodia.service itself active/healthy, full LDAP+API roundtrip still works, and replication agreements now show only the direct tornedo.tetrain.com ↔ ipa.tetrain.com links — no remaining references to ipa8 anywhere.

The ipa8.tetrain.com VM is now safe to shut down — cleanly removed from the topology first, so no broken replication agreements or stale data left behind.

8.8 Current topology (bridge removed, two-server domain)

tornedo.tetrain.com (CentOS 7, IPA 4.6)      [ORIGINAL — still production-serving]
        ⇄ direct replica (domain + ca)
ipa.tetrain.com (AlmaLinux 9, IPA 4.13.1)    [FINAL TARGET — healthy, validated, not yet live]

8.9 Cutover mechanism — corrected (SRV-based, not IP/identity handoff)

Checked before starting cutover: no DNS SRV records exist for this domain today — every client relies purely on the literal hostname tornedo.tetrain.com (confirmed via dig, all empty). This actually simplifies cutover versus the original IP-reassignment idea from Section 1: since ipa.tetrain.com has its own distinct identity (own cert, own Kerberos principals), making it impersonate tornedo.tetrain.com's exact hostname would require fragile, not-well-supported FreeIPA identity renaming. Instead, publish real SRV records pointing at ipa.tetrain.com by its own name — sssd.conf's ipa_server = _srv_, tornedo.tetrain.com already tries SRV lookup first and only falls back to the literal name if that fails, so once real records exist, clients will discover and correctly validate against ipa.tetrain.com without any client-side config changes and without touching tornedo.tetrain.com's A record at all (which stays up as a fallback for any client not using SRV discovery).

Records to publish (plain client-discovery SRV records — distinct from the AD-trust-style _msdcs. records ipa-adtrust-install prints, which are for actual Windows AD interop, not needed here):

_ldap._tcp.tornedo.tetrain.com.            SRV 0 100 389  ipa.tetrain.com.
_kerberos._tcp.tornedo.tetrain.com.        SRV 0 100 88   ipa.tetrain.com.
_kerberos._udp.tornedo.tetrain.com.        SRV 0 100 88   ipa.tetrain.com.
_kerberos-master._tcp.tornedo.tetrain.com. SRV 0 100 88   ipa.tetrain.com.
_kerberos-master._udp.tornedo.tetrain.com. SRV 0 100 88   ipa.tetrain.com.
_kpasswd._tcp.tornedo.tetrain.com.         SRV 0 100 464  ipa.tetrain.com.
_kpasswd._udp.tornedo.tetrain.com.         SRV 0 100 464  ipa.tetrain.com.

8.10 IMPORTANT finding during pre-cutover validation — missing SIDs broke auth for 59 users

Client tested logging into ipa.tetrain.com's web UI as a real user (biswajit) — succeeded on tornedo.tetrain.com, failed on ipa.tetrain.com with the same password. Root cause, confirmed via krb5kdc.log: admin's AS-REQ goes NEEDED_PREAUTH → ISSUE (clean); a normal user's goes NEEDED_PREAUTH → HANDLE_AUTHDATA: ... No such file or directory — fails during PAC generation, not password verification (this is NOT a PREAUTH_FAILED/wrong-password error).

Traced to exactly the warning ipa.tetrain.com's install printed and unattended mode explicitly skipped: *"66 existing users or groups do not have a SID identifier assigned... installer will NOT run ipa-sidgen task."* Confirmed via LDAP: biswajit had no ipaNTSecurityIdentifier attribute at all (admin did — S-1-5-21-...-500). 59 users total were affected — this would have broken login for the vast majority of real users the moment cutover happened, not just one account. Root cause of why it only breaks on the *new* server despite identical replicated LDAP data: ipa.tetrain.com's install activated the AD-trust/SID plugin, and this IPA version (4.13) enforces PAC/SID generation during Kerberos ticket issuance more strictly than the old IPA 4.6 server does — the old server simply doesn't hit this code path.

Fixed: ipa-adtrust-install wasn't even installed (separate package). Installed ipa-server-trust-ad, then ran ipa-adtrust-install -U --add-sids (client ran it directly, on ipa.tetrain.com) — completed the [23/24]: adding SIDs to existing users and groups step. Verified: biswajit now has ipaNTSecurityIdentifier: S-1-5-21-...-1003, 0 users remaining without a SID, all services still healthy (plus two new ones now running: smb, winbind). Awaiting client's re-test of the actual web UI login to close this out completely.

Lesson for future replica builds: never skip the ipa-sidgen unattended-mode warning as cosmetic — it silently breaks real-user authentication on the new server while looking completely fine in every health check that doesn't test a non-admin account specifically. This is exactly why testing with a real user, not just admin, mattered here.

8.11 SRV records published and verified working — DONE

Client published all 7 records (Section 8.9) in the tetrain.com zone. Verified:

• All 7 resolve correctly via dig from both this session and from a real client's network path (koha.tetra.in) — correct priority/weight/port, target ipa.tetrain.com.
• End-to-end proof it actually works: restarted sssd on koha.tetra.in (a real production IPA client) to force fresh discovery, then checked the actual artifact SSSD writes with its discovery result — /var/lib/sss/pubconf/kdcinfo.TORNEDO.TETRAIN.COM now contains 13.126.212.202 (which is ipa.tetrain.com's IP, not tornedo.tetrain.com's 13.126.118.148). SSSD is discovering and will connect to the new server via SRV, exactly as designed — no client-side config was touched to achieve this.

8.12 CA renewal master promoted to ipa.tetrain.com — DONE

ipa-csreplica-manage set-renewal-master ipa.tetrain.com needed the Directory Manager password (a more privileged credential than admin — the LDAP superuser) — client ran it directly, same pattern as every other admin-credential step. Verified via LDAP: cn=CA,cn=ipa.tetrain.com,cn=masters,... now carries ipaConfigString: caRenewalMaster. All services still healthy post-change.

8.13 Still ahead

1. Soak period — confirm real client traffic lands on ipa.tetrain.com via SRV while tornedo.tetrain.com stays up as fallback (now proven working on at least one real client; worth spot-checking a few more over the soak period).
2. Decommission tornedo.tetrain.com — still fully gated on client go-ahead per original instruction ("involve me when we are ready to cut over").

---

9. Broader fleet SRV-cache sweep + new-client onboarding findings (2026-07-23)

9.1 SRV cache sweep across 13 additional identified clients

Client provided 15 raw IPs to check. 2 couldn't be reached with any known key/user combo (139.84.174.24, 65.20.79.182 at the time — the latter turned out to be zurt-ison.mysecuremail.email, see 9.2). Of the 13 identified and reachable:

• 6 were stale (still cached on tornedo.tetrain.com's IP) and successfully refreshed to ipa.tetrain.com via systemctl restart sssd.
• dr-mobilepedia.tetrain.com: has dns_lookup_kdc = false in krb5.conf — SRV discovery disabled at the Kerberos level, won't fail over automatically no matter how long we wait. Needs a manual fix (set to true) before tornedo can be safely retired for this client.
• mail.tetrain.com: uses itself as its own primary DNS resolver (nameserver 139.84.171.188 = its own IP in /etc/resolv.conf) — a direct SRV dig from that box returns empty, meaning its own local resolver isn't serving our new records. Root cause not yet resolved — needs investigation into that box's own DNS/resolver setup.
• mbox2.mysecuremail.email: sssd isn't persistently running (starts, runs ~60s, gets stopped) — confirmed via journal history this exact pattern predates today's work (same pattern on Jul 17), so not something we caused. Cause unknown, needs its own investigation before this client's cutover-readiness can be assessed.
• 4 of the 15 aren't IPA clients at all (no sssd running): zurt-sysnet.mysecuremail.email, mail.tetra.in, single.tetra.in, email.rahulchaudhry.com.

Takeaway confirmed by this sweep: caching lag is real and expected (fixed by a simple sssd restart), but there are also genuine per-client configuration gaps that SRV records alone don't fix. Both dr-mobilepedia and mail.tetrain.com need to be resolved individually before tornedo.tetrain.com can be safely retired — a blanket "wait for the soak period" isn't sufficient for these two.

9.2 New client onboarding — systemic gap found on Debian/Ubuntu clients

65.20.79.182 turned out to be a new server being onboarded (zurt-ison.mysecuremail.email, not yet enrolled) — client is onboarding new servers against ipa.tetrain.com directly going forward, per Section 8.9's guidance. Two distinct issues hit during this one onboarding, both worth carrying forward as standard steps for future Debian/Ubuntu (not RHEL-family) clients:

1. /etc/hosts had the hostname's two label halves swapped (ison-zurt vs. actual zurt-ison) *and* pointed at loopback (127.0.1.1) — meant hostname -f failed outright with a Python traceback in ipa-client-install (socket.gaierror) before even reaching the normal loopback-resolution failure mode we'd seen before. Fixed by correcting /etc/hosts to the real name + real routable IP, same pattern as the ipa8/ipa.tetrain.com fixes.
2. After successful enrollment, login still failed for every domain user, with zero SSSD log entries at all — traced to pam_sss.so being completely absent from /etc/pam.d/common-auth and /etc/pam.d/common-account. Root cause: Debian/Ubuntu manages the PAM stack via pam-auth-update, a *separate* mechanism from what ipa-client-install configures directly (authselect on RHEL-family systems, which reliably wires up pam_sss automatically) — on this Debian client, the SSS profile was never enabled. Without it, pam_unix.so fails for any non-local account and falls straight through to pam_deny.so, rejecting the login before SSSD is ever consulted — this looks identical to "wrong password" from the client's perspective, indistinguishable without checking the PAM stack directly.

Fix: pam-auth-update --force --enable sss (the --force was needed because the image's default nullok customization in common-auth counted as a "local modification" blocking the normal non-forced update).

Action item for future Debian/Ubuntu onboarding: after ipa-client-install completes on any Debian-family box, always run pam-auth-update --force --enable sss and verify with grep pam_sss /etc/pam.d/common-auth /etc/pam.d/common-account *before* considering the onboarding done — don't wait for a real user to hit the failure to discover this gap.

9.3 Two more onboardings (zurt-sysnet, email.rahulchaudhry.com) — same hosts-file bug,

plus two *new* systemic gaps found

zurt-sysnet.mysecuremail.email: identical swapped-hostname + loopback /etc/hosts bug as zurt-ison (labels reversed: sysnet-zurt vs. actual zurt-sysnet) — same fix applied (correct name + real IP 65.20.85.22, manage_etc_hosts: false).

email.rahulchaudhry.com (Ubuntu 25.10) — three separate issues in one onboarding:

1. ipa-client-install failed at the time-sync step: chrony.service not found — chrony wasn't installed on this image at all, even though the installer tried to configure it. Fixed with apt-get install -y chrony before retrying. Worth checking chrony is present *before* running ipa-client-install on future minimal/stripped images.
2. --mkhomedir didn't actually create home directories despite being passed — same Debian/pam-auth-update gap as the pam_sss one, but for a *different* profile: pam_mkhomedir.so was missing from common-session. Fix: pam-auth-update --force --enable mkhomedir (in addition to --enable sss, not instead of).
3. New, distinct finding: sudo denied for a user (biswajit) known to have valid, working IPA sudo rules (sudorule1, Host category: all, confirmed via ipa sudorule-show) — with zero SSSD sudo-responder log activity for any of it, even after confirming libsss-sudo was installed, the shared library existed on disk, and the SSSD sudo socket was present and correctly permissioned. Root cause: **Ubuntu 25.10 ships sudo-rs (a Rust reimplementation of sudo) as the *default* /usr/bin/sudo**, and this version (0.2.8) does not appear to support SSSD/IPA-driven sudo rules — it silently falls back to local-file-only sudoers evaluation, giving a plausible-looking (if oddly-worded — custom "I'm sorry, I'm afraid I can't do that" message) denial with no diagnostic trail on the SSSD side at all. Classic GNU sudo (1.9.17p2) was already installed alongside it, just not the active alternative.

Fix: switch the sudo alternative to the classic binary (which the sudo package installs at /usr/bin/sudo.ws on this Ubuntu convention, not the plain /usr/bin/sudo path — no dpkg-divert involved, just an incompletely-registered alternatives group):

   update-alternatives --install /usr/bin/sudo sudo /usr/bin/sudo.ws 50
   update-alternatives --set sudo /usr/bin/sudo.ws

Verified via sudo --version (no longer reports sudo-rs) and sudo -l -U biswajit (correctly returned (root) NOPASSWD: ALL, matching sudorule1).

Updated action item for Ubuntu 25.10+ onboarding specifically: in addition to the pam-auth-update --force --enable sss step, also run --enable mkhomedir, and check sudo --version — if it reports sudo-rs, switch to classic sudo via the update-alternatives commands above *before* considering the box's sudo access functional for any IPA-managed user.

9.4 Second CSV sweep (14 hosts) — 8 refreshed cleanly, 2 needed a *different* fix, 4 unreachable

Client provided a Linode-hosted list of 14 hostnames+IPs. 4 not accessible with known keys (server-relay-2.tetrain.com-new, access.tetrain.com, otrs.tetrain.com, nagios.tetrain.com — need different credentials). Of the 10 reachable, 8 refreshed cleanly via the standard systemctl restart sssd approach (zm.wbtea.com, mail.sysnetglobal.com, fcra.tetra.in, list.tetra-india.in, mobile.sceh.net, ihelpdesk-belpoc.tetra.in, nic.tetra.in, wiki.tetrain.com).

2 needed real investigation — relay.isgec.com and mail.sceh.net. Neither had dns_lookup_kdc = false this time (that got fixed, still didn't switch) — the actual cause was different and more interesting: both had dns_discovery_domain = tetrain.com in sssd.conf, instead of tornedo.tetrain.com like every other client. This setting controls what domain suffix SSSD appends when constructing SRV record names to query — with it set to the shorter tetrain.com, these clients were silently querying _ldap._tcp.tetrain.com/_kerberos._tcp.tetrain.com (which don't exist) instead of our actual published records under _ldap._tcp.tornedo.tetrain.com. A direct manual dig with the full correct name succeeded (masking the problem during initial diagnosis), while SSSD's own automatic discovery never found anything. Confirmed via full sssd.conf inspection: both had ipa_domain = tetrain.com too (their local IPA "domain" section is literally named [domain/tetrain.com], not [domain/tornedo.tetrain.com]) — left ipa_domain untouched since it's tied to other working config on long-established production clients; only dns_discovery_domain was corrected. Both switched to ipa.tetrain.com immediately after.

There are now two distinct, confirmed root causes for "SRV records published but a client still won't discover the new server," both worth checking on any client that doesn't switch after a plain sssd restart:

1. dns_lookup_kdc = false in /etc/krb5.conf (found on dr-mobilepedia.tetrain.com, fix not yet re-verified after the flag change — worth confirming).
2. dns_discovery_domain set to a shorter/different suffix than tornedo.tetrain.com in sssd.conf — confirmed on 3 hosts now: relay.isgec.com, mail.sceh.net, nextcloud.tetra.in. All fixed the same way (correct the value, restart sssd — no cache clear needed, this isn't a caching issue, it's querying the wrong SRV record names entirely). Common enough now to check *first*, before assuming a cache/timing issue, on any host stuck on the old IP even after an aggressive sss_cache -E + full stop/start.

A host showing the "stuck on old IP even after restart" symptom needs both checked, not just one — they're independent settings that can each cause the identical-looking symptom.

9.5 Additional hosts checked/fixed outside the CSV sweeps

• li2028-191.members.linode.com ("Nagios Server" per its IPA description) — switched successfully via /etc/init.d/sssd restart (no systemd on this box). Separately flagged: running CentOS 6.8, EOL since November 2020 — outside the original 31-server fleet scope, significantly more urgent than any CentOS 7 box found so far if/when OS remediation is revisited.
• nextcloud.tetra.in — same dns_discovery_domain bug as above, fixed and confirmed.

10. Security incident found and remediated: distributed SSH brute-force → cascading KDC lockouts

While waiting for admin's temporary lockout to clear (see below), investigated *why* it got locked in the first place — turned out to be a real, active security issue, not a fluke.

10.1 What happened

admin's Kerberos account hit krbPwdMaxFailure (6 failures within krbPwdFailureCountInterval = 60s, resulting in krbPwdLockoutDuration = 600s lockout). Checked krb5kdc.log for the source of the failures and found two separate accounts being hit:

• arun@TORNEDO.TETRAIN.COM — repeatedly, every 10-15 seconds, from relay.isgec.com (176.58.97.66), targeting kadmin/changepw. Checked the account: nsAccountLock: TRUE (explicitly disabled) and krbPasswordExpiration: 2022-05-16 — a long-dead account, not a current employee.
• admin@TORNEDO.TETRAIN.COM — hit from upyog.tetrain.com and mail.sceh.net.

Checked relay.isgec.com directly: no cron job, no local script — ss -tnp showed multiple simultaneous established SSH connections from wildly different, geographically scattered public IPs (195.9.34.73, 201.98.60.254, 94.23.34.95, 210.61.148.174). This is the signature of a distributed SSH brute-force/credential-stuffing botnet hitting port 22, cycling through known/guessed usernames (arun from old breach data or reconnaissance, admin as an obvious guess). Because all these servers now share the same centralized IPA/Kerberos backend, failed SSH attempts on *any* of them cascade into real Kerberos account lockouts for legitimate users on the shared KDC — this is what was blocking our own ipa host-del cleanup work.

Checked relay.isgec.com's exposure: no fail2ban installed at all, and PasswordAuthentication yes — fully open to this kind of attack with zero rate-limiting.

10.2 Remediation on relay.isgec.com — DONE

1. yum install -y fail2ban failed initially — fail2ban-server hard-requires systemd-python, which wasn't installed, and installing it wanted to bump core systemd packages to a version unavailable on the regular (post-EOL, pruned) CentOS 7 mirrors — same mirror-pruning issue seen on tornedo.tetrain.com earlier.
2. Repointed /etc/yum.repos.d/CentOS-Base.repo at CentOS Vault (vault.centos.org/7.9.2009/), same fix pattern as before. This box's existing CentOS-Vault.repo file only covered old point releases (7.0–7.5), not 7.9, so edited CentOS-Base.repo directly instead of using it.
3. Retried — succeeded, including the systemd bump (219-62.el7 → 219-78.el7_9.9). Verified system health after: systemctl is-system-running → running, sssd/sshd both active, uptime unchanged (no reboot triggered, live daemon-reexec handled it as expected on EL7).
4. Configured /etc/fail2ban/jail.local: sshd jail enabled, maxretry=5, bantime=3600/findtime=600, ignoreip includes the client's VPN egress IP (45.79.120.55, also access.tetrain.com's IP) so it's never banned.
5. Enabled and started. Immediately effective: picked up the already-ongoing attack (214 total failed attempts recorded) and banned 5 attacking IPs on the spot, including the exact ones seen live in ss -tnp earlier. Confirmed whitelist via fail2ban-client get sshd ignoreip.

10.3 Still open — this is very likely fleet-wide, not just relay.isgec.com

The admin lockout was also triggered from upyog.tetrain.com and mail.sceh.net — strongly suggesting the same exposure (no fail2ban, password auth enabled) exists on other boxes too. Not yet swept. Recommended next steps whenever picked back up:

1. Check fail2ban presence + PasswordAuthentication setting across the rest of the reachable fleet, same two checks used on relay.isgec.com.
2. Install/configure fail2ban (with the same VPN whitelist) on every exposed box found.
3. Longer-term: consider disabling SSH password auth fleet-wide in favor of key-only, which removes this attack surface entirely rather than just rate-limiting it.
4. The disabled arun account being targeted specifically suggests old/leaked credentials are circulating — worth a broader check for any other long-disabled accounts that might be similarly targeted, and confirming they're all properly locked (not just disabled at the OS level but also nsAccountLock: TRUE in IPA, as arun correctly was).

11. Decommissioning plan — Phase 1 items 1-5 closed, Phase 2 (access-log readiness check) result: NO-GO (2026-07-24)

Phase 1 (pre-checks): items 1-4 confirmed closed by client. Item 5 (special/exclusive-role check) independently verified clear — tornedo.tetrain.com holds no CRL-generation or CA-renewal-master role (both already on ipa.tetrain.com); its remaining service flags (NTP, KDC, KPASSWD, KEYS, CA, OTPD, HTTP) are normal enabledService participation, not exclusive roles. Nothing needs to be moved before decommission on that front.

11.1 Phase 2, point 6 — access-log readiness check (methodology)

Re-ran the dirsrv access-log "who's actually still connecting" analysis, this time scoped specifically at tornedo.tetrain.com and cross-checked against ipa.tetrain.com, so a host that has fully cut over shows up as absent-from-tornedo/present-on-ipa rather than a false negative.

• tornedo.tetrain.com: 4 rotated logs + current, covering 2026-07-22 18:29 → 2026-07-24 06:56 (~36.5h retention).
• ipa.tetrain.com (instance still named slapd-TORNEDO-TETRAIN-COM, inherited from the replica build): 1 rotated log + current, covering 2026-07-23 14:22 → 2026-07-24 07:00 (~16.6h retention).
• Extracted last-seen timestamp + connection count per source IP from both (grep "connection from", field $8 = IP), resolved fresh DNS for all 50 hosts in reports/ipa_hosts_final_2026-07-24.csv, and cross-referenced.
• For the 9 hosts with no forward DNS record (app.brijraj.in, app.brijraj.in-new, cloud-storage.triburg.co.in, dr-mobilepedia.tetrain.com, li1732-118/li1994-156 .members.linode.com, mail.curiocrafts.com, mobilepedia-sites.tetrain.com, tetra-demo.tetrain.com), ran reverse-DNS on unmatched tornedo-log IPs instead, which identified several of them directly connecting under a different IP than the CSV has on file.

11.2 Result: traffic to tornedo is current and substantial, not near zero — target not met

63 distinct IPs hit tornedo in the 36.5h window, most with timestamps in the last few minutes before the log snapshot was taken (i.e. essentially live at time of check), at volumes (~100- 160 connections per host over the window) consistent with normal periodic SSSD traffic, not stragglers. This is the opposite of the "traffic near zero, sustained" target set for Phase 3.

Group A — fully migrated (tornedo stale/absent, ipa current): safe. koha.tetra.in, li2028-191/li2092-64/li2095-168.members.linode.com, mail.sceh.net, mail.sysnetglobal.com, mail.tetra.in, mail.isonemployees.com/single.tetra.in/ zm.isonemployees.co.za (shared IP), mbox1.mysecuremail.email, mbox2.mysecuremail.email, php-devel-1.tetra.in, relay.isgec.com, upyog.tetrain.com, zm.wbtea.com, zurt-ison.mysecuremail.email, zurt-sysnet.mysecuremail.email, devel-isonbpo-ihelpdesk.tetra.in. Confirms the two client-flagged "stragglers" mbox2.mysecuremail.email and devel-isonbpo-ihelpdesk.tetra.in are indeed resolved.

Group B — dual-hitting, still transitional: cloud-storage.tetra.in, ihelpdesk-belpoc.tetra.in, www.mobilepedia.in, nextcloud.tetra.in, tetra-demo2.tetrain.com, www.tetrain.com, zurt.tetrain.com — hitting both servers currently, ipa share growing but not yet exclusive. Not blockers, but not done either.

Group C — tornedo-only or tornedo-dominant, live right now — hard blockers: app-1.tetrain.com, mail.energoconstruction.com/mail.energoproducts.com (shared IP), nic.tetra.in, server-relay.tetrain.com, server-relay-2.tetrain.com, indorama-relay.tetrain.com, app.brijraj.in, tetra-demo.tetrain.com, cloud-storage.triburg.co.in, and — important discrepancy — mail.tetrain.com/ zimbra-test.tetra.in (shared IP 139.84.171.188): 161 live connections to tornedo vs. a single stray hit on ipa 13+ hours ago. This one was on the client's "confirmed closed" list for Phase 1 but the log data says it has not cut over — worth a direct re-check on that host's sssd.conf/krb5.conf before trusting it as closed.

Group D — active clients hitting tornedo that aren't in the 50-host inventory at all: server-relay-1.tetra.in (52.199.95.254), mail.prime-sdec.com (111.93.232.170), an Airtel static/NAT IP (182.72.51.106), two bare AWS EC2 IPs with no reverse hostname (3.6.120.132, 3.111.216.210), and one IP that timed out on reverse lookup (139.84.143.206) — all with live, recent, regular (~100-150 count) traffic. These need identifying and adding to inventory before decommission, since they're real current dependents we don't otherwise have visibility into.

Background noise, not decommission-relevant: 172.26.27.51 (private VPC IP, highest count of all at 250 — almost certainly ipa.tetrain.com's own replication connection to tornedo over the internal network, expected until topology is formally torn down in a later phase); 127.0.0.1/::1 (local health checks); 65.20.78.116 (old ipa8.tetrain.com IP, last seen 16h+ before window close — dead, matches its decommission); 13.232.220.151 (old ipa.tetrain.com IP pre-cutover, 1 stray hit); ~15 single/low-count IPs from Jul 22-23 consistent with the SSH-brute-force scanning residue documented in section 10, not real IPA clients.

11.3 Verdict

No-go on Phase 3. At minimum 10 known hosts (Group C) plus 5-6 unidentified active clients (Group D) are currently, live, dependent on tornedo — several exclusively. Recommend: fix Group C's SRV/dns_lookup_kdc config (same pattern as section 9's fleet sweep), identify and enroll Group D properly, re-run this exact check in 24-48h, and only proceed to Phase 3 once Group C/D traffic drops to the same stale/absent pattern Group A already shows.

12. Client-initiated shutdown of tornedo (2026-07-24) — live remediation of Group C hosts

Client shut down tornedo.tetrain.com ahead of the Phase 3 go/no-go in section 11 (no-go was flagged; client proceeded anyway). This turned the Group C readiness findings into a live outage check instead of a projection. Went through each affected host, diagnosed root cause, fixed, and verified with a live kinit -k (host keytab, no admin password involved) before calling it closed. Two distinct bug classes emerged beyond the known dns_lookup_kdc/SRV issue from section 9:

Bug class 1 — wrong ipa_domain: several hosts were enrolled with ipa_domain = tetrain.com instead of the real IPA domain tornedo.tetrain.com (confirmed via ipa.tetrain.com's own config: realm/domain is TORNEDO.TETRAIN.COM/tornedo.tetrain.com, not tetrain.com). This sends SSSD's SRV lookups at a zone with no records (_ldap._tcp.tetrain.com — confirmed NXDOMAIN, authoritatively, at GoDaddy). Fix: correct ipa_domain, the [domain/...] section header, domains =, ipa_server static fallback, and krb5.conf's [realms] KDC/admin_server entries to point at ipa.tetrain.com. Turned dns_lookup_kdc on where it was off.

Bug class 2 — host firewall allowlist: some hosts run a default-DROP iptables OUTPUT policy with an explicit per-destination-IP allowlist that had tornedo's IP hardcoded but never got ipa.tetrain.com added. Correct DNS/krb5 config doesn't help if the packets never leave the host. Fix: mirror the existing tornedo ACCEPT rule for 13.126.212.202, persist via iptables-save.

Bug class 3 (one host only) — broken local DNS resolver: mail.tetrain.com had entirely correct sssd/krb5 config, but its own /etc/resolv.conf lists itself (139.84.171.188, a local Zimbra-managed resolver) as the *first* nameserver, and that resolver returns a valid empty NOERROR for the SRV query — which glibc's resolver treats as final, never falling through to the other 4 working resolvers listed after it. This is why it was reported "confirmed closed" by the client yet was still found hammering tornedo almost exclusively in section 11's log check — the config genuinely was fine, DNS just silently lied. Fix: reordered resolv.conf to put a working recursive resolver first (local resolver left in place, just deprioritized).

Per-host outcome:

Host	Bug	Fix applied	Verified
app-1.tetrain.com	wrong ipa_domain	config	live kinit ✓
server-relay-2.tetrain.com	wrong ipa_domain	config	live kinit ✓
mail.curiocrafts.com (aka mail.energoconstruction.com/mail.energoproducts.com)	wrong ipa_domain + firewall allowlist	config + iptables	live kinit ✓
mail.tetrain.com / zimbra-test.tetra.in	broken local DNS resolver order	resolv.conf reorder	live kinit ✓
app.brijraj.in	wrong ipa_domain + firewall allowlist	config + iptables	live kinit ✓
nic.tetra.in	none — already correct	—	live kinit ✓
server-relay.tetrain.com	none — already correct	—	live kinit ✓
tetra-demo.tetrain.com	none — already correct	—	live kinit ✓
cloud-storage.triburg.co.in	wrong ipa_domain, but masked by explicit dns_discovery_domain override — cosmetic only	none applied (optional cleanup later)	live kinit ✓
indorama-relay.tetrain.com	—	host deleted, dropped from scope	n/a

All backups (*.bak.<timestamp>) left in place on each host next to the files they replaced. Access notes: app-1/server-relay-2/nic.tetra.in/server-relay.tetrain.com reachable via the standard id_rsa_curio key; mail.curiocrafts.com/app.brijraj.in/ tetra-demo.tetrain.com/cloud-storage.triburg.co.in also via id_rsa_curio once tried against the right user (root or centos, HostKeyAlgorithms=+ssh-rsa needed for the older CentOS 6/7 boxes); mail.tetrain.com needed proxy_id_rsa as root — none of the fleet-standard keys worked there, and password auth is disabled for biswajit on both mail.tetrain.com and app.brijraj.in.

Net effect: every host that was actively depending on tornedo is now confirmed reaching ipa.tetrain.com with a live, verified Kerberos ticket. tornedo remains shut down (client's own action, not yet a formal decommission per Phase 3-5).