Editing Tornedo.tetrain.com Upgrade Plan

Status: '''planning only — nothing in this document has been executed.'''

Strategy per your direction: '''in-place ("in-line") OS upgrade, target CentOS Stream 10.''' No replica/new-host cutover — the existing box is upgraded where it sits.

'''This is the decision log''' (why each choice was made, dead ends, root-cause analysis — the in-place strategy above was later abandoned, see Section 8). For a reproducible, command-by-command sequence you could actually follow to redo this migration, see [[Tornedo.tetrain.com Runbook|<code><nowiki>tornedo.tetrain.com_RUNBOOK.md</nowiki></code>]] instead.

== 1. What this server actually is ==

<code><nowiki>tornedo.tetrain.com</nowiki></code> is not a generic web/mail box. Discovery found it running the full FreeIPA identity stack:

{| class="wikitable"
|-
! Service !! Role
|-
| <code><nowiki>dirsrv@TORNEDO-TETRAIN-COM</nowiki></code> || 389 Directory Server (LDAP)
|-
| <code><nowiki>krb5kdc</nowiki></code> / <code><nowiki>kadmin</nowiki></code> || Kerberos KDC + admin server
|-
| <code><nowiki>pki-tomcatd@pki-tomcat</nowiki></code> || Dogtag PKI — certificate authority
|-
| <code><nowiki>ipa-custodia</nowiki></code> || IPA secret/key storage
|-
| <code><nowiki>certmonger</nowiki></code> || Cert lifecycle tracking
|-
| <code><nowiki>sssd</nowiki></code> || (client-side, also present)
|-
| <code><nowiki>httpd</nowiki></code> || IPA's own web UI/API frontend (mod_wsgi, IPA-issued certs)
|-
| <code><nowiki>postfix</nowiki></code> || Present but purpose unconfirmed — verify before assuming it's unused
|}

Current state: '''CentOS Linux 7''' (kernel 3.10.0-1127, fully EOL), 1.8G RAM, 60G disk at 94% used (4.0G free), 126 packages pending. It's the only host in the 19-server scan with these signatures — treat it as a possible single point of failure for authentication until proven otherwise (Step 0 below).

== 2. Precision note on 'in-place to CentOS Stream 10' — read this before scheduling ==

There is '''no single-hop tool path from CentOS Linux 7 straight to CentOS Stream 10.''' <code><nowiki>leapp</nowiki></code> (the only real in-place conversion mechanism for this OS family) only supports one major version at a time, and each hop requires its own compatible leapp release/actors. To reach Stream 10 in place, this is actually '''three chained conversions''', each a separate maintenance event with its own reboot and validation pass:

<pre>
<nowiki>
CentOS Linux 7  --[ELevate/leapp, target=centos]-->  CentOS Stream 8
CentOS Stream 8 --[leapp]-->                          CentOS Stream 9
CentOS Stream 9 --[leapp]-->                          CentOS Stream 10
</nowiki>
</pre>

Two things this changes about how risky this is compared to a normal single-hop hobbyists:

* '''This is not a commonly documented/validated path for a domain controller.''' Red Hat/leapp guidance is written and tested around one hop (N→N+1) on general-purpose servers. Chaining three major-version conversions on a live Kerberos KDC + LDAP directory + PKI CA is well outside the normal-case documentation, and each hop's <code><nowiki>ipa-server-upgrade</nowiki></code> (which runs automatically post-conversion) has to succeed cleanly before the next hop is safe to attempt.
* '''CentOS Stream is a rolling upstream branch, not a stable point release''' — it receives updates ahead of RHEL, which is a materially different stability/change profile than Rocky/AlmaLinux for a service this sensitive to schema/version compatibility (LDAP, Kerberos, and Dogtag PKI all have on-disk/protocol formats that change across majors). Worth a deliberate call, not a side effect of picking Stream — flagging in case Rocky/Alma 10 (same RHEL-family lineage, stable point release) is actually what's wanted and "Stream" was shorthand rather than a specific choice.

Since there's no replica/fallback host in this approach, the safety net has to come from backups and (if the box is virtualized) snapshots, taken '''before every single hop''', not just once at the start.

== 3. Step 0 — Facts to confirm before touching anything (read-only) ==

# ~~'''Topology'''~~ — '''resolved: single IPA server''', confirmed by client. No replica exists.
# ~~'''Enrolled clients'''~~ — '''resolved''': queried directly via LDAPI (SASL/EXTERNAL, no password needed). '''110 hosts enrolled''', full list in <code><nowiki>reports/tornedo_enrolled_hosts.txt</nowiki></code>. Critically: '''all 31 servers in our fleet inventory are enrolled here''', plus ~79 more (other customer mail/web servers). This is the identity backbone for essentially the whole environment, not just this project's server list — confirms "single point of failure" framing from Section 1 was not overstated.
# ~~'''Is this box virtualized?'''~~ — '''resolved: yes''', snapshot backup in progress as of this update, confirming snapshot/restore is available as the per-hop safety net.
# '''CA role / renewal master''' — not yet directly queried (<code><nowiki>ipa-cacert-manage list</nowiki></code> needs Directory Manager password, not available to this session). Given it's the only server, it's almost certainly the renewal master by default — treat as confirmed unless contradicted.
# ~~'''KRA in use?'''~~ — '''resolved: not used''', confirmed by client and corroborated (<code><nowiki>/var/lib/pki-kra</nowiki></code> does not exist on the box).
# ~~'''AD trust configured?'''~~ — '''resolved: no''', confirmed by client.
# ~~'''What postfix is actually for'''~~ — '''resolved: not needed/unused''', confirmed by client.
# ~~'''Exact current versions'''~~ — '''resolved''', checked directly: <code><nowiki>ipa-server</nowiki></code>, <code><nowiki>389-ds-base</nowiki></code>, <code><nowiki>krb5-server</nowiki></code>, <code><nowiki>pki-ca</nowiki></code> each show '''two versions registered simultaneously''' in the RPM database (e.g. <code><nowiki>ipa-server-4.6.6-11.el7.centos</nowiki></code> AND <code><nowiki>ipa-server-4.6.8-5.el7.centos.11</nowiki></code>). This is abnormal for non-parallel-installable packages and suggests a prior interrupted/incomplete update. '''New action item''': run <code><nowiki>package-cleanup --dupes</nowiki></code> and <code><nowiki>rpm -Va</nowiki></code> / <code><nowiki>yum check</nowiki></code> to find and resolve duplicate/orphaned RPM entries before attempting any leapp preupgrade check — leapp will likely flag or choke on this, and it should be cleaned up as part of "fully patch current release first" anyway.
# ~~'''Disk headroom'''~~ — '''resolved''': after log cleanup, now 44% used / 35G free, confirmed via re-run discovery. Sufficient margin for <code><nowiki>ipa-backup</nowiki></code>, leapp's pre-upgrade staging, and package work.

== 3.1 URGENT — cert auto-renewal is broken, not just pending (found during Step 0 checks) ==

Three certmonger-tracked certificates expire '''2026-08-07''' (roughly 2 weeks from this check): LDAP (<code><nowiki>dirsrv</nowiki></code>) server cert, <code><nowiki>httpd</nowiki></code> server cert, and the '''KDC's own cert''' (<code><nowiki>/var/kerberos/krb5kdc/kdc.crt</nowiki></code>). All three show <code><nowiki>status: MONITORING</nowiki></code> / <code><nowiki>auto-renew: yes</nowiki></code>, which normally means certmonger will renew them automatically — but all three also carry the same <code><nowiki>ca-error</nowiki></code>:

<pre>
<nowiki>
Error setting up ccache for "host" service on client using default keytab:
Keytab contains no suitable keys for host/ip-172-26-27-51.ap-south-1.compute.internal@.
</nowiki>
</pre>

'''Root cause, confirmed''': this is an AWS EC2 instance. Its live system hostname is the AWS internal DNS name (<code><nowiki>ip-172-26-27-51.ap-south-1.compute.internal</nowiki></code>) — not the FQDN <code><nowiki>tornedo.tetrain.com</nowiki></code> it's actually enrolled under in IPA. <code><nowiki>/etc/krb5.keytab</nowiki></code> only contains keys for <code><nowiki>host/tornedo.tetrain.com</nowiki></code>. certmonger's renewal helper derives the principal it authenticates as from the machine's live hostname, so it's looking for a keytab entry that doesn't exist. '''This means the "auto" in auto-renew will not actually fire''' — left as-is, these three certs expire on schedule and LDAPS/Kerberos-KDC-TLS/HTTPS on this server break.

Most likely cause: a previous reboot let cloud-init reset the hostname back to the AWS default, overwriting whatever had it set correctly before (<code><nowiki>hostnamectl</nowiki></code>/<code><nowiki>/etc/hostname</nowiki></code> reads the AWS name right now).

'''Proposed fix''' (not yet executed, awaiting go-ahead — see chat):

# Set the persistent hostname back to <code><nowiki>tornedo.tetrain.com</nowiki></code> (<code><nowiki>hostnamectl set-hostname tornedo.tetrain.com</nowiki></code>), matching what's already in the keytab and IPA's records — this is a correction of config drift, not a novel change.
# Prevent recurrence: set <code><nowiki>preserve_hostname: true</nowiki></code> in cloud-init's config so a future reboot doesn't silently reset it again (this is almost certainly how it broke in the first place).
# Re-run <code><nowiki>getcert resubmit</nowiki></code> for the three near-expiry request IDs and confirm the <code><nowiki>ca-error</nowiki></code> clears and each shows a new, far-future expiry.
# Re-verify with a fresh <code><nowiki>getcert list</nowiki></code> that all 9 tracked certs (not just these 3) are healthy.

'''RESOLVED.''' Hostname corrected to <code><nowiki>tornedo.tetrain.com</nowiki></code> and locked against cloud-init reset (<code><nowiki>preserve_hostname: true</nowiki></code>). All three certs resubmitted and confirmed: <code><nowiki>status: MONITORING</nowiki></code>, no <code><nowiki>ca-error</nowiki></code>, new expiry '''2028-07-23''' on all three. This item is closed — no further action needed here.

== 3.2 Downtime reality check for 'in-place, no downtime' ==

Important correction before proceeding: '''a true zero-downtime in-place upgrade is not possible for this server.''' <code><nowiki>leapp</nowiki></code> works by rebooting the machine into a dedicated upgrade environment, performing the OS transformation, then rebooting again into the new major version. <code><nowiki>dirsrv</nowiki></code>/<code><nowiki>krb5kdc</nowiki></code>/<code><nowiki>pki-tomcatd</nowiki></code>/<code><nowiki>httpd</nowiki></code> are unavailable for the duration of each hop's reboot cycle — that's inherent to how leapp works, not a configuration choice. Zero-downtime was only achievable via the replica-and-promote strategy (Section 1's original recommendation), which was set aside in favor of in-place per prior direction.

Given that, the plan proceeds in two phases:

* '''Phase A (this session, effectively zero downtime)''': everything that can be prepared without a reboot — cert/hostname fix (done above), RPM duplicate-version cleanup, <code><nowiki>ipa-backup</nowiki></code>, and <code><nowiki>leapp preupgrade</nowiki></code> (analysis-only, no reboot). Correction: <code><nowiki>ipa-backup</nowiki></code> itself briefly stops/restarts IPA services for a consistent snapshot (seconds, not the sustained kind a reboot causes) — noted for precision, not reclassifying the phase.
* '''Phase B (needs a scheduled window, real downtime)''': <code><nowiki>leapp upgrade</nowiki></code> + reboot for each of the three hops. Held for explicit go-ahead with an agreed maintenance window once Phase A's findings are reviewed.

== 3.3 Update (2026-07-23, later same day) — patched to latest 7.9 via Vault, still Phase A ==

Client ran a <code><nowiki>yum update</nowiki></code> pulling latest CentOS 7.9 packages from the Vault archive (the only source left now that CentOS 7's regular mirrors are gone). Rechecked via discovery + direct SSH:

* '''Still CentOS Linux 7''' — this was a point-release patch update, not a leapp hop. Kernel/ distro major version unchanged. This is good, necessary Phase A prep (matches "fully patch current release first"), not progress on the 7→Stream8→Stream9→Stream10 chain itself.
* '''Reboot now required, not yet done''': <code><nowiki>needs-restarting -r</nowiki></code> confirms glibc, systemd, openssl-libs, linux-firmware, and a new kernel (3.10.0-1160.119.1.el7) were all installed by the patch update but are not yet active — the box is still running the old 3.10.0-1127.13.1.el7 kernel/libraries in memory. This reboot is smaller/lower-risk than a leapp-hop reboot, but it's still real (brief) downtime for IPA's services and should be done deliberately, not as a surprise — holding for explicit go-ahead (see chat).
* '''Duplicate-RPM problem NOT resolved''': <code><nowiki>package-cleanup --dupes</nowiki></code> still shows ~497 lines of output. The patch update added new package versions without cleaning up the old ones — this is expected (that's what <code><nowiki>--cleandupes</nowiki></code> is for) but confirms the cleanup step is still required before <code><nowiki>leapp preupgrade</nowiki></code> will have a clean baseline.
* '''IPA health check — all green''': <code><nowiki>ipactl status</nowiki></code> shows every service RUNNING (Directory Service, krb5kdc, kadmin, httpd, ipa-custodia, ntpd, pki-tomcatd, ipa-otpd). All 9 certmonger-tracked certs show <code><nowiki>status: MONITORING</nowiki></code> with healthy future expiries, no <code><nowiki>ca-error</nowiki></code> on any of them. The identity service itself is currently in good shape.

'''Reboot completed.''' Verified post-reboot: running kernel is now <code><nowiki>3.10.0-1160.119.1.el7</nowiki></code> (matches installed), <code><nowiki>needs-restarting -r</nowiki></code> clean, all IPA services RUNNING, all 9 certs still healthy. This closes out the "fully patch current release" prep step.

'''<code><nowiki>ipa-backup</nowiki></code> completed''' (client ran it directly): full backup + logs at <code><nowiki>/var/lib/ipa/backup/ipa-full-2026-07-23-10-25-40</nowiki></code> on the box, 81M. Verified post-backup: all IPA services RUNNING again, all 9 certs still <code><nowiki>MONITORING</nowiki></code> with no errors. This item is closed.

Duplicate-RPM cleanup (<code><nowiki>package-cleanup --cleandupes</nowiki></code>) is the next open item before <code><nowiki>leapp preupgrade</nowiki></code> — client is running this step themselves; guidance given in chat (review the transaction list before confirming, watch for grub2/dracut/kernel entries, re-verify <code><nowiki>--dupes</nowiki></code>/<code><nowiki>yum check</nowiki></code>/<code><nowiki>ipactl status</nowiki></code>/<code><nowiki>getcert list</nowiki></code> afterward).

== 4. The three-hop upgrade sequence ==

Each hop follows the same shape — repeat for 7→Stream 8, Stream 8→Stream 9, Stream 9→Stream 10.

=== Per-hop procedure ===

# '''Freeze + snapshot/backup.'''
** <code><nowiki>ipa-backup --data --logs</nowiki></code> (full IPA data + logs backup, stored off-box).
** VM snapshot if virtualized (Step 0.3); if bare metal, a full disk image/LVM snapshot as the equivalent safety net.
# '''Fully patch current release first''': <code><nowiki>yum update</nowiki></code> (or <code><nowiki>dnf</nowiki></code> once on Stream), reboot, confirm all IPA services (<code><nowiki>ipactl status</nowiki></code>) are healthy before proceeding — never leapp from a box with unpatched/broken current state.
# '''Run leapp's pre-upgrade check''' (<code><nowiki>leapp preupgrade</nowiki></code>) and resolve every blocker/inhibitor it reports — do not override inhibitors to force through, they exist because leapp detected a real incompatibility.
# '''Run the conversion''' (<code><nowiki>leapp upgrade</nowiki></code>, or ELevate's wrapper for the 7→Stream 8 hop specifically, since that first hop needs the <code><nowiki>elevate-release</nowiki></code> package pointing at the CentOS Stream target rather than plain leapp).
# '''Reboot into the new major.'''
# '''Let <code><nowiki>ipa-server-upgrade</nowiki></code> complete''' (normally triggered automatically by the package upgrade) and check its log (<code><nowiki>/var/log/ipaupgrade.log</nowiki></code>) for errors before doing anything else.
# '''Validate before moving to the next hop''' (see checklist below) — do not chain straight into the next leapp run without validating the current one.
# '''Only after validation passes''', proceed to the next hop's freeze/snapshot step.

=== Per-hop validation checklist ===

* <code><nowiki>ipactl status</nowiki></code> — every IPA service reports running.
* <code><nowiki>kinit</nowiki></code> a test principal, confirm ticket issuance.
* LDAP search against <code><nowiki>dirsrv</nowiki></code> returns expected data (spot-check record counts against the pre-hop backup).
* <code><nowiki>ipa cert-request</nowiki></code> for a test cert, confirm it's issued and validates.
* IPA web UI loads and authenticates.
* A real enrolled client (from the Step 0.2 list) successfully authenticates against this server post-hop.
* Disk/service logs show no repeating errors over a short soak period before starting the next hop.

== 5. Rollback per hop ==

* '''If <code><nowiki>leapp preupgrade</nowiki></code> reports inhibitors''': nothing has changed yet — resolve and re-run, or stop and reassess.
* '''If the conversion or reboot fails''': restore from the pre-hop snapshot (virtualized) or disk image (bare metal) — this is why the snapshot in step 1 of every hop is mandatory, not optional.
* '''If validation fails post-reboot''': same — restore the pre-hop snapshot rather than trying to hand-fix a partially-converted IPA install; IPA's directory/PKI state is not something to debug live on a domain controller.
* There is no "keep both old and new running" fallback in this approach (that's what the replica strategy would have given you) — the snapshot '''is''' the rollback mechanism here, so don't proceed past a hop without confirming the snapshot is actually restorable.

== 6. Sequencing against the rest of the fleet ==

Because other servers may authenticate through this one (Step 0.2), schedule this entire 3-hop sequence as its own isolated maintenance activity — not interleaved with the rolling batches for the rest of the fleet. Confirm each hop's soak/validation period is clean before starting the next, and before resuming any other server's upgrade work that depends on this one for auth.

== 7. Open items requiring your input before scheduling this ==

* Confirm Stream 10 (vs. Rocky/AlmaLinux 10) is the deliberate choice given the rolling-release tradeoff noted in Section 2.
* Answers to Step 0's topology/virtualization/KRA/trust/postfix questions.
* A maintenance window per hop you're comfortable with (recommend planning for a soak period between hops, not three conversions back-to-back in one sitting).
* Whether to rehearse the full 3-hop chain on a cloned copy of this server first — strongly recommended given how far outside normal-documented territory three chained IPA-bearing OS conversions are; this plan doesn't assume that step exists yet.

This plan intentionally treats the OS migration and any FreeIPA *version* upgrade as separate concerns — <code><nowiki>ipa-server-upgrade</nowiki></code> runs automatically as part of each hop's package upgrade, but don't additionally chase the latest available IPA version in the same maintenance window as these OS conversions.

----

== 8. STRATEGY CHANGE (2026-07-23) — pivoted to replica-and-promote (Strategy A) ==

Sections 1-7 above described the in-place <code><nowiki>leapp</nowiki></code> chain (7→Stream8→Stream9→Stream10), pursued per earlier direction. '''<code><nowiki>leapp preupgrade</nowiki></code> itself blocked this''': one of its 5 inhibitors was <code><nowiki>ipa-server does not support in-place upgrade</nowiki></code> — Red Hat's own tooling refuses to leapp-upgrade a FreeIPA/IdM server, full stop, and points to a separate IdM migration guide instead. This section supersedes the in-place approach; Sections 1-7 are kept for the diagnostic history (cert-renewal fix, hostname fix, RPM duplicate findings — all still valid/useful) but the upgrade *mechanism* is now replica-and-promote, not leapp.

=== 8.1 Chosen approach ===

* Clients (all 110 enrolled hosts) reference the server exclusively by DNS hostname (<code><nowiki>tornedo.tetrain.com</nowiki></code>) in <code><nowiki>sssd.conf</nowiki></code>/<code><nowiki>krb5.conf</nowiki></code> — confirmed by inspecting a real enrolled client (<code><nowiki>nic.tetra.in</nowiki></code>), never a raw IP. This makes an identity handoff (new server assumes the same hostname via DNS repoint at cutover) viable with '''zero client-side reconfiguration''' — the only alternative (repoint 110 clients, ~80 of which are other companies' servers) was ruled out as an operational non-starter.
* '''Two-hop replica bridge''', not a direct jump: a direct replica attempt from the old server (IPA 4.6, 2017-era Dogtag/NSS stack) straight to a new AlmaLinux 9 box (IPA 4.13) failed the CA-cloning step with <code><nowiki>403 Forbidden</nowiki></code> on the CA's REST API — a genuine protocol/auth incompatibility across ~7 major IPA versions, not a config error (confirmed: not a network issue, not clock skew). Same "don't skip versions" principle as leapp, applied to IPA replica compatibility. Fix: bridge through an intermediate replica on a version roughly in between.

=== 8.2 Topology ===

<pre>
<nowiki>
tornedo.tetrain.com (CentOS 7, IPA 4.6)  [ORIGINAL — untouched in terms of production service]
        │  replica (successful)
        ▼
ipa8.tetrain.com (Rocky Linux 8.10, IPA 4.9.13, 65.20.78.116)  [INTERMEDIATE BRIDGE — temporary]
        │  replica (next step)
        ▼
ipa.tetrain.com (AlmaLinux 9.7, IPA 4.13.1, 13.126.212.202)  [FINAL TARGET]
</nowiki>
</pre>

=== 8.3 Status: intermediate replica (<code><nowiki>ipa8.tetrain.com</nowiki></code>) — DONE, validated ===

Built from scratch, full authorization given by client to build without per-step confirmation (cutover explicitly excluded — still gated). Real obstacles hit and resolved, in order:

# Wrong SSH user/key initially assumed — actual: <code><nowiki>root</nowiki></code> / <code><nowiki>proxy_id_rsa</nowiki></code> (not <code><nowiki>id_rsa_curio</nowiki></code>).
# <code><nowiki>firewalld</nowiki></code> active locally (unlike the other two boxes) — opened freeipa-ldap/freeipa-ldaps/dns/http/https/kerberos services + 464 + 8080/8443 explicitly.
# EL8 packages IPA via a DNF '''module''' (<code><nowiki>idm:DL1</nowiki></code>), not a plain package — needed <code><nowiki>dnf module enable idm:DL1</nowiki></code> before <code><nowiki>dnf install @idm:DL1/server</nowiki></code> would resolve.
# Cloud-init's default <code><nowiki>/etc/hosts</nowiki></code> template mapped the box's own hostname to '''both''' <code><nowiki>127.0.0.1</nowiki></code> and <code><nowiki>::1</nowiki></code> — IPA's installer explicitly refuses a hostname that resolves to loopback. Fixed by rewriting <code><nowiki>/etc/hosts</nowiki></code> to map the hostname to its real IP, and disabled <code><nowiki>manage_etc_hosts</nowiki></code> so cloud-init won't revert it on a future reboot.
# Reverse-DNS mismatch for the '''old''' server (same AWS-default-PTR issue as Section 3.1), recurring here since it's a property of the old server, not the new one — same fix, an <code><nowiki>/etc/hosts</nowiki></code> entry for <code><nowiki>tornedo.tetrain.com</nowiki></code> on this box too.
# <code><nowiki>ipa-replica-install</nowiki></code>'s <code><nowiki>check_creds</nowiki></code> needs actual admin Kerberos credentials, not just the host keytab from enrollment — client ran <code><nowiki>kinit admin</nowiki></code> + the install command together in their own session so the admin password never passed through this session.
# Each <code><nowiki>ipa-server-install --uninstall</nowiki></code> cleanup (needed after every failed attempt above) also wipes client enrollment, requiring a fresh OTP each retry (<code><nowiki>ipa host-disable</nowiki></code> + <code><nowiki>ipa host-mod --random</nowiki></code> on the source server, since the host object already exists after the first attempt).

Once all of the above were resolved, <code><nowiki>ipa-replica-install --setup-ca --unattended</nowiki></code> completed successfully. Two non-fatal warnings during install (<code><nowiki>acmeIPAServerCert</nowiki></code> profile import failure, <code><nowiki>dnaHostname</nowiki></code> lookup timeout) — both are known/common transient issues during replica install; resolved by running <code><nowiki>ipa-server-upgrade</nowiki></code> immediately after, which completed successfully and re-migrated certificate profiles.

'''Validated''': <code><nowiki>ipactl status</nowiki></code> — all 7 services RUNNING (Directory Service, krb5kdc, kadmin, httpd, ipa-custodia, pki-tomcatd, ipa-otpd). All certs <code><nowiki>MONITORING</nowiki></code>, no errors. LDAP search via LDAPI succeeds. Kerberos ticket issuance confirmed working.

=== 8.4 Second hop — <code><nowiki>ipa.tetrain.com</nowiki></code> promoted from <code><nowiki>ipa8.tetrain.com</nowiki></code> — DONE, validated ===

Networking between <code><nowiki>ipa8.tetrain.com</nowiki></code> and <code><nowiki>ipa.tetrain.com</nowiki></code> was already fully open both directions on first check (389/636/88/464/80/443/8080/8443) — no security-group work needed for this pair, unlike the first hop.

Obstacles hit and resolved (some recurring from 8.3, some new):

# Stale IPA '''master'''/topology entry for <code><nowiki>ipa.tetrain.com</nowiki></code> — leftover from the original failed direct-to-<code><nowiki>tornedo</nowiki></code> attempt (which got as far as <code><nowiki>[33/40]: adding master entry</nowiki></code> before failing at CA setup). <code><nowiki>host-disable</nowiki></code>/<code><nowiki>host-mod</nowiki></code> couldn't touch it (<code><nowiki>invalid 'hostname': An IPA master host cannot be deleted or disabled</nowiki></code>); needed <code><nowiki>ipa server-del ipa.tetrain.com --force</nowiki></code> instead. '''Recurred 3 times''' across this hop's retries — each failed attempt that got past "adding master entry" left another one, so this became the standard first step of every cleanup cycle.
# My own cleanup mistake: <code><nowiki>systemctl disable pki-tomcatd@pki-tomcat.service</nowiki></code> (to kill an orphaned process) auto-removed the now-empty <code><nowiki>/etc/systemd/system/pki-tomcatd.target.wants/</nowiki></code> directory as a systemd side effect, which then broke the *next* install attempt's symlink creation (<code><nowiki>FileNotFoundError</nowiki></code>). Fixed with <code><nowiki>mkdir -p</nowiki></code> to recreate it. Lesson: prefer <code><nowiki>systemctl stop</nowiki></code> + <code><nowiki>pkill</nowiki></code> over <code><nowiki>disable</nowiki></code> for ad-hoc orphan cleanup mid-troubleshooting.
# Orphaned <code><nowiki>ns-slapd</nowiki></code>/<code><nowiki>pki-tomcatd</nowiki></code> processes recurred after multiple failed attempts — <code><nowiki>ipa-server-install --uninstall</nowiki></code> repeatedly failed to actually stop them despite reporting success; needed manual <code><nowiki>systemctl stop</nowiki></code> + <code><nowiki>pkill -9 ns-slapd</nowiki></code> each time, checked via <code><nowiki>ps aux</nowiki></code> before every retry.
# <code><nowiki>ldap_add: Already exists (68)</nowiki></code> on "adding master entry" — same stale-topology-entry class as #1, same fix (<code><nowiki>ipa server-del --force</nowiki></code>).
# '''Root cause of the actual CA failure''' (as opposed to the cleanup-artifact failures above): this box has 1.6G RAM and '''zero swap'''. The CA database init step spawns a JVM (<code><nowiki>ca-db-init</nowiki></code>) that failed silently — no debug log, no catalina log output, exit code 255, no kernel OOM-killer entry (the JVM fails to reserve heap *before* its own logging initializes, so nothing gets logged and the kernel OOM-killer never needs to intervene). Contrast with <code><nowiki>ipa8.tetrain.com</nowiki></code>, which has 6.2G swap and sailed through the identical step without issue. '''Fixed by adding a 4G swap file''' (<code><nowiki>fallocate</nowiki></code>+<code><nowiki>mkswap</nowiki></code>+<code><nowiki>swapon</nowiki></code>, persisted in <code><nowiki>/etc/fstab</nowiki></code>) — confirmed swap was actively used (140Mi) during the successful run. (One hiccup: the swapfile briefly vanished — client was independently trying to set one up at the same time and it seems to have gotten removed; recreated once and it stuck.)
# Declined to use the client's offer to hand over the actual <code><nowiki>admin</nowiki></code> Kerberos password directly in chat for <code><nowiki>kinit</nowiki></code> — held the line consistent with not handling account passwords, even under explicit repeated request. Client continued running <code><nowiki>kinit admin</nowiki></code> + the install command themselves each time instead.

'''<code><nowiki>ipa-replica-install --setup-ca --skip-mem-check --unattended</nowiki></code> completed successfully.''' Followed immediately by <code><nowiki>ipa-server-upgrade</nowiki></code> (same precautionary step as the first hop) — successful. '''Validated''': all 7 services RUNNING, all certs <code><nowiki>MONITORING</nowiki></code> with no errors, LDAP search via LDAPI succeeds, swap confirmed in active use.

=== 8.5 Current topology (both hops complete) ===

<pre>
<nowiki>
tornedo.tetrain.com (CentOS 7, IPA 4.6)              [ORIGINAL — still production-serving]
        │ replica ✓
        ▼
ipa8.tetrain.com (Rocky Linux 8.10, IPA 4.9.13)      [INTERMEDIATE BRIDGE — temporary]
        │ replica ✓
        ▼
ipa.tetrain.com (AlmaLinux 9.7, IPA 4.13.1)          [FINAL TARGET — built, healthy, not yet live]
</nowiki>
</pre>

=== 8.6 Full validation — DONE ===

* '''Kerberos ticket issuance''': <code><nowiki>kinit -kt /etc/krb5.keytab host/ipa.tetrain.com</nowiki></code> on the box itself issued a valid <code><nowiki>krbtgt/TORNEDO.TETRAIN.COM</nowiki></code> ticket — KDC fully operational.
* '''Full LDAP+API roundtrip''': <code><nowiki>ipa user-show admin</nowiki></code> returned complete, correct user data — proves 389-ds + httpd + the IPA API framework all work together correctly on this server.
* '''Web UI''': <code><nowiki>https://ipa.tetrain.com/ipa/ui/</nowiki></code> responds <code><nowiki>HTTP 200</nowiki></code>.
* '''CA issuance''': implicitly proven — the certs from the install itself (fresh <code><nowiki>2028-07-23</nowiki></code> expiries) were issued live by this server's own CA during setup, not copied from elsewhere.
* '''Real external client reachability''': tested from <code><nowiki>koha.tetra.in</nowiki></code> (an actual production server already enrolled against the *old* <code><nowiki>tornedo.tetrain.com</nowiki></code>) — all 6 ports (389/636/88/464/80/443) open to the new server, and an ad-hoc <code><nowiki>ldapsearch</nowiki></code> returned the correct naming contexts (<code><nowiki>dc=tornedo,dc=tetrain,dc=com</nowiki></code>, <code><nowiki>o=ipaca</nowiki></code>), matching the domain exactly. This is the closest proxy available for "will real clients actually be able to use this server" without touching any production client's persistent config.

<code><nowiki>ipa.tetrain.com</nowiki></code> is a fully healthy, fully validated replica at this point.

=== 8.7 Intermediate bridge (<code><nowiki>ipa8.tetrain.com</nowiki></code>) decommissioned — DONE ===

Attempting <code><nowiki>ipa server-del ipa8.tetrain.com --force</nowiki></code> directly failed with a topology-disconnect error: '''<code><nowiki>ipa8</nowiki></code> was the only link connecting <code><nowiki>tornedo.tetrain.com</nowiki></code> and <code><nowiki>ipa.tetrain.com</nowiki></code>''' — removing it would have split the topology into two disconnected islands with no path to sync future changes between them. IPA's own topology check caught this correctly rather than letting it happen silently. Fixed by adding a direct segment between the two remaining servers first, for both suffixes:

<pre>
<nowiki>
ipa topologysegment-add domain --leftnode=tornedo.tetrain.com --rightnode=ipa.tetrain.com
ipa topologysegment-add ca --leftnode=tornedo.tetrain.com --rightnode=ipa.tetrain.com
</nowiki>
</pre>

Verified both new agreements replicating successfully before retrying. <code><nowiki>ipa server-del ipa8.tetrain.com --force</nowiki></code> then completed cleanly.

'''Post-removal validation''': all 7 services RUNNING, all 9 certs <code><nowiki>MONITORING</nowiki></code> (no errors — ipa-custodia's renewal role wasn't disrupted despite <code><nowiki>ipa8</nowiki></code> having been its install-time "master peer"), <code><nowiki>ipa-custodia.service</nowiki></code> itself active/healthy, full LDAP+API roundtrip still works, and replication agreements now show only the direct <code><nowiki>tornedo.tetrain.com</nowiki></code> ↔ <code><nowiki>ipa.tetrain.com</nowiki></code> links — no remaining references to <code><nowiki>ipa8</nowiki></code> anywhere.

'''The <code><nowiki>ipa8.tetrain.com</nowiki></code> VM is now safe to shut down''' — cleanly removed from the topology first, so no broken replication agreements or stale data left behind.

=== 8.8 Current topology (bridge removed, two-server domain) ===

<pre>
<nowiki>
tornedo.tetrain.com (CentOS 7, IPA 4.6)      [ORIGINAL — still production-serving]
        ⇄ direct replica (domain + ca)
ipa.tetrain.com (AlmaLinux 9, IPA 4.13.1)    [FINAL TARGET — healthy, validated, not yet live]
</nowiki>
</pre>

=== 8.9 Cutover mechanism — corrected (SRV-based, not IP/identity handoff) ===

Checked before starting cutover: '''no DNS SRV records exist for this domain today''' — every client relies purely on the literal hostname <code><nowiki>tornedo.tetrain.com</nowiki></code> (confirmed via <code><nowiki>dig</nowiki></code>, all empty). This actually simplifies cutover versus the original IP-reassignment idea from Section 1: since <code><nowiki>ipa.tetrain.com</nowiki></code> has its own distinct identity (own cert, own Kerberos principals), making it impersonate <code><nowiki>tornedo.tetrain.com</nowiki></code>'s exact hostname would require fragile, not-well-supported FreeIPA identity renaming. Instead, publish real SRV records pointing at <code><nowiki>ipa.tetrain.com</nowiki></code> by its own name — <code><nowiki>sssd.conf</nowiki></code>'s <code><nowiki>ipa_server = _srv_, tornedo.tetrain.com</nowiki></code> already tries SRV lookup first and only falls back to the literal name if that fails, so once real records exist, clients will discover and correctly validate against <code><nowiki>ipa.tetrain.com</nowiki></code> without any client-side config changes and without touching <code><nowiki>tornedo.tetrain.com</nowiki></code>'s A record at all (which stays up as a fallback for any client not using SRV discovery).

Records to publish (plain client-discovery SRV records — distinct from the AD-trust-style <code><nowiki>_msdcs.</nowiki></code> records <code><nowiki>ipa-adtrust-install</nowiki></code> prints, which are for actual Windows AD interop, not needed here):

<pre>
<nowiki>
_ldap._tcp.tornedo.tetrain.com.            SRV 0 100 389  ipa.tetrain.com.
_kerberos._tcp.tornedo.tetrain.com.        SRV 0 100 88   ipa.tetrain.com.
_kerberos._udp.tornedo.tetrain.com.        SRV 0 100 88   ipa.tetrain.com.
_kerberos-master._tcp.tornedo.tetrain.com. SRV 0 100 88   ipa.tetrain.com.
_kerberos-master._udp.tornedo.tetrain.com. SRV 0 100 88   ipa.tetrain.com.
_kpasswd._tcp.tornedo.tetrain.com.         SRV 0 100 464  ipa.tetrain.com.
_kpasswd._udp.tornedo.tetrain.com.         SRV 0 100 464  ipa.tetrain.com.
</nowiki>
</pre>

=== 8.10 IMPORTANT finding during pre-cutover validation — missing SIDs broke auth for 59 users ===

Client tested logging into <code><nowiki>ipa.tetrain.com</nowiki></code>'s web UI as a real user (<code><nowiki>biswajit</nowiki></code>) — succeeded on <code><nowiki>tornedo.tetrain.com</nowiki></code>, '''failed''' on <code><nowiki>ipa.tetrain.com</nowiki></code> with the same password. Root cause, confirmed via <code><nowiki>krb5kdc.log</nowiki></code>: <code><nowiki>admin</nowiki></code>'s AS-REQ goes <code><nowiki>NEEDED_PREAUTH</nowiki></code> → <code><nowiki>ISSUE</nowiki></code> (clean); a normal user's goes <code><nowiki>NEEDED_PREAUTH</nowiki></code> → <code><nowiki>HANDLE_AUTHDATA: ... No such file or directory</nowiki></code> — fails during PAC generation, not password verification (this is NOT a <code><nowiki>PREAUTH_FAILED</nowiki></code>/wrong-password error).

Traced to exactly the warning <code><nowiki>ipa.tetrain.com</nowiki></code>'s install printed and unattended mode explicitly skipped: *"66 existing users or groups do not have a SID identifier assigned... installer will NOT run ipa-sidgen task."* Confirmed via LDAP: <code><nowiki>biswajit</nowiki></code> had no <code><nowiki>ipaNTSecurityIdentifier</nowiki></code> attribute at all (<code><nowiki>admin</nowiki></code> did — <code><nowiki>S-1-5-21-...-500</nowiki></code>). '''59 users total''' were affected — this would have broken login for the vast majority of real users the moment cutover happened, not just one account. Root cause of why it only breaks on the *new* server despite identical replicated LDAP data: <code><nowiki>ipa.tetrain.com</nowiki></code>'s install activated the AD-trust/SID plugin, and this IPA version (4.13) enforces PAC/SID generation during Kerberos ticket issuance more strictly than the old IPA 4.6 server does — the old server simply doesn't hit this code path.

'''Fixed''': <code><nowiki>ipa-adtrust-install</nowiki></code> wasn't even installed (separate package). Installed <code><nowiki>ipa-server-trust-ad</nowiki></code>, then ran <code><nowiki>ipa-adtrust-install -U --add-sids</nowiki></code> (client ran it directly, on <code><nowiki>ipa.tetrain.com</nowiki></code>) — completed the <code><nowiki>[23/24]: adding SIDs to existing users and groups</nowiki></code> step. '''Verified''': <code><nowiki>biswajit</nowiki></code> now has <code><nowiki>ipaNTSecurityIdentifier: S-1-5-21-...-1003</nowiki></code>, '''0 users remaining without a SID''', all services still healthy (plus two new ones now running: <code><nowiki>smb</nowiki></code>, <code><nowiki>winbind</nowiki></code>). Awaiting client's re-test of the actual web UI login to close this out completely.

'''Lesson for future replica builds''': never skip the <code><nowiki>ipa-sidgen</nowiki></code> unattended-mode warning as cosmetic — it silently breaks real-user authentication on the new server while looking completely fine in every health check that doesn't test a non-admin account specifically. This is exactly why testing with a real user, not just <code><nowiki>admin</nowiki></code>, mattered here.

=== 8.11 SRV records published and verified working — DONE ===

Client published all 7 records (Section 8.9) in the <code><nowiki>tetrain.com</nowiki></code> zone. Verified:

* All 7 resolve correctly via <code><nowiki>dig</nowiki></code> from both this session and from a real client's network path (<code><nowiki>koha.tetra.in</nowiki></code>) — correct priority/weight/port, target <code><nowiki>ipa.tetrain.com</nowiki></code>.
* '''End-to-end proof it actually works''': restarted <code><nowiki>sssd</nowiki></code> on <code><nowiki>koha.tetra.in</nowiki></code> (a real production IPA client) to force fresh discovery, then checked the actual artifact SSSD writes with its discovery result — <code><nowiki>/var/lib/sss/pubconf/kdcinfo.TORNEDO.TETRAIN.COM</nowiki></code> now contains <code><nowiki>13.126.212.202</nowiki></code> (which is <code><nowiki>ipa.tetrain.com</nowiki></code>'s IP, not <code><nowiki>tornedo.tetrain.com</nowiki></code>'s <code><nowiki>13.126.118.148</nowiki></code>). SSSD is discovering and will connect to the new server via SRV, exactly as designed — no client-side config was touched to achieve this.

=== 8.12 CA renewal master promoted to <code><nowiki>ipa.tetrain.com</nowiki></code> — DONE ===

<code><nowiki>ipa-csreplica-manage set-renewal-master ipa.tetrain.com</nowiki></code> needed the '''Directory Manager''' password (a more privileged credential than <code><nowiki>admin</nowiki></code> — the LDAP superuser) — client ran it directly, same pattern as every other admin-credential step. Verified via LDAP: <code><nowiki>cn=CA,cn=ipa.tetrain.com,cn=masters,...</nowiki></code> now carries <code><nowiki>ipaConfigString: caRenewalMaster</nowiki></code>. All services still healthy post-change.

=== 8.13 Still ahead ===

# Soak period — confirm real client traffic lands on <code><nowiki>ipa.tetrain.com</nowiki></code> via SRV while <code><nowiki>tornedo.tetrain.com</nowiki></code> stays up as fallback (now proven working on at least one real client; worth spot-checking a few more over the soak period).
# Decommission <code><nowiki>tornedo.tetrain.com</nowiki></code> — still fully gated on client go-ahead per original instruction ("involve me when we are ready to cut over").

----

== 9. Broader fleet SRV-cache sweep + new-client onboarding findings (2026-07-23) ==

=== 9.1 SRV cache sweep across 13 additional identified clients ===

Client provided 15 raw IPs to check. 2 couldn't be reached with any known key/user combo (<code><nowiki>139.84.174.24</nowiki></code>, <code><nowiki>65.20.79.182</nowiki></code> at the time — the latter turned out to be <code><nowiki>zurt-ison.mysecuremail.email</nowiki></code>, see 9.2). Of the 13 identified and reachable:

* 6 were stale (still cached on <code><nowiki>tornedo.tetrain.com</nowiki></code>'s IP) and successfully refreshed to <code><nowiki>ipa.tetrain.com</nowiki></code> via <code><nowiki>systemctl restart sssd</nowiki></code>.
* '''<code><nowiki>dr-mobilepedia.tetrain.com</nowiki></code>''': has <code><nowiki>dns_lookup_kdc = false</nowiki></code> in <code><nowiki>krb5.conf</nowiki></code> — SRV discovery disabled at the Kerberos level, won't fail over automatically no matter how long we wait. Needs a manual fix (set to <code><nowiki>true</nowiki></code>) before <code><nowiki>tornedo</nowiki></code> can be safely retired for this client.
* '''<code><nowiki>mail.tetrain.com</nowiki></code>''': uses itself as its own primary DNS resolver (<code><nowiki>nameserver 139.84.171.188</nowiki></code> = its own IP in <code><nowiki>/etc/resolv.conf</nowiki></code>) — a direct SRV <code><nowiki>dig</nowiki></code> from that box returns empty, meaning its own local resolver isn't serving our new records. Root cause not yet resolved — needs investigation into that box's own DNS/resolver setup.
* '''<code><nowiki>mbox2.mysecuremail.email</nowiki></code>''': <code><nowiki>sssd</nowiki></code> isn't persistently running (starts, runs ~60s, gets stopped) — confirmed via journal history this exact pattern predates today's work (same pattern on Jul 17), so '''not''' something we caused. Cause unknown, needs its own investigation before this client's cutover-readiness can be assessed.
* 4 of the 15 aren't IPA clients at all (no sssd running): <code><nowiki>zurt-sysnet.mysecuremail.email</nowiki></code>, <code><nowiki>mail.tetra.in</nowiki></code>, <code><nowiki>single.tetra.in</nowiki></code>, <code><nowiki>email.rahulchaudhry.com</nowiki></code>.

'''Takeaway confirmed by this sweep''': caching lag is real and expected (fixed by a simple sssd restart), but there are also genuine per-client configuration gaps that SRV records alone don't fix. Both <code><nowiki>dr-mobilepedia</nowiki></code> and <code><nowiki>mail.tetrain.com</nowiki></code> need to be resolved individually before <code><nowiki>tornedo.tetrain.com</nowiki></code> can be safely retired — a blanket "wait for the soak period" isn't sufficient for these two.

=== 9.2 New client onboarding — systemic gap found on Debian/Ubuntu clients ===

<code><nowiki>65.20.79.182</nowiki></code> turned out to be a '''new''' server being onboarded (<code><nowiki>zurt-ison.mysecuremail.email</nowiki></code>, not yet enrolled) — client is onboarding new servers against <code><nowiki>ipa.tetrain.com</nowiki></code> directly going forward, per Section 8.9's guidance. Two distinct issues hit during this one onboarding, both worth carrying forward as standard steps for future Debian/Ubuntu (not RHEL-family) clients:

# '''<code><nowiki>/etc/hosts</nowiki></code> had the hostname's two label halves swapped''' (<code><nowiki>ison-zurt</nowiki></code> vs. actual <code><nowiki>zurt-ison</nowiki></code>) *and* pointed at loopback (<code><nowiki>127.0.1.1</nowiki></code>) — meant <code><nowiki>hostname -f</nowiki></code> failed outright with a Python traceback in <code><nowiki>ipa-client-install</nowiki></code> (<code><nowiki>socket.gaierror</nowiki></code>) before even reaching the normal loopback-resolution failure mode we'd seen before. Fixed by correcting <code><nowiki>/etc/hosts</nowiki></code> to the real name + real routable IP, same pattern as the <code><nowiki>ipa8</nowiki></code>/<code><nowiki>ipa.tetrain.com</nowiki></code> fixes.
# '''After successful enrollment, login still failed for every domain user, with zero SSSD log entries at all''' — traced to <code><nowiki>pam_sss.so</nowiki></code> being completely absent from <code><nowiki>/etc/pam.d/common-auth</nowiki></code> and <code><nowiki>/etc/pam.d/common-account</nowiki></code>. Root cause: Debian/Ubuntu manages the PAM stack via <code><nowiki>pam-auth-update</nowiki></code>, a *separate* mechanism from what <code><nowiki>ipa-client-install</nowiki></code> configures directly (<code><nowiki>authselect</nowiki></code> on RHEL-family systems, which reliably wires up <code><nowiki>pam_sss</nowiki></code> automatically) — on this Debian client, the SSS profile was never enabled. Without it, <code><nowiki>pam_unix.so</nowiki></code> fails for any non-local account and falls straight through to <code><nowiki>pam_deny.so</nowiki></code>, rejecting the login before SSSD is ever consulted — this looks identical to "wrong password" from the client's perspective, indistinguishable without checking the PAM stack directly.

'''Fix''': <code><nowiki>pam-auth-update --force --enable sss</nowiki></code> (the <code><nowiki>--force</nowiki></code> was needed because the image's default <code><nowiki>nullok</nowiki></code> customization in <code><nowiki>common-auth</nowiki></code> counted as a "local modification" blocking the normal non-forced update).

'''Action item for future Debian/Ubuntu onboarding''': after <code><nowiki>ipa-client-install</nowiki></code> completes on any Debian-family box, always run <code><nowiki>pam-auth-update --force --enable sss</nowiki></code> and verify with <code><nowiki>grep pam_sss /etc/pam.d/common-auth /etc/pam.d/common-account</nowiki></code> *before* considering the onboarding done — don't wait for a real user to hit the failure to discover this gap.

=== 9.3 Two more onboardings (<code><nowiki>zurt-sysnet</nowiki></code>, <code><nowiki>email.rahulchaudhry.com</nowiki></code>) — same hosts-file bug, ===

plus two *new* systemic gaps found

'''<code><nowiki>zurt-sysnet.mysecuremail.email</nowiki></code>''': identical swapped-hostname + loopback <code><nowiki>/etc/hosts</nowiki></code> bug as <code><nowiki>zurt-ison</nowiki></code> (labels reversed: <code><nowiki>sysnet-zurt</nowiki></code> vs. actual <code><nowiki>zurt-sysnet</nowiki></code>) — same fix applied (correct name + real IP <code><nowiki>65.20.85.22</nowiki></code>, <code><nowiki>manage_etc_hosts: false</nowiki></code>).

'''<code><nowiki>email.rahulchaudhry.com</nowiki></code>''' (Ubuntu 25.10) — three separate issues in one onboarding:

# <code><nowiki>ipa-client-install</nowiki></code> failed at the time-sync step: <code><nowiki>chrony.service</nowiki></code> not found — chrony wasn't installed on this image at all, even though the installer tried to configure it. Fixed with <code><nowiki>apt-get install -y chrony</nowiki></code> before retrying. Worth checking chrony is present *before* running <code><nowiki>ipa-client-install</nowiki></code> on future minimal/stripped images.
# <code><nowiki>--mkhomedir</nowiki></code> didn't actually create home directories despite being passed — same Debian/pam-auth-update gap as the <code><nowiki>pam_sss</nowiki></code> one, but for a *different* profile: <code><nowiki>pam_mkhomedir.so</nowiki></code> was missing from <code><nowiki>common-session</nowiki></code>. Fix: <code><nowiki>pam-auth-update --force --enable mkhomedir</nowiki></code> (in addition to <code><nowiki>--enable sss</nowiki></code>, not instead of).
# '''New, distinct finding''': sudo denied for a user (<code><nowiki>biswajit</nowiki></code>) known to have valid, working IPA sudo rules (<code><nowiki>sudorule1</nowiki></code>, Host category: all, confirmed via <code><nowiki>ipa sudorule-show</nowiki></code>) — with zero SSSD sudo-responder log activity for any of it, even after confirming <code><nowiki>libsss-sudo</nowiki></code> was installed, the shared library existed on disk, and the SSSD sudo socket was present and correctly permissioned. Root cause: **Ubuntu 25.10 ships <code><nowiki>sudo-rs</nowiki></code> (a Rust reimplementation of sudo) as the *default* <code><nowiki>/usr/bin/sudo</nowiki></code>**, and this version (<code><nowiki>0.2.8</nowiki></code>) does not appear to support SSSD/IPA-driven sudo rules — it silently falls back to local-file-only sudoers evaluation, giving a plausible-looking (if oddly-worded — custom "I'm sorry, I'm afraid I can't do that" message) denial with no diagnostic trail on the SSSD side at all. Classic GNU sudo (<code><nowiki>1.9.17p2</nowiki></code>) was already installed alongside it, just not the active alternative.

'''Fix''': switch the <code><nowiki>sudo</nowiki></code> alternative to the classic binary (which the <code><nowiki>sudo</nowiki></code> package installs at <code><nowiki>/usr/bin/sudo.ws</nowiki></code> on this Ubuntu convention, not the plain <code><nowiki>/usr/bin/sudo</nowiki></code> path — no <code><nowiki>dpkg-divert</nowiki></code> involved, just an incompletely-registered alternatives group):

<pre>
<nowiki>
   update-alternatives --install /usr/bin/sudo sudo /usr/bin/sudo.ws 50
   update-alternatives --set sudo /usr/bin/sudo.ws
</nowiki>
</pre>

Verified via <code><nowiki>sudo --version</nowiki></code> (no longer reports <code><nowiki>sudo-rs</nowiki></code>) and <code><nowiki>sudo -l -U biswajit</nowiki></code> (correctly returned <code><nowiki>(root) NOPASSWD: ALL</nowiki></code>, matching <code><nowiki>sudorule1</nowiki></code>).

'''Updated action item for Ubuntu 25.10+ onboarding specifically''': in addition to the <code><nowiki>pam-auth-update --force --enable sss</nowiki></code> step, also run <code><nowiki>--enable mkhomedir</nowiki></code>, and check <code><nowiki>sudo --version</nowiki></code> — if it reports <code><nowiki>sudo-rs</nowiki></code>, switch to classic sudo via the <code><nowiki>update-alternatives</nowiki></code> commands above *before* considering the box's sudo access functional for any IPA-managed user.

=== 9.4 Second CSV sweep (14 hosts) — 8 refreshed cleanly, 2 needed a *different* fix, 4 unreachable ===

Client provided a Linode-hosted list of 14 hostnames+IPs. 4 not accessible with known keys (<code><nowiki>server-relay-2.tetrain.com-new</nowiki></code>, <code><nowiki>access.tetrain.com</nowiki></code>, <code><nowiki>otrs.tetrain.com</nowiki></code>, <code><nowiki>nagios.tetrain.com</nowiki></code> — need different credentials). Of the 10 reachable, 8 refreshed cleanly via the standard <code><nowiki>systemctl restart sssd</nowiki></code> approach (<code><nowiki>zm.wbtea.com</nowiki></code>, <code><nowiki>mail.sysnetglobal.com</nowiki></code>, <code><nowiki>fcra.tetra.in</nowiki></code>, <code><nowiki>list.tetra-india.in</nowiki></code>, <code><nowiki>mobile.sceh.net</nowiki></code>, <code><nowiki>ihelpdesk-belpoc.tetra.in</nowiki></code>, <code><nowiki>nic.tetra.in</nowiki></code>, <code><nowiki>wiki.tetrain.com</nowiki></code>).

'''2 needed real investigation — <code><nowiki>relay.isgec.com</nowiki></code> and <code><nowiki>mail.sceh.net</nowiki></code>.''' Neither had <code><nowiki>dns_lookup_kdc = false</nowiki></code> this time (that got fixed, still didn't switch) — the actual cause was different and more interesting: both had <code><nowiki>dns_discovery_domain = tetrain.com</nowiki></code> in <code><nowiki>sssd.conf</nowiki></code>, instead of <code><nowiki>tornedo.tetrain.com</nowiki></code> like every other client. This setting controls what domain suffix SSSD appends when constructing SRV record names to query — with it set to the shorter <code><nowiki>tetrain.com</nowiki></code>, these clients were silently querying <code><nowiki>_ldap._tcp.tetrain.com</nowiki></code>/<code><nowiki>_kerberos._tcp.tetrain.com</nowiki></code> (which don't exist) instead of our actual published records under <code><nowiki>_ldap._tcp.tornedo.tetrain.com</nowiki></code>. A direct manual <code><nowiki>dig</nowiki></code> with the full correct name succeeded (masking the problem during initial diagnosis), while SSSD's own automatic discovery never found anything. Confirmed via full <code><nowiki>sssd.conf</nowiki></code> inspection: both had <code><nowiki>ipa_domain = tetrain.com</nowiki></code> too (their local IPA "domain" section is literally named <code><nowiki>[domain/tetrain.com]</nowiki></code>, not <code><nowiki>[domain/tornedo.tetrain.com]</nowiki></code>) — left <code><nowiki>ipa_domain</nowiki></code> untouched since it's tied to other working config on long-established production clients; only <code><nowiki>dns_discovery_domain</nowiki></code> was corrected. Both switched to <code><nowiki>ipa.tetrain.com</nowiki></code> immediately after.

'''There are now two distinct, confirmed root causes for "SRV records published but a client still won't discover the new server," both worth checking on any client that doesn't switch after a plain <code><nowiki>sssd</nowiki></code> restart:'''

# <code><nowiki>dns_lookup_kdc = false</nowiki></code> in <code><nowiki>/etc/krb5.conf</nowiki></code> (found on <code><nowiki>dr-mobilepedia.tetrain.com</nowiki></code>, fix not yet re-verified after the flag change — worth confirming).
# <code><nowiki>dns_discovery_domain</nowiki></code> set to a shorter/different suffix than <code><nowiki>tornedo.tetrain.com</nowiki></code> in <code><nowiki>sssd.conf</nowiki></code> — '''confirmed on 3 hosts now''': <code><nowiki>relay.isgec.com</nowiki></code>, <code><nowiki>mail.sceh.net</nowiki></code>, <code><nowiki>nextcloud.tetra.in</nowiki></code>. All fixed the same way (correct the value, restart sssd — no cache clear needed, this isn't a caching issue, it's querying the wrong SRV record names entirely). Common enough now to check *first*, before assuming a cache/timing issue, on any host stuck on the old IP even after an aggressive <code><nowiki>sss_cache -E</nowiki></code> + full stop/start.

A host showing the "stuck on old IP even after restart" symptom needs both checked, not just one — they're independent settings that can each cause the identical-looking symptom.

=== 9.5 Additional hosts checked/fixed outside the CSV sweeps ===

* <code><nowiki>li2028-191.members.linode.com</nowiki></code> ("Nagios Server" per its IPA description) — switched successfully via <code><nowiki>/etc/init.d/sssd restart</nowiki></code> (no systemd on this box). Separately flagged: running '''CentOS 6.8''', EOL since November 2020 — outside the original 31-server fleet scope, significantly more urgent than any CentOS 7 box found so far if/when OS remediation is revisited.
* <code><nowiki>nextcloud.tetra.in</nowiki></code> — same <code><nowiki>dns_discovery_domain</nowiki></code> bug as above, fixed and confirmed.

== 10. Security incident found and remediated: distributed SSH brute-force → cascading KDC lockouts ==

While waiting for <code><nowiki>admin</nowiki></code>'s temporary lockout to clear (see below), investigated *why* it got locked in the first place — turned out to be a real, active security issue, not a fluke.

=== 10.1 What happened ===

<code><nowiki>admin</nowiki></code>'s Kerberos account hit <code><nowiki>krbPwdMaxFailure</nowiki></code> (6 failures within <code><nowiki>krbPwdFailureCountInterval</nowiki></code> = 60s, resulting in <code><nowiki>krbPwdLockoutDuration</nowiki></code> = 600s lockout). Checked <code><nowiki>krb5kdc.log</nowiki></code> for the source of the failures and found '''two separate accounts being hit''':

* <code><nowiki>arun@TORNEDO.TETRAIN.COM</nowiki></code> — repeatedly, every 10-15 seconds, from <code><nowiki>relay.isgec.com</nowiki></code> (<code><nowiki>176.58.97.66</nowiki></code>), targeting <code><nowiki>kadmin/changepw</nowiki></code>. Checked the account: <code><nowiki>nsAccountLock: TRUE</nowiki></code> (explicitly disabled) and <code><nowiki>krbPasswordExpiration: 2022-05-16</nowiki></code> — a long-dead account, not a current employee.
* <code><nowiki>admin@TORNEDO.TETRAIN.COM</nowiki></code> — hit from <code><nowiki>upyog.tetrain.com</nowiki></code> and <code><nowiki>mail.sceh.net</nowiki></code>.

Checked <code><nowiki>relay.isgec.com</nowiki></code> directly: no cron job, no local script — <code><nowiki>ss -tnp</nowiki></code> showed multiple simultaneous established SSH connections from wildly different, geographically scattered public IPs (<code><nowiki>195.9.34.73</nowiki></code>, <code><nowiki>201.98.60.254</nowiki></code>, <code><nowiki>94.23.34.95</nowiki></code>, <code><nowiki>210.61.148.174</nowiki></code>). This is the signature of a '''distributed SSH brute-force/credential-stuffing botnet''' hitting port 22, cycling through known/guessed usernames (<code><nowiki>arun</nowiki></code> from old breach data or reconnaissance, <code><nowiki>admin</nowiki></code> as an obvious guess). Because all these servers now share the same centralized IPA/Kerberos backend, failed SSH attempts on *any* of them cascade into real Kerberos account lockouts for legitimate users on the shared KDC — this is what was blocking our own <code><nowiki>ipa host-del</nowiki></code> cleanup work.

Checked <code><nowiki>relay.isgec.com</nowiki></code>'s exposure: '''no <code><nowiki>fail2ban</nowiki></code> installed at all''', and '''<code><nowiki>PasswordAuthentication yes</nowiki></code>''' — fully open to this kind of attack with zero rate-limiting.

=== 10.2 Remediation on <code><nowiki>relay.isgec.com</nowiki></code> — DONE ===

# <code><nowiki>yum install -y fail2ban</nowiki></code> failed initially — <code><nowiki>fail2ban-server</nowiki></code> hard-requires <code><nowiki>systemd-python</nowiki></code>, which wasn't installed, and installing it wanted to bump core <code><nowiki>systemd</nowiki></code> packages to a version unavailable on the regular (post-EOL, pruned) CentOS 7 mirrors — same mirror-pruning issue seen on <code><nowiki>tornedo.tetrain.com</nowiki></code> earlier.
# Repointed <code><nowiki>/etc/yum.repos.d/CentOS-Base.repo</nowiki></code> at CentOS Vault (<code><nowiki>vault.centos.org/7.9.2009/</nowiki></code>), same fix pattern as before. This box's existing <code><nowiki>CentOS-Vault.repo</nowiki></code> file only covered old point releases (7.0–7.5), not 7.9, so edited <code><nowiki>CentOS-Base.repo</nowiki></code> directly instead of using it.
# Retried — succeeded, including the <code><nowiki>systemd</nowiki></code> bump (<code><nowiki>219-62.el7</nowiki></code> → <code><nowiki>219-78.el7_9.9</nowiki></code>). '''Verified system health after''': <code><nowiki>systemctl is-system-running</nowiki></code> → <code><nowiki>running</nowiki></code>, <code><nowiki>sssd</nowiki></code>/<code><nowiki>sshd</nowiki></code> both active, uptime unchanged (no reboot triggered, live daemon-reexec handled it as expected on EL7).
# Configured <code><nowiki>/etc/fail2ban/jail.local</nowiki></code>: <code><nowiki>sshd</nowiki></code> jail enabled, <code><nowiki>maxretry=5</nowiki></code>, <code><nowiki>bantime=3600</nowiki></code>/<code><nowiki>findtime=600</nowiki></code>, '''<code><nowiki>ignoreip</nowiki></code> includes the client's VPN egress IP (<code><nowiki>45.79.120.55</nowiki></code>, also <code><nowiki>access.tetrain.com</nowiki></code>'s IP) so it's never banned'''.
# Enabled and started. '''Immediately effective''': picked up the already-ongoing attack (214 total failed attempts recorded) and banned 5 attacking IPs on the spot, including the exact ones seen live in <code><nowiki>ss -tnp</nowiki></code> earlier. Confirmed whitelist via <code><nowiki>fail2ban-client get sshd ignoreip</nowiki></code>.

=== 10.3 Still open — this is very likely fleet-wide, not just <code><nowiki>relay.isgec.com</nowiki></code> ===

The <code><nowiki>admin</nowiki></code> lockout was also triggered from <code><nowiki>upyog.tetrain.com</nowiki></code> and <code><nowiki>mail.sceh.net</nowiki></code> — strongly suggesting the same exposure (no fail2ban, password auth enabled) exists on other boxes too. '''Not yet swept.''' Recommended next steps whenever picked back up:

# Check <code><nowiki>fail2ban</nowiki></code> presence + <code><nowiki>PasswordAuthentication</nowiki></code> setting across the rest of the reachable fleet, same two checks used on <code><nowiki>relay.isgec.com</nowiki></code>.
# Install/configure fail2ban (with the same VPN whitelist) on every exposed box found.
# Longer-term: consider disabling SSH password auth fleet-wide in favor of key-only, which removes this attack surface entirely rather than just rate-limiting it.
# The disabled <code><nowiki>arun</nowiki></code> account being targeted specifically suggests old/leaked credentials are circulating — worth a broader check for any other long-disabled accounts that might be similarly targeted, and confirming they're all properly locked (not just disabled at the OS level but also <code><nowiki>nsAccountLock: TRUE</nowiki></code> in IPA, as <code><nowiki>arun</nowiki></code> correctly was).

== 11. Decommissioning plan — Phase 1 items 1-5 closed, Phase 2 (access-log readiness check) result: NO-GO (2026-07-24) ==

Phase 1 (pre-checks): items 1-4 confirmed closed by client. Item 5 (special/exclusive-role check) independently verified clear — <code><nowiki>tornedo.tetrain.com</nowiki></code> holds no CRL-generation or CA-renewal-master role (both already on <code><nowiki>ipa.tetrain.com</nowiki></code>); its remaining service flags (NTP, KDC, KPASSWD, KEYS, CA, OTPD, HTTP) are normal <code><nowiki>enabledService</nowiki></code> participation, not exclusive roles. Nothing needs to be moved before decommission on that front.

=== 11.1 Phase 2, point 6 — access-log readiness check (methodology) ===

Re-ran the dirsrv access-log "who's actually still connecting" analysis, this time scoped specifically at <code><nowiki>tornedo.tetrain.com</nowiki></code> and cross-checked against <code><nowiki>ipa.tetrain.com</nowiki></code>, so a host that has fully cut over shows up as absent-from-tornedo/present-on-ipa rather than a false negative.

* <code><nowiki>tornedo.tetrain.com</nowiki></code>: 4 rotated logs + current, covering <code><nowiki>2026-07-22 18:29</nowiki></code> → <code><nowiki>2026-07-24 06:56</nowiki></code> (~36.5h retention).
* <code><nowiki>ipa.tetrain.com</nowiki></code> (instance still named <code><nowiki>slapd-TORNEDO-TETRAIN-COM</nowiki></code>, inherited from the replica build): 1 rotated log + current, covering <code><nowiki>2026-07-23 14:22</nowiki></code> → <code><nowiki>2026-07-24 07:00</nowiki></code> (~16.6h retention).
* Extracted last-seen timestamp + connection count per source IP from both (<code><nowiki>grep "connection from"</nowiki></code>, field <code><nowiki>$8</nowiki></code> = IP), resolved fresh DNS for all 50 hosts in <code><nowiki>reports/ipa_hosts_final_2026-07-24.csv</nowiki></code>, and cross-referenced.
* For the 9 hosts with no forward DNS record (<code><nowiki>app.brijraj.in</nowiki></code>, <code><nowiki>app.brijraj.in-new</nowiki></code>, <code><nowiki>cloud-storage.triburg.co.in</nowiki></code>, <code><nowiki>dr-mobilepedia.tetrain.com</nowiki></code>, <code><nowiki>li1732-118</nowiki></code>/<code><nowiki>li1994-156 .members.linode.com</nowiki></code>, <code><nowiki>mail.curiocrafts.com</nowiki></code>, <code><nowiki>mobilepedia-sites.tetrain.com</nowiki></code>, <code><nowiki>tetra-demo.tetrain.com</nowiki></code>), ran reverse-DNS on unmatched tornedo-log IPs instead, which identified several of them directly connecting under a different IP than the CSV has on file.

=== 11.2 Result: traffic to tornedo is current and substantial, not near zero — target not met ===

63 distinct IPs hit tornedo in the 36.5h window, most with '''timestamps in the last few minutes before the log snapshot was taken''' (i.e. essentially live at time of check), at volumes (~100- 160 connections per host over the window) consistent with normal periodic SSSD traffic, not stragglers. This is the opposite of the "traffic near zero, sustained" target set for Phase 3.

'''Group A — fully migrated (tornedo stale/absent, ipa current): safe.''' <code><nowiki>koha.tetra.in</nowiki></code>, <code><nowiki>li2028-191</nowiki></code>/<code><nowiki>li2092-64</nowiki></code>/<code><nowiki>li2095-168.members.linode.com</nowiki></code>, <code><nowiki>mail.sceh.net</nowiki></code>, <code><nowiki>mail.sysnetglobal.com</nowiki></code>, <code><nowiki>mail.tetra.in</nowiki></code>, <code><nowiki>mail.isonemployees.com</nowiki></code>/<code><nowiki>single.tetra.in</nowiki></code>/ <code><nowiki>zm.isonemployees.co.za</nowiki></code> (shared IP), <code><nowiki>mbox1.mysecuremail.email</nowiki></code>, <code><nowiki>mbox2.mysecuremail.email</nowiki></code>, <code><nowiki>php-devel-1.tetra.in</nowiki></code>, <code><nowiki>relay.isgec.com</nowiki></code>, <code><nowiki>upyog.tetrain.com</nowiki></code>, <code><nowiki>zm.wbtea.com</nowiki></code>, <code><nowiki>zurt-ison.mysecuremail.email</nowiki></code>, <code><nowiki>zurt-sysnet.mysecuremail.email</nowiki></code>, <code><nowiki>devel-isonbpo-ihelpdesk.tetra.in</nowiki></code>. Confirms the two client-flagged "stragglers" <code><nowiki>mbox2.mysecuremail.email</nowiki></code> and <code><nowiki>devel-isonbpo-ihelpdesk.tetra.in</nowiki></code> are indeed resolved.

'''Group B — dual-hitting, still transitional:''' <code><nowiki>cloud-storage.tetra.in</nowiki></code>, <code><nowiki>ihelpdesk-belpoc.tetra.in</nowiki></code>, <code><nowiki>www.mobilepedia.in</nowiki></code>, <code><nowiki>nextcloud.tetra.in</nowiki></code>, <code><nowiki>tetra-demo2.tetrain.com</nowiki></code>, <code><nowiki>www.tetrain.com</nowiki></code>, <code><nowiki>zurt.tetrain.com</nowiki></code> — hitting both servers currently, ipa share growing but not yet exclusive. Not blockers, but not done either.

'''Group C — tornedo-only or tornedo-dominant, live right now — hard blockers:''' <code><nowiki>app-1.tetrain.com</nowiki></code>, <code><nowiki>mail.energoconstruction.com</nowiki></code>/<code><nowiki>mail.energoproducts.com</nowiki></code> (shared IP), <code><nowiki>nic.tetra.in</nowiki></code>, <code><nowiki>server-relay.tetrain.com</nowiki></code>, <code><nowiki>server-relay-2.tetrain.com</nowiki></code>, <code><nowiki>indorama-relay.tetrain.com</nowiki></code>, <code><nowiki>app.brijraj.in</nowiki></code>, <code><nowiki>tetra-demo.tetrain.com</nowiki></code>, <code><nowiki>cloud-storage.triburg.co.in</nowiki></code>, and — '''important discrepancy''' — <code><nowiki>mail.tetrain.com</nowiki></code>/ <code><nowiki>zimbra-test.tetra.in</nowiki></code> (shared IP <code><nowiki>139.84.171.188</nowiki></code>): 161 live connections to tornedo vs. a single stray hit on ipa 13+ hours ago. This one was on the client's "confirmed closed" list for Phase 1 but the log data says it has '''not''' cut over — worth a direct re-check on that host's <code><nowiki>sssd.conf</nowiki></code>/<code><nowiki>krb5.conf</nowiki></code> before trusting it as closed.

'''Group D — active clients hitting tornedo that aren't in the 50-host inventory at all:''' <code><nowiki>server-relay-1.tetra.in</nowiki></code> (52.199.95.254), <code><nowiki>mail.prime-sdec.com</nowiki></code> (111.93.232.170), an Airtel static/NAT IP (182.72.51.106), two bare AWS EC2 IPs with no reverse hostname (3.6.120.132, 3.111.216.210), and one IP that timed out on reverse lookup (139.84.143.206) — all with live, recent, regular (~100-150 count) traffic. These need identifying and adding to inventory before decommission, since they're real current dependents we don't otherwise have visibility into.

'''Background noise, not decommission-relevant:''' <code><nowiki>172.26.27.51</nowiki></code> (private VPC IP, highest count of all at 250 — almost certainly <code><nowiki>ipa.tetrain.com</nowiki></code>'s own replication connection to tornedo over the internal network, expected until topology is formally torn down in a later phase); <code><nowiki>127.0.0.1</nowiki></code>/<code><nowiki>::1</nowiki></code> (local health checks); <code><nowiki>65.20.78.116</nowiki></code> (old <code><nowiki>ipa8.tetrain.com</nowiki></code> IP, last seen 16h+ before window close — dead, matches its decommission); <code><nowiki>13.232.220.151</nowiki></code> (old <code><nowiki>ipa.tetrain.com</nowiki></code> IP pre-cutover, 1 stray hit); ~15 single/low-count IPs from Jul 22-23 consistent with the SSH-brute-force scanning residue documented in section 10, not real IPA clients.

=== 11.3 Verdict ===

'''No-go on Phase 3.''' At minimum 10 known hosts (Group C) plus 5-6 unidentified active clients (Group D) are currently, live, dependent on tornedo — several exclusively. Recommend: fix Group C's SRV/<code><nowiki>dns_lookup_kdc</nowiki></code> config (same pattern as section 9's fleet sweep), identify and enroll Group D properly, re-run this exact check in 24-48h, and only proceed to Phase 3 once Group C/D traffic drops to the same stale/absent pattern Group A already shows.

== 12. Client-initiated shutdown of tornedo (2026-07-24) — live remediation of Group C hosts ==

Client shut down <code><nowiki>tornedo.tetrain.com</nowiki></code> ahead of the Phase 3 go/no-go in section 11 (no-go was flagged; client proceeded anyway). This turned the Group C readiness findings into a live outage check instead of a projection. Went through each affected host, diagnosed root cause, fixed, and verified with a live <code><nowiki>kinit -k</nowiki></code> (host keytab, no admin password involved) before calling it closed. Two distinct bug classes emerged beyond the known <code><nowiki>dns_lookup_kdc</nowiki></code>/SRV issue from section 9:

'''Bug class 1 — wrong <code><nowiki>ipa_domain</nowiki></code>:''' several hosts were enrolled with <code><nowiki>ipa_domain = tetrain.com</nowiki></code> instead of the real IPA domain <code><nowiki>tornedo.tetrain.com</nowiki></code> (confirmed via <code><nowiki>ipa.tetrain.com</nowiki></code>'s own config: realm/domain is <code><nowiki>TORNEDO.TETRAIN.COM</nowiki></code>/<code><nowiki>tornedo.tetrain.com</nowiki></code>, not <code><nowiki>tetrain.com</nowiki></code>). This sends SSSD's SRV lookups at a zone with no records (<code><nowiki>_ldap._tcp.tetrain.com</nowiki></code> — confirmed NXDOMAIN, authoritatively, at GoDaddy). Fix: correct <code><nowiki>ipa_domain</nowiki></code>, the <code><nowiki>[domain/...]</nowiki></code> section header, <code><nowiki>domains =</nowiki></code>, <code><nowiki>ipa_server</nowiki></code> static fallback, and krb5.conf's <code><nowiki>[realms]</nowiki></code> KDC/admin_server entries to point at <code><nowiki>ipa.tetrain.com</nowiki></code>. Turned <code><nowiki>dns_lookup_kdc</nowiki></code> on where it was off.

'''Bug class 2 — host firewall allowlist:''' some hosts run a default-DROP <code><nowiki>iptables OUTPUT</nowiki></code> policy with an explicit per-destination-IP allowlist that had tornedo's IP hardcoded but never got ipa.tetrain.com added. Correct DNS/krb5 config doesn't help if the packets never leave the host. Fix: mirror the existing tornedo ACCEPT rule for <code><nowiki>13.126.212.202</nowiki></code>, persist via <code><nowiki>iptables-save</nowiki></code>.

'''Bug class 3 (one host only) — broken local DNS resolver:''' <code><nowiki>mail.tetrain.com</nowiki></code> had entirely correct sssd/krb5 config, but its own <code><nowiki>/etc/resolv.conf</nowiki></code> lists itself (<code><nowiki>139.84.171.188</nowiki></code>, a local Zimbra-managed resolver) as the *first* nameserver, and that resolver returns a valid empty <code><nowiki>NOERROR</nowiki></code> for the SRV query — which glibc's resolver treats as final, never falling through to the other 4 working resolvers listed after it. This is why it was reported "confirmed closed" by the client yet was still found hammering tornedo almost exclusively in section 11's log check — the config genuinely was fine, DNS just silently lied. Fix: reordered <code><nowiki>resolv.conf</nowiki></code> to put a working recursive resolver first (local resolver left in place, just deprioritized).

'''Per-host outcome:'''

{| class="wikitable"
|-
! Host !! Bug !! Fix applied !! Verified
|-
| <code><nowiki>app-1.tetrain.com</nowiki></code> || wrong <code><nowiki>ipa_domain</nowiki></code> || config || live <code><nowiki>kinit</nowiki></code> ✓
|-
| <code><nowiki>server-relay-2.tetrain.com</nowiki></code> || wrong <code><nowiki>ipa_domain</nowiki></code> || config || live <code><nowiki>kinit</nowiki></code> ✓
|-
| <code><nowiki>mail.curiocrafts.com</nowiki></code> (aka <code><nowiki>mail.energoconstruction.com</nowiki></code>/<code><nowiki>mail.energoproducts.com</nowiki></code>) || wrong <code><nowiki>ipa_domain</nowiki></code> + firewall allowlist || config + iptables || live <code><nowiki>kinit</nowiki></code> ✓
|-
| <code><nowiki>mail.tetrain.com</nowiki></code> / <code><nowiki>zimbra-test.tetra.in</nowiki></code> || broken local DNS resolver order || resolv.conf reorder || live <code><nowiki>kinit</nowiki></code> ✓
|-
| <code><nowiki>app.brijraj.in</nowiki></code> || wrong <code><nowiki>ipa_domain</nowiki></code> + firewall allowlist || config + iptables || live <code><nowiki>kinit</nowiki></code> ✓
|-
| <code><nowiki>nic.tetra.in</nowiki></code> || none — already correct || — || live <code><nowiki>kinit</nowiki></code> ✓
|-
| <code><nowiki>server-relay.tetrain.com</nowiki></code> || none — already correct || — || live <code><nowiki>kinit</nowiki></code> ✓
|-
| <code><nowiki>tetra-demo.tetrain.com</nowiki></code> || none — already correct || — || live <code><nowiki>kinit</nowiki></code> ✓
|-
| <code><nowiki>cloud-storage.triburg.co.in</nowiki></code> || wrong <code><nowiki>ipa_domain</nowiki></code>, but masked by explicit <code><nowiki>dns_discovery_domain</nowiki></code> override — cosmetic only || none applied (optional cleanup later) || live <code><nowiki>kinit</nowiki></code> ✓
|-
| <code><nowiki>indorama-relay.tetrain.com</nowiki></code> || — || host deleted, dropped from scope || n/a
|}

All backups (<code><nowiki>*.bak.<timestamp></nowiki></code>) left in place on each host next to the files they replaced. Access notes: <code><nowiki>app-1</nowiki></code>/<code><nowiki>server-relay-2</nowiki></code>/<code><nowiki>nic.tetra.in</nowiki></code>/<code><nowiki>server-relay.tetrain.com</nowiki></code> reachable via the standard <code><nowiki>id_rsa_curio</nowiki></code> key; <code><nowiki>mail.curiocrafts.com</nowiki></code>/<code><nowiki>app.brijraj.in</nowiki></code>/ <code><nowiki>tetra-demo.tetrain.com</nowiki></code>/<code><nowiki>cloud-storage.triburg.co.in</nowiki></code> also via <code><nowiki>id_rsa_curio</nowiki></code> once tried against the right user (<code><nowiki>root</nowiki></code> or <code><nowiki>centos</nowiki></code>, <code><nowiki>HostKeyAlgorithms=+ssh-rsa</nowiki></code> needed for the older CentOS 6/7 boxes); <code><nowiki>mail.tetrain.com</nowiki></code> needed <code><nowiki>proxy_id_rsa</nowiki></code> as <code><nowiki>root</nowiki></code> — none of the fleet-standard keys worked there, and password auth is disabled for <code><nowiki>biswajit</nowiki></code> on both <code><nowiki>mail.tetrain.com</nowiki></code> and <code><nowiki>app.brijraj.in</nowiki></code>.

'''Net effect:''' every host that was actively depending on tornedo is now confirmed reaching <code><nowiki>ipa.tetrain.com</nowiki></code> with a live, verified Kerberos ticket. tornedo remains shut down (client's own action, not yet a formal decommission per Phase 3-5).

[[Category:Infrastructure Documentation]]