Editing Tornedo.tetrain.com Upgrade Plan (section)

== 9. Broader fleet SRV-cache sweep + new-client onboarding findings (2026-07-23) ==

=== 9.1 SRV cache sweep across 13 additional identified clients ===

Client provided 15 raw IPs to check. 2 couldn't be reached with any known key/user combo (<code><nowiki>139.84.174.24</nowiki></code>, <code><nowiki>65.20.79.182</nowiki></code> at the time — the latter turned out to be <code><nowiki>zurt-ison.mysecuremail.email</nowiki></code>, see 9.2). Of the 13 identified and reachable:

* 6 were stale (still cached on <code><nowiki>tornedo.tetrain.com</nowiki></code>'s IP) and successfully refreshed to <code><nowiki>ipa.tetrain.com</nowiki></code> via <code><nowiki>systemctl restart sssd</nowiki></code>.
* '''<code><nowiki>dr-mobilepedia.tetrain.com</nowiki></code>''': has <code><nowiki>dns_lookup_kdc = false</nowiki></code> in <code><nowiki>krb5.conf</nowiki></code> — SRV discovery disabled at the Kerberos level, won't fail over automatically no matter how long we wait. Needs a manual fix (set to <code><nowiki>true</nowiki></code>) before <code><nowiki>tornedo</nowiki></code> can be safely retired for this client.
* '''<code><nowiki>mail.tetrain.com</nowiki></code>''': uses itself as its own primary DNS resolver (<code><nowiki>nameserver 139.84.171.188</nowiki></code> = its own IP in <code><nowiki>/etc/resolv.conf</nowiki></code>) — a direct SRV <code><nowiki>dig</nowiki></code> from that box returns empty, meaning its own local resolver isn't serving our new records. Root cause not yet resolved — needs investigation into that box's own DNS/resolver setup.
* '''<code><nowiki>mbox2.mysecuremail.email</nowiki></code>''': <code><nowiki>sssd</nowiki></code> isn't persistently running (starts, runs ~60s, gets stopped) — confirmed via journal history this exact pattern predates today's work (same pattern on Jul 17), so '''not''' something we caused. Cause unknown, needs its own investigation before this client's cutover-readiness can be assessed.
* 4 of the 15 aren't IPA clients at all (no sssd running): <code><nowiki>zurt-sysnet.mysecuremail.email</nowiki></code>, <code><nowiki>mail.tetra.in</nowiki></code>, <code><nowiki>single.tetra.in</nowiki></code>, <code><nowiki>email.rahulchaudhry.com</nowiki></code>.

'''Takeaway confirmed by this sweep''': caching lag is real and expected (fixed by a simple sssd restart), but there are also genuine per-client configuration gaps that SRV records alone don't fix. Both <code><nowiki>dr-mobilepedia</nowiki></code> and <code><nowiki>mail.tetrain.com</nowiki></code> need to be resolved individually before <code><nowiki>tornedo.tetrain.com</nowiki></code> can be safely retired — a blanket "wait for the soak period" isn't sufficient for these two.

=== 9.2 New client onboarding — systemic gap found on Debian/Ubuntu clients ===

<code><nowiki>65.20.79.182</nowiki></code> turned out to be a '''new''' server being onboarded (<code><nowiki>zurt-ison.mysecuremail.email</nowiki></code>, not yet enrolled) — client is onboarding new servers against <code><nowiki>ipa.tetrain.com</nowiki></code> directly going forward, per Section 8.9's guidance. Two distinct issues hit during this one onboarding, both worth carrying forward as standard steps for future Debian/Ubuntu (not RHEL-family) clients:

# '''<code><nowiki>/etc/hosts</nowiki></code> had the hostname's two label halves swapped''' (<code><nowiki>ison-zurt</nowiki></code> vs. actual <code><nowiki>zurt-ison</nowiki></code>) *and* pointed at loopback (<code><nowiki>127.0.1.1</nowiki></code>) — meant <code><nowiki>hostname -f</nowiki></code> failed outright with a Python traceback in <code><nowiki>ipa-client-install</nowiki></code> (<code><nowiki>socket.gaierror</nowiki></code>) before even reaching the normal loopback-resolution failure mode we'd seen before. Fixed by correcting <code><nowiki>/etc/hosts</nowiki></code> to the real name + real routable IP, same pattern as the <code><nowiki>ipa8</nowiki></code>/<code><nowiki>ipa.tetrain.com</nowiki></code> fixes.
# '''After successful enrollment, login still failed for every domain user, with zero SSSD log entries at all''' — traced to <code><nowiki>pam_sss.so</nowiki></code> being completely absent from <code><nowiki>/etc/pam.d/common-auth</nowiki></code> and <code><nowiki>/etc/pam.d/common-account</nowiki></code>. Root cause: Debian/Ubuntu manages the PAM stack via <code><nowiki>pam-auth-update</nowiki></code>, a *separate* mechanism from what <code><nowiki>ipa-client-install</nowiki></code> configures directly (<code><nowiki>authselect</nowiki></code> on RHEL-family systems, which reliably wires up <code><nowiki>pam_sss</nowiki></code> automatically) — on this Debian client, the SSS profile was never enabled. Without it, <code><nowiki>pam_unix.so</nowiki></code> fails for any non-local account and falls straight through to <code><nowiki>pam_deny.so</nowiki></code>, rejecting the login before SSSD is ever consulted — this looks identical to "wrong password" from the client's perspective, indistinguishable without checking the PAM stack directly.

'''Fix''': <code><nowiki>pam-auth-update --force --enable sss</nowiki></code> (the <code><nowiki>--force</nowiki></code> was needed because the image's default <code><nowiki>nullok</nowiki></code> customization in <code><nowiki>common-auth</nowiki></code> counted as a "local modification" blocking the normal non-forced update).

'''Action item for future Debian/Ubuntu onboarding''': after <code><nowiki>ipa-client-install</nowiki></code> completes on any Debian-family box, always run <code><nowiki>pam-auth-update --force --enable sss</nowiki></code> and verify with <code><nowiki>grep pam_sss /etc/pam.d/common-auth /etc/pam.d/common-account</nowiki></code> *before* considering the onboarding done — don't wait for a real user to hit the failure to discover this gap.

=== 9.3 Two more onboardings (<code><nowiki>zurt-sysnet</nowiki></code>, <code><nowiki>email.rahulchaudhry.com</nowiki></code>) — same hosts-file bug, ===

plus two *new* systemic gaps found

'''<code><nowiki>zurt-sysnet.mysecuremail.email</nowiki></code>''': identical swapped-hostname + loopback <code><nowiki>/etc/hosts</nowiki></code> bug as <code><nowiki>zurt-ison</nowiki></code> (labels reversed: <code><nowiki>sysnet-zurt</nowiki></code> vs. actual <code><nowiki>zurt-sysnet</nowiki></code>) — same fix applied (correct name + real IP <code><nowiki>65.20.85.22</nowiki></code>, <code><nowiki>manage_etc_hosts: false</nowiki></code>).

'''<code><nowiki>email.rahulchaudhry.com</nowiki></code>''' (Ubuntu 25.10) — three separate issues in one onboarding:

# <code><nowiki>ipa-client-install</nowiki></code> failed at the time-sync step: <code><nowiki>chrony.service</nowiki></code> not found — chrony wasn't installed on this image at all, even though the installer tried to configure it. Fixed with <code><nowiki>apt-get install -y chrony</nowiki></code> before retrying. Worth checking chrony is present *before* running <code><nowiki>ipa-client-install</nowiki></code> on future minimal/stripped images.
# <code><nowiki>--mkhomedir</nowiki></code> didn't actually create home directories despite being passed — same Debian/pam-auth-update gap as the <code><nowiki>pam_sss</nowiki></code> one, but for a *different* profile: <code><nowiki>pam_mkhomedir.so</nowiki></code> was missing from <code><nowiki>common-session</nowiki></code>. Fix: <code><nowiki>pam-auth-update --force --enable mkhomedir</nowiki></code> (in addition to <code><nowiki>--enable sss</nowiki></code>, not instead of).
# '''New, distinct finding''': sudo denied for a user (<code><nowiki>biswajit</nowiki></code>) known to have valid, working IPA sudo rules (<code><nowiki>sudorule1</nowiki></code>, Host category: all, confirmed via <code><nowiki>ipa sudorule-show</nowiki></code>) — with zero SSSD sudo-responder log activity for any of it, even after confirming <code><nowiki>libsss-sudo</nowiki></code> was installed, the shared library existed on disk, and the SSSD sudo socket was present and correctly permissioned. Root cause: **Ubuntu 25.10 ships <code><nowiki>sudo-rs</nowiki></code> (a Rust reimplementation of sudo) as the *default* <code><nowiki>/usr/bin/sudo</nowiki></code>**, and this version (<code><nowiki>0.2.8</nowiki></code>) does not appear to support SSSD/IPA-driven sudo rules — it silently falls back to local-file-only sudoers evaluation, giving a plausible-looking (if oddly-worded — custom "I'm sorry, I'm afraid I can't do that" message) denial with no diagnostic trail on the SSSD side at all. Classic GNU sudo (<code><nowiki>1.9.17p2</nowiki></code>) was already installed alongside it, just not the active alternative.

'''Fix''': switch the <code><nowiki>sudo</nowiki></code> alternative to the classic binary (which the <code><nowiki>sudo</nowiki></code> package installs at <code><nowiki>/usr/bin/sudo.ws</nowiki></code> on this Ubuntu convention, not the plain <code><nowiki>/usr/bin/sudo</nowiki></code> path — no <code><nowiki>dpkg-divert</nowiki></code> involved, just an incompletely-registered alternatives group):

<pre>
<nowiki>
   update-alternatives --install /usr/bin/sudo sudo /usr/bin/sudo.ws 50
   update-alternatives --set sudo /usr/bin/sudo.ws
</nowiki>
</pre>

Verified via <code><nowiki>sudo --version</nowiki></code> (no longer reports <code><nowiki>sudo-rs</nowiki></code>) and <code><nowiki>sudo -l -U biswajit</nowiki></code> (correctly returned <code><nowiki>(root) NOPASSWD: ALL</nowiki></code>, matching <code><nowiki>sudorule1</nowiki></code>).

'''Updated action item for Ubuntu 25.10+ onboarding specifically''': in addition to the <code><nowiki>pam-auth-update --force --enable sss</nowiki></code> step, also run <code><nowiki>--enable mkhomedir</nowiki></code>, and check <code><nowiki>sudo --version</nowiki></code> — if it reports <code><nowiki>sudo-rs</nowiki></code>, switch to classic sudo via the <code><nowiki>update-alternatives</nowiki></code> commands above *before* considering the box's sudo access functional for any IPA-managed user.

=== 9.4 Second CSV sweep (14 hosts) — 8 refreshed cleanly, 2 needed a *different* fix, 4 unreachable ===

Client provided a Linode-hosted list of 14 hostnames+IPs. 4 not accessible with known keys (<code><nowiki>server-relay-2.tetrain.com-new</nowiki></code>, <code><nowiki>access.tetrain.com</nowiki></code>, <code><nowiki>otrs.tetrain.com</nowiki></code>, <code><nowiki>nagios.tetrain.com</nowiki></code> — need different credentials). Of the 10 reachable, 8 refreshed cleanly via the standard <code><nowiki>systemctl restart sssd</nowiki></code> approach (<code><nowiki>zm.wbtea.com</nowiki></code>, <code><nowiki>mail.sysnetglobal.com</nowiki></code>, <code><nowiki>fcra.tetra.in</nowiki></code>, <code><nowiki>list.tetra-india.in</nowiki></code>, <code><nowiki>mobile.sceh.net</nowiki></code>, <code><nowiki>ihelpdesk-belpoc.tetra.in</nowiki></code>, <code><nowiki>nic.tetra.in</nowiki></code>, <code><nowiki>wiki.tetrain.com</nowiki></code>).

'''2 needed real investigation — <code><nowiki>relay.isgec.com</nowiki></code> and <code><nowiki>mail.sceh.net</nowiki></code>.''' Neither had <code><nowiki>dns_lookup_kdc = false</nowiki></code> this time (that got fixed, still didn't switch) — the actual cause was different and more interesting: both had <code><nowiki>dns_discovery_domain = tetrain.com</nowiki></code> in <code><nowiki>sssd.conf</nowiki></code>, instead of <code><nowiki>tornedo.tetrain.com</nowiki></code> like every other client. This setting controls what domain suffix SSSD appends when constructing SRV record names to query — with it set to the shorter <code><nowiki>tetrain.com</nowiki></code>, these clients were silently querying <code><nowiki>_ldap._tcp.tetrain.com</nowiki></code>/<code><nowiki>_kerberos._tcp.tetrain.com</nowiki></code> (which don't exist) instead of our actual published records under <code><nowiki>_ldap._tcp.tornedo.tetrain.com</nowiki></code>. A direct manual <code><nowiki>dig</nowiki></code> with the full correct name succeeded (masking the problem during initial diagnosis), while SSSD's own automatic discovery never found anything. Confirmed via full <code><nowiki>sssd.conf</nowiki></code> inspection: both had <code><nowiki>ipa_domain = tetrain.com</nowiki></code> too (their local IPA "domain" section is literally named <code><nowiki>[domain/tetrain.com]</nowiki></code>, not <code><nowiki>[domain/tornedo.tetrain.com]</nowiki></code>) — left <code><nowiki>ipa_domain</nowiki></code> untouched since it's tied to other working config on long-established production clients; only <code><nowiki>dns_discovery_domain</nowiki></code> was corrected. Both switched to <code><nowiki>ipa.tetrain.com</nowiki></code> immediately after.

'''There are now two distinct, confirmed root causes for "SRV records published but a client still won't discover the new server," both worth checking on any client that doesn't switch after a plain <code><nowiki>sssd</nowiki></code> restart:'''

# <code><nowiki>dns_lookup_kdc = false</nowiki></code> in <code><nowiki>/etc/krb5.conf</nowiki></code> (found on <code><nowiki>dr-mobilepedia.tetrain.com</nowiki></code>, fix not yet re-verified after the flag change — worth confirming).
# <code><nowiki>dns_discovery_domain</nowiki></code> set to a shorter/different suffix than <code><nowiki>tornedo.tetrain.com</nowiki></code> in <code><nowiki>sssd.conf</nowiki></code> — '''confirmed on 3 hosts now''': <code><nowiki>relay.isgec.com</nowiki></code>, <code><nowiki>mail.sceh.net</nowiki></code>, <code><nowiki>nextcloud.tetra.in</nowiki></code>. All fixed the same way (correct the value, restart sssd — no cache clear needed, this isn't a caching issue, it's querying the wrong SRV record names entirely). Common enough now to check *first*, before assuming a cache/timing issue, on any host stuck on the old IP even after an aggressive <code><nowiki>sss_cache -E</nowiki></code> + full stop/start.

A host showing the "stuck on old IP even after restart" symptom needs both checked, not just one — they're independent settings that can each cause the identical-looking symptom.

=== 9.5 Additional hosts checked/fixed outside the CSV sweeps ===

* <code><nowiki>li2028-191.members.linode.com</nowiki></code> ("Nagios Server" per its IPA description) — switched successfully via <code><nowiki>/etc/init.d/sssd restart</nowiki></code> (no systemd on this box). Separately flagged: running '''CentOS 6.8''', EOL since November 2020 — outside the original 31-server fleet scope, significantly more urgent than any CentOS 7 box found so far if/when OS remediation is revisited.
* <code><nowiki>nextcloud.tetra.in</nowiki></code> — same <code><nowiki>dns_discovery_domain</nowiki></code> bug as above, fixed and confirmed.