Editing Tornedo.tetrain.com Runbook (section)

== Phase 8 — Client remediation reference (recurring bug classes found during fleet sweep) ==

Not a linear sequence — this is a diagnostic reference for '''any client that doesn't switch to the new server after the Phase 5 SRV records go live''', encountered across ~50 hosts during the actual sweep. Check in this order:

'''8.1 — Simple caching lag (try first, cheapest check):'''

<pre>
<nowiki>
systemctl restart sssd            # or: /etc/init.d/sssd restart  (no-systemd boxes)
</nowiki>
</pre>

Resolves the majority of "stuck on old IP" cases by itself.

'''8.2 — <code><nowiki>dns_lookup_kdc = false</nowiki></code> in <code><nowiki>/etc/krb5.conf</nowiki></code>:''' SRV discovery is disabled at the Kerberos level entirely — no amount of waiting fixes this.

<pre>
<nowiki>
sed -i 's/dns_lookup_kdc = false/dns_lookup_kdc = true/' /etc/krb5.conf
systemctl restart sssd
</nowiki>
</pre>

'''8.3 — Wrong <code><nowiki>dns_discovery_domain</nowiki></code> in <code><nowiki>sssd.conf</nowiki></code>:''' If it's set to a *shorter* suffix than the real IPA domain (e.g. <code><nowiki>tetrain.com</nowiki></code> instead of <code><nowiki>tornedo.tetrain.com</nowiki></code>), SSSD constructs and queries the wrong SRV record names entirely (<code><nowiki>_ldap._tcp.tetrain.com</nowiki></code>, which doesn't exist) — silent failure, <code><nowiki>dig</nowiki></code> with the *correct* full name still works manually, which masks this during casual diagnosis. Confirmed on 3+ hosts.

<pre>
<nowiki>
# in sssd.conf, correct only dns_discovery_domain — leave ipa_domain alone if other config
# depends on it (some long-established clients have ipa_domain = tetrain.com tied to other
# working config; don't touch it, just fix dns_discovery_domain):
sed -i 's/dns_discovery_domain = tetrain.com/dns_discovery_domain = tornedo.tetrain.com/' /etc/sssd/sssd.conf
systemctl restart sssd
</nowiki>
</pre>

A host stuck on the old IP even after an aggressive <code><nowiki>sss_cache -E</nowiki></code> + full stop/start needs '''both''' 8.2 and 8.3 checked — they're independent settings causing an identical-looking symptom.

'''8.4 — Wrong <code><nowiki>ipa_domain</nowiki></code> (distinct from 8.3, found during a later live-outage remediation):'''

<pre>
<nowiki>
# correct ALL of: [domain/...] section header, domains=, ipa_domain, ipa_server static
# fallback in sssd.conf, and [realms] kdc/admin_server in krb5.conf — all need to agree on
# the real domain, tornedo.tetrain.com, not a shortened alias
</nowiki>
</pre>

'''8.5 — Host firewall allowlist (packets never leave the host):''' Some hosts run a default-DROP <code><nowiki>iptables OUTPUT</nowiki></code> policy with an explicit per-destination-IP allowlist that has the old server's IP hardcoded but never got the new one added — correct DNS/krb5 config doesn't help if the packets are dropped before leaving the box.

<pre>
<nowiki>
iptables -A OUTPUT -d <ipa.tetrain.com-IP> -j ACCEPT   # mirror whatever rule pattern the
                                                         # old server's IP already has
iptables-save > /etc/sysconfig/iptables   # or your distro's persistence mechanism
</nowiki>
</pre>

'''8.6 — Local DNS resolver lying (one host, but worth checking anywhere self-hosting DNS):''' A host whose own <code><nowiki>/etc/resolv.conf</nowiki></code> lists itself as the *first* nameserver can return a valid empty <code><nowiki>NOERROR</nowiki></code> for the SRV query — glibc's resolver treats that as final and never falls through to the other working resolvers listed after it, even though they'd succeed. Config looks completely correct; DNS is just silently lying.

<pre>
<nowiki>
# reorder /etc/resolv.conf: put a working recursive resolver FIRST, keep the local one but
# deprioritized (don't remove it if other local config depends on it)
</nowiki>
</pre>

'''8.7 — Debian/Ubuntu-specific: <code><nowiki>ipa-client-install</nowiki></code> doesn't wire up PAM the way RHEL does.''' RHEL-family systems use <code><nowiki>authselect</nowiki></code>, which <code><nowiki>ipa-client-install</nowiki></code> configures automatically. Debian/Ubuntu use <code><nowiki>pam-auth-update</nowiki></code>, a '''separate''' mechanism <code><nowiki>ipa-client-install</nowiki></code> doesn't touch — enrollment can report success while login still fails completely with zero SSSD log activity (looks identical to "wrong password").

<pre>
<nowiki>
# after ipa-client-install completes on any Debian-family box, always:
pam-auth-update --force --enable sss
pam-auth-update --force --enable mkhomedir   # if --mkhomedir was passed to ipa-client-install
                                               # and doesn't seem to be creating home dirs
# --force is needed if the image ships a "local modification" in common-auth (e.g. a custom
# nullok tweak) that would otherwise block the non-forced update
grep pam_sss /etc/pam.d/common-auth /etc/pam.d/common-account   # verify before calling it done
</nowiki>
</pre>

'''8.8 — <code><nowiki>/etc/hosts</nowiki></code> with swapped hostname labels + loopback (recurring on new-onboarding, not just the migration):'''

<pre>
<nowiki>
# e.g. "sysnet-zurt" written where the real hostname is "zurt-sysnet", AND pointed at 127.0.1.1
# — ipa-client-install fails outright with a Python socket.gaierror before even reaching the
# usual loopback-resolution failure. Fix: correct hostname + real routable IP, same as Phase
# 2.2/3.1, plus manage_etc_hosts: false in cloud-init config.
</nowiki>
</pre>

'''8.9 — Ubuntu 25.10+: <code><nowiki>sudo-rs</nowiki></code> silently ignores IPA sudo rules.''' Symptom: a user with a confirmed-valid IPA sudo rule (<code><nowiki>ipa sudorule-show <rule></nowiki></code>) gets denied, with '''zero''' SSSD sudo-responder log activity even though <code><nowiki>libsss-sudo</nowiki></code> is installed and the SSSD sudo socket exists and is correctly permissioned. Root cause: Ubuntu 25.10 ships <code><nowiki>sudo-rs</nowiki></code> (Rust reimplementation) as the *default* <code><nowiki>/usr/bin/sudo</nowiki></code>, and it doesn't support SSSD/IPA-driven sudo rules — falls back to local-file-only sudoers evaluation with a plausible-looking denial and no diagnostic trail. Classic GNU sudo is usually already installed alongside it, just not the active alternative (registered at <code><nowiki>/usr/bin/sudo.ws</nowiki></code> on this Ubuntu convention, no <code><nowiki>dpkg-divert</nowiki></code> involved):

<pre>
<nowiki>
update-alternatives --install /usr/bin/sudo sudo /usr/bin/sudo.ws 50
update-alternatives --set sudo /usr/bin/sudo.ws
sudo --version         # confirm it no longer reports "sudo-rs"
sudo -l -U <user>       # confirm it now shows the real IPA-sourced rule
</nowiki>
</pre>

'''8.10 — Missing <code><nowiki>chrony</nowiki></code> on minimal/stripped images:''' <code><nowiki>ipa-client-install</nowiki></code> tries to configure <code><nowiki>chrony</nowiki></code> for time sync but doesn't install it if missing — fails outright at that step.

<pre>
<nowiki>
apt-get install -y chrony    # or dnf/yum equivalent — before retrying ipa-client-install
</nowiki>
</pre>

----