Editing Tornedo.tetrain.com Upgrade Plan (section)

== 12. Client-initiated shutdown of tornedo (2026-07-24) — live remediation of Group C hosts ==

Client shut down <code><nowiki>tornedo.tetrain.com</nowiki></code> ahead of the Phase 3 go/no-go in section 11 (no-go was flagged; client proceeded anyway). This turned the Group C readiness findings into a live outage check instead of a projection. Went through each affected host, diagnosed root cause, fixed, and verified with a live <code><nowiki>kinit -k</nowiki></code> (host keytab, no admin password involved) before calling it closed. Two distinct bug classes emerged beyond the known <code><nowiki>dns_lookup_kdc</nowiki></code>/SRV issue from section 9:

'''Bug class 1 — wrong <code><nowiki>ipa_domain</nowiki></code>:''' several hosts were enrolled with <code><nowiki>ipa_domain = tetrain.com</nowiki></code> instead of the real IPA domain <code><nowiki>tornedo.tetrain.com</nowiki></code> (confirmed via <code><nowiki>ipa.tetrain.com</nowiki></code>'s own config: realm/domain is <code><nowiki>TORNEDO.TETRAIN.COM</nowiki></code>/<code><nowiki>tornedo.tetrain.com</nowiki></code>, not <code><nowiki>tetrain.com</nowiki></code>). This sends SSSD's SRV lookups at a zone with no records (<code><nowiki>_ldap._tcp.tetrain.com</nowiki></code> — confirmed NXDOMAIN, authoritatively, at GoDaddy). Fix: correct <code><nowiki>ipa_domain</nowiki></code>, the <code><nowiki>[domain/...]</nowiki></code> section header, <code><nowiki>domains =</nowiki></code>, <code><nowiki>ipa_server</nowiki></code> static fallback, and krb5.conf's <code><nowiki>[realms]</nowiki></code> KDC/admin_server entries to point at <code><nowiki>ipa.tetrain.com</nowiki></code>. Turned <code><nowiki>dns_lookup_kdc</nowiki></code> on where it was off.

'''Bug class 2 — host firewall allowlist:''' some hosts run a default-DROP <code><nowiki>iptables OUTPUT</nowiki></code> policy with an explicit per-destination-IP allowlist that had tornedo's IP hardcoded but never got ipa.tetrain.com added. Correct DNS/krb5 config doesn't help if the packets never leave the host. Fix: mirror the existing tornedo ACCEPT rule for <code><nowiki>13.126.212.202</nowiki></code>, persist via <code><nowiki>iptables-save</nowiki></code>.

'''Bug class 3 (one host only) — broken local DNS resolver:''' <code><nowiki>mail.tetrain.com</nowiki></code> had entirely correct sssd/krb5 config, but its own <code><nowiki>/etc/resolv.conf</nowiki></code> lists itself (<code><nowiki>139.84.171.188</nowiki></code>, a local Zimbra-managed resolver) as the *first* nameserver, and that resolver returns a valid empty <code><nowiki>NOERROR</nowiki></code> for the SRV query — which glibc's resolver treats as final, never falling through to the other 4 working resolvers listed after it. This is why it was reported "confirmed closed" by the client yet was still found hammering tornedo almost exclusively in section 11's log check — the config genuinely was fine, DNS just silently lied. Fix: reordered <code><nowiki>resolv.conf</nowiki></code> to put a working recursive resolver first (local resolver left in place, just deprioritized).

'''Per-host outcome:'''

{| class="wikitable"
|-
! Host !! Bug !! Fix applied !! Verified
|-
| <code><nowiki>app-1.tetrain.com</nowiki></code> || wrong <code><nowiki>ipa_domain</nowiki></code> || config || live <code><nowiki>kinit</nowiki></code> ✓
|-
| <code><nowiki>server-relay-2.tetrain.com</nowiki></code> || wrong <code><nowiki>ipa_domain</nowiki></code> || config || live <code><nowiki>kinit</nowiki></code> ✓
|-
| <code><nowiki>mail.curiocrafts.com</nowiki></code> (aka <code><nowiki>mail.energoconstruction.com</nowiki></code>/<code><nowiki>mail.energoproducts.com</nowiki></code>) || wrong <code><nowiki>ipa_domain</nowiki></code> + firewall allowlist || config + iptables || live <code><nowiki>kinit</nowiki></code> ✓
|-
| <code><nowiki>mail.tetrain.com</nowiki></code> / <code><nowiki>zimbra-test.tetra.in</nowiki></code> || broken local DNS resolver order || resolv.conf reorder || live <code><nowiki>kinit</nowiki></code> ✓
|-
| <code><nowiki>app.brijraj.in</nowiki></code> || wrong <code><nowiki>ipa_domain</nowiki></code> + firewall allowlist || config + iptables || live <code><nowiki>kinit</nowiki></code> ✓
|-
| <code><nowiki>nic.tetra.in</nowiki></code> || none — already correct || — || live <code><nowiki>kinit</nowiki></code> ✓
|-
| <code><nowiki>server-relay.tetrain.com</nowiki></code> || none — already correct || — || live <code><nowiki>kinit</nowiki></code> ✓
|-
| <code><nowiki>tetra-demo.tetrain.com</nowiki></code> || none — already correct || — || live <code><nowiki>kinit</nowiki></code> ✓
|-
| <code><nowiki>cloud-storage.triburg.co.in</nowiki></code> || wrong <code><nowiki>ipa_domain</nowiki></code>, but masked by explicit <code><nowiki>dns_discovery_domain</nowiki></code> override — cosmetic only || none applied (optional cleanup later) || live <code><nowiki>kinit</nowiki></code> ✓
|-
| <code><nowiki>indorama-relay.tetrain.com</nowiki></code> || — || host deleted, dropped from scope || n/a
|}

All backups (<code><nowiki>*.bak.<timestamp></nowiki></code>) left in place on each host next to the files they replaced. Access notes: <code><nowiki>app-1</nowiki></code>/<code><nowiki>server-relay-2</nowiki></code>/<code><nowiki>nic.tetra.in</nowiki></code>/<code><nowiki>server-relay.tetrain.com</nowiki></code> reachable via the standard <code><nowiki>id_rsa_curio</nowiki></code> key; <code><nowiki>mail.curiocrafts.com</nowiki></code>/<code><nowiki>app.brijraj.in</nowiki></code>/ <code><nowiki>tetra-demo.tetrain.com</nowiki></code>/<code><nowiki>cloud-storage.triburg.co.in</nowiki></code> also via <code><nowiki>id_rsa_curio</nowiki></code> once tried against the right user (<code><nowiki>root</nowiki></code> or <code><nowiki>centos</nowiki></code>, <code><nowiki>HostKeyAlgorithms=+ssh-rsa</nowiki></code> needed for the older CentOS 6/7 boxes); <code><nowiki>mail.tetrain.com</nowiki></code> needed <code><nowiki>proxy_id_rsa</nowiki></code> as <code><nowiki>root</nowiki></code> — none of the fleet-standard keys worked there, and password auth is disabled for <code><nowiki>biswajit</nowiki></code> on both <code><nowiki>mail.tetrain.com</nowiki></code> and <code><nowiki>app.brijraj.in</nowiki></code>.

'''Net effect:''' every host that was actively depending on tornedo is now confirmed reaching <code><nowiki>ipa.tetrain.com</nowiki></code> with a live, verified Kerberos ticket. tornedo remains shut down (client's own action, not yet a formal decommission per Phase 3-5).

[[Category:Infrastructure Documentation]]