Editing Tornedo.tetrain.com Upgrade Plan (section)

== 8. STRATEGY CHANGE (2026-07-23) — pivoted to replica-and-promote (Strategy A) ==

Sections 1-7 above described the in-place <code><nowiki>leapp</nowiki></code> chain (7→Stream8→Stream9→Stream10), pursued per earlier direction. '''<code><nowiki>leapp preupgrade</nowiki></code> itself blocked this''': one of its 5 inhibitors was <code><nowiki>ipa-server does not support in-place upgrade</nowiki></code> — Red Hat's own tooling refuses to leapp-upgrade a FreeIPA/IdM server, full stop, and points to a separate IdM migration guide instead. This section supersedes the in-place approach; Sections 1-7 are kept for the diagnostic history (cert-renewal fix, hostname fix, RPM duplicate findings — all still valid/useful) but the upgrade *mechanism* is now replica-and-promote, not leapp.

=== 8.1 Chosen approach ===

* Clients (all 110 enrolled hosts) reference the server exclusively by DNS hostname (<code><nowiki>tornedo.tetrain.com</nowiki></code>) in <code><nowiki>sssd.conf</nowiki></code>/<code><nowiki>krb5.conf</nowiki></code> — confirmed by inspecting a real enrolled client (<code><nowiki>nic.tetra.in</nowiki></code>), never a raw IP. This makes an identity handoff (new server assumes the same hostname via DNS repoint at cutover) viable with '''zero client-side reconfiguration''' — the only alternative (repoint 110 clients, ~80 of which are other companies' servers) was ruled out as an operational non-starter.
* '''Two-hop replica bridge''', not a direct jump: a direct replica attempt from the old server (IPA 4.6, 2017-era Dogtag/NSS stack) straight to a new AlmaLinux 9 box (IPA 4.13) failed the CA-cloning step with <code><nowiki>403 Forbidden</nowiki></code> on the CA's REST API — a genuine protocol/auth incompatibility across ~7 major IPA versions, not a config error (confirmed: not a network issue, not clock skew). Same "don't skip versions" principle as leapp, applied to IPA replica compatibility. Fix: bridge through an intermediate replica on a version roughly in between.

=== 8.2 Topology ===

<pre>
<nowiki>
tornedo.tetrain.com (CentOS 7, IPA 4.6)  [ORIGINAL — untouched in terms of production service]
        │  replica (successful)
        ▼
ipa8.tetrain.com (Rocky Linux 8.10, IPA 4.9.13, 65.20.78.116)  [INTERMEDIATE BRIDGE — temporary]
        │  replica (next step)
        ▼
ipa.tetrain.com (AlmaLinux 9.7, IPA 4.13.1, 13.126.212.202)  [FINAL TARGET]
</nowiki>
</pre>

=== 8.3 Status: intermediate replica (<code><nowiki>ipa8.tetrain.com</nowiki></code>) — DONE, validated ===

Built from scratch, full authorization given by client to build without per-step confirmation (cutover explicitly excluded — still gated). Real obstacles hit and resolved, in order:

# Wrong SSH user/key initially assumed — actual: <code><nowiki>root</nowiki></code> / <code><nowiki>proxy_id_rsa</nowiki></code> (not <code><nowiki>id_rsa_curio</nowiki></code>).
# <code><nowiki>firewalld</nowiki></code> active locally (unlike the other two boxes) — opened freeipa-ldap/freeipa-ldaps/dns/http/https/kerberos services + 464 + 8080/8443 explicitly.
# EL8 packages IPA via a DNF '''module''' (<code><nowiki>idm:DL1</nowiki></code>), not a plain package — needed <code><nowiki>dnf module enable idm:DL1</nowiki></code> before <code><nowiki>dnf install @idm:DL1/server</nowiki></code> would resolve.
# Cloud-init's default <code><nowiki>/etc/hosts</nowiki></code> template mapped the box's own hostname to '''both''' <code><nowiki>127.0.0.1</nowiki></code> and <code><nowiki>::1</nowiki></code> — IPA's installer explicitly refuses a hostname that resolves to loopback. Fixed by rewriting <code><nowiki>/etc/hosts</nowiki></code> to map the hostname to its real IP, and disabled <code><nowiki>manage_etc_hosts</nowiki></code> so cloud-init won't revert it on a future reboot.
# Reverse-DNS mismatch for the '''old''' server (same AWS-default-PTR issue as Section 3.1), recurring here since it's a property of the old server, not the new one — same fix, an <code><nowiki>/etc/hosts</nowiki></code> entry for <code><nowiki>tornedo.tetrain.com</nowiki></code> on this box too.
# <code><nowiki>ipa-replica-install</nowiki></code>'s <code><nowiki>check_creds</nowiki></code> needs actual admin Kerberos credentials, not just the host keytab from enrollment — client ran <code><nowiki>kinit admin</nowiki></code> + the install command together in their own session so the admin password never passed through this session.
# Each <code><nowiki>ipa-server-install --uninstall</nowiki></code> cleanup (needed after every failed attempt above) also wipes client enrollment, requiring a fresh OTP each retry (<code><nowiki>ipa host-disable</nowiki></code> + <code><nowiki>ipa host-mod --random</nowiki></code> on the source server, since the host object already exists after the first attempt).

Once all of the above were resolved, <code><nowiki>ipa-replica-install --setup-ca --unattended</nowiki></code> completed successfully. Two non-fatal warnings during install (<code><nowiki>acmeIPAServerCert</nowiki></code> profile import failure, <code><nowiki>dnaHostname</nowiki></code> lookup timeout) — both are known/common transient issues during replica install; resolved by running <code><nowiki>ipa-server-upgrade</nowiki></code> immediately after, which completed successfully and re-migrated certificate profiles.

'''Validated''': <code><nowiki>ipactl status</nowiki></code> — all 7 services RUNNING (Directory Service, krb5kdc, kadmin, httpd, ipa-custodia, pki-tomcatd, ipa-otpd). All certs <code><nowiki>MONITORING</nowiki></code>, no errors. LDAP search via LDAPI succeeds. Kerberos ticket issuance confirmed working.

=== 8.4 Second hop — <code><nowiki>ipa.tetrain.com</nowiki></code> promoted from <code><nowiki>ipa8.tetrain.com</nowiki></code> — DONE, validated ===

Networking between <code><nowiki>ipa8.tetrain.com</nowiki></code> and <code><nowiki>ipa.tetrain.com</nowiki></code> was already fully open both directions on first check (389/636/88/464/80/443/8080/8443) — no security-group work needed for this pair, unlike the first hop.

Obstacles hit and resolved (some recurring from 8.3, some new):

# Stale IPA '''master'''/topology entry for <code><nowiki>ipa.tetrain.com</nowiki></code> — leftover from the original failed direct-to-<code><nowiki>tornedo</nowiki></code> attempt (which got as far as <code><nowiki>[33/40]: adding master entry</nowiki></code> before failing at CA setup). <code><nowiki>host-disable</nowiki></code>/<code><nowiki>host-mod</nowiki></code> couldn't touch it (<code><nowiki>invalid 'hostname': An IPA master host cannot be deleted or disabled</nowiki></code>); needed <code><nowiki>ipa server-del ipa.tetrain.com --force</nowiki></code> instead. '''Recurred 3 times''' across this hop's retries — each failed attempt that got past "adding master entry" left another one, so this became the standard first step of every cleanup cycle.
# My own cleanup mistake: <code><nowiki>systemctl disable pki-tomcatd@pki-tomcat.service</nowiki></code> (to kill an orphaned process) auto-removed the now-empty <code><nowiki>/etc/systemd/system/pki-tomcatd.target.wants/</nowiki></code> directory as a systemd side effect, which then broke the *next* install attempt's symlink creation (<code><nowiki>FileNotFoundError</nowiki></code>). Fixed with <code><nowiki>mkdir -p</nowiki></code> to recreate it. Lesson: prefer <code><nowiki>systemctl stop</nowiki></code> + <code><nowiki>pkill</nowiki></code> over <code><nowiki>disable</nowiki></code> for ad-hoc orphan cleanup mid-troubleshooting.
# Orphaned <code><nowiki>ns-slapd</nowiki></code>/<code><nowiki>pki-tomcatd</nowiki></code> processes recurred after multiple failed attempts — <code><nowiki>ipa-server-install --uninstall</nowiki></code> repeatedly failed to actually stop them despite reporting success; needed manual <code><nowiki>systemctl stop</nowiki></code> + <code><nowiki>pkill -9 ns-slapd</nowiki></code> each time, checked via <code><nowiki>ps aux</nowiki></code> before every retry.
# <code><nowiki>ldap_add: Already exists (68)</nowiki></code> on "adding master entry" — same stale-topology-entry class as #1, same fix (<code><nowiki>ipa server-del --force</nowiki></code>).
# '''Root cause of the actual CA failure''' (as opposed to the cleanup-artifact failures above): this box has 1.6G RAM and '''zero swap'''. The CA database init step spawns a JVM (<code><nowiki>ca-db-init</nowiki></code>) that failed silently — no debug log, no catalina log output, exit code 255, no kernel OOM-killer entry (the JVM fails to reserve heap *before* its own logging initializes, so nothing gets logged and the kernel OOM-killer never needs to intervene). Contrast with <code><nowiki>ipa8.tetrain.com</nowiki></code>, which has 6.2G swap and sailed through the identical step without issue. '''Fixed by adding a 4G swap file''' (<code><nowiki>fallocate</nowiki></code>+<code><nowiki>mkswap</nowiki></code>+<code><nowiki>swapon</nowiki></code>, persisted in <code><nowiki>/etc/fstab</nowiki></code>) — confirmed swap was actively used (140Mi) during the successful run. (One hiccup: the swapfile briefly vanished — client was independently trying to set one up at the same time and it seems to have gotten removed; recreated once and it stuck.)
# Declined to use the client's offer to hand over the actual <code><nowiki>admin</nowiki></code> Kerberos password directly in chat for <code><nowiki>kinit</nowiki></code> — held the line consistent with not handling account passwords, even under explicit repeated request. Client continued running <code><nowiki>kinit admin</nowiki></code> + the install command themselves each time instead.

'''<code><nowiki>ipa-replica-install --setup-ca --skip-mem-check --unattended</nowiki></code> completed successfully.''' Followed immediately by <code><nowiki>ipa-server-upgrade</nowiki></code> (same precautionary step as the first hop) — successful. '''Validated''': all 7 services RUNNING, all certs <code><nowiki>MONITORING</nowiki></code> with no errors, LDAP search via LDAPI succeeds, swap confirmed in active use.

=== 8.5 Current topology (both hops complete) ===

<pre>
<nowiki>
tornedo.tetrain.com (CentOS 7, IPA 4.6)              [ORIGINAL — still production-serving]
        │ replica ✓
        ▼
ipa8.tetrain.com (Rocky Linux 8.10, IPA 4.9.13)      [INTERMEDIATE BRIDGE — temporary]
        │ replica ✓
        ▼
ipa.tetrain.com (AlmaLinux 9.7, IPA 4.13.1)          [FINAL TARGET — built, healthy, not yet live]
</nowiki>
</pre>

=== 8.6 Full validation — DONE ===

* '''Kerberos ticket issuance''': <code><nowiki>kinit -kt /etc/krb5.keytab host/ipa.tetrain.com</nowiki></code> on the box itself issued a valid <code><nowiki>krbtgt/TORNEDO.TETRAIN.COM</nowiki></code> ticket — KDC fully operational.
* '''Full LDAP+API roundtrip''': <code><nowiki>ipa user-show admin</nowiki></code> returned complete, correct user data — proves 389-ds + httpd + the IPA API framework all work together correctly on this server.
* '''Web UI''': <code><nowiki>https://ipa.tetrain.com/ipa/ui/</nowiki></code> responds <code><nowiki>HTTP 200</nowiki></code>.
* '''CA issuance''': implicitly proven — the certs from the install itself (fresh <code><nowiki>2028-07-23</nowiki></code> expiries) were issued live by this server's own CA during setup, not copied from elsewhere.
* '''Real external client reachability''': tested from <code><nowiki>koha.tetra.in</nowiki></code> (an actual production server already enrolled against the *old* <code><nowiki>tornedo.tetrain.com</nowiki></code>) — all 6 ports (389/636/88/464/80/443) open to the new server, and an ad-hoc <code><nowiki>ldapsearch</nowiki></code> returned the correct naming contexts (<code><nowiki>dc=tornedo,dc=tetrain,dc=com</nowiki></code>, <code><nowiki>o=ipaca</nowiki></code>), matching the domain exactly. This is the closest proxy available for "will real clients actually be able to use this server" without touching any production client's persistent config.

<code><nowiki>ipa.tetrain.com</nowiki></code> is a fully healthy, fully validated replica at this point.

=== 8.7 Intermediate bridge (<code><nowiki>ipa8.tetrain.com</nowiki></code>) decommissioned — DONE ===

Attempting <code><nowiki>ipa server-del ipa8.tetrain.com --force</nowiki></code> directly failed with a topology-disconnect error: '''<code><nowiki>ipa8</nowiki></code> was the only link connecting <code><nowiki>tornedo.tetrain.com</nowiki></code> and <code><nowiki>ipa.tetrain.com</nowiki></code>''' — removing it would have split the topology into two disconnected islands with no path to sync future changes between them. IPA's own topology check caught this correctly rather than letting it happen silently. Fixed by adding a direct segment between the two remaining servers first, for both suffixes:

<pre>
<nowiki>
ipa topologysegment-add domain --leftnode=tornedo.tetrain.com --rightnode=ipa.tetrain.com
ipa topologysegment-add ca --leftnode=tornedo.tetrain.com --rightnode=ipa.tetrain.com
</nowiki>
</pre>

Verified both new agreements replicating successfully before retrying. <code><nowiki>ipa server-del ipa8.tetrain.com --force</nowiki></code> then completed cleanly.

'''Post-removal validation''': all 7 services RUNNING, all 9 certs <code><nowiki>MONITORING</nowiki></code> (no errors — ipa-custodia's renewal role wasn't disrupted despite <code><nowiki>ipa8</nowiki></code> having been its install-time "master peer"), <code><nowiki>ipa-custodia.service</nowiki></code> itself active/healthy, full LDAP+API roundtrip still works, and replication agreements now show only the direct <code><nowiki>tornedo.tetrain.com</nowiki></code> ↔ <code><nowiki>ipa.tetrain.com</nowiki></code> links — no remaining references to <code><nowiki>ipa8</nowiki></code> anywhere.

'''The <code><nowiki>ipa8.tetrain.com</nowiki></code> VM is now safe to shut down''' — cleanly removed from the topology first, so no broken replication agreements or stale data left behind.

=== 8.8 Current topology (bridge removed, two-server domain) ===

<pre>
<nowiki>
tornedo.tetrain.com (CentOS 7, IPA 4.6)      [ORIGINAL — still production-serving]
        ⇄ direct replica (domain + ca)
ipa.tetrain.com (AlmaLinux 9, IPA 4.13.1)    [FINAL TARGET — healthy, validated, not yet live]
</nowiki>
</pre>

=== 8.9 Cutover mechanism — corrected (SRV-based, not IP/identity handoff) ===

Checked before starting cutover: '''no DNS SRV records exist for this domain today''' — every client relies purely on the literal hostname <code><nowiki>tornedo.tetrain.com</nowiki></code> (confirmed via <code><nowiki>dig</nowiki></code>, all empty). This actually simplifies cutover versus the original IP-reassignment idea from Section 1: since <code><nowiki>ipa.tetrain.com</nowiki></code> has its own distinct identity (own cert, own Kerberos principals), making it impersonate <code><nowiki>tornedo.tetrain.com</nowiki></code>'s exact hostname would require fragile, not-well-supported FreeIPA identity renaming. Instead, publish real SRV records pointing at <code><nowiki>ipa.tetrain.com</nowiki></code> by its own name — <code><nowiki>sssd.conf</nowiki></code>'s <code><nowiki>ipa_server = _srv_, tornedo.tetrain.com</nowiki></code> already tries SRV lookup first and only falls back to the literal name if that fails, so once real records exist, clients will discover and correctly validate against <code><nowiki>ipa.tetrain.com</nowiki></code> without any client-side config changes and without touching <code><nowiki>tornedo.tetrain.com</nowiki></code>'s A record at all (which stays up as a fallback for any client not using SRV discovery).

Records to publish (plain client-discovery SRV records — distinct from the AD-trust-style <code><nowiki>_msdcs.</nowiki></code> records <code><nowiki>ipa-adtrust-install</nowiki></code> prints, which are for actual Windows AD interop, not needed here):

<pre>
<nowiki>
_ldap._tcp.tornedo.tetrain.com.            SRV 0 100 389  ipa.tetrain.com.
_kerberos._tcp.tornedo.tetrain.com.        SRV 0 100 88   ipa.tetrain.com.
_kerberos._udp.tornedo.tetrain.com.        SRV 0 100 88   ipa.tetrain.com.
_kerberos-master._tcp.tornedo.tetrain.com. SRV 0 100 88   ipa.tetrain.com.
_kerberos-master._udp.tornedo.tetrain.com. SRV 0 100 88   ipa.tetrain.com.
_kpasswd._tcp.tornedo.tetrain.com.         SRV 0 100 464  ipa.tetrain.com.
_kpasswd._udp.tornedo.tetrain.com.         SRV 0 100 464  ipa.tetrain.com.
</nowiki>
</pre>

=== 8.10 IMPORTANT finding during pre-cutover validation — missing SIDs broke auth for 59 users ===

Client tested logging into <code><nowiki>ipa.tetrain.com</nowiki></code>'s web UI as a real user (<code><nowiki>biswajit</nowiki></code>) — succeeded on <code><nowiki>tornedo.tetrain.com</nowiki></code>, '''failed''' on <code><nowiki>ipa.tetrain.com</nowiki></code> with the same password. Root cause, confirmed via <code><nowiki>krb5kdc.log</nowiki></code>: <code><nowiki>admin</nowiki></code>'s AS-REQ goes <code><nowiki>NEEDED_PREAUTH</nowiki></code> → <code><nowiki>ISSUE</nowiki></code> (clean); a normal user's goes <code><nowiki>NEEDED_PREAUTH</nowiki></code> → <code><nowiki>HANDLE_AUTHDATA: ... No such file or directory</nowiki></code> — fails during PAC generation, not password verification (this is NOT a <code><nowiki>PREAUTH_FAILED</nowiki></code>/wrong-password error).

Traced to exactly the warning <code><nowiki>ipa.tetrain.com</nowiki></code>'s install printed and unattended mode explicitly skipped: *"66 existing users or groups do not have a SID identifier assigned... installer will NOT run ipa-sidgen task."* Confirmed via LDAP: <code><nowiki>biswajit</nowiki></code> had no <code><nowiki>ipaNTSecurityIdentifier</nowiki></code> attribute at all (<code><nowiki>admin</nowiki></code> did — <code><nowiki>S-1-5-21-...-500</nowiki></code>). '''59 users total''' were affected — this would have broken login for the vast majority of real users the moment cutover happened, not just one account. Root cause of why it only breaks on the *new* server despite identical replicated LDAP data: <code><nowiki>ipa.tetrain.com</nowiki></code>'s install activated the AD-trust/SID plugin, and this IPA version (4.13) enforces PAC/SID generation during Kerberos ticket issuance more strictly than the old IPA 4.6 server does — the old server simply doesn't hit this code path.

'''Fixed''': <code><nowiki>ipa-adtrust-install</nowiki></code> wasn't even installed (separate package). Installed <code><nowiki>ipa-server-trust-ad</nowiki></code>, then ran <code><nowiki>ipa-adtrust-install -U --add-sids</nowiki></code> (client ran it directly, on <code><nowiki>ipa.tetrain.com</nowiki></code>) — completed the <code><nowiki>[23/24]: adding SIDs to existing users and groups</nowiki></code> step. '''Verified''': <code><nowiki>biswajit</nowiki></code> now has <code><nowiki>ipaNTSecurityIdentifier: S-1-5-21-...-1003</nowiki></code>, '''0 users remaining without a SID''', all services still healthy (plus two new ones now running: <code><nowiki>smb</nowiki></code>, <code><nowiki>winbind</nowiki></code>). Awaiting client's re-test of the actual web UI login to close this out completely.

'''Lesson for future replica builds''': never skip the <code><nowiki>ipa-sidgen</nowiki></code> unattended-mode warning as cosmetic — it silently breaks real-user authentication on the new server while looking completely fine in every health check that doesn't test a non-admin account specifically. This is exactly why testing with a real user, not just <code><nowiki>admin</nowiki></code>, mattered here.

=== 8.11 SRV records published and verified working — DONE ===

Client published all 7 records (Section 8.9) in the <code><nowiki>tetrain.com</nowiki></code> zone. Verified:

* All 7 resolve correctly via <code><nowiki>dig</nowiki></code> from both this session and from a real client's network path (<code><nowiki>koha.tetra.in</nowiki></code>) — correct priority/weight/port, target <code><nowiki>ipa.tetrain.com</nowiki></code>.
* '''End-to-end proof it actually works''': restarted <code><nowiki>sssd</nowiki></code> on <code><nowiki>koha.tetra.in</nowiki></code> (a real production IPA client) to force fresh discovery, then checked the actual artifact SSSD writes with its discovery result — <code><nowiki>/var/lib/sss/pubconf/kdcinfo.TORNEDO.TETRAIN.COM</nowiki></code> now contains <code><nowiki>13.126.212.202</nowiki></code> (which is <code><nowiki>ipa.tetrain.com</nowiki></code>'s IP, not <code><nowiki>tornedo.tetrain.com</nowiki></code>'s <code><nowiki>13.126.118.148</nowiki></code>). SSSD is discovering and will connect to the new server via SRV, exactly as designed — no client-side config was touched to achieve this.

=== 8.12 CA renewal master promoted to <code><nowiki>ipa.tetrain.com</nowiki></code> — DONE ===

<code><nowiki>ipa-csreplica-manage set-renewal-master ipa.tetrain.com</nowiki></code> needed the '''Directory Manager''' password (a more privileged credential than <code><nowiki>admin</nowiki></code> — the LDAP superuser) — client ran it directly, same pattern as every other admin-credential step. Verified via LDAP: <code><nowiki>cn=CA,cn=ipa.tetrain.com,cn=masters,...</nowiki></code> now carries <code><nowiki>ipaConfigString: caRenewalMaster</nowiki></code>. All services still healthy post-change.

=== 8.13 Still ahead ===

# Soak period — confirm real client traffic lands on <code><nowiki>ipa.tetrain.com</nowiki></code> via SRV while <code><nowiki>tornedo.tetrain.com</nowiki></code> stays up as fallback (now proven working on at least one real client; worth spot-checking a few more over the soak period).
# Decommission <code><nowiki>tornedo.tetrain.com</nowiki></code> — still fully gated on client go-ahead per original instruction ("involve me when we are ready to cut over").

----