Editing Tornedo.tetrain.com Upgrade Plan (section)

=== 8.4 Second hop — <code><nowiki>ipa.tetrain.com</nowiki></code> promoted from <code><nowiki>ipa8.tetrain.com</nowiki></code> — DONE, validated ===

Networking between <code><nowiki>ipa8.tetrain.com</nowiki></code> and <code><nowiki>ipa.tetrain.com</nowiki></code> was already fully open both directions on first check (389/636/88/464/80/443/8080/8443) — no security-group work needed for this pair, unlike the first hop.

Obstacles hit and resolved (some recurring from 8.3, some new):

# Stale IPA '''master'''/topology entry for <code><nowiki>ipa.tetrain.com</nowiki></code> — leftover from the original failed direct-to-<code><nowiki>tornedo</nowiki></code> attempt (which got as far as <code><nowiki>[33/40]: adding master entry</nowiki></code> before failing at CA setup). <code><nowiki>host-disable</nowiki></code>/<code><nowiki>host-mod</nowiki></code> couldn't touch it (<code><nowiki>invalid 'hostname': An IPA master host cannot be deleted or disabled</nowiki></code>); needed <code><nowiki>ipa server-del ipa.tetrain.com --force</nowiki></code> instead. '''Recurred 3 times''' across this hop's retries — each failed attempt that got past "adding master entry" left another one, so this became the standard first step of every cleanup cycle.
# My own cleanup mistake: <code><nowiki>systemctl disable pki-tomcatd@pki-tomcat.service</nowiki></code> (to kill an orphaned process) auto-removed the now-empty <code><nowiki>/etc/systemd/system/pki-tomcatd.target.wants/</nowiki></code> directory as a systemd side effect, which then broke the *next* install attempt's symlink creation (<code><nowiki>FileNotFoundError</nowiki></code>). Fixed with <code><nowiki>mkdir -p</nowiki></code> to recreate it. Lesson: prefer <code><nowiki>systemctl stop</nowiki></code> + <code><nowiki>pkill</nowiki></code> over <code><nowiki>disable</nowiki></code> for ad-hoc orphan cleanup mid-troubleshooting.
# Orphaned <code><nowiki>ns-slapd</nowiki></code>/<code><nowiki>pki-tomcatd</nowiki></code> processes recurred after multiple failed attempts — <code><nowiki>ipa-server-install --uninstall</nowiki></code> repeatedly failed to actually stop them despite reporting success; needed manual <code><nowiki>systemctl stop</nowiki></code> + <code><nowiki>pkill -9 ns-slapd</nowiki></code> each time, checked via <code><nowiki>ps aux</nowiki></code> before every retry.
# <code><nowiki>ldap_add: Already exists (68)</nowiki></code> on "adding master entry" — same stale-topology-entry class as #1, same fix (<code><nowiki>ipa server-del --force</nowiki></code>).
# '''Root cause of the actual CA failure''' (as opposed to the cleanup-artifact failures above): this box has 1.6G RAM and '''zero swap'''. The CA database init step spawns a JVM (<code><nowiki>ca-db-init</nowiki></code>) that failed silently — no debug log, no catalina log output, exit code 255, no kernel OOM-killer entry (the JVM fails to reserve heap *before* its own logging initializes, so nothing gets logged and the kernel OOM-killer never needs to intervene). Contrast with <code><nowiki>ipa8.tetrain.com</nowiki></code>, which has 6.2G swap and sailed through the identical step without issue. '''Fixed by adding a 4G swap file''' (<code><nowiki>fallocate</nowiki></code>+<code><nowiki>mkswap</nowiki></code>+<code><nowiki>swapon</nowiki></code>, persisted in <code><nowiki>/etc/fstab</nowiki></code>) — confirmed swap was actively used (140Mi) during the successful run. (One hiccup: the swapfile briefly vanished — client was independently trying to set one up at the same time and it seems to have gotten removed; recreated once and it stuck.)
# Declined to use the client's offer to hand over the actual <code><nowiki>admin</nowiki></code> Kerberos password directly in chat for <code><nowiki>kinit</nowiki></code> — held the line consistent with not handling account passwords, even under explicit repeated request. Client continued running <code><nowiki>kinit admin</nowiki></code> + the install command themselves each time instead.

'''<code><nowiki>ipa-replica-install --setup-ca --skip-mem-check --unattended</nowiki></code> completed successfully.''' Followed immediately by <code><nowiki>ipa-server-upgrade</nowiki></code> (same precautionary step as the first hop) — successful. '''Validated''': all 7 services RUNNING, all certs <code><nowiki>MONITORING</nowiki></code> with no errors, LDAP search via LDAPI succeeds, swap confirmed in active use.