Editing Tetra's Nagios Server Upgrade and Implemenation Process and Runbook

[[category:Infrastructure Documentation]]

This is a reproducible, step-by-step runbook for the migration of '''nagios.tetrain.com''' from CentOS 6 / hand-built Nagios Core 4.1.1 to Rocky Linux 9 with a current Nagios Core, Thruk, PNP4Nagios, NagiosQL and Livestatus stack, plus a full data migration of the existing monitoring configuration (50 hosts / 711 services). It is the companion to the project's internal decision log (<code><nowiki>nagios.tetrain.com_upgrade_plan.md</nowiki></code>), which records why each choice was made, alternatives considered, and open questions; use this page to actually reproduce the work.

'''Provenance note on exactness''': every command and finding below was captured live during this migration session (direct SSH access to both the old production box and the new build box, direct Ansible playbook runs, direct NagiosQL/Thruk/browser interaction). Nothing here is reconstructed after the fact — this is a running log of commands as they were executed, in order, including the real bugs hit and how each was actually fixed.

'''Status as of this writing''': the production cutover has been executed (Phase 16, 2026-07-27) — <code><nowiki>nagios.tetrain.com</nowiki></code> (<code><nowiki>172.105.47.191</nowiki></code>) now serves from the Rocky 9 box described in this runbook. The old CentOS 6 box is parked, not decommissioned — see "Known-incomplete items" at the end for what's still open.

----

== Phase 1 — Old server discovery (confirmed via SSH, read-only) ==

Old box: <code><nowiki>nagios.tetrain.com</nowiki></code> → <code><nowiki>172.105.47.191</nowiki></code>, a Linode VM (<code><nowiki>li2028-191.members.linode.com</nowiki></code>), IPA-enrolled, plan <code><nowiki>g6-nanode-1</nowiki></code> (1 vCPU / 1GB RAM / 25GB disk), region <code><nowiki>ap-west</nowiki></code> (Mumbai).

<pre>
<nowiki>
cat /etc/redhat-release          # CentOS release 6.8 (Final) -- EOL Nov 2020, never patched to 6.10
uname -r                         # 7.0.5-x86_64-linode173 (Linode-managed kernel, decoupled from userland)
free -m                          # 939M total, 74M free, 228M/511M swap in use -- tight
df -h /                          # 25G total, 7.9G used, 35% used
/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg   # hand-built Nagios Core 4.1.1 (2015)
httpd -v                         # Apache/2.2.15 (CentOS), matches external fingerprint
rpm -qa | grep -iE 'ipa-client|sssd-common'   # ipa-client-3.0.0-51, sssd-common-1.13.3-60 -- IPA-enrolled
</nowiki>
</pre>

'''Authoritative monitoring scope''' (corrected later in Phase 7 via Nagios's own parser, not a manual grep): '''50 hosts, 711 services''' spanning many separate client production domains (mail servers, ERPs, relays, VPS instances) — this box is not an internal-only tool, its downtime is client-facing.

'''Access note''': no SSH key existed for this box at the start. The root password was provided once directly in chat for this read-only pass, written to a private session-only scratchpad file, used once, then <code><nowiki>shred -u</nowiki></code>'d immediately — never persisted to the project folder or repeated back in any document. An SSH key was generated for all subsequent access to the '''new''' box (Phase 3); the old box's password should be rotated once handoff is complete.

'''Verify''': <code><nowiki>nagios -v</nowiki></code> exits 0 with the counts above; <code><nowiki>uptime</nowiki></code>/<code><nowiki>last reboot</nowiki></code> confirm last reboot 2026-07-20.

----

== Phase 2 — Target architecture decision ==

No supported <code><nowiki>leapp</nowiki></code>/ELevate in-place path exists from CentOS 6 (unlike the CentOS-7-based <code><nowiki>tornedo.tetrain.com</nowiki></code> project) — CentOS 6 predates that tooling generation entirely. '''Decision: fresh-install migration on a new Rocky 9 VM, not an in-place upgrade.'''

'''Critical constraint''': the production IP <code><nowiki>172.105.47.191</nowiki></code> cannot change — it is hard-allow-listed in many client-side firewalls and NRPE <code><nowiki>allowed_hosts</nowiki></code> entries. A DNS-only cutover would not fix that. Since this is a Linode VM, the fix is Linode's '''IP Transfer''' feature — it moves an IPv4 address between two Linodes in the same data center; nothing downstream needs to know or change.

<pre>
<nowiki>
New Rocky 9 VM, same Linode region, temp IP --configure Nagios + Thruk--> import 50 hosts / 711 services
        |                                                                       |
        +----------------------- run in parallel, validate parity -------------+
                                          |
                     Linode IP Transfer: 172.105.47.191 moves to the new box
                                          |
                 old box kept built + powered off (not deleted) -- instant rollback
                       is "transfer the IP back" if anything's wrong post-swap
</nowiki>
</pre>

Decisions confirmed with the client: target OS '''Rocky Linux 9''', dashboard/UI '''Thruk''' (evaluated against standing up a modern Checkmk instead — declined, out of scope, see Phase 12), VM sizing bumped from an exact 1GB match to a '''2GB plan''' since the old box's RAM was already nearly exhausted before adding Thruk on top.

----

== Phase 3 — New build box provisioned (Linode API) ==

<pre>
<nowiki>
# via Linode API, token from the office infrastructure vault
# confirmed the running production box's exact specs first, then created its replacement:
</nowiki>
</pre>

{| class="wikitable"
! !! Old (production) !! New (build)
|-
| Label || nagios.tetrain.com || nagios-rocky9-build
|-
| Plan || g6-nanode-1 (1 vCPU, 1GB RAM, 25GB disk) || g6-standard-1 (1 vCPU, '''2GB RAM''', 50GB disk)
|-
| Region || ap-west (Mumbai) || ap-west (Mumbai) -- same region, required for IP Transfer
|-
| Image || linode/centos6.8 || linode/rocky9 (booted as Rocky Linux 9.8 "Blue Onyx")
|-
| IP (temporary) || 172.105.47.191 (unchanged, stays live) || 172.105.40.44
|-
| Access || root password || dedicated ED25519 key only, generated for this box
|-
| Hostname || li2028-191.members.linode.com / nagios.tetrain.com || nagios-rocky9-build.tetrain.com
|}

A root password was also set as a Lish-console fallback only, saved to a local <code><nowiki>keys/</nowiki></code> file marked for rotation, never shown in chat.

'''Verify''': SSH with the new key confirms 2GB RAM (1.6GB free right after boot vs 74MB free on the old box) and 49GB disk free. '''The production box was not touched''' — it kept running CentOS 6 / Nagios Core 4.1.1 without interruption for the entire build.

----

== Phase 4 — Stack deploy via Ansible ==

Ran an adapted Ansible playbook (roles: <code><nowiki>common, apache, nagios_core, plugins, database, nrpe_client, ncpa_client, pnp4nagios, nagiosql, livestatus, thruk, backup</nowiki></code>) against <code><nowiki>nagios-rocky9-build</nowiki></code>. Clean run: <code><nowiki>ok=116 changed=7 failed=0</nowiki></code>.

{| class="wikitable"
! Component !! Result
|-
| Nagios Core 4.5.13 || active, config validates clean
|-
| Apache 2.4.62 + TLS (self-signed) || active
|-
| MariaDB || active
|-
| Livestatus (compiled, Nagios-4.5-compatible fork) || loaded, socket open
|-
| PNP4Nagios (current, not the old 0.6.x) || installed, Apache config live
|-
| NagiosQL 3.5.0 || installed, install wizard pending (manual, browser-driven step)
|-
| Thruk 3.28 || installed, correctly auth-gated
|}

'''Five real playbook bugs found and fixed''' (all folded into the durable playbook copy):

# '''TLS download failures''': <code><nowiki>get_url</nowiki></code> (Python <code><nowiki>ssl</nowiki></code> module) unreliably threw <code><nowiki>[ASN1: NOT_ENOUGH_DATA]</nowiki></code> against <code><nowiki>assets.nagios.com</nowiki></code>, GitHub releases, and <code><nowiki>download.thruk.org</nowiki></code>, while <code><nowiki>curl</nowiki></code> worked every time from the same box. Fixed by shelling out to <code><nowiki>curl</nowiki></code> for every download task.
# '''PNP4Nagios configure flags wrong''': <code><nowiki>--with-httpd-conf</nowiki></code> isn't a real flag (copied by mistake from the Nagios Core role). Fixed:
<pre>
<nowiki>
./configure --with-http_server=apache --with-http_server_base=/etc/httpd \
  --with-http_confdir=/etc/httpd/conf.d --disable-selinux
</nowiki>
</pre>
# '''Thruk missing RPM dependency''': <code><nowiki>thruk-plugin-reporting</nowiki></code> is an undeclared dependency of the main <code><nowiki>thruk</nowiki></code> package — depsolve failed without it; added to the install list.
# '''Nagios itself never actually started''': the role only starts Nagios via a handler notified by the systemd-override file task — a no-op on any re-run where that file is already correct. Added an unconditional enable+start task, matching how Apache/MariaDB/firewalld were already handled.
# '''Thruk dual-auth bypass''': the Thruk RPM ships two active auth mechanisms at once — the intended <code><nowiki>AuthType Basic</nowiki></code> in <code><nowiki>thruk.conf</nowiki></code>, and a cookie/rewrite-based auth in <code><nowiki>thruk_cookie_auth_vhost.conf</nowiki></code> granting <code><nowiki>Require all granted</nowiki></code> for nearly all of <code><nowiki>/thruk/</nowiki></code>. Apache's Require-merging let the blanket grant win, so anonymous requests reached the app unauthenticated. Fixed by disabling the cookie-auth vhost include and populating <code><nowiki>/etc/thruk/htpasswd</nowiki></code> with the same admin credentials as Nagios Core.

'''Verify''': all four web UIs (<code><nowiki>/nagios/</nowiki></code>, <code><nowiki>/pnp4nagios/</nowiki></code>, <code><nowiki>/nagiosql/</nowiki></code>, <code><nowiki>/thruk/</nowiki></code>) return '''401''' with no credentials and '''200''' with the generated admin login.

----

== Phase 5 — NagiosQL wizard + post-deploy fixes ==

The NagiosQL install wizard is a manual, browser-driven, one-time step by design (not automated by the playbook) — completed at <code><nowiki>https://<build-ip>/nagiosql/install/</nowiki></code>.

'''Three more real bugs surfaced once the box was actually used''', all fixed and redeployed clean (<code><nowiki>ok=119 failed=0</nowiki></code>):

# '''firewalld runtime/permanent drift''': <code><nowiki>http</nowiki></code>/<code><nowiki>https</nowiki></code> were added to firewalld's '''permanent''' config, but the running box's '''runtime''' ruleset had gone stale (only pre-existing <code><nowiki>ssh</nowiki></code> was actually enforced) — HTTPS was blocked while SSH kept working. Fixed with a full restart, not a reload:
<pre>
<nowiki>
systemctl restart firewalld   # --reload alone was confirmed NOT reliable here
</nowiki>
</pre>
# '''NagiosQL install-wizard requirements missing''': <code><nowiki>php-pear</nowiki></code> was never installed (added to the Apache role's package list); <code><nowiki>date.timezone</nowiki></code> was never set in <code><nowiki>php.ini</nowiki></code> (set to <code><nowiki>Asia/Kolkata</nowiki></code>). Also learned that <code><nowiki>php-fpm</nowiki></code> caches <code><nowiki>php.ini</nowiki></code> in memory — restarting only <code><nowiki>httpd</nowiki></code> left stale values in effect; added a proper <code><nowiki>Restart php-fpm</nowiki></code> handler.
# '''PNP4Nagios graphs structurally broken''': two stacked gaps — (a) <code><nowiki>npcd</nowiki></code>'s systemd unit installs but was never enabled/started (same missing-enablement pattern as bug 4 in Phase 4); (b) the role never set the perfdata file templates in <code><nowiki>nagios.cfg</nowiki></code>. Without them Nagios writes perfdata in its own raw positional format, but PNP4Nagios's bulk-mode parser expects its own tagged <code><nowiki>DATATYPE::.../HOSTNAME::...</nowiki></code> format (per PNP4Nagios's own shipped <code><nowiki>nagios-bulk.cfg</nowiki></code> reference) — every line was silently logged "No Perfdata. Skipping line" and no RRDs were ever created. Fix, in <code><nowiki>nagios.cfg</nowiki></code> (single-quoted YAML in the Ansible source — double-quotes would convert <code><nowiki>\t</nowiki></code> to a real tab and break Nagios's own escape parsing):
<pre>
<nowiki>
host_perfdata_file_template=DATATYPE::HOSTPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\t...
service_perfdata_file_template=DATATYPE::SERVICEPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\t...
</nowiki>
</pre>

'''Verify''': real RRDs generate, e.g. <code><nowiki>rrd/localhost/Current_Users.rrd</nowiki></code>.

----

== Phase 6 — NagiosQL / Nagios Core alignment incident ==

After completing the NagiosQL wizard and clicking "Write monitoring data," Nagios Core crash-looped (<code><nowiki>systemctl status nagios</nowiki></code> showed <code><nowiki>activating (auto-restart)</nowiki></code>, restart counter over 190).

'''Root cause''': NagiosQL's freshly-written config collided with Nagios Core's own stock quickstart sample files (<code><nowiki>windows.cfg, printer.cfg, switch.cfg, localhost.cfg, templates.cfg</nowiki></code>, installed by <code><nowiki>make install-config</nowiki></code> and never removed by the playbook) — identical sample host names and template names defined in both places at once.

'''Fixed''':
# Retired the 5 stock sample files (moved outside the recursively-scanned <code><nowiki>objects/</nowiki></code> tree, kept for reference).
# Removed 6 redundant <code><nowiki>cfg_dir</nowiki></code> entries the playbook had added — <code><nowiki>make install-config</nowiki></code>'s default <code><nowiki>cfg_dir=/usr/local/nagios/etc/objects/</nowiki></code> is already recursive, so the extras double-loaded every file underneath them.
# '''Permission gap''': NagiosQL (running as <code><nowiki>apache</nowiki></code>) recreates its managed subdirectories as <code><nowiki>apache:apache</nowiki></code> mode <code><nowiki>750</nowiki></code>, blocking the <code><nowiki>nagios</nowiki></code> user (not in the <code><nowiki>apache</nowiki></code> group). A one-time <code><nowiki>chgrp</nowiki></code> wouldn't survive NagiosQL doing this again, so added mutual group membership instead — <code><nowiki>nagios</nowiki></code> added to the <code><nowiki>apache</nowiki></code> group (alongside the existing reverse membership).
# '''Duplicate <code>check_ncpa</code> command''': the <code><nowiki>ncpa_client</nowiki></code> role's own command definition duplicated what NagiosQL's default command set already defines identically. Removed the role's copy — NagiosQL is authoritative for object definitions now.

Also fixed while completing the wizard: NagiosQL's DB-creation step needs genuine <code><nowiki>CREATE DATABASE</nowiki></code>/<code><nowiki>CREATE USER</nowiki></code> privileges, which MariaDB's socket-authenticated <code><nowiki>root</nowiki></code> can't grant from a web form by design. Created a temporary password-authenticated installer account for that one step only (revoke after handoff).

'''Verify''': <code><nowiki>Total Warnings: 0, Total Errors: 0</nowiki></code> after applying all four fixes and restarting.

----

== Phase 7 — Migration data pull from the old box ==

Pulled a full read-only copy of <code><nowiki>/usr/local/nagios/etc/</nowiki></code> from the old box (excluding the htpasswd file) — nothing changed on the old box; the transfer tarball was deleted afterward.

Most of the old box's <code><nowiki>objects/</nowiki></code> tree is a decade of untouched <code><nowiki>.bkp</nowiki></code>/<code><nowiki>.bak</nowiki></code> files Nagios never actually loads (only 96 of 243 files under <code><nowiki>service/</nowiki></code> even end in <code><nowiki>.cfg</nowiki></code>). What's actually live, per <code><nowiki>nagios.cfg</nowiki></code>'s own directives:

<pre>
<nowiki>
objects/commands.cfg, contacts.cfg, timeperiods.cfg, templates.cfg, linux-box.cfg,
escalation.cfg, servicegroup.cfg, localhost.cfg, windows.cfg
service/*.cfg   (96 files, one per client/host)
</nowiki>
</pre>

'''Authoritative scope''' (confirmed via <code><nowiki>nagios -v</nowiki></code>, 0 errors):

{| class="wikitable"
! Object !! Count
|-
| Hosts || 50
|-
| Services || 711
|-
| Host groups || 5
|-
| Service groups || 3
|-
| Contacts / contact groups || 7 / 7
|-
| Commands || 327
|-
| Time periods || 6
|-
| Host / service escalations || 50 / 717
|}

'''6 pre-existing duplicate-service warnings''' found in the source itself (Nagios tolerates these — first definition wins — but flagged as real cruft, not carried forward silently): most notably "Load Average" on <code><nowiki>mail.tetrain.com</nowiki></code> independently redefined in three separate client config files.

----

== Phase 8 — Bulk import into NagiosQL ==

Imported the full config (105 files: 9 base object files + 96 per-client host/service files) via NagiosQL's own HTTP import mechanism, driven directly (logged in as NagiosQL's own <code><nowiki>admin</nowiki></code> user) rather than clicking through 105 files by hand.

'''Reverse-engineering the import form''' — NagiosQL's PHP source comments use different names than the real HTML fields, cost a couple of failed attempts before finding the real ones:

<pre>
<nowiki>
mselValue1[]   -- multi-select file list (NOT chkMselValue1[] as the source comments imply)
chbValue1      -- overwrite checkbox (NOT chkChbValue1)
hidStatus=1    -- required hidden trigger field, normally set by JS on submit in the real UI
</nowiki>
</pre>

'''Two fresh bugs found and fixed during this pass''':
# The 105 staged import files sat in <code><nowiki>/usr/local/nagios/etc/objects/objects/</nowiki></code> — itself inside the recursively-scanned <code><nowiki>objects/</nowiki></code> tree, so Nagios loaded both the raw staging copies and NagiosQL's freshly-written official config at once. Moved staging files to <code><nowiki>/usr/local/nagios/etc/_import_staging/</nowiki></code> (outside the scan path).
# Ansible's own <code><nowiki>commands/pnp4nagios.cfg</nowiki></code> duplicated <code><nowiki>process-host-perfdata-file</nowiki></code>, which NagiosQL's own imported command set also defines — retired, same fix as the <code><nowiki>check_ncpa</nowiki></code> duplicate in Phase 6.
# One data bug from the source itself: the <code><nowiki>passive_service</nowiki></code> template imported with an empty <code><nowiki>register</nowiki></code> field (should default to <code><nowiki>'0'</nowiki></code>) — Nagios treated it as a real service instance missing a host. Fixed via a direct <code><nowiki>UPDATE</nowiki></code> on <code><nowiki>tbl_servicetemplate</nowiki></code> (single row), then re-triggered NagiosQL's "Write monitoring data" to regenerate the file from the corrected DB state.

'''Confirmed live''' (<code><nowiki>nagios -v</nowiki></code>, then restart, <code><nowiki>nagiostats</nowiki></code>): 53 hosts / 724 services loaded (+3 hosts / +13 services vs. the old box's 50/711 — NagiosQL's own leftover seed hosts, cleaned up in Phase 9). '''Total Warnings: 0, Total Errors: 0.'''

----

== Phase 9 — Seed-data cleanup ==

Deleted the 3 leftover NagiosQL seed hosts (<code><nowiki>winserver, hplj2605dn, linksys-srw224p</nowiki></code>) and their services/link-table rows.

<pre>
<nowiki>
# NagiosQL's delete-from-list action isn't in hosts.php itself -- it's generic shared logic
# in prepend_content.php, triggered by POSTing to the object's own admin page:
POST admin/hosts.php   modus=checkform&selModify=delete&hidListId=<id>   (one ID per request)
</nowiki>
</pre>

'''Found''': deleting the DB rows alone left orphaned <code><nowiki>.cfg</nowiki></code> files on disk — NagiosQL's "Write" only regenerates files for objects that currently exist, it does not prune files for objects that no longer do. The 3 stale host/service <code><nowiki>.cfg</nowiki></code> files had to be removed manually before re-validating.

'''Result: exact match to the old box's authoritative counts''' — 50 hosts, 711 services, 0 errors, confirmed via <code><nowiki>nagiostats</nowiki></code> after restart.

----

== Phase 10 — PNP4Nagios graph routing fix ==

Thruk's graph links (<code><nowiki>/pnp4nagios/graph?host=X</nowiki></code>) were 404ing straight from Apache — the request never reached PHP.

'''Root cause''': the Apache config only had a bare <code><nowiki>Alias /pnp4nagios</nowiki></code> to the share directory, missing PNP4Nagios's documented short-URL rewrite convention that Thruk's links rely on.

'''First attempt (failed)''': a top-level <code><nowiki>RewriteRule</nowiki></code> per endpoint (<code><nowiki>graph, image, json, xml, popup</nowiki></code>) — didn't work, because <code><nowiki>mod_alias</nowiki></code>'s <code><nowiki>Alias</nowiki></code> directive had already translated the URL to a filesystem path before <code><nowiki>mod_rewrite</nowiki></code> ever got a chance to run.

'''Actual fix''' — matched PNP4Nagios's own shipped reference config (<code><nowiki>etc/config_samples/http_server_apache/pnp4nagios.conf</nowiki></code>): the rewrite has to live '''inside''' the <code><nowiki><Directory></nowiki></code> block, operating on the per-directory URL-path, with one generic catch-all rule rather than one rule per endpoint:

<pre>
<nowiki>
<Directory "/usr/local/pnp4nagios/share">
    ...
    RewriteEngine On
    RewriteCond %{THE_REQUEST} ^GET\ /pnp4nagios/(.+)/(.*)\ HTTP
    RewriteCond %{DOCUMENT_ROOT}/%1 !-f
    RewriteCond %{DOCUMENT_ROOT}/%1 !-d
    RewriteRule ^(.+)$ index.php/$1 [L]
</Directory>
</nowiki>
</pre>

'''Verify''': <code><nowiki>localhost</nowiki></code> (which has accumulated real perfdata) returns HTTP 200 on its graph URL. Freshly-imported hosts correctly show PNP4Nagios's own "perfdata directory does not exist" message until Nagios has actually run checks against them long enough to accumulate history — expected, not a bug.

----

== Phase 11 — Perfdata pipeline silently broken ==

After the Phase 10 routing fix, graphs still failed for every freshly-imported real host, even minutes after their checks had clearly succeeded (<code><nowiki>has_been_checked=1</nowiki></code> in <code><nowiki>status.dat</nowiki></code>).

'''Root cause''': the Phase 8 NagiosQL import brought in its own <code><nowiki>process-host-perfdata-file</nowiki></code>/<code><nowiki>process-service-perfdata-file</nowiki></code> command definitions, pointing at <code><nowiki>/usr/local/pnp4nagios/var/{host,service}-perfdata</nowiki></code> (the vendor's documented convention). But <code><nowiki>nagios.cfg</nowiki></code>'s own <code><nowiki>host_perfdata_file</nowiki></code>/<code><nowiki>service_perfdata_file</nowiki></code> directives (set by the Ansible <code><nowiki>pnp4nagios</nowiki></code> role) still pointed at <code><nowiki>/usr/local/nagios/var/{host,service}-perfdata</nowiki></code> — a different path entirely. Nagios wrote real perfdata to one file; the processing command bulk-read a different, stale one. No error anywhere — it just silently processed nothing, indefinitely.

'''Fixed''' by realigning <code><nowiki>nagios.cfg</nowiki></code> to the vendor-standard path that NagiosQL's imported commands already expect, rather than fighting NagiosQL's convention. After restart, force-scheduled an immediate check for all 50 hosts via the external command pipe (<code><nowiki>SCHEDULE_FORCED_HOST_CHECK</nowiki></code>) rather than waiting out the normal 30-minute check-spread windows.

'''Verify''': all 50 real hosts + <code><nowiki>localhost</nowiki></code> now have RRD data and return HTTP 200 on their graph URLs.

----

== Phase 12 — Check_MK evaluated, nothing to migrate ==

The old box also runs Check_MK at <code><nowiki>nagios.tetrain.com/check_mk</nowiki></code> — checked whether it needed migrating too.

<pre>
<nowiki>
# Check_MK 1.2.2p2 (2013) -- an old, non-OMD, Nagios-add-on-style install
cat main.mk    # all_hosts = [ ]  -- zero hosts configured natively
</nowiki>
</pre>

The web login does still work, but it was purely a Livestatus-based UI layered on the same Nagios core — not a separate monitoring system with its own data (matches the client's own recollection of it predating Check_MK's modern full product). '''Conclusion: nothing was missed''' — all real monitoring config already lived in the raw Nagios object files migrated in Phase 8. '''Decision: stick with Thruk, do not stand up modern Checkmk''' — that would be a materially different, much bigger addition (its own agents, host/service discovery), out of scope for this migration. Thruk is architecturally the same "Livestatus UI on the same core" pattern the client liked about the old Check_MK setup.

----

== Phase 13 — Duplicate-service cleanup ==

7 duplicate-service warnings existed post-import (0 blocking — Nagios just uses the first definition and ignores the rest — but real cruft). Turned out to be '''three genuinely different root causes''', not one kind of cruft — a blind "delete the duplicate" pass would have been wrong for 5 of the 7:

# '''Stale NagiosQL seed data''' (<code><nowiki>Swap Usage</nowiki></code>/<code><nowiki>Root Partition</nowiki></code> on <code><nowiki>localhost</nowiki></code>, 2 warnings) — NagiosQL's factory-default <code><nowiki>localhost</nowiki></code> checks were never removed after the real import. Verified the correct values against the old box's actual production config before deleting: kept <code><nowiki>Root Partition</nowiki></code> at 10%/5% and <code><nowiki>Swap Usage</nowiki></code> at 20/10 (real values), deleted the 20%/10% seed leftovers.
# '''Two genuinely different checks sharing a name''' (<code><nowiki>Website Status</nowiki></code> on reachingcriticalwill.org, <code><nowiki>NTP</nowiki></code> on tetrain.com, 2 warnings) — one custom check command, one generic NRPE fallback, both real. Deleting either would have dropped real coverage — renamed to disambiguate instead: <code><nowiki>Website Status - NRPE</nowiki></code> and <code><nowiki>NTP - Default</nowiki></code>. (First attempt used parentheses, e.g. <code><nowiki>NTP (Default)</nowiki></code> — parentheses are in Nagios's <code><nowiki>illegal_object_name_chars</nowiki></code> list, hit a real "illegal characters" error; fixed with dashes.)
# '''A real config bug inherited from the old box''' (<code><nowiki>Load Average</nowiki></code> on <code><nowiki>mail.tetrain.com</nowiki></code> via 3 unrelated client files, 3 warnings) — <code><nowiki>email.planetpci.com</nowiki></code>, <code><nowiki>mail21c.pahwa.com</nowiki></code>, and <code><nowiki>zm.wbtea.com</nowiki></code> each define their own <code><nowiki>Load Average</nowiki></code> check but also tag <code><nowiki>hostgroup_name Tetra servers</nowiki></code>; since <code><nowiki>mail.tetrain.com</nowiki></code> is a member of that hostgroup too, all three unintentionally applied to it as well. Confirmed accidental (not intended as hostgroup-wide) — removed the <code><nowiki>hostgroup_name</nowiki></code> link from all 3 so each stays scoped to its own host only.

'''Verify''': <code><nowiki>Total Warnings: 0, Total Errors: 0</nowiki></code>, service restarted, 50 hosts active.

'''Learned along the way''': NagiosQL's "Write monitoring data" button (<code><nowiki>butValue1</nowiki></code>) does '''not''' rewrite <code><nowiki>commands.cfg</nowiki></code> — only "Write additional data" (<code><nowiki>butValue2</nowiki></code>) does. Also, NagiosQL web sessions expire — a second write attempt after idle time needs a fresh login (POST to <code><nowiki>tfUsername</nowiki></code>/<code><nowiki>tfPassword</nowiki></code>) before it will actually save.

----

== Phase 14 — Notification integrations rebuilt ==

Pulled the full SMS/Telegram/ihelpdesk scripts from the old box (only headers had been reviewed before) and rebuilt them on the new box, per the client's own credential-handling decisions (kept-vs-rotate calls made explicitly, not assumed):

'''SMS''' — <code><nowiki>sms.sh/php</nowiki></code> (own number, one gateway), <code><nowiki>hero-sms.sh/php</nowiki></code> and <code><nowiki>VCS-sms.sh/php</nowiki></code> (a second contact group, a second gateway) copied over as-is, credentials kept per the client's call. Confirmed <code><nowiki>VCS-sms.php</nowiki></code> and <code><nowiki>hero-sms.php</nowiki></code> were byte-identical — the long-standing "<code><nowiki>VCS-sms.sh</nowiki></code> actually calls <code><nowiki>hero-sms.php</nowiki></code> instead of its own file" bug had been a no-op in practice, but the routing was fixed anyway for future-proofing. '''Open item''': one gateway's credentials look like a demo/trial account — verifying it's still active was deferred by the client.

'''Telegram''' — kept the existing bot token per the client's call, but '''hardened how it's used''': rewrote <code><nowiki>nagios_telegram.py</nowiki></code> to call Telegram's plain HTTP Bot API directly (dropping a third-party client library and an EOL Python-version pin that doesn't even exist on Rocky 9) and to read the token from a <code><nowiki>--token-file</nowiki></code> argument instead of a <code><nowiki>--token</nowiki></code> CLI argument:

<pre>
<nowiki>
#!/usr/bin/env python3
# Token read from --token-file, not passed as a CLI argument -- a value passed on the
# command line is visible to any local user via `ps aux` for the life of the process;
# a file readable only by the nagios user is not.
import argparse, json, urllib.request, urllib.parse

def send_notification(token, chat_id, message):
    url = f"https://api.telegram.org/bot{token}/sendMessage"
    data = urllib.parse.urlencode({"chat_id": chat_id, "text": message}).encode()
    req = urllib.request.Request(url, data=data, method="POST")
    with urllib.request.urlopen(req, timeout=15) as resp:
        result = json.load(resp)
    if not result.get("ok"):
        raise RuntimeError(f"Telegram API error: {result}")
</nowiki>
</pre>

Token now lives at <code><nowiki>/usr/local/nagios/etc/telegram_bot_token</nowiki></code>, mode <code><nowiki>600</nowiki></code>, owned <code><nowiki>nagios</nowiki></code> — previously it sat in plaintext directly inside <code><nowiki>commands.cfg</nowiki></code>. Updated the corresponding <code><nowiki>tbl_command</nowiki></code> rows (host + service notify commands) to match, applied via a <code><nowiki>.sql</nowiki></code> file against NagiosQL's database rather than fighting nested-quoting on the CLI.

'''ihelpdesk ticketing''' — <code><nowiki>/opt/ihelpdesk_alert.sh</nowiki></code> (HMAC-signed API calls) copied as-is, secret kept per the client's call.

Config validates clean (<code><nowiki>Total Warnings: 0, Total Errors: 0</nowiki></code>), Nagios restarted and active. '''Only the Telegram channel was actually live-tested''' (a real message sent and confirmed received) — SMS and ihelpdesk were deliberately left untested pending the client's own go-ahead, since firing them sends a real message/ticket.

----

== Phase 15 — Parity validation: real gaps found and fixed ==

A full status check on the new box initially showed '''685 of 713 services CRITICAL'''. Rather than assume this was entirely the expected pre-cutover IP-allowlist gap, every failure was categorized by its actual error message — good thing, because only '''356''' were the expected NRPE/firewall gap (client-side firewalls only trust <code><nowiki>172.105.47.191</nowiki></code>, not the build box's temporary IP — resolves automatically once the IP Transfer happens). The other '''329''' were real gaps: custom/third-party plugins present on the old box outside the standard <code><nowiki>nagios-plugins</nowiki></code> package, never copied during migration.

{| class="wikitable"
! Missing !! Affected checks !! Root cause / fix
|-
| <code><nowiki>check_dnsbl</nowiki></code> || 302 || Perl module <code><nowiki>Net::DNS</nowiki></code> missing -- <code><nowiki>dnf install perl-Net-DNS</nowiki></code>
|-
| <code><nowiki>check_ssl_cert</nowiki></code> || 13 || Plugin binary missing -- copied from old box
|-
| <code><nowiki>Crypt::X509</nowiki></code> (used by check_ssl_validity) || 6 || CPAN-installed module, not RPM. First copied to the version-agnostic <code><nowiki>/usr/local/share/perl5/</nowiki></code>, but Perl 5.32's <code><nowiki>@INC</nowiki></code> only searches the version-specific <code><nowiki>/usr/local/share/perl5/5.32/</nowiki></code> -- moved into the correct subdirectory
|-
| <code><nowiki>check_https_1</nowiki></code> || 5 || Plugin binary missing -- copied from old box
|-
| <code><nowiki>check_zimbra_outgoing_mail.sh</nowiki></code> || 1 || Runs via NRPE on the remote Zimbra host itself, not the Nagios server -- copied for parity but was never actually a local gap
|}

All plugins individually verified working after the fix (real, correct output — e.g. <code><nowiki>check_ssl_cert</nowiki></code> returned a genuine "valid until Sep 8 2026" result). A full recheck of all 713 services was forced afterward; <code><nowiki>check_dnsbl</nowiki></code> in particular is slow (up to 302 checks × 60s timeout each), so results took several minutes to fully settle.

'''Practical implication''': full service-level parity can't be definitively confirmed until '''after''' the IP actually transfers, since ~356 checks are structurally blocked by client firewalls until then — that was always the expected shape of this migration, not a new problem. What '''is''' confirmed now: every plugin this box needs is present and working, config is clean, and the only remaining "failures" are exactly the ones the IP Transfer is meant to resolve.

== Phase 16 — Production cutover: Linode IP Transfer (exact — run in this session) ==

Held over the weekend per an explicit hold instruction, executed with explicit go-ahead the following Monday (2026-07-27).

'''Pre-flight''': confirmed via the Linode API that both Linodes were still <code><nowiki>running</nowiki></code> in the same region (<code><nowiki>ap-west</nowiki></code>), same IPs as Phase 3; confirmed both boxes' live service health (old box still serving 401 on its Nagios CGI = up and monitoring; new box's full stack -- Nagios, Apache, MariaDB, npcd -- all active).

'''Admin access decision''': asked the client directly whether to widen Apache's <code><nowiki>Require ip</nowiki></code> (locked to the build session's IP) and the Cloud Firewall's SSH allow-list before cutover. Client chose to leave both as-is and widen access themselves afterward -- nothing was changed on either.

'''Two prep fixes applied first''' (no client input needed for either):
<pre>
<nowiki>
# TLS cert regenerated with the real final identity
openssl req -x509 -nodes -newkey rsa:2048 -keyout /etc/pki/tls/private/nagios.key \
  -out /etc/pki/tls/certs/nagios.crt -days 825 -subj "/CN=nagios.tetrain.com/O=Tetra" \
  -addext "subjectAltName=DNS:nagios.tetrain.com,DNS:nagios-rocky9-build.tetrain.com,IP:172.105.47.191,IP:172.105.40.44,IP:127.0.0.1"
systemctl reload httpd

hostnamectl set-hostname nagios.tetrain.com
</nowiki>
</pre>

'''The transfer itself''' -- via the Linode API, the same call Cloud Manager's own "IP Transfer" button makes:
<pre>
<nowiki>
curl -X POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  https://api.linode.com/v4/networking/ips/assign \
  -d '{
    "region": "ap-west",
    "assignments": [
      {"address": "172.105.47.191", "linode_id": 101291311},
      {"address": "172.105.40.44", "linode_id": 19854563}
    ]
  }'
</nowiki>
</pre>
Confirmed immediately after via <code><nowiki>GET /v4/linode/instances</nowiki></code> that both addresses had actually moved.

'''Network-level follow-through''': both Linode config profiles have Network Helper enabled (<code><nowiki>"network": true</nowiki></code>), but the new box's own <code><nowiki>/etc/sysconfig/network-scripts/ifcfg-eth0</nowiki></code> had the old IP '''hardcoded''' (<code><nowiki>BOOTPROTO=none</nowiki></code>, static <code><nowiki>IPADDR0</nowiki></code>) -- the guest OS would not have picked up the new address on its own. Rebooted both Linodes via the API (not via SSH, since the IP swap itself would drop any existing session):
<pre>
<nowiki>
curl -X POST -H "Authorization: Bearer $TOKEN" https://api.linode.com/v4/linode/instances/101291311/reboot
curl -X POST -H "Authorization: Bearer $TOKEN" https://api.linode.com/v4/linode/instances/19854563/reboot
</nowiki>
</pre>
Both came back <code><nowiki>running</nowiki></code>; confirmed reachable at the new addresses within about a minute of reboot completion.

'''A genuine new bug caught by the reboot itself''': <code><nowiki>npcd</nowiki></code> (PNP4Nagios's RRD processor) came back <code><nowiki>inactive</nowiki></code> despite <code><nowiki>systemctl is-enabled</nowiki></code> reporting <code><nowiki>enabled</nowiki></code>. Root cause: the <code><nowiki>npcd.service</nowiki></code> unit file that PNP4Nagios's own <code><nowiki>make install-init</nowiki></code> ships contains <code><nowiki>WantedBy=no.service</nowiki></code> in its <code><nowiki>[Install]</nowiki></code> section -- not a real systemd target. <code><nowiki>systemctl enable</nowiki></code> still "succeeds" (it creates a symlink under a <code><nowiki>no.service.wants/</nowiki></code> directory that nothing ever pulls in), so this looked fine through every in-session restart during the whole build and only surfaced on an actual reboot -- which hadn't happened since npcd was first enabled. Fixed live:
<pre>
<nowiki>
sed -i 's/WantedBy=no.service/WantedBy=multi-user.target/' /usr/lib/systemd/system/npcd.service
systemctl daemon-reload
systemctl enable npcd
systemctl start npcd
</nowiki>
</pre>
Folded into the durable Ansible role (<code><nowiki>ansible-nagios-rocky9/roles/pnp4nagios/tasks/main.yml</nowiki></code>) as a <code><nowiki>lineinfile</nowiki></code> fix immediately after <code><nowiki>make install-init</nowiki></code>, so a future rebuild doesn't hit this same blind spot.

'''Verify''' (all on the production IP <code><nowiki>172.105.47.191</nowiki></code> now):
* <code><nowiki>nagios, httpd, mariadb, npcd, php-fpm</nowiki></code> all <code><nowiki>active</nowiki></code>.
* <code><nowiki>nagios -v</nowiki></code> still validates clean: 0 warnings, 0 errors, 50 hosts.
* Perfdata spool files actively growing with fresh timestamps -- the graphing pipeline survived the network change intact.
* Forced a recheck pass; of currently-critical services, '''zero''' show a timeout/connection-refused signature in plugin output -- the strongest available signal that the ~356 checks structurally blocked by client-side firewalls (Phase 15) have started clearing now that checks originate from the trusted production IP again. Full settling still follows the normal ~30-minute check-spread window.
* Old box confirmed still up and reachable at its new address (<code><nowiki>172.105.40.44</nowiki></code>) -- kept powered on, not decommissioned, as the rollback path: reversing the same API <code><nowiki>assign</nowiki></code> call (plus a reboot of both sides) would move <code><nowiki>172.105.47.191</nowiki></code> back if anything surfaces later.

----

== Phase 17 — Post-cutover access issues found and fixed (exact — run in this session) ==

The client reported "Unable to connect", redirecting to <code><nowiki>https://nagios-rocky9-build.tetrain.com/</nowiki></code>. Two real, separate bugs, both fixed:

'''Bug 1 -- stale hardcoded HTTP-to-HTTPS redirect'''. A leftover port-80 vhost from the build phase (<code><nowiki>/etc/httpd/conf.d/01-nagios-redirect.conf</nowiki></code>) had <code><nowiki>ServerName nagios-rocky9-build.tetrain.com</nowiki></code> and <code><nowiki>Redirect permanent / https://nagios-rocky9-build.tetrain.com/</nowiki></code> hardcoded. Since no other vhost declares a <code><nowiki>ServerName</nowiki></code> on port 80, Apache used this as the default vhost for '''any''' Host header -- so plain-HTTP requests to <code><nowiki>nagios.tetrain.com</nowiki></code> got 301'd to the old build hostname, which now resolves to the parked old box (stale DNS <code><nowiki>A</nowiki></code> record, never cleaned up after the build phase). Fixed by replacing the hardcoded redirect with one that echoes back whatever hostname was actually requested:
<pre>
<nowiki>
<VirtualHost *:80>
    RewriteEngine On
    RewriteCond %{HTTPS} off
    RewriteRule ^(.*)$ https://%{HTTP_HOST}$1 [R=301,L]
</VirtualHost>
</nowiki>
</pre>
'''Verify''': a request for <code><nowiki>nagios.tetrain.com</nowiki></code> now redirects to itself, not the build hostname.

'''Bug 2 -- the known <code>Require ip</code> restriction, now actually blocking real usage'''. With bug 1 fixed, the next thing hit would have been a 403 -- <code><nowiki>nagios.conf</nowiki></code>/<code><nowiki>nagiosql.conf</nowiki></code>/<code><nowiki>pnp4nagios.conf</nowiki></code> still had <code><nowiki>Require ip 45.79.120.55/32</nowiki></code> (the build session's IP) ANDed with <code><nowiki>Require valid-user</nowiki></code>. Asked the client directly since they were actively trying to use it -- '''client chose to open the web UIs to everyone''' (still Basic-Auth-gated, same as Thruk already was). Removed the <code><nowiki><RequireAny>...Require ip...</RequireAny></nowiki></code> block from all three:
<pre>
<nowiki>
# before:
<RequireAll>
    Require valid-user
    <RequireAny>
        Require ip 127.0.0.0/8
        Require ip ::1
        Require ip 45.79.120.55/32
    </RequireAny>
</RequireAll>

# after:
Require valid-user
</nowiki>
</pre>
Confirmed all four (<code><nowiki>/nagios/</nowiki></code>, <code><nowiki>/nagiosql/</nowiki></code>, <code><nowiki>/pnp4nagios/</nowiki></code>, <code><nowiki>/thruk/</nowiki></code>) now return '''401''' (auth-gated) instead of 403 (blocked) from an arbitrary outside IP. SSH / the Cloud Firewall's port-22 allow-list were '''not''' touched -- that remains a separate, still-open decision.

'''Note for anyone reproducing this''': a client browser may have cached the old "permanent" 301 redirect; a hard-refresh clears it.

----

== Known-incomplete items ==

These items are unchanged by the Phase 16 cutover — they were the client's own calls, not blockers on the technical migration, and remain open exactly as before.

* '''Admin network access (web UIs)''' -- resolved in Phase 17: <code><nowiki>Require ip</nowiki></code> removed from Nagios/NagiosQL/PNP4Nagios, now Basic-Auth-only like Thruk. '''SSH / Cloud Firewall port-22 allow-list is still narrow''' -- that was a separate decision and remains open.
* '''SMS gateway credential verification''' (Phase 14) — one gateway's credentials look like a demo/trial account; the client deferred confirming whether it's still active.
* '''SMS and ihelpdesk notification channels''' — rebuilt and config-validated, but never actually fired live (only Telegram was), pending the client's go-ahead.
* '''Old box decommissioning''' — kept fully built and powered on (not off yet, not deleted) at its real IP for rollback safety; a decision on when to power it off/decommission comes only after a validated cutover.