Editing DGARM Nagios Core - Implementation Runbook


'''Status: Final — based on all 8 source recordings''' (2026-07-10 through 2026-07-24).

'''Sources:'''
* <code>Dgarm-Nagios-Core-infosys-Kick-off-Meeting-2026-07-10_11-30-52.mkv</code>
* <code>DGARM-Nagios-Core-Fixed-PNP4Nagios-2026-07-17_11-33-20.mkv</code>
* <code>DGARM-Nagios-Core-Fixed-NagiosQL-2026-07-17_11-33-20.mkv-2026-07-17_14-24-09.mkv</code>
* <code>DGARM-Infosys-Nagios-core-Deployment-Thruk-issue-rectification-2026-07-18_11-23-23.mkv</code>
* <code>DGARM-Infosys-Nagios-Core-deployment-Process.mp4</code>
* <code>DGRAM-Infosys-Nagios-Core-Deployment-Network-switch.mp4</code>
* <code>DGRAM-Infosys-Nagios-Core-Deployment-.storage-1.mp4</code>
* <code>DGRAM-Infosys-Nagios-Core-Deployment-storage-2.mp4</code>

== 1. Background & scope ==

DGARM's device monitoring currently lives in '''Nagios XI''', which has hit a host-count limit. Devices that don't fit in XI are being onboarded into a new '''Nagios Core''' deployment instead, reusing XI's existing templates, plugins, and inventory where possible rather than building from scratch.

Device categories in scope: '''Windows servers''', '''Linux servers (physical + virtual)''', '''network/security devices''' (switches, routers, WAF, sandbox appliances), and '''storage (NetApp)'''. Laptops are explicitly out of scope.

Primary hosts referenced throughout: Nagios Core server <code>dcprdlmonitor02</code> (<code>10.150.24.167</code>), existing Nagios XI server <code>dcprodnagios01</code>.

== 2. Architecture decisions ==

{| class="wikitable"
|-
! Concern
! XI (old)
! Core (new) — decision
|-
| Config management
| Built-in GUI
| '''NagiosQL 3.5.0''' (web-based CCM)
|-
| Graphing
| Built-in
| '''PNP4Nagios''' (RRD-based) — not present in stock Core
|-
| Web dashboard
| Built-in
| '''Thruk''' — Core's default UI was judged too primitive for client-facing use
|-
| Live data backend
| N/A
| '''MK-Livestatus''' — required broker module; Thruk has no data of its own without it
|-
| Host/service templates
| Native
| Copy templates out of XI and reuse in Core via NagiosQL
|-
| Plugins
| Pre-compiled/working in XI
| '''Not assumed portable''' — each plugin must be re-obtained/recompiled and tested against the Core host
|-
| Network device discovery
| Manual per-device in XI GUI
| '''Custom SNMP-based script''' (see §5) — manual entry doesn't scale (one switch had 92 interfaces)
|-
| Notifications
| XI's own PHP handler scripts
| XI already has <code>handle_nagioscore_event.php</code> / <code>handle_nagioscore_notification.php</code> under <code>/usr/local/nagiosxi/scripts/</code> — evaluate reusing these before writing new ones
|}

== 3. Install sequence ==

# '''Base install''': Nagios Core + dependencies on the target host (~1 week estimated for base install/config).
# '''Install NagiosQL''' as the configuration manager:
** Deploy at a path served by the web server (<code>/var/www/html/nagiosql</code> → <code>http://<host>/nagiosql/</code>).
** Create the expected directory tree '''before''' first use:
<syntaxhighlight lang="bash">
     sudo mkdir -p /etc/nagiosql/hosts /etc/nagiosql/services \
          /etc/nagiosql/hostgroups /etc/nagiosql/servicegroups \
          /etc/nagiosql/backup/hosts /etc/nagiosql/backup/services
     sudo chown -R apache:apache /etc/nagiosql
     sudo chmod -R 775 /etc/nagiosql
</syntaxhighlight>
** If SELinux is <code>Enforcing</code> (check with <code>getenforce</code>), also fix SELinux context/labeling so Apache can write there, not just POSIX permissions.
** In '''Administration → Config targets''', confirm <code>Nagios base directory</code> (<code>/usr/local/nagios/etc/</code>), config/cgi/resource file paths all match the actual install.
# '''Install PNP4Nagios''' for graphing:
** If the UI throws <code>sizeof(): Parameter must be an array or an object that implements Countable</code> in <code>application/models/data.php</code>, that's a PHP 8.x compatibility break in PNP4Nagios 0.6.26 — see fix in §6.
** Wire hosts/services to graphs by adding to <code>/usr/local/nagios/etc/objects/templates.cfg</code>:
<syntaxhighlight lang="text">
     define host {
         name        host-pnp
         action_url  /pnp4nagios/index.php/graph?host=$HOSTNAME$&srv=_HOST_
         register    0
     }
     define service {
         name        service-pnp
         action_url  /pnp4nagios/index.php/graph?host=$HOSTNAME$&srv=$SERVICEDESC$
         register    0
     }
</syntaxhighlight>
# '''Install Thruk''' (web dashboard):
** If <code>dnf</code>/<code>subscription-manager</code> can't reach Red Hat's CDN (DNS/outbound restrictions are common on this network), download RPMs manually from <code>download.thruk.org</code> and install with <code>rpm -ivh</code>.
** Build/install '''MK-Livestatus''' and load it as a Nagios event broker module — without this, Thruk has literally no data to show. Confirm via the Nagios log: <code>Event broker module '/usr/local/lib/mk-livestatus.o' initialized successfully.</code> This creates a socket at <code>/usr/local/nagios/var/rw/livestatus/live</code>.
** Verify the socket directly before blaming Thruk:
<syntaxhighlight lang="bash">
     printf 'GET hosts\nColumns: name state plugin_output\nLimit: 10\n\n' | \
       /usr/local/bin/unixcat /usr/local/nagios/var/rw/livestatus/live
</syntaxhighlight>
** Ensure <code>apache</code> has group access to the socket (<code>id apache</code>, test with <code>sudo -u apache unixcat ...</code>), then <code>systemctl restart httpd</code>.
# '''Import existing config objects''' via NagiosQL (Administration → Tools → Data import), pointed at <code>/usr/local/nagios/etc/objects/*.cfg</code>. Import in NagiosQL's recommended order:
<syntaxhighlight lang="text">
   commands → timeperiods → contacttemplates → contacts → contactgroups →
   hosttemplates → hosts → hostgroups → servicetemplates → services → servicegroups
</syntaxhighlight>
# '''Verify the Nagios service''':
<syntaxhighlight lang="bash">
   systemctl status nagios.service
   # expect: Active: active (running)
</syntaxhighlight>
# '''Recompile/validate plugins per device category''' before relying on them. Confirmed present/needed: NCPA plugin (<code>/root/nagios-4.5.9/nagios-plugins/plugins-python/check_ncpa.ini</code>, for Windows/Linux agent monitoring) and <code>check_nwc_health</code> (for network device status/bandwidth via SNMP).

== 4. Device onboarding process (Windows/Linux via NCPA) ==

Established host groups: '''linux-servers''', '''windows-servers''', '''network-printers''', '''switches'''. A <code>linux-server</code> host template exists and is reused across hosts.

# '''Add host''' (NagiosQL → Supervision → Host): hostname, IP, host group, template, mark Active/Registered, save.
# '''Faster path for repeat hosts''': clone an existing host (clone icon in the host list) rather than rebuilding from scratch — edit IP/name/description only.
# '''Add services per host''' (Supervision → Services): naming convention <code>SvcT_NCPA_<check>_usage</code> / description e.g. "CPU usage", choose the NCPA check command, supply the plugin argument (copy from a known-working example).
** '''Cloning a host is safe; cloning a service is not''' — each service's check command/arguments must be verified for the new context, they don't just work by virtue of being copied.
# '''Force an immediate check''' to validate rather than waiting for the schedule.
# '''Verify in Thruk''' once the check cycle completes.
# Watch for:
** '''NCPA port blocked by firewall''' → <code>firewall-cmd --add-port=5693/tcp --permanent && firewall-cmd --reload</code> on the target host.
** '''NCPA token/secret mismatch''' between agent config and command argument → causes blank/failed checks, re-verify the exact (case-sensitive) secret.
** '''Host-group miscategorization''' (e.g. a Linux box accidentally under <code>windows-servers</code>) → silently applies the wrong template; audit periodically.

== 5. Network device onboarding (switches/routers) — automated ==

Manual per-interface entry doesn't scale (one switch had 92 interfaces). A custom discovery script solves this:

# '''Prerequisite''': confirm SNMP is reachable on the target device first. This environment uses '''SNMPv3''' — username/auth/priv credentials must match exactly between the device and the script, or you'll see "credential does not match" / "unknown user" errors that look like script bugs but aren't.
# '''Discovery/generation script''' (custom, Python + shell, under <code>/opt</code> on <code>dcprdlmonitor02</code>): takes a target IP, SNMP-walks it, and generates a '''host''' file plus a '''services''' file under <code>/opt/generated/</code> — one status + one bandwidth service '''per discovered interface''', using the <code>check_nwc_health</code> plugin:
<syntaxhighlight lang="text">
   check_command   check_nwc_health_ifstatus!<if>
   check_command   check_nwc_health_ifusage!<if>!80!90
</syntaxhighlight>
   Applies to '''any device with ports/bandwidth''' — switch, router, or firewall. Also handles '''LAG (link aggregation) interfaces''', not just physical ports.
# '''Import into Nagios''': '''empty the import directory first''' (<code>/usr/local/nagios/etc/objects/import/</code> — leftover files from a previous device cause confused/wrong imports), copy in the generated host + services files, then in NagiosQL: '''Tools → Data import''', select both, import.
# '''Verify''': NagiosQL → Tools → Check configuration files (should report host/service counts with no errors), then Tools → write all config files, then wait a few minutes for status to move off "pending."
# '''Router/firewall caveat''': if a device has no SNMP enabled/reachable, this pipeline can't help — that's a networking-team blocker, not a Nagios config issue.
# '''Port bandwidth graphing gap''': <code>check_nwc_health_ifusage</code> reports numeric usage, but historical MRTG-based port graphing needs its own MRTG log file set up per device separately — don't assume it comes free.

== 6. Storage device onboarding (NetApp) ==

# '''Plugin''': <code>check-netapp-ng.pl</code> (Perl), reused from the existing XI server (<code>/usr/local/nagios/libexec/</code> on <code>dcprodnagios01</code>) — handles <code>SHELFINFO</code>, <code>FSSTATUS</code> (via SNMP OID <code>snmpfsOverallStatus</code>), and more.
# '''Reuse XI's existing NetApp service configs''' rather than rebuilding — six standard files: <code>Netapp_cpu.cfg</code>, <code>Netapp_disksummary.cfg</code>, <code>Netapp_fan.cfg</code>, <code>Netapp_shelf.cfg</code>, <code>NetApp_top_volumes.cfg</code>, <code>Netapp_uptime.cfg</code>, copied via <code>scp</code> from XI to Core.
# Import the same way as other devices (NagiosQL data import); expect the same transient <code>PENDING</code> status immediately after import — clears after the first active check cycle, not a fault.
# '''Grafana''' was discussed as a possible alternative/addition to PNP4Nagios for storage graphing specifically — not yet decided/implemented as of the source recordings.

== 7. Rollout plan & pacing (per kick-off agreement, confirmed in later sessions) ==

* Day 1: onboard one representative device from each category (Windows, Linux, network) to validate the end-to-end pipeline — subsequent devices in the same category replicate quickly once the first is proven.
* Vendor engineering pre-stages setup; client team (Chala, Narendra, Ashutosh and others) onboards hands-on with the vendor lead available for escalation only.
* Target: ~2 weeks from kick-off to a fully handed-off state.
* Inventory source: reuse the existing Excel device inventory from the original XI onboarding (expect minor additions, not a full rebuild).
* Network devices use '''SNMP walk (polling)''', not SNMP trap — an explicit design choice, not an oversight.
* Recurring practical blocker across sessions: '''finding a reference host''' already known-good in XI (e.g. "which IP is an already-monitored Windows server") — when the right person isn't available, this stalls onboarding until confirmed. Keep an up-to-date reference list per device category to avoid repeating this.

== 8. Access & operations notes ==

* Remote access: VPN (pending DGARM security-team approval) is preferred over WebEx/Zoom screen-share — enables after-hours/independent debugging without a client-side team member present at all times. In practice, sessions ran over Zoom remote control with frequent clipboard/control handoff friction — worth pushing on VPN approval to reduce this overhead.
* Escalation model: client's own team handles day-to-day onboarding; only genuine blockers get escalated back to the vendor lead.

== 9. Known issues & fixes log ==

{| class="wikitable"
|-
! Date
! Component
! Symptom
! Root cause
! Fix
|-
| 2026-07-17
| NagiosQL
| Cannot write/backup Nagios config files
| <code>/etc/nagiosql/{hosts,services,hostgroups,servicegroups,backup}</code> missing or not owned by web server user
| Create directory tree, <code>chown -R apache:apache</code>, <code>chmod -R 775</code> on <code>/etc/nagiosql</code>
|-
| 2026-07-17
| PNP4Nagios
| <code>sizeof(): Parameter must be an array or an object that implements Countable</code> in <code>data.php:979</code>
| PHP 8.x strict-typing break in PNP4Nagios 0.6.26
| Back up <code>data.php</code>, guard the <code>sizeof()</code> call with <code>is_array($pages) &&</code>
|-
| 2026-07-17
| PNP4Nagios
| No graphs/performance data at all
| Stock Core has no graphing; hosts/services not wired to PNP4Nagios
| Add <code>host-pnp</code>/<code>service-pnp</code> templates to <code>templates.cfg</code>
|-
| 2026-07-18
| Thruk
| <code>dnf</code>/<code>subscription-manager</code> can't install packages
| Outbound DNS/internet blocked to Red Hat's CDN on this network
| Download RPMs manually from <code>download.thruk.org</code>, install via <code>rpm -ivh</code>
|-
| 2026-07-18
| Thruk
| Shows no host/service data even though Nagios is running
| MK-Livestatus broker module not built/loaded
| Build/load <code>mk-livestatus</code>, verify via Nagios log + direct <code>unixcat</code> query
|-
| 2026-07-18
| Thruk/Apache
| Still can't read Livestatus data after module loads
| <code>apache</code> lacks group access to the livestatus socket
| Add <code>apache</code> to the required group, <code>systemctl restart httpd</code>
|-
| 2026-07-20
| NagiosQL/onboarding
| New NCPA host unreachable
| Host-level firewall blocking NCPA port
| <code>firewall-cmd --add-port=5693/tcp --permanent && firewall-cmd --reload</code>
|-
| 2026-07-20
| NagiosQL/onboarding
| NCPA check returns blank data
| Token/secret mismatch between agent and command argument
| Re-verify exact secret on both sides
|-
| 2026-07-20
| Thruk
| Switch port/bandwidth shows UNKNOWN: <code>check_mrtgtraf: Unable to open MRTG log</code>
| MRTG log file for that interface doesn't exist yet
| Set up MRTG logging for the switch's ports before relying on this check
|-
| 2026-07-20
| NagiosQL
| Cloned service doesn't work even though host is fine
| Cloning a service only duplicates config, not correctness of check command/args
| Verify/edit check command + <code>$ARGx$</code> values per service after cloning
|-
| 2026-07-21
| NagiosQL
| Host behaves per wrong template
| Host placed in the wrong host group (e.g. Linux box under <code>windows-servers</code>)
| Audit host-group membership, correct it
|-
| 2026-07-21
| Discovery script
| "Credential does not match" / "unknown user" against a switch
| SNMPv3 credentials in the script don't match the device
| Re-verify SNMPv3 username/auth/priv on both sides
|-
| 2026-07-21
| NagiosQL import
| Wrong/duplicate services imported after repeated script runs
| Leftover files in <code>/usr/local/nagios/etc/objects/import/</code> from a previous device
| Clear the import directory before each new import
|-
| 2026-07-21
| Discovery script
| Router won't onboard via the automation at all
| SNMP not enabled/reachable on that router
| Escalate to network team, not a Nagios-side fix
|-
| 2026-07-24
| NagiosQL
| Newly imported storage host shows <code>PENDING</code>
| Expected transient state — object hasn't been checked yet
| Wait for the first active check cycle; not a fault
|}

== 10. Open items / follow-ups from source material ==

* Confirm final decision on VPN vs. WebEx/Zoom for vendor remote access.
* Confirm inventory Excel has been shared/received and is current.
* Track plugin recompilation status per device category (Windows/Linux/network) — the highest-risk, least-templated part of the migration.
* Decide whether to reuse XI's <code>handle_nagioscore_event.php</code> / <code>handle_nagioscore_notification.php</code> for Core's notification handling.
* Decide on Grafana vs. PNP4Nagios (or both) for storage graphing.
* Set up MRTG logging per switch port as a standard step in the switch-onboarding checklist (currently a gap discovered mid-rollout, not a planned step).
* Maintain a living "known-good reference host per category" list (Windows/Linux/ network/storage) to avoid repeated stalls when the right person is unavailable.

== 11. Security note ==

One frame in the Thruk troubleshooting video shows a Red Hat <code>subscription-manager</code> username/password typed in plaintext in a terminal during screen-share. That credential is '''not reproduced anywhere in this documentation''' — recommend rotating it since it was captured on a recording that's now been reviewed/analyzed.

== See also ==
* [[DGARM Nagios Core - Infosys Kick-off Meeting Summary]]
* [[DGARM Nagios Core - PNP4Nagios Fix Session Summary]]
* [[DGARM Nagios Core - NagiosQL Fix Session Summary]]
* [[DGARM Nagios Core - Thruk Issue Rectification Summary]]
* [[DGARM Nagios Core - Deployment Process Summary]]
* [[DGARM Nagios Core - Network Switch Onboarding Summary]]
* [[DGARM Nagios Core - Storage Onboarding Summary]]

[[Category:INFOSYS_DGARM_NAGIOS]]