DGARM Nagios Core - Implementation Runbook

Status: Final — based on all 8 source recordings (2026-07-10 through 2026-07-24).

Sources:

• Dgarm-Nagios-Core-infosys-Kick-off-Meeting-2026-07-10_11-30-52.mkv
• DGARM-Nagios-Core-Fixed-PNP4Nagios-2026-07-17_11-33-20.mkv
• DGARM-Nagios-Core-Fixed-NagiosQL-2026-07-17_11-33-20.mkv-2026-07-17_14-24-09.mkv
• DGARM-Infosys-Nagios-core-Deployment-Thruk-issue-rectification-2026-07-18_11-23-23.mkv
• DGARM-Infosys-Nagios-Core-deployment-Process.mp4
• DGRAM-Infosys-Nagios-Core-Deployment-Network-switch.mp4
• DGRAM-Infosys-Nagios-Core-Deployment-.storage-1.mp4
• DGRAM-Infosys-Nagios-Core-Deployment-storage-2.mp4

1. Background & scope

DGARM's device monitoring currently lives in Nagios XI, which has hit a host-count limit. Devices that don't fit in XI are being onboarded into a new Nagios Core deployment instead, reusing XI's existing templates, plugins, and inventory where possible rather than building from scratch.

Device categories in scope: Windows servers, Linux servers (physical + virtual), network/security devices (switches, routers, WAF, sandbox appliances), and storage (NetApp). Laptops are explicitly out of scope.

Primary hosts referenced throughout: Nagios Core server dcprdlmonitor02 (10.150.24.167), existing Nagios XI server dcprodnagios01.

2. Architecture decisions

Concern	XI (old)	Core (new) — decision
Config management	Built-in GUI	NagiosQL 3.5.0 (web-based CCM)
Graphing	Built-in	PNP4Nagios (RRD-based) — not present in stock Core
Web dashboard	Built-in	Thruk — Core's default UI was judged too primitive for client-facing use
Live data backend	N/A	MK-Livestatus — required broker module; Thruk has no data of its own without it
Host/service templates	Native	Copy templates out of XI and reuse in Core via NagiosQL
Plugins	Pre-compiled/working in XI	Not assumed portable — each plugin must be re-obtained/recompiled and tested against the Core host
Network device discovery	Manual per-device in XI GUI	Custom SNMP-based script (see §5) — manual entry doesn't scale (one switch had 92 interfaces)
Notifications	XI's own PHP handler scripts	XI already has handle_nagioscore_event.php / handle_nagioscore_notification.php under /usr/local/nagiosxi/scripts/ — evaluate reusing these before writing new ones

3. Install sequence

1. Base install: Nagios Core + dependencies on the target host (~1 week estimated for base install/config).
2. Install NagiosQL as the configuration manager:

• • Deploy at a path served by the web server (/var/www/html/nagiosql → http://<host>/nagiosql/).
  • Create the expected directory tree before first use:

<syntaxhighlight lang="bash">

    sudo mkdir -p /etc/nagiosql/hosts /etc/nagiosql/services \
         /etc/nagiosql/hostgroups /etc/nagiosql/servicegroups \
         /etc/nagiosql/backup/hosts /etc/nagiosql/backup/services
    sudo chown -R apache:apache /etc/nagiosql
    sudo chmod -R 775 /etc/nagiosql

</syntaxhighlight>

• • If SELinux is Enforcing (check with getenforce), also fix SELinux context/labeling so Apache can write there, not just POSIX permissions.
  • In Administration → Config targets, confirm Nagios base directory (/usr/local/nagios/etc/), config/cgi/resource file paths all match the actual install.

1. Install PNP4Nagios for graphing:

• • If the UI throws sizeof(): Parameter must be an array or an object that implements Countable in application/models/data.php, that's a PHP 8.x compatibility break in PNP4Nagios 0.6.26 — see fix in §6.
  • Wire hosts/services to graphs by adding to /usr/local/nagios/etc/objects/templates.cfg:

<syntaxhighlight lang="text">

    define host {
        name        host-pnp
        action_url  /pnp4nagios/index.php/graph?host=$HOSTNAME$&srv=_HOST_
        register    0
    }
    define service {
        name        service-pnp
        action_url  /pnp4nagios/index.php/graph?host=$HOSTNAME$&srv=$SERVICEDESC$
        register    0
    }

</syntaxhighlight>

1. Install Thruk (web dashboard):

• • If dnf/subscription-manager can't reach Red Hat's CDN (DNS/outbound restrictions are common on this network), download RPMs manually from download.thruk.org and install with rpm -ivh.
  • Build/install MK-Livestatus and load it as a Nagios event broker module — without this, Thruk has literally no data to show. Confirm via the Nagios log: Event broker module '/usr/local/lib/mk-livestatus.o' initialized successfully. This creates a socket at /usr/local/nagios/var/rw/livestatus/live.
  • Verify the socket directly before blaming Thruk:

<syntaxhighlight lang="bash">

    printf 'GET hosts\nColumns: name state plugin_output\nLimit: 10\n\n' | \
      /usr/local/bin/unixcat /usr/local/nagios/var/rw/livestatus/live

</syntaxhighlight>

• • Ensure apache has group access to the socket (id apache, test with sudo -u apache unixcat ...), then systemctl restart httpd.

1. Import existing config objects via NagiosQL (Administration → Tools → Data import), pointed at /usr/local/nagios/etc/objects/*.cfg. Import in NagiosQL's recommended order:

<syntaxhighlight lang="text">

  commands → timeperiods → contacttemplates → contacts → contactgroups →
  hosttemplates → hosts → hostgroups → servicetemplates → services → servicegroups

</syntaxhighlight>

1. Verify the Nagios service:

<syntaxhighlight lang="bash">

  systemctl status nagios.service
  # expect: Active: active (running)

</syntaxhighlight>

1. Recompile/validate plugins per device category before relying on them. Confirmed present/needed: NCPA plugin (/root/nagios-4.5.9/nagios-plugins/plugins-python/check_ncpa.ini, for Windows/Linux agent monitoring) and check_nwc_health (for network device status/bandwidth via SNMP).

4. Device onboarding process (Windows/Linux via NCPA)

Established host groups: linux-servers, windows-servers, network-printers, switches. A linux-server host template exists and is reused across hosts.

1. Add host (NagiosQL → Supervision → Host): hostname, IP, host group, template, mark Active/Registered, save.
2. Faster path for repeat hosts: clone an existing host (clone icon in the host list) rather than rebuilding from scratch — edit IP/name/description only.
3. Add services per host (Supervision → Services): naming convention SvcT_NCPA_<check>_usage / description e.g. "CPU usage", choose the NCPA check command, supply the plugin argument (copy from a known-working example).

• • Cloning a host is safe; cloning a service is not — each service's check command/arguments must be verified for the new context, they don't just work by virtue of being copied.

1. Force an immediate check to validate rather than waiting for the schedule.
2. Verify in Thruk once the check cycle completes.
3. Watch for:

• • NCPA port blocked by firewall → firewall-cmd --add-port=5693/tcp --permanent && firewall-cmd --reload on the target host.
  • NCPA token/secret mismatch between agent config and command argument → causes blank/failed checks, re-verify the exact (case-sensitive) secret.
  • Host-group miscategorization (e.g. a Linux box accidentally under windows-servers) → silently applies the wrong template; audit periodically.

5. Network device onboarding (switches/routers) — automated

Manual per-interface entry doesn't scale (one switch had 92 interfaces). A custom discovery script solves this:

1. Prerequisite: confirm SNMP is reachable on the target device first. This environment uses SNMPv3 — username/auth/priv credentials must match exactly between the device and the script, or you'll see "credential does not match" / "unknown user" errors that look like script bugs but aren't.
2. Discovery/generation script (custom, Python + shell, under /opt on dcprdlmonitor02): takes a target IP, SNMP-walks it, and generates a host file plus a services file under /opt/generated/ — one status + one bandwidth service per discovered interface, using the check_nwc_health plugin:

<syntaxhighlight lang="text">

  check_command   check_nwc_health_ifstatus!<if>
  check_command   check_nwc_health_ifusage!<if>!80!90

</syntaxhighlight>

  Applies to any device with ports/bandwidth — switch, router, or firewall. Also handles LAG (link aggregation) interfaces, not just physical ports.

1. Import into Nagios: empty the import directory first (/usr/local/nagios/etc/objects/import/ — leftover files from a previous device cause confused/wrong imports), copy in the generated host + services files, then in NagiosQL: Tools → Data import, select both, import.
2. Verify: NagiosQL → Tools → Check configuration files (should report host/service counts with no errors), then Tools → write all config files, then wait a few minutes for status to move off "pending."
3. Router/firewall caveat: if a device has no SNMP enabled/reachable, this pipeline can't help — that's a networking-team blocker, not a Nagios config issue.
4. Port bandwidth graphing gap: check_nwc_health_ifusage reports numeric usage, but historical MRTG-based port graphing needs its own MRTG log file set up per device separately — don't assume it comes free.

6. Storage device onboarding (NetApp)

1. Plugin: check-netapp-ng.pl (Perl), reused from the existing XI server (/usr/local/nagios/libexec/ on dcprodnagios01) — handles SHELFINFO, FSSTATUS (via SNMP OID snmpfsOverallStatus), and more.
2. Reuse XI's existing NetApp service configs rather than rebuilding — six standard files: Netapp_cpu.cfg, Netapp_disksummary.cfg, Netapp_fan.cfg, Netapp_shelf.cfg, NetApp_top_volumes.cfg, Netapp_uptime.cfg, copied via scp from XI to Core.
3. Import the same way as other devices (NagiosQL data import); expect the same transient PENDING status immediately after import — clears after the first active check cycle, not a fault.
4. Grafana was discussed as a possible alternative/addition to PNP4Nagios for storage graphing specifically — not yet decided/implemented as of the source recordings.

7. Rollout plan & pacing (per kick-off agreement, confirmed in later sessions)

• Day 1: onboard one representative device from each category (Windows, Linux, network) to validate the end-to-end pipeline — subsequent devices in the same category replicate quickly once the first is proven.
• Vendor engineering pre-stages setup; client team (Chala, Narendra, Ashutosh and others) onboards hands-on with the vendor lead available for escalation only.
• Target: ~2 weeks from kick-off to a fully handed-off state.
• Inventory source: reuse the existing Excel device inventory from the original XI onboarding (expect minor additions, not a full rebuild).
• Network devices use SNMP walk (polling), not SNMP trap — an explicit design choice, not an oversight.
• Recurring practical blocker across sessions: finding a reference host already known-good in XI (e.g. "which IP is an already-monitored Windows server") — when the right person isn't available, this stalls onboarding until confirmed. Keep an up-to-date reference list per device category to avoid repeating this.

8. Access & operations notes

• Remote access: VPN (pending DGARM security-team approval) is preferred over WebEx/Zoom screen-share — enables after-hours/independent debugging without a client-side team member present at all times. In practice, sessions ran over Zoom remote control with frequent clipboard/control handoff friction — worth pushing on VPN approval to reduce this overhead.
• Escalation model: client's own team handles day-to-day onboarding; only genuine blockers get escalated back to the vendor lead.

9. Known issues & fixes log

Date	Component	Symptom	Root cause	Fix
2026-07-17	NagiosQL	Cannot write/backup Nagios config files	/etc/nagiosql/{hosts,services,hostgroups,servicegroups,backup} missing or not owned by web server user	Create directory tree, chown -R apache:apache, chmod -R 775 on /etc/nagiosql
2026-07-17	PNP4Nagios	sizeof(): Parameter must be an array or an object that implements Countable in data.php:979	PHP 8.x strict-typing break in PNP4Nagios 0.6.26	Back up data.php, guard the sizeof() call with is_array($pages) &&
2026-07-17	PNP4Nagios	No graphs/performance data at all	Stock Core has no graphing; hosts/services not wired to PNP4Nagios	Add host-pnp/service-pnp templates to templates.cfg
2026-07-18	Thruk	dnf/subscription-manager can't install packages	Outbound DNS/internet blocked to Red Hat's CDN on this network	Download RPMs manually from download.thruk.org, install via rpm -ivh
2026-07-18	Thruk	Shows no host/service data even though Nagios is running	MK-Livestatus broker module not built/loaded	Build/load mk-livestatus, verify via Nagios log + direct unixcat query
2026-07-18	Thruk/Apache	Still can't read Livestatus data after module loads	apache lacks group access to the livestatus socket	Add apache to the required group, systemctl restart httpd
2026-07-20	NagiosQL/onboarding	New NCPA host unreachable	Host-level firewall blocking NCPA port	firewall-cmd --add-port=5693/tcp --permanent && firewall-cmd --reload
2026-07-20	NagiosQL/onboarding	NCPA check returns blank data	Token/secret mismatch between agent and command argument	Re-verify exact secret on both sides
2026-07-20	Thruk	Switch port/bandwidth shows UNKNOWN: check_mrtgtraf: Unable to open MRTG log	MRTG log file for that interface doesn't exist yet	Set up MRTG logging for the switch's ports before relying on this check
2026-07-20	NagiosQL	Cloned service doesn't work even though host is fine	Cloning a service only duplicates config, not correctness of check command/args	Verify/edit check command + $ARGx$ values per service after cloning
2026-07-21	NagiosQL	Host behaves per wrong template	Host placed in the wrong host group (e.g. Linux box under windows-servers)	Audit host-group membership, correct it
2026-07-21	Discovery script	"Credential does not match" / "unknown user" against a switch	SNMPv3 credentials in the script don't match the device	Re-verify SNMPv3 username/auth/priv on both sides
2026-07-21	NagiosQL import	Wrong/duplicate services imported after repeated script runs	Leftover files in /usr/local/nagios/etc/objects/import/ from a previous device	Clear the import directory before each new import
2026-07-21	Discovery script	Router won't onboard via the automation at all	SNMP not enabled/reachable on that router	Escalate to network team, not a Nagios-side fix
2026-07-24	NagiosQL	Newly imported storage host shows PENDING	Expected transient state — object hasn't been checked yet	Wait for the first active check cycle; not a fault

10. Open items / follow-ups from source material

• Confirm final decision on VPN vs. WebEx/Zoom for vendor remote access.
• Confirm inventory Excel has been shared/received and is current.
• Track plugin recompilation status per device category (Windows/Linux/network) — the highest-risk, least-templated part of the migration.
• Decide whether to reuse XI's handle_nagioscore_event.php / handle_nagioscore_notification.php for Core's notification handling.
• Decide on Grafana vs. PNP4Nagios (or both) for storage graphing.
• Set up MRTG logging per switch port as a standard step in the switch-onboarding checklist (currently a gap discovered mid-rollout, not a planned step).
• Maintain a living "known-good reference host per category" list (Windows/Linux/ network/storage) to avoid repeated stalls when the right person is unavailable.

11. Security note

One frame in the Thruk troubleshooting video shows a Red Hat subscription-manager username/password typed in plaintext in a terminal during screen-share. That credential is not reproduced anywhere in this documentation — recommend rotating it since it was captured on a recording that's now been reviewed/analyzed.

See also

• DGARM Nagios Core - Infosys Kick-off Meeting Summary
• DGARM Nagios Core - PNP4Nagios Fix Session Summary
• DGARM Nagios Core - NagiosQL Fix Session Summary
• DGARM Nagios Core - Thruk Issue Rectification Summary
• DGARM Nagios Core - Deployment Process Summary
• DGARM Nagios Core - Network Switch Onboarding Summary
• DGARM Nagios Core - Storage Onboarding Summary