Spectra Nagios Project Completion Report

Auto-generated from the uploaded PDF Spectra_Nagios_Project_Completion_Report.pdf. This is an extracted-text rendering for searchability; see the original PDF for exact formatting, diagrams, tables, and images.

Project Completion Report

Project Title: Nagios XI Upgrade and HA Implementation
Project Duration: March 2025 – June 2025
Prepared For: spectra.co
Prepared By: Tetra Information Services LTD.
Date: 03 July 2025


Table of Contents

  1.   Executive Summary
  2.   Project Objectives
  3.   Scope of Work
  4.   Implementation Timeline
  5.   Detailed Activities and Outcomes
          o Module 1: Design HA Architecture
          o Module 2: OS and Application Upgrade
          o Module 3: HA Configuration for Main
          o Module 4: HA Configuration for Probe
          o Module 5: Database Issue Resolution
          o Module 6: Main-Probe Synchronization
          o Module 7: API & Integration
          o Module 8: Hardware and Software Planning
  6.   Key Challenges & Resolutions
  7.   Recommendations
  8.   Conclusion
  9.   Appendices

1. Executive Summary

This report outlines the successful completion of the Nagios XI Upgrade and
High Availability (HA) Implementation project carried out for spectra.co by
Tetra Information Services LTD between March and June 2025.

The goal was to revamp the monitoring system by upgrading the outdated
CentOS-based setup, introducing redundancy through HA configurations,
and improving reliability, database performance, and scalability. This project
aligns the system for anticipated growth over the next 3–5 years and
provides a resilient infrastructure to support 15,000+ hosts and over 150,000
monitored services.

All modules planned under this initiative have been executed and validated
in collaboration with the client. Deliverables include new HA architecture,
upgraded OS/Nagios versions, optimized databases, real-time configuration
sync, and hardware recommendations for future scaling.


2. Project Objectives

2.1 Core Objectives

      To implement a robust and fault-tolerant High Availability (HA)
       configuration for both Main and Probe monitoring nodes.
      To upgrade the Operating System from CentOS 7 (EOL) to Ubuntu
       20.04 LTS.
      To upgrade Nagios XI from version 5.8.9 to the latest stable release,
       2024R1.4.4.
      To address database performance issues, backup failures, and table
       bloating.
      To ensure real-time synchronization between Main and Probe
       environments.
      To stabilize integration with API and Service Request (SR)
       automation tools.

2.2 Key Deliverables

      HA design documentation finalized with the client
      Clean OS installations and configuration
      Upgraded Nagios XI instances
      Master-Master MySQL Replication and DRBD setup
      Automated database cleanup configurations
      Resolved main-probe sync lags
      Reviewed and stabilized API/SR behavior
      Capacity roadmap to scale up to 150K services


3. Scope of Work

The project scope included infrastructure redesign, HA implementation, and
full-stack upgrades with minimal downtime. The following major
components were included:

      Planning and designing of Main/Probe HA infrastructure
      Execution of OS and Nagios XI upgrades on both nodes
      Re-establishing DRBD and MySQL replication clusters
      Resolving historical database errors
      Ensuring production-ready probe-main synchronization
      Addressing post-upgrade API/SR issues
      Providing a future hardware expansion strategy


4. Implementation Timeline

Instead of a table, here’s the timeline as a step-wise breakdown with date
markers:

      19–25 March 2025 – HA architecture finalized with Spectra
      26 March – 07 May 2025 – SOC (Secondary) Node OS and Nagios
       installation completed
      28 March – 13 May 2025 – Configuration and database migration to
       new node
      16–19 May 2025 – Switchover to new SOC node completed
      20–23 May 2025 – Primary Node OS/Nagios rebuilt and upgraded
      24 May – 10 June 2025 – Main and Probe HA clusters established
      10–25 June 2025 – Final testing, sync tuning, and hardware roadmap
       delivered

Each activity followed strict checklists, supported by configuration
screenshots and validations reviewed by the client.


5. Detailed Activities and Outcomes

Module 1: HA Architecture Design

      Multiple HA models were proposed, tested, and validated.
      Final decision involved using MySQL master-master replication
       and DRBD for file-level sync.
      No major new hardware purchases were required at this stage.
      Documentation of cluster failover scenarios was shared with Spectra.

Module 2: OS and Application Upgrade

2.1 HA Break and Initial Preparation

      HA was temporarily disabled.
      Services temporarily shifted to the Main node.

2.2 Secondary Node Installation

      Ubuntu 22.04 LTS was first used, but changed to 20.04 LTS as per
       client feedback.
      Nagios XI 5.7 installed successfully, preparing for upgrade.

2.3 Configuration Migration

      Nagios configurations, DBs, custom scripts were moved from the old
       system.
      Data validation, backup recovery, and plugin compatibility tests were
       performed.

2.4 SOC Go-Live

      Final production cutover took place on 19 May 2025.
      Monitoring services resumed without loss of data or downtime.

2.5 Primary Node Upgrade

      Rebuilt Primary Node from scratch with Ubuntu 20.04 LTS.
      Upgraded Nagios XI to 2024R1.4.4.
      Configured DRBD and MySQL replication to secondary node.

Module 3: HA Configuration – Main

     Cluster fencing, shared IP configuration, and failover scripting
      implemented.
     DRBD replication verified for logs, configs, and backups.
     MySQL master-master pair synchronized and validated.
     All services continued without interruption during switchover
      simulations.


Module 4: HA Configuration – Probe

     Probe node was duplicated and configured with identical packages.
     MySQL cluster and DRBD replication implemented as in Main node.
     Configured to handle real-time alerts and monitoring data from
      remote devices.


Module 5: Database Issue Resolution

     Backups were previously failing due to large .ibd files and table
      locks.
     Old tables (over 5GB each) were split and purged with automation.
     Enabled automated cleanup scripts in Nagios XI.
     Post-cleanup system is stable, with backup jobs running nightly.

Module 6: Main–Probe Synchronization

     Prior to the upgrade, there were 15–20 minute lags in alarm sync.
     Post-upgrade, the sync is real-time, with instant propagation of
      changes.
     This led to faster deployment, reduced troubleshooting, and higher
      monitoring accuracy.


Module 7: API & Integration

     SR automation previously created duplicate or missing records.
     Integration scripts were revalidated and corrected with vendor
      support.
     Delay in SR acknowledgment reduced from 5–10 mins to <2 mins on
      average.
     Final integration success rate is now above 97%.


Module 8: Hardware and Software Planning

     Final architecture allows for growth up to 150K services.
     Main HA servers:
      / = 600GB, /data = 5.2TB, RAM = 128GB, vCPU = 12
     Probe nodes:
      / = 500GB, /data = 500GB, RAM = 32GB, vCPU = 4–6
     Hardware scaling roadmap provided and accepted by Spectra.

6. Key Challenges & Resolutions

      OS version conflict resolved by switching from 22.04 LTS to 20.04
       LTS.
      DRBD sync stalls fixed by tuning resource parameters.
      Partition size mismatches addressed via disk resizing.
      Backup failures resolved through database optimization.


7. Recommendations

      Continue monitoring synchronization and replication status weekly.
      Maintain quarterly health checks on MySQL size and HA logs.
      Keep probe configuration updated for future monitoring growth.
      Begin planning for cloud-based hybrid Nagios XI extension in 2026.


8. Conclusion

The project has been executed successfully and fulfills all design,
functionality, and availability expectations. The Nagios XI infrastructure is
now fault-tolerant, stable, and scalable for years to come.

Final Handover Date: 25 June 2025
Project Closure Status: ✔ Fully Completed

9. Appendices

9.1 Reference Document List

The following documents were created, reviewed, and shared with
spectra.co throughout the project lifecycle. Each document supports the
execution and validation of key modules in the HA architecture, upgrade
strategy, configuration, and optimization.

   1. Proposed Monitoring Architecture – Spectra
      Describes the final HA design including main and probe node layout,
      resource planning, and IP failover strategy.
   2. Solution Document – Nagios HA Comparison with Architecture
      Diagram
      Provides comparative analysis of different HA deployment methods
      and justifies selected approach for Spectra.
   3. PCS Cluster Failover – Process
      Step-by-step documentation outlining cluster fencing, VIP migration,
      and automatic failover process validation.
   4. Spectra_SOC Nagios Optimization – Project Status (Final Sheet)
      Status tracking sheet showing all completed activities, timelines,
      dependencies, and mitigation actions.
   5. Nagios XI Cluster Setup Checklist
      Comprehensive deployment checklist used by engineers to ensure no
      step was missed during HA and sync configuration.