PCS Cluster failover-process

Auto-generated from the uploaded PDF PCS_Cluster_failover-process.pdf. This is an extracted-text rendering for searchability; see the original PDF for exact formatting, diagrams, tables, and images.

SOP: PCS Cluster Working Process & Failover
Management
Cluster Details

      Cluster Name: nagios_cluster
      Cluster Stack: Corosync / Pacemaker
      Preferred Node: app2-2.spectra.co
      Failover Node: app2-1.spectra.co
      DRBD Resource: r0
      MySQL: Master-Master Replication
      DRBD Mount Point Resource: fs_drbd


1. Normal Cluster Behavior
      By default, all resources (VIP, DRBD master, services) are hosted on app2-
       2.spectra.co.
      If app2-2.spectra.co becomes unreachable or is manually set to standby, cluster
       resources failover to app2-1.spectra.co.


2. Failover Behavior
      When app2-2.spectra.co is offline/unreachable, resources move to app2-
       1.spectra.co.
      Once app2-2.spectra.co comes back online, PCS automatically migrates resources
       back to the preferred node (as per location constraints).
      This migration may take a few minutes depending on service/resource status.


3. Manual Standby & Recovery Steps
Put Node in Standby

pcs node standby app2-2.spectra.co
Un-standby Node (Bring Back to Cluster)

pcs node unstandby app2-2.spectra.co

4. Cleanup Failed Resource Actions
After any failover or recovery, always clean failed resource states:

pcs resource cleanup


5. Verification Steps After Failover or Recovery
A. Check DRBD Sync Status

drbdadm status r0

      Ensure both nodes are in sync.
      Preferred node should show as Primary.

B. Check MySQL Replication (Master-Master)

 mysql -e "SHOW SLAVE STATUS\G"

      Look for Seconds_Behind_Master, Slave_IO_Running, Slave_SQL_Running values.
      Both should show replication in healthy state.

C. Check Cluster Status

pcs status

 Verify all resources are running on the correct node ( app2-2.spectra.co).

 Example:

* Promoted: [ app2-2.spectra.co ]

* Started: [ app2-2.spectra.co ]

6. Logs & History
Check logs for any anomalies:

journalctl -xe | grep pacemaker

cat /var/log/pacemaker.log


7. Important Notes
      Always verify DRBD and MySQL replication after failover or maintenance.
      Ensure fencing (fence_ipmilan) is working correctly for both nodes.
      If frequent failures occur, check the monitor action results and logs for root cause.