Admin: Auto-created from uploaded PDF text extraction

2026-07-26T16:27:10Z

Auto-created from uploaded PDF text extraction

New page

''Auto-generated from the uploaded PDF [[:File:PCS_Cluster_failover-process.pdf|PCS_Cluster_failover-process.pdf]]. This is an extracted-text rendering for searchability; see the original PDF for exact formatting, diagrams, tables, and images.''

<pre>
<nowiki>
SOP: PCS Cluster Working Process & Failover
Management
Cluster Details

 Cluster Name: nagios_cluster
 Cluster Stack: Corosync / Pacemaker
 Preferred Node: app2-2.spectra.co
 Failover Node: app2-1.spectra.co
 DRBD Resource: r0
 MySQL: Master-Master Replication
 DRBD Mount Point Resource: fs_drbd

1. Normal Cluster Behavior
 By default, all resources (VIP, DRBD master, services) are hosted on app2-
2.spectra.co.
 If app2-2.spectra.co becomes unreachable or is manually set to standby, cluster
resources failover to app2-1.spectra.co.

2. Failover Behavior
 When app2-2.spectra.co is offline/unreachable, resources move to app2-
1.spectra.co.
 Once app2-2.spectra.co comes back online, PCS automatically migrates resources
back to the preferred node (as per location constraints).
 This migration may take a few minutes depending on service/resource status.

3. Manual Standby & Recovery Steps
Put Node in Standby

pcs node standby app2-2.spectra.co
Un-standby Node (Bring Back to Cluster)

pcs node unstandby app2-2.spectra.co

4. Cleanup Failed Resource Actions
After any failover or recovery, always clean failed resource states:

pcs resource cleanup

5. Verification Steps After Failover or Recovery
A. Check DRBD Sync Status

drbdadm status r0

 Ensure both nodes are in sync.
 Preferred node should show as Primary.

B. Check MySQL Replication (Master-Master)

mysql -e "SHOW SLAVE STATUS\G"

 Look for Seconds_Behind_Master, Slave_IO_Running, Slave_SQL_Running values.
 Both should show replication in healthy state.

C. Check Cluster Status

pcs status

 Verify all resources are running on the correct node ( app2-2.spectra.co).

 Example:

* Promoted: [ app2-2.spectra.co ]

* Started: [ app2-2.spectra.co ]

6. Logs & History
Check logs for any anomalies:

journalctl -xe | grep pacemaker

cat /var/log/pacemaker.log

7. Important Notes
 Always verify DRBD and MySQL replication after failover or maintenance.
 Ensure fencing (fence_ipmilan) is working correctly for both nodes.
 If frequent failures occur, check the monitor action results and logs for root cause.
</nowiki>
</pre>

PCS Cluster failover-process - Revision history

Admin: Auto-created from uploaded PDF text extraction