Editing CSC Cloudstack Mariadb Galera Cluster Corruption issue

[[category:CSC]]
[[Category:Cloudstack]]

Issue : The Mariadb Galera Cluster Got Stuck , Data Corruption and Cloudstack Failed 

Diagnosis and Solution Process 

1. Stop the Mariadb process on both the nodes . if not successful , disable , mask and then kill the Process on both the node .
 
 ''ps -ef | grep mysql'' 
 OR
 ''systemctl status mariadb''
 if not successful then do 
 ''systemctl disable mariadb'' 
 ''systemctl mask mariadb''
 ''pkill -9 -u mysql'' 
 ''systemctl status mariadb''

2.This step is only preferred , but you can use own judgement for which node's data will be more reliable . 
 sudo -u mysql mariadbd --wsrep-recover # run on both nodes 
 What you are looking for: You will see a line that looks like this: ... [Note] WSREP: Recovered position: 5a7b3b-.... : 10540
 The number after the colon (e.g., 10540) is your Sequence Number.
 The Winner: The node with the HIGHER number.

3. On the Winner Node ONLY:( Or on the Preferred node )

 Edit /var/lib/mysql/grastate.dat.
 Set safe_to_bootstrap: 1.
 Unmask and start:
 systemctl unmask mariadb
 galera_new_cluster

4. Verify Mysql is running on the Node 
 mysql -u root -p -e "SHOW STATUS LIKE 'wsrep_cluster_size';"
 Target Output: Value should be 1.

5. Now we need to bring other Node back into the cluster. Since we "masked" it earlier to stop the restart loop, we must unmask it first.
 systemctl unmask mariadb
 systemctl start mariadb
 SHOW STATUS LIKE 'wsrep_cluster_size';
 Target Output: Value should be 2.

6, Enable Autostart: If you disabled the services earlier, re-enable them so they start on reboot:
 systemctl enable mariadb