CSC Cloudstack Mariadb Galera Cluster Corruption issue

Issue : The Mariadb Galera Cluster Got Stuck , Data Corruption and Cloudstack Failed

Diagnosis and Solution Process

1. Stop the Mariadb process on both the nodes . if not successful , disable , mask and then kill the Process on both the node .

ps -ef | grep mysql 
OR
systemctl status mariadb
if not successful then do 
systemctl disable mariadb 
systemctl mask mariadb
pkill -9 -u mysql 
systemctl status mariadb

2.This step is only preferred , but you can use own judgement for which node's data will be more reliable .

sudo -u mysql mariadbd --wsrep-recover # run on both nodes 
What you are looking for: You will see a line that looks like this: ... [Note] WSREP: Recovered position: 5a7b3b-.... : 10540
The number after the colon (e.g., 10540) is your Sequence Number.
The Winner: The node with the HIGHER number.

3. On the Winner Node ONLY:( Or on the Preferred node )

Edit /var/lib/mysql/grastate.dat.
Set safe_to_bootstrap: 1.
Unmask and start:
systemctl unmask mariadb
galera_new_cluster

4. Verify Mysql is running on the Node

mysql -u root -p -e "SHOW STATUS LIKE 'wsrep_cluster_size';"
Target Output: Value should be 1.

5. Now we need to bring other Node back into the cluster. Since we "masked" it earlier to stop the restart loop, we must unmask it first.

systemctl unmask mariadb
systemctl start mariadb
SHOW STATUS LIKE 'wsrep_cluster_size';
Target Output: Value should be 2.

6, Enable Autostart: If you disabled the services earlier, re-enable them so they start on reboot:

systemctl enable mariadb