Fall back Strategy for the cluster-infrastructure - NIIT Tobacco Project

Objective

The objective of this document is when the primary node of cluster go down and the cluter shift on the secondary node then how we can shift it back on the primary node without any problem and data lose. This is to ensure that our Fail Back to primary server is ensured with no data loss.

Fail Back Steps

The following steps has been followed to bring the cluster on the primary node

1. Submit a down time resquest to the Concerning officials.
2. After getting approval for the down time insure that no activity happens on the servers.
3. Make the Primary Up
4. Insure the data replication (drbd ) should be completed from the secondary node to primary node. Check the DRBD checks for referance .
5. Down the secondary node manually.
6. Chcek all the application and data base integrity test.
7. Once the cluster shift to the primary node again check the data and application.
8. Make the Secondary Up . Secondary will Join the cluster automatically
9. Follow the Basic Santity test with application tests before releasing the Application .

Do's and Dont's

Dont's

1. No activity shold be happen on the any of server while shifting the cluster. Ensure that downtime process is followed
2. Don't do any change management or change any file on any of servers.
3. Don't login on the server via ssh or any other sevices like vnc.
4. Don't power off the any server.
5. Don't plug out the network cable from any server.
6. Don't switch off any network device i.e gateway or firewall.
7. Don't change any configuration file of any servers.

Do's

1. Make sure replication has been done of data from secondary to primary.
2. Bring the secondary server down manually so that cluster will be shift from secondary to primary.
3. During the time the Primary is down , please increase the frequency of basic sanity and health checks to ensuring the availability of application on secondary server .
4. Check all network devices and make sure all would be active.
5. Once cluster shifting has been done check for data and application via data integraty checks , sanity checks , health checks and configuration tests.