Editing RCA REPORT FOR RAMNATHPURA - NIIT Tobacco Cluster Failure

[[category:NIIT-Tobacco Board Project]]


<center>Root Cause Analysis (RCA)</center>


{| style="border-spacing:0;"
| style="border-top:0.5pt solid #000000;border-bottom:0.5pt solid #000000;border-left:0.5pt solid #000000;border-right:none;padding-top:0in;padding-bottom:0in;padding-left:0.075in;padding-right:0.075in;"| # 


| style="border-top:0.5pt solid #000000;border-bottom:0.5pt solid #000000;border-left:0.5pt solid #000000;border-right:none;padding-top:0in;padding-bottom:0in;padding-left:0.075in;padding-right:0.075in;"| Location 
| style="border:0.5pt solid #000000;padding-top:0in;padding-bottom:0in;padding-left:0.075in;padding-right:0.075in;"| Ramnathpuram , Karnataka , Plaform No 63

|-
| style="border-top:0.5pt solid #000000;border-bottom:0.5pt solid #000000;border-left:0.5pt solid #000000;border-right:none;padding-top:0in;padding-bottom:0in;padding-left:0.075in;padding-right:0.075in;"| # 


| style="border-top:0.5pt solid #000000;border-bottom:0.5pt solid #000000;border-left:0.5pt solid #000000;border-right:none;padding-top:0in;padding-bottom:0in;padding-left:0.075in;padding-right:0.075in;"| Incident ticket Number
| style="border:0.5pt solid #000000;padding-top:0in;padding-bottom:0in;padding-left:0.075in;padding-right:0.075in;"| 2013020110000595

|-
| style="border-top:0.5pt solid #000000;border-bottom:0.5pt solid #000000;border-left:0.5pt solid #000000;border-right:none;padding-top:0in;padding-bottom:0in;padding-left:0.075in;padding-right:0.075in;"| # 


| style="border-top:0.5pt solid #000000;border-bottom:0.5pt solid #000000;border-left:0.5pt solid #000000;border-right:none;padding-top:0in;padding-bottom:0in;padding-left:0.075in;padding-right:0.075in;"| Priority
| style="border:0.5pt solid #000000;padding-top:0in;padding-bottom:0in;padding-left:0.075in;padding-right:0.075in;"| 1

|-
| style="border-top:0.5pt solid #000000;border-bottom:0.5pt solid #000000;border-left:0.5pt solid #000000;border-right:none;padding-top:0in;padding-bottom:0in;padding-left:0.075in;padding-right:0.075in;"| # 


| style="border-top:0.5pt solid #000000;border-bottom:0.5pt solid #000000;border-left:0.5pt solid #000000;border-right:none;padding-top:0in;padding-bottom:0in;padding-left:0.075in;padding-right:0.075in;"| Date & Time of Incident Reported 
| style="border:0.5pt solid #000000;padding-top:0in;padding-bottom:0in;padding-left:0.075in;padding-right:0.075in;"| 01-Feb-2013 11:06 hrs

|-
| style="border-top:0.5pt solid #000000;border-bottom:0.5pt solid #000000;border-left:0.5pt solid #000000;border-right:none;padding-top:0in;padding-bottom:0in;padding-left:0.075in;padding-right:0.075in;"| # 


| style="border-top:0.5pt solid #000000;border-bottom:0.5pt solid #000000;border-left:0.5pt solid #000000;border-right:none;padding-top:0in;padding-bottom:0in;padding-left:0.075in;padding-right:0.075in;"| RCA Date
| style="border:0.5pt solid #000000;padding-top:0in;padding-bottom:0in;padding-left:0.075in;padding-right:0.075in;"| 02-Feb-2012

|-
| style="border-top:0.5pt solid #000000;border-bottom:0.5pt solid #000000;border-left:0.5pt solid #000000;border-right:none;padding-top:0in;padding-bottom:0in;padding-left:0.075in;padding-right:0.075in;"| # 


| style="border-top:0.5pt solid #000000;border-bottom:0.5pt solid #000000;border-left:0.5pt solid #000000;border-right:none;padding-top:0in;padding-bottom:0in;padding-left:0.075in;padding-right:0.075in;"| Description of the Incident and how the incident was detected
| style="border:0.5pt solid #000000;padding-top:0in;padding-bottom:0in;padding-left:0.075in;padding-right:0.075in;"| Platform services was not available

Customer reported 




|-
| style="border-top:0.5pt solid #000000;border-bottom:0.5pt solid #000000;border-left:0.5pt solid #000000;border-right:none;padding-top:0in;padding-bottom:0in;padding-left:0.075in;padding-right:0.075in;"| # 


| style="border-top:0.5pt solid #000000;border-bottom:0.5pt solid #000000;border-left:0.5pt solid #000000;border-right:none;padding-top:0in;padding-bottom:0in;padding-left:0.075in;padding-right:0.075in;"| Impact of the Incident
| style="border:0.5pt solid #000000;padding-top:0in;padding-bottom:0in;padding-left:0.075in;padding-right:0.075in;"| Platform 63 HHT services was not available

|-
| style="border-top:0.5pt solid #000000;border-bottom:0.5pt solid #000000;border-left:0.5pt solid #000000;border-right:none;padding-top:0in;padding-bottom:0in;padding-left:0.075in;padding-right:0.075in;"| # 


| style="border-top:0.5pt solid #000000;border-bottom:0.5pt solid #000000;border-left:0.5pt solid #000000;border-right:none;padding-top:0in;padding-bottom:0in;padding-left:0.075in;padding-right:0.075in;"| Investigation Findings

# 
#* What happened
#* How did it happen
#* Why it happened
#* Recommendations


| style="border:0.5pt solid #000000;padding-top:0in;padding-bottom:0in;padding-left:0.075in;padding-right:0.075in;"| 11:06 AM – Phone call .

11.15 to 13 PM – Screen shot send to Rajneesh ( not to Service help desk ) . Rajneesh has guided the local team ( Raj kumar ) to make cluster up .

13:00 – 16:00 – Tetra remote support desk got involved and tried to diagnose the issue. Due to lack of basic Linux knowledge at the local end and no team-viewer access , major delay in understanding the issue by Tetra Team .

16:00 – Decision was taken to manual start the services on Node 2

18:30 PM – Node 2 was on-line (with out cluster ) and Platform services was Up with Single Node 

19:00 – 04:00 – Efforts to make the cluster up . Senior Engineer , Bhushan , reached site. Provided logs . Logs were analyzed .

8:00 AM – Production time started and one node was on production . Tetra team was still analyzing and cause and fix . 

16:30 PM – production hours were over .

16:45 PM – Cluster was tested UP and running 

Analyzing the logs , the following conclusion was inferred 

* '''The Cause''' of issue was repeated power failures . This has made data sync between the cluster node , inconsistent
* Logs also revealed that process inconsistency on process DRBD.
* It was also discovered that drbd process was not on for both the nodes at the boot time . 
* We had forcefully made the data consistent . 
* Switching on the DRBD process on boot time made the cluster run .
* Logs are attached 



|-
| style="border-top:0.5pt solid #000000;border-bottom:0.5pt solid #000000;border-left:0.5pt solid #000000;border-right:none;padding-top:0in;padding-bottom:0in;padding-left:0.075in;padding-right:0.075in;"| # 


| style="border-top:0.5pt solid #000000;border-bottom:0.5pt solid #000000;border-left:0.5pt solid #000000;border-right:none;padding-top:0in;padding-bottom:0in;padding-left:0.075in;padding-right:0.075in;"| Recommended measures for prevention of similar incidents

* The recommendation should consider the following:

# Any process improvement 
# Any technical measures
# Any training/awareness


| style="border:0.5pt solid #000000;padding-top:0in;padding-bottom:0in;padding-left:0.075in;padding-right:0.075in;"| After analysis we recommend that 

* It is 100% mandatory to have proper UPS system . The power should be available 100% without fail to the Cluster 
* Team viewer access 
* At least 3-6 month experience Linux L0 resources , major delay has happen due to no basic knowledge
* Remote training to L0/L1 local resource to have fast turnaround
* Monthly health report generation



|-
| style="border-top:0.5pt solid #000000;border-bottom:0.5pt solid #000000;border-left:0.5pt solid #000000;border-right:none;padding-top:0in;padding-bottom:0in;padding-left:0.075in;padding-right:0.075in;"| # 


| style="border-top:0.5pt solid #000000;border-bottom:0.5pt solid #000000;border-left:0.5pt solid #000000;border-right:none;padding-top:0in;padding-bottom:0in;padding-left:0.075in;padding-right:0.075in;"| Follow up actions on prevention of similar incidents
| style="border:0.5pt solid #000000;padding-top:0in;padding-bottom:0in;padding-left:0.075in;padding-right:0.075in;"| * SOS UPS availability
* Training to L0 / L1 Resource .



|}