Editing Mahuaa cluster issue

'''Scenerio :''' 
There are two nodes (mail.mahuaatv.com & mail1.mahuaatv.com) of cluster running at Mahuaa. Basically what happened was that 
mail1.mahuaatv.com stopped working. Neither approachable via ping nor by ssh. 

What We Did To Resue the Issue : 

>  Shutted down both the systems. 

>  Node 1 was brought up. 

>  After proper startup of node 1, node 2 was brought up.

>  Here we face an issue that node 1 was not coming up and stuck at boot process. Issue was regarding the disk issue. 

>  Then finally after 2-3 tries, We removed lan cable & it automatically started.

>  Both machines were up but neither if them was in network(i.e. not accessible via global ip)

>  Then machine was brought in local n/w and it was accessed via taking teamviewer of a local machine and through that machine to server.

>  Then following was done on server  :

On '''Node 2''' :

  /etc/init.d/heartbeat stop
Stopped heartbeat

  tailf /var/log/ha-log
checked logs weather the heartbeat is properly shutting down or not

  cat /proc/drbd
checked weather this machine is now secondary or not.

  df -h
checked wether the /home is unmounted or not. 

now to '''Node 1''' :
  
  /etc/init.d/heartbeat stop
  /etc/init.d/heartbeat star
restart the service 

 tail -f /var/log/ha-log
checked the logs wether the heartbeat is properly started or not. 

 cat /proc/drbd 
check wether the server is primary or not.

 cat /etc/ha.d/haresources 
in this file we check what ip has to be assigned to node 1 , what services to be started, which partition to be mounter and where.

now check ip address 
 ifconfig 
   
Check wether the /home partition is mounted or not.
  df-h

now check wether all the services are up or not which are defined in /etc/ha.d/haresources /etc/ha.d/haresources file.

example :  
 [root@mail ~]# cat /etc/ha.d/haresources
 mail.mahuaatv.com  IPaddr::193.168.0.222/24/eth1 drbddisk::r0 Filesystem::/dev/drbd0::/home::ext3 mysqld dovecot clam_permissions clamd amavisd  postfix httpd crond

 [root@mail ~]# /etc/init.d/mysqld status 
 mysqld (pid 24486) is running... 

 [root@mail ~]# /etc/init.d/dovecot status 
 dovecot (pid  24536) is running... 

 [root@mail ~]# /etc/init.d/clamd status 
 clamd (pid 24617) is running... 

 [root@mail ~]# /etc/init.d/amavisd status 
 amavisd (pid 25009 25008 25006 25004 25003 25002 25001 25000 24999 24998 24660) is running... 
 amavis-milter is stopped 

 [root@mail ~]# /etc/init.d/postfix status 
 master (pid 24743) is running... 

 [root@mail ~]# /etc/init.d/httpd  status 
 httpd (pid  24785) is running... 

 [root@mail ~]# /etc/init.d/crond status 
 crond (pid  24823) is running... 


now check weather the node is primary and is uptodate :

 [root@mail ~]# cat /proc/drbd
 version: 8.0.16 (api:86/proto:86)
 GIT-hash: d30881451c988619e243d6294a899139eed1183d build by mockbuild@v20z-x86-64.home.local, 2009-08-22 13:23:56
 0: cs:Connected st:Primary/Secondary ds:UpToDate/UpToDate C r---
 ns:30232760 nr:8 dw:29180096 dr:2144235 al:35473 bm:787 lo:0 pe:0 ua:0 ap:0
 resync: used:0/61 hits:0 misses:0 starving:0 dirty:0 changed:0
 act_log: used:0/257 hits:7259549 misses:36475 starving:0 dirty:1002 changed:35473


---DONE---