Jump to content

Mahuaa cluster issue: Difference between revisions

From TetraWiki
Newer edit →
No edit summary
 
No edit summary
Line 1: Line 1:
  Scenerio :  
'''Scenerio :'''
      There are two nodes (mail.mahuaatv.com & mail1.mahuaatv.com) of cluster running at Mahuaa. Basically what happened was that  
There are two nodes (mail.mahuaatv.com & mail1.mahuaatv.com) of cluster running at Mahuaa. Basically what happened was that  
      mail1.mahuaatv.com stopped working. Neither approachable via ping nor by ssh.  
mail1.mahuaatv.com stopped working. Neither approachable via ping nor by ssh.  


  What We Did To Resue the Issue :  
What We Did To Resue the Issue :  
  >  Shutted down both the systems.  
>  Shutted down both the systems.  
  >  Node 1 was brought up.  
>  Node 1 was brought up.  
  >  After proper startup of node 1, node 2 was brought up.
>  After proper startup of node 1, node 2 was brought up.
  >  Here we face an issue that node 1 was not coming up and stuck at boot process. Issue was regarding the disk issue.  
>  Here we face an issue that node 1 was not coming up and stuck at boot process. Issue was regarding the disk issue.  
  >  Then finally after 2-3 tries, We removed lan cable & it automatically started.
>  Then finally after 2-3 tries, We removed lan cable & it automatically started.
  >  Both machines were up but neither if them was in network(i.e. not accessible via global ip)
>  Both machines were up but neither if them was in network(i.e. not accessible via global ip)
  >  Then machine was brought in local n/w and it was accessed via taking teamviewer of a local machine and through that machine to server.
>  Then machine was brought in local n/w and it was accessed via taking teamviewer of a local machine and through that machine to server.
  >  Then following was done on server  :
>  Then following was done on server  :


On Node 2 :
On Node 2 :


   /etc/init.d/heartbeat stop
   /etc/init.d/heartbeat stop
  Stopped heartbeat
Stopped heartbeat


   tailf /var/log/ha-log
   tailf /var/log/ha-log
  checked logs wether the heartbeat is properly shutting down or not
checked logs weather the heartbeat is properly shutting down or not


   cat /proc/drbd
   cat /proc/drbd
  checked wether this machine is now secondary or not.
checked weather this machine is now secondary or not.


   df -h
   df -h
  checked wether the /home is unmounted or not.  
checked wether the /home is unmounted or not.  


now to Node 1 :
now to Node 1 :
    
    
   /etc/init.d/heartbeat stop
   /etc/init.d/heartbeat stop
   /etc/init.d/heartbeat start
   /etc/init.d/heartbeat star
  restart the service  
restart the service  


  tail -f /var/log/ha-log
tail -f /var/log/ha-log
  checked the logs wether the heartbeat is properly started or not.  
checked the logs wether the heartbeat is properly started or not.  


  cat /proc/drbd  
cat /proc/drbd  
  check wether the server is primary or not.
check wether the server is primary or not.


  cat /etc/ha.d/haresources  
cat /etc/ha.d/haresources  
  in this file we check what ip has to be assigned to node 1 , what services to be started, which partition to be mounter and where.
in this file we check what ip has to be assigned to node 1 , what services to be started, which partition to be mounter and where.


  now check ip address  
now check ip address  
  ifconfig  
ifconfig  
    
    
  Check wether the /home partition is mounted or not.
Check wether the /home partition is mounted or not.
   df-h
   df-h


Line 77: Line 77:




now check wether the node is primary and is uptodate :
now check weather the node is primary and is uptodate :


  [root@mail ~]# cat /proc/drbd
  [root@mail ~]# cat /proc/drbd

Revision as of 08:31, 3 December 2013

Scenerio : There are two nodes (mail.mahuaatv.com & mail1.mahuaatv.com) of cluster running at Mahuaa. Basically what happened was that mail1.mahuaatv.com stopped working. Neither approachable via ping nor by ssh.

What We Did To Resue the Issue : > Shutted down both the systems. > Node 1 was brought up. > After proper startup of node 1, node 2 was brought up. > Here we face an issue that node 1 was not coming up and stuck at boot process. Issue was regarding the disk issue. > Then finally after 2-3 tries, We removed lan cable & it automatically started. > Both machines were up but neither if them was in network(i.e. not accessible via global ip) > Then machine was brought in local n/w and it was accessed via taking teamviewer of a local machine and through that machine to server. > Then following was done on server  :

On Node 2 : 

 /etc/init.d/heartbeat stop

Stopped heartbeat 

 tailf /var/log/ha-log

checked logs weather the heartbeat is properly shutting down or not 

 cat /proc/drbd

checked weather this machine is now secondary or not. 

 df -h

checked wether the /home is unmounted or not.

now to Node 1 : 

 /etc/init.d/heartbeat stop
 /etc/init.d/heartbeat star

restart the service 

tail -f /var/log/ha-log

checked the logs wether the heartbeat is properly started or not. 

cat /proc/drbd

check wether the server is primary or not. 

cat /etc/ha.d/haresources

in this file we check what ip has to be assigned to node 1 , what services to be started, which partition to be mounter and where.

now check ip address 

ifconfig

Check wether the /home partition is mounted or not. 

 df-h

now check wether all the services are up or not which are defined in /etc/ha.d/haresources /etc/ha.d/haresources file.

example : 

[root@mail ~]# cat /etc/ha.d/haresources
mail.mahuaatv.com  IPaddr::193.168.0.222/24/eth1 drbddisk::r0 Filesystem::/dev/drbd0::/home::ext3 mysqld dovecot clam_permissions clamd amavisd  postfix httpd crond

[root@mail ~]# /etc/init.d/mysqld status 
mysqld (pid 24486) is running... 

[root@mail ~]# /etc/init.d/dovecot status 
dovecot (pid  24536) is running... 

[root@mail ~]# /etc/init.d/clamd status 
clamd (pid 24617) is running... 

[root@mail ~]# /etc/init.d/amavisd status 
amavisd (pid 25009 25008 25006 25004 25003 25002 25001 25000 24999 24998 24660) is running... 
amavis-milter is stopped 

[root@mail ~]# /etc/init.d/postfix status 
master (pid 24743) is running... 

[root@mail ~]# /etc/init.d/httpd  status 
httpd (pid  24785) is running... 

[root@mail ~]# /etc/init.d/crond status 
crond (pid  24823) is running...


now check weather the node is primary and is uptodate : 

[root@mail ~]# cat /proc/drbd
version: 8.0.16 (api:86/proto:86)
GIT-hash: d30881451c988619e243d6294a899139eed1183d build by mockbuild@v20z-x86-64.home.local, 2009-08-22 13:23:56
0: cs:Connected st:Primary/Secondary ds:UpToDate/UpToDate C r---
ns:30232760 nr:8 dw:29180096 dr:2144235 al:35473 bm:787 lo:0 pe:0 ua:0 ap:0
resync: used:0/61 hits:0 misses:0 starving:0 dirty:0 changed:0
act_log: used:0/257 hits:7259549 misses:36475 starving:0 dirty:1002 changed:35473


---DONE---

Retrieved from "https://wiki.tetrain.com/index.php?title=Mahuaa_cluster_issue&oldid=1517"