Mahuaa cluster issue
Scenerio : There are two nodes (mail.mahuaatv.com & mail1.mahuaatv.com) of cluster running at Mahuaa. Basically what happened was that mail1.mahuaatv.com stopped working. Neither approachable via ping nor by ssh.
What We Did To Resue the Issue : > Shutted down both the systems. > Node 1 was brought up. > After proper startup of node 1, node 2 was brought up. > Here we face an issue that node 1 was not coming up and stuck at boot process. Issue was regarding the disk issue. > Then finally after 2-3 tries, We removed lan cable & it automatically started. > Both machines were up but neither if them was in network(i.e. not accessible via global ip) > Then machine was brought in local n/w and it was accessed via taking teamviewer of a local machine and through that machine to server. > Then following was done on server :
On Node 2 :
/etc/init.d/heartbeat stop
Stopped heartbeat
tailf /var/log/ha-log
checked logs weather the heartbeat is properly shutting down or not
cat /proc/drbd
checked weather this machine is now secondary or not.
df -h
checked wether the /home is unmounted or not.
now to Node 1 :
/etc/init.d/heartbeat stop /etc/init.d/heartbeat star
restart the service
tail -f /var/log/ha-log
checked the logs wether the heartbeat is properly started or not.
cat /proc/drbd
check wether the server is primary or not.
cat /etc/ha.d/haresources
in this file we check what ip has to be assigned to node 1 , what services to be started, which partition to be mounter and where.
now check ip address
ifconfig
Check wether the /home partition is mounted or not.
df-h
now check wether all the services are up or not which are defined in /etc/ha.d/haresources /etc/ha.d/haresources file.
example :
[root@mail ~]# cat /etc/ha.d/haresources mail.mahuaatv.com IPaddr::193.168.0.222/24/eth1 drbddisk::r0 Filesystem::/dev/drbd0::/home::ext3 mysqld dovecot clam_permissions clamd amavisd postfix httpd crond
[root@mail ~]# /etc/init.d/mysqld status mysqld (pid 24486) is running...
[root@mail ~]# /etc/init.d/dovecot status dovecot (pid 24536) is running...
[root@mail ~]# /etc/init.d/clamd status clamd (pid 24617) is running...
[root@mail ~]# /etc/init.d/amavisd status amavisd (pid 25009 25008 25006 25004 25003 25002 25001 25000 24999 24998 24660) is running... amavis-milter is stopped
[root@mail ~]# /etc/init.d/postfix status master (pid 24743) is running...
[root@mail ~]# /etc/init.d/httpd status httpd (pid 24785) is running...
[root@mail ~]# /etc/init.d/crond status crond (pid 24823) is running...
now check weather the node is primary and is uptodate :
[root@mail ~]# cat /proc/drbd version: 8.0.16 (api:86/proto:86) GIT-hash: d30881451c988619e243d6294a899139eed1183d build by mockbuild@v20z-x86-64.home.local, 2009-08-22 13:23:56 0: cs:Connected st:Primary/Secondary ds:UpToDate/UpToDate C r--- ns:30232760 nr:8 dw:29180096 dr:2144235 al:35473 bm:787 lo:0 pe:0 ua:0 ap:0 resync: used:0/61 hits:0 misses:0 starving:0 dirty:0 changed:0 act_log: used:0/257 hits:7259549 misses:36475 starving:0 dirty:1002 changed:35473
---DONE---