Mahuaa cluster issue - Revision history

Biswajit at 08:33, 3 December 2013

2013-12-03T08:33:43Z

← Older revision		Revision as of 08:33, 3 December 2013
Line 4:		Line 4:

	What We Did To Resue the Issue :		What We Did To Resue the Issue :

	> Shutted down both the systems.		> Shutted down both the systems.

	> Node 1 was brought up.		> Node 1 was brought up.

	> After proper startup of node 1, node 2 was brought up.		> After proper startup of node 1, node 2 was brought up.

	> Here we face an issue that node 1 was not coming up and stuck at boot process. Issue was regarding the disk issue.		> Here we face an issue that node 1 was not coming up and stuck at boot process. Issue was regarding the disk issue.

	> Then finally after 2-3 tries, We removed lan cable & it automatically started.		> Then finally after 2-3 tries, We removed lan cable & it automatically started.

	> Both machines were up but neither if them was in network(i.e. not accessible via global ip)		> Both machines were up but neither if them was in network(i.e. not accessible via global ip)

	> Then machine was brought in local n/w and it was accessed via taking teamviewer of a local machine and through that machine to server.		> Then machine was brought in local n/w and it was accessed via taking teamviewer of a local machine and through that machine to server.

	> Then following was done on server :		> Then following was done on server :

	On Node 2 :		On '''Node 2''' :

	/etc/init.d/heartbeat stop		/etc/init.d/heartbeat stop
Line 27:		Line 35:
	checked wether the /home is unmounted or not.		checked wether the /home is unmounted or not.

	now to Node 1 :		now to '''Node 1''' :

	/etc/init.d/heartbeat stop		/etc/init.d/heartbeat stop

Biswajit at 08:31, 3 December 2013

2013-12-03T08:31:52Z

← Older revision		Revision as of 08:31, 3 December 2013
Line 1:		Line 1:
	Scenerio :		'''Scenerio :'''
	There are two nodes (mail.mahuaatv.com & mail1.mahuaatv.com) of cluster running at Mahuaa. Basically what happened was that		There are two nodes (mail.mahuaatv.com & mail1.mahuaatv.com) of cluster running at Mahuaa. Basically what happened was that
	mail1.mahuaatv.com stopped working. Neither approachable via ping nor by ssh.		mail1.mahuaatv.com stopped working. Neither approachable via ping nor by ssh.

	What We Did To Resue the Issue :		What We Did To Resue the Issue :
	> Shutted down both the systems.		> Shutted down both the systems.
	> Node 1 was brought up.		> Node 1 was brought up.
	> After proper startup of node 1, node 2 was brought up.		> After proper startup of node 1, node 2 was brought up.
	> Here we face an issue that node 1 was not coming up and stuck at boot process. Issue was regarding the disk issue.		> Here we face an issue that node 1 was not coming up and stuck at boot process. Issue was regarding the disk issue.
	> Then finally after 2-3 tries, We removed lan cable & it automatically started.		> Then finally after 2-3 tries, We removed lan cable & it automatically started.
	> Both machines were up but neither if them was in network(i.e. not accessible via global ip)		> Both machines were up but neither if them was in network(i.e. not accessible via global ip)
	> Then machine was brought in local n/w and it was accessed via taking teamviewer of a local machine and through that machine to server.		> Then machine was brought in local n/w and it was accessed via taking teamviewer of a local machine and through that machine to server.
	> Then following was done on server :		> Then following was done on server :

	On Node 2 :		On Node 2 :

	/etc/init.d/heartbeat stop		/etc/init.d/heartbeat stop
	Stopped heartbeat		Stopped heartbeat

	tailf /var/log/ha-log		tailf /var/log/ha-log
	checked logs ~~wether~~ the heartbeat is properly shutting down or not		checked logs weather the heartbeat is properly shutting down or not

	cat /proc/drbd		cat /proc/drbd
	checked ~~wether~~ this machine is now secondary or not.		checked weather this machine is now secondary or not.

	df -h		df -h
	checked wether the /home is unmounted or not.		checked wether the /home is unmounted or not.

	now to Node 1 :		now to Node 1 :

	/etc/init.d/heartbeat stop		/etc/init.d/heartbeat stop
	/etc/init.d/heartbeat ~~start~~		/etc/init.d/heartbeat star
	restart the service		restart the service

	tail -f /var/log/ha-log		tail -f /var/log/ha-log
	checked the logs wether the heartbeat is properly started or not.		checked the logs wether the heartbeat is properly started or not.

	cat /proc/drbd		cat /proc/drbd
	check wether the server is primary or not.		check wether the server is primary or not.

	cat /etc/ha.d/haresources		cat /etc/ha.d/haresources
	in this file we check what ip has to be assigned to node 1 , what services to be started, which partition to be mounter and where.		in this file we check what ip has to be assigned to node 1 , what services to be started, which partition to be mounter and where.

	now check ip address		now check ip address
	ifconfig		ifconfig

	Check wether the /home partition is mounted or not.		Check wether the /home partition is mounted or not.
	df-h		df-h

Line 77:		Line 77:


	now check ~~wether~~ the node is primary and is uptodate :		now check weather the node is primary and is uptodate :

	[root@mail ~]# cat /proc/drbd		[root@mail ~]# cat /proc/drbd

Shashanksharma at 07:14, 3 December 2013

2013-12-03T07:14:53Z

New page

Scenerio :
There are two nodes (mail.mahuaatv.com & mail1.mahuaatv.com) of cluster running at Mahuaa. Basically what happened was that
mail1.mahuaatv.com stopped working. Neither approachable via ping nor by ssh.

What We Did To Resue the Issue :
> Shutted down both the systems.
> Node 1 was brought up.
> After proper startup of node 1, node 2 was brought up.
> Here we face an issue that node 1 was not coming up and stuck at boot process. Issue was regarding the disk issue.
> Then finally after 2-3 tries, We removed lan cable & it automatically started.
> Both machines were up but neither if them was in network(i.e. not accessible via global ip)
> Then machine was brought in local n/w and it was accessed via taking teamviewer of a local machine and through that machine to server.
> Then following was done on server :

On Node 2 :

/etc/init.d/heartbeat stop
Stopped heartbeat

tailf /var/log/ha-log
checked logs wether the heartbeat is properly shutting down or not

cat /proc/drbd
checked wether this machine is now secondary or not.

df -h
checked wether the /home is unmounted or not.

now to Node 1 :

/etc/init.d/heartbeat stop
/etc/init.d/heartbeat start
restart the service

tail -f /var/log/ha-log
checked the logs wether the heartbeat is properly started or not.

cat /proc/drbd
check wether the server is primary or not.

cat /etc/ha.d/haresources
in this file we check what ip has to be assigned to node 1 , what services to be started, which partition to be mounter and where.

now check ip address
ifconfig

Check wether the /home partition is mounted or not.
df-h

now check wether all the services are up or not which are defined in /etc/ha.d/haresources /etc/ha.d/haresources file.

example :
[root@mail ~]# cat /etc/ha.d/haresources
mail.mahuaatv.com IPaddr::193.168.0.222/24/eth1 drbddisk::r0 Filesystem::/dev/drbd0::/home::ext3 mysqld dovecot clam_permissions clamd amavisd postfix httpd crond

[root@mail ~]# /etc/init.d/mysqld status
mysqld (pid 24486) is running...

[root@mail ~]# /etc/init.d/dovecot status
dovecot (pid 24536) is running...

[root@mail ~]# /etc/init.d/clamd status
clamd (pid 24617) is running...

[root@mail ~]# /etc/init.d/amavisd status
amavisd (pid 25009 25008 25006 25004 25003 25002 25001 25000 24999 24998 24660) is running...
amavis-milter is stopped

[root@mail ~]# /etc/init.d/postfix status
master (pid 24743) is running...

[root@mail ~]# /etc/init.d/httpd status
httpd (pid 24785) is running...

[root@mail ~]# /etc/init.d/crond status
crond (pid 24823) is running...

now check wether the node is primary and is uptodate :

[root@mail ~]# cat /proc/drbd
version: 8.0.16 (api:86/proto:86)
GIT-hash: d30881451c988619e243d6294a899139eed1183d build by mockbuild@v20z-x86-64.home.local, 2009-08-22 13:23:56
0: cs:Connected st:Primary/Secondary ds:UpToDate/UpToDate C r---
ns:30232760 nr:8 dw:29180096 dr:2144235 al:35473 bm:787 lo:0 pe:0 ua:0 ap:0
resync: used:0/61 hits:0 misses:0 starving:0 dirty:0 changed:0
act_log: used:0/257 hits:7259549 misses:36475 starving:0 dirty:1002 changed:35473

---DONE---