Cluster of energoindia drbd and heartbeat

We had challenge to create cluster with drbd and heartbeat of qmail because alreay there was running server of qmail. We have decided to create cluster by following below steps:

STEP 1. create a server with same configuration as running server.( Os, harddisk, drbd size)

STEP 2. create DRBD and HEARTBEAT on this server.

STEP 3. Migrate the users, password, and groups from running server to new server.

STEP 4. make link of required directory and file of qmail, ldap, clamd, spamassassin, squid, fetchmail in /home partition and copied data from running server.

STEP 5. Live this new server as primary node of cluster.

STEP 6. Create DRBD and HEARTBEAT on old server as secondary node.

STEP 7. Make link of required directory and file as created in primary node.

STEP 8. Test the cluster.

############### NODE1: configuration ################

OS- Red Hat Enterprise Linux Server release 5.3 (Tikanga)

ARCH - 64 bit

Note :

1. Before start DRBD and Heartbeat implementation , make sure NTP synchronization between node1 and node2 , otherwise it will not work.

2. Make both node or machine keyless, so it will not promt password at the time of login from one node to other node.

[root@node1 ~]# cat /etc/hosts

 # Do not remove the following line, or various programs
 # that require network functionality will fail.
 127.0.0.1		mail.energoindia.com mail localhost.localdomain localhost
 #::1		localhost6.localdomain6 localhost6
 192.168.10.149  node1.energoindia.com node1  node.energoindia.com mail
 192.168.1.150  node.energoindia.com node  mail.energoindia.com mail
 10.0.0.1        node1.energoindia.com node1 node.energoindia.com 
 10.0.0.2        node2.energoindia.com node2 node.energoindia.com
 192.168.10.155  node2.energoindia.com node2 node.energoindia.com mail

NOTE: The virtual ip is 192.168.1.150.

                       :::: Installing DRBD ::::

[root@node1 ~]# yum install drbd83-utils kmod-drbd83

[root@node1 ~]# mv /etc/drbd.d/global_common.conf /etc/drbd.d/global_common.sample

[root@node1 ~]# vim /etc/drbd.d/global_common.conf

 global { usage-count no; }
 common {
 syncer { rate 100M; }
 } 

Create a new resource file that I called “main”,

[root@node1 ~]# vim /etc/drbd.d/main.res

  resource main {
  protocol C;
  startup { wfc-timeout 0; degr-wfc-timeout 120; }
  disk { on-io-error detach; }
  on node1.energoindia.com {
  device /dev/drbd0;
  disk /dev/VolGroup00/LogVol02;
  meta-disk internal;
  address 10.0.0.1:7788;
 }
 on node2.energoindia.com {
 device /dev/drbd0;
 disk /dev/mapper/VolGroup00-LogVol02;
 meta-disk internal;
 address 10.0.0.2:7788;
 }
 }

Now we have to create the metadata on node1:

[root@node1 ~]#dd if=/dev/zero of= /dev/VolGroup00/LogVol02 bs=1M count=50

[root@node1 ~]# drbdadm create-md main

and we can now start DRBD

[root@node1 ~]# service drbd start

Starting DRBD resources: [ d(main) s(main) n(main) ].

We will tell DRBD that server1 is the Primary server:

[root@node1 ~]# drbdsetup /dev/drbd0 primary –o

[root@node1 ~]# cat /proc/drbd

 version: 8.3.16 (api:88/proto:86-97)
 GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R5, 2013-09-27 15:48:18
 0: cs:Connected ro:Primary/Secondary ds:UpToDate/unknown C r-----
 ns:182767328 nr:116152 dw:182883616 dr:829873130 al:1019560 bm:344 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

[root@node1 ~]# mkfs.ext3 /dev/drbd0

Now DRBD is configured , now you ready to mount your /dev/drbd0 to /home

[root@node1 ~]# mount –t ext3 /dev/drbd0 /home

[root@node1 ~]# df -h

 Filesystem            Size  Used Avail Use% Mounted on
 /dev/mapper/VolGroup00-LogVol00                         96G     41G     50G    46% /
 /dev/sda1               935M     27M    860M     4%   /boot
 tmpfs                   3.8G       0     3.8G     0%   /dev/shm
 /dev/drbd0               446G     346G     78G    82%   /home

                        ::::::::::::Installing HEARBEAT:::::::::::

[root@node1 ~]# yum install heartbeat

Create the following file on node1 servers:

[root@node1 ~]# vim /etc/ha.d/ha.cf

 keepalive 2
 deadtime 30
 bcast eth0 
 node node1.energoindia.com node2.energoindia.com
 logfile /var/log/ha-log
 debugfile /var/log/ha-debug

The next step is to create the resource file for heartbeat on node1 servers:

[root@node1 ~]# vim /etc/ha.d/haresources

  node1.energoindia.com IPaddr::192.168.10.150/24/eth0 IPaddr::192.168.20.150/24/eth0 
  IPaddr::192.168.1.150/22/eth0 IPaddr::192.168.30.150/24/eth0 drbddisk::main Filesystem::/dev/drbd0::
  /home::ext3 ldap httpd spamassassin clamd authdaemond imap imaps squid qmailctl

The last thing is to create the authentication file on node1 servers:

[root@node1 ~]# vim /etc/ha.d/authkeys

  auth 3
  3 md5 asd1234f 

This password file should only be readable by the root user:

[root@node1 ~]# chmod 600 /etc/ha.d/authkeys

start heartbeat on node 1:

[root@node1 ~]# service heartbeat start

[root@node1 ~]# ifconfig

eth0 Link encap:Ethernet HWaddr E4:1F:13:C0:E9:CC

             inet addr:192.168.10.149  Bcast:192.168.10.255  Mask:255.255.255.0
            inet6 addr: fe80::e61f:13ff:fec0:e9cc/64 Scope:Link
            UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
            RX packets:171872956 errors:0 dropped:0 overruns:0 frame:0
            TX packets:229211317 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:1000 
            RX bytes:80916792381 (75.3 GiB)  TX bytes:190588412808 (177.4 GiB)
            Interrupt:169 Memory:92000000-92012100 

eth0:0 Link encap:Ethernet HWaddr E4:1F:13:C0:E9:CC

              inet addr:192.168.10.150  Bcast:192.168.10.255  Mask:255.255.255.0
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              Interrupt:169 Memory:92000000-92012100 

eth0:1 Link encap:Ethernet HWaddr E4:1F:13:C0:E9:CC

         inet addr:192.168.20.150  Bcast:192.168.20.255  Mask:255.255.255.0
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         Interrupt:169 Memory:92000000-92012100 

eth0:2 Link encap:Ethernet HWaddr E4:1F:13:C0:E9:CC

         inet addr:192.168.1.150  Bcast:192.168.3.255  Mask:255.255.252.0
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         Interrupt:169 Memory:92000000-92012100 

eth0:3 Link encap:Ethernet HWaddr E4:1F:13:C0:E9:CC

         inet addr:192.168.30.150  Bcast:192.168.30.255  Mask:255.255.255.0
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         Interrupt:169 Memory:92000000-92012100 

eth1 Link encap:Ethernet HWaddr E4:1F:13:C0:E9:CD

         inet addr:10.0.0.1  Bcast:10.255.255.255  Mask:255.0.0.0
         inet6 addr: fe80::e61f:13ff:fec0:e9cd/64 Scope:Link
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         RX packets:32136747 errors:0 dropped:0 overruns:0 frame:0
         TX packets:92544365 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:1000 
         RX bytes:3368179779 (3.1 GiB)  TX bytes:128171113009 (119.3 GiB)
         Interrupt:138 Memory:94000000-94012100 

lo Link encap:Local Loopback

         inet addr:127.0.0.1  Mask:255.0.0.0
         inet6 addr: ::1/128 Scope:Host
         UP LOOPBACK RUNNING  MTU:16436  Metric:1
         RX packets:7571848 errors:0 dropped:0 overruns:0 frame:0
         TX packets:7571848 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:0 
         X bytes:5164760682 (4.8 GiB)  TX bytes:5164760682 (4.8 GiB)

:MIGRATION OF QMAIL USERS, SYSTEM USERS, NORMAL USERS FROM A RUNNING SERVER TO NODE1:

Commands to type on old Linux system:

1. First create a tar ball of old uses (old Linux system). Create a directory:

1. mkdir /root/move/

2. Setup UID filter limit:

1. export UGIDLIMIT=500

3. Now copy /etc/passwd accounts to /root/move/passwd.mig using awk to filter out system account (i.e. only copy user accounts)

1. awk -v LIMIT=$UGIDLIMIT -F: '($3>=LIMIT) && ($3!=65534)' /etc/passwd > /root/move/passwd.mig

4. Copy /etc/group file:

1. awk -v LIMIT=$UGIDLIMIT -F: '($3>=LIMIT) && ($3!=65534)' /etc/group > /root/move/group.mig

5. Copy /etc/shadow file:

1. awk -v LIMIT=$UGIDLIMIT -F: '($3>=LIMIT) && ($3!=65534) {print $1}' /etc/passwd | tee - |egrep -f - /etc/shadow > /root/move/shadow.mig

6. Copy /etc/gshadow (rarely used):

1. cp /etc/gshadow /root/move/gshadow.mig

7. Use scp or usb pen or tape to copy /root/move to a new Linux system.

1. scp -r /root/move/* user@new.linuxserver.com:/path/to/location

Commands to type on new Linux system

1. First, make a backup of current users and passwords:

[root@node1 ~]# mkdir /root/newsusers.bak

[root@node1 ~]# cp /etc/passwd /etc/shadow /etc/group /etc/gshadow /root/newsusers.bak

2. Now restore passwd and other files in /etc/

[root@node1 ~]# cd /path/to/location

[root@node1 ~]# cat passwd.mig >> /etc/passwd

[root@node1 ~]# cat group.mig >> /etc/group

[root@node1 ~]# cat shadow.mig >> /etc/shadow

[root@node1 ~]# /bin/cp gshadow.mig /etc/gshadow

Now reboot system; when the Linux comes back, your user accounts will work as they did before on old system:

[root@node1 ~]# reboot

:COPY ALL FILES OF QMAIL, LDAP ,SPAMASSASSIN , CLAMAV FROM A SERVER TO NODE1 USING LINK:

create below directory, make link of this and copy the data from running server.

[root@node1 ~]# mkdir -p /home/command

[root@node1 ~]# ln -s /home/command /command ( copy data from running server )

[root@node1 ~]# mkdir -p /home/downloads

[root@node1 ~]# ln -s /home/downloads /downloads

[root@node1 ~]# mkdir -p /home/packages

[root@node1 ~]# ln -s /home/packages /packages

[root@node1 ~]# mkdir -p /home/service

[root@node1 ~]# ln -s /home/service /service

[root@node1 ~]# mkdir -p /home/usr/local

[root@node1 ~]# ln -s /home/usr/local /usr/local

[root@node1 ~]# mkdir -p /home/var/qmail

[root@node1 ~]# ln -s /home/var/qmail /var/qmail

[root@node1 ~]# mkdir -p /home/usr/src

[root@node1 ~]# ln -s /home/usr/src /usr/src

[root@node1 ~]# mkdir -p /home/var/log/qmail

[root@node1 ~]# ln -s /home/var/log/qmail /var/log/qmail

[root@node1 ~]# touch /home/etc/tcp.smtp

[root@node1 ~]# ln -s /home/etc/tcp.smtp /etc/tcp.smtp

[root@node1 ~]# touch /home/etc/tcp.smtp.cdb

[root@node1 ~]# ln -s /home/etc/tcp.smtp.cdb /etc/tcp.smtp.cdb

[root@node1 ~]# mkdir -p /home/qmaildoc

[root@node1 ~]# ln -s /home/qmaildoc /qmaildoc

Install openldap:

[root@node1 ~]# yum install openldap

[root@node1 ~]# mkdir -p /home/etc/openldap

[root@node1 ~]# ln -s /home/etc/openldap /etc/openldap

[root@node1 ~]# mkdir -p /home/var/lib/ldap

[root@node1 ~]# ln -s /home/var/lib/ldap /var/lib/ldap

[root@node1 ~]# mkdir -p /home/var/www/html

[root@node1 ~]# ln -s /home/var/www/html /var/www/html

[root@node1 ~]# mkdir -p /home/var/www/sarg

[root@node1 ~]# ln -s /home/var/www/sarg /var/www/sarg

Install clamav:

[root@node1 ~]# yum install clamav clamd

[root@node1 ~]# mkdir -p /home/var/spool/qscan

[root@node1 ~]# ln -s /home/var/spool/qscan /var/spool/qscan

install spamassassin:

[root@node1 ~]# yum install spamassassin

[root@node1 ~]# mkdir -p /home/etc/mail/spamassassin/

[root@node1 ~]# touch /home/etc/mail/spamassassin/local.cf

[root@node1 ~]# ln -s /home/etc/mail/spamassassin/local.cf /etc/mail/spamassassin/local.cf

install squid:

[root@node1 ~]# yum install squid

[root@node1 ~]# mkdir -p /home/etc/squid

[root@node1 ~]# ln -s /home/etc/squid /etc/squid

configuration of fetchmail:

[root@node1 ~]# crontab -lu tetra

• /5 * * * * /usr/bin/fetchmail -v -U -F >> /home/tetra/fetchmail.log

copy the script of fetchmail...

[root@node1 ~]# rsync -aP 192.168.10.155:/usr/bin/fetchmail /usr/bin/

copy the rc.local file........

[root@node1 ~]# rsync -aP 192.168.10.155:/etc/rc.local /etc/

Make link of below file.......

[root@node1 ~]# ln -s /usr/local/sbin/authdaemond /etc/init.d/authdaemond

[root@node1 ~]# ln -s /var/qmail/bin/qmailctl /usr/bin/qmailctl

[root@node1 ~]# ln -s /usr/bin/qmailctl /etc/init.d/qmailctl

Now reboot the node1 to check all required files are copied properly and server is starting normally. Afterthen try below to check qmail, imap, ldap is working or not.

[root@node1 ~]# telnet localhost 25

               Trying 127.0.0.1...
               Connected to localhost.localdomain (127.0.0.1).
               Escape character is '^]'.
               220 mail.energoindia.com ESMTP 

[root@node1 ~]# telnet localhost 110

              Trying 127.0.0.1...
              Connected to localhost.localdomain (127.0.0.1).
              Escape character is '^]'.
             +OK <2081.1436400429@mail.energoindia.com>

[root@node1 ~]# telnet localhost 143

                Trying 127.0.0.1...
                Connected to localhost.localdomain (127.0.0.1).
                Escape character is '^]'.
             *  OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA 
                IDLE ACL ACL2=UNION STARTTLS] Courier-IMAP ready. Copyright 1998-2005 Double Precision, Inc.  See COPYING 
                for distribution information.

[root@node1 ~]# telnet localhost 389

              Trying 127.0.0.1...
              Connected to localhost.localdomain (127.0.0.1).
              Escape character is '^]'.

copy the /home from running server to node1......

[root@node1 ~]# rsync -alkvHP 192.168.10.155:/home/* /home/

After proper sync live this node1.

############## ON NODE2 configuration ##############

[root@node2 ~]# cat /etc/hosts

# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1               mail.energoindia.com mail localhost.localdomain localhost
#::1            localhost6.localdomain6 localhost6
192.168.10.149  node1.energoindia.com node1  node.energoindia.com mail
192.168.1.150  node.energoindia.com node  mail.energoindia.com mail
10.0.0.1        node1.energoindia.com node1 node.energoindia.com
10.0.0.2        node2.energoindia.com node2 node.energoindia.com
192.168.10.155  node2.energoindia.com node2 node.energoindia.com mail

                   :::::::::Installing DRBD:::::::

[root@node2 ~]# yum install drbd83-utils kmod-drbd83

Configuration should be same as in node1:

[root@node2 ~]# mv /etc/drbd.d/global_common.conf /etc/drbd.d/global_common.sample

[root@node2 ~]# vim /etc/drbd.d/global_common.conf

global { usage-count no; }
common {
syncer { rate 100M; }
} 

Create a new resource file that I called “main”,

[root@node2 ~]# vim /etc/drbd.d/main.res

resource main {
protocol C;
startup { wfc-timeout 0; degr-wfc-timeout 120; }
disk { on-io-error detach; }
on node1.energoindia.com {
device /dev/drbd0;
disk /dev/VolGroup00/LogVol02;
meta-disk internal;
address 10.0.0.1:7788;
}
on node2.energoindia.com {
device /dev/drbd0;
disk /dev/mapper/VolGroup00-LogVol02;
meta-disk internal;
address 10.0.0.2:7788;
}
}

Now we have to create the metadata on node2:

[root@node2 ~]# dd if=/dev/zero of= /dev/VolGroup00/LogVol02  bs=1M count=50 

[root@node2 ~]# drbdadm create-md main

Now start DRBD on this (node2) server:

[root@node2 ~]# service drbd start

Starting DRBD resources: [ d(main) s(main) n(main) ].

After starting drbd on node2 server, it start to synchronization between both nodes. It will take a little while to be completed. Please wait until it is done and move to the next step.

[root@node2 ~]# cat /proc/drbd

version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R5, 2013-09-27 15:48:18
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
ns:503808 nr:0 dw:0 dr:504472 al:0 bm:30 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:41437916
[>....................] sync'ed: 1.3% (40464/40956)M
finish: 1:06:40 speed: 10,340 (10,280) K/sec 

                     ::::::Installing heartbeat::::::

[root@node2 ~]# yum install heartbeat

Create the following file on node2 servers with the exact same content as on node1:

[root@node2 ~]# vim /etc/ha.d/ha.cf

 keepalive 2
 deadtime 30
 bcast eth0 
 node node1.energoindia.com node2.energoindia.com
 logfile /var/log/ha-log
 debugfile /var/log/ha-debug

The next setp is to create the resource file for heartbeat on node2 servers with exact same content as on node1 again:

[root@node2 ~]# vim /etc/ha.d/haresources

 node1.energoindia.com IPaddr::192.168.10.150/24/eth0 IPaddr::192.168.20.150/24/eth0   IPaddr::192.168.1.150/22/eth0 IPaddr::192.168.30.150/24/eth0 drbddisk::main Filesystem::/dev/drbd0::/home::ext3 ldap httpd spamassassin clamd authdaemond imap imaps squid qmailctl

The last thing is to create the authentication file on node2 servers again with the same content as on node1:

[root@node2 ~]# vim /etc/ha.d/authkeys

 auth 3
 3 md5 asd1234f 

This password file should only be readable by the root user:

[root@node2 ~]# chmod 600 /etc/ha.d/authkeys

[root@node2 ~]# service heartbeat start

Again create same directory on node2 as in node1 and make links but before take backup of these directory:

[root@node2 ~]# mkdir -p /home/command

[root@node2 ~]# ln -s /home/command /command

[root@node2 ~]# mkdir -p /home/downloads

[root@node2 ~]# ln -s /home/downloads /downloads

[root@node2 ~]# mkdir -p /home/packages

[root@node2 ~]# ln -s /home/packages /packages

[root@node2 ~]# mkdir -p /home/service

[root@node2 ~]# ln -s /home/service /service

[root@node2 ~]# mkdir -p /home/usr/local

[root@node2 ~]# ln -s /home/usr/local /usr/local

[root@node2 ~]# mkdir -p /home/var/qmail

[root@node2 ~]# ln -s /home/var/qmail /var/qmail

[root@node2 ~]# mkdir -p /home/usr/src

[root@node2 ~]# ln -s /home/usr/src /usr/src

[root@node2 ~]# mkdir -p /home/var/log/qmail

[root@node2 ~]# ln -s /home/var/log/qmail /var/log/qmail

[root@node2 ~]# touch /home/etc/tcp.smtp

[root@node2 ~]# ln -s /home/etc/tcp.smtp /etc/tcp.smtp

[root@node2 ~]# touch /home/etc/tcp.smtp.cdb

[root@node2 ~]# ln -s /home/etc/tcp.smtp.cdb /etc/tcp.smtp.cdb

[root@node2 ~]# mkdir -p /home/qmaildoc

[root@node2 ~]# ln -s /home/qmaildoc /qmaildoc

[root@node2 ~]# mkdir -p /home/etc/openldap

[root@node2 ~]# ln -s /home/etc/openldap /etc/openldap

[root@node2 ~]# mkdir -p /home/var/lib/ldap

[root@node2 ~]# ln -s /home/var/lib/ldap /var/lib/ldap

[root@node2 ~]# mkdir -p /home/var/www/html

[root@node2 ~]# ln -s /home/var/www/html /var/www/html

[root@node2 ~]# mkdir -p /home/var/www/sarg

[root@node2 ~]# ln -s /home/var/www/sarg /var/www/sarg

[root@node2 ~]# mkdir -p /home/var/spool/qscan

[root@node2 ~]# ln -s /home/var/spool/qscan /var/spool/qscan

[root@node2 ~]# mkdir -p /home/etc/mail/spamassassin/

[root@node2 ~]# touch /home/etc/mail/spamassassin/local.cf

[root@node2 ~]# ln -s /home/etc/mail/spamassassin/local.cf /etc/mail/spamassassin/local.cf

[root@node2 ~]# mkdir -p /home/etc/squid

[root@node2 ~]# ln -s /home/etc/squid /etc/squid

Make link of below file.......

[root@node2 ~]# ln -s /usr/local/sbin/authdaemond /etc/init.d/authdaemond

[root@node2 ~]# ln -s /var/qmail/bin/qmailctl /usr/bin/qmailctl

[root@node2 ~]# ln -s /usr/bin/qmailctl /etc/init.d/qmailctl

                           == ### Test the cluster ### ==

1. Stop heartbeat on node1:

[root@node1 ~]# /etc/init.d/heartbeat stop

2. Watch log of heartbeat on node2, cluster will shift on node2.

[root@node2 ~]# tailf /var/log/ha-log

3. On node2, check the mails are working and /home is mounted.

[root@node2 ~]# telnet localhost 25

  Trying 127.0.0.1...
  Connected to localhost.localdomain (127.0.0.1).
  Escape character is '^]'.
  220 mail.energoindia.com ESMTP 

[root@node2 ~]# telnet localhost 110

  Trying 127.0.0.1...
  Connected to localhost.localdomain (127.0.0.1).
  Escape character is '^]'.
  +OK <14050.1436639325@mail.energoindia.com>

[root@node2 ~]# telnet localhost 143

  Trying 127.0.0.1...
  Connected to localhost.localdomain (127.0.0.1).
  Escape character is '^]'.
  * OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT   QUOTA IDLE ACL ACL2=UNION STARTTLS] Courier-IMAP ready. Copyright 1998-2005 Double Precision, Inc.  See COPYING for distribution information.

[root@node2 ~]# telnet localhost 389

 Trying 127.0.0.1...
 Connected to localhost.localdomain (127.0.0.1).
 Escape character is '^]'.

[root@node2 ~]# df -h

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      95G   74G   17G  83% /
/dev/sda1             965M   24M  892M   3% /boot
tmpfs                 2.0G     0  2.0G   0% /dev/shm
/dev/drbd0            446G  359G   65G  85% /home

4. now, to shift cluster again on node1.

  stop heartbeat on node2 and start heartbeat on node1.

[root@node2 ~]# /etc/init.d/heartbeat stop

[root@node1 ~]# /etc/init.d/heartbeat start

5. Now test the cluster is shifting to node2, if node1 goes down. It will shift.

6. Now start node1, the cluster will shift automatically on node1 from node2.