Cluster of energoindia drbd and heartbeat
We had challenge to create cluster with drbd and heartbeat of qmail because alreay there was running server of qmail. We have decided to create cluster by following below steps:
STEP 1. create a server with same configuration as running server.( Os, harddisk, drbd size)
STEP 2. create DRBD and HEARTBEAT on this server.
STEP 3. Migrate the users, password, and groups from running server to new server.
STEP 4. make link of required directory and file of qmail, ldap, clamd, spamassassin, squid, fetchmail in /home partition and copied data from running server.
STEP 5. Live this new server as primary node of cluster.
STEP 6. Create DRBD and HEARTBEAT on old server as secondary node.
STEP 7. Make link of required directory and file as created in primary node.
STEP 8. Test the cluster.
== ###NODE1: configuration ### ==
OS- Red Hat Enterprise Linux Server release 5.3 (Tikanga)
ARCH - 64 bit
Note :
1. Before start DRBD and Heartbeat implementation , make sure NTP synchronization between node1 and node2 , otherwise it will not work.
2. Make both node or machine keyless, so it will not promt password at the time of login from one node to other node.
[root@node1 ~]# cat /etc/hosts
- Do not remove the following line, or various programs
- that require network functionality will fail.
127.0.0.1 mail.energoindia.com mail localhost.localdomain localhost
- 1 localhost6.localdomain6 localhost6
192.168.10.149 node1.energoindia.com node1 node.energoindia.com mail
192.168.1.150 node.energoindia.com node mail.energoindia.com mail
10.0.0.1 node1.energoindia.com node1 node.energoindia.com
10.0.0.2 node2.energoindia.com node2 node.energoindia.com
192.168.10.155 node2.energoindia.com node2 node.energoindia.com mail
NOTE: The virtual ip is 192.168.1.150.
:::: Installing DRBD ::::
[root@node1 ~]# yum install drbd83-utils kmod-drbd83
[root@node1 ~]# mv /etc/drbd.d/global_common.conf /etc/drbd.d/global_common.sample
[root@node1 ~]# vim /etc/drbd.d/global_common.conf global { usage-count no; }
common {
syncer { rate 100M; }
}
Create a new resource file that I called “main”,
[root@node1 ~]# vim /etc/drbd.d/main.res
resource main {
protocol C;
startup { wfc-timeout 0; degr-wfc-timeout 120; }
disk { on-io-error detach; }
on node1.energoindia.com {
device /dev/drbd0;
disk /dev/VolGroup00/LogVol02;
meta-disk internal;
address 10.0.0.1:7788;
}
on node2.energoindia.com {
device /dev/drbd0;
disk /dev/mapper/VolGroup00-LogVol02;
meta-disk internal;
address 10.0.0.2:7788;
}
}
Now we have to create the metadata on node1:
[root@node1 ~]#dd if=/dev/zero of= /dev/VolGroup00/LogVol02 bs=1M count=50
[root@node1 ~]# drbdadm create-md main
and we can now start DRBD
[root@node1 ~]# service drbd start
Starting DRBD resources: [ d(main) s(main) n(main) ].
We will tell DRBD that server1 is the Primary server:
On node1:
[root@node1 ~]# drbdsetup /dev/drbd0 primary –o
[root@node1 ~]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R5, 2013-09-27 15:48:18
0: cs:Connected ro:Primary/Secondary ds:UpToDate/unknown C r-----
ns:182767328 nr:116152 dw:182883616 dr:829873130 al:1019560 bm:344 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[root@node1 ~]# mkfs.ext3 /dev/drbd0
Now DRBD is configured , now you ready to mount your /dev/drbd0 to /home
[root@node1 ~]# mount –t ext3 /dev/drbd0 /home
[root@node1 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00 96G 41G 50G 46% /
/dev/sda1 935M 27M 860M 4% /boot
tmpfs 3.8G 0 3.8G 0% /dev/shm
/dev/drbd0 446G 346G 78G 82% /home
::::::::::::Installing HEARBEAT:::::::::::
[root@node1 ~]# yum install heartbeat
Create the following file on node1 servers:
[root@node1 ~]# vim /etc/ha.d/ha.cf
keepalive 2
deadtime 30
bcast eth0
node node1.energoindia.com node2.energoindia.com
logfile /var/log/ha-log
debugfile /var/log/ha-debug
The next step is to create the resource file for heartbeat on node1 servers:
[root@node1 ~]# vim /etc/ha.d/haresources
node1.energoindia.com IPaddr::192.168.10.150/24/eth0 IPaddr::192.168.20.150/24/eth0
IPaddr::192.168.1.150/22/eth0 IPaddr::192.168.30.150/24/eth0 drbddisk::main Filesystem::/dev/drbd0::
/home::ext3 ldap httpd spamassassin clamd authdaemond imap imaps squid qmailctl
The last thing is to create the authentication file on node1 servers:
[root@node1 ~]# vim /etc/ha.d/authkeys
auth 3
3 md5 asd1234f
This password file should only be readable by the root user:
[root@node1 ~]# chmod 600 /etc/ha.d/authkeys
start heartbeat on node 1:
[root@node1 ~]# service heartbeat start
[root@node1 ~]# ifconfig
eth0 Link encap:Ethernet HWaddr E4:1F:13:C0:E9:CC inet addr:192.168.10.149 Bcast:192.168.10.255 Mask:255.255.255.0 inet6 addr: fe80::e61f:13ff:fec0:e9cc/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:171872956 errors:0 dropped:0 overruns:0 frame:0 TX packets:229211317 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:80916792381 (75.3 GiB) TX bytes:190588412808 (177.4 GiB) Interrupt:169 Memory:92000000-92012100
eth0:0 Link encap:Ethernet HWaddr E4:1F:13:C0:E9:CC
inet addr:192.168.10.150 Bcast:192.168.10.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Interrupt:169 Memory:92000000-92012100
eth0:1 Link encap:Ethernet HWaddr E4:1F:13:C0:E9:CC
inet addr:192.168.20.150 Bcast:192.168.20.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Interrupt:169 Memory:92000000-92012100
eth0:2 Link encap:Ethernet HWaddr E4:1F:13:C0:E9:CC
inet addr:192.168.1.150 Bcast:192.168.3.255 Mask:255.255.252.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Interrupt:169 Memory:92000000-92012100
eth0:3 Link encap:Ethernet HWaddr E4:1F:13:C0:E9:CC
inet addr:192.168.30.150 Bcast:192.168.30.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Interrupt:169 Memory:92000000-92012100
eth1 Link encap:Ethernet HWaddr E4:1F:13:C0:E9:CD
inet addr:10.0.0.1 Bcast:10.255.255.255 Mask:255.0.0.0
inet6 addr: fe80::e61f:13ff:fec0:e9cd/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:32136747 errors:0 dropped:0 overruns:0 frame:0
TX packets:92544365 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3368179779 (3.1 GiB) TX bytes:128171113009 (119.3 GiB)
Interrupt:138 Memory:94000000-94012100
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:7571848 errors:0 dropped:0 overruns:0 frame:0
TX packets:7571848 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:5164760682 (4.8 GiB) TX bytes:5164760682 (4.8 GiB)
:::::::MIGRATION OF QMAIL USERS, SYSTEM USERS, NORMAL USERS FROM A RUNNING SERVER TO NODE1::::::::::
Commands to type on old Linux system:
1. First create a tar ball of old uses (old Linux system). Create a directory:
- mkdir /root/move/
2. Setup UID filter limit:
- export UGIDLIMIT=500
3. Now copy /etc/passwd accounts to /root/move/passwd.mig using awk to filter out system account (i.e. only copy user accounts)
- awk -v LIMIT=$UGIDLIMIT -F: '($3>=LIMIT) && ($3!=65534)' /etc/passwd > /root/move/passwd.mig
4. Copy /etc/group file:
- awk -v LIMIT=$UGIDLIMIT -F: '($3>=LIMIT) && ($3!=65534)' /etc/group > /root/move/group.mig
5. Copy /etc/shadow file:
- awk -v LIMIT=$UGIDLIMIT -F: '($3>=LIMIT) && ($3!=65534) {print $1}' /etc/passwd | tee - |egrep -f - /etc/shadow > /root/move/shadow.mig
6. Copy /etc/gshadow (rarely used):
- cp /etc/gshadow /root/move/gshadow.mig
7. Use scp or usb pen or tape to copy /root/move to a new Linux system.
- scp -r /root/move/* user@new.linuxserver.com:/path/to/location
Commands to type on new Linux system
1. First, make a backup of current users and passwords:
[root@node1 ~]# mkdir /root/newsusers.bak [root@node1 ~]# cp /etc/passwd /etc/shadow /etc/group /etc/gshadow /root/newsusers.bak
2. Now restore passwd and other files in /etc/ [root@node1 ~]# cd /path/to/location [root@node1 ~]# cat passwd.mig >> /etc/passwd [root@node1 ~]# cat group.mig >> /etc/group [root@node1 ~]# cat shadow.mig >> /etc/shadow [root@node1 ~]# /bin/cp gshadow.mig /etc/gshadow
Now reboot system; when the Linux comes back, your user accounts will work as they did before on old system: [root@node1 ~]# reboot
- COPY ALL FILES OF QMAIL, LDAP ,SPAMASSASSIN , CLAMAV FROM A SERVER TO NODE1 USING LINKS:::::::::::
create below directory, make link of this and copy the data from running server.
[root@node1 ~]# mkdir -p /home/command [root@node1 ~]# ln -s /home/command /command ( copy data from running server )
[root@node1 ~]# mkdir -p /home/downloads [root@node1 ~]# ln -s /home/downloads /downloads
[root@node1 ~]# mkdir -p /home/packages [root@node1 ~]# ln -s /home/packages /packages
[root@node1 ~]# mkdir -p /home/service [root@node1 ~]# ln -s /home/service /service
[root@node1 ~]# mkdir -p /home/usr/local [root@node1 ~]# ln -s /home/usr/local /usr/local
[root@node1 ~]# mkdir -p /home/var/qmail [root@node1 ~]# ln -s /home/var/qmail /var/qmail
[root@node1 ~]# mkdir -p /home/usr/src [root@node1 ~]# ln -s /home/usr/src /usr/src
[root@node1 ~]# mkdir -p /home/var/log/qmail [root@node1 ~]# ln -s /home/var/log/qmail /var/log/qmail
[root@node1 ~]# touch /home/etc/tcp.smtp [root@node1 ~]# ln -s /home/etc/tcp.smtp /etc/tcp.smtp
[root@node1 ~]# touch /home/etc/tcp.smtp.cdb [root@node1 ~]# ln -s /home/etc/tcp.smtp.cdb /etc/tcp.smtp.cdb
[root@node1 ~]# mkdir -p /home/qmaildoc [root@node1 ~]# ln -s /home/qmaildoc /qmaildoc
Install openldap:
[root@node1 ~]# yum install openldap
[root@node1 ~]# mkdir -p /home/etc/openldap [root@node1 ~]# ln -s /home/etc/openldap /etc/openldap
[root@node1 ~]# mkdir -p /home/var/lib/ldap [root@node1 ~]# ln -s /home/var/lib/ldap /var/lib/ldap
[root@node1 ~]# mkdir -p /home/var/www/html [root@node1 ~]# ln -s /home/var/www/html /var/www/html
[root@node1 ~]# mkdir -p /home/var/www/sarg [root@node1 ~]# ln -s /home/var/www/sarg /var/www/sarg
Install clamav:
[root@node1 ~]# yum install clamav clamd
[root@node1 ~]# mkdir -p /home/var/spool/qscan [root@node1 ~]# ln -s /home/var/spool/qscan /var/spool/qscan
install spamassassin:
[root@node1 ~]# yum install spamassassin
[root@node1 ~]# mkdir -p /home/etc/mail/spamassassin/ [root@node1 ~]# touch /home/etc/mail/spamassassin/local.cf [root@node1 ~]# ln -s /home/etc/mail/spamassassin/local.cf /etc/mail/spamassassin/local.cf
install squid:
[root@node1 ~]# yum install squid
[root@node1 ~]# mkdir -p /home/etc/squid [root@node1 ~]# ln -s /home/etc/squid /etc/squid
configuration of fetchmail:
[root@node1 ~]# crontab -lu tetra
- /5 * * * * /usr/bin/fetchmail -v -U -F >> /home/tetra/fetchmail.log
copy the script of fetchmail... [root@node1 ~]# rsync -aP 192.168.10.155:/usr/bin/fetchmail /usr/bin/
copy the rc.local file........ [root@node1 ~]# rsync -aP 192.168.10.155:/etc/rc.local /etc/
Make link of below file....... [root@node1 ~]# ln -s /usr/local/sbin/authdaemond /etc/init.d/authdaemond [root@node1 ~]# ln -s /var/qmail/bin/qmailctl /usr/bin/qmailctl [root@node1 ~]# ln -s /usr/bin/qmailctl /etc/init.d/qmailctl
Now reboot the node1 to check all required files are copied properly and server is starting normally. Afterthen try below to check qmail, imap, ldap is working or not.
[root@node1 ~]# telnet localhost 25 Trying 127.0.0.1... Connected to localhost.localdomain (127.0.0.1). Escape character is '^]'. 220 mail.energoindia.com ESMTP
[root@node1 ~]# telnet localhost 110 Trying 127.0.0.1... Connected to localhost.localdomain (127.0.0.1). Escape character is '^]'. +OK <2081.1436400429@mail.energoindia.com>
[root@node1 ~]# telnet localhost 143 Trying 127.0.0.1... Connected to localhost.localdomain (127.0.0.1). Escape character is '^]'.
- OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION STARTTLS] Courier-IMAP ready. Copyright 1998-2005 Double Precision, Inc. See COPYING for distribution information.
[root@node1 ~]# telnet localhost 389 Trying 127.0.0.1... Connected to localhost.localdomain (127.0.0.1). Escape character is '^]'.
copy the /home from running server to node1......
[root@node1 ~]# rsync -alkvHP 192.168.10.155:/home/* /home/
After proper sync live this node1.
### ON NODE2 configuration ###
[root@node2 ~]# cat /etc/hosts
- Do not remove the following line, or various programs
- that require network functionality will fail.
127.0.0.1 mail.energoindia.com mail localhost.localdomain localhost
- 1 localhost6.localdomain6 localhost6
192.168.10.149 node1.energoindia.com node1 node.energoindia.com mail 192.168.1.150 node.energoindia.com node mail.energoindia.com mail 10.0.0.1 node1.energoindia.com node1 node.energoindia.com 10.0.0.2 node2.energoindia.com node2 node.energoindia.com 192.168.10.155 node2.energoindia.com node2 node.energoindia.com mail
:::::::::Installing DRBD:::::::
[root@node2 ~]# yum install drbd83-utils kmod-drbd83
Configuration should be same as in node1:
[root@node2 ~]#mv /etc/drbd.d/global_common.conf /etc/drbd.d/global_common.sample
[root@node2 ~]# vim /etc/drbd.d/global_common.conf global { usage-count no; }
common {
syncer { rate 100M; }
}
Create a new resource file that I called “main”,
[root@node2 ~]# vim /etc/drbd.d/main.res resource main { protocol C; startup { wfc-timeout 0; degr-wfc-timeout 120; } disk { on-io-error detach; } on node1.energoindia.com { device /dev/drbd0; disk /dev/VolGroup00/LogVol02; meta-disk internal; address 10.0.0.1:7788; } on node2.energoindia.com { device /dev/drbd0; disk /dev/mapper/VolGroup00-LogVol02; meta-disk internal; address 10.0.0.2:7788; } }
Now we have to create the metadata on node2:
[root@node2 ~]#dd if=/dev/zero of= /dev/VolGroup00/LogVol02 bs=1M count=50
[root@node2 ~]# drbdadm create-md main
Now start DRBD on this (node2) server:
[root@node2 ~]# service drbd start
Starting DRBD resources: [ d(main) s(main) n(main) ].
After starting drbd on node2 server, it start to synchronization between both nodes. It will take a little while to be completed. Please wait until it is done and move to the next step.
[root@node2 ~]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97) GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R5, 2013-09-27 15:48:18 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r----- ns:503808 nr:0 dw:0 dr:504472 al:0 bm:30 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:41437916 [>....................] sync'ed: 1.3% (40464/40956)M finish: 1:06:40 speed: 10,340 (10,280) K/sec
::::::Installing heartbeat::::::
[root@node2 ~]# yum install heartbeat
Create the following file on node2 servers with the exact same content as on node1:
[root@node2 ~]# vim /etc/ha.d/ha.cf keepalive 2 deadtime 30 bcast eth0 node node1.energoindia.com node2.energoindia.com logfile /var/log/ha-log debugfile /var/log/ha-debug
The next setp is to create the resource file for heartbeat on node2 servers with exact same content as on node1 again:
[root@node2 ~]# vim /etc/ha.d/haresources node1.energoindia.com IPaddr::192.168.10.150/24/eth0 IPaddr::192.168.20.150/24/eth0 IPaddr::192.168.1.150/22/eth0 IPaddr::192.168.30.150/24/eth0 drbddisk::main Filesystem::/dev/drbd0::/home::ext3 ldap httpd spamassassin clamd authdaemond imap imaps squid qmailctl
The last thing is to create the authentication file on node2 servers again with the same content as on node1: [root@node2 ~]# vim /etc/ha.d/authkeys auth 3 3 md5 asd1234f
This password file should only be readable by the root user:
[root@node2 ~]# chmod 600 /etc/ha.d/authkeys [root@node2 ~]# service heartbeat start
Again create same directory on node2 as in node1 and make links but before take backup of these directory:
[root@node2 ~]# mkdir -p /home/command [root@node2 ~]# ln -s /home/command /command
[root@node2 ~]# mkdir -p /home/downloads [root@node2 ~]# ln -s /home/downloads /downloads
[root@node2 ~]# mkdir -p /home/packages [root@node2 ~]# ln -s /home/packages /packages
[root@node2 ~]# mkdir -p /home/service [root@node2 ~]# ln -s /home/service /service
[root@node2 ~]# mkdir -p /home/usr/local [root@node2 ~]# ln -s /home/usr/local /usr/local
[root@node2 ~]# mkdir -p /home/var/qmail [root@node2 ~]# ln -s /home/var/qmail /var/qmail
[root@node2 ~]# mkdir -p /home/usr/src [root@node2 ~]# ln -s /home/usr/src /usr/src
[root@node2 ~]# mkdir -p /home/var/log/qmail [root@node2 ~]# ln -s /home/var/log/qmail /var/log/qmail
[root@node2 ~]# touch /home/etc/tcp.smtp [root@node2 ~]# ln -s /home/etc/tcp.smtp /etc/tcp.smtp
[root@node2 ~]# touch /home/etc/tcp.smtp.cdb [root@node2 ~]# ln -s /home/etc/tcp.smtp.cdb /etc/tcp.smtp.cdb
[root@node2 ~]# mkdir -p /home/qmaildoc [root@node2 ~]# ln -s /home/qmaildoc /qmaildoc
[root@node2 ~]# mkdir -p /home/etc/openldap [root@node2 ~]# ln -s /home/etc/openldap /etc/openldap
[root@node2 ~]# mkdir -p /home/var/lib/ldap [root@node2 ~]# ln -s /home/var/lib/ldap /var/lib/ldap
[root@node2 ~]# mkdir -p /home/var/www/html [root@node2 ~]# ln -s /home/var/www/html /var/www/html
[root@node2 ~]# mkdir -p /home/var/www/sarg [root@node2 ~]# ln -s /home/var/www/sarg /var/www/sarg
[root@node2 ~]# mkdir -p /home/var/spool/qscan [root@node2 ~]# ln -s /home/var/spool/qscan /var/spool/qscan
[root@node2 ~]# mkdir -p /home/etc/mail/spamassassin/ [root@node2 ~]# touch /home/etc/mail/spamassassin/local.cf [root@node2 ~]# ln -s /home/etc/mail/spamassassin/local.cf /etc/mail/spamassassin/local.cf
[root@node2 ~]# mkdir -p /home/etc/squid [root@node2 ~]# ln -s /home/etc/squid /etc/squid
Make link of below file....... [root@node2 ~]# ln -s /usr/local/sbin/authdaemond /etc/init.d/authdaemond [root@node2 ~]# ln -s /var/qmail/bin/qmailctl /usr/bin/qmailctl [root@node2 ~]# ln -s /usr/bin/qmailctl /etc/init.d/qmailctl
### Test the cluster ###
1. Stop heartbeat on node1: [root@node1 ~]# /etc/init.d/heartbeat stop
2. Watch log of heartbeat on node2, cluster will shift on node2. [root@node2 ~]# tailf /var/log/ha-log
3. On node2, check the mails are working and /home is mounted.
[root@node2 ~]# telnet localhost 25 Trying 127.0.0.1... Connected to localhost.localdomain (127.0.0.1). Escape character is '^]'. 220 mail.energoindia.com ESMTP
[root@node2 ~]# telnet localhost 110 Trying 127.0.0.1... Connected to localhost.localdomain (127.0.0.1). Escape character is '^]'. +OK <14050.1436639325@mail.energoindia.com>
[root@node2 ~]# telnet localhost 143 Trying 127.0.0.1... Connected to localhost.localdomain (127.0.0.1). Escape character is '^]'.
- OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION STARTTLS] Courier-IMAP ready. Copyright 1998-2005 Double Precision, Inc. See COPYING for distribution information.
[root@node2 ~]# telnet localhost 389 Trying 127.0.0.1... Connected to localhost.localdomain (127.0.0.1). Escape character is '^]'.
[root@node2 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00
95G 74G 17G 83% /
/dev/sda1 965M 24M 892M 3% /boot tmpfs 2.0G 0 2.0G 0% /dev/shm /dev/drbd0 446G 359G 65G 85% /home
4. now, to shift cluster again on node1.
stop heartbeat on node2 and start heartbeat on node1.
[root@node2 ~]# /etc/init.d/heartbeat stop [root@node1 ~]# /etc/init.d/heartbeat start
5. Now test the cluster is shifting to node2, if node1 goes down. It will shift. 6. Now start node1, the cluster will shift automatically on node1 from node2.