HPC High Processing Cluster in BITS PILANI

HIGH PROCESSING CLUSTER USING MPICH2

FOR 2 NODES AND 9 NODES CLUSTER

NAVDEEP MATHUR

ARCHITECTURE – VLSI LAB DATA CENTER (BITS-PILANI)

ARCHITECTURE DETAILS

• They have two IBM servers and nine Sunfire servers.
• On two IBM servers have installed HPC 2- Node Cluster, High Availability over NFS and multimaster LDAP.
• All home directories are stored in these two ibm servers have separate HDD and they don’t have SAN , So need to rsync home directories from one to another IBM Server.
• Other nine Sunfire servers have HPC-9 Node cluster Installed.
• All sunfire servers are LDAP Client and NFS Client
• HPC 2- Node cluster uses only two home directories /staff and /projects have GID is 501 and 502 and permission is 770 because of group of members access this directory from another user as well.
• HPC 9-Node cluster uses another two home directories that is mounted in all 9 machines /ugassignmnets and /pgassignments have GID is 503 and 504 and permission is 770 because of group of members access this directory from another user as well.
• LDAP have four different OU’s for these distribution of users according GID wise.
• Members of staff and projects can’t able to use sunfire systems because GID is mismatched.
• Once if you want to run application by mpi mode then first you need to login from ldap user then switch to wolf/lam user on 2-Node IBM HPC or wolf2/lam1 user on 9-Node sunfire HPC accordingly.
• All binaries like cadence,mentor,etc. are stored in /linuxeda on all 11 servers.
• All clients are use VNC as well as Sunray.
• VNC Server are installed in all 11 servers.
• Both master node handle ganglia for hpc monitoring http://172.20.4.25/ganglia and http://172.20.4.39/ganglia
• Script is running for syncing home directories in every 4 hours located at on 172.20.4.39 “/root/script/homedir.sh”
• Ldap user add script is located at on both servers under “/root/script/”, by default password is “user123”.

IBM 2-Node Machines IP/HOSTNAME Details :

172.20.4.39ibm-39.bits.ac

172.20.4.40ibm-40.bits.ac

Virtual HA IP : 172.20.4.43

Sunfire 9-Node Machines IP/HOSTNAMES Details :

172.20.4.25sunfire-25

172.20.4.26sunfire-26

172.20.4.27sunfire-27

172.20.4.28sunfire-28

172.20.4.29sunfire-29

172.20.4.30sunfire-30

172.20.4.31sunfire-31

172.20.4.32sunfire-32

172.20.4.33sunfire-33

INTRODUCTION OF HIGH PROCESSING CLUSTER

HPC cluster are used to run parallel programs for time-intensive computations and are of special interest to the scientific community. It commonly run simulations and other CPU-intensive programs that would take an inordinate amount of time to run on regular hardware.

The basic cluster Design

Grid computing is a broad term that typically refers to a service-oriented architecture (SOA) with collaboration among loosely coupled systems. Cluster-based HPC is a special case of grid computing in which the nodes are tightly coupled.

REQUIREMENT OF BITS-PILANI

BITS had a requirement of High Processing Cluster for there applications .For this one slave node server with one master node server and Eight Slave node server with one master node server were used. RHEL5 server where installed in two node cluster and CENTOS5 server installed in 9 node cluster.

All users authenticate via 389 ldap, IBM nodes handle Multi-master LDAP and other are LDAP Clients.

All users have home directories shared via NFS to other nodes.

Around 10 Clients they have windows machines and 35 Clients have Sunray Terminals and need to implement VNC Enterprise Server 5.03 with LDAP Authentication for Windows .All terminals handle by Sunray Server software.

High Availability over NFS is implemented on IBM Machines.

INSTALLATION OF 2-Node(IBM) HIGH PROCESSING CLUSTER

Master node

• NFS server running in High Availability.
• Ganglia daemon “gomnd” and “gmetad” running.
• apache running to provide web interface for ganglia
• mpd (MPI) daemon running.
• .mpd.conf file is created in wolf and lam home directory
• user wolf and group Beowulf responsible for operating cluster for all staff members, here “staff” is an OU of ldap.All “staff’ members use only wolf user for HPC.
• user lam and group prolam responsible for operating for all projects members,here “projects” is an OU of ldap. All “projects” members use only lam user for HPC

Slave node

• NFS server running in High Availability
• Ganglia daemon “gomnd” and “gmetad” running.
• mpd (MPI) is started by master node on each slave node.
• .mpd.conf file is created in wolf and lam home directory
• user wolf and group Beowulf responsible for operating cluster for all staff members, here “staff” is an OU of ldap.All “staff’ members use only wolf user for HPC.
• user lam and group prolam responsible for operating for all projects members,here “projects” is an OU of ldap. All “projects” members use only lam user for HPC

Note : wolf and lam are local system users in master and slave nodes.,

Preparation of Master node(IBM-39)

• First the entries where made in /etc/hosts of each cluster as follows

172.20.4.39 ibm-39.bits.ac ibm-39 172.20.4.40 ibm-40.bits.ac ibm-40

• Two groups was created and a two users who will be responsible for operating cluster was added as follows.

groupadd beowulf 
useradd -g beowulf wolf 
groupadd prolam
useradd –g prolam lam

Note : This document is concern about wolf user.

• Following line were added in /home/wolf/.bash_profile

umask 007 

• NFS server was created as follows in master node.

mkdir /linuxeda 

This file have all the binaries and need to rsync with all when changes occurred.

chmod 770 /linuxeda 

All home directories are in lvm mode name and location

/staff :This Home directory is for 2 node IBM HPC.

/projects:This Home directory is for 2 node IBM HPC.

/ugassignments:This Home directory is for 9 node Sunfire HPC.

/pgassignments:This Home directory is for 9 node Sunfire HPC.

Following entry is made in /etc/exports file

/staff * (rw,no_root_squash,sync) 
/projects * (rw,no_root_squash,sync) 
/ugassignments* (rw,no_root_squash,sync) 
/pgassignments* (rw,no_root_squash,sync) 

# service nfs start ; chkconfig nfs on

• Communication among the nodes takes place through ssh, so ssh login for user “wolf” has to be without need of entering the password. This was acheived as follow.

Uncomment the following in /etc/sshd/sshd_config

#RSAAuthentication yes 
#AuthorizedKeysFile .ssh/authorized_keys 

• Reboot and login as wolf because operation of cluster will always be done by user “wolf”
• To generate your public and private SSH keys, do this:

ssh-keygen –t rsa 

This will generate the two public/private key pain in .ssh directory. Copy the id_rsa.pub file into a file called "authorized_keys" right there in the .ssh directory.This key will be used later. Modify the security of file and directory

chmod 644 ~/.ssh/auth* 
chmod 755 ~/.ssh 

• Add these entries is /home/wolf/.bash_profile

export LAMRSH='ssh -x' 

ssh-agent sh -c 'ssh-add && bash'

INSTALLATION OF MPI(MESSAGE PASSING INTERFACE)

The MPI mpich2 can be downloaded from http://www.mpich.org/static/tarballs/1.2/mpich2-1.2.tar.gz

1. Unpack the tar file.

tar -zxvf mpich2-1.2.tar.gz

2. Choose an installation directory (the default is /usr/local/bin):

mkdir /home/wolf/mpich2-install

cd /home/wolf/mpich2-install/mpich2-install

./configure –prefix=/home/wolf/mpich2-install/ 2>&1 | tee c.txtmake 2>&1 | tee m.txt

make check

make install 2>&1 | tee mi.txt

Set the enviroment variable for user “wolf” in /home/wolf/.bash_profile

Add following lines.

MPICH2=/home/wolf/mpich2-install/bin/

PATH=$PATH:$HOME/bin:$MPICH2/bin

2.MPICH2, uses an external process manager for scalable startup of large MPI jobs. The default process manager is called MPD, which is a ring of daemons on the machines where you will run your MPI programs.

For security reasons, mpd looks in your home directory for a file named .mpd.conf containing the line in home directory. So create this file as follows

cd $HOME

touch .mpd.conf

chmod 600 .mpd.conf

add a line

secretword=<anysecretword>

we used “redhat123” as secretword in our case.

3.The first sanity check consists of bringing up a ring of one mpd on the local machine, testing one mpd command, and bringing the “ring” down.

mpd &

mpdtrace

mpdallexit

The output of mpdtrace should be the hostname of the machine you

are running on. The mpdallexit causes the mpd daemon to exit

4.Now we will bring up a ring of mpd’s on a set of machines. Create a file consisting of a list of machine names, one per line. Name this file .mpd.hosts in home directory of user wolf. These hostnames will be used as targets for ssh so include full domain names if necessary. Check that you can reach these machines with ssh without entering a password. You can test by doing.

Ssh node1 –l wolf

1. Start the daemons on (some of) the hosts in the file mpd.hosts

mpdboot -n <number to start> -f mpd.hosts

The number to start can be less than 1 + number of hosts in the file,but cannot be greater than 1 + the number of hosts in the file. One mpd is always started on the machine where mpdboot is run, and is counted in the number to start, whether or not it occurs in the file. By default, mpdboot will only start one mpd per machine even if the machine name appears in the hosts file multiple times.

mpdboot -n 5 -f mpd.hosts

This will start the cluster.

mpdtrace

This command will show you the nodes in the cluster like below

ibm-39.bits.ac

ibm-40.bits.ac

PREPERATION OF SLAVE NODE(IBM-40)

a. beowulf and wolf user were created as in master node

b. Edit file /home/wolf/.bash_pofile

export LAMRSH='ssh -x'

ssh-agent

MPICH2=/home/wolf/mpich2-install/mpich2

PATH=$HOME/bin:$MPICH2/bin

Make /linuxeda directory

mkdir /linuxeda

This file have all the binaries and need to rsync with all when changes occurred.

chmod 770 /linuxeda

All home directories are in lvm mode name and location

/staff :This Home directory is for 2 node IBM HPC.

/projects:This Home directory is for 2 node IBM HPC.

/ugassignments:This Home directory is for 9 node Sunfire HPC.

/pgassignments:This Home directory is for 9 node Sunfire HPC.

Following entry is made in /etc/exports file

/staff * (rw,no_root_squash,sync)

/projects * (rw,no_root_squash,sync)

/ugassignments* (rw,no_root_squash,sync)

/pgassignments* (rw,no_root_squash,sync)

# service nfs start ; chkconfig nfs on

c. Reboot all the nodes.

d. login as wolf users and install MPI as described above in MPI installation section above.

Note: Firewall should be disabled in all nodes as MPI uses randoms ports each time it is started.

“mpdboot -n 2 -f mpd.hosts” command should be executed as wolf user from master node. This will start mpd daemon in slave nodes

Now we will run an MPI job, using the mpiexec command as specified in the MPI-2 standard.

As part of the build process for MPICH2, a simple program to compute the value of pi by numerical integration is created in the mpich2-1.2/examples directory. If the current directory is the top level MPICH2 build directory, then you can run this program with

mpiexec -n 2 ./examples/cpi

The cpi example will tell you which hosts it is running on.

How to Run Cadence Application :

1. Firstly copy your application into /linuxeda
2. Copy user database into the wolf home directory

Note : Each users have their own data

1. Then Rsync /home /wolf/cadence6 to other nodes of same /home/wolf/cadence6
2. Ensure keyless ssh to execute commands to other nodes.
3. Command to execute application by mpi

mpiexec –np 4 /home/wulf/cadence6/spectre/linuxeda/cadence/(whatever)

Then it executes on both nodes and divide equally

Check with command

Mpdlistjobs on both nodes by wolf user

INSTALLATION OF GANGLIA (CLUSTER MONITERING TOOL) ON EACH NODE.

Ganglia can be downloaded from http://ganglia.sourceforge.net/downloads.php

Install ganglia as follows

1. rpm –Uvh rrdtool-x.x.x.rpm ( this is dependency of ganglia)
2. rpm –Uvh rrdtool-devel-.x.x.x.rpm
3. rpm –Uvh ganglia-gmond-x.x.x.rpm
4. rpm –Uvh ganglia-gmetad-x.x.x.rpm
5. rpm –Uvh ganglia-web-x.x.x.rpm ( web interface for ganglia to be installed only in master node)
6. create a file /etc/httpd/conf.d/ganglia.conf only in master node.

Alias /ganglia /usr/share/ganglia

open the file /etc/gmetad.conf and edit the file with following entry only in master node

data_source “unspecified” ibm-39 ibm-40

start the ganglia service in each node.

/etc/init.d/gmond start

/etc/init.d/gmetad start

gateway of all nodes should be master_node.

ganglia can be accessed in any browser with the url http://172.20.4.39/ganglia

Note : lam have same configuration as wolf

INSTALLATION OF 9-Node(SUNFIRE) HIGH PROCESSING CLUSTER

Master node

• NFS Client
• Ganglia daemon “gomnd” and “gmetad” running.
• apache running to provide web interface for ganglia
• mpd (MPI) daemon running.
• .mpd.conf file is created in wolf and lam home directory
• user wolf2 and group beowulf2 responsible for operating cluster for all pgassignments members, here “pgassignments” is an OU of ldap.All “pgassignments’ members use only wolf2 user for HPC.
• user lam1 and group prolam1 responsible for operating for all projects members,here “ugassignments” is an OU of ldap. All “ugassignments” members use only lam user for HPC

Slave node

• NFS client
• Ganglia daemon “gomnd” and “gmetad” running.
• mpd (MPI) is started by master node on each slave node.
• .mpd.conf file is created in wolf and lam home directory
• user wolf2 and group beowulf2 responsible for operating cluster for all pgassignments members, here “pgassignments” is an OU of ldap.All “pgassignments’ members use only wolf2 user for HPC.
• user lam1 and group prolam1 responsible for operating for all projects members,here “ugassignments” is an OU of ldap. All “ugassignments” members use only lam user for HPC

Note : wolf2 and lam1 are local system users in master and slave nodes.,

Preparation of Master node(IBM-39)

• First the entries where made in /etc/hosts of each cluster as follows

172.20.4.25sunfire-25

172.20.4.26sunfire-26

172.20.4.27sunfire-27

172.20.4.28sunfire-28

172.20.4.29sunfire-29

172.20.4.30sunfire-30

172.20.4.31sunfire-31

172.20.4.32sunfire-32

172.20.4.33sunfire-33

• Two groups was created and a two users who will be responsible for operating cluster was added as follows.

groupadd beowulf2

useradd -g beowulf2 wolf2

groupadd prolam1

useradd –g prolam1 lam1

Note : This document is concern about wolf2 user.

• Following line were added in /home/wolf2/.bash_profile

umask 007

• Mount all home directories for these HPC2 Cluster i.e.

Mkdir /ugassignments /pgassignments

mount –t nfs 172.20.4.43: /ugassignments /ugassignments

mount –t nfs 172.20.4.43: /pgassignments /pgassignments

Then make permanent entry on fstab

• Communication among the nodes takes place through ssh, so ssh login for user “wolf” has to be without need of entering the password. This was acheived as follow.

Uncomment the following in /etc/sshd/sshd_config

#RSAAuthentication yes

#AuthorizedKeysFile .ssh/authorized_keys

• Reboot and login as wolf because operation of cluster will always be done by user “wolf”
• To generate your public and private SSH keys, do this:

ssh-keygen –t rsa

This will generate the two public/private key pain in .ssh directory. Copy the id_rsa.pub file into a file called "authorized_keys" right there in the .ssh directory.This key will be used later. Modify the security of file and directory

chmod 644 ~/.ssh/auth*

chmod 755 ~/.ssh

• Add these entries is /home/wolf/.bash_profile

export LAMRSH='ssh -x'

ssh-agent sh -c 'ssh-add && bash'

• Install MPI as above HPC1 section
• Install slave nodes as above section but home directories mount as a nfs.
• Install Ganglia as above HPC1 section