HPC High Processing Cluster in IIT DELHI

Documentation of High Processing Cluster in IIT ( 2008 )

IIT had a requirement of High Processing Cluster for there application called “Fluent”.For this four slave node computer and one master node computer were used. RHEL5 client where installed in each slave server and RHEL 5 server was installed in master node.

Diagramatic View of High Processing Cluster

Master node

1. NFS server running
2. Ganglia daemon “gomnd” and “gmetad” running.
3. apache running to provide web interface for ganglia
4. mpd (MPI) daemon running.
5. .mpd.conf file is created in wolf home directory
6. user wolf and group Beowulf responsible for operating cluster

Slave node

1. NFS share is mounted on four slave nodes
2. Ganglia daemon “gomnd” and “gmetad” running.
3. mpd (MPI) is started by master node on each slave node.
4. .mpd.conf file is created in wolf home directory in each slave node.

User wolf and group Beowulf responsible for operating cluster

Preparation of Master node.

a.First the entries where made in /etc/hosts of each cluster as follows

10.52.3.111 master_node

10.52.3.112node1

10.52.3.113node2

10.52.3.114node3

10.52.3.115node5

b.A group was created and a user who will be responsible for operating cluster was added as follows.

groupadd beowulf

useradd -g beowulf wolf

c.Following line were added in /home/wolf/.bash_profile

umask 007

d.NFS server was created as follows in master node.

mkdir /shared

chmod 770 /shared

chown -R wolf:beowulf /shared

Following entry is made in /etc/exports file

/wolf * (rw,no_root_squash,sync)

e.Communication among the nodes takes place through ssh, so ssh login for user “wolf” has to be without need of entering the password. This was acheived as follow.

Uncomment the following in /etc/sshd/sshd_config

#RSAAuthentication yes

#AuthorizedKeysFile .ssh/authorized_keys

f.Reboot and login as wolf because operation of cluster will always be done by user “wolf”

g.To generate your public and private SSH keys, do this:

ssh-keygen -b 1024 -f ~/.ssh/id_rsa -t rsa -N ""

This will generate the two public/private key pain in .ssh directory. Copy the id_rsa.pub file into a file called "authorized_keys" right there in the .ssh directory.This key will be used later. Modify the security of file and directory

chmod 644 ~/.ssh/auth*

chmod 755 ~/.ssh

h.Add these entries is /home/wolf/.bash_profile

export LAMRSH='ssh -x'

ssh-agent sh -c 'ssh-add && bash'

g. Installation of MPI ( Message Passing Interface)

Initally LAM was used as MPI but the application Fluent doesn't work with this MPI so we installed MPICH2. The MPI mpich2 can be downloaded from http://www.mcs.anl.gov/research/projects/mpich2

1. Unpack the tar file.

tar xfz mpich2.x.x.x.tar.gz

2. Choose an installation directory (the default is /usr/local/bin):

mkdir /usr/local/mpich2/

cd /mpich2

./configure –prefix=/usr/local/mpich2

make

make install.

Set the enviroment variable for user “wolf” in /home/wolf/.bash_profile

Add following lines.

MPICH2=/usr/local/mpich2

PATH=$PATH:$HOME/bin:$MPICH2/bin

2.MPICH2, uses an external process manager for scalable startup of large MPI jobs. The default process manager is called MPD, which is a ring of daemons on the machines where you will run your MPI programs.

For security reasons, mpd looks in your home directory for a file named

.mpd.conf containing the line in home directory. So create this file as follows

cd $HOME

touch .mpd.conf

chmod 600 .mpd.conf

add a line

secretword=<anysecretword>

we used “fluentwolf” as secretword in our case.

3.The first sanity check consists of bringing up a ring of one mpd on

the local machine, testing one mpd command, and bringing the “ring”

down.

mpd &

mpdtrace

mpdallexit

The output of mpdtrace should be the hostname of the machine you

are running on. The mpdallexit causes the mpd daemon to exit

4.Now we will bring up a ring of mpd’s on a set of machines. Create a

file consisting of a list of machine names, one per line. Name this file

.mpd.hosts in home directory of user wolf. These hostnames will be used as targets for ssh so include full domain names if necessary. Check that you can reach these machines with ssh without entering a password. You can test by doing.

Ssh node1 –l wolf

1. Start the daemons on (some of) the hosts in the file mpd.hosts

mpdboot -n <number to start> -f mpd.hosts

The number to start can be less than 1 + number of hosts in the file,but cannot be greater than 1 + the number of hosts in the file. One mpd is always started on the machine where mpdboot is run, and is counted in the number to start, whether or not it occurs in the file. By default, mpdboot will only start one mpd per machine even if the machine name appears in the hosts file multiple times.

mpdboot -n 5 -f mpd.hosts

This will start the cluster.

mpdtrace

This command will show you the nodes in the cluster like below

Master_node

Node1

Node2

Preparation of slave nodes

a. beowulf and wolf user were created as in master node

b. Edit file /home/wolf/.bash_pofile

export LAMRSH='ssh -x'

ssh-agent

MPICH2=/usr/local/mpich2

PATH=$HOME/bin:$MPICH2/bin

Make /shared directory

mkdir shared

c. Create entry in /etc/fstab/

master_node:/shared /shared nfs rw,hard,intr 0 0

This will mount the nfs shared drive at boot time

c. Reboot all the nodes.

d. login as wolf users and install MPI as described above in MPI installation section.

Note: Firewall should be disabled in all nodes as MPI uses randoms ports each time it is started.

“mpdboot -n 5 -f mpd.hosts” command should be executed as root user from master node. This will start mpd daemon in slave nodes

First master node must boot then slave node so that nfs share are mounted on slave machine.

2 . Finally we installed the IIT application “Fluent” on to the shared directory /shared which is shared by all nodes too. Mdp daemon has to be started on all the nodes as mentioned above and then only master node as user wolf can start the application.

Installation of ganglia (cluster monitoring tool) on each node.

Ganglia can be downloaded from http://ganglia.sourceforge.net/downloads.php

1. 1. Install ganglia as follows

1. rpm –Uvh rrdtool-x.x.x.rpm ( this is dependency of ganglia)
2. rpm –Uvh rrdtool-devel-.x.x.x.rpm
3. rpm –Uvh ganglia-gmond-x.x.x.rpm
4. rpm –Uvh ganglia-gmetad-x.x.x.rpm
5. rpm –Uvh ganglia-web-x.x.x.rpm ( web interface for ganglia to be installed only in master node)
6. create a file /etc/httpd/conf.d/ganglia.conf only in master node.

Alias /ganglia /usr/share/ganglia

1. open the file /etc/gmetad.conf and edit the file with following entry only in master node

data_source “unspecified” master_node node1 node2 node3 node4

1. start the ganglia service in each node.

/etc/init.d/gmond start

/etc/init.d/gmetad start

1. gateway of all nodes should be master_node.

1. ganglia can be accessed in any browser with the url http://10.52.3.110/ganglia