Editing HPC High Processing Cluster in BITS PILANI

[[category:Bits pilani]]

==<center>'''HIGH PROCESSING CLUSTER USING MPICH2'''</center>==

'''FOR 2 NODES AND 9 NODES CLUSTER'''

'''NAVDEEP MATHUR'''


== <center>'''ARCHITECTURE – VLSI LAB DATA CENTER (BITS-PILANI)'''</center>==


<center>[[Image:bits_p.png]]</center>





==<center>'''ARCHITECTURE DETAILS'''</center>==

* They have two IBM servers and nine Sunfire servers.
* On two IBM servers have installed HPC 2- Node Cluster, High Availability over NFS and multimaster LDAP.
* All home directories are stored in these two ibm servers have separate HDD and they don’t have SAN , So need to rsync home directories from one to another IBM Server.
* Other nine Sunfire servers have HPC-9 Node cluster Installed.
* All sunfire servers are LDAP Client and NFS Client
* HPC 2- Node cluster uses only two home directories /staff and /projects have GID is 501 and 502 and permission is 770 because of group of members access this directory from another user as well.
* HPC 9-Node cluster uses another two home directories that is mounted in all 9 machines /ugassignmnets and /pgassignments have GID is 503 and 504 and permission is 770 because of group of members access this directory from another user as well.
* LDAP have four different OU’s for these distribution of users according GID wise.
* Members of staff and projects can’t able to use sunfire systems because GID is mismatched.
* Once if you want to run application by mpi mode then first you need to login from ldap user then switch to wolf/lam user on 2-Node IBM HPC or wolf2/lam1 user on 9-Node sunfire HPC accordingly.
* All binaries like cadence,mentor,etc. are stored in /linuxeda on all 11 servers.
* All clients are use VNC as well as Sunray.
* VNC Server are installed in all 11 servers.
* Both master node handle ganglia for hpc monitoring [http://172.20.4.25/ganglia http://172.20.4.25/ganglia] and [http://172.20.4.39/ganglia http://172.20.4.39/ganglia]
* Script is running for syncing home directories in every 4 hours located at on 172.20.4.39 “/root/script/homedir.sh”
* Ldap user add script is located at on both servers under “/root/script/”, by default password is “user123”.

='''IBM 2-Node Machines IP/HOSTNAME Details :'''=


172.20.4.39ibm-39.bits.ac

172.20.4.40ibm-40.bits.ac


'''Virtual HA IP : 172.20.4.43'''


=='''Sunfire 9-Node Machines IP/HOSTNAMES Details :'''==


172.20.4.25sunfire-25

172.20.4.26sunfire-26

172.20.4.27sunfire-27

172.20.4.28sunfire-28

172.20.4.29sunfire-29

172.20.4.30sunfire-30

172.20.4.31sunfire-31

172.20.4.32sunfire-32

172.20.4.33sunfire-33


== <center>'''INTRODUCTION OF HIGH PROCESSING CLUSTER'''</center>==

'''HPC cluster''' are used to run parallel programs for time-intensive computations and are of special interest to the scientific community. It commonly run simulations and other CPU-intensive programs that would take an inordinate amount of time to run on regular hardware. 

<center>'''The basic cluster Design'''</center>

<center>[[Image:bits-1.png]]</center>

Grid computing is a broad term that typically refers to a service-oriented architecture (SOA) with collaboration among loosely coupled systems. Cluster-based HPC is a special case of grid computing in which the nodes are tightly coupled.


==<center>'''REQUIREMENT OF BITS-PILANI'''</center>==

BITS had a requirement of High Processing Cluster for there applications .For this one slave node server with one master node server and Eight Slave node server with one master node server were used. RHEL5 server where installed in two node cluster and CENTOS5 server installed in 9 node cluster. 

All users authenticate via 389 ldap, IBM nodes handle Multi-master LDAP and other are LDAP Clients.

All users have home directories shared via NFS to other nodes. 

Around 10 Clients they have windows machines and 35 Clients have Sunray Terminals and need to implement VNC Enterprise Server 5.03 with LDAP Authentication for Windows .All terminals handle by Sunray Server software.

High Availability over NFS is implemented on IBM Machines.


=<center>'''INSTALLATION OF 2-Node(IBM) HIGH PROCESSING CLUSTER'''</center>=

=='''Master node'''==

* NFS server running in High Availability.
* Ganglia daemon “gomnd” and “gmetad” running. 
* apache running to provide web interface for ganglia 
* mpd (MPI) daemon running. 
* .mpd.conf file is created in wolf and lam home directory 
* user wolf and group Beowulf responsible for operating cluster for all staff members, here “staff” is an OU of ldap.All “staff’ members use only wolf user for HPC.
* user lam and group prolam responsible for operating for all projects members,here “projects” is an OU of ldap. All “projects” members use only lam user for HPC

=='''Slave node'''==

* NFS server running in High Availability
* Ganglia daemon “gomnd” and “gmetad” running. 
* mpd (MPI) is started by master node on each slave node. 
* .mpd.conf file is created in wolf and lam home directory 
* user wolf and group Beowulf responsible for operating cluster for all staff members, here “staff” is an OU of ldap.All “staff’ members use only wolf user for HPC.
* user lam and group prolam responsible for operating for all projects members,here “projects” is an OU of ldap. All “projects” members use only lam user for HPC

Note : wolf and lam are local system users in master and slave nodes.,

== Preparation of Master node(IBM-39) ==
* First the entries where made in /etc/hosts of each cluster as follows 


172.20.4.39		 ibm-39.bits.ac		ibm-39 
172.20.4.40       	 ibm-40.bits.ac		ibm-40

* Two groups was created and a two users who will be responsible for operating cluster was added as follows. 

 groupadd beowulf 
 useradd -g beowulf wolf 
 groupadd prolam
 useradd –g prolam lam

'''Note : This document is concern about wolf user.'''

* Following line were added in /home/wolf/.bash_profile 

 umask 007 

* NFS server was created as follows in master node. 

 mkdir /linuxeda 

This file have all the binaries and need to rsync with all when changes occurred.

 chmod 770 /linuxeda 

All home directories are in lvm mode name and location 

/staff :This Home directory is for 2 node IBM HPC.

/projects:This Home directory is for 2 node IBM HPC.

/ugassignments:This Home directory is for 9 node Sunfire HPC.

/pgassignments:This Home directory is for 9 node Sunfire HPC.

'''Following entry is made in /etc/exports file '''

 /staff <nowiki>* (rw,no_root_squash,sync) </nowiki>
 /projects <nowiki>* (rw,no_root_squash,sync) </nowiki>
 /ugassignments<nowiki>* (rw,no_root_squash,sync) </nowiki>
 /pgassignments<nowiki>* (rw,no_root_squash,sync) </nowiki>

 <nowiki># service nfs start ; chkconfig nfs on</nowiki>


* Communication among the nodes takes place through ssh, so ssh login for user “wolf” has to be without need of entering the password. This was acheived as follow. 

Uncomment the following in /etc/sshd/sshd_config 

 <nowiki>#RSAAuthentication yes </nowiki>
 <nowiki>#AuthorizedKeysFile .ssh/authorized_keys </nowiki>

* Reboot and login as wolf because operation of cluster will always be done by user “wolf”
* To generate your public and private SSH keys, do this: 

 ssh-keygen –t rsa 

This will generate the two public/private key pain in .ssh directory. Copy the id_rsa.pub file into a file called "authorized_keys" right there in the .ssh directory.This key will be used later. Modify the security of file and directory 

 chmod 644 ~/.ssh/auth* 
 chmod 755 ~/.ssh 

* Add these entries is /home/wolf/.bash_profile 

 export LAMRSH='ssh -x' 

 ssh-agent sh -c 'ssh-add && bash'

=<center>'''INSTALLATION OF MPI(MESSAGE PASSING INTERFACE)'''</center>=

The MPI mpich2 can be downloaded from [http://www.mpich.org/static/tarballs/1.2/mpich2-1.2.tar.gz http://www.mpich.org/static/tarballs/1.2/mpich2-1.2.tar.gz]

1. Unpack the tar ﬁle. 

 tar -zxvf mpich2-1.2.tar.gz

2. Choose an installation directory (the default is /usr/local/bin): 


 mkdir /home/wolf/mpich2-install
 cd /home/wolf/mpich2-install/mpich2-install
 ./configure --prefix=/home/wolf/mpich2-install/ 2>&1 | tee c.txt
 make 2>&1 | tee m.txt
 make check
 make install 2>&1 | tee mi.txt 

Set the enviroment variable for user “wolf” in /home/wolf/.bash_profile 

Add following lines. 

 MPICH2=/home/wolf/mpich2-install/bin/
 PATH=$PATH:$HOME/bin:$MPICH2/bin 

2.MPICH2, uses an external process manager for scalable startup of large MPI jobs. The default process manager is called MPD, which is a ring of daemons on the machines where you will run your MPI programs. 

For security reasons, mpd looks in your home directory for a ﬁle named .mpd.conf containing the line in home directory. So create this file as follows 

 cd $HOME 
 touch .mpd.conf 
 chmod 600 .mpd.conf 

add a line 

 <nowiki>secretword=<anysecretword> </nowiki>

we used “redhat123” as secretword in our case. 

3.The ﬁrst sanity check consists of bringing up a ring of one mpd on the local machine, testing one mpd command, and bringing the “ring” down. 

 mpd &
 mpdtrace
 mpdallexit


The output of mpdtrace should be the hostname of the machine you 

are running on. The mpdallexit causes the mpd daemon to exit 

4.Now we will bring up a ring of mpd’s on a set of machines. Create a ﬁle consisting of a list of machine names, one per line. Name this ﬁle .mpd.hosts in home directory of user wolf. These hostnames will be used as targets for ssh so include full domain names if necessary. Check that you can reach these machines with ssh without entering a password. You can test by doing. 

 Ssh node1 –l wolf 

# Start the daemons on (some of) the hosts in the ﬁle mpd.hosts 

 <nowiki>mpdboot -n <number to start> -f mpd.hosts </nowiki>

The number to start can be less than 1 + number of hosts in the ﬁle,but cannot be greater than 1 + the number of hosts in the ﬁle. One mpd is always started on the machine where mpdboot is run, and is counted in the number to start, whether or not it occurs in the ﬁle. By default, mpdboot will only start one mpd per machine even if the machine name appears in the hosts ﬁle multiple times. 

 mpdboot -n 5 -f mpd.hosts 

This will start the cluster. 

 mpdtrace 

This command will show you the nodes in the cluster like below 

ibm-39.bits.ac

ibm-40.bits.ac

== PREPERATION OF SLAVE NODE(IBM-40) ==
a. beowulf and wolf user were created as in master node 

b. Edit file /home/wolf/.bash_pofile 

 export LAMRSH='ssh -x' 
 ssh-agent 
 MPICH2=/home/wolf/mpich2-install/mpich2 
 PATH=$HOME/bin:$MPICH2/bin 

Make /linuxeda directory 

 mkdir /linuxeda 

This file have all the binaries and need to rsync with all when changes occurred.

 chmod 770 /linuxeda 

All home directories are in lvm mode name and location 

/staff :This Home directory is for 2 node IBM HPC.

/projects:This Home directory is for 2 node IBM HPC.

/ugassignments:This Home directory is for 9 node Sunfire HPC.

/pgassignments:This Home directory is for 9 node Sunfire HPC.

Following entry is made in /etc/exports file 

 /staff <nowiki>* (rw,no_root_squash,sync) </nowiki>
 /projects <nowiki>* (rw,no_root_squash,sync) </nowiki>
 /ugassignments<nowiki>* (rw,no_root_squash,sync) </nowiki>
 /pgassignments<nowiki>* (rw,no_root_squash,sync) </nowiki>
 <nowiki># service nfs start ; chkconfig nfs on</nowiki>

c. Reboot all the nodes. 

d. login as wolf users and install MPI as described above in MPI installation section above.

'''Note''': Firewall should be disabled in all nodes as MPI uses randoms ports each time it is started. 

 “mpdboot -n 2 -f mpd.hosts”  ** command should be executed as wolf user from master node. This will start mpd daemon in slave nodes **

Now we will run an MPI job, using the mpiexec command as specified in the MPI-2 standard.

As part of the build process for MPICH2, a simple program to compute the value of ''pi ''by numerical integration is created in the mpich2-1.2/examples directory. If the current directory is the top level MPICH2 build directory, then you can run this program with


 mpiexec -n 2 ./examples/cpi


The cpi example will tell you which hosts it is running on.

='''How to Run Cadence Application : '''=

# Firstly copy your application into /linuxeda
# Copy user database into the wolf home directory 

Note : Each users have their own data

# Then Rsync /home /wolf/cadence6 to other nodes of same /home/wolf/cadence6
# Ensure keyless ssh to execute commands to other nodes.
# Command to execute application by mpi

mpiexec –np 4 /home/wulf/cadence6/spectre/linuxeda/cadence/(whatever)

Then it executes on both nodes and divide equally

Check with command

Mpdlistjobs on both nodes by wolf user

= INSTALLATION OF GANGLIA (CLUSTER MONITERING TOOL) ON EACH NODE. =
Ganglia can be downloaded from [http://ganglia.sourceforge.net/downloads.php http://ganglia.sourceforge.net/downloads.php] 


Install ganglia as follows 

# rpm –Uvh rrdtool-x.x.x.rpm ( this is dependency of ganglia) 
# rpm –Uvh rrdtool-devel-.x.x.x.rpm 
# rpm –Uvh ganglia-gmond-x.x.x.rpm 
# rpm –Uvh ganglia-gmetad-x.x.x.rpm 
# rpm –Uvh ganglia-web-x.x.x.rpm ( web interface for ganglia to be installed only in master node) 
# create a file /etc/httpd/conf.d/ganglia.conf only in master node. 

Alias /ganglia /usr/share/ganglia 

open the file /etc/gmetad.conf and edit the file with following entry only in master node 

data_source “unspecified” ibm-39 ibm-40 


start the ganglia service in each node. 

/etc/init.d/gmond start 

/etc/init.d/gmetad start 

gateway of all nodes should be master_node. 

ganglia can be accessed in any browser with the url [http://172.20.4.39/ganglia http://172.20.4.39/ganglia] 


'''Note : lam have same configuration as wolf'''

= <center>'''INSTALLATION OF 9-Node(SUNFIRE) HIGH PROCESSING CLUSTER'''</center>=

=='''Master node'''==

* NFS Client
* Ganglia daemon “gomnd” and “gmetad” running. 
* apache running to provide web interface for ganglia 
* mpd (MPI) daemon running. 
* .mpd.conf file is created in wolf and lam home directory 
* user wolf2 and group beowulf2 responsible for operating cluster for all pgassignments members, here “pgassignments” is an OU of ldap.All “pgassignments’ members use only wolf2 user for HPC.
* user lam1 and group prolam1 responsible for operating for all projects members,here “ugassignments” is an OU of ldap. All “ugassignments” members use only lam user for HPC

=='''Slave node'''==

* NFS client
* Ganglia daemon “gomnd” and “gmetad” running. 
* mpd (MPI) is started by master node on each slave node. 
* .mpd.conf file is created in wolf and lam home directory 
* user wolf2 and group beowulf2 responsible for operating cluster for all pgassignments members, here “pgassignments” is an OU of ldap.All “pgassignments’ members use only wolf2 user for HPC.
* user lam1 and group prolam1 responsible for operating for all projects members,here “ugassignments” is an OU of ldap. All “ugassignments” members use only lam user for HPC

Note : wolf2 and lam1 are local system users in master and slave nodes., 

== Preparation of Master node(IBM-39) ==
* First the entries where made in /etc/hosts of each cluster as follows 

172.20.4.25sunfire-25

172.20.4.26sunfire-26

172.20.4.27sunfire-27

172.20.4.28sunfire-28

172.20.4.29sunfire-29

172.20.4.30sunfire-30

172.20.4.31sunfire-31

172.20.4.32sunfire-32

172.20.4.33sunfire-33


* Two groups was created and a two users who will be responsible for operating cluster was added as follows. 

 groupadd beowulf2 
 useradd -g beowulf2 wolf2 
 groupadd prolam1
 useradd –g prolam1 lam1

'''Note : This document is concern about wolf2 user.'''

* Following line were added in /home/wolf2/.bash_profile 

 umask 007 

* '''Mount all home directories for these HPC2 Cluster i.e.'''

Mkdir /ugassignments /pgassignments

 mount –t nfs 172.20.4.43: /ugassignments /ugassignments 
 mount –t nfs 172.20.4.43: /pgassignments /pgassignments 

Then make permanent entry on fstab

* Communication among the nodes takes place through ssh, so ssh login for user “wolf” has to be without need of entering the password. This was acheived as follow. 

Uncomment the following in /etc/sshd/sshd_config 

 <nowiki>#RSAAuthentication yes </nowiki>
 <nowiki>#AuthorizedKeysFile .ssh/authorized_keys </nowiki>

* Reboot and login as wolf because operation of cluster will always be done by user “wolf”
* To generate your public and private SSH keys, do this: 

 ssh-keygen –t rsa 

This will generate the two public/private key pain in .ssh directory. Copy the id_rsa.pub file into a file called "authorized_keys" right there in the .ssh directory.This key will be used later. Modify the security of file and directory 

 chmod 644 ~/.ssh/auth* 
 chmod 755 ~/.ssh 

* Add these entries is /home/wolf/.bash_profile 

 export LAMRSH='ssh -x' 
 ssh-agent sh -c 'ssh-add && bash' 

* Install MPI as above HPC1 section 
* Install slave nodes as above section but home directories mount as a nfs.
* Install Ganglia as above HPC1 section