Jump to content

HPC High Processing Cluster in BITS PILANI: Difference between revisions

From TetraWiki
← Older edit
Navdeep (talk | contribs)
42 edits
 
(4 intermediate revisions by one other user not shown)
Line 11: Line 11:




<center>[[Image:bits_p.png]]<center>
<center>[[Image:bits_p.png]]</center>




Line 36: Line 36:
* Ldap user add script is located at on both servers under “/root/script/”, by default password is “user123”.
* Ldap user add script is located at on both servers under “/root/script/”, by default password is “user123”.


'''IBM 2-Node Machines IP/HOSTNAME Details :'''
='''IBM 2-Node Machines IP/HOSTNAME Details :'''=




Line 47: Line 47:




'''Sunfire 9-Node Machines IP/HOSTNAMES Details :'''
=='''Sunfire 9-Node Machines IP/HOSTNAMES Details :'''==




Line 69: Line 69:




# <center>'''INTRODUCTION OF HIGH PROCESSING CLUSTER'''</center>
== <center>'''INTRODUCTION OF HIGH PROCESSING CLUSTER'''</center>==


'''HPC cluster''' are used to run parallel programs for time-intensive computations and are of special interest to the scientific community. It commonly run simulations and other CPU-intensive programs that would take an inordinate amount of time to run on regular hardware.  
'''HPC cluster''' are used to run parallel programs for time-intensive computations and are of special interest to the scientific community. It commonly run simulations and other CPU-intensive programs that would take an inordinate amount of time to run on regular hardware.  
Line 75: Line 75:
<center>'''The basic cluster Design'''</center>
<center>'''The basic cluster Design'''</center>


<center>[[Image:]]</center>
<center>[[Image:bits-1.png]]</center>


Grid computing is a broad term that typically refers to a service-oriented architecture (SOA) with collaboration among loosely coupled systems. Cluster-based HPC is a special case of grid computing in which the nodes are tightly coupled.
Grid computing is a broad term that typically refers to a service-oriented architecture (SOA) with collaboration among loosely coupled systems. Cluster-based HPC is a special case of grid computing in which the nodes are tightly coupled.




# <center>'''REQUIREMENT OF BITS-PILANI'''</center>
==<center>'''REQUIREMENT OF BITS-PILANI'''</center>==


BITS had a requirement of High Processing Cluster for there applications .For this one slave node server with one master node server and Eight Slave node server with one master node server were used. RHEL5 server where installed in two node cluster and CENTOS5 server installed in 9 node cluster.  
BITS had a requirement of High Processing Cluster for there applications .For this one slave node server with one master node server and Eight Slave node server with one master node server were used. RHEL5 server where installed in two node cluster and CENTOS5 server installed in 9 node cluster.  
Line 93: Line 93:




# <center>'''INSTALLATION OF 2-Node(IBM) HIGH PROCESSING CLUSTER'''</center>
=<center>'''INSTALLATION OF 2-Node(IBM) HIGH PROCESSING CLUSTER'''</center>=


'''Master node'''
=='''Master node'''==


* NFS server running in High Availability.
* NFS server running in High Availability.
Line 105: Line 105:
* user lam and group prolam responsible for operating for all projects members,here “projects” is an OU of ldap. All “projects” members use only lam user for HPC
* user lam and group prolam responsible for operating for all projects members,here “projects” is an OU of ldap. All “projects” members use only lam user for HPC


'''Slave node'''
=='''Slave node'''==


* NFS server running in High Availability
* NFS server running in High Availability
Line 119: Line 119:
* First the entries where made in /etc/hosts of each cluster as follows  
* First the entries where made in /etc/hosts of each cluster as follows  


172.20.4.39 ibm-39.bits.acibm-39


#
172.20.4.39 ibm-39.bits.ac ibm-39
##
172.20.4.40      ibm-40.bits.ac ibm-40
###
####  ibm-40.bits.acibm-40


* Two groups was created and a two users who will be responsible for operating cluster was added as follows.  
* Two groups was created and a two users who will be responsible for operating cluster was added as follows.  


groupadd beowulf  
groupadd beowulf  
 
useradd -g beowulf wolf  
useradd -g beowulf wolf  
groupadd prolam
 
useradd –g prolam lam
groupadd prolam
 
useradd –g prolam lam


'''Note : This document is concern about wolf user.'''
'''Note : This document is concern about wolf user.'''
Line 140: Line 134:
* Following line were added in /home/wolf/.bash_profile  
* Following line were added in /home/wolf/.bash_profile  


umask 007  
umask 007  


* NFS server was created as follows in master node.  
* NFS server was created as follows in master node.  


mkdir /linuxeda  
mkdir /linuxeda  


This file have all the binaries and need to rsync with all when changes occurred.
This file have all the binaries and need to rsync with all when changes occurred.


chmod 770 /linuxeda  
chmod 770 /linuxeda  


All home directories are in lvm mode name and location  
All home directories are in lvm mode name and location  
Line 162: Line 156:
'''Following entry is made in /etc/exports file '''
'''Following entry is made in /etc/exports file '''


/staff <nowiki>* (rw,no_root_squash,sync) </nowiki>
/staff <nowiki>* (rw,no_root_squash,sync) </nowiki>
/projects <nowiki>* (rw,no_root_squash,sync) </nowiki>
/ugassignments<nowiki>* (rw,no_root_squash,sync) </nowiki>
/pgassignments<nowiki>* (rw,no_root_squash,sync) </nowiki>


/projects <nowiki>* (rw,no_root_squash,sync) </nowiki>
<nowiki># service nfs start ; chkconfig nfs on</nowiki>
 
/ugassignments<nowiki>* (rw,no_root_squash,sync) </nowiki>
 
/pgassignments<nowiki>* (rw,no_root_squash,sync) </nowiki>
 
<nowiki># service nfs start ; chkconfig nfs on</nowiki>




Line 177: Line 168:
Uncomment the following in /etc/sshd/sshd_config  
Uncomment the following in /etc/sshd/sshd_config  


<nowiki>#RSAAuthentication yes </nowiki>
<nowiki>#RSAAuthentication yes </nowiki>
 
<nowiki>#AuthorizedKeysFile .ssh/authorized_keys </nowiki>
<nowiki>#AuthorizedKeysFile .ssh/authorized_keys </nowiki>


* Reboot and login as wolf because operation of cluster will always be done by user “wolf”
* Reboot and login as wolf because operation of cluster will always be done by user “wolf”
* To generate your public and private SSH keys, do this:  
* To generate your public and private SSH keys, do this:  


ssh-keygen –t rsa  
ssh-keygen –t rsa  


This will generate the two public/private key pain in .ssh directory. Copy the id_rsa.pub file into a file called "authorized_keys" right there in the .ssh directory.This key will be used later. Modify the security of file and directory  
This will generate the two public/private key pain in .ssh directory. Copy the id_rsa.pub file into a file called "authorized_keys" right there in the .ssh directory.This key will be used later. Modify the security of file and directory  


chmod 644 ~/.ssh/auth*  
chmod 644 ~/.ssh/auth*  
 
chmod 755 ~/.ssh  
chmod 755 ~/.ssh  


* Add these entries is /home/wolf/.bash_profile  
* Add these entries is /home/wolf/.bash_profile  


export LAMRSH='ssh -x'  
export LAMRSH='ssh -x'  


ssh-agent sh -c 'ssh-add && bash'  
ssh-agent sh -c 'ssh-add && bash'


# <center>'''INSTALLATION OF MPI(MESSAGE PASSING INTERFACE)'''</center>
=<center>'''INSTALLATION OF MPI(MESSAGE PASSING INTERFACE)'''</center>=


The MPI mpich2 can be downloaded from [http://www.mpich.org/static/tarballs/1.2/mpich2-1.2.tar.gz http://www.mpich.org/static/tarballs/1.2/mpich2-1.2.tar.gz]
The MPI mpich2 can be downloaded from [http://www.mpich.org/static/tarballs/1.2/mpich2-1.2.tar.gz http://www.mpich.org/static/tarballs/1.2/mpich2-1.2.tar.gz]
Line 204: Line 193:
1. Unpack the tar file.  
1. Unpack the tar file.  


tar -zxvf mpich2-1.2.tar.gz
tar -zxvf mpich2-1.2.tar.gz


2. Choose an installation directory (the default is /usr/local/bin):  
2. Choose an installation directory (the default is /usr/local/bin):  




mkdir /home/wolf/mpich2-install
mkdir /home/wolf/mpich2-install
 
cd /home/wolf/mpich2-install/mpich2-install
cd /home/wolf/mpich2-install/mpich2-install
./configure --prefix=/home/wolf/mpich2-install/ 2>&1 | tee c.txt
 
make 2>&1 | tee m.txt
./configure –prefix=/home/wolf/mpich2-install/ 2>&1 | tee c.txtmake 2>&1 | tee m.txt
make check
 
make install 2>&1 | tee mi.txt  
make check
 
make install 2>&1 | tee mi.txt  


Set the enviroment variable for user “wolf” in /home/wolf/.bash_profile  
Set the enviroment variable for user “wolf” in /home/wolf/.bash_profile  
Line 223: Line 209:
Add following lines.  
Add following lines.  


MPICH2=/home/wolf/mpich2-install/bin/
MPICH2=/home/wolf/mpich2-install/bin/
 
PATH=$PATH:$HOME/bin:$MPICH2/bin  
PATH=$PATH:$HOME/bin:$MPICH2/bin  


2.MPICH2, uses an external process manager for scalable startup of large MPI jobs. The default process manager is called MPD, which is a ring of daemons on the machines where you will run your MPI programs.  
2.MPICH2, uses an external process manager for scalable startup of large MPI jobs. The default process manager is called MPD, which is a ring of daemons on the machines where you will run your MPI programs.  
Line 231: Line 216:
For security reasons, mpd looks in your home directory for a file named .mpd.conf containing the line in home directory. So create this file as follows  
For security reasons, mpd looks in your home directory for a file named .mpd.conf containing the line in home directory. So create this file as follows  


cd $HOME  
cd $HOME  
 
touch .mpd.conf  
touch .mpd.conf  
chmod 600 .mpd.conf  
 
chmod 600 .mpd.conf  


add a line  
add a line  


<nowiki>secretword=<anysecretword> </nowiki>
<nowiki>secretword=<anysecretword> </nowiki>


we used “redhat123” as secretword in our case.  
we used “redhat123” as secretword in our case.  
Line 245: Line 228:
3.The first sanity check consists of bringing up a ring of one mpd on the local machine, testing one mpd command, and bringing the “ring” down.  
3.The first sanity check consists of bringing up a ring of one mpd on the local machine, testing one mpd command, and bringing the “ring” down.  


mpd &
mpd &
 
mpdtrace
 
mpdallexit
mpdtrace
 
 
mpdallexit




Line 260: Line 239:
4.Now we will bring up a ring of mpd’s on a set of machines. Create a file consisting of a list of machine names, one per line. Name this file .mpd.hosts in home directory of user wolf. These hostnames will be used as targets for ssh so include full domain names if necessary. Check that you can reach these machines with ssh without entering a password. You can test by doing.  
4.Now we will bring up a ring of mpd’s on a set of machines. Create a file consisting of a list of machine names, one per line. Name this file .mpd.hosts in home directory of user wolf. These hostnames will be used as targets for ssh so include full domain names if necessary. Check that you can reach these machines with ssh without entering a password. You can test by doing.  


Ssh node1 –l wolf  
Ssh node1 –l wolf  


# Start the daemons on (some of) the hosts in the file mpd.hosts  
# Start the daemons on (some of) the hosts in the file mpd.hosts  


<nowiki>mpdboot -n <number to start> -f mpd.hosts </nowiki>
<nowiki>mpdboot -n <number to start> -f mpd.hosts </nowiki>


The number to start can be less than 1 + number of hosts in the file,but cannot be greater than 1 + the number of hosts in the file. One mpd is always started on the machine where mpdboot is run, and is counted in the number to start, whether or not it occurs in the file. By default, mpdboot will only start one mpd per machine even if the machine name appears in the hosts file multiple times.  
The number to start can be less than 1 + number of hosts in the file,but cannot be greater than 1 + the number of hosts in the file. One mpd is always started on the machine where mpdboot is run, and is counted in the number to start, whether or not it occurs in the file. By default, mpdboot will only start one mpd per machine even if the machine name appears in the hosts file multiple times.  


mpdboot -n 5 -f mpd.hosts  
mpdboot -n 5 -f mpd.hosts  


This will start the cluster.  
This will start the cluster.  


mpdtrace  
mpdtrace  


This command will show you the nodes in the cluster like below  
This command will show you the nodes in the cluster like below  
Line 285: Line 264:
b. Edit file /home/wolf/.bash_pofile  
b. Edit file /home/wolf/.bash_pofile  


export LAMRSH='ssh -x'  
export LAMRSH='ssh -x'  
 
ssh-agent  
ssh-agent  
MPICH2=/home/wolf/mpich2-install/mpich2  
 
PATH=$HOME/bin:$MPICH2/bin  
MPICH2=/home/wolf/mpich2-install/mpich2  
 
PATH=$HOME/bin:$MPICH2/bin  


Make /linuxeda directory  
Make /linuxeda directory  


mkdir /linuxeda  
mkdir /linuxeda  


This file have all the binaries and need to rsync with all when changes occurred.
This file have all the binaries and need to rsync with all when changes occurred.


chmod 770 /linuxeda  
chmod 770 /linuxeda  


All home directories are in lvm mode name and location  
All home directories are in lvm mode name and location  
Line 313: Line 289:
Following entry is made in /etc/exports file  
Following entry is made in /etc/exports file  


/staff <nowiki>* (rw,no_root_squash,sync) </nowiki>
/staff <nowiki>* (rw,no_root_squash,sync) </nowiki>
 
/projects <nowiki>* (rw,no_root_squash,sync) </nowiki>
/projects <nowiki>* (rw,no_root_squash,sync) </nowiki>
/ugassignments<nowiki>* (rw,no_root_squash,sync) </nowiki>
 
/pgassignments<nowiki>* (rw,no_root_squash,sync) </nowiki>
/ugassignments<nowiki>* (rw,no_root_squash,sync) </nowiki>
<nowiki># service nfs start ; chkconfig nfs on</nowiki>
 
/pgassignments<nowiki>* (rw,no_root_squash,sync) </nowiki>
 
<nowiki># service nfs start ; chkconfig nfs on</nowiki>


c. Reboot all the nodes.  
c. Reboot all the nodes.  
Line 329: Line 301:
'''Note''': Firewall should be disabled in all nodes as MPI uses randoms ports each time it is started.  
'''Note''': Firewall should be disabled in all nodes as MPI uses randoms ports each time it is started.  


“mpdboot -n 2 -f mpd.hosts” command should be executed as wolf user from master node. This will start mpd daemon in slave nodes  
“mpdboot -n 2 -f mpd.hosts” ** command should be executed as wolf user from master node. This will start mpd daemon in slave nodes **


Now we will run an MPI job, using the mpiexec command as specified in the MPI-2 standard.
Now we will run an MPI job, using the mpiexec command as specified in the MPI-2 standard.
Line 336: Line 308:




mpiexec -n 2 ./examples/cpi
mpiexec -n 2 ./examples/cpi




The cpi example will tell you which hosts it is running on.
The cpi example will tell you which hosts it is running on.


'''How to Run Cadence Application : '''
='''How to Run Cadence Application : '''=


# Firstly copy your application into /linuxeda
# Firstly copy your application into /linuxeda
Line 393: Line 365:
'''Note : lam have same configuration as wolf'''
'''Note : lam have same configuration as wolf'''


# <center>'''INSTALLATION OF 9-Node(SUNFIRE) HIGH PROCESSING CLUSTER'''</center>
= <center>'''INSTALLATION OF 9-Node(SUNFIRE) HIGH PROCESSING CLUSTER'''</center>=


'''Master node'''
=='''Master node'''==


* NFS Client
* NFS Client
Line 405: Line 377:
* user lam1 and group prolam1 responsible for operating for all projects members,here “ugassignments” is an OU of ldap. All “ugassignments” members use only lam user for HPC
* user lam1 and group prolam1 responsible for operating for all projects members,here “ugassignments” is an OU of ldap. All “ugassignments” members use only lam user for HPC


'''Slave node'''
=='''Slave node'''==


* NFS client
* NFS client
Line 440: Line 412:
* Two groups was created and a two users who will be responsible for operating cluster was added as follows.  
* Two groups was created and a two users who will be responsible for operating cluster was added as follows.  


groupadd beowulf2  
groupadd beowulf2  
 
useradd -g beowulf2 wolf2  
useradd -g beowulf2 wolf2  
groupadd prolam1
 
useradd –g prolam1 lam1
groupadd prolam1
 
useradd –g prolam1 lam1


'''Note : This document is concern about wolf2 user.'''
'''Note : This document is concern about wolf2 user.'''
Line 452: Line 421:
* Following line were added in /home/wolf2/.bash_profile  
* Following line were added in /home/wolf2/.bash_profile  


umask 007  
umask 007  


* '''Mount all home directories for these HPC2 Cluster i.e.'''
* '''Mount all home directories for these HPC2 Cluster i.e.'''
Line 458: Line 427:
Mkdir /ugassignments /pgassignments
Mkdir /ugassignments /pgassignments


mount –t nfs 172.20.4.43: /ugassignments /ugassignments  
mount –t nfs 172.20.4.43: /ugassignments /ugassignments  
 
mount –t nfs 172.20.4.43: /pgassignments /pgassignments  
mount –t nfs 172.20.4.43: /pgassignments /pgassignments  


Then make permanent entry on fstab
Then make permanent entry on fstab
Line 468: Line 436:
Uncomment the following in /etc/sshd/sshd_config  
Uncomment the following in /etc/sshd/sshd_config  


<nowiki>#RSAAuthentication yes </nowiki>
<nowiki>#RSAAuthentication yes </nowiki>
 
<nowiki>#AuthorizedKeysFile .ssh/authorized_keys </nowiki>
<nowiki>#AuthorizedKeysFile .ssh/authorized_keys </nowiki>


* Reboot and login as wolf because operation of cluster will always be done by user “wolf”
* Reboot and login as wolf because operation of cluster will always be done by user “wolf”
* To generate your public and private SSH keys, do this:  
* To generate your public and private SSH keys, do this:  


ssh-keygen –t rsa  
ssh-keygen –t rsa  


This will generate the two public/private key pain in .ssh directory. Copy the id_rsa.pub file into a file called "authorized_keys" right there in the .ssh directory.This key will be used later. Modify the security of file and directory  
This will generate the two public/private key pain in .ssh directory. Copy the id_rsa.pub file into a file called "authorized_keys" right there in the .ssh directory.This key will be used later. Modify the security of file and directory  


chmod 644 ~/.ssh/auth*  
chmod 644 ~/.ssh/auth*  
 
chmod 755 ~/.ssh  
chmod 755 ~/.ssh  


* Add these entries is /home/wolf/.bash_profile  
* Add these entries is /home/wolf/.bash_profile  


export LAMRSH='ssh -x'  
export LAMRSH='ssh -x'  
 
ssh-agent sh -c 'ssh-add && bash'  
ssh-agent sh -c 'ssh-add && bash'  


* Install MPI as above HPC1 section  
* Install MPI as above HPC1 section  
* Install slave nodes as above section but home directories mount as a nfs.
* Install slave nodes as above section but home directories mount as a nfs.
* Install Ganglia as above HPC1 section
* Install Ganglia as above HPC1 section

Latest revision as of 10:08, 22 January 2013


HIGH PROCESSING CLUSTER USING MPICH2
[edit]

FOR 2 NODES AND 9 NODES CLUSTER

NAVDEEP MATHUR


ARCHITECTURE – VLSI LAB DATA CENTER (BITS-PILANI)
[edit]



ARCHITECTURE DETAILS
[edit]

  • They have two IBM servers and nine Sunfire servers.
  • On two IBM servers have installed HPC 2- Node Cluster, High Availability over NFS and multimaster LDAP.
  • All home directories are stored in these two ibm servers have separate HDD and they don’t have SAN , So need to rsync home directories from one to another IBM Server.
  • Other nine Sunfire servers have HPC-9 Node cluster Installed.
  • All sunfire servers are LDAP Client and NFS Client
  • HPC 2- Node cluster uses only two home directories /staff and /projects have GID is 501 and 502 and permission is 770 because of group of members access this directory from another user as well.
  • HPC 9-Node cluster uses another two home directories that is mounted in all 9 machines /ugassignmnets and /pgassignments have GID is 503 and 504 and permission is 770 because of group of members access this directory from another user as well.
  • LDAP have four different OU’s for these distribution of users according GID wise.
  • Members of staff and projects can’t able to use sunfire systems because GID is mismatched.
  • Once if you want to run application by mpi mode then first you need to login from ldap user then switch to wolf/lam user on 2-Node IBM HPC or wolf2/lam1 user on 9-Node sunfire HPC accordingly.
  • All binaries like cadence,mentor,etc. are stored in /linuxeda on all 11 servers.
  • All clients are use VNC as well as Sunray.
  • VNC Server are installed in all 11 servers.
  • Both master node handle ganglia for hpc monitoring http://172.20.4.25/ganglia and http://172.20.4.39/ganglia
  • Script is running for syncing home directories in every 4 hours located at on 172.20.4.39 “/root/script/homedir.sh”
  • Ldap user add script is located at on both servers under “/root/script/”, by default password is “user123”.

IBM 2-Node Machines IP/HOSTNAME Details :[edit]

172.20.4.39ibm-39.bits.ac

172.20.4.40ibm-40.bits.ac


Virtual HA IP : 172.20.4.43


Sunfire 9-Node Machines IP/HOSTNAMES Details :[edit]

172.20.4.25sunfire-25

172.20.4.26sunfire-26

172.20.4.27sunfire-27

172.20.4.28sunfire-28

172.20.4.29sunfire-29

172.20.4.30sunfire-30

172.20.4.31sunfire-31

172.20.4.32sunfire-32

172.20.4.33sunfire-33


INTRODUCTION OF HIGH PROCESSING CLUSTER
[edit]

HPC cluster are used to run parallel programs for time-intensive computations and are of special interest to the scientific community. It commonly run simulations and other CPU-intensive programs that would take an inordinate amount of time to run on regular hardware.

The basic cluster Design

Grid computing is a broad term that typically refers to a service-oriented architecture (SOA) with collaboration among loosely coupled systems. Cluster-based HPC is a special case of grid computing in which the nodes are tightly coupled.


REQUIREMENT OF BITS-PILANI
[edit]

BITS had a requirement of High Processing Cluster for there applications .For this one slave node server with one master node server and Eight Slave node server with one master node server were used. RHEL5 server where installed in two node cluster and CENTOS5 server installed in 9 node cluster.

All users authenticate via 389 ldap, IBM nodes handle Multi-master LDAP and other are LDAP Clients.

All users have home directories shared via NFS to other nodes.

Around 10 Clients they have windows machines and 35 Clients have Sunray Terminals and need to implement VNC Enterprise Server 5.03 with LDAP Authentication for Windows .All terminals handle by Sunray Server software.

High Availability over NFS is implemented on IBM Machines.


INSTALLATION OF 2-Node(IBM) HIGH PROCESSING CLUSTER
[edit]

Master node[edit]

  • NFS server running in High Availability.
  • Ganglia daemon “gomnd” and “gmetad” running.
  • apache running to provide web interface for ganglia
  • mpd (MPI) daemon running.
  • .mpd.conf file is created in wolf and lam home directory
  • user wolf and group Beowulf responsible for operating cluster for all staff members, here “staff” is an OU of ldap.All “staff’ members use only wolf user for HPC.
  • user lam and group prolam responsible for operating for all projects members,here “projects” is an OU of ldap. All “projects” members use only lam user for HPC

Slave node[edit]

  • NFS server running in High Availability
  • Ganglia daemon “gomnd” and “gmetad” running.
  • mpd (MPI) is started by master node on each slave node.
  • .mpd.conf file is created in wolf and lam home directory
  • user wolf and group Beowulf responsible for operating cluster for all staff members, here “staff” is an OU of ldap.All “staff’ members use only wolf user for HPC.
  • user lam and group prolam responsible for operating for all projects members,here “projects” is an OU of ldap. All “projects” members use only lam user for HPC

Note : wolf and lam are local system users in master and slave nodes.,

Preparation of Master node(IBM-39)[edit]

  • First the entries where made in /etc/hosts of each cluster as follows


172.20.4.39 ibm-39.bits.ac ibm-39 172.20.4.40 ibm-40.bits.ac ibm-40

  • Two groups was created and a two users who will be responsible for operating cluster was added as follows.
groupadd beowulf 
useradd -g beowulf wolf 
groupadd prolam
useradd –g prolam lam

Note : This document is concern about wolf user.

  • Following line were added in /home/wolf/.bash_profile
umask 007
  • NFS server was created as follows in master node.
mkdir /linuxeda

This file have all the binaries and need to rsync with all when changes occurred. 

chmod 770 /linuxeda

All home directories are in lvm mode name and location

/staff :This Home directory is for 2 node IBM HPC.

/projects:This Home directory is for 2 node IBM HPC.

/ugassignments:This Home directory is for 9 node Sunfire HPC.

/pgassignments:This Home directory is for 9 node Sunfire HPC.

Following entry is made in /etc/exports file 

/staff * (rw,no_root_squash,sync) 
/projects * (rw,no_root_squash,sync) 
/ugassignments* (rw,no_root_squash,sync) 
/pgassignments* (rw,no_root_squash,sync) 

# service nfs start ; chkconfig nfs on


  • Communication among the nodes takes place through ssh, so ssh login for user “wolf” has to be without need of entering the password. This was acheived as follow.

Uncomment the following in /etc/sshd/sshd_config 

#RSAAuthentication yes 
#AuthorizedKeysFile .ssh/authorized_keys
  • Reboot and login as wolf because operation of cluster will always be done by user “wolf”
  • To generate your public and private SSH keys, do this:
ssh-keygen –t rsa

This will generate the two public/private key pain in .ssh directory. Copy the id_rsa.pub file into a file called "authorized_keys" right there in the .ssh directory.This key will be used later. Modify the security of file and directory 

chmod 644 ~/.ssh/auth* 
chmod 755 ~/.ssh
  • Add these entries is /home/wolf/.bash_profile
export LAMRSH='ssh -x' 

ssh-agent sh -c 'ssh-add && bash'

INSTALLATION OF MPI(MESSAGE PASSING INTERFACE)
[edit]

The MPI mpich2 can be downloaded from http://www.mpich.org/static/tarballs/1.2/mpich2-1.2.tar.gz

1. Unpack the tar file. 

tar -zxvf mpich2-1.2.tar.gz

2. Choose an installation directory (the default is /usr/local/bin): 


mkdir /home/wolf/mpich2-install
cd /home/wolf/mpich2-install/mpich2-install
./configure --prefix=/home/wolf/mpich2-install/ 2>&1 | tee c.txt
make 2>&1 | tee m.txt
make check
make install 2>&1 | tee mi.txt

Set the enviroment variable for user “wolf” in /home/wolf/.bash_profile

Add following lines. 

MPICH2=/home/wolf/mpich2-install/bin/
PATH=$PATH:$HOME/bin:$MPICH2/bin

2.MPICH2, uses an external process manager for scalable startup of large MPI jobs. The default process manager is called MPD, which is a ring of daemons on the machines where you will run your MPI programs.

For security reasons, mpd looks in your home directory for a file named .mpd.conf containing the line in home directory. So create this file as follows 

cd $HOME 
touch .mpd.conf 
chmod 600 .mpd.conf

add a line 

secretword=<anysecretword>

we used “redhat123” as secretword in our case.

3.The first sanity check consists of bringing up a ring of one mpd on the local machine, testing one mpd command, and bringing the “ring” down. 

mpd &
mpdtrace
mpdallexit


The output of mpdtrace should be the hostname of the machine you

are running on. The mpdallexit causes the mpd daemon to exit

4.Now we will bring up a ring of mpd’s on a set of machines. Create a file consisting of a list of machine names, one per line. Name this file .mpd.hosts in home directory of user wolf. These hostnames will be used as targets for ssh so include full domain names if necessary. Check that you can reach these machines with ssh without entering a password. You can test by doing. 

Ssh node1 –l wolf
  1. Start the daemons on (some of) the hosts in the file mpd.hosts
mpdboot -n <number to start> -f mpd.hosts

The number to start can be less than 1 + number of hosts in the file,but cannot be greater than 1 + the number of hosts in the file. One mpd is always started on the machine where mpdboot is run, and is counted in the number to start, whether or not it occurs in the file. By default, mpdboot will only start one mpd per machine even if the machine name appears in the hosts file multiple times. 

mpdboot -n 5 -f mpd.hosts

This will start the cluster. 

mpdtrace

This command will show you the nodes in the cluster like below

ibm-39.bits.ac

ibm-40.bits.ac

PREPERATION OF SLAVE NODE(IBM-40)[edit]

a. beowulf and wolf user were created as in master node

b. Edit file /home/wolf/.bash_pofile 

export LAMRSH='ssh -x' 
ssh-agent 
MPICH2=/home/wolf/mpich2-install/mpich2 
PATH=$HOME/bin:$MPICH2/bin

Make /linuxeda directory 

mkdir /linuxeda

This file have all the binaries and need to rsync with all when changes occurred. 

chmod 770 /linuxeda

All home directories are in lvm mode name and location

/staff :This Home directory is for 2 node IBM HPC.

/projects:This Home directory is for 2 node IBM HPC.

/ugassignments:This Home directory is for 9 node Sunfire HPC.

/pgassignments:This Home directory is for 9 node Sunfire HPC.

Following entry is made in /etc/exports file 

/staff * (rw,no_root_squash,sync) 
/projects * (rw,no_root_squash,sync) 
/ugassignments* (rw,no_root_squash,sync) 
/pgassignments* (rw,no_root_squash,sync) 
# service nfs start ; chkconfig nfs on

c. Reboot all the nodes.

d. login as wolf users and install MPI as described above in MPI installation section above.

Note: Firewall should be disabled in all nodes as MPI uses randoms ports each time it is started. 

“mpdboot -n 2 -f mpd.hosts”  ** command should be executed as wolf user from master node. This will start mpd daemon in slave nodes **

Now we will run an MPI job, using the mpiexec command as specified in the MPI-2 standard.

As part of the build process for MPICH2, a simple program to compute the value of pi by numerical integration is created in the mpich2-1.2/examples directory. If the current directory is the top level MPICH2 build directory, then you can run this program with 


mpiexec -n 2 ./examples/cpi


The cpi example will tell you which hosts it is running on.

How to Run Cadence Application : [edit]

  1. Firstly copy your application into /linuxeda
  2. Copy user database into the wolf home directory

Note : Each users have their own data

  1. Then Rsync /home /wolf/cadence6 to other nodes of same /home/wolf/cadence6
  2. Ensure keyless ssh to execute commands to other nodes.
  3. Command to execute application by mpi

mpiexec –np 4 /home/wulf/cadence6/spectre/linuxeda/cadence/(whatever)

Then it executes on both nodes and divide equally

Check with command

Mpdlistjobs on both nodes by wolf user

INSTALLATION OF GANGLIA (CLUSTER MONITERING TOOL) ON EACH NODE.[edit]

Ganglia can be downloaded from http://ganglia.sourceforge.net/downloads.php


Install ganglia as follows

  1. rpm –Uvh rrdtool-x.x.x.rpm ( this is dependency of ganglia)
  2. rpm –Uvh rrdtool-devel-.x.x.x.rpm
  3. rpm –Uvh ganglia-gmond-x.x.x.rpm
  4. rpm –Uvh ganglia-gmetad-x.x.x.rpm
  5. rpm –Uvh ganglia-web-x.x.x.rpm ( web interface for ganglia to be installed only in master node)
  6. create a file /etc/httpd/conf.d/ganglia.conf only in master node.

Alias /ganglia /usr/share/ganglia

open the file /etc/gmetad.conf and edit the file with following entry only in master node

data_source “unspecified” ibm-39 ibm-40


start the ganglia service in each node.

/etc/init.d/gmond start

/etc/init.d/gmetad start

gateway of all nodes should be master_node.

ganglia can be accessed in any browser with the url http://172.20.4.39/ganglia


Note : lam have same configuration as wolf

INSTALLATION OF 9-Node(SUNFIRE) HIGH PROCESSING CLUSTER
[edit]

Master node[edit]

  • NFS Client
  • Ganglia daemon “gomnd” and “gmetad” running.
  • apache running to provide web interface for ganglia
  • mpd (MPI) daemon running.
  • .mpd.conf file is created in wolf and lam home directory
  • user wolf2 and group beowulf2 responsible for operating cluster for all pgassignments members, here “pgassignments” is an OU of ldap.All “pgassignments’ members use only wolf2 user for HPC.
  • user lam1 and group prolam1 responsible for operating for all projects members,here “ugassignments” is an OU of ldap. All “ugassignments” members use only lam user for HPC

Slave node[edit]

  • NFS client
  • Ganglia daemon “gomnd” and “gmetad” running.
  • mpd (MPI) is started by master node on each slave node.
  • .mpd.conf file is created in wolf and lam home directory
  • user wolf2 and group beowulf2 responsible for operating cluster for all pgassignments members, here “pgassignments” is an OU of ldap.All “pgassignments’ members use only wolf2 user for HPC.
  • user lam1 and group prolam1 responsible for operating for all projects members,here “ugassignments” is an OU of ldap. All “ugassignments” members use only lam user for HPC

Note : wolf2 and lam1 are local system users in master and slave nodes.,

Preparation of Master node(IBM-39)[edit]

  • First the entries where made in /etc/hosts of each cluster as follows

172.20.4.25sunfire-25

172.20.4.26sunfire-26

172.20.4.27sunfire-27

172.20.4.28sunfire-28

172.20.4.29sunfire-29

172.20.4.30sunfire-30

172.20.4.31sunfire-31

172.20.4.32sunfire-32

172.20.4.33sunfire-33


  • Two groups was created and a two users who will be responsible for operating cluster was added as follows.
groupadd beowulf2 
useradd -g beowulf2 wolf2 
groupadd prolam1
useradd –g prolam1 lam1

Note : This document is concern about wolf2 user.

  • Following line were added in /home/wolf2/.bash_profile
umask 007
  • Mount all home directories for these HPC2 Cluster i.e.

Mkdir /ugassignments /pgassignments 

mount –t nfs 172.20.4.43: /ugassignments /ugassignments 
mount –t nfs 172.20.4.43: /pgassignments /pgassignments

Then make permanent entry on fstab

  • Communication among the nodes takes place through ssh, so ssh login for user “wolf” has to be without need of entering the password. This was acheived as follow.

Uncomment the following in /etc/sshd/sshd_config 

#RSAAuthentication yes 
#AuthorizedKeysFile .ssh/authorized_keys
  • Reboot and login as wolf because operation of cluster will always be done by user “wolf”
  • To generate your public and private SSH keys, do this:
ssh-keygen –t rsa

This will generate the two public/private key pain in .ssh directory. Copy the id_rsa.pub file into a file called "authorized_keys" right there in the .ssh directory.This key will be used later. Modify the security of file and directory 

chmod 644 ~/.ssh/auth* 
chmod 755 ~/.ssh
  • Add these entries is /home/wolf/.bash_profile
export LAMRSH='ssh -x' 
ssh-agent sh -c 'ssh-add && bash'
  • Install MPI as above HPC1 section
  • Install slave nodes as above section but home directories mount as a nfs.
  • Install Ganglia as above HPC1 section
Retrieved from "https://wiki.tetrain.com/index.php?title=HPC_High_Processing_Cluster_in_BITS_PILANI&oldid=1147"