Editing SAMBA CLUSTER PUNJAB GOVT IFMS-IWDMS PROJECT (section)

= Installation of the cluster =
The Samba service will be clustered using CTDB (Clustered TDB), with RHCS supporting it. The Red Hat Enterprise Linux 6.1 release provides support for running Clustered Samba. This requires that you install and configure CTDB on all nodes in a cluster, which you use in conjunction with GFS2 clustered file systems.


Active node: pbifmsntp1.pbifmsiwdms.pjb 

Standby node: pbifmsntp2.pbifmsiwdms.pjb

Cluster service hostname: samba.pbifmsiwdms.pjb

Mounted HDD: /dev/mapper/csmb_vg-csmb_lv ; /dev/mapper/csmb_vg-csmb_lv

== CTDB Overview  ==
CTDB is a cluster implementation of the TDB database used by Samba. To use CTDB, a clustered file 

system must be available and shared on all nodes in the cluster. CTDB provides clustered features 

on top of this clustered file system. CTDB manages node membership, recovery/failover, IP relocation and Samba services. 


== Required Packages  ==
In addition to the standard packages required to run the Red Hat High Availability Add-On and the Red Hat Resilient Storage Add-On, running Samba with Red Hat Enterprise Linux clustering requires the following packages. Install via 'yum': 

• ctdb 

• samba 

• samba-common 

• samba-winbind-clients

• pacemaker

• gfs2-utils


== GFS2 Configuration  ==
Configuring Samba with the Red Hat Enterprise Linux clustering requires two GFS2 file systems: One small file system for CTDB, and a second file system for the Samba share. 


Before creating the GFS2 file systems, first create an LVM logical volume for each of the file systems. 


We use the following volumes: 

• /dev/csmb_vg/csmb_lv, which will hold the user data that will be exported via a Samba share. 

• /dev/csmb_vg/ctdb_lv, which will store the shared CTDB state information and needs to be 1GB in size. 


== Clustered Samba Configuration  ==
You create clustered volume groups and logical volumes on one node of the cluster only. 


To create a GFS2 file system on a logical volume, run the mkfs.gfs2 command. You run this command on one cluster node only. 

To create the file system to host the Samba share on the logical volume /dev/csmb_vg/csmb_lv, execute the following command: 


<nowiki># mkfs.gfs2 -j3 -p lock_dlm -t csmb:gfs2 /dev/csmb_vg/csmb_lv</nowiki> 


The meaning of the parameters is as follows: 

-j 

Specifies the number of journals to create in the filesystem. This example uses a cluster with two nodes, so we create one journal per node. One journal is extra for future use.


-p 

Specifies the locking protocol. lock_dlm is the locking protocol GFS2 uses for inter-node communication. 


-t 

Specifies the lock table name and is of the format cluster_name:fs_name. In our structure, the cluster name as specified in the cluster.conf file is csmb, and we use gfs2 as the name for the file system. 


The /dev/csmb_vg/csmb_lv file system will be mounted at /mnt/gfs2 on all nodes. This mount point must match the value that you specify as the location of the share directory with the path = option in the /etc/samba/smb.conf file. 


To create the file system to host the CTDB state information on the logical volume /dev/csmb_vg/ ctdb_lv, execute the following command: 


<nowiki># mkfs.gfs2 -j3 -p lock_dlm -t csmb:ctdb_state /dev/csmb_vg/ctdb_lv</nowiki> 


Note that this command specifies a different lock table name than the lock table in the example that created the filesystem on /dev/csmb_vg/csmb_lv. This distinguishes the lock table names for the different devices used for the file systems. 


In this example, the /dev/csmb_vg/ctdb_lv file system will be mounted at /mnt/ctdb on all nodes. This mount point must match the value that you specify as the location of the .ctdb.lock file with the CTDB_RECOVERY_LOCK option in the /etc/sysconfig/ctdb file. 


== CTDB Configuration  ==
The CTDB configuration file is located at /etc/sysconfig/ctdb. The mandatory fields that must be configured for CTDB operation are as follows: 

• CTDB_NODES 

• CTDB_PUBLIC_ADDRESSES 

• CTDB_RECOVERY_LOCK 

• CTDB_MANAGES_SAMBA (must be enabled) 


The following is our configuration file for CTDB operation: 


CTDB_RECOVERY_LOCK="/mnt/ctdb/.ctdb.lock" 

CTDB_PUBLIC_ADDRESSES=/etc/ctdb/public_addresses 

CTDB_MANAGES_SAMBA=yes 

CTDB_MANAGES_WINBIND=no 

CTDB_NODES=/etc/ctdb/nodes 

CTDB_DEBUGLEVEL=ERR 


The meaning of these parameters is as follows. 


CTDB_NODES 

Specifies the location of the file which contains the cluster node list. 


The /etc/ctdb/nodes file that CTDB_NODES references simply lists the IP addresses of the cluster nodes, as in the following example: 

192.168.10.81 

192.168.10.82 


CTDB_PUBLIC_ADDRESSES 

Specifies the location of the file that lists the IP addresses that can be used to access the Samba shares exported by this cluster. These are the IP addresses that you should configure in DNS for the name of the clustered Samba server and are the addresses that CIFS clients will connect to. 


The contents of the /etc/ctdb/public_addresses file on node1 is: 

192.168.10.93/0 eth1 


And node 2: 

192.168.10.93/0 eth0 


CTDB_RECOVERY_LOCK 

Specifies a lock file that CTDB uses internally for recovery. This file must reside on shared storage such that all the cluster nodes have access to it. We use the GFS2 file system that will be mounted at /mnt/ctdb on all nodes. This is different from the GFS2 file system that will host the Samba share that will be exported. This recovery lock file is used to prevent split-brain scenarios. 


CTDB_MANAGES_SAMBA 

When enabling by setting it to yes, specifies that CTDB is allowed to start and stop the Samba service as it deems necessary to provide service migration/failover. 

When CTDB_MANAGES_SAMBA is enabled, you should disable automatic init startup of the smb daemons by executing the following commands: 

<nowiki># chkconfig smb off </nowiki>


CTDB_MANAGES_WINBIND 

When enabling by setting it to yes, specifies that CTDB is allowed to start and stop the winbind daemon as required. 


The smb.conf configuration file must be identical on all of the cluster nodes. We will modify the file when we integrate it with 389-DS.


After starting up the cluster, you must mount the GFS2 file systems that you created. The permissions on the Samba share directory and user accounts on the cluster nodes should be set up for client access. 


== Fencing  ==
We now change cluster.conf manually to define our fencing devices. 

(While luci-ricci can be used, I personally prefer editing the conf file directly due to the random bugs that keep popping up in luci-ricci.) 


The fence that we are using is HP's ILO v3. This version specifically uses 'fence_ipmilan', and NOT 'fence_ilo' as was the case with v2. 

The cluster packages also need to be updated, (at least the files from RHEL 6.2) else it will not work. 


The fence-agents used here was "fence-agents-3.1.5-10.el6.x86_64". 


We need the IP address, username and password for access the ILO interface. 


/etc/cluster.conf is edited as follows: 


<nowiki><fencedevices></nowiki>

<nowiki><fencedevice agent="fence_ipmilan" ipaddr="192.168.10.181" lanplus="1" login="user" name="smb1ilo" passwd="password" power_wait="15"/> </nowiki>

<nowiki><fencedevice agent="fence_ipmilan" ipaddr="192.168.10.182" lanplus="1" login="user" name="smb2ilo" passwd="password" power_wait="15"/> </nowiki>

<nowiki></fencedevices> </nowiki>


Please note that 'lanplus' needs to be enabled, and 'power_wait' has to be at least 15, else the fence will fail. Also ensure that the user has the privileges to manage the machine in the ILO. To check whether the fencing is working, we can run the following command: 


fence_ipmilan -a 192.168.10.181 -l username -p password \

-o status -P -T 15 


rsync the edited file to the standby node as well. 


== Starting the cluster ==
After the cluster configuration file is copied to the standby node, you can start the Red Hat Cluster Suite Daemons. 


Important: In order to start the cluster daemons correctly, you must be 

logged on to each node before proceeding, and to see any errors, you should have two sessions open for each node. You enter a command for one node, then enter the same command for the second. You must enter each command on both nodes, before proceeding to the next command. 


Important: 

Run tail -f /var/log/messages, on each node to watch for any errors. 


To recap, we need two sessions on each node, one running 'tailf /var/log/messages', and the other to run commands. tmux is highly recommended to make all of this easier. 


Log on to each node as root. 


To start the Red Hat Cluster Service on a member, execute the following commands on all of the nodes to start up the cluster. Since we have configured CTDB with CTDB_MANAGES_SAMBA=yes, CTDB will also start up the Samba service on all nodes and export all configured Samba shares. 


<nowiki># service cman start</nowiki>


This is the cluster manager. This starts all the relevant services. 


<nowiki># service ctdb start </nowiki>


This is the ctdb daemon.


It can take a couple of minutes for CTDB to start Samba, export the shares, and stabilize. Executing ctdb status shows the status of CTDB, as in the following example: 


<nowiki># ctdb status </nowiki>


Number of nodes:2 

pnn:0 192.168.10.81 OK (THIS NODE) 

pnn:1 192.168.10.82 OK 

Generation:1101525957 

Size:2 

hash:0 lmaster:0 

hash:1 lmaster:1 

Recovery mode:NORMAL (0) 

Recovery master:1 


When you see that all nodes are "OK", it is safe to move on to use the clustered Samba server. 


== Problem with NMB ==
The NMB daemon fails when ctdb starts. When started manually, it works as expected. This has been found to be a bug, and was prevented by running a script in crontab. 


Execute the following to start it manually:

<nowiki># service nmb start</nowiki>


The script is located in /root/scripts on both the servers and is listed in crontab:

<nowiki>*/3 * * * * /root/scripts/nmbstartup.sh </nowiki>


The script is as follows:


<nowiki>#!/bin/sh </nowiki>

if ! /sbin/service nmb status 

then 

/sbin/service nmb start 

fi 

== Testing the Cluster Set up  ==
To perform a quick test to see if failover works: 


1. Log in to the remote power switch and turn off the active node. 


2. Run tail -f /var/log/messages on the standby node. You will observe the cluster becomes aware of the failed node, I/O fence it, and bring up the failed service on the standby node. 


== Configuration files ==
Node1: 

<nowiki># cat /etc/ctdb/public_addresses </nowiki>

192.168.10.93/0 eth1 


Node2: 

<nowiki># cat /etc/ctdb/public_addresses </nowiki>

192.168.10.93/0 eth0 


Common to both:

<nowiki># cat /etc/ctdb/nodes </nowiki>

192.168.10.81 

192.168.10.82 


<nowiki># grep gfs2 /etc/fstab </nowiki>

/dev/mapper/csmb_vg-csmb_lv /mnt/gfs2 gfs2 defaults 0 0 

/dev/mapper/csmb_vg-ctdb_lv /mnt/ctdb gfs2 defaults 0 0


<nowiki># cat /etc/cluster/cluster.conf </nowiki>

<nowiki><?xml version="1.0"?> </nowiki>

<nowiki><cluster name="csmb" config_version="1"> </nowiki>

<nowiki><clusternodes> </nowiki>

<nowiki><clusternode name="pbifmsntp1.pbifmsiwdms.pjb" nodeid="1"> </nowiki>

<nowiki><fence> </nowiki>

<nowiki><method name="fence-smb1ilo"> </nowiki>

<nowiki><device name="smb1ilo"/> </nowiki>

<nowiki></method> </nowiki>

<nowiki></fence> </nowiki>

<nowiki></clusternode> </nowiki>

<nowiki><clusternode name="pbifmsntp2.pbifmsiwdms.pjb" nodeid="2"> </nowiki>

<nowiki><fence> </nowiki>

<nowiki><method name="fence-smb2ilo"> </nowiki>

<nowiki><device name="smb2ilo"/> </nowiki>

<nowiki></method> </nowiki>


<nowiki></clusternode> </nowiki>

<nowiki></clusternodes> </nowiki>

<nowiki><cman two_node="1" expected_votes="1"/> </nowiki>

<nowiki><fencedevices> </nowiki>

<nowiki><fencedevice agent="fence_ipmilan" ipaddr="192.168.10.181" lanplus="1" login="user" name="smb1ilo" passwd="password" power_wait="15"/> </nowiki>

<nowiki><fencedevice agent="fence_ipmilan" ipaddr="192.168.10.182" lanplus="1" login="user" name="smb2ilo" passwd="password" power_wait="15"/> </nowiki>

<nowiki></fencedevices> </nowiki>

<nowiki></cluster> </nowiki>


<center>End of PART 1 – Samba cluster installation</center>