Initial GPFS Cluster Setup This section will provide information on setting up an initial GPFS cluster on an AIX system Installation Setting up SSH Create Cluster Verification Steps: Create the GPFS cluster using mmcrcluster. Before the cluster can be built, GPFS has to be installed on the required nodes. Then enable password less login enabled from each of the nodes to other nodes. Step 1 Installing GPFS fileset is fairly easy for an AIX administrator by using the smitty installp tool, select the fileset or use the installp command depending on your choice. The filesets required are gpfs.base gpfs.msg.en_US gpfs.base gpfs.docs.data Step 2 Next step is to enable the ssh password less login rsh or ssh can be used. Even though rsh can also be used, ssh is the preferred method as it offers more security. On the one of the nodes (preferably the node which you plan as the primary node) generate ssh key and copy the private and the public key to /root/.ssh directory on all the nodes which are part of GPFS cluster. Generate keypair #ssh-keygen -t dsa Verify if key-pair is generated There should be 2 files generated on /root/.ssh: id_dsa and id_dsa.pub Modify parameter to YES Look for PermitRootLogin parameter in /etc/ssh/sshd_config file Before copying the ssh keys to all other nodes make sure that the parameter PermitRootLogin is set to yes on all nodes. If the parameter is not set to yes then change it to 'yes' Then refresh the ssh daemon.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Initial GPFS Cluster Setup
This section will provide information on setting up an initial GPFS cluster on an AIX system
InstallationSetting up SSHCreate ClusterVerificationSteps: Create the GPFS cluster using mmcrcluster.
Before the cluster can be built, GPFS has to be installed on the required nodes. Then enable password less login enabled from each of thenodes to other nodes.
Step 1
Installing GPFS fileset is fairly easy for an AIX administrator by usingthe smitty installp tool, select the fileset or use the installp commanddepending on your choice.
Next step is to enable the ssh password less login
rsh or ssh can be used. Even though rsh can also be used, ssh is the preferred method as it offers more security.
On the one of the nodes (preferably the node which you plan as the primary node) generate ssh key and copy the private and the public key to /root/.ssh directory on all the nodes which are part of GPFS cluster.
Generate keypair
#ssh-keygen -t dsa
Verify if key-pair is generated
There should be 2 files generated on /root/.ssh: id_dsa and id_dsa.pub
Modify parameter to YES
Look for PermitRootLogin parameter in /etc/ssh/sshd_config file
Before copying the ssh keys to all other nodes make sure that the parameter PermitRootLogin is set to yes on all nodes.
If the parameter is not set to yes then change it to 'yes'
Then refresh the ssh daemon.
Copy the key-pair id_dsa and id_dsa.pub to all other nodes to the same location /root/.ssh
Append the ssh public key to /etc/ssh/authorized_keys on all the nodes including the primary gpfs node .
Create a nodelist in root home directory which has all the node names (FQDN) which will be part of gpfs cluster
For eg:
If the nodenames are node1.test.com (node1) , node2.test.com (node2) , etc
Create a file /root/nodelist
Add the FQDN’s or shortnames of these nodes in the file one after the other (remember to put these entries in /etc/hosts file as well)
/root/nodelist is input file with a list of node names:designations.
Designations are “manager or client” and “quorum or nonquorum” (Tip: To make a node as quorum node specify quorum alone , to make it a client quorum node specify as quorum-client)
where /root/nodelist contains the the list of nodes, node1 the primary configuration serve which is specified by –p optionr, node2 the secondary configuration server which is specified by –s option, ssh is the shell used for GPFS command execution specified by –R option and scpis the copy command used by GPFS to copy in between nodes specified by –r option and testcluster is the cluster name specified by –C option and finally –A option denotes GPFS will auto start during reboot.
Step 4
Verify status of the cluster using the mmlscluster
# mmlscluster
GPFS cluster information
GPFS cluster name: testcluster
GPFS cluster id: 12399838388936568191
GPFS UID domain: testcluster
Remote shell command: /usr/bin/ssh
Remote file copy command: /usr/bin/scp
GPFS cluster configuration servers:
Primary server : node1
Secondary server: node2
Node Daemon node name IP address Admin node name Designation
This means the hdisk1 is the LUN which is to be used for the NSD, node1 is the primary server for this NSD, node2 is the backup server for this NSD, dataAndMetadata indicates it can contain data as well as metadata. 0 is the failure group and test_nsd is the name of the NSD
After the descriptor file is created use the mmcrnsd command to creat the NSD
Usage: mmcrnsd -F DescFile [-v {yes | no}]
Eg:
#mmcrnsd -F /root/disklist
mmcrnsd: Processing disk hdisk1
mmcrnsd: 6027-1371 Propagating the cluster configuration data to all affected nodes. This is an asynchronous process.
Verify the NSD’s were created
Use mmlsnsd to verify the NSD’s were properly created
After the NSD has been created the "disk descriptor" file /root/disklistin our case will have been rewritten and now has the NSD disk names. This newly written "disk
descriptor" file now is used as input to the mmcrfs command.
# cat disks
# hdisk1:node1:node2:dataAndMetadata:0:test_nsd
test_nsd:::dataAndMetadata:0::
Creating the GPFS Filesystem
Before creating the gpfs filesystem make sure to create the mount point.
For example if /gpfsFS1 is the filesystem name to be used then create the mount point using
Command mkdir –p /gpfsFS1.
Create the gpfs filesystem using the below command.
This will create a Filesystem gpfsFS1 with device /dev/gpfsFS1 (whose underlying raw device will the disk mentioned in /root/disklist) with block size of 64K and maximum copies if data (m) and metadata (M) set to2. Incase if you add an extra NSD like above and mention the failure group as 1, you can also specify the –r 2 and –R 2 which implies that 2 replicas of data and metadata will be created (like mirroring).
GPFS startup , shutdown ,Status and add a new node
This section will explain about GPFS operations as below
To Check Status of GPFS ClusterTo Startup GPFS ClusterTo Shutdown GPFS ClusterAdd a new node to a running GPFS ClusterTo change designation of the running GPFS clusterSteps: To Check Status of the GPFS Cluster
# mmgetstate –aLs àTo get the status of GPFS
Node number Node name Quorum Nodes up Total nodes GPFS state Remarks
Wed Dec 3 00:11:44 CDT 2010: 6027-1341 mmshutdown: Starting force unmount of GPFS file systems
Wed Dec 3 00:11:49 CDT 2010: 6027-1344 mmshutdown: Shutting down GPFS daemons
node1: Shutting down!
node1: 'shutdown' command about to kill process 5701702
Wed Dec 3 00:11:56 CDT 2010: 6027-1345 mmshutdown: Finished
To shutdown all the nodes together
# mmshutdown –N all
To check status of the nodes
# mmgetstate -a
Node number Node name GPFS state
------------------------------------------
1 node1 arbitrating
2 node2 down
This shows that node2 is either down or GPFS is not started and hence the node1 is arbitrating to find the quorum. Solution is to start GPFS in node2 which will be described below
This Section will explain more in detail of the following actions with examples.
To List Characteristics of the GPFS filesystemTo add a tiebreaker diskTo mount all the GPFS filesystemsTo list all the physical disks which are part of a GPFS filesystemTo display the GPFS filesystemTo Unmount a GPFS filesystem from one nodeTo remove a GPFS filesystemTo remove a disk from the filesystem
To remove the NSDTo replace the diskTo add a new disk to the GPFS filesystemTo suspend a diskTo Resume the diskSteps:
Remember that you should remove only a disk after confirming that
adequate space is left in the other disks which are part of this filesystem (you can check this by using mmdf <GPFS filesystem name> so that when the disk is removed it will respan and data will then be shared across other available disks.OR
2 data replicas are available which can be checked using the mmlsfs <GPFS filesysem>
Syntax: mmdeldisk <GPFS filesystem> <NSD name> -r
-r option is very important as it will resync the data and balnce the data across other available disks in this filesystem.
# mmdeldisk gpfs_fs1 test_nsd2 -r
Deleting disks ...
Scanning system storage pool
GPFS: 6027-589 Scanning file system metadata, phase 1 ...
GPFS: 6027-552 Scan completed successfully.
GPFS: 6027-589 Scanning file system metadata, phase 2 ...
GPFS: 6027-552 Scan completed successfully.
GPFS: 6027-589 Scanning file system metadata, phase 3 ...
GPFS: 6027-552 Scan completed successfully.
GPFS: 6027-589 Scanning file system metadata, phase 4 ...
GPFS: 6027-552 Scan completed successfully.
GPFS: 6027-565 Scanning user file metadata ...
GPFS: 6027-552 Scan completed successfully.
Checking Allocation Map for storage pool 'system'
GPFS: 6027-370 tsdeldisk64 completed.
mmdeldisk: 6027-1371 Propagating the cluster configuration data to all
Prerequisites for replacing or adding a new disk. The physical disk / LUN should be added to a NSD as mentioned in the “Creating Network shared Disk (NSD)” section
Syntax : mmrpldisk <GPFS filesystem name> <NSD to be replaced> <new NSD>-v {yes|no}
Yes – checks will be done if any data is there in new NSD
No -- no checks will be done if any data is there in new NSD
In this example you have 3 existing nsd’s nsd2, nsd3 and nsd4 and a newly added nsd nsd1 which is not part of GPFS filesystem. This procedure explains how to replace nsd4 with nsd1
Prerequisites for adding a new disk. The physical disk / LUN should be added to a NSD as mentioned in the “Creating Network shared Disk (NSD)” section and the same disk descriptor file used for creating the NSD has to be used with –F option
Step 1: create a disk descriptor file with any name. here we use the file name as “disks” and the disk used is hdisk10
# cat disks
hdisk10:::dataAndMetadata:0:test_nsd10
Step 2: Create the NSD with the new disk hdisk10
# /usr/lpp/mmfs/bin/mmcrnsd -F disks
mmcrnsd: Processing disk hdisk10
mmcrnsd: 6027-1371 Propagating the cluster configuration data to all