7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
1/64
Copyright IBM Corporation 2015. Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM.
PowerHA SystemMirror CommonAdministrator Tasks
Michael HerreraPower Systems Strategic Initiatives Team
[email protected] Coppell, TX
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
2/64
2IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Agenda
Deployment Methods
Available methods
Tips & Options
Recovery (snapshots | alt disk install | replication)
Management
Version in use
Start / Stop / Move Operations
Maintenance Tasks
SW Updates
Available Migration Methods
Recommendations
Configuration Optimization
Tunables
Custom Verification Methods
Custom Events
Pager Notification Events
Application Monitoring
CLI Usage
V7 Changes
Monitoring
Troubleshooting
Methods to collect cluster
information
Useful Cluster Logs
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
3/64
3IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Deployment Methods
PowerHA SystemMirror for AIX
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
4/64
4IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
V7 Deployment Methods
There are a number of different ways to achieve the same result:
smitty sysmirror
Initial | Discovery
Custom
IBM Systems Director Plug-In
Available GUI Interface
clmgr cluster copy !cluster cloning from snapshot
Command Line Interface (clmgr)1.
clmgr add cluster repository= nodes=,
2. clmgr add service_ip network=
3.
clmgr add application_controller startscript= stopscript=
4.
clmgr add nodes=, startup=ohn fallback=nfb service_label=
volume_group=application=5. clmgr sync cluster
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
5/64
5IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
PowerHA SystemMirror: Cluster Snapshots
/usr/es/sbin/cluster/snapshots/ .info
.odm
Cluster Configuration
HACMPcluster
...info
HACMPnodeinfo
HACMPadapter
info.
Cluster Report & CLI output
cllsnode
..
cllscf..
cllsif
..
Snapshot C .odm
clusterODM stanzas
Snapshot files:
Snapshot C .info
cluster reportSnapshot B .odm
clusterODM stanzas
Snapshot A .odm
clusterODM stanzas
Snapshot B .info
cluster reportSnapshot A .info
cluster report
" Snapshots are saved off
automatically any time a
Verify / Sync operation is
invoked
" The .info file is not
necessary in order toable to restore the
configuration
" The snapshot menu willask for a and a
as the only
required fields
" The snapshot upgrademigration path requires
the entire cluster to be
down
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
6/64
6IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Cluster Copy Function with clmgr
V713 provides new fast-start mechanism called cluster copying
Good for environments with many, similar clusters
Traditional snapshot capture & restore:
# clmgr add snapshot
# clmgr manage snapshot restore \
[ configure = yes | no ] \[ force = no | yes ]
Copy function on new cluster:
# clmgr manage snapshot restore \
nodes=, \repositories=, [] [: [,]] \
[ cluster_name = ] \
[ configure = yes | no ]
[ force = no | yes
Snapshot must bemanually copied onto
new nodes
Service Labels are notpreserved
Will perform a new
discovery but will not
automatically
synchronize the cluster
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
7/64
7IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Planning for Environment Recoverability
Restore from mksysb | NIM
Cluster snapshots
Rootvg Replication
Cloning a cluster
25GB
25GB
25GB
rootvg
rootvg
Alt disk copy
25GB
Storage
Replication
25GB
StorageReplication
LPAR
LPAR
VSCSI or NPIV
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
8/64
8IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Removing a cluster software
Removal Options:
Delete CAA cluster
# rmcluster fn
Delete cluster from
one or all nodes
Other Considerations:
/etc/cluster/rhosts
/etc/hosts
ECM VG conversion
Application Monitoring
Custom Pre / Post Events
Steps & Considerations:
smitty remove !cluster.*
Hard Set IPs
Auto-varyon of volume groups
Auto-mount of file systems
Automated Application start
The goal is not have to remove the software, but if you have to
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
9/64
9IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Removing a partition
that has a CAA cluster configured
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
10/64
10IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Management
PowerHA SystemMirror for AIX
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
11/64
11IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
What version of code do you have installed?
# lslpp l cluster.es.server.rte
cluster.es.server.rte 7.1.3.3 COMMITTED Base Server Runtime
# halevel s
7.1.3 SP3
# lssrc ls clstrmgrES | grep fix
cluster fix level is 3
# clmgr query version
SystemMirror Information:=========================
Version: 7.1.3 SP3Build Level: 1509B_hacmp713 (3/6/15, 10:59:11)
Cluster Type: Multi Site Cluster Deployment (Linked Cluster)
CAA Information:================
May 16 201417:01:161419D_71N
@(#) _kdb_buildinfo unix_64 May 16 2014 17:01:16 1419D_71NCluster Configured: Yes.
Host Information:
=================HOSTNAME: mhoracle1.dfw.ibm.com
IPADDRESS: 9.19.51.211
LOCALHOST: true
HAVERSION: 7.1.3.3VERSION_NUMBER: 15HAEDITION: ENTERPRISE
AIX_LEVEL: 7100-03-03-1415
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
12/64
12IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
So how do you start up Cluster Services ?
smitty sysmirror !System Management !PowerHA SystemMirror Services !Start / Stop
smitty clstart (FastPath)
clmgr start cluster
-clmgr online node nodeA
- clmgr start node nodeA
IBM Systems Director Plug-In
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
13/64
13IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
PowerHA SM: Cluster Start up Behavior
The cluster manager daemon is now running all of the time
# clshowsrv -v
Status of the RSCT subsystems used by HACMP:
Subsystem Group PID Status
cthags cthags 4980948 active
ctrmc rsct 4063376 active
Status of the HACMP subsystems:
Subsystem Group PID Status
clstrmgrES cluster 4915234 active
clcomd caa 6422738 active
Status of the optional HACMP subsystems:
Subsystem Group PID Status
clinfoES cluster 8847544 active
Settings can be altered within the cluster panels:
# lssrc -ls clstrmgrES | grep stateCurrent state: ST_STABLE
Verify Cluster should be left to true
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
14/64
14IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Automatic Cluster Services Startup Behavior
What is the Best Practice ?
If entry exists thecluster Services are
set to automaticallystart up on boot up
All currently supported
releases perform acluster verification on
start up and will validate
whether the node canenter the cluster
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
15/64
15IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
PowerHA SystemMirror: Cluster Stop Options
What is the purpose of each option ?
For non-disruptive updatesstop services on only one
node at a time to allow forone node to retain the status
of the cluster resources
" You cannotNon-Disruptively upgrade from pre-version 7.X to newer releases
" Avoid Unmanaging cluster from all nodes at the same time
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
16/64
16IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Moving Resources between Nodes
clRGmove g n -m
clmgr move rg node=
IBM Systems Director Plug-In
smitty cl_admin
If multiple RGs are selectedthe operation and
resources will be processedsequentially
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
17/64
17IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Moving Resource Groups with Dependencies
DependantWorkload
Production
Workload
Resource Groups:
Parent / ChildDependency
ODM Stanza:
PowerHA Support:
Parent / Child Dependencies
Location Dependencies (Priority)
Start After | Stop After
CSPOC ll
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
18/64
18IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
CSPOC allows
for a multitude of DARE operations
The Cluster Single Point of Control options facilitate dynamic operations
Follow these panels to
dynamically add or remove
resources from the cluster or
perform resource group
movements between nodes
There are CSPOC specific
logs in the HA cluster that will
provide details in the event of
a problem
# smitty cl_admin
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
19/64
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
20/64
20IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Cluster Report Functions
Prior to version 7 an HTML cluster report was available
Required Online Planning Worksheets (discontinued in V7)
In V7 feature was added in IBM Systems Director plug-in
V713 introduces a native HTML cluster report via clmgr # clmgr view report cluster file= type=html
Only officiallysupported on
IE & Firefox
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
21/64
21IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Sample HTML Report
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
22/64
22IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Maintenance
PowerHA SystemMirror for AIX
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
23/64
23IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Updating your Multipath Driver with CAA cluster
Stopping cluster services does not close the CAA private volume group
root@mhoracle1 /> lspv | grep private
hdisk9 00f626d13aa3645a caavg_private active
root@mhoracle1 /> lsvg -l caavg_private
caavg_private:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
caalv_private1 boot 1 1 1 closed/syncd N/A
caalv_private2 boot 1 1 1 closed/syncd N/A
caalv_private3 boot 4 4 1 open/syncd N/A
powerha_crlv boot 1 1 1 closed/syncd N/A
" New option in Version 713 SP1 to stop CAA with it:
root@mhoracle1 /> clmgr stop cluster STOP_CAA=yes
root@mhoracle1 /> clmgr start cluster START_CAA=yes
Use of CAA option typically not required
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
24/64
24IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Upgrade Considerations
Operating System: Should you do AIX first or HA code?
Should you combine the upgrade
New OS requirements for HA
What is your back-out plan? Alternate disk install
Mksysb
BOS updates will typically require areboot (hence a disruption)
Cluster Software Code:
What type of Migration
Snapshot Migration
Rolling Migration
Non-Disruptive Update
Remove & Reconfigure
Evaluate source to target level
Can you user NDU?
Any new critical requirements?
Any configuration changes required?
There are two main areas that you need to consider OS & HA Software
Change Controls: what is your ability to apply and test the updates ?
Consider things like Interim Fixes locking down the system
Will they need to be reapplied?
Will they need to be rebuilt?
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
25/64
25IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Flow of Non-Disruptive PowerHA Software Upgrade (NDU)
Active Production Environment
-
HA Version 7.1.3
-
UNMANAGE resources
-
Application is still running
-
smit update_all
-
HA Level & Patches
-
Be mindful of new base filesets
-
smit clstart
-
Start scripts will get reinvoked
Node Running at New 7.2.0 version
-
Application still active
Hypothetical Example 2 Node Cluster HA version 7.1.3 to 7.2.0
Starting Point Standby System
- HA Version 7.1.3
-
UNMANAGE resources
-
smit update_all
-
smit clstart
Node Running Version 7.2.0
We advise against stopping thecluster with the UNMANAGE
option on more than one node ata time. Note that it can be done
but there are various factors toconsider
Common Question: How long can the cluster run in a mixed mode ? What operations are supported ?
Important Note: This process flowassumes that the minimum AIX levels for
target release are already in place
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
26/64
26IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Migration Scenario to PowerHA SM Version 7.X
Active Production Environment
-
PowerHA SM Version 6.1(OS & CAA pre-reqs must be there to proceed)
- Populate /etc/cluster/rhosts
- Refresh clcomd
Stop Cluster Services with Takeover
Run clmigcheck utility
Install PowerHA V7.X + fixes (2 steps)
Reboot Server
Start Cluster Services (cluster conversion is run)26
Starting Point Standby System
-
PowerHA SM Version 6.1(OS & CAA pre-reqs must be there to proceed)
Perform pre-migration check (clmigcheck)
- option 1
- checks for unsupported elements
- if valid proceed to option 3
- option 3
- specify repository disk
- define multicast address
- Populate /etc/cluster/rhosts
- Refresh clcomd
Stop Cluster Services
Install PowerHA V7.X + fixes (2 steps)Reboot Server
Reintegrate Node into cluster (migration not complete)
Issue rg_move or contine to run on 2ndNode
Resources & applications are interrupted thenresumed after during this fallover operation
At this stage of the migration the cluster willbe temporarily running in a mixed mode
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
27/64
27IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Migration from V6 to V7 Recommendations
IBM Knowledge Center upgrade recommendations
http://www-01.ibm.com/support/knowledgecenter/SSPHQG_7.1.0/com.ibm.powerha.insgd/ha_install_mig61_plan.htm
V7.1.3 Recommend:
AIX 6.1 TL9 SP4
AIX 7.1 TL3 SP4
APAR Service Pack Level
IV58849 Included in SP4
IV59761 Included in SP4
IV60736 Included in SP4
IV65472 Interim Fix available for download
IV66606 Interim Fix available for download
Ensure thatthese APARs
are installed
Configurations that will not be automatically converted:o
IPAT with HWATo
FDDI, ATM, X.25, token ringo
LVM Split-Site configurations with disks assigned to each site
New clmigcheckutility available for download (additional checks & enhancements)
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
28/64
28IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Additional Tools & References
PowerHA SystemMirror Internal Wiki
https://w3-connections.ibm.com/wikis/home?lang=en-us#!/wiki/hacmp/page/PowerHA%20for%20AIX
Detailed checklist puttogether by HA
development resources
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
29/64
29IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Configuration Optimization
PowerHA SystemMirror for AIX
Node initiated IP traffic
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
30/64
30IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Node initiated IP traffic
Default behavior with first alias is to set service IP first in routing table
Firewall
9.20.20.7 (service IP)
En2 9.20.20.10 (base) En2 9.20.20.20
9.20.20.10 (base)
En2 9.20.20.7 (service IP) En2 9.20.20.20
* Example using first alias behavior
Disabling First Alias behavior (V713)
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
31/64
31IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Disabling First Alias behavior (V713)
This option willprompt you for the
network
The default is for the service IP to beplaced before the base address.
This is ideal for environments with afirewall between the boxes that wantto see node initiated traffic go out of
the service IP and not the baseaddress
# smitty sysmirror !Cluster Applications & Resources !Resources
A t ti C ti V if & S
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
32/64
32IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Automatic Corrections on Verify & Sync
There are Verify & Sync options in thefirst two paths, however, note that they
do not include the Auto-Corrective option.
You need to follow the Custom Cluster
Configuration Path for that.
The custom path will allow to make
corrective actions only if ALL cluster nodesare not running cluster services. By default
it will not perform any corrective actions.
Cluster Custom Verification Methods
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
33/64
33IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Cluster Custom Verification Methods
Cluster Verification is made up of a bunch of data collectors
Checks will return PASSED or FAILED
Will often provide more details than what is reported in the smit.log output
CustomVerification Methods may be defined to run during the Verify / Syncoperations
Note: Automatic verify & sync on node start up does not include any custom verification methods
Adding Custom Verification Methods
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
34/64
34IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Adding Custom Verification Methods
Problem Determination Tools > PowerHA SystemMirror Verification > Configure Custom Verification Method
> Add a Custom Verification Method and press Enter
Output in smit.logand clverify.logfiles:
Currently Loaded Interim Fixes:
NODE mutiny.dfw.ibm.com
PACKAGE INSTALLER LABEL
======================================================== =========== ==========bos.rte.security installp passwdLock
NODE munited.dfw.ibm.com
PACKAGE INSTALLER LABEL
======================================================== =========== ==========
bos.rte.security installp passwdLock
Please Ensure that they are consistent between the nodes!
Custom Verification Methods
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
35/64
35IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Custom Verification Methods
Custom methods should be in a common path between the cluster members
-
ie. /usr/local/hascripts/custom_ver_check.sh
The Methods are stored in the cluster ODM stanzas
Script Logic & Return Codes
How fancy do you want to get#!/bin/kshecho "Currently Loaded Interim Fixes:"
clcmd emgr -Pecho "Please Ensure that they are consistent between the nodes!"
PowerHA SystemMirror Tuning Differences
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
36/64
36IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
PowerHA SystemMirror Tuning Differences
Version 7 clusters & CAA changed DMS behavior
" CAA DMS tunable (deadman_mode) allows two different actions
Assert (crash) the system (default behavior)
Generate AHAFS event
..............
Client Failure Scenario
Repository Disk LUN had beenlocked and had not been
responsive for days. Client wasunaware and standby node had
a problem. Primary system wasbrought down when it was
unable to write to repository disk
Tuning the Failure Detection Rate in V7 X
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
37/64
37IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Tuning the Failure Detection Rate in V7.X
Note that the SMIT menu to alter values was missing prior to HA 7.1.1 SP1
" Checking current settings:
root@mhoracle1 /> clctrl -tune -o node_down_delay
sapdemo71_cluster(07552a84-057b-11e1-b7cb-46a6ba546402).node_down_delay = 10000
root@mhoracle1 /> clctrl -tune -o node_timeout
sapdemo71_cluster(07552a84-057b-11e1-b7cb-46a6ba546402).node_timeout = 20000
" Modifying via command line:
clmgr modify cluster HEARTBEAT_FREQUENCY= 10000 GRACE_PERIOD=5000
*** The settings will take effect only after the next sync
Attributesstored in
HACMPclusterobject class
Defining Pre / Post Events
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
38/64
38IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Defining Pre / Post Events
Pre/Post-Event Commands are NOT the same thing as User Defined Events
A custom Event will neverget invoked unless you
explicitly define it as a Preor Post event command to
an existing Cluster Event
PowerHA SystemMirror: File Collections
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
39/64
39IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
PowerHA SystemMirror: File Collections
Configuration_Files
/etc/hosts
/etc/services /etc/snmpd.conf
/etc/snmpdv3.conf
/etc/rc.net
/etc/inetd.conf
/usr/es/sbin/cluster/netmon.cf
/usr/es/sbin/cluster/etc/clhosts
/usr/es/sbin/cluster/etc/rhosts
/usr/es/sbin/cluster/etc/clinfo.rc
" Introduced in HA 5.2
Ability to automatically push files every 10 min from source node specified Default collections created but not enabled by default
" SystemMirror_Files
Pre, Post & Notification
Start & Stop scripts Scripts specified in monitors
Custom pager text messages
SNA scripts
Scripts for tape support
Custom snapshot methods
User defined events
" Not intended to maintain users & passwords between cluster nodes
File Collections Application script Scenario
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
40/64
40IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
File Collections Application script Scenario
If set to yes fileswill be
propagated every10 minutes
# smitty sysmirror !System Management !File Collections
#!/bin/ksh
Application Start Logic
RED Updates#!/bin/ksh
Application Stop Logic
RED Updates
/usr/local/hascripts/app*
#!/bin/ksh
Application Start Logic
BLUE Logic#!/bin/ksh
Application Stop Logic
Blue Logic
/usr/local/hascripts/app*
Node A Node B
PowerHA SM: User & Group Administration
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
41/64
41IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
p
Can select
Local (files)
LDAP
Select Nodes byResource Group
No selection
means all nodes
# smitty sysmirror !System Management !Security and Users
" Users will be
propagated to all of
the cluster nodes
applicable
" Password
command can be
altered to ensure
consistency across
al nodes
PowerHA SM: User Passwords (clpasswd)
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
42/64
42IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
( p )
Optional List ofUsers whosepasswords willbe propagatedto all clusternodes
passwdcommand isaliased toclpasswd
Functionalityavailable sinceHACMP 5.2
(Fall 2004)
# smitty sysmirror !System Management !Security and Users !Passwords in a PowerHA SystemMirror cluster
Pager Notification Events
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
43/64
43IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
g
As long as sendmail is enabled you can easily receive EVENT notification
smitty sysmirror !Custom Cluster Configuration !Events !Cluster Events !RemoteNotification Methods !Add a Custom Remote Notification Method
Sample Email:
From: root 10/23/2012 Subject: HACMPNode mhoracle1: Event acquire_takeover_addr occurred at Tue Oct 23 16:29:36 2012, object =
Pager Notification Methods
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
44/64
44IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Action Taken: Halt of Node connor
HACMPpager:
methodname = "Herrera_notify"desc = Lab Systems Pager Event"
nodename = "connor kaitlyn"
dialnum = "[email protected]"
filename = "/usr/es/sbin/cluster/samples/pager/sample.txt"
eventname = "acquire_takeover_addr config_too_long event_errornode_down_complete node_up_complete"
retrycnt = 3
timeout = 45
# cat /usr/es/sbin/cluster/samples/pager/sample.txt
Node %n: Event %e occurred at %d, object = %o
Sample Email:
From: root 09/01/2009 Subject: HACMPNode kaitlyn: Event acquire_takeover_addr occurred at Tue Sep 1 16:29:36 2009, object =
* Sendmail must be working and accessible via the firewall to receive notifications
Application Monitoring within Cluster
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
45/64
45IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Some are provided in Smart Assistants
ie. cluster.es.assist.oracle !/usr/es/sbin/cluster/sa/oracle/sbin/DBInstanceMonitor
A Monitor is bound to the Application Controller
Example OracleDB
StartupMonitor
ProcessMonitor
CustomMonitor
Confirm thestartup of the
application
Invokes thecustom logic
Checks theprocess table
NewApplication
Startup Mode
in HA 7.1.1
60 secinterval
60 secinterval
Onlyinvoked onapplication
startup
Long RunningMonitors will continue
run locally with the
running application
Application Monitoringwithin the cluster
configuration is optional
Monitoring can beconfigured to perform
restarts | notify | fallover
If the source LPAR
remains ONLINE andonly the application goes
offline - withoutmonitoring the cluster will
not attempt to relocatethe workload/s
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
46/64
46IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
V7 Command Line Interface
PowerHA SystemMirror for AIX
Monitoring the Cluster Status
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
47/64
47IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
clcmd lssrc ls clstrmgrES | grep state
Current state: ST_STABLE
Current state: ST_STABLE
clmgr v a name,state,raw_state query node
NAME="mhoracle1"
STATE="OFFLINE"
RAW_STATE="ST_INIT"
NAME="mhoracle2"
STATE="OFFLINE"
RAW_STATE="ST_INIT
clRGinfo p
Cluster Name: sapdemo71_cluster
Resource Group Name: SAP_rg
Node Group State
---------------------------- ---------------
mhoracle1 ONLINE
mhoracle2 OFFLINE
Provided by the CAA packages
Part of new CLI interface in V7 clusters
Legacy cluster command (location of resources)
PowerHA V7.1: lscluster command
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
48/64
48IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Flags
-i Lists the cluster configuration interfaces on the local node
-n Allows the cluster name to be queried for all interfaces-s Lists the cluster network statistics on the local node
-m Lists the cluster node configuration information
-d Lists the cluster storage interfaces
-c Lists the cluster configuration
root@mhoracle2 /> lscluster -cCluster Name: MH_Oracle_SVC_EE_cluster
Cluster UUID: 3662ad2a-8d22-11e3-8d70-46a6ba546402Number of nodes in cluster = 3
Cluster ID for node mhoracle1.dfw.ibm.com: 2
Primary IP address for node mhoracle1.dfw.ibm.com: 9.19.51.211Cluster ID for node mhoracle2.dfw.ibm.com: 3
Primary IP address for node mhoracle2.dfw.ibm.com: 9.19.51.212Cluster ID for node mhoracle3.dfw.ibm.com: 4
Primary IP address for node mhoracle3.dfw.ibm.com: 9.19.51.198Number of disks in cluster = 2
Disk = UUID = 497a007c-a3e9-fcc4-d362-2db9002bdfbb cluster_major = 0 cluster_minor = 2
Disk = hdisk9 UUID = d3ce4fd5-3003-ac21-9789-6d9a590242fd cluster_major = 0 cluster_minor = 1Multicastfor site Site2: IPv4 228.19.51.212IPv6 ff05::e413:33d4
Multicastfor site Site1: IPv4 228.19.51.211IPv6 ff05::e413:33d3
Communication Mode: unicastLocal node maximum capabilities: HNAME_CHG, UNICAST, IPV6, SITE
Effective cluster-wide capabilities: HNAME_CHG, UNICAST, IPV6, SITE
Example of 3 node EE clusterusing Sites & SVC replication
Validating SANCOMM Heartbeat Link
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
49/64
49IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
lscluster i
Interface number 2, sfwcomIFNET type = 0 (none)
NDD type = 304 (NDD_SANCOMM)
Smoothed RTT across interface = 7
Mean deviation in network RTT across interface = 3
Probe interval for interface = 100 ms
Interface state = UP
lscluster mInterface State Protocol Status
-----------------------------------------------
dpcom DOWN none RESTRICTED
en0 UP IPv4 none
sfwcom UP none none
lscluster -sstorage pkts sent: 168493709 storage pkts recv: 82575360
New in version 7.1.3
In version 7 clusters improperly
configured storage framework
communication interfaces could
be perceived as being
operational
clras sancomm_status
NAME UUID STATUS
nodeA.dfw.ibm.com | e9b4d6a4-5e71-11-e2-af42-00145ee726e1 | UP |
Monitoring interface traffic
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
50/64
50IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
# lscluster i
Node mhoracle1.dfw.ibm.com
Node UUID = 1dfc2d5a-6ab0-11e2-ace9-46a6ba546402
Number of interfaces discovered = 3Interface number 1, en0IFNET type = 6 (IFT_ETHER)NDD type = 7 (NDD_ISO88023)MAC address length = 6MAC address = 46:A6:BA:54:64:02Smoothed RTT across interface = 7Mean deviation in network RTT across interface = 3Probe interval for interface = 100 msIFNET flags for interface = 0x1E080863NDD flags for interface = 0x0021081BInterface state = UPNumber of regular addresses configured on interface = 2
IPv4 ADDRESS: 9.19.51.211 broadcast 9.19.51.255 netmask 255.255.255.0IPv4 ADDRESS: 9.19.51.239 broadcast 9.19.51.255 netmask 255.255.255.0Number of cluster multicast addresses configured on interface = 1IPv4 MULTICAST ADDRESS: 228.19.51.211
Interface number 2, sfwcomIFNET type = 0 (none)NDD type = 304 (NDD_SANCOMM)Smoothed RTT across interface = 7
Mean deviation in network RTT across interface = 3Probe interval for interface = 100 msIFNET flags for interface = 0x00000000
NDD flags for interface = 0x00000009
Interface state = UPInterface number 3, dpcom
IFNET type = 0 (none)
NDD type = 305 (NDD_PINGCOMM)Smoothed RTT across interface = 750Mean deviation in network RTT across interface = 1500Probe interval for interface = 22500 msIFNET flags for interface = 0x00000000NDD flags for interface = 0x00000009Interface state = UP RESTRICTED AIX_CONTROLLED
# lscluster s
Cluster Network Statistics:
pkts seen: 7441874 passed: 1698499IP pkts: 6026783 UDP pkts: 5768975gossip pkts sent: 1154441 gossip pkts recv: 2304475cluster address pkts: 0 CP pkts: 5743449bad transmits: 0 bad posts: 33
Bad transmit (overflow): 0Bad transmit (host unreachable): 0Bad transmit (net unreachable): 0Bad transmit (network down): 0Bad transmit (no connection): 0short pkts: 0 multicast pkts: 5449950
cluster wide errors: 0 bad pkts: 0dup pkts: 185857 pkt fragments: 5112fragments queued: 0 fragments freed: 0pkts pulled: 0 no memory: 0rxmit requests recv: 292 requests found: 250requests missed: 66 ooo pkts: 40
requests reset sent: 66 reset recv: 48remote tcpsock send: 0 tcpsock recv: 0rxmit requests sent: 332alive pkts sent: 0 alive pkts recv: 0
ahafs pkts sent: 14 ahafs pkts recv: 4nodedown pkts sent: 0 nodedown pkts recv: 0
socket pkts sent: 11673 socket pkts recv: 11711cwide pkts sent: 462885 cwide pkts recv: 463555socket pkts no space: 0 pkts recv notforhere: 0
Pseudo socket pkts sent: 0 Pseudo socket pkts recv: 0Pseudo socket pkts dropped: 0arp pkts sent: 3 arp pkts recv: 1stale pkts recv: 0 other cluster pkts: 2
storage pkts sent: 2814171 storage pkts recv: 2722069disk pkts sent: 3115 disk pkts recv: 3267unicast pkts sent: 203803 unicast pkts recv: 318887out-of-range pkts recv: 0IPv6 pkts sent: 0 IPv6 pkts recv: 0
IPv6 frags sent: 0 IPv6 frags recv: 0Unhandled large pkts: 0
Updates to clras command
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
51/64
51IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Options to monitor the health of the heartbeat networks
Cluster Repository disk heartbeating:
SANCOMM heartbeating:
Notice that this
output doesntshow both nodes
Dump Repository Disk Information:
# clras dumprepos
Dump Repository Disk information
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
52/64
52IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
The cluster configuration is stored in ODM stanzas and in the repository disk!
# clras dumprepos
New Command Line Interface clmgr
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
53/64
53IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Several useful references:
Rapid Deployment Guide
IBM Systems Magazine article
IBM Redbooks
Documentation
MAN page
An much more robust extension of the original clvt command
The command is the back-end to the IBM Systems Director plug-in
Uses:
CLI Cluster Deployment
Cluster snapshots | Cluster Copy functions
Cluster Reporting
Start | Stop | Move resource operations
Dynamically adding / removing resources
Detailed MAN page:
CAA clcmd command
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
54/64
54IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Introduced in bos.cluster.rte
I typically probeclstrmgrES to get
the cluster state
Think of the many uses:
Identify hdisk#s for differentPVIDs or volume groups
Compare file set levels betweenall of the cluster members
Check scriptlocations and
permissions
Leveraging new V7 Command Line Interface
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
55/64
55IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
The clmgr command can be used for many different operations:
Dynamic LVM changes (sample syntax) Description
clmgr modify vg add= Add physical volume to VG
clmgr add lv type=jfs2 vg logical_partitions= Add new logical volume
clmgr add fs type= lv
lv_for_log=inline inline_log_size=
Add file system to existing
logical volume
Cluster Operations (sample syntax) Description
clmgr move rg node= Relocate RG
clmgr start | stop cluster Start / Stop services cluster wide
clmgr online | offline node Start / Stop on individual node
clmgr sync cluster Verify & synchronize clusterclmgr replace repository Dynamically replace CAA repo disk
clmgr query cluster Query cluster attributes / status
Creating & Managing Application Monitors
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
56/64
56IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Earlier releases did NOThave a command to suspend / resume monitors:
- clmgr manage application_controller suspend test_app1
- clmgr resume application_controller resume test_app1
"
Active application monitors will have a corresponding clapp_mon process
" Creation can be performed via SMIT, Director or CLI:
clmgr add mon appA_mon TYPE=Custom APPLICATION=appA MONITORINTERVAL=60FAILUREACTION=fallover STABILIZATION=300 RESTARTINTERVAL=1200
CLEANUPMETHOD=/usr/local/hascripts/appA_cleanup.sh RESTARTMETHOD=/usr/local/
hascripts/appA_restart.sh RESTARTCOUNT=3 MONITORMETHOD=/usr/local/hascripts/appA_monitor.sh
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
57/64
57IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Troubleshooting
PowerHA SystemMirror for AIX
Capturing Cluster Information
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
58/64
58IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Quick Troubleshooting:
Topology & Resource configuration
(netstat in | ifconfig a)
Individual Logs
(Error report | hacmp.out file | cluster.log)
Cluster snapshot
# clmgr add snapshot
Data Detailed Analysis:
PowerHA Information !snap e
/tmp/ibmsupt/snap.pax.Z
RSCT logs
/usr/sbin/rsct/bin/phoenix.snap !collects the RSCT information & logs
Includes everythingincluding RSCT logs,
PowerHA config/log
Consider using cllsif | clshowres orvariations using the clmgr command
Review the .info report file & the .odmcluster configuration stanzas
Troubleshooting Cluster Problems
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
59/64
59IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Most widely used logs:
Log Files Description
/var/hacmp/adm/cluster.log High level cluster EVENTs
/var/hacmp/adm/hacmp.out Detailed EVENT processing
/var/hacmp/log/clstrmgr.debug Detailed information from clstrmgrES daemon
/var/hacmp/log/clutils.log Generated by cluster utilities
/var/adm/ras/syslog.caa CAA logging & Troubleshooting
Log Files Description
/var/hacmp/clverify/clverify.log Detailed verification check output
/var/hacmp/clcomd/clcomd.log Troubleshooting communication issues
/var/hacmp/log/cspoc.log.long Detailed information from CSPOC
/var/hacmp/log/autoverify.log Generated by Nightly Verification
/var/hamp/adm/history/cluster.mmddyyyy Cluster history files generated daily
Troubleshooting specific areas:
Automatic Nightly Cluster Verification
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
60/64
60IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
By Default the cluster will run a nightly Verification check at midnight
"
The clutils.log file should show the results of the nightly check
On V7 clusters check /var/hacmp/log/autoverify.log
Summary
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
61/64
61IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
Goal of the session:
-
Highlight useful information for new & experienced users- Cover scenarios an Administrator may run into
V6.1 is now officially out of Support
- Review migration options and attend appropriate sessions
There are a number of useful features for HA Administrators
- File collections
- Pager Notification
- Custom Events
- Custom Verification Checks
- Application Monitoring Framework
V7 Clusters are significantly different:
- Deployment Methods
- Command Line Interface
- Monitoring
Please fill out an evaluation!
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
62/64
62IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposia
materials may not be reproduced in whole or in part without the prior
written permission of IBM.
!"#$ &'$() *'+,$-)" .$ /"01
@ IBMtechU
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
63/64
63IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposiamaterials may not be reproduced in whole or in part without the prior
written permission of IBM.63
Questions?
Thank you for your time!
Continue growing your IBM skills
7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287
64/64
64IBM Systems Technical University, October 5-9 | Hilton Orlando
Copyright IBM Corporation 2015. Technical University/Symposiamaterials may not be reproduced in whole or in part without the prior
written permission of IBM.
ibm.com/trainingprovides acomprehensive portfolio of skills and careeraccelerators that are designed to meet allyour training needs.
Training in cities local to you- where andwhen you need it, and in the format you want
Use IBM Training Searchto locate public training classes
near to you with our five Global Training Providers
Private training is also available with our Global TrainingProviders
Demanding a high standard of quality view the paths to success
Browse Training Pathsand Certificationsto find thecourse that is right for you
If you cant find the training that is right for youwith our Global Training Providers, we can help.
Contact IBM Training at [email protected]
64
Global Skills Initiative