HA17

© 2010 IBM Corporation

IBM Power Systems Technical University

October 18–22, 2010 — Las Vegas, NV

Session Title:Designing a PowerHA SystemMirror for AIX High Availability Solution

Speaker Name: Michael Herrera

Session ID: HA17(AIX)

Michael Herrera ([email protected])

Advanced Technical Skills (ATS)

Certified IT Specialist

Workload-Optimizing Systems

+

Best Practices for Designing a PowerHA

SystemMirror for AIX High Availability Solution

3

Agenda

• Common Misconceptions & Mistakes

• Infrastructure Considerations

• Differences in 7.1

• Virtualization & PowerHA SystemMirror

• Licensing Scenarios

• Cluster Management & Testing

• Summary

4

HACMP is now PowerHA SystemMirror for AIX!

• Current Release: 7.1.0.X– Available on: AIX 6.1 TL06 & 7.1

• Packaging Changes:– Standard Edition - Local Availability– Enterprise Edition - Local & Disaster Recovery

• Licensing Changes:– Small, Medium, Large Server Class

N/AOct 20, 2009PowerHA SystemMirror 6.1.0

N/ANov 14, 2008PowerHA 5.5.0

Sept, 2011Nov 6, 2007HACMP 5.4.1

N/ASept 10, 2010PowerHA SystemMirror 7.1.0

End of Support DateRelease DateVersion

Product Lifecycle:

HA & DR solutions from IBM for your mission-critical AIX applications

* These dates are subject to change per Announcement Flash

5

PowerHA SystemMirror Minimum Requirements

7.1.0.1 – Sep

6.1.0.2 – May 21

5.5.0.6 – June 7

5.4.1.8 – May 13

PowerHA SystemMirror 7.1• AIX 7.1 • AIX 6.1 TL 6 - SP1

PowerHA SystemMirror 6.1• AIX 7.1• AIX 6.1 TL 2 with RSCT 2.5.4.0• AIX 5.3 TL 9, with RSCT 2.4.12.0

PowerHA Version 5.5• AIX 7.1• AIX 6.1 TL2, SP1 and APAR IZ31208, RSCT 2.5.2.0(Async GLVM - APARs IZ31205 and IZ31207)• AIX 5L 5.3 TL 9 with RSCT version 2.4.10.0

HACMP 5.4.1• AIX 6.1 with RSCT version 2.5.0.0 or higher• AIX 5.3 TL4 with RSCT version 2.4.5 (IY84920) or higher• AIX 5.2 TL8 with RSCT version 2.3.9 (IY84921) or higher

6

Common Misconceptions

• PowerHA SystemMirror is an out of the box solution– Scripting & Testing of application Start / Stop scripts– Application monitors will also require scripting & testing

• PowerHA SystemMirror is installed we are completely protected– Consider all single points of failure – ie. SAN, LAN, I/O drawers, etc..

• Heartbeats go over a dedicated link– All interfaces defined to the cluster will pass heartbeats (IP & Non-IP)– CAA definitely changes this behavior

• With clustering I need two of everything – hence idle resources

Fact:

Clustering will highlight what you are & are NOT doing right in your environment

7

Common Mistakes Beyond Base Cluster Functionality

• Not knowing Expected fallover behaviors

• Lack of application monitoring

• Not knowing what to monitor or check– CLI

– Logs

• Not propagating changes appropriately

• No change history

• Down / Missing Serial Networks• EtherChannel Links Down• ERRORs in Verification• Inconsistent AIX levels• Down level cluster filesets• Fallback Policy not set to desired behavior• Missing Filesets• Missing Custom Disk Methods• SAN not built in a robust fashion• Bootlist Issues• Dump Devices

– Insufficient size– Mirrored– Lack Secondary

• I/O Pacing Enabled (old values)• HBA Levels at GA code• Fiber Channel Tunable settings not enabled• Interim Fixes not loaded on all cluster nodes

Lack of Education / Experience

Solutions:

• IBM Training / Redbooks / Proof of Concepts / ATS Health-check Reviews

Poor Change Controls

8

Identify & Eliminate Points of Failure

• LAN Infrastructure

– Redundant Switches

• SAN Infrastructure

– Redundant Fabric

• Application Availability

– Application Monitoring

– Availability Reports

9

Infrastructure Considerations

Important:Identify & Eliminate Single Points of Failure!

Site A Site B

Node A Node B

DWDM DWDM

SITEAMETROVG

50GB

50GB

50GB

50GB

50GB

50GB

50GB

50GB

LAN

SANSAN

LAN

All links through one pipe

10


Important:Identify Single Points of Failure & design the solution around them

Site A Site B

Node A

WAN

Node B

XD_rs232

XD_IP

net_ether_0

DWDM DWDM

SITEAMETROVG

1GBECM VG: diskhb_vg1

hdisk2 000fe4111f25a1d1

ECM VG: diskhb_vg1

hdisk3 000fe4111f25a1d1

1GBECM VG: diskhb_vg2

hdisk3 000fe4112f998235

ECM VG: diskhb_vg2

hdisk4 000fe4112f998235

LAN

SAN

LAN

SAN

50GB

50GB

50GB

50GB

50GB

50GB

50GB

50GB

11


• Power Redundancy

• I/O Drawers

• SCSI Backplane

• SAN HBAs

• Virtualized Environments

• Application Fallover Protection

I/O drawer

I/O drawer

I/O drawer

I/O drawer

I/O drawer

1

2

3

4

5

6

7

8

9

10

Ie 1. Two nodes sharing I/O drawer

Real Customer Scenarios:

Ie 2. Application Failure with no monitoring – box remains up : no cluster fallover

Moral of the Story:

* High Availability goes beyond just installing the cluster software

12

PowerHA SystemMirror 7.1: Topology managementHeartbeating differences between earlier cluster releases

LPAR 1

LPAR 3

LPAR 2LPAR 4

diskhb_net4 diskhb_net3

diskhb_net1diskhb_net2

RSCT Subnet

Heartbeat Rings

PowerHA SM 6.1 & Earlier

RSCT Based Heartbeating• Leader, Successor, Mayor, etc..• Strict Subnet Rules• No Heartbeating over HBAs

Multiple Disk Heartbeat Networks• Point to Point only• Each Network requires LUN with ECM VG

LPAR 1

LPAR 3

LPAR 2LPAR 4

Multicasting

PowerHA SM 7.1 with CAA

Kernel Based Cluster Message Handling• Multi cast based protocol• Use Network & SAN as needed• Discover and use as many adapters as possible

• All monitors are implemented at low levels of the AIX Kernel & are largely insensitive to system load

Single Repository Disk• Used to heartbeat & store information

13

Transition of PowerHA Topology IP Networks

en1

(persistent IP) 9.19.51.11

( base address) 192.168.100.2en0

9.19.51.20 (service IP)

9.19.51.10 (persistent IP)

192.168.100.1 (base address) en0

en1 ( base address) 192.168.101.2

Traditional heartbeating rules no longer apply. However, route stripping is still a potential issue. When two interfaces have routable IPs on the same subnet AIX will send half the traffic out of either interface

9.19.51.21 (service IP)

192.168.101.1 (base address)

VLAN

Network: Net_ether0

Methods to circumvent this:

• Link Aggregation / EtherChannel• Virtualized Interfaces with dual VIO servers


9.19.51.21 (service IP)

9.19.51.20 (service IP)


VLAN

ent0 ent1 ent0 ent1

HB Rings

In 6.1 & below

14

PowerHA SM 7.1: Additional Heartbeating Differences

Serial Networks removed:• No more rs232 support• No more traditional disk heartbeating over ECM VG• No more slow takeover w/disk heartbeat device as last device on selective takeover

Critical Volume Groups• Replace Multi-node Disk Heartbeating

– Oracle RAC three disk volume group - Voting Files– Unlike MNDHB, no more general use– Migration is a manual operation, and customer responsibility– Any Concurrent Access Volume Group can be marked as “Critical”


9.19.51.21 (service IP)

9.19.51.20 (service IP)


VLAN

ent0 ent1 ent0 ent1

Heartbeating:• Self Tuning Failure Detection Rate (FDR)• All interfaces are used even if not in cluster networks

en3 en3

15

CAA – Cluster Aware AIX

What is it:

• A set of services/tools embedded in AIX to help manage a cluster of AIX nodes and/or help run cluster software on AIX

• IBM cluster products (including RSCT, PowerHA, and the VIOS) will use and/or call CAA services/tools

• CAA services can assist in the management and monitoring of an arbitrary set of nodes and/or running a third-party cluster

• CAA does not form a cluster by itself. It is a tool set.

• There is no notion of quorum (If 20 nodes of a 21 node cluster are down, CAA still runs on the remaining node)

• CAA does not eject nodes from a cluster. CAA provides tools to fence a node but never fences a node and will continue to run on a fenced node

Major Benefits:

• Enhanced Health Management (Integrated Health Monitoring)

• Cluster Wide Device Naming

Enabling tighter Integration with PowerHA SystemMirror

16

Cluster Aware AIX Exploiters

Legacy AIX

PowerHA

System MirrorTSA HMC

IBM

StorageHPC

DB2IBM

Director

Monitoring

API

Cluster

Monitoring

Group Services

Cluster Admin

UI

Cluster CFG

Repository

Resource Mgr Services

Bundled Resource Managers

Cluster

Messaging

Messaging

API

Legacy RSCT

RSCT Consumers

VIOS

Monitoring

API

Cluster

Monitoring

Group Services

Cluster Admin

UI

Cluster CFG

Repository

Resource Mgr Services

Bundled Resource Managers

Cluster

Messaging

Messaging

API

RSCT With Cluster Aware AIX

Cluster Aware AIX

Cluster

Repository

Cluster

Messaging

Cluster

Monitoring

Cluster

Events

CAA APIs and UIs

Redesigned Layers Integrated to CAA Capabilities

• RSCT and Cluster Aware AIX together provide the foundation of strategic Power Systems SW

• RSCT-CAA integration enables compatibility with a diverse set of dependent IBM products

• RSCT integration with CAA extends simplified cluster management along with optimized and robust cluster monitoring, failure detection, and recovery to RSCT exploiters on Power / AIX

17

Cluster Aware AIX: Central Repository Disk

Aids in:

• Global Device Naming• Inter node synchronization• Centrally managed configuration• Heartbeating device

Host 1 Host 2 Host 3

HA ODM HA ODM HA ODM

Cluster Synchronization

PowerHA SystemMirror 6.1 & Prior:

Host 1 Host 2 Host 3

Central

Repository

PowerHA SystemMirror will continue to

run if Central Repository Disk goes away

However, no changes may take place

within the cluster.

PowerHA SystemMirror 7.1 & CAA:

Contrast from previous releases

Direction:

• In the first release, support is confined to shared storage

• Will eventually evolve into a general AIX device rename interface

• Future direction is to enable cluster-wide storage policy settings

• PowerHA ODM will eventually also entirely move to the repository disk

18

Multi Channel Health Management – Out of the Box

Network

SAN

Repository Disk

Heartbeats HeartbeatsReliable

Messaging

Reliable

Messaging

First Line of Defense

Second Line of Defense

Third Line of Defense

Highlights:

• RSCT Topology services no longer used for cluster Heartbeating• All customers now have multiple communication paths by default

Hardened Environments with new communication protocol

Faster detection & more efficient communication

LPAR 1 LPAR 2

19

Basic Cluster vs. Advanced Cluster Features

Basic Cluster

– Network Topology – Resource Group/s

• IPs• VGs• Application Server

– Application Monitoring– Pager Events

Advanced Cluster

– Multiple Networks• Crossover Connections• Virtualized Resources

– Multiple Resource Groups• Mutual Takeover• Custom Resource Groups

– Adaptive Fallover

– NFS Cross-Mounts– File Collections– Dependencies

• Parent / Child• Location• Start After• Stop After

IP

VGs

App Server

IP Network

SAN Network

Resource Group

IP

VGs

App1

IP Network

SAN Network

Resource Group

IP Network

IP

VGs

App2

Resource Group

– Smart Assistants– Multiple Sites– Cross Site LVM Configs– Storage Replication– IP Replication– Application Monitoring– Pager Events– DLPAR Integration

• Grow LPAR on Fallover

– Director Management– WebSMIT Management– Dynamic Node Priority

IP

VGs

Dev App

Resource Group

Site A Site B

Disk Replication

20

One to one One to any

Any to anyAny to one

PowerHA SystemMirror: Fallover PossibilitiesCluster Scalable to 32 nodes

21

Methods to Circumvent Unused Resources

Frame 1

hdisk2

hdisk4

rootvg {

oracle_vg1 {

hdisk1

hdisk4

}rootvg

}oracle_vg1

SAN SAN

VIO

HBA

VIO

Node B

Frame 2

Node A

VIOHBA

VIO

NIC

NIC

NIC

NIC

HBA

HBA

Storage Subsystem

Resource Group A

Node A, Node BShared IPVG/s & filesystemsApp 1

Resource Group B

Node A, Node BShared IPVG/s & filesystemsApp 2

Resource Group C

Node B, Node AShared IPVG/s & filesystemsApp 3

Resource Group D

Node B, Node AShared IPVG/s & filesystemsApp 4

RGDependency

RGDependency

Mutual

Takeover

Virtualization

22

Power Virtualization & PowerHA SystemMirror

• LPAR / DLPAR

• Micropartitioning & Shared Processor Pools

• Virtual I/O Server

– Virtual Ethernet

– Virtual SCSI

– Virtual Fiber

• Live Partition Mobility

• Active Memory Sharing

• WPAR (AIX 6.1)

LAN

SAN

LPAR X

AIXRootvg

Data

vfc1vfc0en0

LPAR Y

VIO1 A VIO2 A

Power HA_node 1

AIXRootvg

Data

vfc1vfc0en0

LPAR Z

VIO1 B VIO2 B

Power HA_node 2

Rootvgvolumes

Data

Power HA Cluster

External

Storage Enclosure

23

PowerHA SystemMirror Virtualization Considerations

• Ethernet Virtualization– Topology should look the same as environment using link aggregation– Version 7.1 no longer uses netmon.cf file– As a best practice dual VIO Servers are recommended

• SEA Fallover Backend

• Storage Virtualization– Both methods of virtualizing storage are supported

• VSCSI vs. Virtual fiber (NPIV)

– In DR implementations leveraging disk replication consider the implications of using either option

• Benefits of virtualization:– Maximize utilization of resources– Less PCI slots & physical adapters– Foundation for advanced functions like Live Partition Mobility– Migrations to newer Power Hardware are simplified

* Live Partition Mobility & PowerHA SM compliment each otherMaintenance vs. High Availability(non-reactive . reactive)

Chapter 2.4 PowerVM Virtualization Considerations

24

Frame 1

Virtual I/O Server (VIOS2)

Virtual Ethernet & PowerHA SystemMirror

Hypervisor

PowerHA LPAR 1

en0

ent0(virt)


ent4(SEA)

ent2(virt)

PowerHA LPAR 2

ent0(virt)

en0

ent0(phy)

Ethernet Switch

ent0(phy)

ent4(SEA)

ent2(virt)

Ethernet Switch

ent6(virt)

ent5(virt)

Control

Channel

Control

Channel

ent5(virt)

en6

ent6(virt)

en6

PVID 99

PVID 10

This is a diagram of the configuration required for SEA fallover across VIO Servers. Note that Ethernet traffic will not be load balanced across the VIO Servers. The lower trunk priority on the “ent2” virtual adapter would designate the primary VIO Server to use.

No Link Aggregation / Same Frame

25


Virtual Ethernet & PowerHA SystemMirror

Hypervisor

PowerHA LPAR 1

en0

ent0(virt)


ent0(phy)

ent4(SEA)

ent2(virt)

ent1(phy)

Ethernet Switch

ent3(LA)

ent0(phy)

ent4(SEA)

ent2(virt)

ent1(phy)

ent3(LA)


Hypervisor

PowerHA LPAR 2

en0

ent0(virt)


ent0(phy)

ent4(SEA)

ent2(virt)

ent1(phy)

ent3(LA)

ent0(phy)

ent4(SEA)

ent2(virt)

ent1(phy)

ent3(LA)

Ethernet Switch

Frame1

Frame2

ent5(virt)

ent5(virt)

ent5(virt)

ent5(virt)

Control

Channel

Control

Channel

Control

ChannelControl

Channel

Independent Frames & Link Aggregation

26

PowerHA Node 1

PowerHA SystemMirror 6.1 & Below

PowerHA Node 2

en0

FRAME 2

net_ether_0

9.19.51.10

( base address)

9.19.51.11

( base address)

FRAME 1

en0

9.19.51.20 (service IP) (service IP) 9.19.51.21

Topsvcs heartbeating

serial_net_0


Hypervisor

AIX Client LPAR

en0

ent0(virt)


ent0(phy)

ent4(SEA)

ent2(virt)

ent1(phy)

ent3(LA)

ent0(phy)

ent4(SEA)

ent2(virt)

ent1(phy)

ent3(LA)

FRAME X

ent5(virt)

ent5(virt)

Control

ChannelControl

Channel

* Netmon.cf file used for single adapter networks

27

PowerHA SystemMirror 7.1 - Topology

All nodes are monitored:

Cluster Aware AIX tells you what nodes are in the cluster and information on those nodes including state. A special “gossip”protocol is used over the multicast address to determine node information and implement scalable reliable multicast. No traditional heartbeat mechanism is employed. Gossip packets travel over all interfaces including storage.

Differences:

• RSCT Topology services is no longer used for heartbeat monitoring• Subnet Requirements no longer need to be followed• Netmon.cf file is no longer required or used• All interfaces are used for monitoring even if they are not in an HA network(this may be tunable in a future release)IGMP Snooping must be enabled on the switches

28

LUNSVSCSI

VSCSI Mapping vs. NPIV (virtual fiber)

FRAME 1

Hypervisor

FRAME 2

Node 1

VIOS 2

VIOS 1

LUNS

VIOS 2

VIOS 1

hdisk3

}npiv_vg

STORAGE

SUBSYSTEM

MPIO

NPIV HBA

NPIVHBA

fcs0

fcs1

Hypervisor

Node 2

MPIO

NPIVHBA

NPIVHBA

fcs0

fcs1

vhost0

vhost0

vhost0

vhost0

hdisk4

hdisk3

}npiv_vghdisk4

hdisk1 }vscsi_vghdisk2

hdisk0 } rootvg

hdisk1 }vscsi_vghdisk2

hdisk0 } rootvgMPIO

vscsi0

vscsi1

NPIV

MPIOvscsi0

vscsi1

hdisk

hdisk

hdisk

hdisk

HBA

HBA

HBA

HBA

29

Live Partition Mobility Support with IBM PowerHA

rootvg

Considerations:

• This is a planned move

• It assumes that all resources are virtualized through VIO (Storage & Ethernet connections)

• PowerHA should only experiencea minor disruption to theheartbeats during a move

• IVE / HEA virtual Ethernet is not supported for LPM

• VSCSI & NPIV virtual fiber mappings are supported

VIOS 1

Frame 1 Frame 2

VIOS 2

PowerHANode 1

VIOS 1

VIOS 2

PowerHANode 2

rootvg

datavg

SANStorage

PowerHANode 1

PowerHANode 2

The two solutions compliment each other by providing the ability to perform non-disruptivemaintenance while retaining the ability to fallover in the event of a system or application outage

How does it all work?

30

PowerHA and LPM Feature Comparison

�Automated failover upon vg access loss

�Server Workload Management**

�Automated failover upon any specified AIX error

(via customized error notification of error report entry)

�Automated failover upon HW failure

�Automated failover upon App failure

�Live OS/App move between physical frames*

�Automated failover upon System Failure (OS or HW)

�Software Maintenance

��Hardware Maintenance

�Energy Management**

Live

Partition

Mobility

PowerHA

SystemMirror

*~ 2 seconds of total interruption time** Require free system resources on target system

31

PowerHA SystemMirror: DLPAR Value Proposition

Pros:

• Automated action on acquisition of resources (bound to the PowerHA application server)

• HMC Verification Checking for connectivity to the HMC

• Ability to Grow LPAR on Failover

• Save $ on PowerHA SM Licensing–Thin Standby node

Cons:

• Requires Connectivity to HMC

• Potentially Slower FailoverSystem Specs:– 32-way (2.3GHz) Squad-H+

– 256GB of memory

Results:– 120GB DLPAR add took 1min 55 sec

– 246GB DLPAR add took 4 min 25 sec

– 30% busy running artificial load the add took

4 minutes 36 seconds

• Lacks ability to grow LPAR on-fly

HMCHMC

LPAR A LPAR B

Backup

ssh communication ssh communication

Minimal CPU CountDLPAR CPU Count

Application ServerMinimal CPU Count

32

Oracle DB 1 CPU

Banner DB 1 CPU

Standby 1 CPU

Standby 1 CPU

Standby 1 CPU

Standby 1 CPU

PeopleSoft 1 CPU

Financial DB 1 CPU

System A System B

Power7 740 – 16 Way

Print Server 2 CPU TSM 2 CPU

+ 1 CPU

Power7 740 – 16 Way

Capacity 10 CPU Capacity 10 CPU

+ 2 CPU

+ 1 CPU

+ 2 CPU

Acquired

via DLPAR

with App

Acquired

via DLPAR

with App

Cluster 1

Cluster 2

Cluster 3

Cluster 4

8 GB2Tivoli Storage Manager 5.5.2.0

32 GB3Production Financial DB

32 GB3Banner Financial DB

4 GB2AIX Print Server

8 GB2Production PeopleSoft

16 GB2Production Oracle DB

MemoryCPUApplications

DLPAR Licensing ScenarioHow does it all work?

33

Oracle DB 1 CPU

Banner DB 1 CPU

Standby 1 CPU

Standby 1 CPU

Standby 1 CPU

Standby 1 CPU

PeopleSoft 1 CPU

Financial DB 1 CPU

System A System B

Cluster 1

Cluster 2

Cluster 3

Cluster 4

Environment: PowerHA App Server Definitions

Oracle DB 1 CPU

Banner DB 1 CPU

Standby 1 CPU

Standby 1 CPU

Standby 1 CPU

Standby 1 CPU

PeopleSoft 1 CPU

Financial DB 1 CPU

System A System B

+ 1 CPU

+ 2 CPU

+ 1 CPU

+ 2 CPU

Acquired

via DLPAR

with App

Acquired

via DLPAR

with App

Cluster 1

Cluster 2

Cluster 3

Cluster 4

The actual application requirements are stored in the PowerHA SystemMirror definitions and enforced during the acquisition or release of application server resources

During acquisition of resourcesin the cluster start up the host will ssh to the pre-defined HMC/s to perform the DLPAR operation automatically

Application ServerMin 1Desired 2

HMC


34

Environment: DLPAR Resource Processing Flow

Oracle DB 1 CPU

Banner DB 1 CPU

Standby 1 CPU

Standby 1 CPU

System A System B

DLPAR

Cluster 1

Cluster 2

HMC

- 1 CPU

- 2 CPU

- 1 CPU

- 2 CPU

+ 1 CPU

+ 2 CPU

+ 1 CPU

+ 2 CPU

DLPAR

Oracle DB 2 CPU

Banner DB 3 CPU

Oracle DB 2 CPU

Banner DB 3 CPU

LPAR ProfileMin 1Desired 1Max 2



Max 2


Max 3



1. Activate LPARs Activate LPARs2. Start PowerHA

3. Release resources

Fallover or RG_move

4. Release resources

Stop cluster without takeover


Max 2


Max 3

Read Requirements

Take Aways:

• CPU allocations follow the application server wherever it is being hosted (this model allows you to lower the HA license count)• DLPAR resources will only get processed during the acquisition or release of cluster resources• PowerHA 6.1+ allows provide micro-partitioning support and the ability to also alter virtual processor counts• DLPAR resources can come from free CPUs in shared processor pool or CoD resources

35

Oracle DB 2 CPU

Banner DB 3 CPU

Standby 2 CPU

Standby 3 CPU

Standby 2 CPU

Standby 3 CPU

PeopleSoft 2 CPU

Financial DB 3 CPU

System A System B

Cluster 1

Cluster 2

Cluster 3

Cluster 4

PowerHA SystemMirror: DLPAR Value Proposition

Oracle DB 1 CPU

Banner DB 1 CPU

Standby 1 CPU

Standby 1 CPU

Standby 1 CPU

Standby 1 CPU

PeopleSoft 1 CPU

Financial DB 1 CPU

System A System B

+ 1 CPU

+ 2 CPU

+ 1 CPU

+ 2 CPU

Acquired

via DLPAR

with App

Acquired

via DLPAR

with App

Cluster 1

Cluster 2

Cluster 3

Cluster 4

PowerHA license counts:

Cluster 1 : 4 CPUsCluster 2 : 6 CPUsCluster 3 : 4 CPUsCluster 4 : 6 CPUs

Total : 20 licenses

HMC

Environment using dedicated CPU model (No DLPAR)

Environment using DLPAR modelPowerHA license counts:


Total : 14 licenses

36

PowerHA SystemMirror: DLPAR Modified Model

Oracle DB

Banner DB 1 CPU

Standby 1 CPU

Standby 1 CPU

PeopleSoft

Financial DB 1 CPU

System A System B

+ 4 CPU

+ 4 CPU

Acquired

via DLPAR

with App

Acquired

via DLPAR

with App

Cluster 1

Cluster2

HMC

Environment using modified DLPAR modelPowerHA license counts:

Cluster 1 : 6 CPUsCluster 2 : 6 CPUs

Total : 12 licenses

* Consolidated both Prod LPARs into one LPAR. Control separated by Resource Groups

Oracle DB 1 CPU

Banner DB 1 CPU

Standby 1 CPU

Standby 1 CPU

Standby 1 CPU

Standby 1 CPU

PeopleSoft 1 CPU

Financial DB 1 CPU

System A System B

+ 1 CPU

+ 2 CPU

+ 1 CPU

+ 2 CPU

Acquired

via DLPAR

with App

Acquired

via DLPAR

with App

Cluster 1

Cluster 2

Cluster 3

Cluster 4

HMCEnvironment using DLPAR model

* Same as previous slide

PowerHA license counts:


Total : 14 licenses

37

Data Protection with PowerHA SM 7.1 & CAA

ECM VGs were introduced in version 5.1

• Fast Disk Takeover

• Fast Failure Detection

• Disk heartbeating

Disk Fencing in CAA

• Fencing is automatic and transparent

• Cannot be turned off

• Fence group created by cl_vg_fence_init called from node_up

CAA Storage Framework fencing support

• Ability to specify level of disk access allowed by device driver

– Read/Write

– Read Only

– No Access (I/O is held until timeout)

– Fast Failure

Enhanced Concurrent Mode Volume Groups are now required

38

Data Protection with PowerHA SM 7.1 & CAAECM Volume groups and the newly added protection

ACTIVEread/write

PASSIVEread only

/data/app

/data/db

/dataDatavgECM VG

Node A Node B

Shared

LUNs

No Access

Fail all I/Os

In the event

of a failure

on node B

• LVM Enhanced Concurrent Mode VGs (Passive Mode)• Prevent writes to logical volume or volume group devices• Prevent filesystems from being mounted or any change requiring access to it

• CAA Fencing – prevents writes to the disk itself (ie. dd which runs below LVM level)

Cluster

Services

Cluster

Services

read/writeCAA

read onlyCAA

39

Management Console: WebSMIT vs. IBM Director

IBM Systems Director Plug-in

• New for PowerHA SystemMirror 7.1

• Only for management of

7.1 & above

• Same look and feel as IBM suite of products

• Will leverage existing Director implementation

• Uses clvt & clmgr CLI behind the covers

CLI & SMIT sysmirror panels still the most common management interfaces

WebSMIT

• Available since HACMP 5.2

• Required web server to run on host until HACMP 5.5 (Gateway server)

• Did not fall in line with look and feel of other IBM offerings

40

WebSMIT Gateway Model: One-to-Many (6.1 & Below)

WebSMIT converted from a one-to-one architecture to one-to-many

User_1 User_2 User_3 User_4

Cluster_A Cluster_B Cluster_C

*One* WebSMIT server…

managing multiple

clusters…

Multiple WebSMIT users…

accessing multiple clusters…

through *one* WebSMIT

server Standalone

WebSMIT

Server

41

WebSMIT Screenshot: Associations Tab

42

Three tier architecture provides scalability:

User Interface �� Management Server �� Director Agent

Director Server

� Central point of control

� Supported on AIX, Linux, and Windows

� Agent manager

User Interface

� Web-based interface

� Command-line interface

PowerHA

AIX

Director

AgentP D

P D

P D

P D

P D

P D

P DDiscovery of clusters

and resources

Director Agent

� Automatically installed on AIX 7.1 & AIX V6.1 TL06

Secure communication

PowerHA SystemMirror Cluster ManagementNew GUI User Interface for Version 7.1 Clusters

43

PowerHA SystemMirror Director Integration

• Accessing the SystemMirror Plug-ins

44

IBM Systems Director: Monitoring Status of Clusters

• Accessing the SystemMirror Plug-ins

45

PowerHA SystemMirror Configuration Wizards

• Wizards

46

PowerHA SystemMirror Smart Assistant EnhancementsDeploy HA Policy for Popular Middleware

47

PowerHA SystemMirror Detailed Views

• SystemMirror Management View

48

IBM Director: Management Dashboard

49

Do you know about clvt & clmgr ?

• “clmgr” announced in PowerHA SM 7.1

– clvt available since HACMP 5.4.1 for Smart Assists

– Hard linked “clmgr” to “clvt”

– Originally clmgr was intended for the Director team & rapidly evolved into a major, unintended, informal line item.

– allows for deviation from clvt without breaking the Smart Assists

• From this release forward, only “clmgr” is supported for customer use

– clvt is strictly for use by the Smart Assists

• New Command Line Infrastructure

– Ease of Management

• Stop

• Start

• Move Resources

# clmgr on cluster � Start Cluster Services on all nodes

# clmgr sync cluster � Verify & Sync Cluster

# clmgr rg appAgroup node=node2 � Move a resource group

50

Do you know about “clcmd” in CAA ?

# lslpp -w /usr/sbin/clcmd

/usr/sbin/clcmd bos.cluster.rte

# clcmd lssrc -g caa

-------------------------------

NODE mutiny.dfw.ibm.com

-------------------------------

Subsystem Group PID Status

clcomd caa 9502848 active

cld caa 10551448 active

clconfd caa 10092716 active

solid caa 7143642 active

solidhac caa 7340248 active

-------------------------------

NODE munited.dfw.ibm.com

-------------------------------

Subsystem Group PID Status

cld caa 4390916 active

clcomd caa 4587668 active

clconfd caa 6357196 active

solidhac caa 6094862 active

solid caa 6553698 active

Allows commands to be run across all cluster nodes

# clcmd lspv

-------------------------------

NODE mutiny.dfw.ibm.com

-------------------------------

hdisk0 0004a99c161a7e45 rootvg active

caa_private0 0004a99cd90dba78 caavg_private active

hdisk2 0004a99c3b06bf99 None

hdisk3 0004a99c3b076c86 None

hdisk4 0004a99c3b076ce3 None

hdisk5 0004a99c3b076d2d None

-------------------------------

NODE munited.dfw.ibm.com

-------------------------------

hdisk0 0004a99c15ecf25d rootvg active

caa_private0 0004a99cd90dba78 caavg_private active

hdisk2 0004a99c3b06bf99 None

hdisk3 0004a99c3b076c86 None

hdisk4 0004a99c3b076ce3 None

hdisk5 0004a99c3b076d2d None

51

PowerHA SystemMirror: Sample Application Monitor

# cat /usr/local/hascripts/ora_monitor.sh#!/bin/kshps –ef | grep ora_pmon_hatest

52

PowerHA SystemMirror: Pager Events

HACMPpager:methodname = "Herrera_notify"desc = “Lab Systems Pager Event"nodename = "connor kaitlyn"dialnum = "[email protected]"

filename = "/usr/es/sbin/cluster/samples/pager/sample.txt"eventname = "acquire_takeover_addr config_too_long event_errornode_down_complete node_up_complete"

retrycnt = 3timeout = 45

# cat /usr/es/sbin/cluster/samples/pager/sample.txtNode %n: Event %e occurred at %d, object = %o

Sample Email:

From: root 09/01/2009Subject: HACMPNode kaitlyn: Event acquire_takeover_addr occurred at Tue Sep 1 16:29:36 2009, object =

Attention:

Sendmail must be working and accessible via the firewall to receive notifications

� Action Taken: Halted Node Connor

53

PowerHA SystemMirror Tunables

• AIX I/O Pacing (High & Low Watermark)– Typically only enable if recommended after performance evaluation

– Historical values 33 & 24 have been updated to 513 & 256 on AIX 5.3 and 8193 & 4096 on AIX 6.1

http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.prftungd/doc/prftungd/disk_io_pacing.htm

• Syncd Setting– Default value of 60 � recommended change to 10

• Failure Detection Rate (FDR) – only for Version 6.1 & below

– Normal Settings should suffice in most environments (note that it can be tuned further)

– Remember to enable FFD when using disk heartbeating

• Pre & Post Custom EVENTs– Entry points for notifications or actions required before phases in a takeover

54

PowerHA SystemMirror: Testing Best Practices

• Test Application scripts and Application monitors thoroughly– Common problems include edits to scripts within scripts

• Test fallovers in all directions– Will confirm start & stop scripts on both locations

• Test Cluster– Lpars within same frame

– Virtual resources

• Utilize Available Tools – Cluster Test Tool

• Testing upgrades “Alternate disk install” is your friend

Best Practice:

Testing should be the foundation for your documentation in the event that someone not PowerHA savvy is there when a failure occurs.

55

How to be successful with PowerHA SystemMirror

• Strict Change Controls– Available test environment

– Testing of changes

• Leverage CSPOC functions– Create / Remove / Change - VGs, LVs, Filesystems

– User Administration

• Know what to look for– cluster.log / hacmp.out / clstrmgr.debug log files

– /var/hacmp/log/clutils.log � Summary of nightly verification

– /var/hacmp/clverify/clverify.log � detailed verification output

munited /# cltopinfo -m

Interface Name Adapter Total Missed Current MissedAddress Heartbeats Heartbeats

--------------------------------------------------------------------------------------------------------------------en0 192.168.1.103 0 0rhdisk1 255.255.10.0 1 1

Cluster Services Uptime: 30 days 0 hours 31 minutes

56

Summary

• Review your infrastructure for potential single points of failure

– Be aware of the potential pitfalls listed in the common mistakes slide

• Leverage Features like:

– File Collections

– Application monitoring

– Pager Notification Events

• Keep up with feature changes in each release

– New dependencies & fallover behaviors

• Virtualizing P7 or P6 environments is the foundation for Live Partition Mobility

– NPIV capable adapters can help simplify the configuration & management

• WebSMIT & IBM Director are the available GUI front-ends

– The cluster release will determine which one to use

57

http://www-03.ibm.com/systems/power/software/availability/aix/index.html

PowerHA SystemMirror IBM Portal

( … or Google ‘PowerHA SystemMirror’ and click I’m Feeling Lucky)

Learn More About PowerHA SystemMirror

Popular Topics:

* Frequently Asked Questions

* Customer References

* Documentation

* White Papers

58

Questions?

Thank you for your time!

59

Additional Resources

• New - Disaster Recovery Redbook

SG24-7841 - Exploiting PowerHA SystemMirror Enterprise Edition for AIX

http://www.redbooks.ibm.com/abstracts/sg247841.html?Open

• New - RedGuide: High Availability and Disaster Recovery Planning: Next-Generation Solutions for Multi server IBM Power Systems Environments

http://www.redbooks.ibm.com/abstracts/redp4669.html?Open

• Online Documentationhttp://www-03.ibm.com/systems/p/library/hacmp_docs.html

• PowerHA SystemMirror Marketing Pagehttp://www-03.ibm.com/systems/p/ha/

• PowerHA SystemMirror Wiki Pagehttp://www-941.ibm.com/collaboration/wiki/display/WikiPtype/High+Availability

• PowerHA SystemMirror (“HACMP”) Redbookshttp://www.redbooks.ibm.com/cgi-bin/searchsite.cgi?query=hacmp

HA17

Documents

higher aix

iz31207 aix

powerha version

sp1powerha systemmirror

rsct version

packaging changes

base cluster functionality

change history