HA With Sap en Queue Replication and Sun Cluster V02

Sun Microsystems GmbH


High Availabil ity with SAP enqueue replication and Sun Cluster

Author: Jörg Meiners, Sun Microsystems GmbHCreated: 16.01.2009Last Changed: 17.03.2009Version: 02

Page 1 of 21


Table of Contents1 - Conceptual overview...........................................................................................................................................................32 - About this paper..................................................................................................................................................................43 - Description of the target setup.............................................................................................................................................5

3.1 - Cluster configuration...................................................................................................................................................53.2 - Oracle database..........................................................................................................................................................53.3 - SAP System Central Services (SCS)................................................................................................................................63.4 - SAP Replicated Enqueue Server....................................................................................................................................63.5 - SAP Central Instance....................................................................................................................................................6

4 - Performing the installation..................................................................................................................................................84.1 - Solaris and Sun Cluster installation..............................................................................................................................84.2 - Basic Cluster configuration...........................................................................................................................................84.3 - Cluster data services configuration (part I)....................................................................................................................84.4 - SAP/Oracle Installation..............................................................................................................................................104.5 - Prepare SAP/Oracle switchover...................................................................................................................................114.6 - Test SAP/Oracle switchover........................................................................................................................................124.7 - Enable enqueue replication........................................................................................................................................124.8 - Test the enqueue replication......................................................................................................................................134.9 - Cluster data services configuration (part II).................................................................................................................144.10 - Enable and test the cluster.......................................................................................................................................16

5 - Appendix...........................................................................................................................................................................175.1 - Default profile...........................................................................................................................................................175.2 - ERS11 start profile......................................................................................................................................................175.3 - ERS11 instance profile................................................................................................................................................175.4 - SCS01 start profile......................................................................................................................................................185.5 - SCS01 instance profile................................................................................................................................................195.6 - JC00 start profile........................................................................................................................................................195.7 - JC00 instance profile...................................................................................................................................................205.8 - Script startsap_cluster.sh...........................................................................................................................................205.9 - Script stopsap_cluster.sh...........................................................................................................................................21

Page 2 of 21


1 - Conceptual overview

The Enqueue Server is a critical single point of failure in SAP architectures. And furthermore - as the Enqueue Server holds its data in memory - a failure would mean to loose the central SAP enqueue table. As a consequence, all ongoing SAP transactions having an active lock would abort. And special only to the SAP JAVA-stack, all SAP application instances will automatically restart as soon as the Enqueue Server is online again (and having a new empty enqueue table with a new ID).

To overcome this drawback, SAP strongly recommends to use the standalone Enqueue Server (as part of the SAP System Central Services) together with the Replication Enqueue Server especially in productive JAVA-stack systems.The Replication Enqueue Server runs on another host and contains a replica of the enqueue table (the replication table). Enqueue Server and Replicated Enqueue Server are running in a synchronous replication mode. Thus if the Enqueue Server fails and becomes online again, the enqueue table is not lost. Instead, the Enqueue Server can recover the enqueue table from the replication table.

The SAP enqueue replication concept is based on an underlying HA software and can't work without that. Sun Cluster supports this architecture out-of-the-box by providing best suitable product features:

• Provides a robust and reliable cluster framework.• Provides data agents for the basic infrastructure (Storage, Network).• Provides data agents for the SAP and database applications. In this context especially for the SAP SCS

(System Central Services: Message Server and Enqueue Server) and the Replicated Enqueue Server.• Handles the particular needs of Enqueue Server and Replicated Enqueue Server in a failover environment

➢ If Enqueue Server fails, Sun Cluster will restart the Enqueue Server on the cluster node (= physical server) that is currently hosting the Replicated Enqueue Server. This is mandatory, because the restarted Enqueue Server will recover the enqueue table from the replication table using a server-local access to the shared memory segment that holds the replicated table.

➢ After table recovery has finished, the Replicated Enqueue Server will be stopped. Sun Cluster will restart the Replicated Enqueue Server on another cluster node (different than the node hosting the Enqueue Server) – if another cluster node is available. Thus keeping up the enqueue replication concept and maximal availability.

To learn more about the SAP enqueue replication and Sun Cluster, please refer to:• Sun Cluster 3.2 documentation (http://docs.sun.com/app/docs/prod/sun.cluster32)• Sun Cluster 3.2 Data Service for SAP Web Application Server Guide

(http://docs.sun.com/app/docs/doc/820-2568)• SAP documentation (http://help.sap.com → SAP NetWeaver 7.0 Library → Look for the chapter

"Standalone Enqueue Server")

Page 3 of 21

http://docs.sun.com/app/docs/prod/sun.cluster32

http://help.sap.com/

http://docs.sun.com/app/docs/doc/820-2568


2 - About this paper

This paper describes the implementation of a SAP High Availability Architecture based on the “SAP enqueue replication” concept and a Sun infrastructure.An installation-”HowTo” is being provided, using:

• Sun Solaris 10 (Update 5/08)• Sun Cluster 3.2.0 (Update 2/08)• Oracle Database 10g Enterprise Edition Release 10.2.0.2.0• SAP NetWeaver Portal 7.0 (SP16, Kernel 7.00 patch# 159)

The HowTo is focused on all necessary implementation and configuration tasks that are specific for the “SAP Enqueue replication” scenario. A “standard” installation of Solaris, Sun Cluster and SAP Portal is assumed and a description of this process is out of scope.

Also it is not intended to describe all aspects of an “High End” SAP/Oracle/Cluster implementation. Many things in this example installation are implemented as simple as possible (like filesystem layout, parameter tuning, ...) to hold this paper small and to focus on “SAP enqueue replication” setup.

Page 4 of 21


3 - Description of the target setup

3.1 - Cluster configuration

Cluster Name: sap1Node Names: sscsapx1,sscsapx2Transport Adapters: bge1,e1000g1 (on both nodes)Quorum Device Name: d11 (c4t600015D00022B30000000000000072CDd0)Public Network sscsapx1

• IPMP-group: sc_ipmp0➢ bge0 (sscsapx1 192.168.14.35)➢ e1000g0 (sscsapx1b 192.168.14.1)

Public Network sscsapx4• IPMP-group: sc_ipmp0

➢ bge0 (sscsapx2 192.168.14.36)➢ e1000g0 (sscsapx2b 192.168.14.6)

3.2 - Oracle database

DB name PE2, installed on logical hostname pe2oralh.

Cluster-Resourcegroup: pe2ora; preferred node: sscsapx1

Cluster-resource Type Properties

pe2ora_stor SUNW.HAStoragePlus zpool, zfs

Devices: d10 (c4t600015D00022B30000000000000072D1d0)

Failover filesystem

Mountpoints: /oracle/PE2 (150 GB)

Page 5 of 21

Figure 1: Overview Cluster Resourcegroups

Local (root FS):/oracle/client/oracle/stage/102_64

SAP SCS Group

Logical Hostname

SAP Message Server

NFS Share (/sapmnt/PE2, /usr/sap/trans)

SAP Enqueue Server

Global Filesystem UFS+SVM (/usr/sap/PE2, /sapmnt/PE2, /usr/sap/trans)

Oracle DB Group

Logical Hostname

Oracle Database

Oracle Listener

Failover Filesystem ZFS (/oracle/PE2)

SAP CI

Logical Hostname

SAP NW WAS

Local (root FS):/oracle/client/oracle/stage/102_64

SAP Replicated Enqueue Group

Logical Hostname

SAP Replicated Enqueue Server


pe2ora_lh SUNW.LogicalHostname ip / hostname = 192.168.14.24 / pe2oralh

pe2ora_lsnr SUNW.oracle_listener

pe2ora_db SUNW.oracle_server

Notes regarding SAP/Oracle filesystem configuration:• /oracle/client: Local directory in root-fs. To be copied on both cluster nodes.• /oracle/stage/102_64: Local directory in root-fs of the install server. Not needed for running the database.• /sapmnt/PE2 and /usr/sap/PE2: Will be accessible to the database by the cluster global filesystem. No

need to copy anything.

3.3 - SAP System Central Services (SCS)

SAP SID PE2, Instance No. 01 (instance name SCS01), installed on logical hostname pe2scslh.

Cluster-Resourcegroup: pe2scs; preferred node: sscsapx1


pe2scs_stor SUNW.HAStoragePlus SVM diskset pe2scs_stor

Devices: d11 (c4t600015D00022B30000000000000072CDd0)

Global filesystems

Mountpoints: /usr/sap/PE2 (20 GB), /sapmnt/PE2 (10 GB), /usr/sap/trans (10 GB)

pe2scs_lh SUNW.LogicalHostname ip / hostname = 192.168.14.23 / pe2scslh

pe2scs_ms SUNW.sapscs (Just FYI: SAP Message Server)

pe2scs_enq SUNW.sapenq (Just FYI: SAP Enqueue Server)

pe2scs_nfs SUNW.nfs Exports “/sapmnt/PE2” and “/usr/sap/trans”

Note: Any necessary application shared data access (for example to “/sapmnt/PE2”) is done by using the global filesystem(s) within the cluster. This is also to avoid any problems with “NFS cross-mounts” between cluster nodes (that are generally not supported in a Sun Cluster configuration). The NFS server configured in this resourcegroup is to provide all cluster-external NFS-clients – like additional SAP application instances – with access to the shared data.

3.4 - SAP Replicated Enqueue Server

SAP SID PE2, Instance No. 11 (instance name ERS11), installed on logical hostname pe2erslh.

Cluster-Resourcegroup: pe2ers; preferred node: sscsapx2


pe2ers_lh SUNW.LogicalHostname ip / hostname = 192.168.14.25 / pe2erslh

pe2ers_enq SUNW.saprepl (Just FYI: SAP Replicated Enqueue Server)

Notes regarding SAP/Oracle filesystem configuration:• No dedicated filesystems needed. Global filesystems (/usr/sap/PE2, /sapmnt/PE2) will be used for

instance specific data.

3.5 - SAP Central Instance

SAP SID PE2, Instance No. 00 (instance name JC00), installed on logical hostname pe2cilh.

Cluster-Resourcegroup: pe2ci; preferred node: sscsapx1


Page 6 of 21


pe2ci_lh SUNW.LogicalHostname ip / hostname = 192.168.14.22 / pe2cilh

pe2ci_jc00 SUNW.sapwebas (Just FYI: SAP Web Application Server)

Notes regarding SAP/Oracle filesystem configuration:• No dedicated filesystem needed. Global filesystems (/usr/sap/PE2, /sapmnt/PE2) will be used for instance

specific data.

Page 7 of 21


4 - Performing the installation

4.1 - Solaris and Sun Cluster installation

Install Sun Solaris 10 (Update 5/08) and Sun Cluster 3.2.0 (Update 01) on the servers sscsapx1 and sscsapx2. A description is out of scope of this paper.

Note: The documentation “Sun Cluster Quick Start Guide for Solaris OS” (http://docs.sun.com/app/docs/doc/820-2556) provides a good example (cookbook) for setting up the cluster hardware, installing the Solaris and Sun Cluster Software.

Note: Using Solaris Volume Manager (SVM), take care not to use the same metadevice name for the filesystem “/globaldevices” on both nodes. The cluster will not work properly. Instead take different metadevice names (for example d200 on sscsapx1 and d400 on sscsapx2).

4.2 - Basic Cluster configuration

This step describes some basic cluster configurations. To be executed, if not done before.

Add Quorum Device# cldevice list v (… FYI: to view the available did devices)# clquorum add d11 (… adds d11 as quorum device)# clquorum list (… FYI: to view the configured quorum devices)# clquorum status (… FYI: to view the quorum status)

Network configuration# eeprom localmacaddress?=true (… to be executed on both cluster nodes)

Data Services registration# clresourcetype register SUNW.HAStoragePlus# clresourcetype register SUNW.LogicalHostname# clresourcetype register SUNW.oracle_listener# clresourcetype register SUNW.oracle_server# clresourcetype register SUNW.sapscs# clresourcetype register SUNW.sapenq# clresourcetype register SUNW.saprepl# clresourcetype register SUNW.nfs# clresourcetype register SUNW.sapwebas

HostnamesAdd all logical hostnames needed to the local /etc/hosts on sscsapx1 and sscsapx2

192.168.14.24 pe2oralh pe2oralh.abc.de # logical host for Oracle DB192.168.14.23 pe2scslh pe2scslh.abc.de # logical host for SAP SCS01192.168.14.25 pe2erslh pe2erslh.abc.de # logical host for SAP ERS11192.168.14.22 pe2cilh pe2cilh.abc.de # logical host for SAP JC00

4.3 - Cluster data services configuration (part I)

This step will configure the storage, filesystems and IP addresses. Hereby providing the necessary infrastructure platform to install SAP/Oracle afterwards.

Oracle databasea) Create failover filesystem (zpool and zfs filesystem)

# zpool create pe2ora_stor /dev/dsk/c4t600015D00022B30000000000000072D1d0# zfs create pe2ora_stor/oraclePE2# zfs set mountpoint=/oracle/PE2 pe2ora_stor/oraclePE2

Test:# zpool export pe2ora_stor (… on the currently hosting node)# zpool import pe2ora_stor (… on the target node)

b) Create resourcegroup with storage and logical hostname

Page 8 of 21




# clresourcegroup create n sscsapx1,sscsapx2 pe2ora# clresource create t HAStoragePlus g pe2ora p zpools=pe2ora_stor p AffinityOn=true pe2ora_stor# clreslogicalhostname create g pe2ora h pe2oralh N sc_ipmp0@sscsapx1,sc_ipmp0@sscsapx2 pe2ora_lh

c) Enable and test Resourcegroup# clresourcegroup online pe2ora# clresourcegroup switch n sscsapx2 pe2ora# clresourcegroup switch n sscsapx1 pe2ora

SAP System Central Services (SCS)a) Create SVM Diskset and cluster devicegroupCheck if /kernel/drv/md.conf identical on both cluster nodes

# cldevice list v (… FYI: Just to review; look for d11)# metaset s pe2scs_stor a h sscsapx1 sscsapx2 (… FYI: creates devicegroup “pe2scs_stor”)# cldevicegroup show (… FYI: To view existing device groups)# cldevicegroup status (… FYI: To view status of device groups)# cldevicegroup set p <name>=<value> pe2scs_stor (… Optional. Nothing mandatory here !!)

Test cluster devicegroup:# cldevicegroup switch n sscsapx2 pe2scs_stor# cldevicegroup switch n sscsapx1 pe2scs_stor

b) Create global filesystems (UFS)# metaset s pe2scs_stor a /dev/did/rdsk/d11# metainit s pe2scs_stor d100 p e /dev/did/rdsk/d11 10G# metainit s pe2scs_stor d101 p /dev/did/rdsk/d11s0 20G# metainit s pe2scs_stor d102 p /dev/did/rdsk/d11s0 all# newfs /dev/md/pe2scs_stor/rdsk/d100# newfs /dev/md/pe2scs_stor/rdsk/d101# newfs /dev/md/pe2scs_stor/rdsk/d102# mkdir p /sapmnt/PE2 (… to be executed on both cluster nodes)# mkdir p /usr/sap/PE2 (… to be executed on both cluster nodes)# mkdir p /usr/sap/trans (… to be executed on both cluster nodes)

c) Add entries to /etc/vfstab (both cluster nodes):/dev/md/pe2scs_stor/dsk/d100 /dev/md/pe2scs_stor/rdsk/d100 /sapmnt/PE2 ufs 2 yes global,logging/dev/md/pe2scs_stor/dsk/d101 /dev/md/pe2scs_stor/rdsk/d101 /usr/sap/PE2 ufs 2 yes global,logging/dev/md/pe2scs_stor/dsk/d102 /dev/md/pe2scs_stor/rdsk/d102 /usr/sap/trans ufs 2 yes global,logging# sccheck# mount /sapmnt/PE2# mount /usr/sap/PE2# mount /usr/sap/trans

d) Create resourcegroup with storage, logical hostname and NFS server# clresourcegroup create n sscsapx1,sscsapx2 pe2scs# clresource create t HAStoragePlus g pe2scs p GlobalDevicePaths=pe2scs_stor p AffinityOn=true pe2scs_stor# clreslogicalhostname create g pe2scs h pe2scslh N sc_ipmp0@sscsapx1,sc_ipmp0@sscsapx2 pe2scs_lh# clresourcegroup set p Pathprefix=/usr/sap/PE2 pe2scs# mkdir p /usr/sap/PE2/SUNW.nfs

Add these two entries (shares) to /usr/sap/PE2/SUNW.nfs/dfstab.pe2scs_nfsshare F nfs o rw /sapmnt/PE2share F nfs o rw /usr/sap/trans# clresource create g pe2scs t SUNW.nfs p Resource_dependencies=pe2scs_stor pe2scs_nfs

e) Enable and test Resourcegroup# clresourcegroup online pe2scs# clresourcegroup switch n sscsapx2 pe2scs# clresourcegroup switch n sscsapx1 pe2scs

SAP Central Instancea) Create Resourcegroup with logical hostname

# clresourcegroup create n sscsapx1,sscsapx2 pe2ci# clreslogicalhostname create g pe2ci h pe2cilh N sc_ipmp0@sscsapx1,sc_ipmp0@sscsapx2 pe2ci_lh

b) Enable and test Resourcegroup# clresourcegroup online pe2ora# clresourcegroup switch n sscsapx2 pe2ci# clresourcegroup switch n sscsapx1 pe2ci

SAP Replicated Enqueue Servera) Create Resourcegroup with logical hostname

# clresourcegroup create n sscsapx2,sscsapx1 pe2ers

Page 9 of 21


# clreslogicalhostname create g pe2ers h pe2erslh N sc_ipmp0@sscsapx2,sc_ipmp0@sscsapx1 pe2ers_lh

b) Enable and test Resourcegroup# clresourcegroup switch n sscsapx1 pe2ci# clresourcegroup switch n sscsapx2 pe2ci

4.4 - SAP/Oracle Installation

SAP/Oracle standard installationPerform a “standard” installation of SAP NetWeaver Portal 7.0 and Oracle Database 10g on the prepared storage and logical hosts. Any special configuration needs regarding failover, Sun Cluster or enqueue replication will be described/done not here, but in the following steps.

• Oracle DB on node sscsapx1: Name PE2, configured on logical hostname pe2oralh• SAP SCS on node sscsapx1: SAP SID PE2, Instance No. 01 (instance name SCS01), configured on

logical hostname pe2scslh• SAP CI on node sscsapx1: SAP SID PE2, Instance No. 00 (instance name JC00), installed on logical

hostname pe2cilh

SAP Replicated Enqueue Server installationPerform a “standard” installation of the SAP Replicated Enqueue Server on the prepared storage and logical hosts. Any special configuration needs regarding failover, Sun Cluster or Enqueue replication will be described/done not here, but in the following steps.•SAP Replicated Enqueue Server on node sscsapx2: SAP SID PE2, Instance No. 11 (instance name ERS11), installed on logical hostname pe2erslh

As of today (January 2009, SAP NetWeaver 7.0 EHP 1, SPS02) there is no SAP tool like “sapinst” to install the SAP Replicated Enqueue Server. Instead, it has to be done manually. You find the guide in the SAP documentation (http://help.sap.com → SAP NetWeaver 7.0 Library → Look for the chapter "Setting Up the Replication Server").

Here is a summary. It is simplified and adopted to the special needs of the cluster configuration:• Directory structure

Create the instance-specific subdirectories in the global filesystem “/usr/sap/PE2”:/usr/sap/PE2/ERS11/exe/usr/sap/PE2/ERS11/log/usr/sap/PE2/ERS11/data/usr/sap/PE2/ERS11/work

• Instance profileCreate the instance profile “/sapmnt/PE2/profile/PE2_ERS11_pe2erslh”. You find the file for reuse in the appendix (Chapter “ERS11 instance profile”). Please pay attention to comments given in the file. Some changes have to be done (compared to the SAP documentation) because of Sun Cluster specifics.

• Start profileYou may want to create a start profile. This is optional, because Sun Cluster will start the server-process directly (by executing the command: “/usr/sap/PE2/SYS/exe/run/enrepserver pf=/sapmnt/PE2/profile/PE2_ERS11_pe2erslh”). In this way, no start profile is being used. If you want to start the server-process manually, you can do it in the same way. Just if you want to use “startsap” (“startsap r3 ERS11 pe2erslh”), you have to provide the start profile “/sapmnt/PE2/profile/START_ERS11_pe2erslh”. You find the file for reuse in the appendix (Chapter “ERS11 start profile”).

Disable “sapcpe”Copying the SAP Kernel from “/sapmnt/PE2/exe” to the instance-specific directories “/usr/sap/PE2/<instancename>/exe” does not make sense, because it's all on the global filesystem(s). Instead start directly from /usr/sap/PE2/SYS/exe/run (that is a softlink to /sapmnt/PE2/exe).

• Set these parameters in all start profiles and instance profiles:DIR_CT_RUN = /usr/sap/PE2/SYS/exe/runDIR_EXECUTABLE = /usr/sap/PE2/SYS/exe/run

• Check and adjust environment of OS users pe2adm and orape2 to point to the correct directoryPATH=...:/usr/sap/PE2/SYS/exe/run:...DIR_LIBRARY=.../usr/sap/PE2/SYS/exe/run:...LD_LIBRARY_PATH=.../usr/sap/PE2/SYS/exe/run:...

• In the instance start profiles, disable all of the “sapcpe” calls like this one:Execute_00 = immediate $(DIR_CT_RUN)/sapcpe$(FT_EXE) pf=$(_PF) $(_CPARG0)

Page 10 of 21



• Delete the existing (already copied) files in the instance-specific directories /usr/sap/PE2/<instancename>/exe.

4.5 - Prepare SAP/Oracle switchover

Prepare both nodes (sscsapx1 and sscsapx2) to be capable to host the SAP/Oracle applications. If not done before (during the installation), please check if these configurations are available and identical on both cluster nodes.

Home directoriesRelocate the home directories of the OS users pe2adm and orape2 onto global filesystems.

• Homedir of pe2adm: /usr/sap/PE2/pe2adm• Homedir of orape2: /oracle/PE2/orape2

Create OS users and groupsCreate the accounts on both hosts, if missing.

# groupadd g 100 sapsys# groupadd g 101 dba# groupadd g 102 oper# groupadd g 103 sapinst# useradd g 100 G 101,102,103 u 110 d /usr/sap/PE2/pe2adm s /bin/csh pe2adm# useradd g 101 G 102,103 u 111 d /oracle/PE2/orape2 s /bin/csh orape2

Solaris projectCheck “/etc/project” for this entry:

PE2:200:SAP System PE2:orape2,pe2adm,root::process.maxsemnsems=(priv,2048,deny);project.maxsemids=(priv,1024,deny);project.maxshmids=(priv,256,deny);project.maxshmmemory=(priv,18446744073709551615,deny)

Definition of default projectCheck file “/etc/user_attr” for these entries:

pe2adm::::project=PE2orape2::::project=PE2

ServicesCheck file “/etc/services” for these entries:

sapdp00 3200/tcp # SAP System Dispatcher Portsapdp01 3201/tcp # SAP System Dispatcher Portsapdp11 3211/tcp # SAP System Dispatcher Portsapgw00 3300/tcp # SAP System Gateway Portsapgw01 3301/tcp # SAP System Gateway Portsapgw11 3311/tcp # SAP System Gateway Portsapdp00s 4700/tcp # SAP System Dispatcher Security Portsapdp01s 4701/tcp # SAP System Dispatcher Security Portsapdp11s 4711/tcp # SAP System Dispatcher Security Portsapgw00s 4800/tcp # SAP System Gateway Security Portsapgw01s 4801/tcp # SAP System Gateway Security Portsapgw11s 4811/tcp # SAP System Gateway Security Port

HostsCheck file “/etc/hosts” to contain all logical hosts and physical nodes.

Mountpoints for failover filesystemsMake sure, that the mountpoint “/oracle/PE2” is available on all nodes.

Java J2SEInstall the most recent version of J2SE 1.4.2.x SDK on a global filesystem.

• Download self-extracting version von http://java.sun.com• Install into directory /usr/sap/PE2/j2se• Modify environment of user pe2adm: JAVA_HOME=/usr/sap/PE2/j2se• Modify JC00 instance profile: jstartup/vm/home = /usr/sap/PE2/j2se

Note: For Solaris x64 you need to download a version from http://java.sun.com/j2se/1.4.2/SAPsite/download.html.

JCE policy filesDownload from http://java.sun.com/javase/downloads and extract to:

Page 11 of 21

http://java.sun.com/javase/downloads

http://java.sun.com/j2se/1.4.2/SAPsite/download.html

http://java.sun.com/


/usr/sap/PE2/j2se/jre/lib/security/local_policy.jar/usr/sap/PE2/j2se/jre/lib/security/US_export_policy.jar

saposcol directoryCreate a local working directory “/usr/sap/tmp” for saposcol on both nodes (owner: root, group: sapsys, permissions: drwxrwxr-x).

Oracle clientCopy directory “/oracle/client” (from the Oracle-installation node) to the other nodes.

Oracle inventoryRelocate the Oracle inventory to shared storage. On the Oracle-installation node;

• Modify the entry in “/var/opt/oracle/oraInst.loc”: inventory_loc=/oracle/PE2/oraInventory• Copy directory “/var/opt/oracle” to the other nodes• Move directory “/oracle/oraInventory” to “/oracle/PE2/oraInventory”

4.6 - Test SAP/Oracle switchover

Check if SAP/Oracle is able to run on both cluster nodes. It makes sense to test application switchover at this point in time, without a fully completed Sun Cluster integration. Because in the case of any errors, you could be sure to have no impact of a potentially inaccurate cluster configuration. And you could focus your error analysis on the native SAP stuff.Start/Stop SAP and Oracle manually, relocate filesystems and IP adresses using Sun Cluster commands.

To stop SAP, use following commands and OS user pe1adm:# stopsap r3 JC00 pe2cilh# stopsap r3 SCS01 pe2scslh# stopsap r3 ERS11 pe2erslh(or kill the “enrepserver” process, if you have not provided a start profile for ERS11)

To stop Oracle, use following commands and OS user orape2:# sqlplusEnter username: / as sysdbaSQL> shutdown# lsnrclt stop

To start Oracle, use following commands and OS user orape2:# lsnrclt start# sqlplusEnter username: / as sysdbaSQL> startup

To start SAP, use following commands and OS user pe1adm:# startsap r3 SCS01 pe2scslh# startsap r3 JC00 pe2cilh# startsap r3 ERS11 pe2erslh(or “# /usr/sap/PE2/SYS/exe/run/enrepserver pf=/sapmnt/PE2/profile/PE2_ERS11_pe2erslh”, if you have not provided a start profile for ERS11)

To switch resourcegroups (containing storage and IP) between the cluster nodes:# clrg switch n <targetnode> <resourcegroup>

with <target-node> = sscsapx1 or sscsapx2with <resourcegroup> = pe2scs, pe2ers, pe2ora or pe2ci

4.7 - Enable enqueue replication

You need to set some SAP parameters to enable the enqueue replication. For complete list of enqueue-parameters please refer to the SAP documentation (http://help.sap.com → SAP NetWeaver 7.0 Library → Look for the chapter "Profile Parameter Settings for the Standalone Enqueue Server").

This overview shows the enqueue (and enqueue replication) related parameterisation of this example installation. All profiles (including all parameters) are listed in the appendix of this paper for reuse.

Parameters for SCS01Configured in the instance profile:

Page 12 of 21



ipc/shm_psize_34 = 0enque/server/replication = trueenque/serverinst = 01enque/table_size = 4096rdisp/enqname = $(rdisp/myname)enque/snapshot_pck_ids = 100

Parameters for JC00 (and all other application instances)Configured in the default profile:

enque/deque_wait_answer = TRUE

Parameters for ERS11Nothing more to do, if you have reused the instance profile from the appendix of this paper.

4.8 - Test the enqueue replication

It makes sense to test the enqueue replication at this point in time, without a fully completed Sun Cluster integration. Because in the case of any errors, you could be sure to have no impact of a potentially inaccurate cluster configuration. And you could focus your error analysis on the native SAP stuff.

For more infos, please refer to the SAP documentation (http://help.sap.com → SAP NetWeaver 7.0 Library → Look for the chapter "Replication Server: Check Installation").

Execute the following SAP-related commands (ensmon, enqt, …) with OS user pe2adm.

Attention: Take care to execute the commands on the correct nodes as shown below. Some of the commands are working on both sides (Enqueue and Replicated Enqueue), others not. It may be astonishingly to create the enqueue table test-entries on the Replicated Enqueue Server side (and not on the Enqueue Server side). But this is NOT a typo.

1) Switch applications to their preferred nodeStart the applications on their preferred node. Oracle DB, SCS01, JC00 on sscsapx1. ERS11 on sscsapx2. See chapter “Test SAP/Oracle switchover” on how to start, stop and switch the applications (in this installation phase, without having a complete cluster integration).

2) Check if replication is activeOn sscsapx2 (the current node hosting ERS11, containing the Replicated Enqueue Server):

# ensmon pf=/sapmnt/PE2/profile/PE2_ERS11_pe2erslh 2

The output should look like this:Try to connect to host pe2scslh service sapdp01get replinfo request executed successfully

Replication is enabled in server, repl. server is connectedReplication is active…

3) Check the enqueue table IDOn sscsapx1 (the current node hosting SCS01, containing the Enqueue Server):

# enqt pf=/sapmnt/PE2/profile/PE2_SCS01_pe2scslh 97On sscsapx2 (the current node hosting ERS11, containing the Replicated Enqueue Server):

# enqt pf=/sapmnt/PE2/profile/PE2_ERS11_pe2erslh 97

The output should be identical and look like this:REQEnqId: EnqTabCreaTime/RandomNumber = 14.01.2009 18:44:31 1231955071 / 2868ReqOrd at Srv: TimeInSecs/ReqNumberThisSec = 18.01.2009 11:26:32 1232274392 / 1

4) Create enqueue table entriesOn sscsapx2 (the current node hosting ERS11, containing the Replicated Enqueue Server):

# enqt pf=/sapmnt/PE2/profile/PE2_ERS11_pe2erslh 11 20This command will create 20 dummy entries in the table.

5) View enqueue table entries

Page 13 of 21



On sscsapx1 (the current node hosting SCS01, containing the Enqueue Server):# enqt pf=/sapmnt/PE2/profile/PE2_SCS01_pe2scslh 20

On sscsapx2 (the current node hosting ERS11, containing the Replicated Enqueue Server):# enqt pf=/sapmnt/PE2/profile/PE2_ERS11_pe2erslh 20

The output should be identical and contain 20 entries like this (and maybe some others):000 ENQT ENQ_PERF X enqueue performance test 11111111000

6) Perform switchoverStop the instance SC01 on node sscsapx1 and start it on node sscsapx2. See chapter “Test SAP/Oracle switchover” on how to start, stop and switch the applications (in this installation phase, without having a complete cluster integration).

7) Check the success• The Enqueue Server should have stopped the Replicated Enqueue Server on node sscsapx2. Check if the

OS process is gone. It is the process command name “er.sapPE2_ERS11”, if you have started ERS11 with “startssap” (having a start profile). Otherwise, if you have started the Replicated Enqueue Server directly, the process command name is “enrepserver”.

• Check if the enqueue table ID is still the same from before the switchover. Repeat step 3, solely regarding SCS01 (because ERS11 should be stopped). Remember that the node has changed from sscsapx1 to sscsapx2 !

• Check if the 20 enqueue table entries still exist. Repeat step 4, solely regarding SCS01 (because ERS11 should be stopped). Remember that the node has changed from sscsapx1 to sscsapx2 !

Note. In the case, that the replication does not work correctly, you can use these files foe debugging:• Look for the developer traces “dev_en*” in the directories “/usr/sap/PE2/SCS01/work” and

“/usr/sap/ERS11/SCS01/work”.• Attention: If you have started the Replicated Enqueue Server directly (not using “startsap” and a start

profile), then you will find the trace files of the Replicated Enqueue Server in the home directory of user pe2adm. Not yet, but in the final configuration, Sun Cluster will also start the Enqueue Server directly (not using “startsap” and a start profile). Thus, you will find also these trace files in the home directory of user pe2adm.

• If needed, increase the trace level. Set parameter “rdisp/TRACE = 3” in the instance profiles of SCS01 and ERS11.

4.9 - Cluster data services configuration (part II)

This step will finalize the integration of SAP/Oracle into the Sun Cluster.

Provide start/stop script for SAP instancesProvide these two scripts (owner: pe2adm, group: sapsys, permissions: -rwxr—r--):

/usr/sap/PE2/pe2adm/startsap_cluster.sh/usr/sap/PE2/pe2adm/stopsap_cluster.sh

You find them in the appendix. Sun Cluster will call them to start and stop the SAP instances SCS01 and JC00.

Oracle databasea) Create Oracle monitor-user and tableStart the oracle database.Use OS user orape2 to connect to the db:

# sqlplus "/ as sysdba"sql> create user haoracle identified by haoracle;sql> alter user haoracle default tablespace system quota 1m on system;sql> grant select on v_$sysstat to haoracle;sql> grant select on v_$archive_dest to haoracle;sql> grant create session to haoracle;sql> grant create table to haoracle;sql> exit;

b) Resource for Oracle databaseStop database, if running.Then create the cluster resource:

# clresource create g pe2ora \t SUNW.oracle_server \p Resource_project_name=PE2 \

Page 14 of 21


p Connect_string=haoracle/haoracle \p ORACLE_SID=PE2 \p ORACLE_HOME=/oracle/PE2/102_64 \p Alert_log_file=/oracle/PE2/saptrace/background/alert_PE2.log \p Restart_type=RESOURCE_RESTART \p resource_dependencies=pe2ora_stor \pe2ora_db

c) Resource for Oracle listenerStop listener, if running.Then create the cluster resource:

# clresource create g pe2ora \t SUNW.oracle_listener \p Resource_project_name=PE2 \p LISTENER_NAME=LISTENER \p ORACLE_HOME=/oracle/PE2/102_64 \p resource_dependencies=pe2ora_stor \pe2ora_lsnr

SAP System Central Services (SCS)a) Remove the Enqueue Server from instance startDisable the startup in the start profile "/sapmnt/PE2/profile/START_SCS01_pe2scslh". Remove these lines in the section ”Start SAP locking service”:

_EN = en.sap$(SAPSYSTEMNAME)_$(INSTANCE_NAME)Execute_03 = local rm f $(_EN)Execute_04 = local ln s f $(DIR_EXECUTABLE)/enserver$(FT_EXE) $(_EN)Start_Program_01 = local $(_EN) pf=$(DIR_PROFILE)/PE2_SCS01_pe2scslh

Note. You can start the Enqueue Server directly (and Cluster will do it in this way):# /usr/sap/PE2/SCS01/exe/enserver pf=/sapmnt/PE2/profile/PE2_SCS01_pe2scslh

b) Resource for Enqueue ServerStop instance SCS01, if running.Then create the cluster resource:

# clresource create d g pe2scs t SUNW.sapenq \p Resource_project_name=PE2 \p enqueue_profile=/sapmnt/PE2/profile/PE2_SCS01_pe2scslh \p enqueue_server=/usr/sap/PE2/SYS/exe/run/enserver \p sap_user=pe2adm \p enqueue_instance_number=01 \p resource_dependencies=pe2scs_stor,pe2ora_db{ANY_NODE},pe2ora_lsnr{ANY_NODE} \pe2scs_enq

c) Resource for SCS01 (remaining Message Server)Stop instance SCS01, if running.Then create the cluster resource:

# clresource create d g pe2scs t SUNW.sapscs \p Resource_project_name=PE2 \p sap_sid=PE2 \p sap_instance_number=01 \p sap_instance_name=SCS01 \p msg_server_port=3901 \p scs_startup_script=/usr/sap/PE2/pe2adm/startsap_cluster.sh \p scs_shutdown_script=/usr/sap/PE2/pe2adm/stopsap_cluster.sh \p resource_dependencies=pe2scs_stor,pe2ora_db{ANY_NODE},pe2ora_lsnr{ANY_NODE} \pe2scs_ms

SAP Central Instancea) Resource for JC00Stop instance JC00, if running.Then create the cluster resource:

# clresource create d g pe2ci t SUNW.sapwebas \p Resource_project_name=PE2 \p sap_sid=PE2 \p sap_instance_number=00 \p sap_instance_name=JC00 \p sap_instance_type=J2EE \p webas_startup_script=/usr/sap/PE2/pe2adm/startsap_cluster.sh \p webas_shutdown_script=/usr/sap/PE2/pe2adm/stopsap_cluster.sh \p resource_dependencies=pe2scs_stor{ANY_NODE},pe2ora_db{ANY_NODE}, \ pe2ora_lsnr{ANY_NODE},pe2scs_enq_1{ANY_NODE},pe2scs_ms{ANY_NODE} \pe2ci_jc00

Page 15 of 21


SAP Replicated Enqueue Servera) Resource for Replicated Enqueue Server (instance ERS11)Stop Replicated Enqueue Server (Instance ERS11), if running.Then create the cluster resource:

# clresource create d g pe2ers t SUNW.saprepl \p replica_profile=/sapmnt/PE2/profile/PE2_ERS11_pe2erslh \p replica_server=/usr/sap/PE2/SYS/exe/run/enrepserver \p sap_user=pe2adm \p resource_dependencies=pe2scs_stor{ANY_NODE},pe2scs_enq{ANY_NODE} \pe2ers_enq

Group affinitiesConfigure these resourcegroup affinities to reflect the special failover requirements of the SAP enqueue replication concept.

# clresourcegroup set p RG_affinities=+pe2ers pe2scsThe weak positive affinity setting ensures that, in case of failover, the group pe2scs (SCS01, including Enqueue Server) fails over to the node where the group pe2ers (ERS11, Replicated Enqueue Server) is running.

# clresourcegroup set p RG_affinities=pe2scs pe2ersThe strong negative affinity setting: The group pe2ers (ERS11, Replicated Enqueue Server) is never brought online nor allowed to remain online on a node on which the group pe2scs (SCS01, including Enqueue Server) is online.

4.10 - Enable and test the cluster

This chapter does not intend to describe a complete cluster acceptance test. Just the minimum functional test is shown here.

Enable the resourcesEnable the cluster resources, configured in the step before:

# clresource enable pe2ora_lsnr# clresource enable pe2ora_db# clresource enable pe2scs_enq# clresource enable pe2scs_ms# clresource enable pe2ci_jc00# clresource enable pe2ers_enq

SwitchoverTest if every resourcegroup can be switched over to every node.

# clrg switch n <targetnode> <resourcegroup>with <target-node> = sscsapx1 or sscsapx2with <resourcegroup> = pe2scs, pe2ers, pe2ora or pe2ci

Check if the evacuation of the Replicated Enqueue Server is working, as specified in the section “Group affinities” above.

SAP enqueue replicationTest the enqueue replication again, as already described in the chapter “Test the enqueue replication”.Verify that JC00 (as enqueue client) is not doing a restart, after group pe2scs (SCS01, including Enqueue Server) has done an switchover or failover.

Check the enqueue clientsCheck, if the JAVA enqueue clients (all SAP application instances) will continue to work without a restart, after a SCS01 failover or switchover.

Page 16 of 21


5 - Appendix

5.1 - Default profile

File “/sapmnt/PE2/profile/DEFAULT.PFL”

SAPSYSTEMNAME = PE2SAPGLOBALHOST = pe2scslh## SAP Central Service Instance for J2EE#j2ee/scs/host = pe2scslhj2ee/scs/system = 01j2ee/ms/port = 3901SAPDBHOST = pe2oralhj2ee/dbtype = oraj2ee/dbname = PE2j2ee/dbhost = pe2oralhDIR_PUT = /usr/sap/$(SAPSYSTEMNAME)/put

# Enable enqueue replication for all clients (= app instances)enque/deque_wait_answer = TRUE

5.2 - ERS11 start profile

File “/sapmnt/PE2/profile/START_ERS11_pe2erslh”

SAPSYSTEMNAME = PE2

# See comments in the ERS11 instance profile.SAPSYSTEM = 01

INSTANCE_NAME = ERS11

# See comments in the ERS11 instance profile.DIR_CT_RUN = /usr/sap/PE2/SYS/exe/runDIR_EXECUTABLE = /usr/sap/PE2/SYS/exe/run

SETENV_00 = PATH=$(DIR_INSTANCE)/exe:%(PATH)SETENV_01 = LD_LIBRARY_PATH=$(DIR_EXECUTABLE)_PF = $(DIR_PROFILE)/PE2_ERS11_pe2erslh

## Disabled! See comments in the ERS11 instance profile.## _CPARG0 = list:$(DIR_EXECUTABLE)/ers.lst# Execute_00 = immediate $(DIR_EXECUTABLE)/sapcpe$(FT_EXE) $(_CPARG0) pf=$(_PF)# OS_UNICODE = uc

## Start Replicated Enqueue Server#_ER = er.sap$(SAPSYSTEMNAME)_$(INSTANCE_NAME)Execute_01 = immediate rm f $(_ER)Execute_02 = local ln s f $(DIR_EXECUTABLE)/enrepserver $(_ER)

# Not starting it in the SAPdescribed way.# See comments in the ERS11 instance profile.# Restart_Program_00 = local $(_ER) pf=$(_PF) NR=01Restart_Program_00 = local $(_ER) pf=$(_PF)

5.3 - ERS11 instance profile

File “/sapmnt/PE2/profile/PE2_ERS11_pe2erslh”

SAPSYSTEMNAME = PE2

Page 17 of 21


# !!ATTENTION!!# Even this instance was installed on #11 and instance name is ERS11,# you have to set the internal number to 01. That is the instance number of SCS01.# This assures that both instances (SAP Enqueue Server AND SAP Replicated Enqueue Server)# are using the same internal memory representation for the LOCK table. Thus being able to# synchronize in the case of a failover (otherwise the synchronisation will fail!)# The SAP documentation is describing a different way, which comes obviously to the same# result. It starts the process within the start profile with this parameterisation:# “Restart_Program_00 = local $(_ER) pf=$(_PF) NR=01” (note the ending NR=01).# But as the cluster starts the Replicated Enqueue process directly (without using the# start profile), we have to set this parameter in the instance profile.SAPSYSTEM = 01

INSTANCE_NAME = ERS11

# Start directly from /sapmnt/PE2/exe (the dirs specified here are softlinks to /sapmnt/PE2/exe).# No “sapcpe” to instancespecific dirs make sense, because it's all on the global filesystem(s).DIR_CT_RUN = /usr/sap/PE2/SYS/exe/runDIR_EXECUTABLE = /usr/sap/PE2/SYS/exe/run

SAPLOCALHOST = pe2erslhOS_UNICODE = uc

# Infos about SAP Enqueue Server (runnning in SCS01)rdisp/enqname = $(rdisp/myname)enque/process_location = REMOTESAenque/serverinst = 01enque/serverhost = pe2scslh

# General instance parameteripc/shm_psize_10 = 130000000ipc/shm_psize_40 = 115000000

5.4 - SCS01 start profile

File “/sapmnt/PE2/profile/START_SCS01_pe2scslh”

SAPSYSTEMNAME = PE2SAPSYSTEM = 01INSTANCE_NAME = SCS01# DIR_CT_RUN = $(DIR_EXE_ROOT)/run# DIR_EXECUTABLE = $(DIR_INSTANCE)/exeDIR_CT_RUN = /usr/sap/PE2/SYS/exe/runDIR_EXECUTABLE = /usr/sap/PE2/SYS/exe/runSAPLOCALHOST = pe2scslhDIR_PROFILE = $(DIR_INSTALL)/profile_PF = $(DIR_PROFILE)/PE2_SCS01_pe2scslhSETENV_00 = LD_LIBRARY_PATH=$(DIR_LIBRARY):%(LD_LIBRARY_PATH)SETENV_01 = SHLIB_PATH=$(DIR_LIBRARY):%(SHLIB_PATH)SETENV_02 = LIBPATH=$(DIR_LIBRARY):%(LIBPATH)## Copy SAP Executables !!! Disabled !!!## _CPARG0 = list:$(DIR_CT_RUN)/scs.lst# Execute_00 = immediate $(DIR_CT_RUN)/sapcpe$(FT_EXE) pf=$(_PF) $(_CPARG0)# OS_UNICODE = uc## Start SAP messaging service#_MS = ms.sap$(SAPSYSTEMNAME)_$(INSTANCE_NAME)Execute_01 = local rm f $(_MS)Execute_02 = local ln s f $(DIR_EXECUTABLE)/msg_server$(FT_EXE) $(_MS)Start_Program_00 = local $(_MS) pf=$(DIR_PROFILE)/PE2_SCS01_pe2scslh## Start SAP locking service# !!! Disabled !!! Will be started directly by Sun Cluster## _EN = en.sap$(SAPSYSTEMNAME)_$(INSTANCE_NAME)# Execute_03 = local rm f $(_EN)# Execute_04 = local ln s f $(DIR_EXECUTABLE)/enserver$(FT_EXE) $(_EN)# Start_Program_01 = local $(_EN) pf=$(DIR_PROFILE)/PE2_SCS01_pe2scslh

Page 18 of 21


5.5 - SCS01 instance profile

File “/sapmnt/PE2/profile/PE2_SCS01_pe2scslh”

SAPSYSTEMNAME = PE2SAPSYSTEM = 01INSTANCE_NAME = SCS01# DIR_CT_RUN = $(DIR_EXE_ROOT)/run# DIR_EXECUTABLE = $(DIR_INSTANCE)/exeDIR_CT_RUN = /usr/sap/PE2/SYS/exe/runDIR_EXECUTABLE = /usr/sap/PE2/SYS/exe/runSAPLOCALHOST = pe2scslhOS_UNICODE = uc## SAP Messaging Service for Java#rdisp/msserv = 0rdisp/msserv_internal = 3901ms/standalone = 1ms/server_port_0 = PROT=HTTP,PORT=81$$## SAP Locking Service#enque/serverinst = 01enque/table_size = 4096rdisp/enqname = $(rdisp/myname)enque/snapshot_pck_ids = 100enque/server/replication = true

ipc/shm_psize_34 = 0ipc/shm_psize_10 = 130000000ipc/shm_psize_40 = 115000000

5.6 - JC00 start profile

File “/sapmnt/PE2/profile/START_JC00_pe2cilh”

SAPSYSTEMNAME = PE2SAPSYSTEM = 00INSTANCE_NAME = JC00# DIR_CT_RUN = $(DIR_EXE_ROOT)/run# DIR_EXECUTABLE = $(DIR_INSTANCE)/exeDIR_CT_RUN = /usr/sap/PE2/SYS/exe/runDIR_EXECUTABLE = /usr/sap/PE2/SYS/exe/runSAPLOCALHOST = pe2cilhDIR_PROFILE = $(DIR_INSTALL)/profile_PF = $(DIR_PROFILE)/PE2_JC00_pe2cilhSETENV_00 = LD_LIBRARY_PATH=$(DIR_LIBRARY):%(LD_LIBRARY_PATH)SETENV_01 = SHLIB_PATH=$(DIR_LIBRARY):%(SHLIB_PATH)SETENV_02 = LIBPATH=$(DIR_LIBRARY):%(LIBPATH)## Copy SAP Executables !!! Disabled !!!## _CPARG0 = list:$(DIR_CT_RUN)/j2eeinst.lst# Execute_00 = immediate $(DIR_CT_RUN)/sapcpe$(FT_EXE) pf=$(_PF) $(_CPARG0)## Copy SAP Executables## _CPARG1 = list:$(DIR_CT_RUN)/igsexe.lst# Execute_01 = immediate $(DIR_CT_RUN)/sapcpe$(FT_EXE) pf=$(_PF) $(_CPARG1)## Start Java application server#_JC = jc.sap$(SAPSYSTEMNAME)_$(INSTANCE_NAME)Execute_02 = local rm f $(_JC)Execute_03 = local ln s f $(DIR_EXECUTABLE)/jcontrol$(FT_EXE) $(_JC)Start_Program_00 = local $(_JC) pf=$(_PF)## Start internet graphics server#_IG = ig.sap$(SAPSYSTEMNAME)_$(INSTANCE_NAME)Execute_04 = local rm f $(_IG)

Page 19 of 21


Execute_05 = local ln s f $(DIR_EXECUTABLE)/igswd_mt $(_IG)Start_Program_01 = local $(_IG) mode=profile pf=$(_PF)

5.7 - JC00 instance profile

File “/sapmnt/PE2/profile/PE2_JC00_pe2cilh”

SAPSYSTEMNAME = PE2SAPSYSTEM = 00INSTANCE_NAME = JC00# DIR_CT_RUN = $(DIR_EXE_ROOT)/run# DIR_EXECUTABLE = $(DIR_INSTANCE)/exeDIR_CT_RUN = /usr/sap/PE2/SYS/exe/runDIR_EXECUTABLE = /usr/sap/PE2/SYS/exe/runSAPLOCALHOST = pe2cilhjstartup/trimming_properties = offjstartup/protocol = onjstartup/vm/home = /usr/sap/PE2/j2sejstartup/max_caches = 500jstartup/release = 700jstartup/instance_properties = $(jstartup/j2ee_properties):$(jstartup/sdm_properties)j2ee/dbdriver = /oracle/client/10x_64/instantclient/ojdbc14.jarigs/listener/rfc/disable = 1PHYS_MEMSIZE = 512exe/saposcol = $(DIR_CT_RUN)/saposcol

## Jcontrol: Migrated Profile Parameter# create at Thu Jun 12 15:06:30 2008#j2ee/instance_id = ID0016265#

5.8 - Script startsap_cluster.sh

File “/usr/sap/PE2/pe2adm/startsap_cluster.sh”

#!/bin/sh

###################### please customize .... #######################HOME=`getent passwd pe2adm | cut d: f 6 s`. $HOME/.profile

INSTANCE=$2

LOGICAL_HOST=ùname nìf [ "$INSTANCE" = "SCS01" ]; then LOGICAL_HOST=pe2scslh;fiif [ "$INSTANCE" = "JC00" ]; then LOGICAL_HOST=pe2cilh;fi

SAPSID=PE2STARTSAP="/usr/sap/${SAPSID}/SYS/exe/run/startsap"####################### end of customizing .... #######################

SCRIPTNAME=`basename $0 `LOG_FILE=/tmp/${SCRIPTNAME}_${INSTANCE}.log

exec >> $LOG_FILEecho "================================================================================"echo "`date`: Script $SCRIPTNAME started with options $* ... "

GREP=/usr/bin/grepNEWTASK=/usr/bin/newtaskPS=/usr/bin/psSLEEP=/usr/bin/sleepINSTID=ècho $INSTANCE | sed 's/^[AZaz]*//g'`

# start $INSTANCE ....

Page 20 of 21


echo " execute $STARTSAP $INSTANCE $LOGICAL_HOST "$NEWTASK $STARTSAP $INSTANCE $LOGICAL_HOST

5.9 - Script stopsap_cluster.sh

File “/usr/sap/PE2/pe2adm/stopsap_cluster.sh”

#!/bin/sh

###################### please customize .... #######################HOME=`getent passwd pe2adm | cut d: f 6 s`. $HOME/.profile

INSTANCE=$2;

LOGICAL_HOST=ùname nìf [ "$INSTANCE" = "SCS01" ]; then LOGICAL_HOST=pe2scslh;fiif [ "$INSTANCE" = "JC00" ]; then LOGICAL_HOST=pe2cilh;fi

SAPSID=PE2STOPSAP="/usr/sap/${SAPSID}/SYS/exe/run/stopsap"SAPCCMSR="/usr/sap/${SAPSID}/SYS/exe/run/sapccmsr"####################### end of customizing .... #######################

SCRIPTNAME=`basename $0 `LOG_FILE=/tmp/${SCRIPTNAME}_${INSTANCE}.log

exec >> $LOG_FILEecho "================================================================================"echo "`date`: Script $SCRIPTNAME mit Optionen $* gestartet ... "

###################### stop ccmsr ######################echo " execute ${SAPCCMSR} stop j2ee pf=/usr/sap/${SAPSID}/SYS/profile/${SAPSID}_${INSTANCE}_${LOGICAL_HOST} "${SAPCCMSR} stop j2ee pf=/usr/sap/${SAPSID}/SYS/profile/${SAPSID}_${INSTANCE}_${LOGICAL_HOST}

###################### stop $INSTANCE .... ######################echo " execute $STOPSAP startupsrv $INSTANCE2 $LOGICAL_HOST "$STOPSAP startupsrv $INSTANCE $LOGICAL_HOST

echo " execute $STOPSAP $INSTANCE $LOGICAL_HOST "$STOPSAP $INSTANCE $LOGICAL_HOST

# IGS needs some time for stoppin all precesses, so wait ...if [ "`pgrep f /usr/sap/${SAPSID}/${INSTANCE}/exe/igs`" != "" ]then/usr/bin/sleep 30fi

###################### stop jlaunch processes ######################INSTANCEtemp=ècho ${INSTANCE} | awk '/^D[09][09]$/{print $1}'ìf [ "$INSTANCEtemp" ]; then echo " killing jlaunch processes " /usr/bin/pkill f /usr/sap/${SAPSID}/${INSTANCE}/exe/jlaunchfi

###################### cleanup IPC ######################instance_nr=ècho $INSTANCE | sed 's/[AZ]*//'ècho " /usr/sap/${SAPSID}/SYS/exe/run/cleanipc $instance_nr remove"/usr/sap/${SAPSID}/SYS/exe/run/cleanipc "$instance_nr" remove

###################### stop saposcol ####################### Note: Not really correct. saposcol is not instance specific# Best would be to start it outside of SAP or Cluster control as system service (SMF)echo " execute /usr/sap/${SAPSID}/SYS/exe/run/saposcol k"/usr/sap/${SAPSID}/SYS/exe/run/saposcol k

echo "`date`: Script $SCRIPTNAME finished

Page 21 of 21

HA With Sap en Queue Replication and Sun Cluster V02

Documents

scriptnamebasename0

sun microsystems

os user pe1adm

log execgt

standalone

native sap

sap central

replicated