Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global Mirror: Solution installation and configuration
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
7/31/2019 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global…
About high availability .............................................................................................................................. 1
About disaster recovery ........................................................................................................................... 1
About IBM ................................................................................................................................................ 2
About Symantec ....................................................................................................................................... 2
About Veritas Storage Foundation HA..................................................................................................... 2
About IBM Storwize V7000 system ......................................................................................................... 2
Setting up the Storwize V7000 remote copy partnership ...................................................................... 11
Creating Storwize V7000 Metro Mirror, Global Mirror, and FlashCopy consistency groups ................. 12
Creating Storwize V7000 Metro Mirror, Storwize V7000 Global Mirror relationship, and FlashCopymappings ............................................................................................................................................... 13
Installing Veritas Storage Foundation .................................................................................. 14
Disk space ............................................................................................................................................. 15
Environment variables ........................................................................................................................... 15Virtual IP address ................................................................................................................................... 16
Prerequisites for local and remote cluster installation ........................................................................... 16
Mounting a software disk ....................................................................................................................... 16
Installing SFHA 5.1 SP1PR1 using the Veritas product installers ......................................................... 17
Installing Veritas Storage Foundation HA using webinstaller interface ................................................. 20
Adding the VCS SAN Volume Controller copy services resource ......................................................... 26
Before you configure the agent for SVCCopyServices .................................................................. 26
Adding the agent resource type ..................................................................................................... 27
SVCCopyServices resource type definition ................................................................................... 27
Attribute definitions for the SVCCopyServices agent ..................................................................... 28
Required attributes ......................................................................................................................... 28Setting up Fire Drill ................................................................................................................................ 29
About Fire Drill resource................................................................................................................. 29
SVCCopyServicesSnap resource type definition ........................................................................... 30
Attribute definitions for the SVCCopyServicesSnap agent ............................................................ 30
This paper describes how Symantec and IBM have installed, configured, and validated high availability (HA) and disaster recovery (DR) configurations for Oracle with IBM Storwize V7000 systems. These validations include local HA configurations using Veritas Storage Foundation and Veritas Cluster Server (VCS). The configuration was extended to a DR configuration using
IBM Storwize V7000 Metro Mirror for synchronous replication and Storwize V7000 Global Mirror for asynchronous replication using the VCS agent for IBM SVCCopyServices and VCS Global Cluster Option (GCO) for alternate site failover and failback capability.
Introduction
Infrastructure for mission-critical applications must be able to meet the organization's recovery time
objective (RTO) and recovery point objective (RPO) for resuming operation in the event of a site disaster.
This solution addresses environments where the RPOs and RTOs are in the range of minutes to a few
hours. While backup is the foundation for any DR plan, typical RTOs for tape-based backup are well
beyond these objectives. Also, replication alone is not enough as having the application data at a DR site
is of limited use without also having the ability to start the correct sequence of database management
systems, application servers, and business applications.
Symantec‟s DR solutions, Metro Clustering and Global Clustering, are extensions of local HA clustering
using Veritas Storage Foundation and Veritas Cluster Server. This validated and documented solution is
an example of Global Clustering, which is a collection of two or more VCS clusters at separate locations
linked together with VCS Global Cluster option to enable wide-area failover and disaster recovery. Each
local cluster within the global cluster is connected to its own shared storage. Local clustering provides
local failover for each site. IBM® Storwize® V7000 storage system Metro Mirror replicates data between
sites to maintain synchronized copies of storage at the two sites. For a disaster that affects an entire site,
the customer makes a decision on whether or not to move operations to the disaster recovery site. When
that decision to move the operations is made, the application is automatically migrated to a system at the
DR site. IBM Storwize V7000 Global Mirror replicates data asynchronously between sites and appliesrecovery at the DR sites.
The term high availability or HA refers to a state where data and applications are highly available because
software or hardware is in place to maintain the continued functioning in the event of computer failure. HA
can refer to any software or hardware that provides fault tolerance, but generally the term has become
associated with clustering. Local clustering provides HA through database and application failover.
Veritas Storage Foundation Enterprise HA (SF/HA) includes Veritas Storage Foundation and Veritas
Cluster Server and provides the capability for local clustering. The Storwize V7000 disk system includes a
wide range of HA features as well.
Wide-area disaster recovery provides the ultimate protection for data and applications in the event of a
disaster. With an appropriate disaster recovery solution in place, if a disaster affects a local or
metropolitan area, data and critical services can be failed over to a site hundreds or even thousands of
miles away. IBM Storwize V7000 Metro Mirror and Global Mirror, combined with Veritas Storage
Foundation Enterprise HA/DR provide the capability for implementing disaster recovery.
7/31/2019 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global…
IBM is one of the world's largest information technology company, with over 80 years of leadership in
helping businesses innovate by delivering a wide range of solutions and technologies that enable
customers, large and small, to deliver more-efficient and effective services. IBM's comprehensive server,
storage, software, and services portfolio are designed to help you create new business insight by
integrating, analyzing, and optimizing information on demand. From its foundations of virtualization,
openness, and innovation through collaboration, IBM can optimize management of information through
technology innovations and infrastructure simplification to help achieve maximum business productivity.
You can visit IBM at ibm.com.
Symantec is a global leader in infrastructure software, enabling businesses and consumers to have
confidence in a connected world. The company helps customers protect their infrastructure, information,
and interactions by delivering software and services that address risks to security, availability,
compliance, and performance. Headquartered in Cupertino, California, Symantec has operations in more
than 40 countries. You can visit Symantec at symantec.com
Veritas Storage Foundation HA is a comprehensive solution that delivers data and application availability
by bringing together two industry-leading products: Veritas Storage Foundation and Veritas Cluster
Server.
Veritas Storage Foundation provides a complete solution for heterogeneous online storage management
Based on the industry-leading Veritas Volume Manager and Veritas File System, it provides a standard
set of integrated tools to centrally manage explosive data growth, maximize storage hardware
investments, provide data protection, and adapt to changing business requirements. Unlike point
solutions, Storage Foundation enables IT organizations to manage their storage infrastructure with one
tool. With advanced features, such as centralized storage management, nondisruptive configuration and
administration, dynamic storage tiering, dynamic multipathing, data migration, and local and remote
replication, Storage Foundation enables organizations to reduce operational costs and capital
expenditures across the data center.
Veritas Cluster Server is an industry leading clustering solution for reducing both planned and unplanned
downtime. By monitoring the status of applications and automatically moving them to another server in
the event of a fault, Veritas Cluster Server can dramatically increase the availability of an application or
database. Veritas Cluster Server can detect faults in an application and all its dependent components,
including the associated database, operating system, network, and storage resources. When a failure is
detected, Veritas Cluster Server gracefully shuts down the application, restarts it on an available server,
connects it to the appropriate storage device, and resumes normal operations. Veritas Cluster Server cantemporarily move applications to a standby server when routine maintenance, such as upgrades or
patches, requires that the primary server be taken offline.
IBM Storwize V7000 system is a large midrange storage platform, which in addition to new functionality,
leverages proven virtualization technologies available previously in other popular IBM storage products.
7/31/2019 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global…
Table 3 lists the product documentation set that is required for installation, configuration, and
troubleshooting the setup.
Product Use this guide
Storwize V7000 Storwize V7000 Infocenter
http://publib.boulder.ibm.com/infocenter/storwize/ic/index.jspVeritas StorageFoundationEnterprise HA
Veritas Storage Foundation and High Availability Installation Guidehttps://sort.symantec.com/public/documents/sfha/5.1sp1pr1/aix/productguides/pdf/sf_install_51sp1pr1_aix.pdf
Veritas Cluster Server
Veritas Cluster Server Installation Guidehttps://sort.symantec.com/public/documents/sfha/5.1sp1pr1/aix/productguides/pdf/vcs_install_51sp1pr1_aix.pdf Veritas Cluster Server Administrator‟s Guidehttps://sort.symantec.com/public/documents/sfha/5.1sp1pr1/aix/productguides/pdf/vcs_admin_51sp1pr1_aix.pdf
Veritas Cluster Server Agent for Oracle Installation and Configuration Guidehttps://sort.symantec.com/public/documents/sfha/5.1sp1pr1/aix/productguides/pdf/vcs_oracle_agent_51sp1pr1_aix.pdf Veritas Cluster Server Agent for IBM SVCCopyServices Installation andConfiguration Guide
Included here are some general guidelines for configuration of the Storwize V7000 in preparation for the
Storage Foundation for High Availability installation.
For more details on zoning requirements for Storwize V7000, review the sections on Zoning details and
Zoning examples available in the Storwize V7000 Information Center at
http://publib.boulder.ibm.com/infocenter/storwize/ic/index.jsp. Table 4 shows the zones created for this
configuration.
ZoneZone configuration – Fabric 1 and 2 (SAN01, SAN03 /
SAN02, SAN04)
ISV7K4 includes aliases for two Storwize V7000 ports and a port each
from ISVP12 / ISVP13
ISV7KD10includes aliases for two Storwize V7000 ports and a port eachfrom ISVP14 / ISVP15
ISV7KD10_ISV7K4_Mirror includes one Storwize V7000 port from each disk system toallow formation of intercluster link, for remote copy (GlobalMirror and Metro Mirror)
Table 4: Fabric zoning
7/31/2019 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global…
This section assumes that the Storwize V7000 storage subsystems are up and running. Provided you
have configured the Secure Shell (SSH) client on each participating node in your cluster, you can
SSH/login to the Storwize V7000 CLI from any VCS cluster node or you can connect to the StorwizeV7000 integrated GUI console using the IP address of the controller canister. You can find detailed
instruction on configuring SSH between AIX hosts and the Storwize V7000 disk system in “ Appendix A:
OpenSSH client configuration between AIX and Storwize V7000”.
The method shown in this section demonstrates the Storwize V7000 CLI interface that is used to
configure volumes from managed disks (MDisks) available to the Storwize V7000 storage system. With
Storwize V7000, MDisks can be either LUNS from an external SAN-attached storage device or
alternatively (as in this case) from SAS drives or SSDs available inside the Storwize V7000 disk system.
In this configuration, the storage is Storwize V7000 physical disk. The volumes were created from a
storage pool containing seven MDisks, each of which represents an individual RAID5 array spanning 7 or
8 x 300 GB SAS drives – delivering a usable capacity of 1.6 TB or 1.9 TB, respectively.
Typical disk configuration consists of initial disk discovery (in the case of externally-attached SAN disks),
or configuration of existing physical disks in the Storwize V7000 disk enclosures into MDisks/arrays of a
particular RAID type, then adding these to one or more storage pools. Volumes are then created from the
available capacity in the storage pools and finally are mapped to the hosts.
The volume and host mapping required for this setup is shown in this section. Table 5 shows the volumes
configured for Metro and Global Mirror used in this configuration. The following commands given provide
an example of creating storage pools (formerly manged disk groups), volumes, and mapping volumes to
hosts.
Creating a storage pool (formerly known as managed disk group):
This section provides a walk through of the installation prerequisites, as well as detailed command lineoutput resulting from the installation of our first SFHA cluster.
Symantec SFHA licensing is typically done during product installation. During the installation process,
both of the Symantec installer options provide opportunity to enter the product license key.
The VRTSvlic package enables product licensing. After VRTSvlic is installed, the following commands
and their manual pages are available on the system:
vxlicinst installs a license key for a Symantec product
vxlicrep displays currently installed licenses vxlictest retrieves features and their descriptions encoded in a license key
You need to make sure that you have activated the Veritas Storage Foundation Enterprise HA/DR
AIX 5.1 license key.
7/31/2019 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global…
This release of Veritas Storage Foundation (5.1, SP1, PR1) operates on:
IBM POWER7 processor running
AIX 7.1 TL0 or higher
AIX 6.1 TL5 with SP1 or higher
AIX 5.3 in POWER6 or IBM Power6+™ compatibility at TL11, with Service Pack 2 or higher, or
TL 10 with Service Pack 4 or later.
IBM POWER6® processors, at one of the following levels:
AIX 7.1, TL0 or higher
AIX 6.1, TL2 or higher
AIX 5.3 at one of the following levels:
− TL7 with Service Pack 6 or higher
− TL8 with Service Pack 4 or higher
Both of the Symantec product installer interfaces verify the required update levels. The installation
process terminates if the target systems do not meet maintenance-level requirements.
For any Veritas cluster product, all nodes in the cluster need to have the same operating system version
and update level.
The Oracle database version supported by Veritas high availability release 5.1SP1, PR1 is Oracle 11gR2,but generally the VCS agent supports a particular Oracle release as long as Oracle already supports the
release on that particular AIX level.
Use the Perform a Preinstallation Check (P) option from the product installer to determine whether
there is sufficient space.
Most of the commands used in the installation are in the /sbin or /usr/sbin directory. However, there are
additional variables that are required to use a Veritas Storage Foundation product after installation.
7/31/2019 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global…
Note: Veritas products are installed under the /opt directory on the specified host systems. Ensure that
the directory /opt exists and has write permissions for root before starting an installation procedure.
The Veritas product installer is the recommended method to license and install the product. There are
command-line-driven and web-based versions of the product installer available. Both the versions enable
you to verify preinstallation requirements, configure the product, and view the product‟s description.
You can use the product installer to install Veritas Storage Foundation and Veritas Storage FoundationEnterprise HA. Usually, during an installation, you can type b ( back ) to return to a previous section of the
installation procedure. The back feature of the installation scripts is context-sensitive, so it returns to the
beginning of a grouped section of questions. If an installation procedure hangs, press Ctrl+C to stop and
exit the program. There is a short delay before the script exits.
To install a Storage Foundation product, run the following steps from one node in each cluster.
1. If the installation file sets are on a DVD media, make sure the disk is mounted. Refer to
the “Mounting a software disk” section. This installation scenario uses downloaded
compressed tarfiles that have been unpacked into the /tmp directory on one of the cluster
nodes.
2. To invoke the common installer, run the installer command on the disk as shown in this example:# /tmp/installer (to invoke the menu-based installation interface)
or, alternatively
# /tmp/webinstaller (to invoke the web-based installation interface)
3. Enter I to install a product and press Enter to begin.
4. When the list of available products is displayed, select the product you want to install and enter the
corresponding number and press Enter. The product installation begins automatically.
5. Enter the Storage Foundation Enterprise HA/DR product license information.
Enter a product_name license key for isvp_sfha_clusterA: [?] XXXX-XXXX-XXXX-XXXX-XXXX-
XXXX-X
XXXX-XXXX-XXXX-XXXX-XXXX-XXXX-X successfully registered on isvp_sfha_clusterA Do youwant to enter another license key for isvp_sfha_clusterA? [y,n,q,?] (n)Enter a product_name license key for isvp_sfha_clusterB: [?] XXXX-XXXX-XXXX-XXXX-XXXX-
XXXX-X XXXX-XXXX-XXXX-XXXX-XXXX-XXXX-X successfully registered on isvp_sfha_clusterBDo you want to enter another license key for isvp_sfha_clusterB? [y,n,q,?] (n)Enter n if you have no further license keys to add for a system.
You are then prompted to enter the keys for the next system.
7/31/2019 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global…
Note: Each system requires a product license before installation. License keys for additional product
features should also be added at this time.
6. Choose to install all file sets.
SF can be installed without optional filesets to conserve disk space. Additionalfilesets are typically installed to simplify future upgrades.1) Required Veritas Storage Foundation filesets - 928 MB required2) All Veritas Storage Foundation filesets - 1063 MB requiredSelect the filesets to be installed on all systems? [1-2,q,?] (2)
7. At the installed product list page, enter y or press Enter to configure the Storage Foundation and VCSproducts.
It is possible to install SF filesets without performing configuration.
It is optional to configure SF now. If you choose to configure SF later, you
can do so manually or run the installsf -configure command.
Are you ready to configure SF? [y,n,q] (y)
Do you want to configure VCS on these systems at this time? [y,n,q] (y)
8. The installer prompts for details for configuring the VCS cluster for Storage Foundation (SF) Enter theunique cluster name and cluster ID number.
Enter the unique cluster name: [?]isvp_sfha_clusterA
Enter the unique Cluster ID number between 0-65535: [b,?] 0
The installer discovers the NICs available on the first system and reports them,
Discovering NICs on isvp_sfha_clusterA...discovered en0 en1 en2
9. Enter private heartbeat NIC information for each host.
Enter the NIC for the first private heartbeat link on isvp12:[b,?] en1
Would you like to configure a second private heartbeat link?[y,n,q,b,?] (y) y
Enter the NIC for the second private heartbeat link on isvp12:[b,?] en2
Would you like to configure a third private heartbeat link?[y,n,q,b,?] (n) n
Do you want to configure an additional low priority heartbeat link? [y,n,q,b,?]
(n) n
Are you using the same NICs for private heartbeat links on all systems?
[y,n,q,b,?] (y) y
Note: When answering as y, be sure that the same NICs are available on each system; the installer does not verify this.
Notice that in this example, en0 is not selected for use as a private heartbeat NIC because it isalready in use as the public network interface.
10. A summary of the information you entered is given. When prompted, confirm that theinformation is correct.
Is this information correct? [y,n,q] (y)
7/31/2019 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global…
Installing VCS agent for IBM SAN Volume Controller copy
services
The VCS agent used to support remote copy services on IBM Storwize V7000 system is the same agentthat has been used in previous VCS configurations with IBM SAN Volume Controller. Despite the agent
name (IBM SVCCopyServices) the required resource attributes are applicable to both SAN Volume
Controller and Storwize V7000 disk systems.
For complete details refer to the Veritas Cluster Server Agent for IBM SVCCopyServices Installation and
You must install the IBM SAN Volume Controller copy services agent on each node in the cluster. In
global cluster environments, install the agent on each node in each cluster. These instructions assume
that the Veritas Cluster Server is already installed. Perform the following steps to install the agent.
1. Make sure the disk is mounted. See “Mounting a Software Disk”.
2. Run the following command to navigate to the location of the agent packages:
# cd /cdrom/aix/replication/svccopyservices_agent /version/pkgs The variable version represents the version of the agent. In this scenario, version 5.0.3.0 hasbeen installed.
3. Run the following command to add the file sets for the software.
# installp -ac -d VRTSvcssvc.rte.bff VRTSvcssvc
All the required software components have now been installed. You should be able to list out the filesetsthat are mentioned in the “ Appendix D: Veritas Software file sets listing” on each application host.
Installing and configuring Oracle
Installing and configuring Oracle involves the following tasks:
Installation of Oracle software
Creation of an Oracle instance
Creation of the database
First, you need to install Oracle on all nodes in both clusters. Make sure that the installation prerequisites
are met and are identical on all nodes, especially the user and group ID, passwords, owner and group
permissions and listener port ID. Refer to the appropriate “ Appendix B: Setting up the databaseapplications” section for instructions to setup the database.
In this configuration, a database representing testmm schema is built. A database workload application is
used to populate and simulate a Transaction Processing Performance Council OLTP (TPC-C) workload.
You will need a workload application to exercise the database load.
7/31/2019 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global…
1. Make sure that all of objects mentioned in VCS configuration file are created and available.
2. Halt the cluster server from any node in the clusters in Site A and Site B#/opt/VRTSvcs/hastop –all
3. Cut and paste the VCS configuration file (main.cf) from Appendix C: Sample main.cf file -
isvp_sfha_clusterA to /etc/VRTSvcs/conf/config directory as shown here.
On cluster nodes isvp12, isvp13 in Site A as:main.cf.siteA On cluster nodes isvp14, isvp15 in Site B as:main.cf.siteB
4. Modify the values of hostnames, IP addresses, mount points, volumes and disk group resources,cluster names, passwords, and so on to match your site specific configuration.
5. Run the following commands to copy the VCS agent type definition files, if they are not present inthe destination directory.#cp /etc/VRTSagents/ha/conf/Oracle/OracleTypes.cf /etc/VRTSvcs/conf/config/
6. Run the following commands to overwrite the existing main.cf file on the primary node on each of the respective clusters, then use remote shell or SSH to copy that file to its local partner.
For example main.cf on Site A cluster node 1 (isvp12)
#cd /etc/VRTSvcs/conf/config
7/31/2019 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global…
9. Start the cluster manager from any node in the cluster Site A. Login to one of the nodes with theadministrator userID as admin and password as password.
10. On the first node at Site A, login to the cluster manager and bring the app_grp1 service grouponline if it is not already online.
11. Now you are ready to manage the clusters from the cluster manager GUI. To test HA and DRscenarios, proceed with the instructions in the “Failover scenarios” section.
General configuration steps
In this section, the steps mentioned in the setup are explained in detail. Refer to the documents listed inTable 3 for additional details of configuration procedures. You will need the following guides:
Veritas Cluster Server User’s Guide
Veritas Cluster Server Bundled Agents Reference Guide Veritas Cluster Server Agent for Oracle Installation and Configuration Guide
Veritas Cluster Server Agent for IBM SVCCopyServices installation and Configuration Guide
Refer to “Setting up the Storwize V7000 remote copy partnership” section for more information.
For more details, refer to the following sections: “Creating Storwize V7000 Metro Mirror, Global Mirror,
and FlashCopy consistency groups” and “Creating Storwize V7000 Metro Mirror, Storwize V7000 Global
Mirror relationship, and FlashCopy mappings”.
7/31/2019 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global…
Refer to the section Linking clusters in the Veritas Cluster Server User’s Guide. The Remote Cluster
Configuration wizard provides an easy interface to link clusters. Before linking clusters, verify that the
virtual IP address for the ClusterAddress attribute for each cluster is set. Use the same IP address as the
one assigned to the IP resource in the ClusterService group. Have the following information ready: The
active host name or IP address of each cluster in the global configuration and of the cluster being added
to the configuration, the administrator login name and password for each cluster in the configuration.
The VCS cluster management GUI (Java™ Console) is no longer packaged in the latest 5.1 release of
VCS. If you attempt to issue the command used to manage previous installations, you will see the
following message:
isvp12> /opt/VRTSvcs/bin/hagui &[1] 13369510 isvp12> VCS Single Cluster Manager (Java Console) is no longer packaged with VCS.Symantec recommends use of the Veritas Operations Manager (VOM) to manage, monitor and report on multi-cluster environments. You can download VOM at
http://go.symantec.com/vom . If you wish to continue using the VCS Single Cluster Manager, you can get it at no charge at the http://go.symantec.com/vcsm_download website.
If you elect to download and use the VCS single-cluster manager, you can still click Edit Add/Delete
Remote Cluster or alternatively use the Veritas Operations Manager (VOM)
Configuring global cluster
From any node in the clusters in Site A and Site B, run the GCO Configuration wizard to create or update
the ClusterService group. The wizard verifies your configuration and validates it for a global cluster setup.
#/opt/VRTSvcs/bin/gcoconfig
The wizard discovers the NIC devices on the local system and prompts you to enter the device to be used
for the global cluster.
Specify the name of the device and press Enter. If you do not have NIC resources in your configuration,
the wizard asks you whether the specified NIC will be the public NIC used by all systems. Enter y if it is
the public NIC; otherwise enter n. If you entered n, the wizard prompts you to enter the names of NICs on
all systems.
Enter the virtual IP to be used for the global cluster, which you have already identified. If you do not have
IP resources in your configuration, the wizard prompts you for the netmask associated with the virtual IP.
The wizard detects the netmask; you can accept the suggested value or enter another value.
The wizard starts running the commands to create or update the ClusterService group. Various
messages indicate the status of these commands. After running these commands, the wizard brings the
ClusterService group online.
7/31/2019 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global…
The VCS service groups can be setup once the VCS agents have been installed. In this setup Oracle is
installed on all the cluster nodes.
Figure 3 shows the dependency graph of the VCS resources within each service group. There are two
service groups. The application service group test_shared_DG and the cluster service group which iscreated when you enable Global Cluster Option. The application group has been configured as a global
service group, which means it will be able to fail over to local and remote cluster nodes, if required. You
also need to observe that based on the resource dependency view, the Oracle database resource
depends on the mount, volume, diskgroup, and SVCCopyServices resources being available.
Refer to the Veritas Cluster Server Agent for Oracle Installation and Configuration Guide for adding and
configuring Oracle resource types.
Configure the application service group on the primary and secondary clusters. It involves the following
steps:
1. Start the VCS Single Cluster Manager (Java Console), or alternatively use the Veritas Operations
Manager (VOM) and log on to the cluster.
2. If the agent resource type (Oracle / SVCCopyServices ) is not added to your configuration, add it.
From the Cluster Manager File menu, click Import Types and select the appropriate file types for
the agent in the following directory path : /etc/VRTSagents/ha/conf/Oracle/OracleTypes.cf,
/etc/VRTSvcs/conf/SVCCopyServicesTypes.cf
3. Click Import.
4. Save the configuration.
5. Create the service groups. In this case, you can create the test_shared_DG service group. Refer
to chapter 4 of the Veritas Cluster Server Agent for Oracle Installation and Configuration guide tocreate the service group.
6. Add the resources mentioned in the test_shared_DG service group in main.cf for the respective
cluster.
7. Configure the service group as a global group using the Global Group Configuration wizard. Refe
to the Veritas Cluster Server User’s Guide for more information.
7/31/2019 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global…
Attribute definitions for the SVCCopyServices agent
Review the description of the agent attributes.
Required attributes
You need to assign values to the required attributes.
Attribute DescriptionGroupName Name of the replication relationship or consistency group that is
managed by the agent.Type-dimension string-scalar IsConsistencyGroup
Indicates whether the value specified in the GroupNameattribute is the name of a single replication relationship or of aconsistency group consisting of several replicationrelationships. Attribute value is either 0 or 1. Default is 1.
Type-dimension string-scalar SSHBinary Contains the absolute path to the SSH binary. SSH is the mode
of communication with the SAN Volume Controller cluster that isconnected to the node.Default is /usr/bin/ssh.
Type-dimension string-scalar SSHPathToID File Contains the absolute path to the identity file used for
authenticating the host with the SAN Volume Controller cluster.The corresponding public key must be uploaded on the SANVolume Controller cluster so that the SAN Volume Controller cluster can correctly authenticate the host.
SVCClusterIP Is the IP address of the SAN Volume Controller cluster in thedot notation. The agent uses this IP address to communicatewith the SAN Volume Controller cluster.
Type-dimension string-scalar SVCUserName Is the user name that authenticates the SSH connection with
the SAN Volume Controller cluster.Default is admin.
StopTakeover Determines whether the agent makes read-write accessavailable to the host when the replication is in a stopped state(that is consistent_stopped).The status of the replication goes into a stopped state when theuser fires the stoprcrelationship or the stoprcconsistgrp command. Thus, no replication occurs between the primary andsecondary SAN Volume Controller clusters. Attribute value is either 0 or 1. Default value is 0. If it is set to 1,there is a possibility for data loss if after the replication wasstopped, the application continues to write data on the primarycluster. Thus, when the agent enables read-write access on the
secondary SAN Volume Controller cluster, the secondary SANVolume Controller cluster does not have up-to-date data on it.The possible stopped states are: inconsistent_stopped andconsistent_stoppedWhen the state of the replication is consistent_stopped andStopTakeover = 1, the agent enables read-write access for theSAN Volume Controller cluster.When the state of the replication is inconsistent_stopped, the
7/31/2019 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global…
agent does not enable read-write access for the SAN VolumeController cluster.
Type-dimension string-scalar DisconnectTakeover
Determines whether the agent makes read-write accessavailable to the host when the replication is in a disconnectedstate (that is consistent_disconnected).
The status of the replication goes into a disconnected statewhen the primary and secondary SAN Volume Controller clusters lose communication with each other. Thus, noreplication occurs between the primary and secondarySAN Volume Controller clusters. Attribute value is either 0 or 1. Default is 0.The possible disconnected states are:
idling_disconnected
inconsistent_disconnected consistent_disconnected
When the state of the replication is consistent_disconnectedand DisconnectTakeover = 1, the agent enables read-writeaccess for the SAN Volume Controller cluster. When the stateof the replication is idling_disconnected, the agent does notenable read-write access for the SAN Volume Controller cluster.
Bringing the Fire Drill service group online demonstrates the ability of the application service group to
come online at the remote site when a failover occurs.
The SVCCopyServicesSnap agent is the Fire Drill agent for IBM SVCCopyServices. It is included in the
VCS agent for IBM SVCCopyServices agent package. The agent manages the replication relationshipbetween the source and target arrays when running a Fire Drill. You can configure the
SVCCopyServicesSnap resource in the Fire Drill service group, in place of the SVCCopyServices
resource. VCS supports the Gold, Silver and Bronze Fire Drill configurations for the agent. Refer to the
Veritas Cluster Server Agent for IBM SVCCopyServices Installation and Configuration Guide mentioned
in Table 3 for more details.
SVCCopyServicesSnap resource type definition
The IBM SVCCopyServicesSnap agent is represented by the SVCCopyServicesSnap type in VCS.
type SVCCopyServicesSnap (static int MonitorInterval = 300static int OpenTimeout = 180
static int NumThreads = 1static int OfflineMonitorInterval = 0static int OnlineTimeout = 6000static int RestartLimit = 1static str ArgList[] = { TargetResName, MountSnapshot, UseSnapshot,RequireSnapshot, FCMapGroupName }str TargetResNameint MountSnapshotint UseSnapshotint RequireSnapshotstr FCMapGroupNametemp str Responsibility)
Attribute definitions for the SVCCopyServicesSnap agent
Review the description of the agent attributes. You can find additional details on the
SVCCopyServicesSnap agent in the Veritas Cluster Server Agent for IBM SVCCopyServices Installation
and Configuration Guide, mentioned in Table 3.
Required attributes
You must assign values to required attributes.
Attribute DescriptionTargetResName Name of the resource managing the LUNs that you want to take
a snapshot of. Set this attribute to the name of theSVCCopyServices resource if you want to take a snapshot of replicated data. Set this attribute to the name of the DiskGroupresource if the data is not replicated.For example, in a typical Oracle setup, you might replicate datafiles and redo logs, but you might choose to avoid replicatingtemporary tablespaces. The temporary tablespace must stillexist at the DR site and might be part of its own disk group.
Type-dimension string-scalar
7/31/2019 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global…
UseSnapshot Specifies whether the SVCCopyServicesSnap resource takes alocal snapshot of the target array. Set this attribute to 1 for Goldand Silver configurations. For Bronze, set this attribute to 0.
Type-dimension integer-scalar RequireSnapshot Specifies whether the SVCCopyServicesSnap resource must
take a snapshot before coming online. Set this attribute to 1 if
you want the resource to come online only after it succeeds intaking a snapshot. Set this attribute to 0 if you do want theresource to come online even if it fails to take a snapshot.Setting this attribute to 0 creates the Bronze configuration.Note: Set this attribute to 1 only if UseSnapshot is set to 1
Type-dimension integer-scalar MountSnapshot Specifies whether the resource uses the snapshot to bring the
service group online. Set this attribute to 1 for Goldconfiguration. For Silver and Bronze configurations, set theattribute to 0.Note: Set this attribute to 1 only if UseSnapshot is set to 1.
Type-dimension integer-scalar
ResponsibilityDo not modify. For internal use only. Used by the agent to keeptrack of desynchronizing snapshots.
Type-dimension temporary stringFCMapGroupName Name of the FlashCopy mapping or FlashCopy consistency
group. If the target SVCCopyServices resource contains aconsistency group, set FCMapGroupName to a FlashCopyconsistency group. If the target SVCCopyServices resourcecontains a relationship, set FCMapGroupName to a FlashCopymapping. This attribute is optional for Bronze configurations.
While the application is running, from the Storwize V7000 console, ensure that the Metro Mirror
consistency group is in the consistent_synchronized state and the primary is the master cluster
(isvp_sfha_clusterA).
In this scenario, a node in the cluster at Site A where the application is online is lost. The application fails
over to the second node in the cluster. Next, that node is also lost. As all the nodes in the cluster are
down, the application fails over to a cluster node at Site B.
To test the host failure across nodes and clusters, perform the following steps:
1. On the switch, disable a host port associated with the of cluster node at site A, where the
application is online. This action introduces a fault. The service group fails over to the second
node of the cluster at Site A. You need to wait until the service group comes online on the second
node at Site A, and then verify that the DB application data is consistent. While the Application is
running, from the Storwize V7000 console or CLI interface ensure that the Metro Mirror
consistency group is in the consistent_synchronized state and the primary is the master cluster
(ISV7K4).
IBM_2076:ISV7K4:admin>svcinfo lsrcconsistgrp
2. Bring the Fire Drill service group at Site B offline, if it is online.
3. Disable the host port on the switch of the second cluster node at Site A.
4. A cluster down alert appears and gives the administrator an opportunity to fail over the service
group manually to one of the cluster nodes (for example, isvp14) at Site B.
5. Wait till the service group comes online on isvp14 and then verify that the DB application data is
consistent. While the application is running, from the IBM Storwize V7000 console, ensure thatthe Metro Mirror consistency group is in the consistent_synchronized state and the primary is the
auxiliary cluster (ISV7KD10).
6. Follow step 1 to fail over the application to isvp15 (the second node at Site B) and verify the
status.
7. Enable the switch ports of the two cluster nodes at Site A and ensure that the cluster comes up
and the service group is offline.
8. From the active node at Site B, move the service group to its original host (in this example,
isvp12). In the Service Groups tab of the Cluster Manager Configuration tree, right-click the
service group. Click Switch To and click isvp12 on which the service group was originally online.
9. The service group comes online on isvp_sfha_clusterA. From the Storwize V7000 console, verify
that the Storwize V7000 at the Site A becomes the primary again. Wait until the service group
comes online on isvp_sfha_clusterA, check that the DB application data is consistent.
7/31/2019 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global…
You need to test cluster failover in case of a disaster and failback after restoration of the cluster at the
disaster site (Site A). In this case, simulate a disaster by introducing fault to all the nodes in the cluster
and storage at Site A simultaneously.
To test the cluster failover and failback, perform the following steps:
1. Make sure that the application is up and running on one of the cluster nodes, for example, isvp12
in Site A.
2. Bring the Fire Drill service group to offline at Site B, if it is online.
3. Disable the host ports and storage switch ports of cluster Site A.
4. A cluster down alert appears and gives the administrator an opportunity to fail over the service
group manually to one of the cluster nodes (for example, isvp14) at Site B. The Storwize V7000
Metro Mirror consistency group lists primary as the auxiliary cluster (ISV7KD10). You can also
check the status from the VCS log at /var/VRTSvcs/log/engine_A.log.
5. Enable the disabled switch ports of cluster Site A. Ensure that the cluster comes up and the
ClusterService group goes online. The application service group is offline.
6. Manually resynchronize the data between the primary and secondary sites. If the current primary
is retained as the primary for the replication relationship and the replication is not in a stopped
state, stop the replication manually.
Use the stoprcrelationship or the stoprcconsistgrp command, without the -access option. Use
the startrcrelationship or the startrcconsistgrp command with the -force option on the
Storwize V7000 disk system that the user wants to retain as primary.
Alternatively, use the update action entry point with no arguments specified.If you change the primary for the replication relationship, stop the replication manually (if it is not
stopped already).
Enable read / write access on both Storwize V7000 disk systems by using the
stoprcrelationship or the stoprcconsistgrp command with the –access option.
Use the startrcrelationship or the startrcconsistgrp command with both the -force and
-primary options. Alternatively, use the update action entry point with exactly one argument that
is the Storwize V7000 disk system which the user retains as primary.
7. After the resynchronization is complete, switch application to the cluster at Site A.
8. The service group comes online on isvp12. Now, from the Storwize V7000 CLI or web interface,verify that the subsystem at Site A becomes the primary again. Wait until the service group
comes online on isvp12, and check that the DB application data is consistent.
7/31/2019 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global…
In this example, the Fire Drill service group is in the Gold configuration.
Refer to Chapter 5 in the Veritas Cluster Server Agent for IBM SVCCopyServices installation and
Configuration Guide for details on Bronze, Silver, and Gold configuration for the Fire Drill resource.
A local snapshot of the target LUN is taken and the fire drill service group in brought online by mountingthe replication target LUN based on the Fire Drill configuration. VCS creates a lock file to indicate that the
resource is online.
To check the failover readiness with Fire Drill:
1. Ensure that the application service group, test_shared_DG, is offline at cluster Site B.
2. Bring the fire drill service, firedrill_appgrp1, online on a node (isvp14) at site B.
This action validates your disaster recovery configuration. The production service group can failover to the secondary site in the event of an actual failure (disaster) at the primary site. If the fire
drill service group does not come online, review the VCS engine log for more information.
3. Take the fire drill offline after its functioning has been validated.
Figure 8 – Defining a new user on Storwize V7000 for SSH-only access
You will not require a password for these IDs but will need to upload the public SSH key that you saved
locally in the previous step. This key need to match the private key (which means they should have been
generated together).
After creating the user ID on the Storwize V7000 storage system, you can obtain a password-less SSH
session between the client LPAR and the storage subsystem. Be sure to use the -l flag specifying adminas the user, because this is the only one that Storwize V7000 will accept for sessions without providing a
password. This communication is required for the SVCCopyServices agent to function correctly in SFHA
This section shows the output collected by the test team during creation of testmm Oracle instance.
These include group and user creation as well as creation of the Oracle home directory, along withcommands used in the environment to create the database instance.
You can use the following commands to create groups.
7/31/2019 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global…
startup nomount pfile="/u01/app/oracle/admin/testmm/scripts/init.ora";@/u01/app/oracle/admin/testmm/scripts/rmanRestoreDatafiles.sql;spool off
CreateDBFiles.sql
connect SYS/&&sysPassword as SYSDBAset echo onspool /oracle/orahome/assistants/dbca/logs/CreateDBFiles.logCREATE TABLESPACE "USERS1" LOGGING DATAFILE'/oradata/&&DBNAME/mnt2/users01.dbf' SIZE 5M REUSE AUTOEXTEND ON NEXT 1280KMAXSIZE UNLIMITED EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT
AUTO ;CREATE TABLESPACE "USERS2" LOGGING DATAFILE'/oradata/&&DBNAME/mnt3/users02.dbf' SIZE 5M REUSE AUTOEXTEND ON NEXT 1280KMAXSIZE UNLIMITED EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO ;CREATE TABLESPACE "USERS3" LOGGING DATAFILE'/oradata/&&DBNAME/mnt4/users03.dbf' SIZE 5M REUSE AUTOEXTEND ON NEXT 1280K
7/31/2019 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global…
'/v7k_mm_testmount/testmm/system01.dbf','/v7k_mm_testmount/testmm/sysaux01.dbf','/v7k_mm_testmount/testmm/undotbs01.dbf','/v7k_mm_testmount/testmm/users01.dbf'LOGFILE GROUP 1 ('/v7k_mm_testmount/testmm/redo01.log') SIZE 51200K,GROUP 2 ('/v7k_mm_testmount/testmm/redo02.log') SIZE 51200K,GROUP 3 ('/v7k_mm_testmount/testmm/redo03.log') SIZE 51200K RESETLOGS;alter system enable restricted session;alter database "testmm" open resetlogs;exec dbms_service.delete_service('seeddata');exec dbms_service.delete_service('seeddataXDB');alter database rename global_name to "testmm"; ALTER TABLESPACE TEMP ADD TEMPFILE '/v7k_mm_testmount/testmm/temp01.dbf' SIZE
20480K REUSE AUTOEXTEND ON NEXT 640K MAXSIZE UNLIMITED;select tablespace_name from dba_tablespaces where tablespace_name='USERS';select sid, program, serial#, username from v$session;alter database character set INTERNAL_CONVERT WE8MSWIN1252;alter database national character set INTERNAL_CONVERT AL16UTF16;alter user sys account unlock identified by "&&sysPassword";alter user system account unlock identified by "&&systemPassword";alter system disable restricted session;
7/31/2019 Disaster recovery using Veritas Storage Foundation Enterprise HA with IBM Storwize V7000 Metro Mirror and Global…
References in this document to IBM products or services do not imply that IBM intends to make them
available in every country.
IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business
Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked
terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these
symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information
was published. Such trademarks may also be registered or common law trademarks in other countries. A
current list of IBM trademarks is available on the Web at "Copyright and trademark information" at
www.ibm.com/legal/copytrade.shtml.
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or
its affiliates.
Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in theUnited States, other countries, or both.
Intel, Intel Inside (logos), MMX, and Pentium are trademarks of Intel Corporation in the United States,
other countries, or both.
UNIX is a registered trademark of The Open Group in the United States and other countries.
Linux is a trademark of Linus Torvalds in the United States, other countries, or both.
SET and the SET Logo are trademarks owned by SET Secure Electronic Transaction LLC.
DIA Data Integrity Assurance®, Storewiz®, Storwize®, and the Storwize® logo are trademarks or
registered trademarks of Storwize, Inc., an IBM Company.
Other company, product, or service names may be trademarks or service marks of others.
Information is provided "AS IS" without warranty of any kind.
All customer examples described are presented as illustrations of how those customers have used IBM
products and the results they may have achieved. Actual environmental costs and performance
characteristics may vary by customer.
Information concerning non-IBM products was obtained from a supplier of these products, published
announcement material, or other publicly available sources and does not constitute an endorsement of
such products by IBM. Sources for non-IBM list prices and performance numbers are taken from publicly
available information, including vendor announcements and vendor worldwide homepages. IBM has not
tested these products and cannot confirm the accuracy of performance, capability, or any other claimsrelated to non-IBM products. Questions on the capability of non-IBM products should be addressed to the
supplier of those products.
All statements regarding IBM future direction and intent are subject to change or withdrawal without
notice, and represent goals and objectives only. Contact your local IBM office or IBM authorized reseller
for the full text of the specific Statement of Direction.