Maximum Availability Architecture Oracle Best Practices for High Availability Database Rolling Upgrade Using Transient Logical Standby: Oracle Data Guard 11g Oracle Maximum Availability Architecture White Paper June 2010
Nov 07, 2014
Maximum Availability Architecture Oracle Best Practices for High Availability
Database Rolling Upgrade Using Transient Logical Standby:
Oracle Data Guard 11g Oracle Maximum Availability Architecture White Paper
June 2010
Maximum Availability Architecture
MAA - Database Rolling Upgrade Using Transient Logical Standby: Oracle Data Guard 11g Page 1
Database Rolling Upgrade Using Transient Logical Standby:
Oracle Data Guard 11g
Executive Summary .......................................................................................... 2 Overview: Rolling Database Upgrade with Transient Logical Standby .... 3 General Restrictions .......................................................................................... 5 Rolling Upgrades Using a Transient Logical Standby .................................. 6
Confirm Data Type Support ....................................................................... 6 Conventions ................................................................................................... 9 Preparation Steps .......................................................................................... 9 Conversion to Logical Standby Steps ...................................................... 13 Create an Archived Redo Log Repository .............................................. 13 Perform Patch Apply or Upgrade Steps .................................................. 14 Create an Archived Redo Log Repository .............................................. 15 Switchover.................................................................................................... 15 Retransformation Into a Physical Standby Database ............................ 16 Switchback steps ......................................................................................... 18 Cleanup steps ............................................................................................... 18
Fallback Best Practices ................................................................................... 20 Backups ........................................................................................................ 20 Flashback Database .................................................................................... 21 Downgrade .................................................................................................. 22 Transient Logical Standby Fallback Considerations .............................. 22
Appendix A- Cloning the ORACLE_HOME to Apply a Patch set ....... 23 Appendix B- Using EDS with a Transient Logical Standby ..................... 24
Preparing EDS ............................................................................................ 24 Configuring EDS Logging Table Triggers .............................................. 25 Removing EDS ........................................................................................... 25
References ........................................................................................................ 26
Maximum Availability Architecture
MAA - Database Rolling Upgrade Using Transient Logical Standby: Oracle Data Guard 11g Page 2
Database Rolling Upgrade with Transient Logical Standby:
Oracle Data Guard 11g
EXECUTIVE SUMMARY
Oracle Maximum Availability Architecture (MAA) [1] is Oracle's best practices
blueprint for implementing Oracle high-availability technologies. This MAA white
paper provides best practices for utilizing a physical standby database to perform a
rolling upgrade with minimal downtime using the new Data Guard 11g feature,
Transient Logical Standby.
Rolling upgrades eliminates downtime associated with the conventional database
upgrade, including PL/SQL recompilation. Additionally, once the standby
database has been upgraded, it can be used to evaluate performance and stability of
the new release before there is any impact to production. The only database
downtime required by the entire upgrade process is the time it takes to perform a
Data Guard switchover (seconds).
A rolling upgrade using Transient Logical Standby:
1. Temporarily converts an existing physical standby to a logical standby
database.
2. Executes a rolling upgrade.
3. Returns the standby back to its original status as a physical standby
database once the upgrade is complete.
This process allows physical standby users to obtain the same benefits of using
logical standby for rolling database upgrades, without requiring a second set of
storage and maintenance for an extra logical standby database. Transient Logical
Standby is supported for upgrades beginning with Oracle Database 11g.
Rolling database upgrades are also supported with Oracle Database 10g Release 2
when using the standard rolling upgrade procedure described in the white paper,
“Rolling Database Upgrades Using Data Guard SQL Apply ”. The standard rolling
upgrade process is different from Transient Logical Standby in that the upgrade
process concludes with the standby remaining a logical standby database.
Oracle’s enterprise eMail application, hosting almost 70,000 worldwide users, reduced downtime for a database upgrade by 96%. Using SQL Apply rolling upgrade compared to the conventional upgrade reduced downtime, from 48 minutes to 2
minutes.
Maximum Availability Architecture
MAA - Database Rolling Upgrade Using Transient Logical Standby: Oracle Data Guard 11g Page 3
OVERVIEW: ROLLING DATABASE UPGRADE WITH TRANSIENT
LOGICAL STANDBY
Figure 1: Rolling Upgrade Using a Transient Logical Standby
Users who have chosen to implement physical standby databases for their HA/DR
requirements can benefit from the minimal downtime advantages of a SQL Apply
Rolling Upgrade by temporarily converting a physical standby database to a logical
standby for the duration of the upgrade. When the upgrade is complete, the logical
standby database is returned to its role as a physical standby for normal processing.
This eliminates the requirement for physical standby users to make a second set of
storage available to implement a rolling upgrade. Performing a rolling upgrade
using a transient logical standby is shown in Figure 1.
The following high-level steps describe a rolling upgrade using a transient logical
standby database:
1. Review prerequisites and preparation best practices and
a. create a guaranteed restore point
b. install upgraded ORACLE_HOME on each node
2. Convert the existing Physical Standby to a Logical Standby
Maximum Availability Architecture
MAA - Database Rolling Upgrade Using Transient Logical Standby: Oracle Data Guard 11g Page 4
3. Perform the upgrade on the logical standby and resynchronize with the
original primary
4. Switchover
5. Flashback the original primary (now a logical standby) to the guaranteed
restore point from #1
6. Remount the original primary (now a logical standby) under the new
Oracle Home
7. Convert the logical standby back to a physical standby and resynchronize
with the new primary, automatically completing the upgrade process
8. Wait for physical standby to be upgraded via the redo stream
9. Switch back to the original configuration (optional)
10. Raise the COMPATIBLE settings
Performing a rolling upgrade using a transient logical standby is similar to the
standard SQL Apply rolling upgrade with the following differences:
A guaranteed restore point is created on the primary database for the purpose
of flashing it back to become a physical standby after the switchover.
The conversion of a physical standby to a logical standby uses the new KEEP
IDENTITY clause to retain the same DB_NAME and DBID as that of its
primary database..
The ALTER DATABASE CONVERT TO PHYSICAL STANDBY
statement is used to convert the original primary from a logical standby to a
physical standby.
The original primary is actually upgraded by means of Redo Apply after it is
converted from a logical standby to a physical standby.
As with a standard SQL Apply rolling upgrade, the transient logical standby is
upgraded to release n while the primary runs uninterrupted at the older release m.
When the standby upgrade is deemed successful, the administrator allows Data
Guard to resynchronize the standby with the primary, and a switchover is
performed to transition the standby database to the primary role running on the
new release. While the standby site database operates in the primary role, the
database on the original primary is flashed back to the guaranteed restore point and
converted back to a physical standby. Once the original primary site database is
converted to a physical standby it is then restarted with version n and upgraded to
release n through Redo Apply. If desired, you can then perform a second
switchover to return all databases to their original roles. Total database downtime
is limited to the time it takes to execute a Data Guard switchover and the
Maximum Availability Architecture
MAA - Database Rolling Upgrade Using Transient Logical Standby: Oracle Data Guard 11g Page 5
application or client reconnection time. Compare this to the longer downtime
required for a conventional database upgrade.
Data Guard SQL Apply (logical standby database) is required to perform a database
rolling upgrade. The improved availability is achieved by offloading the following
to the logical standby:
The database upgrade process
Any unexpected upgrade problems
Any preliminary performance troubleshooting
Proof-of-concept tests using a transient logical standby at a large Oracle customer
demonstrated that primary and standby databases can be upgraded to a new release
with less than one minute of total database downtime.
Oracle internal testing with the database supporting Oracle‟s enterprise eMail
application hosting 70,000 users showed a 96% reduction in downtime, from 48
minutes to 2 minutes, when comparing the conventional upgrade method to the
SQL Apply rolling upgrade method.
Using SQL Apply to perform an upgrade will always result in a significant
reduction in planned downtime compared to the conventional database upgrade
method.
The generic upgrade steps for any Oracle database are detailed in the Oracle
Database Upgrade Guide 11g (11.1) [7]. Review this documentation before performing
a rolling upgrade using a transient logical standby. The best practices provided in
this paper complement the documentation, but the information in this white paper
is not intended to duplicate information already provided in the documentation.
GENERAL RESTRICTIONS
Some restrictions apply to using the transient logical standby rolling upgrade
process.
The databases must not be part of a Data Guard Broker configuration. Hence,
Oracle Enterprise Manager Grid Control does not support the rolling upgrade
process. To disable and enable the Broker configuration, see the “Preparation
Steps” section.
The Data Guard protection mode must be set to either maximum availability
or maximum performance. Query the PROTECTION_LEVEL column in the
V$DATABASE view to determine the current protection mode setting.
Maximum Availability Architecture
MAA - Database Rolling Upgrade Using Transient Logical Standby: Oracle Data Guard 11g Page 6
The LOG_ARCHIVE_DEST_n initialization parameter for the logical standby
database destination must be set to OPTIONAL to ensure the primary database
can proceed while the logical standby database is being upgraded.
The COMPATIBLE initialization parameter must match the software release
prior to the upgrade. That is, a rolling upgrade from release X to new release
X’ requires that the COMPATIBLE initialization parameter be set to X on
both the primary and standby databases. Then, after the upgrade and all
assurance tests have passed, you can update the COMPATIBLE parameter to
the new target release, X’.
NOTE: once you have updated the COMPATIBLE parameter to the target
database release, you cannot downgrade to an earlier release with flashback
database nor the database downgrade procedure
When you perform the initial switchover in the rolling upgrade process, you
cannot not use the PREPARE TO SWITCHOVER operation because the
primary and standby databases are running different Oracle releases.
You cannot have any bystander logical standby databases. A bystander logical
standby database is a logical standby database that exists in the same Data
Guard configuration as the physical standby that will become the transient
logical standby. If you already have a logical standby database then use that for
the rolling upgrade.
ROLLING UPGRADES USING A TRANSIENT LOGICAL STANDBY
The complete procedure documented here applies to any upgrade with a source
(starting) release of Oracle Database 11.1.0.6 or 11.1.0.7. You can also use a
transient logical standby to perform a rolling upgrade of Oracle Database 10.2, but
doing so requires additional steps. For a complete description about rolling
upgrades for release 10.2, including upgrades in a Real Application Cluster (Oracle
RAC) environment, see Oracle MetaLink note: 300479.1.
CONFIRM DATA TYPE SUPPORT
It is important to identify unsupported database objects on the primary database
before you create a logical standby database because changes made to unsupported
data types and tables on the primary database will be automatically skipped by SQL
Apply on the logical standby database. See Appendix C, “Data Type and DDL
Support on a Logical Standby Database” in the Oracle Data Guard Concepts and
Administration guide [5] for complete details on how to verify data type support.
Appendix C.9 has specific information related to unsupported tables.
Even if unsupported data types are identified, there are cases when a transient
logical standby procedure can still be used. The determination has to be made if
there is a satisfactory way to handle the unsupported data types. Options for using
rolling upgrade when unsupported data types exist are as follows:
Maximum Availability Architecture
MAA - Database Rolling Upgrade Using Transient Logical Standby: Oracle Data Guard 11g Page 7
Suspend or prohibit changes to the unsupported data type objects.
Temporarily suspend or prohibit changes to the unsupported tables for the
period of time it takes to perform the upgrade procedure.
Use DBA_LOGSTDBY_EVENTS with Oracle Data Pump or with the
Export/Import utility. If you cannot prevent changes to unsupported tables
during the upgrade, any unsupported transactions that occur are recorded in
the DBA_LOGSTDBY_EVENTS table on the logical standby database. After
the upgrade is completed, use Oracle Data Pump or the Export/Import utility
to import the changed tables to the upgraded databases. These features are
described in the Oracle Database Utilities documentation [10].
See Section 12.4 in Oracle Data Guard Concepts and Administration [5] for
further information.
Use Extended Datatype Support (EDS). EDS enables SQL Apply to replicate
changes to tables that contain some data types not natively supported.
Beginning with Oracle Database 11g Release 11.1.0.7, SQL Apply supports
the ability for triggers to fire on the logical standby database, which provides
the basis of EDS. Thus, the target release must be 11.1.0.7 or higher. Using
EDS does require some additional steps that are detailed in Appendix B,
“Using EDS with a Transient Logical Standby.”
For an overview of EDS, see the MAA white paper Extended Datatype
Support. For details and examples of using EDS to support data types that are
not natively supported by SQL Apply, see the following Support notes:
1. Pre 11.2 - My Oracle Support Note 559353.1
2. 11.2+ - My Oracle Support Note 949516.1
PREPARATION BEST PRACTICES
Preparing properly for the upgrade will give you the necessary knowledge and
confidence to complete a successful upgrade. Preparation for an upgrade using a
transient logical standby is very similar to preparation for a normal upgrade.
Chapter 2 of the Oracle Database Upgrade Guide [7] has an excellent set of steps to
follow. In addition to those steps, the following practices are also recommended:
Review the documentation resources
o Read the Oracle Database Upgrade Guide [7]
o Read the Oracle Database Readme [11]
o Review known issues and post-release Oracle MetaLink notes derived
from Note: 161818.1
o Review and follow the Oracle 11g Upgrade Companion in Oracle
MetaLink Note: 601807.1. This is an important document to review
Maximum Availability Architecture
MAA - Database Rolling Upgrade Using Transient Logical Standby: Oracle Data Guard 11g Page 8
because it contains a comprehensive set of steps and best practices
for upgrading.
Prepare a new ORACLE_HOME:
o If you are planning to do a patch or patch set upgrade, then clone a
new ORACLE_HOME and apply the patch or patch set to it on both
the primary and standby systems. This is not mandatory when
applying patch sets but it is a recommended practice. Having a
separate installation rather than directly applying the patch or
patchset to the existing installation allows for easy switching between
Oracle homes, including if a fallback is necessary. Using a new
ORACLE_HOME is also termed as an out-of-place patch set apply (see
“Cloning ORACLE_HOME for a Patchset Apply”).
o If you are planning to do a release upgrade, then install the new
Oracle Database software into a new ORACLE_HOME on both the
primary and the standby systems.
Create a test plan that includes the best practices outlined in the 11g Upgrade
Companion in Oracle MetaLink Note: 601807.1, and the following additional
practices:
o Upgrade testing.
o Fallback testing: Testing the fallback methods that will be used in the
event that any step fails. Various fallback methods are discussed in
the “Fallback” section later in this paper.
o Handle unsupported tables and objects: If you have any unsupported
tables and have determined a method to handle them within the
rolling upgrade, then ensure that this is in your test plan
Understand the fallback restoration options for each upgrade step. The
following fallback methods are described in the “Fallback” section later in this
paper:
o Backups
o Flashback Database
o Downgrade procedure
Maintain the same database COMPATIBLE initialization parameter setting on
all databases in the Data Guard configuration until the upgrade has been
completely evaluated. For example, if upgrading from release 11.1.0.6 to
11.1.0.7, then set the COMPATIBLE initialization parameter to 11.1.0.6.0.
Follow the best practices described in the MAA “Oracle Database 10g Release
2 Best Practices: Data Guard Switchover and Failover” [4] white paper to
optimize Data Guard switchover time.
Maximum Availability Architecture
MAA - Database Rolling Upgrade Using Transient Logical Standby: Oracle Data Guard 11g Page 9
Ensure beforehand that you have followed the guidelines for relocating Database
Services to the new primary database. See the MAA best practices paper “Client
Failover Best Practices for Highly Available Oracle Databases: Oracle Database
10g Release 2” for more information.
CONVENTIONS
The following text conventions are used in this document:
PROD PSTBY
Left column is always the primary site Right column is always the standby site
Other labels used in the table header are as follows:
PROD = Database is in primary role
PSTBY = Database is in the physical standby role
LSTBY = Database is in the logical standby role
PREPARATION STEPS
The preparation steps are the basis for an efficient and successful rolling upgrade.
1. If adding EDS for tables that contain unsupported data types, then perform the
“Preparing EDS” steps listed in Appendix B.
2. If you are using the Data Guard Broker to manage the Data Guard
configuration, you must stop the broker before you begin the rolling upgrade
process. You can reenable the broker after you switch back to the original
configuration. However, if you perform only one switchover, in which you
leave the original physical standby site database running in the primary
database role, then you must remove the Broker configuration and re-create it.
If you have implemented Fast-Start Failover, then you must disable fast-start
failover before you can disable broker management of the configuration.
PROD PSTBY
DGMGRL> disable configuration;
ALTER SYSTEM SET
DG_BROKER_START=FALSE SCOPE=BOTH;
ALTER SYSTEM SET
DG_BROKER_START=FALSE SCOPE=BOTH;
3. If standby redo log files do not already exist, then create standby redo log,files
of the same size as the online redo log files on the primary and the standby
databases so that you can enable real-time apply later. See Oracle Data Guard
Concepts and Administration [5] for guidelines to create standby redo log files
PROD PSTBY
ALTER DATABASE ADD STANDBY LOGFILE ALTER DATABASE ADD STANDBY LOGFILE
Maximum Availability Architecture
MAA - Database Rolling Upgrade Using Transient Logical Standby: Oracle Data Guard 11g Page 10
(’+PROD’) SIZE 50M;
...
(’+PSTBY’) SIZE 50M;
...
4. Set the protection level to MAXIMUM AVAILABILITY or MAXIMUM
PERFORMANCE for the rolling upgrade process.
PROD PSTBY
SELECT PROTECTION_MODE FROM
V$DATABASE;
SELECT PROTECTION_MODE FROM
V$DATABASE;
5. Set and adjust the LOG_ARCHIVE_DEST_n parameters and ensure that none
include the MANDATORY attribute, all use the VALID_FOR attribute, and
eliminate any DELAY attributes. Including the DB_UNIQUE_NAME attribute is
required only if the LOG_ARCHIVE_CONFIG parameter is configured, but
specifying DB_UNIQUE_NAME is a recommended practice
PROD PSTBY
ALTER SYSTEM SET
LOG_ARCHIVE_DEST_1='
LOCATION=USE_DB_RECOVERY_FILE_DEST
VALID_FOR=(ONLINE_LOGFILES,
ALL_ROLES) DB_UNIQUE_NAME=prod'
SCOPE=BOTH;
ALTER SYSTEM SET
LOG_ARCHIVE_DEST_2=
'SERVICE=pstby LGWR ASYNC
NOAFFIRM DELAY=0 OPTIONAL
REOPEN=15 MAX_FAILURE=10
VALID_FOR=(ONLINE_LOGFILES,
PRIMARY_ROLE)
DB_UNIQUE_NAME=pstby' SCOPE=BOTH;
ALTER SYSTEM SET
LOG_ARCHIVE_DEST_3=
'LOCATION=USE_DB_RECOVERY_FILE_DES
T VALID_FOR=(STANDBY_LOGFILES,
STANDBY_ROLE)
DB_UNIQUE_NAME=prod' SCOPE=BOTH;
ALTER SYSTEM SET
LOG_ARCHIVE_DEST_STATE_1=ENABLE
SCOPE=BOTH;
ALTER SYSTEM SET
LOG_ARCHIVE_DEST_STATE_2=ENABLE
SCOPE=BOTH;
ALTER SYSTEM SET
LOG_ARCHIVE_DEST_STATE_3=ENABLE
SCOPE=BOTH;
ALTER SYSTEM SET
LOG_ARCHIVE_DEST_1='
LOCATION=USE_DB_RECOVERY_FILE_DEST
VALID_FOR=(ONLINE_LOGFILES,
ALL_ROLES) DB_UNIQUE_NAME=pstby'
SCOPE=BOTH;
ALTER SYSTEM SET
LOG_ARCHIVE_DEST_2=
'SERVICE=prod LGWR ASYNC NOAFFIRM
DELAY=0 OPTIONAL REOPEN=15
MAX_FAILURE=10
VALID_FOR=(ONLINE_LOGFILES,
PRIMARY_ROLE)
DB_UNIQUE_NAME=prod' SCOPE=BOTH;
ALTER SYSTEM SET
LOG_ARCHIVE_DEST_3=
'LOCATION=USE_DB_RECOVERY_FILE_DES
T VALID_FOR=(STANDBY_LOGFILES,
STANDBY_ROLE)
DB_UNIQUE_NAME=pstby' SCOPE=BOTH;
ALTER SYSTEM SET
LOG_ARCHIVE_DEST_STATE_1=ENABLE
SCOPE=BOTH;
ALTER SYSTEM SET
LOG_ARCHIVE_DEST_STATE_2=ENABLE
SCOPE=BOTH;
ALTER SYSTEM SET
LOG_ARCHIVE_DEST_STATE_3=ENABLE
SCOPE=BOTH;
6. Ensure you correctly set the FAL_SERVER, FAL_CLIENT, LOG_ARCHIVE_CONFIG,
and STANDBY_FILE_MANAGEMENT initialization parameters:
Maximum Availability Architecture
MAA - Database Rolling Upgrade Using Transient Logical Standby: Oracle Data Guard 11g Page 11
Maximum Availability Architecture
MAA - Database Rolling Upgrade Using Transient Logical Standby: Oracle Data Guard 11g Page 12
PROD PSTBY
ALTER SYSTEM SET
FAL_SERVER=’pstby’ SCOPE=BOTH;
ALTER SYSTEM SET FAL_CLIENT=’prod’
SCOPE=BOTH;
ALTER SYSTEM SET
LOG_ARCHIVE_CONFIG=’dg_config=
(prod,pstby)’ SCOPE=BOTH;
ALTER SYSTEM SET
STANDBY_FILE_MANAGEMENT=’AUTO’
SCOPE=BOTH;
ALTER SYSTEM SET FAL_SERVER=’prod’
SCOPE=BOTH;
ALTER SYSTEM SET
FAL_CLIENT=’pstby’ SCOPE=BOTH;
ALTER SYSTEM SET
LOG_ARCHIVE_CONFIG=’dg_config=
(prod,pstby)’ SCOPE=BOTH;
ALTER SYSTEM SET
STANDBY_FILE_MANAGEMENT=’AUTO’
SCOPE=BOTH;
7. Set the LOG_FILE_NAME_CONVERT parameter whether or not the file
system structure is identical, to pre-clear the online redo logs and to optimize
switchover time (See the MAA white paper: “Oracle Database 10g Release 2
Best Practices: Data Guard Switchover and Failover” [4] for more information).
Do this in conjunction with the following step (to enable Flashback Database),
because the LOG_FILE_NAME_CONVERT parameter is static and can only be
switched during a restart. If the files systems are identical, then just set each side
of the convert pair to a blank (for example:. ALTER SYSTEM SET
LOG_FILE_NAME_CONVERT= „ ‟,‟ ‟ SCOPE=SPFILE;)
PROD PSTBY
ALTER SYSTEM SET
LOG_FILE_NAME_CONVERT= ’+PSTBY/’,
’+PROD/’ SCOPE=SPFILE;
ALTER SYSTEM SET
LOG_FILE_NAME_CONVERT= ’+PROD/’,
’+PSTBY/’ SCOPE=SPFILE;
8. Enable Flashback Database on both the primary and standby databases.
When you enable Flashback Database on the standby database, it allows you
to flash back the standby database if failures occur during the upgrade or
patch process (prior to changing the database COMPATIBLE parameter).
Enabling Flashback Database on the primary system requires that you restart
the database, allow sufficient I/O bandwidth, and provide additional disk
space for the flashback log files located in the Flash Recovery Area. This is
important because when a guaranteed restore point is created, a lack of
sufficient free space in the Flash Recovery Area will cause the database
operation to stop and wait for additional space to become available.
See the Oracle Database Backup and Recovery Users Guide [9] for complete details
about using Flashback Database.
Maximum Availability Architecture
MAA - Database Rolling Upgrade Using Transient Logical Standby: Oracle Data Guard 11g Page 13
PROD PSTBY
SHUTDOWN IMMEDIATE
(In an Oracle RAC environment for all instances)
STARTUP MOUNT
ALTER DATABASE FLASHBACK ON;
ALTER DATABASE OPEN;
Optional:
SHUTDOWN IMMEDIATE
STARTUP MOUNT
ALTER DATABASE FLASHBACK ON;
9. Create a guaranteed restore point on the primary database PROD. To provide the
ability to flashback the database on the standby site as well (for example, in case an
error occurs while applying a patch or upgrading the database upgrade), then create
a guaranteed restore point on PSTBY, too.
PROD PSTBY
CREATE RESTORE POINT PRE_UPGRADE_1
GUARANTEE FLASHBACK DATABASE;
Optional:
CREATE RESTORE POINT PRE_UPGRADE_2
GUARANTEE FLASHBACK DATABASE;
RECOVER MANAGED STANDBY DATABASE
USING CURRENT LOGFILE DISCONNECT;
CONVERSION TO LOGICAL STANDBY STEPS
Now the physical standby is converted to a logical standby by creating a LogMiner
dictionary on PROD and synchronizing the transient logical standby with PSTBY .
Including the new Oracle Database 11g KEEP IDENTITY clause (on the
SQL*Plus ALTER DATABASE RECOVER TO LOGICAL STANDBY statement)
ensures that the logical standby database keeps its DBID, which is identical to the ID
of the PROD database.
At this point the new logical standby LSTBY receives the necessary redo
information from PROD to populate its own LogMiner dictionary. This step can
take a few minutes but it does not affect availability. You can monitor the
dictionary build by querying the V$LOGSTDBY_STATE view. When it returns
”IDLE” that indicates the build completed and SQL Apply is proceeding. The
output of querying V$LOGSTDBY_STATE will change from ”INITIALIZING”
to ”WAITING FOR DICTIONARY LOGS” to ”LOADING DICTIONARY” to
”APPLYING” and finally to “IDLE.”
CREATE AN ARCHIVED REDO LOG REPOSITORY
Optionally, you can create an archived redo log repository with the same database
release (the target release) and COMPATIBLE setting as the primary database so
that redo is still received while applying the patch or upgrading the database. See
The new Oracle Database 11g KEEP
IDENTITY clause ensures that the
logical standby database keeps its database ID (DBID) identical to the primary. This facilitates the later step when you convert the transient logical standby database back to its original role as a physical standby.
Maximum Availability Architecture
MAA - Database Rolling Upgrade Using Transient Logical Standby: Oracle Data Guard 11g Page 14
Oracle MetaLink note 434164.1. Creating a repository ensures you can meet the
recovery point objective (RPO) in the event the primary site fails during the
upgrade of the logical.
PERFORM PATCH APPLY OR UPGRADE STEPS
Because you will perform the upgrade or apply the patch or patch set first on the
logical standby database, you must stop SQL Apply before you can perform the
necessary upgrade steps. At this point, it is recommended to create another
guaranteed restore point on the logical standby as a fallback possibility.
PROD LSTBY
ALTER SYSTEM SET
LOG_ARCHIVE_DEST_STATE_2=DEFER
SCOPE=MEMORY;
ALTER DATABASE STOP LOGICAL
STANDBY APPLY;
CREATE RESTORE POINT PRE_UPGRADE_3
GUARANTEE FLASHBACK DATABASE;
SHUTDOWN IMMEDIATE
1. Set the environment to the new ORACLE_HOME.
Perform the database upgrade on the logical standby using either the Database
Upgrade Assistant (DBUA) or the manual upgrade method with
catupgrd.sql Using the DBUA is the recommended method because it
handles all parameter changes and any updates to the Oracle Cluster Registry
(OCR) in an Oracle RAC environment. See the “Oracle Database Upgrade Guide
11.1 ” [7] for complete upgrade instructions.
Note: Although you can change the COMPATIBLE parameter on the logical
standby site to the new version, it is not required or recommended.
o Changing the COMPATIBLE parameter now is against best
practice. The best practice is to change COMPATIBLE after
successfully completing the rolling upgrade process and after you
have successfully verified and tested the configuration. Changing the
COMPATIBLE parameter requires that you restart both databases.
o Changing COMPATIBLE at this stage will prevent use of the
downgrade procedure or the ability to flash back the database
to a point prior to this. You cannot revert the COMPATIBLE
parameter back to its earlier value once it has been set to a higher
release value. If you raise the value of the COMPATIBLE parameter
at this point, the only way to revert LSTBY to its original role as a
physical standby database is to restore an earlier backup or
completely recreate the standby database.
2. The upgrade is now complete. If you are using EDS then perform the steps in
“Configuring EDS Logging Table Triggers” in Appendix B.
Maximum Availability Architecture
MAA - Database Rolling Upgrade Using Transient Logical Standby: Oracle Data Guard 11g Page 15
3. If an archived redo log repository was used then you can use the RMAN
CATALOG command to catalog the archived redo log repository archivelog
destination to avoid resending those logs. e.g.:
PROD PSTBY
RMAN> CATALOG START WITH
'+PSTBY/ALOGREP/ARCHIVELOG/’;
4. Start SQL Apply
PROD LSTBY
ALTER SYSTEM SET
LOG_ARCHIVE_DEST_STATE_2=ENABLE
SCOPE=MEMORY;
STARTUP
Note: If using EDS, perform the steps in “Configuring
EDS Logging Table Triggers” in Appendix B before starting
SQL Apply.
ALTER DATABASE START LOGICAL
STANDBY APPLY IMMEDIATE;
Up to this point there‟s no impact on availability of the primary database.
CREATE AN ARCHIVED REDO LOG REPOSITORY
Optionally, you can create an archived redo log repository with the same database
release (the target release) and COMPATIBLE setting as the primary database so
that redo is still received while applying the patch or upgrading the database. See
Oracle MetaLink note 434164.1. Creating a repository ensures you can meet the
recovery point objective (RPO) in the event the primary site fails during this step.
SWITCHOVER
Perform a switchover to relocate the primary role to the standby site. Although a
switchover incurs some downtime you can minimize the application downtime to
less than 1 minute if you applied the switchover best practices described in the
“Preparation Best Practices” section.
The “SWITCHOVER TO LOGICAL STANDBY;” command cannot use the
“WITH SESSION SHUTDOWN” clause so you need to manually shutdown
sessions.
PROD LSTBY PROD
Wait for this query to display “TO STANDBY” or proceed
to switchover with session disconnect
SELECT SWITCHOVER_STATUS FROM
V$DATABASE;
ALTER DATABASE COMMIT TO
SWITCHOVER TO LOGICAL STANDBY;
Wait for this query to display “TO PRIMARY” or proceed
Maximum Availability Architecture
MAA - Database Rolling Upgrade Using Transient Logical Standby: Oracle Data Guard 11g Page 16
PROD LSTBY PROD
to switchover with session disconnect:
SELECT SWITCHOVER_STATUS FROM
V$DATABASE;
ALTER DATABASE COMMIT TO
SWITCHOVER TO LOGICAL PRIMARY;
RETRANSFORMATION INTO A PHYSICAL STANDBY DATABASE
The former primary database, which is currently running as the transient logical
standby (LSTBY), cannot provide disaster recovery because it is running under a
lower database version than the new primary database, and cannot receive redo.
Thus, you must convert LSTBY back into a physical standby database.
1. If using EDS, remove EDS components by performing the steps in
“Removing EDS” in Appendix B.
2. Defer the redo transmission from the primary to the standby and then flash
back the standby to the guaranteed restore point.
LSTBY PSTBY PROD
ALTER SYSTEM SET
LOG_ARCHIVE_DEST_STATE_2=DEFER
SCOPE=MEMORY;
SHUTDOWN IMMEDIATE;
STARTUP MOUNT;
FLASHBACK DATABASE TO RESTORE
POINT PRE_UPGRADE_1;
3. Set your environment to use the new ORACLE_HOME. Issue the ALTER
DATABASE CONVERT TO PHYSICAL STANDBY statement to convert the logical
standby to a physical standby database.
LSTBY PSTBY PROD
SHUTDOWN IMMEDIATE;
Switch to the new ORACLE_HOME
STARTUP MOUNT;
ALTER DATABASE CONVERT TO PHYSICAL
STANDBY;
4. Because the CONVERT TO PHYSICAL statement dismounts the database,
you must remount the PSTBY database.
PSTBY PROD
SHUTDOWN IMMEDIATE
STARTUP MOUNT;
Maximum Availability Architecture
MAA - Database Rolling Upgrade Using Transient Logical Standby: Oracle Data Guard 11g Page 17
5. If you previously set the COMPATIBLE parameter to the higher target database
release number on the LSTBY site, then you must set COMPATIBLE to the same
value on the PROD site. Remember that changing the value of the
COMPATIBLE parameter eliminates most options for downgrading.
PSTBY PROD
ALTER SYSTEM SET COMPATIBLE=’...’
SCOPE=SPFILE;
SHUTDOWN IMMEDIATE
STARTUP MOUNT;
6. If an archived redo log repository was used then you can use the RMAN
CATALOG command to catalog the archived redo log repository archivelog
destination to avoid resending those logs. e.g.:
PSTBY PROD
RMAN> CATALOG START WITH
'+PROD/ALOGREP/ARCHIVELOG/’;
7. Start Redo Apply on the physical standby database PSTBY. Data Guard will
synchronize the physical standby database with the primary database PROD.
PSTBY PROD
RECOVER MANAGED STANDBY DATABASE
USING CURRENT LOGFILE DISCONNECT;
ALTER SYSTEM SET
LOG_ARCHIVE_DEST_STATE_2=ENABLE;
8. When Redo Apply starts, it registers a new incarnation from the primary
database. During that time, Redo Apply will be in a wait state, looping every
10 seconds until the new incarnation is registered. A sample of that sequence
of alert log messages is similar to the following:
Managed Standby Recovery starting Real Time Apply
MRP0: Incarnation has changed! Retry recovery...
Errors in file /u01/app/oracle/diag/rdbms/PROD/PROD/trace/PROD_mrp0_741.trc:
ORA-19906: recovery target incarnation changed during recovery
<10 second gap >
Managed Standby Recovery not using Real Time Apply
Errors in file /u01/app/oracle/diag/rdbms/PROD/PROD/trace/PROD_mrp0_741.trc:
ORA-19906: recovery target incarnation changed during recovery
Fast Parallel Media Recovery enabled
Managed Standby Recovery starting Real Time Apply
. . .
. . .
Setting recovery target incarnation to 4
RFS[3]: Incarnation entry added for Branch(resetlogs_id): 647358813 (PROD)
Fri Feb 22 14:10:36 2008
Setting recovery target incarnation to 4
Fast Parallel Media Recovery enabled
Managed Standby Recovery starting Real Time Apply
Maximum Availability Architecture
MAA - Database Rolling Upgrade Using Transient Logical Standby: Oracle Data Guard 11g Page 18
RFS[1]: Archived Log: '+PROD/PROD/archivelog/2008_02_22/thread_1_seq_3.2334.647359837'
RFS[2]: Archived Log: '+PROD/PROD/archivelog/2008_02_22/thread_1_seq_1.285.647359835'
RFS[3]: Archived Log: '+PROD/PROD/archivelog/2008_02_22/thread_1_seq_4.622.647359837'
parallel recovery started with 3 processes
. . .
Media Recovery start incarnation depth : 1, target inc# : 4, irscn : 1002171
Waiting for all non-current ORLs to be archived...
Clearing online redo logfile 1 +PROD/PROD/onlinelog/group_1.1629.647333483
Clearing online log 1 of thread 1 sequence number 13
If the primary and standby databases are running on identical systems, then there‟s
no reason to perform a switchover to have PSTBY return to the standby role.
Thus, the whole rolling upgrade has incurred just one switchover operation with
minimal downtime. You can wait to perform another switchover to return the
PROD database to run in the role of the primary database during the next planned
maintenance operation.
SWITCHBACK STEPS
This section describes how to perform a second switchover to return the original
primary database (PROD) back in the role as the primary database and to return
the original physical standby (PSTBY) into the role of the physical standby
database. This step is optional as long as your standby site can support your
production operations.
PSTBY PROD PROD PSTBY
Wait for this query to display “TO STANDBY” or proceed
to switchover with session disconnect:
SELECT SWITCHOVER_STATUS
FROM V$DATABASE;
ALTER DATABASE COMMIT TO
SWITCHOVER TO STANDBY WITH SESSION
SHUTDOWN;
Wait for this query to display “TO PRIMARY” or proceed
to switchover with session disconnect:
SELECT SWITCHOVER_STATUS FROM
V$DATABASE;
ALTER DATABASE COMMIT TO
SWITCHOVER TO PRIMARY;
ALTER DATABASE OPEN; SHUTDOWN IMMEDIATE;
STARTUP MOUNT;
RECOVER MANAGED STANDBY DATABASE
USING CURRENT LOGFILE DISCONNECT;
CLEANUP STEPS
The following example shows how to drop the existing guaranteed restore point and
any others that are no longer needed to prevent running out of disk space.
Maximum Availability Architecture
MAA - Database Rolling Upgrade Using Transient Logical Standby: Oracle Data Guard 11g Page 19
PROD PSTBY
SELECT NAME FROM V$RESTORE_POINT
WHERE GUARANTEE_FLASHBACK_DATABASE
=’YES’;
SELECT NAME FROM V$RESTORE_POINT
WHERE GUARANTEE_FLASHBACK_DATABASE
=’YES’;
DROP RESTORE POINT PRE_UPGRADE_1; DROP RESTORE POINT <name>;
If the Data Guard Broker has been used at the beginning of this process, reenable it as
shown in the following example.
PROD PSTBY
ALTER SYSTEM SET
DG_BROKER_START=TRUE SCOPE=BOTH;
ALTER SYSTEM SET
DG_BROKER_START=TRUE SCOPE=BOTH;
DGMGRL> enable configuration;
Maximum Availability Architecture
MAA - Database Rolling Upgrade Using Transient Logical Standby: Oracle Data Guard 11g Page 20
FALLBACK BEST PRACTICES
This section describes the following fallback practices:
1. Backups
2. Flashback Database
3. Downgrades
The Flowchart in Figure 2 presents an overview of which fallback option is viable
at each step of the SQL Apply Rolling Upgrade.
BACKUPS
Ensure that you take database and software backups on the primary and the
standby databases prior to starting the upgrade process. The software backups
should include the oraInventory directory tree. Taking software backups are
Figure 2: SQL Apply Rolling Upgrade with Transient Logical Fallback Options
Maximum Availability Architecture
MAA - Database Rolling Upgrade Using Transient Logical Standby: Oracle Data Guard 11g Page 21
necessary only if they have never been done and if you are applying the patch set
directly to the existing ORACLE_HOME tree rather than applying the patch set to a
newly installed separate ORACLE_HOME.
FLASHBACK DATABASE
Flashback Database is a method to achieve very fast point-in-time recovery and is
most often the quickest way to fall back to the previous release. Note that data
from any transactions that occur after the point in time to which the database is
recovered are lost. Flashback Database is an excellent tool for backing out the
database upgrade on the logical standby before the first switchover in the SQL
Apply rolling upgrade process. Using Flashback Database with sufficient space and
creating a guaranteed restore point immediately prior to the upgrade is the fastest
method to fallback as compared to a restore operation or a database downgrade.
Following switchover, and after the production workload is running on the original
standby database, any Flashback operation will result in data loss. Deciding to use
Flashback Database versus downgrading the database typically entails a tradeoff
between the speed at which you need to return the primary database to the
previous release and the amount of data loss that you can tolerate.
NOTE: You can use Flashback Database to fall back to the pre-upgrade release
only if you have not changed the COMPATIBLE database parameter to the target
release. See the “General Restrictions” section in this white paper for additional
information.
The steps for flashing back the database after a failed upgrade (due to a failure
while running the catupgrd.sql script or a DBUA failure) are as follows:
1. Shut down the upgraded database.
2. Startup mount the database under the new ORACLE_HOME
3. Use Flashback Database to return to the guaranteed restore point that was
taken prior to the upgrade:
SQL> flashback database to restore point PRE_LOGICAL_UPGRADE;
4. Shut down the database
5. Startup mount the database under the old ORACLE_HOME
6. Issue the OPEN RESETLOGS command to open the database:
SQL> alter database open resetlogs;
Maximum Availability Architecture
MAA - Database Rolling Upgrade Using Transient Logical Standby: Oracle Data Guard 11g Page 22
DOWNGRADE
To run the downgrade procedure, use the manual downgrade steps documented in
Chapter 7 of the Oracle Database Upgrade Guide 11g Release 1 (11.1) [7]. Performing a
downgrade generally takes as long as the upgrade procedure and backs out the
patch set changes while maintaining any transactional changes. In cases where
transactional changes have taken place since the upgrade and there is no other way
to restore the data, then you should perform a downgrade procedure. A
prerequisite to downgrading is that the COMPATIBLE database parameter must
have remained at the release to which you are downgrading .
TRANSIENT LOGICAL STANDBY FALLBACK CONSIDERATIONS
The fallback practices when using a Transient Logical Standby differ from the
fallback practices you would use after performing a standard SQL Apply Rolling
Upgrade. The flowchart shown in Figure 3 presents an overview of which fallback
options are viable at each step for a rolling upgrade with transient logical standby.
In general, ensure you have taken the proper backups prior to invoking the rolling
upgrade.
Maximum Availability Architecture
MAA - Database Rolling Upgrade Using Transient Logical Standby: Oracle Data Guard 11g Page 23
APPENDIX A- CLONING THE ORACLE_HOME TO APPLY A PATCH SET
For an out-of-place (creating a new ORACLE_HOME) patch set apply, it is
recommended to clone the existing ORACLE_HOME. The following procedure
shows the commands you would use on a Linux system:
1. As the root user, on each node in the cluster, copy the existing ORACLE_HOME
to the new ORACLE_HOME location. (This requires no downtime.) For example:
cp –pr /u01/app/oracle/product/11.1.0.6
/u01/app/oracle/product/11.1.0.7
2. Validate that any file changes between the source ORACLE_HOME and the copied ORACLE_HOME are ok (only log files may differ): > diff -q -r /u01/app/oracle/product/11.1.0.6
/u01/app/oracle/product/11.1.0.7
3. As the OS user that owns the Oracle software, add the new ORACLE_HOME
software to the Oracle Inventory using a cloning script. For example: #!/bin/sh
echo "Clone started at `date`" | tee -a clone.log
perl /u01/app/oracle/product/11.1.0.7/clone/bin/clone.pl
ORACLE_HOME=/u01/app/oracle/product/11.1.0.7
ORACLE_HOME_NAME=11gR1P2 '-O"CLUSTER_NODES={chi01,chi02}"'
'-O"LOCAL_NODE=chi01"'
echo "Clone ended at `date`" | tee -a clone.log
4. As root user, run the root.sh script on each node.
5. Apply the patch set to the newly cloned ORACLE_HOME
(/u01/app/oracle/product/11.1.0.7)
Note: If the source ORACLE_HOME contains any hard links or symbolic links
these will also be cloned. If any of these links in the cloned ORACLE_HOME
reference the source ORACLE_HOME then you must manually change them to
point to the cloned ORACLE_HOME path.
Maximum Availability Architecture
MAA - Database Rolling Upgrade Using Transient Logical Standby: Oracle Data Guard 11g Page 24
APPENDIX B- USING EDS WITH A TRANSIENT LOGICAL STANDBY
For an overview of Extended Datatype Support (EDS), see the MAA white paper
Extended Datatype Support.
For details and examples of using EDS to support data types that are not natively
supported by SQL Apply, see the following Support notes:
3. Pre 11.2 - My Oracle Support Note 559353.1
4. 11.2+ - My Oracle Support Note 949516.1
Using EDS with a transient logical is intended for use while the standby is a logical
standby and once the standby is converted back to a physical standby EDS is no
longer necessary.
PREPARING EDS
Prior to converting the physical standby to a logical standby perform the following
steps to setup EDS. These steps are performed on the primary database for each
table that requires EDS support.
1. On the primary database, for each schema that contains a table that requires EDS support, grant execute privilege on the DBMS_LOGSTDBY package.
e.g. (where OBJUSER is the schema)
SQL> connect system/<password>
SQL> grant execute on dbms_logstdby to OBJUSER;
2. Create the logging table on the primary database e.g. OBJUSER.PLAYERS_LOG
3. Create the base table trigger on the primary database e.g. OBJUSER.PLAYERS_PRI_TRG
4. Create the logging table trigger on the primary database e.g. OBJUSER.PLAYERS_STBY_TRG
NOTE: If the starting release is Oracle Database 11g Release 1 (11.1.0.6) then create all logging table triggers disabled. For example:
SQL> create trigger PLAYERS_STBY_TRG disable ...
In Oracle Database 11g Release 1 (11.1.0.6), logging table triggers will generate warnings during compilation and will impact operations on the base table if they are not disabled. Invalid logging table triggers will be successfully compiled and enabled on the logical standby database once it has been upgraded to the later release.
The table and trigger names provided follow the examples in MetaLink Note 565069.1. See MetaLink Note 559353.1 for additional EDS logging table and EDS trigger examples.
Maximum Availability Architecture
MAA - Database Rolling Upgrade Using Transient Logical Standby: Oracle Data Guard 11g Page 25
CONFIGURING EDS LOGGING TABLE TRIGGERS
After the physical standby has become a logical standby and the logical standby has
been upgraded configure the logging table triggers. These commands are run on
the upgraded logical standby database for each logging table trigger used by EDS.
NOTE: The following steps must be done prior to starting SQL Apply for
the first time. If SQL Apply is started before these steps are performed,
updates performed on the primary database to EDS-supported tables during
the upgrade will not be applied to the upgraded logical standby database.
1. As SYS, temporarily disable GUARD, then enable and compile logging table triggers
SQL> alter session disable guard; SQL> alter trigger OBJUSER.PLAYERS_STBY_TRG compile;
SQL> alter trigger OBJUSER.PLAYERS_STBY_TRG enable; SQL> alter session enable guard;
2. Set the FIRE_ONCE trigger firing property for all logging table triggers to FALSE
SQL> execute dbms_ddl.set_trigger_firing_property('OBJUSER',
'PLAYERS_STBY_TRG', fire_once => FALSE);
REMOVING EDS
Once the switchover takes place and the upgraded logical standby becomes the new primary, drop all the EDS objects on the primary.
SQL> drop trigger OBJUSER.PLAYERS_STBY_TRG;
SQL> drop trigger OBJUSER.PLAYERS_PRI_TRG; SQL> drop table OBJUSER.PLAYERS_LOG;
Before starting SQL Apply for the first time, upgrade the logical standby database and configure the logging table triggers.
Maximum Availability Architecture
MAA - Database Rolling Upgrade Using Transient Logical Standby: Oracle Data Guard 11g Page 26
REFERENCES
1. Oracle Maximum Availability Architecture
http://www.oracle.com/technology/deploy/availability/htdocs/maa.htm
2. Oracle Data Guard
http://www.oracle.com/technology/deploy/availability/htdocs/DataGuardOverview.html
3. Oracle Database 10g Release 2 Best Practices: Data Guard SQL Apply
www.oracle.com/technology/deploy/availability/pdf/MAA_WP_10gR2_SQLApplyBestPr
actices.pdf
4. Oracle Database 10g Release 2 Best Practices: Data Guard Switchover and
Failover
http://www.oracle.com/technology/deploy/availability/pdf/MAA_WP_10gR2_Switchove
rFailoverBestPractices.pdf
5. Oracle Data Guard Concepts and Administration 11g Release 1 (11.1) (Part #B28294)
http://otn.oracle.com/pls/db111/db111.to_toc?partno=b28294
6. Oracle Database High Availability Overview 11g Release 1 (11.1) (Part #B28281)
http://otn.oracle.com/pls/db111/db111.to_toc?partno=b28281
7. Oracle Database Upgrade Guide 11g Release 1 (11.1) (Part #B28300)
http://otn.oracle.com/pls/db111/db111.to_toc?partno=b28300
8. Oracle Data Guard Broker 11g Release 1 (11.1) (Part Number B28295)
http://otn.oracle.com/pls/db111/db111.to_toc?partno=b28295
9. Oracle Database Backup and Recovery Users Guide 11g Release 1 (11.1)
(Part Number B28270)
http://otn.oracle.com/pls/db111/db111.to_toc?partno=b28270
10. Oracle Database Utilities 11g Release 1 (11.1) (Part Number B28319)
http://otn.oracle.com/pls/db111/db111.to_toc?partno=b28319
11. Oracle Database Readme 11g Release 1 (11.1) Part Number B28280
http://otn.oracle.com/pls/db111/db111.to_toc?partno=b28280
Database Rolling Upgrade Using Transient Logical Standby, Oracle Data Guard 11g
June 2010
Authors: Ray Dutcher, Mike Dietrich
Contributing Authors: Lawrence To, Joe Meeks, Viv Schupmann, Doug Utzig
Oracle USA, Inc.
World Headquarters
500 Oracle Parkway
Redwood Shores, CA 94065
U.S.A.
Worldwide Inquiries:
Phone: +1.650.506.7000
Fax: +1.650.506.7200
oracle.com
Copyright © 2008, Oracle. All rights reserved.
This document is provided for information purposes only and the
contents hereof are subject to change without notice.
This document is not warranted to be error-free, nor subject to any
other warranties or conditions, whether expressed orally or implied
in law, including implied warranties and conditions of merchantability
or fitness for a particular purpose. We specifically disclaim any
liability with respect to this document and no contractual obligations
are formed either directly or indirectly by this document. This document
may not be reproduced or transmitted in any form or by any means,
electronic or mechanical, for any purpose, without our prior written permission.
Oracle, JD Edwards, PeopleSoft, and Siebel are registered trademarks of
Oracle Corporation and/or its affiliates. Other names may be trademarks
of their respective owners.