Top Banner
Upgrade with Minimal Downtime Data Guard Transient Logical Standby by Kalyan Kallepally Senior Database Administrator Wellcome Trust Sanger Institute Cambridge, UK [email protected]
23

Upgrade with Minimal Downtime - Oracle · Upgrade with Minimal Downtime ... • 10+ years of experience in Oracle ... (MRC) • Primarily funded by the Wellcome Trust

Apr 16, 2018

Download

Documents

phungbao
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Upgrade with Minimal Downtime - Oracle · Upgrade with Minimal Downtime ... • 10+ years of experience in Oracle ... (MRC) • Primarily funded by the Wellcome Trust

Upgrade with Minimal Downtime

Data Guard Transient Logical Standby

by Kalyan Kallepally

Senior Database Administrator

Wellcome Trust Sanger Institute Cambridge, UK

[email protected]

Page 2: Upgrade with Minimal Downtime - Oracle · Upgrade with Minimal Downtime ... • 10+ years of experience in Oracle ... (MRC) • Primarily funded by the Wellcome Trust

Who am I • Working as Senior Database Administrator at

– Wellcome Trust Sanger Institute • 10+ years of experience in Oracle (8-11gr2) • 5+ years of experience in MySQL, SQL Server • Recently started blogging

– www.dbakalyan.wordpress.com – You can reach me on [email protected]

Page 3: Upgrade with Minimal Downtime - Oracle · Upgrade with Minimal Downtime ... • 10+ years of experience in Oracle ... (MRC) • Primarily funded by the Wellcome Trust

Wellcome Trust Sanger Institute

• One of the leading genomics centres in the world • Located in Hinxton, south of Cambridge, UK • Founded in 1993 by the Wellcome Trust and the UK Medical Research Council

(MRC) • Primarily funded by the Wellcome Trust • The institute is named after Dr Fred Sanger – The double Nobel prize-winning

biochemist • Responsible for the completion of 1/3 of the human genome

Page 4: Upgrade with Minimal Downtime - Oracle · Upgrade with Minimal Downtime ... • 10+ years of experience in Oracle ... (MRC) • Primarily funded by the Wellcome Trust

• Current computational capacity of 16k cores: – 12k cores on blade format – 4k cores in racked servers

• Current usable storage capacity 16 Petabytes: – Estimated growth 3

Petabytes/year • Data shared globally across

the world with many scientific research institutes

Page 5: Upgrade with Minimal Downtime - Oracle · Upgrade with Minimal Downtime ... • 10+ years of experience in Oracle ... (MRC) • Primarily funded by the Wellcome Trust

Database Layout • Primary database (300G)

– ASM – 2 Physical standbys (local and remote)

• Local standby is within same site as of the primary but placed in a different quadrant of the datacentre

• DR standby is 10 miles away from the Primary with dedicated private 10Gb/s WAN to DR datacentre

Page 6: Upgrade with Minimal Downtime - Oracle · Upgrade with Minimal Downtime ... • 10+ years of experience in Oracle ... (MRC) • Primarily funded by the Wellcome Trust

Upgrade Process • Upgraded database from 11.1.0.6 to 11.2.0.3

using Transient Logical standby – Down time reduced by 98%

• Upgrade process is split into 2 phases: – Phase 1 (Upgrade Primary Database) – Phase 2 (Upgrade Standbys & Switchback)

Page 7: Upgrade with Minimal Downtime - Oracle · Upgrade with Minimal Downtime ... • 10+ years of experience in Oracle ... (MRC) • Primarily funded by the Wellcome Trust

Phase - 1

Upgrade Primary Database

Page 8: Upgrade with Minimal Downtime - Oracle · Upgrade with Minimal Downtime ... • 10+ years of experience in Oracle ... (MRC) • Primarily funded by the Wellcome Trust

Physical Standby Database

11.1.0.6

Production Database

11.1.0.6

11.1.0.6

Sync or Async Redo shipping

Sync or Async Redo shipping

Local Site

DR Site

Read Write Clients Read Only Clients

Redo Logs

Redo Logs

ASM

ASM

ASM

Broker

Backup

Redo apply

Redo apply

Oracle Home (11.1.0.6)

Oracle Home (11.1.0.6)

Oracle Home (11.1.0.6)

Physical Standby Database

Page 9: Upgrade with Minimal Downtime - Oracle · Upgrade with Minimal Downtime ... • 10+ years of experience in Oracle ... (MRC) • Primarily funded by the Wellcome Trust

Upgrade Pre-requisites • Disable Data Guard Broker • Protection level must be either Maximum

availability or Maximum performance • Ensure Flashback is enabled on all databases in

the Data Guard configuration • Create guaranteed restore points • Parameter “compatible” must be set to same on

all the databases

Page 10: Upgrade with Minimal Downtime - Oracle · Upgrade with Minimal Downtime ... • 10+ years of experience in Oracle ... (MRC) • Primarily funded by the Wellcome Trust

Physical Standby Database

11.1.0.6

Primary Database

11.1.0.6

11.1.0.6

Local Site

DR Site

Read Write Clients Read Only Clients

Redo Logs

ASM

ASM

ASM

Broker

Backup

Redo apply

Oracle Home (11.1.0.6) Physical Standby

Database

• Disable Broker • Enable Flashback • Create Guaranteed Restore points

Redo Logs

Redo apply

Oracle Home (11.1.0.6)

Oracle Home (11.1.0.6)

Sync or Async Redo shipping

Sync or Async Redo shipping

Page 11: Upgrade with Minimal Downtime - Oracle · Upgrade with Minimal Downtime ... • 10+ years of experience in Oracle ... (MRC) • Primarily funded by the Wellcome Trust

11.1.0.6

Primary Database

11.1.0.6

11.1.0.6

Local Site

DR Site

Read Write Clients Read Only Clients

Redo Logs

Redo Logs

ASM

ASM

ASM

Broker

Backup

Physical Standby Database Redo

apply

Oracle Home (11.1.0.6)

Oracle Home (11.1.0.6)

Oracle Home (11.1.0.6)

Transform Redo to SQL

Logical Standby Database

SQL apply

• Convert the DR physical standby to Transient Logical Standby • Check if Logical standby is in sync

Physical Standby Database

Redo apply

Sync or Async Redo shipping

Sync or Async Redo shipping

Page 12: Upgrade with Minimal Downtime - Oracle · Upgrade with Minimal Downtime ... • 10+ years of experience in Oracle ... (MRC) • Primarily funded by the Wellcome Trust

Physical Standby Database

11.1.0.6

Primary Database

11.1.0.6

11.2.0.3

Local Site

DR Site

Read Write Clients Read Only Clients

Redo Logs

Redo Logs

ASM

ASM

ASM

Broker

Backup

Redo apply

SQL apply

Oracle Home (11.1.0.6)

Oracle Home (11.1.0.6)

Oracle Home (11.1.0.6)

Logical Standby Database

Transform Redo to SQL

• Install new version of Oracle database Software • Upgrade the LSTBY using DBUA from the new ORACLE HOME • After successful upgrade check if LSTBY is in sync again

Oracle Home (11.2.0.3)

Sync or Async Redo shipping

Sync or Async Redo shipping

11.1.0.6

Page 13: Upgrade with Minimal Downtime - Oracle · Upgrade with Minimal Downtime ... • 10+ years of experience in Oracle ... (MRC) • Primarily funded by the Wellcome Trust

Redo Logs Online Redo Logs

Physical Standby Database

11.1.0.6

Primary Database

11.1.0.6

11.2.0.3

Sync or Async Redo shipping

Sync or Async Redo shipping

Local Site

DR Site

Read Write Clients Read Only Clients

Redo Logs

ASM

ASM

ASM

Broker

Backup

Redo apply

SQL apply

Oracle Home (11.1.0.6)

Oracle Home (11.1.0.6)

Oracle Home (11.1.0.6)

Logical Standby Database

Transform Redo to SQL

• Plan for switch over to LSTBY • Stop the redo transmission to PSTBY • Up to this point absolutely there is no impact on the availability of the primary database. • Switch over • Re-point the clients, services

Oracle Home (11.2.0.3)

Transform Redo to SQL

SQL apply

Page 14: Upgrade with Minimal Downtime - Oracle · Upgrade with Minimal Downtime ... • 10+ years of experience in Oracle ... (MRC) • Primarily funded by the Wellcome Trust

Phase - 2

Upgrade Standbys & Switch Back

Page 15: Upgrade with Minimal Downtime - Oracle · Upgrade with Minimal Downtime ... • 10+ years of experience in Oracle ... (MRC) • Primarily funded by the Wellcome Trust

Physical Standby Database

11.1.0.6

Primary Database

11.1.0.6

11.2.0.3

Sync or Async Redo shipping

Sync or Async Redo shipping

Local Site

DR Site

Read Write Clients Read Only Clients

Redo Logs

ASM

ASM

ASM

Broker

Backup

Redo apply

Oracle Home (11.1.0.6)

Oracle Home (11.1.0.6)

Oracle Home (11.1.0.6)

• LSTBY (previous Primary) is in a Transient state • It is Running lower version than primary • Cannot understand/apply changes received from the higher version of primary database • Cannot provide disaster recovery

Oracle Home (11.2.0.3)

Online Redo Logs

Logical Standby Database

Transform Redo to SQL

SQL apply

Page 16: Upgrade with Minimal Downtime - Oracle · Upgrade with Minimal Downtime ... • 10+ years of experience in Oracle ... (MRC) • Primarily funded by the Wellcome Trust

Physical Standby Database

11.1.0.6

Primary Database

11.1.0.6

11.2.0.3

Sync or Async Redo shipping

Sync or Async Redo shipping

Local Site

DR Site

Read Write Clients Read Only Clients

Redo Logs

ASM

ASM

ASM

Broker

Backup

Redo apply

Oracle Home (11.1.0.6)

Oracle Home (11.1.0.6)

Oracle Home (11.1.0.6)

• Flashback LSTBY to guaranteed restore point and shut down • Install new oracle version • Mount database from new home and convert LSTBY to PSTBY • When Redo Apply starts, it registers a new incarnation from the primary database

Oracle Home (11.2.0.3)

Online Redo Logs

Logical Standby Database

Transform Redo to SQL

SQL apply

Oracle Home (11.2.0.3)

Redo apply

Physical Standby Database

Page 17: Upgrade with Minimal Downtime - Oracle · Upgrade with Minimal Downtime ... • 10+ years of experience in Oracle ... (MRC) • Primarily funded by the Wellcome Trust

Physical Standby Database

11.1.0.6

Primary Database

11.2.0.3

Sync or Async Redo shipping

Sync or Async Redo shipping

Local Site

DR Site

Read Write Clients Read Only Clients

ASM

ASM

ASM

Broker

Backup

Redo apply

Oracle Home (11.1.0.6)

Oracle Home (11.1.0.6)

Oracle Home (11.1.0.6)

• Wait for physical standby to be upgraded via the redo stream • At this point you have a second interruption to the database availability • Switch Back Primary to its Original location • Re-point the clients accordingly

Oracle Home (11.2.0.3)

Online Redo Logs

Oracle Home (11.2.0.3)

Physical Standby Database

11.2.0.3

Redo Logs

Redo apply

Redo Logs

Redo apply

Redo Logs

Online Redo Logs

11.1.0.6

Page 18: Upgrade with Minimal Downtime - Oracle · Upgrade with Minimal Downtime ... • 10+ years of experience in Oracle ... (MRC) • Primarily funded by the Wellcome Trust

Physical Standby Database

11.1.0.6

Primary Database

11.2.0.3

Sync or Async Redo shipping

Sync or Async Redo shipping

Local Site

DR Site

Read Write Clients Read Only Clients

ASM

ASM

ASM

Broker

Backup

Redo apply

Oracle Home (11.1.0.6)

Oracle Home (11.1.0.6)

Oracle Home (11.1.0.6)

• Install new database home on other standbys • Mount PSTBY from new home • Flashback database to guaranteed restore point Oracle Home (11.2.0.3)

Oracle Home (11.2.0.3) Physical Standby Database

11.2.0.3 Redo Logs

Redo apply

Redo Logs

Online Redo Logs

Oracle Home (11.2.0.3)

Page 19: Upgrade with Minimal Downtime - Oracle · Upgrade with Minimal Downtime ... • 10+ years of experience in Oracle ... (MRC) • Primarily funded by the Wellcome Trust

Physical Standby Database

11.1.0.6

Primary Database

11.2.0.3

Sync or Async Redo shipping

Sync or Async Redo shipping

Local Site

DR Site

Read Write Clients Read Only Clients

ASM

ASM

ASM

Broker

Backup

Redo apply

Oracle Home (11.1.0.6)

Oracle Home (11.1.0.6)

Oracle Home (11.1.0.6)

• Enable the redo transmission to local PSTBY to catch up to new incarnation • PSTBY will be upgraded via redo stream • Enable Broker

Oracle Home (11.2.0.3)

Oracle Home (11.2.0.3) Physical Standby Database

11.2.0.3 Redo Logs

Redo apply

Redo Logs

Online Redo Logs

Oracle Home (11.2.0.3)

11.2.0.3

Page 20: Upgrade with Minimal Downtime - Oracle · Upgrade with Minimal Downtime ... • 10+ years of experience in Oracle ... (MRC) • Primarily funded by the Wellcome Trust

Physical Standby Database

Primary Database

11.2.0.3

Sync or Async Redo shipping

Sync or Async Redo shipping

Local Site

DR Site

Read Write Clients Read Only Clients

ASM

ASM

ASM

Broker

Backup

Redo apply

Oracle Home (11.1.0.6)

Oracle Home (11.1.0.6)

Oracle Home (11.1.0.6)

• Drop guaranteed restore points • Rolling upgrade has Finished and databases are back in the original positions • With very minimal interruptions Oracle Home (11.2.0.3)

Oracle Home (11.2.0.3) Physical Standby Database

11.2.0.3 Redo Logs

Redo apply

Redo Logs

Online Redo Logs

Oracle Home (11.2.0.3)

11.2.0.3

Page 21: Upgrade with Minimal Downtime - Oracle · Upgrade with Minimal Downtime ... • 10+ years of experience in Oracle ... (MRC) • Primarily funded by the Wellcome Trust

Pros & Cons Regular /Conventional Upgrade • Upgrade using DBUA/CLI

(50mins) • Upgrade using datapump -

Complete Down time required for exports, scp & imports (approx 3 hrs) – Manually recreate standbys

after the upgrade • Errors at any stage can

increase down time

Rolling Upgrade using Transient LSTBY

• Down time is required for only switchover (reduced by 98% - 5mins)

• Standbys are automatically upgraded

• Fail safe options at each step • Down time is independent of

database size

Page 22: Upgrade with Minimal Downtime - Oracle · Upgrade with Minimal Downtime ... • 10+ years of experience in Oracle ... (MRC) • Primarily funded by the Wellcome Trust

Caveats • Need to upgrade grid/ASM before the database upgrade • Need to manually update the OCR (srvctl) registries • If Upgrading from 11gR1 to 11gR2 then move the database

spfile from $OH/dbs to ASM - if not, DBUA will fail • If the “compatible” parameter is changed after LSTBY upgrade

- database cannot be rolled back • To change the “compatible” parameter after the upgrade - another downtime is needed

Page 23: Upgrade with Minimal Downtime - Oracle · Upgrade with Minimal Downtime ... • 10+ years of experience in Oracle ... (MRC) • Primarily funded by the Wellcome Trust

References • http://www.oracle.com/technetwork/database/features/availability/maa-

wp-11g-transientlogicalrollingu-1-131927.pdf • http://www.oracle.com/technetwork/database/features/availability/maa-

wp-11g-upgrades-made-easy-131972.pdf