Reducing Costs of Environment Copies using Oracle - Recovery Manager Rich Bernat ChevronTexaco Corp Session: 612.

Reducing Costs of Environment Copies using Oracle - Recovery ManagerRich BernatChevronTexaco CorpSession: 612

Agenda

Who We Are

Tuning RMAN for Performance

Q & A

How We Use RMAN

What We GainedThe ever-present

“Gotcha’s”

ChevronTexaco SAP Landscape

Why We Chose RMAN

Who am I ???

Rich Bernat

ChevronTexaco Corp.

Concord, CA (SF East Bay)

Basis Admin (HR & ITS)

Oracle DBA– Backup & Recovery– Disaster Recovery

ChevronTexaco – Our Company

• One of the world’s largest global energy companies• Active in more than 180 countries• 53,000 employees worldwide• $106 billion in revenue for 2001

Agenda

Who We Are

Tuning RMAN for Performance

Q & A

How We Use RMAN

What We GainedThe ever-present

“Gotcha’s”

ChevronTexaco SAP Landscape

Why We Chose RMAN

ChevronTexaco Landscape

• ~13 HP-UX N-class or RP database servers (1-10) DB’s per server

• ~35 development, staging, or project databases – ~6.0Tb total ranging from 80Gb to 1Tb

• 6 production databases– SAP Modules - FI, CO, HR, BW, SD

• Oracle 8.1.7.4+ in shared $ORACLE_HOME

Refresh FrequencyHow are we special…

• Staging environments are refreshed monthly for Prod Change window (or on demand)

• Project environments are refreshed on demand• Projects create, copy, refresh, restore many

times for different project phases• Several refreshes can occur at in the same day

…6+Tb of data movement!!!• Customers expect us to deliver environments in

a narrow time window.

Look what we used to do…

Take an offline backup (outage required) Restore (recover) as source DB on separate

hardware… Risk: having 2 DB’s briefly named the same

Lots of SQL to parse & rebuild control file Lots of SQL & shell scripting logic to rename

datafiles Mount point madness for sapdata’s

Had to recreate the Oracle DBID (also risky!!!)

Agenda

Who We Are

Tuning RMAN for Performance

Q & A

How We Use RMAN

What We GainedThe ever-present

“Gotcha’s”

ChevronTexaco SAP Landscape

Why We Chose RMAN

Why we needed to change…

• Projects wanted more flexibility in refresh options

• Prior method required restore to separate DB server

• Refresh of environments was spilling into project time

• Required offline backup of source DB.

• Buying more hardware was too expensive

Why we needed to change…

• Tape Capacity – Tape silo reaching capacity

• Impact of backups on Production cycles

• Reduction of downtime

What RMAN provides...

Ability to copy/create an instance on the same host!!! Huge benefit in our world!

No outage for the source environment required!!! Point-In-Time recovery for projects that want an

env from Thursday at 3:14 pm.!!!No Mountpoint madness…No controlfile worries…No DBID concerns…No copy named the same SID as source.Delegation of work to Operation Analysts and

Tech Team

RMAN Terminology

• Channel – pathway for MML I/O tape or disk• DBID – Unique Oracle database Identifier• Incarnation – Version of a Database (after

resetlogs or duplicate)• RMAN Catalog – Repository of Database

backup activity (required for DB Duplication!)• Duplexing – data streams to multiple backup

devices• Backup types – Whole, Incr 0, incr 1

RMAN Catalog

• RMAN stores data in a separate DB known as the RMAN catalog.

• It is exported with EACH backup and archivelog.

• Over time, export grows so keep enough space in temp location for export.

• These are HP Omniback II implementation ‘features’… your MML may vary.

Agenda

Who We Are

Tuning RMAN for Performance

Q & A

How We Use RMAN

What We GainedThe ever-present

“Gotcha’s”

ChevronTexaco SAP Landscape

Why We Chose RMAN

An RMAN DB Backup

rman connect target ${SRCSTRING} connect catalog ${CATSTRING}

run { allocate channel 'dev_1' type 'sbt_tape' parms BLKSIZE=524288,

ENV=(OB2BARTYPE=Oracle8,OB2APPNAME=SID,OB2BARLIST=host_SID_incr0)'; allocate channel 'dev_2' type 'sbt_tape‘ parms 'BLKSIZE=524288,

ENV=(OB2BARTYPE=Oracle8,OB2APPNAME=SID,OB2BARLIST=host_SID_incr0)'; set limit channel 'dev_1' maxopenfiles 4; set limit channel 'dev_2' maxopenfiles 4; backup incremental level 0 cumulative filesperset 24 format ‘host_SID_incr0<SID_%s:%t>.dbf' tag='1050127202' database; backup format ‘host_SID_incr0<SID_%s:%t>.dbf' current controlfile; sql 'alter database backup controlfile to trace'; resync catalog; }

And now for your archivelogs

rman connect target ${SRCSTRING} connect catalog ${CATSTRING}

run { set duplex=2; sql 'alter system archive log current'; host 'sleep 60'; allocate channel 'ch1' type 'sbt_tape' parms ENV=(OB2BARTYPE=Oracle8,OB2APPNAME=SID,OB2BARLIST=host_SID_arch%c)';

set limit channel 'ch1' maxopenfiles 4; backup filesperset 24 format ‘host_SID_arch%c<SID_%s:%t>.dbf' archivelog all delete input; resync catalog; }

Steps in a CVX Refresh

• Export SAP Security

• SM59/SMLG Settings

• Validate target filesystem size (sapdatas)

• Validate RMAN connectivity

This all happens during the day prior to refresh or create. Source env is up & active.

RMAN Processing

Typical RMAN command for Production to QA Refresh

rman connect target ${SRCSTRING} connect catalog ${CATSTRING} connect auxiliary sys/${SYSPWD}@QA1}; run {allocate auxiliary channel ch1 type 'sbt_tape'; allocate auxiliary channel ch2 type 'sbt_tape'; allocate auxiliary channel ch3 type 'sbt_tape'; allocate auxiliary channel ch4 type 'sbt_tape'; allocate auxiliary channel ch5 type 'sbt_tape'; allocate auxiliary channel ch6 type 'sbt_tape'; set until time “08-16-02 15:32:11”; duplicate target database to QA1 }

Post Processing

• Configure RMAN ID – Grant sysdba to rman;

• Configure OBII(HP Backup Utility) for new SID• Update Transport Tables• Update Printers tables• Delete Scheduled Jobs• Re-create SID-ADM user and SAPUSER table

Post Processing (Cont’d)

• Validate Oracle Users (Basis SA’s)• Delete Old Security• Import New Security• Reconfigure Taxware• Delete/Import SM59 & SMLG entries• Apply SAP license Very Cool! I hate typing the

LicenseKey– Saplicense –pinstall –ifile=license.key

• Set archivelogmode on

Voila`… a working SAP env.

• Reset profiles and operation modes via RZ04/RZ10

• Turn over to our TMS team to apply necessary transports

• Released to our customers early in the morning (depending on security and transports)

Agenda

Who We Are

Tuning RMAN for Performance

Q & A

How We Use RMAN

What We GainedThe ever-present

“Gotcha’s”

ChevronTexaco SAP Landscape

Why We Chose RMAN

RMAN “Gains”

Copy/Refresh to any server!Drastically reduced errors…

predictable availability for projectsAutomates most of refresh processReduce risk in rename processReduced tape resource consumption using

incremental backupsReduced operational impact of backup process

RMAN is not perfect… we’ve gone through some trouble… but it was worth it.

Agenda

Who We Are

Tuning RMAN for Performance

Q & A

How We Use RMAN

What We GainedThe ever-present

“Gotcha’s”

ChevronTexaco SAP Landscape

Why We Chose RMAN

RMAN Catalog Housekeeping

• Clean your catalog by – Removing old DB’s and old incarnations

• execute dbms_rcvcat.unregisterdatabase(4624689,1632315282);

– Removing expired backup entries from the catalog

• change backupset 12345 crosscheck;• change backupset 12345 delete;• Clean up old control file backups…

RMAN “Gotcha’s”

• ORA-4031 – shared pool consumed… either increase SHARED_POOL_SIZE or enable LARGE_POOL_SIZE (100M)

• Duplicate DBID… from our old DB copies… zero_dbid procedure and new controlfiles – MetaLink note: 174625.1

• Need to maintain your catalog… identify and delete old DB incarnations

More RMAN “Gotcha’s…”

• Controlfile captured at beginning of backup… also needs to be at end and with each set of archlogs.– backup format ‘your format here' current controlfile;

• Need to create sapdata directory structure prior to “Duplicate” (generated in backup step via post-processing script)

• Need to have 1 archivelog– "sql 'alter system archivelog current';" (archivelogmode)

or

– "sql 'alter system switch logfile';" (noarchivelogmode)– "host 'sleep 45';"

Agenda

Who We Are

Tuning RMAN for Performance

Q & A

How We Use RMAN

What We GainedThe ever-present

“Gotcha’s”

ChevronTexaco SAP Landscape

Why We Chose RMAN

Getting the squirrels to run!

• Balance memory consumption– (shared_pool_size or large_pool_size)

• network bandwidth - Gigabit for network devices.• Tape device speed – probably the bottleneck for

incr0 and whole!– Your bottleneck may vary

• Disk Subsystem - maximum read rates• Schedule during low impact times• Duplex devices… More = better!

Getting the squirrels to run (cont’d)

• Multiplex – filesperset ~20 – lower values create many catalog entries. Catalog space management. – disk is fast– tape and network are slow

• MaxOpenFiles = 4 Governs # of files open for backup… still allows for efficient use of FilesPerSet

• Channel Parallelism – 1 or 2 per device (nominal gain on 2 channels. 3 & more are not beneficial in incr0 or whole backup.– # of channels impacts restore speed. For faster restores, use 1

channel in your backups • Channels & filesperset consume your SGA memory• I/O buffer tuning - BLKSIZE on channel parms

How is this all done??

• Automation utilizing Unix shell scripting• Standards in DB layout & file systems on ALL

servers• Understanding the nuances of RMAN• In-depth knowledge about SAP tables• Years of experience and building on ideas of

others• Management support to try new methods• Like all our work… it has evolved

Other RMAN Uses

• Backup & Recovery (Obviously)• Disaster Recovery!!!

– Restore Oracle binaries– Restore RMAN catalog– Configure MML – RMAN

communications– Restore DB’s!

• Used for D/R drills last 2 years

Acknowledgements

• Jason Singleton – ChevronTexaco Corp.

• George Cramm – ChevronTexaco Corp.

• Joel Albert – ChevronTexaco Corp.

Agenda

Who We Are

Tuning RMAN for Performance

Q & A

How We Use RMAN

What We GainedThe ever-present

“Gotcha’s”

ChevronTexaco SAP Landscape

Why We Chose RMAN

Other Exciting Sessions…

Wed. 8:30 AM - 911A – How to do a Painless QA DB Refresh ( SAP focus) – Tracy Ferrell

Thank You!

Rich Bernat

ChevronTexaco Corp.

[email protected]

Thank you for attending!Please remember to complete and return your evaluation form following this session.

Session Code: 908-A