01/06/2018 1 Oracle Golden Gate Oracle Golden Gate Oracle Golden Gate Oracle Golden Gate . . . from AIX to Exadata . . . . . . from AIX to Exadata . . . . . . from AIX to Exadata . . . . . . from AIX to Exadata . . . Even voorstellen ...... Even voorstellen ...... Even voorstellen ...... Even voorstellen ...... 1-6-2018 2 Rob Lasonder In dienst bij Qualogy als Exadata Specialist Exadata Specialist Exadata Specialist Exadata Specialist ... Sinds 2010 bezig met Exadata (V2) ... Momenteel project X2-6 bij int. verzekeraar ... Laatste tijd vooral bezig met Golden Gate OCP: Oracle 8, 8i, 9i, 10g, 11g, 12c OCE: Exadata X3, X4, RAC11g, ZFS Storage Appliance, Big Data & GG 11g Impl. specialist Email: [email protected]www.rob.lasonder.org
64
Embed
Oracle Golden Gate - nlOUG · Oracle Golden Gate. . . from AIX to Exadata . . . Even voorstellen ..... 2 1-6-2018 Rob Lasonder In dienst bij Qualogy als Exadata SpecialistExadata
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
01/06/2018
1
Oracle Golden GateOracle Golden GateOracle Golden GateOracle Golden Gate
. . . from AIX to Exadata . . .. . . from AIX to Exadata . . .. . . from AIX to Exadata . . .. . . from AIX to Exadata . . .
Even voorstellen ......Even voorstellen ......Even voorstellen ......Even voorstellen ......
1-6-20182
Rob LasonderIn dienst bij Qualogy als Exadata SpecialistExadata SpecialistExadata SpecialistExadata Specialist
• 1 Golden Gate Use Case• 2 Golden Gate introduction• 3 Golden Gate customer architecture• 4 Golden Gate installation & configuration• 5 Golden Gate findings & troubleshooting• 6 Golden Gate Monitoring• 7 Golden Gate High Availability• 8 Lessons learned
1 Golden Gate Use Case1 Golden Gate Use Case1 Golden Gate Use Case1 Golden Gate Use Case
01/06/2018
3
1 Golden Gate Use Case1 Golden Gate Use Case1 Golden Gate Use Case1 Golden Gate Use Case
1-6-20185
• The customer purchased an Exadata 1/8 Rack and already migrated some of their databases to Exadata
• De core legacy database is too complex to migrate at this stage, the migration is planned for 2019.
• In order to already use Exadata for this core legacy database, the database will be replicated to Exadata.
• This Golden Gate Clone on Exadata will be used to build data services, mainly for reporting services.
2 Golden Gate introduction2 Golden Gate introduction2 Golden Gate introduction2 Golden Gate introduction
01/06/2018
4
2 Golden Gate Introduction : What is Golden Gate?2 Golden Gate Introduction : What is Golden Gate?2 Golden Gate Introduction : What is Golden Gate?2 Golden Gate Introduction : What is Golden Gate?
1-6-20187
Oracle GoldenGate provides lowlowlowlow----impact captureimpact captureimpact captureimpact capture, routingroutingroutingrouting, transformationtransformationtransformationtransformation, and deliverydeliverydeliverydelivery of transactional data across heterogeneousheterogeneousheterogeneousheterogeneous environments in real timereal timereal timereal time
• Golden Gate replicates data or subsets of data, in real time
• Golden Gate can transform data during the replication
• Golden Gate supports heterogenous environments• Different platforms (Linux, Windows, etc.)• Different databases (Oracle, MySQL, SQLServer, etc)• Different database versions
2 GG Introduction: Golden Gate core functionality2 GG Introduction: Golden Gate core functionality2 GG Introduction: Golden Gate core functionality2 GG Introduction: Golden Gate core functionality
1-6-20188
1 Database transactions are captured in trail files
2 Trail files are transported to target
3 Transactions are applied in the target
Real time (asynchronous) processing, Decoupled ArchitectureReal time (asynchronous) processing, Decoupled ArchitectureReal time (asynchronous) processing, Decoupled ArchitectureReal time (asynchronous) processing, Decoupled Architecture
01/06/2018
5
2 GG Introduction: available GG architectures2 GG Introduction: available GG architectures2 GG Introduction: available GG architectures2 GG Introduction: available GG architectures
2 GG Introductie: Golden Gate 12.2 New Features2 GG Introductie: Golden Gate 12.2 New Features2 GG Introductie: Golden Gate 12.2 New Features2 GG Introductie: Golden Gate 12.2 New Features
1-6-201811
• Self describing trail files: Metadata records used to interpret DML
• Oracle Data Pump Integration for Table Instantiation
• Support for Invisible Columns (Oracle Only)
• New parameter – MAPINVISIBLECOLUMNS (Integrated only)
2 GG Introductie: Golden Gate 12.2 New Features2 GG Introductie: Golden Gate 12.2 New Features2 GG Introductie: Golden Gate 12.2 New Features2 GG Introductie: Golden Gate 12.2 New Features
• Only applicable to Oracle Databases.• STREAMS_POOL_SIZE 1250 MB per Extract/Replicat process !
01/06/2018
7
2 GG Introduction: Golden Gate 12.2 New Features2 GG Introduction: Golden Gate 12.2 New Features2 GG Introduction: Golden Gate 12.2 New Features2 GG Introduction: Golden Gate 12.2 New Features
GGSCI> DBLOGIN USERIDALIAS ALIAS_GGADMINGGSCI> REGISTER EXTRACT ext1 DATABASEGGSCI> ADD EXTRACT ext1 INTEGRATED TRANLOG, BEGIN NOW
2 GG Introduction: Golden Gate 12.2 New Features2 GG Introduction: Golden Gate 12.2 New Features2 GG Introduction: Golden Gate 12.2 New Features2 GG Introduction: Golden Gate 12.2 New Features
1-6-201814
Reference: Oracle GoldenGate Performance Tuning for Oracle Database Integration CON7773
Doc in HandoutDoc in HandoutDoc in HandoutDoc in Handout
01/06/2018
8
2 GG Introduction: Golden Gate 12.2 New Features2 GG Introduction: Golden Gate 12.2 New Features2 GG Introduction: Golden Gate 12.2 New Features2 GG Introduction: Golden Gate 12.2 New Features
Oracle by Example:Oracle by Example:Oracle by Example:Oracle by Example:
Docs in HandoutDocs in HandoutDocs in HandoutDocs in Handout
2 GG Introduction: Golden Gate versus Dataguard2 GG Introduction: Golden Gate versus Dataguard2 GG Introduction: Golden Gate versus Dataguard2 GG Introduction: Golden Gate versus Dataguard
1-6-201816
Oracle ships 2 seperate products for database replication: Dataguard and Golden Gate
• Optimized for disaster recovery anddisaster recovery anddisaster recovery anddisaster recovery and data protectiondata protectiondata protectiondata protection.
• (physical) standby DB is a physical block-level copy of the database
• Part of Oracle Maximum Availability Architecture (MAA)
Oracle Golden Gate:• Optimized for database replicationreplicationreplicationreplication and datatransformation in datatransformation in datatransformation in datatransformation in
2 GG Introduction: Golden Gate versus Dataguard2 GG Introduction: Golden Gate versus Dataguard2 GG Introduction: Golden Gate versus Dataguard2 GG Introduction: Golden Gate versus Dataguard
1-6-201817
2 GG Introduction: Golden Gate versus ODI2 GG Introduction: Golden Gate versus ODI2 GG Introduction: Golden Gate versus ODI2 GG Introduction: Golden Gate versus ODI
1-6-201818
Oracle Data Integrator and Golden Gate complement each other
Oracle ODI: Oracle ODI: Oracle ODI: Oracle ODI: is based on a new technological concept called E –LT
(Extract – Load, Transform), as opposed to traditional ETL tools
Disadvantages ETL architecture: Disadvantages ETL architecture: Disadvantages ETL architecture: Disadvantages ETL architecture: • CPU/RAM / IO overhead• Data transform process at
application level
Advantages EAdvantages EAdvantages EAdvantages E----LT architecture: LT architecture: LT architecture: LT architecture: • Less CPU/RAM/IO overhead• Data transform processes at
DB level
01/06/2018
10
2 GG Introduction: Golden Gate versus ODI2 GG Introduction: Golden Gate versus ODI2 GG Introduction: Golden Gate versus ODI2 GG Introduction: Golden Gate versus ODI
1-6-201819
Best of both worlds:• Oracle Golden Gate Oracle Golden Gate Oracle Golden Gate Oracle Golden Gate provides a cross-platform data replication and
changed data capture. (Extract)• Oracle ODI is used for the transformation and loading
3 Golden Gate customer architecture3 Golden Gate customer architecture3 Golden Gate customer architecture3 Golden Gate customer architecture
01/06/2018
11
3 Golden Gate customer architecture: the “old” world3 Golden Gate customer architecture: the “old” world3 Golden Gate customer architecture: the “old” world3 Golden Gate customer architecture: the “old” world
1-6-201821
Typical characteristics:• Maintenance silo’s: network, sysadmin, storage, DBA• DBA has no control over (v)CPU & shared storage• DBA lacks required privileges by default (root, and even oracle for production)
IBM Power8 series (up to 8 threads per Power8 CPU) & IBM PowerVM IBM Power8 series (up to 8 threads per Power8 CPU) & IBM PowerVM IBM Power8 series (up to 8 threads per Power8 CPU) & IBM PowerVM IBM Power8 series (up to 8 threads per Power8 CPU) & IBM PowerVM
3 Golden Gate customer architecute : the “new” world3 Golden Gate customer architecute : the “new” world3 Golden Gate customer architecute : the “new” world3 Golden Gate customer architecute : the “new” world
1-6-201822
DB(M
)A in F
ull C
ontr
ol, T
ota
l Sta
ck M
ain
tenance
DB(M
)A in F
ull C
ontr
ol, T
ota
l Sta
ck M
ain
tenance
DB(M
)A in F
ull C
ontr
ol, T
ota
l Sta
ck M
ain
tenance
DB(M
)A in F
ull C
ontr
ol, T
ota
l Sta
ck M
ain
tenance
DB +
Sysadam
in +
Sto
rage +
Netw
ork
DB +
Sysadam
in +
Sto
rage +
Netw
ork
DB +
Sysadam
in +
Sto
rage +
Netw
ork
DB +
Sysadam
in +
Sto
rage +
Netw
ork
01/06/2018
12
3 Golden Gate customer architecture: new versus old3 Golden Gate customer architecture: new versus old3 Golden Gate customer architecture: new versus old3 Golden Gate customer architecture: new versus old
1-6-201823
• Sample of 10 report in old (AIX) and new (Exadata) DB• All 10 reports were faster on Exadata (“out-of-the-box”)• Performance gains between 1 and 181 (!) times faster on Exadata
(especially with "big volume data reports")
New versus old: realNew versus old: realNew versus old: realNew versus old: real----life comparison in customer landscape.........life comparison in customer landscape.........life comparison in customer landscape.........life comparison in customer landscape.........
3 Golden Gate customer architecture: GG Setup3 Golden Gate customer architecture: GG Setup3 Golden Gate customer architecture: GG Setup3 Golden Gate customer architecture: GG Setup
1-6-201824
GoldenGate 12.2GoldenGate 12.2GoldenGate 12.2GoldenGate 12.2• 1 technical test environment• 1 functional test environment• 1 production environment
old world (AIX)old world (AIX)old world (AIX)old world (AIX)
new world (Exadata)new world (Exadata)new world (Exadata)new world (Exadata)
01/06/2018
13
4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration
4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration
Analyse phase :Analyse phase :Analyse phase :Analyse phase :• Which data needs to be replicated (schemas, object types)• What is the replication architecture: unilateral / bilateral, etc.• Is any filtering / transformation required• Type of data changes (DML only or DML and DDL)• What is the allowed latency• Golden Gate usage, DR & High Availability requirements• Are any non supported mechanisms in place, like nologging
operations, transportable tablespaces (TTS), etc
Use Oracle CHECK scripts to investigate source dataUse Oracle CHECK scripts to investigate source dataUse Oracle CHECK scripts to investigate source dataUse Oracle CHECK scripts to investigate source data• Oracle GoldenGate database Schema Profile check script for
Oracle DB (Doc ID 1296168.1). • Oracle GoldenGate database Complete Database Profile check
script for Oracle DB (All Schemas) (Doc ID 1298562.1)
01/06/2018
15
4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration
Results analysis II : Results analysis II : Results analysis II : Results analysis II : Many tables without PMK/UK: 650+ tables !!
Problem with tables without PMK/UK:Problem with tables without PMK/UK:Problem with tables without PMK/UK:Problem with tables without PMK/UK:• On the source DB: Storage & I/O overhead PMK/UK required for
supplemental logging (alternatively all columns will be used )• On the target: Performance overhead when processing updates &
deletes => Full tablescans
Oracle has a solution for this problemOracle has a solution for this problemOracle has a solution for this problemOracle has a solution for this problemHow to Handle Tables Without Primary Keys or Unique Indexes With Oracle GoldenGate (Doc ID 1271578.1)
01/06/2018
16
4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration
Oracle Solution for tables without PMK/UK : Oracle Solution for tables without PMK/UK : Oracle Solution for tables without PMK/UK : Oracle Solution for tables without PMK/UK : How to Handle Tables Without Primary Keys or Unique Indexes With Oracle GoldenGate (Doc ID 1271578.1)
on the source database: add a column & autofill/backfill with unique valueon the source database: add a column & autofill/backfill with unique valueon the source database: add a column & autofill/backfill with unique valueon the source database: add a column & autofill/backfill with unique valueSQL> alter table table_1 add column OGG_KEY_ID raw(16);SQL> alter table table_1 modify OGG_KEY_ID default sys_guid();
Configure this column at GG Level as the unique identifier:Configure this column at GG Level as the unique identifier:Configure this column at GG Level as the unique identifier:Configure this column at GG Level as the unique identifier:GGSCI> add trandata <owner>.<table_name>, COLS (OGG_ KEY_ID), nokey
on the target database: on the target database: on the target database: on the target database: Create a UNIQUE index on this columnCreate a UNIQUE index on this columnCreate a UNIQUE index on this columnCreate a UNIQUE index on this columnSQL> create unique index ix_table_1 on table_1 (OGG_KEY_ID);
4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration
Compilation in many stored procedures: ORA-00947: not enough valuesWe did not even try the Forms & other applications.
This error occurs when you do not specify all the columns in your coding:Bad coding: INSERT INTO TABLE_1 VALUES ('A','B');Good coding: INSERT INTO TABLE_1 (COL1, COL2) VALUE S ('A','B');
SQL> alter table HR.TABLE_1 modify OGG_KEY_ID invis ible
4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration
1-6-2018Agenda37
Install GGInstall GGInstall GGInstall GG
4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration
1-6-201838
Prepare the Prepare the Prepare the Prepare the targettargettargettarget
1 Create the target database
2 Prepare database settings for Golden Gate
3 Create GGADMIN user and related objects
4 Start GG Manager process
5 Configure & start the GG Manager process
01/06/2018
20
4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration
1-6-201839
Prepare the Prepare the Prepare the Prepare the sourcesourcesourcesource
1 Size archive log destination (increased due to Supplemental Logging
2 Create local trail file (/oracle/trail) destination, ++ sufficient size
3 Prepare database settings for Golden Gate
4 Create GGADMIN user and related objects
5 Enable Supplemental Logging
6 Configure & Start GG Manager process
7 Configure & Start GG Extract process
8 Configure & Start GG Pump process
potential DB Impactpotential DB Impactpotential DB Impactpotential DB Impact
potential DB Impactpotential DB Impactpotential DB Impactpotential DB Impact
4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration
1-6-201840
Prepare the Prepare the Prepare the Prepare the sourcesourcesourcesource
Some details step 3: Prepare settings for source databaseSome details step 3: Prepare settings for source databaseSome details step 3: Prepare settings for source databaseSome details step 3: Prepare settings for source database
SQL> alter database force logging;SQL> alter database add supplemental log data;
SQL> alter system set undo_retention=86400 scope=bo th;SQL> alter system set "_log_read_buffers"=64 scope= spfile; SQL> alter system set "_log_read_buffer_size" = 128 scope=spfile; SQL> alter system set enable_goldengate_replication = true;SQL> alter system set shared_pool_size = <current_v alue + 1250M>;SQL> alter system set sga_target = <current_value + 1250M>;
Database in force logging !! May have effects on batch operations. Check for unrecoverable actions (v$datafile.unrecoverable_change#)
Several parameter changes. More memory required for integrated extract (logminer) processes.
01/06/2018
21
4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration
1-6-2018Aenda41
Prepare the Prepare the Prepare the Prepare the sourcesourcesourcesource
The data stored in the online redo logfiles contain the ROWIDROWIDROWIDROWID of the row being changed and the new values for the column(s) that are changed. The ROWID is used to identify the changed row.
The ROWID is a mapping to a physical block address and cannot be used by Golden Gate to identify the changed row. Therefore, the PMK or UK are added to the online redo logfile as supplemental logging.
Supplemental logging can take place at the table level or at the schema level:
Redo size increased by 60% due to Supplemental Logging. Reason is the many tables without PMK / UK
Supplemental logging was (initially) implemented at the SCHEMA level
01/06/2018
22
4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration
1-6-201843
Prepare the Prepare the Prepare the Prepare the sourcesourcesourcesource
Advantages of enabling supplemental logging at the schema levelAdvantages of enabling supplemental logging at the schema levelAdvantages of enabling supplemental logging at the schema levelAdvantages of enabling supplemental logging at the schema level
• Automatically enables supplemental logging for new tables created with a CREATE TABLE DDL command.
• Updates supplemental logging for tables affected by an ALTER TABLE DDL command that adds or drops columns
• Updates supplemental logging for tables affected by RENAME TABLE command
• Updates supplemental logging for tables affected by adding or dropping of unique or primary key constraints
4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration
1-6-201844
Prepare the Prepare the Prepare the Prepare the sourcesourcesourcesource
Some details step 7: Configure the Extract process parameter fileSome details step 7: Configure the Extract process parameter fileSome details step 7: Configure the Extract process parameter fileSome details step 7: Configure the Extract process parameter file
-- include DDL-- MAPPED means we only replicat DDL from the mapped objects-- and we only want tables and indexes, no other obj ect typesDDL &INCLUDE MAPPED OBJTYPE TABLE &INCLUDE MAPPED OBJTYPE INDEX
-- we want all the tables from user HRtable HR.*
-- etc. More specifications & options.
01/06/2018
23
4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration
1-6-201845
Prepare the Prepare the Prepare the Prepare the sourcesourcesourcesource
Some details step 7: Configure & start the Extract ProcessSome details step 7: Configure & start the Extract ProcessSome details step 7: Configure & start the Extract ProcessSome details step 7: Configure & start the Extract Process
GGSCI> dblogin useridalias alias_ggadminSuccessfully logged into database.
GGSCI> register extract X1PROD database2017-09-14 12:43:29 INFO OGG-02003 Extract X1P ROD successfully registered with database at SCN 672577 3753243.
GGSCI> add extract X1PROD, integrated tranlog, begi n now
4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration
1-6-201846
Prepare the Prepare the Prepare the Prepare the sourcesourcesourcesource
Some details step 7: Local trail files Some details step 7: Local trail files Some details step 7: Local trail files Some details step 7: Local trail files afterafterafterafter Extract is started....Extract is started....Extract is started....Extract is started....
01/06/2018
24
4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration
1-6-201847
Prepare the Prepare the Prepare the Prepare the sourcesourcesourcesource
Some details Step 8, remote trail files Some details Step 8, remote trail files Some details Step 8, remote trail files Some details Step 8, remote trail files afterafterafterafter Pump is configured & started:Pump is configured & started:Pump is configured & started:Pump is configured & started:
4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration
1-6-201848
Prepare the Prepare the Prepare the Prepare the sourcesourcesourcesource
Status of the Golden Gate processes on the source:Status of the Golden Gate processes on the source:Status of the Golden Gate processes on the source:Status of the Golden Gate processes on the source:
GGSCI> info allProgram Status Group Lag at Chkpt T ime Since ChkptMANAGER RUNNINGJAGENT STOPPEDEXTRACT RUNNING P1PROD 00:12:18 0 0:00:04EXTRACT RUNNING X1PROD 00:00:04 0 0:00:15
01/06/2018
25
4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration
1-6-201849
Prepare the Prepare the Prepare the Prepare the sourcesourcesourcesource
After the prepare source operation part of the Golden Gate configuration is up and running:• MGR, Extract & Pump processes running on source• Local trail files are being written• Local trail files are being transferred to remote trail
4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration
1-6-201850
Instantiation: inital copy of the source database + replication startInstantiation: inital copy of the source database + replication startInstantiation: inital copy of the source database + replication startInstantiation: inital copy of the source database + replication start
• Several methods available: GG, SQL*Loader, Oracle Datapump• We use Oracle Datapump because of performance• Instantiation needs to be a consistent, automated and well
performing operation:• Functional test environment (source DB) is being cloned from
Production every 3 months, and GG needs to be re-instantiated.• In case of very serious issues, we want to be able to re-
instantiate the Golden Gate Production target quickly
ObjectivesObjectivesObjectivesObjectives: • (re)instantiate the target database in maximum 8 hours.
(Source DB is 2.111 GB)• Impact on Production should be as little as possible.• (re)instantiation should run as an automated batch job
No DDL allowed during instantiation !! Prevent error below:ORA-01466: Unable to read data -- Table definition h as changed" (during the export)
01/06/2018
26
4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration
1-6-201851
Instantiation architecure: NO (!) impact on source databaseInstantiation architecure: NO (!) impact on source databaseInstantiation architecure: NO (!) impact on source databaseInstantiation architecure: NO (!) impact on source database
4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration
1-6-201855
Start the replicat Start the replicat Start the replicat Start the replicat afterafterafterafter the IMPORT operationthe IMPORT operationthe IMPORT operationthe IMPORT operation
GGSCI> register replicat rprod database2017-08-23 13:08:44 INFO OGG-02528 REPLICAT RPROD successfully registered with database as inbound server OGG$RPROD.
4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration
1-6-201856
Everything is scripted and automated runs in Batch SchedulerEverything is scripted and automated runs in Batch SchedulerEverything is scripted and automated runs in Batch SchedulerEverything is scripted and automated runs in Batch Scheduler
. . . Mitigate the whole chain when issues occur. . . Mitigate the whole chain when issues occur. . . Mitigate the whole chain when issues occur. . . Mitigate the whole chain when issues occur
Major issue 1: Issues with supplemental logging on the sourceMajor issue 1: Issues with supplemental logging on the sourceMajor issue 1: Issues with supplemental logging on the sourceMajor issue 1: Issues with supplemental logging on the source
ORA-22328: object "HR"."GENERIC_OTC_QUEUE_TABLE" ha s errors. ORA-62009: An attempt to encode invalid XML charact er 0x4 in supplemental log has occurred.
Even though we excluded the Advanced Queuing tables from the Extract process, this error still occurred, because supplemental logging was enabled at the SCHEMA level, including these AQ tables inside the HR schema.
There is NO solution for this problem. You cannot exclude objects when you add supplemental logging at the SCHEMA Level.⇒ Confirmed by Oracle (Service Request)⇒ Enhancement request rejected. (works as designed)
Major issue 1: Issues with supplemental logging on the sourceMajor issue 1: Issues with supplemental logging on the sourceMajor issue 1: Issues with supplemental logging on the sourceMajor issue 1: Issues with supplemental logging on the source
The best solution would be to move the AQ$ tables to another schema. However, this requires code & application changes & testing. Impact analysis planned for next year.
For now: move from SCHEMA level logging to TABLE level logging. GGSCI> add trandata HR.table_1 (instead of add sche matrandata HR)
In Extract parameter file: map every single table. In stead of HR.*;TABLE HR.table_1;TABLE HR.table_2;.......
Disadvantate of this approach:Disadvantate of this approach:Disadvantate of this approach:Disadvantate of this approach:1. GG extract configuration much more complex. 2000+ tables listed.2. New tables are not automatically added to the GG replication.3. Certain DDL operations are not supported with table level logging:
Major issue 2: Very bad performance due to tables without PMK/UKMajor issue 2: Very bad performance due to tables without PMK/UKMajor issue 2: Very bad performance due to tables without PMK/UKMajor issue 2: Very bad performance due to tables without PMK/UK
• Tables without PMK/UK on source were initially ignored. (No adequate solution with invisible OGG_KEY_ID column)
• However, they cause very bad performance on the target, during updates and deletes: => full tablescans for every (grouped) transaction.
Major issue 2: Very bad performance due to tables without PMK/UKMajor issue 2: Very bad performance due to tables without PMK/UKMajor issue 2: Very bad performance due to tables without PMK/UKMajor issue 2: Very bad performance due to tables without PMK/UK
Major issue 2: Very bad performance due to tables without PMK/UKMajor issue 2: Very bad performance due to tables without PMK/UKMajor issue 2: Very bad performance due to tables without PMK/UKMajor issue 2: Very bad performance due to tables without PMK/UK
Major issue 2: Very bad performance due to tables without PMK/UKMajor issue 2: Very bad performance due to tables without PMK/UKMajor issue 2: Very bad performance due to tables without PMK/UKMajor issue 2: Very bad performance due to tables without PMK/UK
Subsequent table fragmentation causes performance to degrade even worse !!!
Major issue 2: Very bad performance due to tables without PMK/UKMajor issue 2: Very bad performance due to tables without PMK/UKMajor issue 2: Very bad performance due to tables without PMK/UKMajor issue 2: Very bad performance due to tables without PMK/UK
Permanent solution (impact analysis planned for next years program) ⇒ create PMK/UK or implement visible OGG_KEY_ID⇒ Requires code changes and extensive testing
Temporary workarounds 1: exclude tables from the GG replicationTemporary workarounds 1: exclude tables from the GG replicationTemporary workarounds 1: exclude tables from the GG replicationTemporary workarounds 1: exclude tables from the GG replication=> Discussion with data architects.=> Discussion with data architects.=> Discussion with data architects.=> Discussion with data architects.
Temporary workarounds 2: define logical keysTemporary workarounds 2: define logical keysTemporary workarounds 2: define logical keysTemporary workarounds 2: define logical keys=> Discussion with data architect to identify tables=> Discussion with data architect to identify tables=> Discussion with data architect to identify tables=> Discussion with data architect to identify tables
Temporary workarounds 3: use IGNOREDELETES for “snapshot tables”Temporary workarounds 3: use IGNOREDELETES for “snapshot tables”Temporary workarounds 3: use IGNOREDELETES for “snapshot tables”Temporary workarounds 3: use IGNOREDELETES for “snapshot tables”=> Discussion with data architect to identify tables=> Discussion with data architect to identify tables=> Discussion with data architect to identify tables=> Discussion with data architect to identify tables
Temporary workarounds 4: create all column index on remaining tables Temporary workarounds 4: create all column index on remaining tables Temporary workarounds 4: create all column index on remaining tables Temporary workarounds 4: create all column index on remaining tables (on target) (on target) (on target) (on target)
Major issue 2: Very bad performance due to tables without PMK/UKMajor issue 2: Very bad performance due to tables without PMK/UKMajor issue 2: Very bad performance due to tables without PMK/UKMajor issue 2: Very bad performance due to tables without PMK/UK
Table Analasis sheet, with colour coded “solutions”/workarounds
Major issue 2: Very bad performance due to tables without PMK/UKMajor issue 2: Very bad performance due to tables without PMK/UKMajor issue 2: Very bad performance due to tables without PMK/UKMajor issue 2: Very bad performance due to tables without PMK/UK..... Workaround with logical keys..... Workaround with logical keys..... Workaround with logical keys..... Workaround with logical keys
On the source:GGSCI> add trandata HR.TABLE_1 COLS (A,B) NOKEY
In extract parameter file: table HR.TABLE_1 (A,B);
On target:SQL> Create unique index hr.ix_table_1 on hr.table_1 (A,B);
In replicat parameter file:MAP HR.TABLE_1, target HR.TABLE_1, KEYCOLS (A,B);
Major issue 2: Very bad performance due to tables without PMK/UKMajor issue 2: Very bad performance due to tables without PMK/UKMajor issue 2: Very bad performance due to tables without PMK/UKMajor issue 2: Very bad performance due to tables without PMK/UK..... Workaround with ignoredeletes..... Workaround with ignoredeletes..... Workaround with ignoredeletes..... Workaround with ignoredeletes
• Only updates and deletes are affected negatively. Inserts are fine• Customer has ‘snapshot’ tables which only contains inserts and
periodic deletes (overnight, cleaning up old entries)
Solution: ignore the deletes and perform them on the target outside of the Golden Gate scope, in a nightly batch. Configure this in the source extract parameter file:
. . . Several minor issues, some examples. . .. . . Several minor issues, some examples. . .. . . Several minor issues, some examples. . .. . . Several minor issues, some examples. . .
1-6-201871
Example 1: Performance optimization
Example 2: DDL Error, table with added column
WARNING OGG-01431 Aborted grouped transaction on 'HR.TBBU_REPROCESS_BUYER_STATUSES', Mapping error
. . . Several minor issues: example 1 performance optimization. . . Several minor issues: example 1 performance optimization. . . Several minor issues: example 1 performance optimization. . . Several minor issues: example 1 performance optimization
. . . Several minor issues: example 1 performance optimization. . . Several minor issues: example 1 performance optimization. . . Several minor issues: example 1 performance optimization. . . Several minor issues: example 1 performance optimization
1-6-201873
Changing parameters
EXTRACT-- to minimize the latency during low volume changesEOFDELAYCSECS 1FLUSHCSECS 1
PUMP-- to minimize the latency during low volume changesEOFDELAYCSECS 1FLUSHCSECS 1
REPLICAT-- to minimize the latency during low volume changesEOFDELAYCSECS 1
Reference: How can I optimize (reduce delay, lag ) throughput from NSK to a target platform in a low volume situation. (Doc ID 1489540.1)
. . . Several minor issues: example 1 performance optimization. . . Several minor issues: example 1 performance optimization. . . Several minor issues: example 1 performance optimization. . . Several minor issues: example 1 performance optimization
. . . Several minor issues: example 2 new column added. . . Several minor issues: example 2 new column added. . . Several minor issues: example 2 new column added. . . Several minor issues: example 2 new column added
1-6-201875
Normally, new columns can be added to a table, and this DDL is captured and applied by Golden Gate.
However, when it concerns a table with no PMK or UK:• as a consequence, all columns are required as supplemental logging. • In those cases, the new column is not automatically added to the
supplemental log group.
Solution:⇒ Temporarily exclude the table from the Replicat (tableexclude
HR.TABLE_1;) and restart the replicat. ⇒ Process all the “bad” trails with the missing column information.⇒ Add the new column as Supplemental Logging on the source⇒ Reinstatiate the new table, remove the tableexclude from the
Replicat and restart the Replicat
WARNING OGG-01431 Aborted grouped transaction on 'HR.TBBU_REPROCESS_BUYER_STATUSES', Mapping error
. . . Several minor issues: example 3 system generated names. . . Several minor issues: example 3 system generated names. . . Several minor issues: example 3 system generated names. . . Several minor issues: example 3 system generated names
Try to avoid these type of errors by naming your constraints !!
Use Replicat Error Handling to handle this type of errors:-- DDL ErrorsDDLERROR DEFAULT ABENDDDLERROR 2443 IGNORE
Monitor your Replicat report files and discard files errors !!
Oracle claims in an Oracle note that you can treat DML & DDL errors with the same error handling clause (REPERROR). I did not get it to work. In stead:DDL Errors => DDLERROR clauseDML Errors => REPERROR clause
. . . Several minor issues: example 3 system generated names. . . Several minor issues: example 3 system generated names. . . Several minor issues: example 3 system generated names. . . Several minor issues: example 3 system generated names
1-6-201877
Sample section in Replicat parameter file to handle DML errors
6 Golden Gate monitoring6 Golden Gate monitoring6 Golden Gate monitoring6 Golden Gate monitoring
01/06/2018
40
6 Golden Gate monitoring6 Golden Gate monitoring6 Golden Gate monitoring6 Golden Gate monitoring
Challenge:• Report on current Golden Gate status• Alert if the replication lag increases or GG processes are down• No access (as Oracle) to source database server. Only PUID access
Detailed work instruction in HandoutDetailed work instruction in HandoutDetailed work instruction in HandoutDetailed work instruction in Handout
Customization 1: making reports accessible for PUIDCustomization 2: remote monitoringCustumization 3: monitoring replication lagCustomization 4: reporting on Golden Gate Statistics
01/06/2018
47
6 Golden Gate monitoring6 Golden Gate monitoring6 Golden Gate monitoring6 Golden Gate monitoring
. . . . . . . . . . . . Customization 1: making reports availableCustomization 1: making reports availableCustomization 1: making reports availableCustomization 1: making reports available
1-6-201893
-- parameter setting in ./GLOBALS on source -- allows user read access to the report files-- overriding the default (026)OUTPUTFILEUMASK 022
6 Golden Gate monitoring6 Golden Gate monitoring6 Golden Gate monitoring6 Golden Gate monitoring
. . . . . . . . . . . . Customization 1: making reports availableCustomization 1: making reports availableCustomization 1: making reports availableCustomization 1: making reports available
1-6-201894
01/06/2018
48
6 Golden Gate monitoring6 Golden Gate monitoring6 Golden Gate monitoring6 Golden Gate monitoring
External tables call OS scripts:preprocessor GG_DIR2:'gg_status_02.sh '
SQL> querySQL> querySQL> querySQL> query
Advantage of this setup: we do not need access (as oracle) the source system to view the Advantage of this setup: we do not need access (as oracle) the source system to view the Advantage of this setup: we do not need access (as oracle) the source system to view the Advantage of this setup: we do not need access (as oracle) the source system to view the current GG statuscurrent GG statuscurrent GG statuscurrent GG status
DB LINK GG_SYMPUser GGADMIN_READ with Read Acces to External Table
Detailed work instruction in HandoutDetailed work instruction in HandoutDetailed work instruction in HandoutDetailed work instruction in Handout
6 Golden Gate monitoring6 Golden Gate monitoring6 Golden Gate monitoring6 Golden Gate monitoring
ON THE SOURCE• Create Heartbeat tables on source with an automatically updated
insert_on_source date/time column• Create some dependant objects (sequences). • Create DBMS_SCHEDULER_JOBS to insert records every minute
ON THE TARGET• Create a trigger to update the insert_on_target date/time column• Enable the trigger (triggers are by default disabled by GoldenGate)• Create some reports based on the hearbeat tables
6 Golden Gate monitoring6 Golden Gate monitoring6 Golden Gate monitoring6 Golden Gate monitoring
6 Golden Gate High Availability6 Golden Gate High Availability6 Golden Gate High Availability6 Golden Gate High Availability
. . . Relevant Golden Gate processes for High Availability. . . Relevant Golden Gate processes for High Availability. . . Relevant Golden Gate processes for High Availability. . . Relevant Golden Gate processes for High Availability
1-6-2018105
• The only process we have to care about (when configured correctly) is the ManagerManagerManagerManager process
• The Extract, Pump and Replicat Extract, Pump and Replicat Extract, Pump and Replicat Extract, Pump and Replicat processes are managed by the Manager process (via AUTOSTART & AUTORESTART parameter)
• But … the manager process should be managed by some external mechanism
• On AIX: startup scripts• On Exadata: by Oracle Clusterware
6 Golden Gate High Availability6 Golden Gate High Availability6 Golden Gate High Availability6 Golden Gate High Availability
. . . Relevant Golden Gate processes for High Availability. . . Relevant Golden Gate processes for High Availability. . . Relevant Golden Gate processes for High Availability. . . Relevant Golden Gate processes for High Availability
1-6-2018106
• The only process we have to care about (when configured correctly) is the ManagerManagerManagerManager process
• The Extract, Pump and Replicat Extract, Pump and Replicat Extract, Pump and Replicat Extract, Pump and Replicat processes are managed by the Manager process
• But … the manager process should be managed by some external mechanism
• On AIX: startup scripts• On Exadata: by Oracle Clusterware
01/06/2018
54
6 Golden Gate High Availability6 Golden Gate High Availability6 Golden Gate High Availability6 Golden Gate High Availability
. . . Golden Gate extract and replicat processes. . . Golden Gate extract and replicat processes. . . Golden Gate extract and replicat processes. . . Golden Gate extract and replicat processes
1-6-2018107
• The GG Manager (can) start / restart the Extract and Replicat processes.
• This is configured with parameters in the MANAGER parameter file:
AUTOSTART ER *-- Starts Extract and Replicat processes when Manage r starts
AUTORESTART ER *, retries 12, WAITMINUTES 5, RESETM INUTES 60-- restarts Extract and Replicat processes again aft er abnormal termination.
HA events are recorded in the ggserr.log file and m anager report file:
6 Golden Gate High Availability6 Golden Gate High Availability6 Golden Gate High Availability6 Golden Gate High Availability
. . . Golden Gate High Availability on the target (Exadata). . . Golden Gate High Availability on the target (Exadata). . . Golden Gate High Availability on the target (Exadata). . . Golden Gate High Availability on the target (Exadata)
1-6-2018108
Application VIPApplication VIPApplication VIPApplication VIP• Separate IP Address on public network (10 Gb/s)• Added to DNS as dm1db01-appvip
Application resource ggate_prod• hard dependency on Application VIP (pullup,
shutdown)• action script to stop/start/check gg processes
01/06/2018
55
6 Golden Gate High Availability6 Golden Gate High Availability6 Golden Gate High Availability6 Golden Gate High Availability
Detailed work instruction in HandoutDetailed work instruction in HandoutDetailed work instruction in HandoutDetailed work instruction in Handout
6 Golden Gate High Availability6 Golden Gate High Availability6 Golden Gate High Availability6 Golden Gate High Availability
. . . Golden Gate Failover steps. . . Golden Gate Failover steps. . . Golden Gate Failover steps. . . Golden Gate Failover steps
1-6-2018110
1 Check current status of cluster resources
2 Check current status of GG processes
01/06/2018
56
6 Golden Gate High Availability6 Golden Gate High Availability6 Golden Gate High Availability6 Golden Gate High Availability
. . . Golden Gate Failover steps. . . Golden Gate Failover steps. . . Golden Gate Failover steps. . . Golden Gate Failover steps
1-6-2018111
3 Perform the failover
4 Check the GG processes
The Golden Gate PUMP process on the source will temporarily ABEND, but will be restarted automatically by the Manager process on the source.
6 Golden Gate High Availability6 Golden Gate High Availability6 Golden Gate High Availability6 Golden Gate High Availability
. . . Golden Gate Failover, confusing ...... . . Golden Gate Failover, confusing ...... . . Golden Gate Failover, confusing ...... . . Golden Gate Failover, confusing .....
1-6-2018112
MGR and Replicat processes appear to be running on both nodes simultaneously ?????
01/06/2018
57
6 Golden Gate High Availability6 Golden Gate High Availability6 Golden Gate High Availability6 Golden Gate High Availability
. . . Golden Gate Failover, confusing ...... . . Golden Gate Failover, confusing ...... . . Golden Gate Failover, confusing ...... . . Golden Gate Failover, confusing .....
1-6-2018113
The GGSCI Interface uses the shared checkpoint and process locations on the ZFS-SA to display this information. The ACTUAL GG processes are only running on only one node
Ps –ef|grep mgr on first node
Ps –ef|grep mgr on second node
6 Golden Gate High Availability6 Golden Gate High Availability6 Golden Gate High Availability6 Golden Gate High Availability
. . . Golden Gate Failover steps. . . Golden Gate Failover steps. . . Golden Gate Failover steps. . . Golden Gate Failover steps
1-6-2018114
3 Perform the failover
4 Check the GG processes
The Golden Gate PUMP process on the source will temporarily ABEND, but will be restarted automatically by the Manager process on the source.
Lessons Learned 1: DBMA not for every DBALessons Learned 1: DBMA not for every DBALessons Learned 1: DBMA not for every DBALessons Learned 1: DBMA not for every DBA
The Exadata Platform requires a scaling up and scaling out of the
Lessons Learned 2: Make use of existing structuresLessons Learned 2: Make use of existing structuresLessons Learned 2: Make use of existing structuresLessons Learned 2: Make use of existing structures
Make use of existing structures & procedures already in place at
the customer site.
• It will save you work
• It will change “opponents” into supporters
Examples applied in this project:
• Existing snapshot & cloning procedures were used for GG
database instantiation
• Batch scheduling team was asked to run & monitor GG batches
1-6-2018117
Lessons Learned 3: Make use of Oracle Support !!Lessons Learned 3: Make use of Oracle Support !!Lessons Learned 3: Make use of Oracle Support !!Lessons Learned 3: Make use of Oracle Support !!
• When working on technical issues, make use of Oracle Support
to increase your project team.
• Once in place, it only takes about 20 minutes to create a
Service Request.
• Quality of Oracle Support analysists varies heavily.
(example: one SR (not being able to discover GG in OEM) took 2 months
with hardly any progress. Then support analyst went on holiday and
colleague took over. Within a few days the SR was resolved.
1-6-2018118
01/06/2018
60
Lessons Learned 4: Document the solution !!Lessons Learned 4: Document the solution !!Lessons Learned 4: Document the solution !!Lessons Learned 4: Document the solution !!
1-6-2018119
Preferably on a WIKI
Lessons Learned 5: Data analysis of source DB !!Lessons Learned 5: Data analysis of source DB !!Lessons Learned 5: Data analysis of source DB !!Lessons Learned 5: Data analysis of source DB !!
1-6-2018120
Perform the data analysis of the source DB first, with Oracle provided scripts. Reports on Golden Gate challenges:• Non supported objects (like AQ$ tables)• Tables without PMK/UK• Non supported datatypes (like IDENTITY columns)• Etc.
Try and avoid making promises up front, do a POC
Contact the data administrators who know their data and discuss the outcome and possible solutions with them.
References:Oracle GoldenGate database Schema Profile check script for Oracle DB (Doc ID 1296168.1). ScriptOracle GoldenGate database Complete Database Profile check script for Oracle DB (All Schemas) (Doc ID 1298562.1) GoldenGate Integrated Capture and Integrated Replicat Healthcheck Script (Doc ID 1448324.1
If possible, choose an incremental approach and evaluate every step over time:• Place source DB in archive log & force logging mode• Add supplemental logging to the source DB• Configure and start the extract process• Configure and start the pump process• Instantiate the target database• Automate the instantiation• Configure High Availability• Configure monitoring & reporting
• Configure the TESTTESTTESTTEST environment first, then PRODUCTIONPRODUCTIONPRODUCTIONPRODUCTION
Lessons Learned 7: Take time to work out the solutionLessons Learned 7: Take time to work out the solutionLessons Learned 7: Take time to work out the solutionLessons Learned 7: Take time to work out the solution
1-6-2018122
Take time to work out the solution.
As soon as you are outside the “Golden Gate sweetspot”, you are much on your own, with little to no help from Oracle Support.
Documentation on internet/My Oracle Support hard to find. Available documentation just touches the surface.
01/06/2018
62
Lessons Learned 8: Supersize if possibleLessons Learned 8: Supersize if possibleLessons Learned 8: Supersize if possibleLessons Learned 8: Supersize if possible
1-6-2018123
Try and supersize your Golden Gate trail file locations, both the local trail and the remote trail.
This will allow you room to manage Golden Gate issues. While you fix the issues, there is still sufficient storage for Golden Gate to keep on shipping trail files.
In this Customer setup:• GG local trail file location with 200 GB of storage. Enough for +10
days of data changes• GG remote trail file location of “unlimited” storage capacity (hosted
GoldenGate Extract and Pump processes turned out to be VERY stable.
GoldenGate Manager process monitors other processes VERY succesfully. (For example AUTORESTART Pump after network issue)
Most problems occur with the GoldenGate Replicat processes.⇒ Important to configure CDR (Conflict Detection & Resolution) in
Replicat parameter file⇒ Important to have sufficient remote trail file storage. While error
is being analyzed and resolved, remote trail file processing stalls and new trails keep coming !
01/06/2018
63
Lessons Learned 10: Reporting is importantLessons Learned 10: Reporting is importantLessons Learned 10: Reporting is importantLessons Learned 10: Reporting is important
1-6-2018125
Customers like to know the status of their GG replication. Take time to build some custom reports:• Golden Gate availability• Golden Gate lag (latency) => use the Heartbeat tables• Golden Gate table stats => use GV$GOLDENGATE_TABLE_STATS
Do not promise the impossible! For real up to date reporting, products like GoldenGate Veridata need to be purchased.
DBA and developers should be aware of the consequences of GoldenGate and adopt their standards:• No NOLOGGING operations allowed• No Transportable Tablespace methods allowed• Always create tables with PMK or Unique Key• Make sure all new objects within the replicated schema(s) are
supported by GoldenGate• Do not use objects that are not supported by GoldenGate, like AQ$
tables and IDENTITY columns• Name your constraints and indexes, avoid system generated names.• Etc.
01/06/2018
64
End of PresentationEnd of PresentationEnd of PresentationEnd of Presentation