Oracle 11g Real Application Testing: Avoiding Performance Regressions with SQL Performance Analyzer Khaled Yagoub, Pete Belknap, Benoit Dageville, Karl.

<Insert Picture Here>

Oracle 11g Real Application Testing: Avoiding Performance Regressions with SQL Performance Analyzer

Khaled Yagoub, Pete Belknap, Benoit Dageville, Karl DiasDatabase Manageability Group

<Insert Picture Here>

Presentation Outline

• SQL Performance Analyzer Overview

• Common Usage Scenarios

• Workflow Phases

• SQL Performance Analyzer Interface

• Enterprise Manager

• Command Line

• Q/A

SQL Performance Analyzer: Overview

• Helps users predict the impact of system changes on SQL workload response time

• Builds different versions of SQL workload performance (i.e., SQL execution plans and statistics)

• Analyzes performance differences

• Offers fine-grained performance analysis on individual SQL

• Integrated with SQL tuning set, SQL plan baselines, and SQL tuning advisor to form an End-to-end solution

Analysis Report

Compare

SQL Performance

SQL plans + stats SQL plans + stats

Pre-change Post-change

SQL Workload

(SQL text + binds)

Common Usage Scenarios

• Database upgrades and patches • Database parameter changes• Schema changes• Statistics gathering • Implementation of tuning recommendations• OS/hardware changes

SQL Performance Analyzer: Workflow

Production Test

Capture SQL (STS)

Transport STS

Execute SQL Pre-change

Execute SQL Post-change

Compare Perf.

Steps (1) (2) (3) (4) (5)

(6) Reiterate

(7)

No

Yes

(7)

Done?

Make Change

Production Change / Tuning Deployment

Tuned System

Step 1: Capture SQL Workload

• SQL Tuning Set (STS) used to store & manage SQL workload

• STS includes complete execution context

• SQL Text, bind variables, parsing schema

• Execution plans and run-time statistics

• Number of executions

• STS can be built from AWR, Cursor Cache, user workload and existing STS

• Incremental capture used to populate STS from cursor cache over a time period

• Negligible performance overhead (<2 %)

• Filtering and ranking capabilities filters out undesirable SQL

Production Database

Cursor Cache

SQL Tuning Set

Incremental Capture

Step 2: Move SQL Workload to Test SystemProduction

Cursor Cache

SQL Tuning Set

Test

SQL Tuning SetExport/Import

• Test system should resemble production (stats… )

• Copy SQL tuning set to staging table (“pack”)

• Transport staging table to test system (datapump, db link, etc.)

• Copy SQL tuning set from staging table (“unpack”)

Step 3: Execute SQL Before Making Change• Establishes SQL workload performance

baseline before change

• Each SQL “test-executed” once in isolation with production context

• SQL Trial: SQL execution plans and run-time statistics captured

• DDL skipped

• Only query part of DML is executed

• Customizable global and per-SQL time limit to handle long running/run-away SQL

• Option to do Explain Plan only analysis

• Remote test-execute using database link (on 10.2 and up)

SQL Tuning Set

Fetch Next SQL

SQL Performance Analyzer

Execution Plan &

Statistics

Test Execute

Save Result

Trial:

Before Change

Step 4: Execute SQL After Making Change

• Re-execute SQL after change

• Gathers new SQL execution plans and run-time statistics (SQL Trial)

SQL Tuning Set

SQL Performance Analyzer

Trial:

Before Change

Fetch Next SQL

Execution Plan &

Statistics

Test Execute

Save Result

Trial:

After Change

Completed

• Manually implement the planned change • Database upgrade, patches

• Optimizer statistics refresh

• Schema changes

• Database parameter changes

• Validate tuning actions, e.g., SQL Profile creation, Partitioning, etc.

• Hardware changes, e.g., Enabling direct IO, File System to ASM conversion, etc.

Step 5: Compare & Analyze Performance

Trail:

Before Change

Completed

Trial:

After Change

Completed

Compare

SQL Performance

SQL Performance Analyzer

Analysis Report

• Compare performance using different metrics, e.g.,

• Buffer Gets• CPU Time• Elapsed Time• Optimizer Cost

• Weigh SQL using original number of executions

• SPA Report shows change impact on both workload, individual SQL

• Improved SQL• Regressed SQL

• Unchanged SQL• SQL with plan changes

• Fix regressed SQL using SQL Tuning Advisor or SQL Plan Baselines

SQL Performance Analyzer Report

4

1

3

2

SQL Performance Analyzer Report Regressed SQL Statements

Two recommended solutions

to fix regressed SQL:

• Create SQL Plan Baselines: Helps revert to previous stable plans

• SQL Tuning Advisor: Helps explore better execution plans

Step 6: ReiterateRegression Remediation

1

2

Step 6: ReiterateReport before using Plan Baselines

1

2

Change Accepted No Regressions after SPM fix

Step 6: ReiterateReport after using Plan Baselines

Step 7: Production Change & Tuning Deployment

• Implement validated changes permanently on production

• Transport any tuning solution to production• Export/Import SQL Plan Baselines• Export/Import SQL Profiles• Make schema changes • Gather new optimizer statistics• Change parameter values

Enterprise Manager Interface SQL Performance Analyzer Home Page

Enterprise Manager Interface Guided Workflow

Enterprise Manager Interface Parameter Change Workflow

Parameter Changes SPA Report

1

2

Parameter Changes SQL Tuning Advisor Recommendations

Parameter ChangesSPA Report After Profiles Implemented

Change Accepted

1

Command Line Interface dbms_sqlpa PL/SQL Package

-- create an empty sql tuning set

exec dbms_sqltune.create_sqlset(‘my_sts’);

-- incrementally capture sql from the sql cache

exec dbms_sqltune.capture_cursor_cache_sqlset( -

sqlset_name => 'my_sts', -

time_limit => 24*60*60, -

repeat_interval => 5*60);

-- create an analysis task

Var tname varchar2(30);

exec :tname := dbms_sqlpa.create_analysis_task(-

sqlset_name => ‘my_sts’ –

task_name => ‘my_task’);

Command Line Interface dbms_sqlpa PL/SQL Package-- before change execution

exec dbms_sqlpa.execute_analysis_task(:tname, -

execution_type=>'test execute');

-- make your change here

-- alter session set …

-- after change execution

exec dbms_sqlpa.execute_analysis_task(:tname, -

execution_type=>'test execute');

-- compare the before and after execution using buffer_gets

exec dbms_sqlpa.set_analysis_task_parameter(:tname, -

'comparison_metric', 'buffer_gets'); exec dbms_sqlpa.execute_analysis_task(task_name => :tname, -

execution_type=>'compare performance');

Command Line Interface dbms_sqlpa PL/SQL Package

Projected Workload Change Impact:

-------------------------------------------

Overall Impact : 49.18%

Improvement Impact : 61.48%

Regression Impact : -12.3%

SQL Statement Count

-------------------------------------------

SQL Category SQL Count Plan Change Count

Overall 12 9

Improved 1 1

Regressed 3 3

Unchanged 8 5

SQL Statements Sorted by their Absolute Value of Change Impact on the Workload

--------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------------------

| | | Impact on | Metric | Metric | Impact | % Workload | % Workload | Plan |

| object_id | sql_id | Workload | Before | After | on SQL | Before | After | Change |

---------------------------------------------------------------------------------------------------------

| 26 | 8answpy3djcj5 | 61.48% | 67 | 7 | 89.55% | 68.65% | 14.11% | y |

| 29 | gcfysssf6hykh | -4.1% | 3 | 7 | -133.33% | 3.07% | 14.11% | y |

| 30 | 8yyjua0yhcc92 | -4.1% | 3 | 7 | -133.33% | 3.07% | 14.11% | y |

| 32 | 0782mpfq27aq5 | -4.1% | 3 | 7 | -133.33% | 3.07% | 14.11% | y |

---------------------------------------------------------------------------------------------------------

SQL Performance Analyzer Data Dictionary Views

• Modified views in Oracle Database 11g:• DBA{USER}_ADVISOR_TASKS:details about the advisor task• DBA{USER}_ADVISOR_FINDINGS:analysis findings

• New views in Oracle Database 11g:• DBA{USER}_ADVISOR_EXECUTIONS:metadata information for

a task execution• DBA{USER}_ADVISOR_SQLPLANS: SQL execution plans• DBA{USER}_ADVISOR_SQLSTATS: SQL compilation and

execution statistics

• Enables identification of SQL performance changes (e.g., regressions) before end-users can be impacted

• Helps with any change that impacts SQL execution plans (e.g., DB upgrades, Optimizer statistics refresh, schema changes, etc.)

• Automates SQL performance tracking of hundreds of thousands of SQL statements – impossible to do manually

• Prevents side effects to database content when executing SQL.

• Keeps history of all experiments performed on SQL workloads.

• Integrated with SQL Tuning Advisor and SQL Plan Baselines for regression remediation

• GUI provides predefined workflows for Database upgrades and parameter changes

Summary

AQ&

SQL Performance AnalyzerIn-house Testing • SQL Performance Regression Bugs

• Errors + performance degradations caused by code changes • SQL tuning set with 500K SQL statements • Nightly runs

• SQL Plan Management • Failures in reproducing execution plans

• Partitioning Advisor • Verify partition recommendations

• SQL Tuning Advisor• SQL Profiles, index recommendations

• SQL Repair Advisor

Real Application Testing: Tools of the Trade

SQL Performance Analyzer Database Replay

What is it? • Predicts SQL performance deviations before end-users can be impacted

• Replays real database workload on test system

What Purpose? • Assess impact of change on SQL response time

• Assess impact of change on workload throughput

How it works? • Executes each SQL, stored in SQL Tuning Set, in isolation using production context and then compares before and after execution plans and run-time statistics

• Captures workloads and replays it with production context including concurrency, synchronization & dependencies

When to use? • Unit testing of SQL with the goal to identify the set of SQL statements with improved/regressed performance

• Comprehensive testing of all sub-systems of the database server using real production workload

SQL Performance Analyzer and Database Replay are complimentary