Query Optimizer, - AIOUG · 2018-07-20 · Optimizer Statistics Advisor Oracle Database 12.2 introduce new feature called Optimizer Statistics Advisor The goal of the statistics advisor

Copyright © 2018 Oracle and/or its affiliates. All rights reserved. | 1

Query Optimizer, Who Influences & How it works

++ optimization techniques

Chandan TanwaniSenior Application EngineerOracle Financial Services Software Ltd.

AIOUG : ODevC Yatra 2018, India


Journey so far...

Working on various Oracle Database version from Oracle 7 to Oracle 12c…

Working on Oracle Application Server 10g, Weblogic server 8 to 12c…

Worked on forms/reports 3, 6i, 9i…

Passionate for Oracle Technologies…

Frequent Blogger & Speaker in various events…

Frequent Contributor in OTN database forum…

https://tanwanichandan.blogspot.com

@tanwanichandan

/tanwanichandan

[email protected]


Introduction to Query Optimization

How Query Optimizer Works?

Optimizer Components and Operations

Who influences optimizer?

Adaptive Query Optimization

Optimizer Limitation vs Improvements

New Optimization Techniques

Q&A

Agenda


Safe Harbor Statement

The Technical Observations & Views here are my own and not necessarily those of Oracleor its affiliates. These are purely based on my understanding, learning and resolution ofvarious issues.


Introduction to Query Optimizer



I am asked to Tune Query.

I do generate explain plan,

Checking if there is full table scan or not,

If yes, looking why index not used,

Checking if cost is optimal,

Looking for better Plan,

Go here and there and tried few stuff,

Hope something works, but no success.



Is there any guidelines

which I follow and my query

will be Tune itself ?



Are you looking something like this ?



Why don’t we give this task to our Optimizer.

Optimizer is the Brain of Oracle Database.

Feed Optimizer and Get Best Performance


How Optimizer Works?


How Optimizer Works?… Continue…

Query Transformer

OR Expansion

You wrote Optimizer Transformation

SELECT *

FROM sales

WHERE region_id=2

OR prod_id=11;

SELECT *

FROM sales

WHERE prod_id=11

UNION ALL

SELECT *

FROM sales

WHERE region_id=2

AND LNNVL(prod_id=11);



Query Transformer

Subquery Un-nesting


SELECT *

FROM sales

WHERE cust_id IN ( SELECT cust_id

FROM customers );

SELECT sales.*

FROM sales, customers

WHERE sales.cust_id = customers.cust_id;

Query Transformer

Subquery Un-nesting


SELECT *

FROM sales

WHERE cust_id IN ( SELECT cust_id

FROM customers );

SELECT sales.*

FROM sales, customers

WHERE sales.cust_id = customers.cust_id;



Query Transformer

Predicate Pushing


CREATE VIEW empvw AS

( SELECT emp_id, last_name, job_id,

commission_pct, dept_id

FROM employees )

UNION

( SELECT emp_id, last_name, job_id,


FROM contract_workers );

You then query the view as follows:

SELECT last_name

FROM empvw

WHERE dept_id = 50;

SELECT last_name

FROM ( SELECT emp_id, last_name, job_id,


FROM employees

WHERE dept_id=50

UNION

SELECT emp_id, last_name, job_id,


FROM contract_workers

WHERE dept_id=50 );



Estimator

Estimator

Selectivity

Cardinality

Cost



Plan Generator


Optimizer OperationsEvaluation of expressions and

conditions

The optimizer first evaluates expressions and conditions

containing constants as fully as possible.

Statement transformation For complex statements involving, for example, correlated

subqueries or views, the optimizer might transform the

original statement into an equivalent join statement.

Choice of optimizer goals The optimizer determines the goal of

optimization. Optimizer mode plays role to choose goal.

Choice of access paths For each table accessed by the statement, the optimizer

chooses one or more of the available access paths to obtain

table data.

Choice of join orders For a join statement that joins more than two tables, the

optimizer chooses which pair of tables is joined first, and

then which table is joined to the result, and so on.


Do You Know ?

Selectivity is an internal calculation that isnot visible in execution plans.


Do You Know ?

If no statistics are available then theoptimizer either uses dynamic sampling oran internal default value


Do You Know ?

The cost calculation is done based on diskI/O, CPU usage, and memory usage as unitsof work.

The cost of a query plan is the number of work unitsthat are expected to be acquired when the query isexecuted and produce the result.


Do You Know ?

Cost is the measurement for the Optimizer tocompare plans for the same query. Youcannot tune or change cost.


Optimizeris

smartenough…

Right ???

Then,

Why

SQLsperformingBad ?


Who Influences Optimizer?


Who Influences Optimizer?


Controlling Behavior of Optimizer

Optimizer Mode

Cursor Sharing

Optimizer Dynamic Sampling

Optimizer Features Enable

Optimizer Index Caching

Optimizer Index Cost Adj

Db File Multiblock Read Count

Star Transformation Enabled

Optimizer use pending statistics

Optimizer use sql plan baseline

init Parameters are theone who ControlOptimizer behavior.


Controlling Behavior of Optimizer

Optimizer Adaptive Features

Optimizer In-memory Aware

Optimizer Adaptive Features (Obsoleted)

Optimizer Adaptive Plans

Optimizer Adaptive Statistics

Optimizer In-memory Aware






Adaptive Query Optimization … How it works

Adaptive query optimization enable optimizer to adjust execution plan at run timewhen better information is available.

Adaptive Plans:

• Different Join Methods (change NL to HASH) or Parallel Distribution

• Adaptive Plans does not pick the final plan until execution time based onstatistics collection. Information learned at execution time is used in futureexecutions.

There are three types of Adaptive Statistics:

• Dynamic Statistics (previously dynamic sampling in 10g/11g) or runtime statistics

• Automatic Reoptimization or statistics generated after the initial execution

• SQL Plan Directives direct optimizer to dynamic statistics & gets accuratecardinality

In 12cR2, Oracle introduced Continuous Adaptive Query Plans (CAQP) where certainqueries, based on input data, can benefit from continuous adaptive join methods .


Adaptive Query Optimization … How it works

Adaptive Plans – changed from default on 1st execution

Automatic Re-optimization – 2nd execution

1. Determine default plan and include alternative sub-plans which are pre-computed

2. Sub-plans stored in the cursor

3. Stats collector inserted before join

4. Rows buffered until final decision is made

5. Number of rows seen in statistics collector if exceeds threshold, Plan switches toother join method

6. Statistics collector disabled

7. Plan resolved on first execution & remains the same for subsequent executions


Do You Know ?

How long same plan is used by optimizer?

Till the plan ages out from the cache, or a different optimizerfeature (for example, adaptive cursor sharing or statisticsfeedback) invalidates the plan.


Do You Know ?

When optimizer choose nested loop andwhen chooses hash join?

The statistics collector buffers enough rows to determine whichjoin method to use. If the row count is below the thresholdidentified by the optimizer, then optimizer chooses the nestedloops join; otherwise, optimizer chooses the hash join.


Optimizer Limitations vs Improvements


Optimizer Limitation vs Improvements


11g Limitations

There will be variations of statistics by optimizer for many reasons i.e. stale or incomplete

statistics, complex expressions used etc.…

No way to adopt run time statistics.

12c Improvements

Run-time adjustments to query plans.

Discovery of additional information at execution time leads to better statistics.


Optimizer Limitation vs Improvements… … continue … …

Histograms

11g Limitations

Histograms can underestimation of cardinality for nearly popular values.

Number of histogram buckets is limited to 254.

12c Improvements

Top-frequency histograms providing more accurate information on the popular values

Actual frequencies of bucket endpoint values are recorded in histograms.

More histogram buckets.

Hybrid histogram with more endpoint and buckets.



SQL Plan Directives

11g Limitations

Better plan and SQL information stored in cursor only.

We cannot share this information across statements.

Information is lost when the cursor ages out from shared pool.

12c Improvements

SQL plan directives allow to persist information learnt during compilation and execution.

Collected on query expressions not at a statement level.

Automatically created and maintained for you.



Online Statistics Gathering for Direct Path Load

11g Limitations

After inserting rows in newly created table statistics are required to ensure accurate

cardinality estimate.

Gathering statistics for the first time requires a full table scan.

Potentially expensive in terms of time and resource.

12c Improvements

Statistics will be gathered as part of the direct path load operations.

Statistics available directly after data load.

No additional table scan required to gather statistics.



Statistic gathering options for GTTs

11g Limitations

All Sessions can have same statistics, which is gathered on a GTT

This needs to understand that not all session in GTT will have the same data distribution

12c Improvements

Each session can gather their own set of statistics for GTT.

Provides more accurate cardinality estimates within each session.

Controlled by new preference GLOBAL_TEMP_TABLE_STATS.

Default value is SESSION (non shared).



Adaptive SQL Plan Management

11g Limitations

DBA needs to monitor SQL plan baselines to find new unaccepted plans.

SPM evolve process needed to manually invoked.

12c Improvements

New evolve auto task running in the maintenance window.

Ranks all non-accepted plans and runs evolve process for them.

Automatically accepts any new plan that performs better than exist plan.


Do You Know ?

Dynamic stats are stored in memory andreused by other SQLs too.

(In result cache).


New Query Optimization Techniques in 12c onwards


New Optimizer Techniques

Approximate Query Processing

Partial Join Evaluation (PJE)

Multi-Table Left Outer Join

Optimizer Statistics Advisor



Approximate query processing is a set of optimization techniques that speed

analytic queries by calculating results within an acceptable range of error.

In 12c R1APPROX_COUNT_DISTINCT

In 12c R2 (newly added) In 18c (extended in this release)APPROX_COUNT_DISTINCT_AGG APPROX_COUNT

APPROX_COUNT_DISTINCT_DETAIL APPROX_SUM

APPROX_MEDIAN APPROX_RANK

APPROX_PERCENTILE

APPROX_PERCENTILE_AGG

APPROX_PERCENTILE_DETAIL

TO_APPROX_COUNT_DISTINCT

TO_APPROX_PERCENTILE



To control behavior approximate query processing, following parameters are introduce

in 12c R2, which can be set at system and session level.

Initialization Parameter Default Description

APPROX_FOR_AGGREGATION FALSE Approximate query processing is used for aggregation queries

and analytic queries.

APPROX_FOR_COUNT_DISTINCT FALSE If TRUE converts COUNT(DISTINCT ...) calls to

APPROX_COUNT_DISTINCT calls.

APPROX_FOR_PERCENTILE NONE This can be set to NONE PERCENTILE_CONT,

PERCENTILE_CONT DETERMINISTIC, PERCENTILE_DISC,

PERCENTILE_DISC DETERMINISTIC, ALL, ALL DETERMINISTIC.



Exact query

Approx query

Lots of temp

no temp


Join Elimination

SELECT e.ename, d.deptno

FROM employees e,

departments d

WHERE e.deptno = d.deptno;

SELECT e.ename, e.deptno

FROM employees e

WHERE e.deptno is not null;

SELECT e.ename

FROM employees e

WHERE exists ( SELECT 1

FROM departments d

WHERE e.deptno = d.deptno);

SELECT e.ename

FROM employees e

WHERE e.deptno is not null;

SELECT e.ename, e.project_id

FROM employees e,

projects p

WHERE e.project_id = p.project_id (+);

SELECT e.ename, e.project_id

FROM employees e;


Multi-table left outer Join



Oracle Database 12.2 introduce new feature called Optimizer Statistics Advisor

The goal of the statistics advisor is to analyze how statistics are gathered, validate

statistics and check the status of auto stats gathering.

Consists of 4 components

Rules – Define the best way to manage statistics

Findings – Violations of the rules in the system

Recommendations – Solutions to the violations

Actions – Implementation of the recommendations

Rules Findings Recommendations Actions



Auto

• Happens daily in the maintenance window

• Partial cycle: Rules -> Findings -> Recommendations, Actions not

implemented

• Report can be viewed any time and Actions implemented

Manual

• Can be run anytime

• Full cycle: Rules -> Findings -> Recommendations -> Actions

• The Actions can generate a script instead of implementation


Wrapping it all up

Query optimizer is the brain of Oracle Database. More you feed more powerful it becomes.

Statistics calculation, User, SQL profiles, Hints, Plan Management, InitParameters are the components who Influences to Optimizer.

Many Optimizer Limitation in 11g which is improved in 12c and many more in 18c version.

Adaptive Optimizer, one of the smartest way to perform SQL better and faster

New Query Optimization Techniques in 12c onwards which increases query performance.


Query Optimizer, - AIOUG · 2018-07-20 · Optimizer Statistics Advisor Oracle Database 12.2 introduce new feature called Optimizer Statistics Advisor The goal of the statistics advisor

Documents