43749305 Oracle Db 11g Readiness Technical Uk Sep07 2

PL/SQL Enhancements in 11g

Enhanced SQL –PIVOT and UNPIVOT

100Q3Jeans

1000Q4Jeans

500Q3Jeans

1000Q1Jeans

1000Q2Shoes

2000Q1Shoes

AMOUNTQUARTERPROD

1000600Null1000Jeans

NullNull10002000Shoes

Q4Q3Q2Q1PROD

Enhanced performance of ETL in your warehouse

SELECT * FROM salesPIVOT

(sum(amount)FOR quarter in

‘Q1’,’Q2’,’Q3’,’Q4’);

PL/SQL Enhancements in 11gFine Grained Dependency Tracking

The challenge

• View v ends up invalid in 10.2 because we know only that its dependency parent has changed – at the granularity of the whole object

create table t(a number)/create view v as select a from t/alter table t add(Unheard_Of number)/select status from User_Objectswhere Object_Name = 'V'/

Fine Grained Dependency Tracking

• In 11.1 we track dependencies at the level of element within unit

• so we know that these changes have no consequence

create table t(a number)/create view v as select a from t/alter table t add(Unheard_Of number)/select status from User_Objectswhere Object_Name = 'V'/

PL/SQL Enhancements in 11gReal Native Compilation

The challenge

• Through 10.2, PL/SQL compiled to a native DLL is significantly faster than PL/SQL compiled for interpretation by the PVM

• Oracle translates PL/SQL source to C code and leaves the last step to a 3rd partyC compiler

• BUT… some customers’ religion forbids a C compiler on a production box!

• AND… other customers’ religion forbids paying to license a C compiler when they’ve already paid to license Oracle Database!

Real native compilation

• In 11.1, Oracle translates PL/SQL source directly to the DLL for the current hardware

• Moreover, Oracle does the linking and loading so that the filesystem directories are no longer needed

• So PL/SQL native compilation will work out of the box – and without compromising religion

• Only one parameter remains: the on/off switch, PLSQL_Code_Type

Real native compilation

• As a bonus, it’s faster!• Compilation with real native is twice as fast as with C native

• The Whetstone benchmark is 2.5x faster with real nativethan with C native at run-time

• Contrived tests have shown 20x run-time speedup

PL/SQL Enhancements in 11gSequence in a PL/SQL expression

create or replace trigger Trgbefore insert on My_Table for each row

declares number;

begin-- Annoying locutionselect My_Seq.Nextval into s from Dual;:New.PK := s;

end;/

The challenge

Sequence in a PL/SQL expression

create or replace trigger Trgbefore insert on My_Table for each row

begin

:New.ID := My_Seq.Nextval;end;/

PL/SQL Enhancements in 11gTools support

ChallengeInherit code, find a problem in one area that links elsewhere, got lost following reference after referenceImpact Analysis: how do you determine who references a particular variable?

SolutionPL/Scope

Usage

PL/Scope

alter session set PLScope_Settings = 'identifiers:all'

create or replace...

Select …. from … User_Identifiers where Object_Name = ...

UI expected in SQL Developer 2.0

ChallengeWhere is time really being spent in PL/SQL code?

SolutionPL/SQL Hierarchical Profiler

reports dynamic execution profile organised by subprogramaccounts for SQL and PL/SQL separatelygenerates hypertexted HTML reports

Usage

PL/SQL Hierarchical Profiler

> EXECUTE DBMS_HPROF.START_PROFILING('PLSHPROF_DIR', 'test.trc');

% plshprof

UI expected in SQL Developer 2.0

XML DB Enhancements in 11gFast, better and easier

See:- New Features in Oracle XML DB, for Oracle Database 11g Release 1,

An Oracle White Paper, June 2007

In Place Schema Evolution

• 10.2 copyEvolve() unloads and reloads the data when the XML Schema changes• very flexible but expensive: time to evolve schema is proportionate

to the amount of data being managed • 11.1 In Place Schema Evolution allows simple changes to

registered XML schemas with zero down-time• no data copy required• schema change takes a few seconds regardless of amount of data.• changes to the XML must not invalidate existing documents

• add optional elements and attributes• adding new values to enumerations• increase in length

XMLType Partitioning

Challenge• repeating elements typify XML documents and are

managed using nested tables

Solution• leverage 11.1 REF partitioning• nested Tables are partitioned using the same key

as the top level XMLType table• supports “partition maintenance” operations • supports “partition pruning”

<PurchaseOrder><Date>…</Date><Ref>…</Ref><LineItems>

<LineItem>…<LineItem>…<LineItem>

<LineItems>

XML Misc Improvements

• Significant performance improvements• Large node handling eliminates current 64K limit on

size of a text-node.• Stream based replication support for LOB-based

XML storage• Events: trigger support for the XML DB repository.

• e.g. inserting a purchase order into “approved” folder triggers ordering process

Binary XML

Challenge• 10.2 schema-less XML not well optimizedSolution• 11.1 binary XML optimized for schema-less and

flexible schema-based XML• Post Parse compact representation of XML• Flexible XML Schema Support• High performance fragment access and extraction

with new XML Index

Binary XML

AppServer

WebCache

DatabaseClient

Binary XML Binary XML Binary XML

Oracle Binary XML

• Single format for “on-disk”, “in-memory” and “on-the-wire”• Reduced CPU and Memory overhead

• single format avoids parse and serialize issue between tiers• Reduced Network Overhead

• post-parse format used when XML moves between tiers

XML Index

• New universal index for Binary and LOB basedXMLType

• All possible paths are indexed by default• XML index optimization

• asynchronous operation possible• path sub-setting controls what is indexed• partial re-indexing (binary XML and securefiles only)

• Repository integration with XML Index on XML DB repository

Java Enhancements in 11gDoing it inside like you do outside

See:- Java Developer Center on OTNhttp://www.oracle.com/technology/tech/java/index.html

JDBC PerformanceSaving 1 Million Roundtrips per Day

Prefetch in First Roundtrip (JDBC-Thin) • saves 1 database roundtrip per SELECT Statement • benefit for ad-hoc queries

New Native AQ protocol (JDBC-Thin) • AQ/OJMS operations 40%-300% Faster• reduced server-side CPU Utilization

Java StandardsJDBC 4.0

• SQLXML• RowId• NChar & Nclob• Wrapper Interface• SQLException Hierarchy• conn.createBlob, createClob, etc

Disconnect Java VM upgrade from database• Java 5 (i.e. JDK 1.5) in base 11.1 release• Java 6 (i.e. JDK 1.6) and newer JDKs, in patch sets

From Black Box to White Box Monitoring• JMX Instrumentation• Support standard JMX-enabled tools(i.e., JConsole)

Java in the Database Mass Adoption

Challenge• PerformanceSolution• New JIT compiler compiles Java in the database natively and transparently (on

the fly) without the need of a C compiler• Enabled by default, Out-of-the-Box• Up to an Order of magnitude (~x 10) speed-up for pure Java, 10g versus 11g,

out-of-the-box

Challenge• Ease of UseSolution• JDK-Like interface, with code on filesystem, using familiar notation,

e.g. –{classpath}, -D system properties• Eases the migration for developers from JDK VM world

Enhancements for PHP developersAccessing the database will never be the same

See:- PHP Developer Center on OTNhttp://www.oracle.com/technology/tech/php/index.html

PHP users have asked for Database Connection Pooling

• Web apps issue hundreds connects per second• Connection establishment is expensive• PHP does not allow real multi-threading

• persistent vs non persistent connections• Existing client-side connection pooling not possible • Options:

• apache process can repeatedly connect/disconnect• apache process can hold onto private connection

• Results in inefficient use of the database resources

Database Resident Connection Pool

No Connection Pooling 11g Database Resident Connection Pooling

DB

WS

Native Database Web Service

PL/SQL

SQL

XQuery

SOA

DBA

?

WSDLAuto-

Genera

te

SOAP

Easy: a few steps, no other software installation and no coding

Secure: HTTP authentication and easily integrate with WS Security Products

High performance : C-based engine, natively build in to the Oracle DB

Oracle Scheduler enhancements in Oracle Database 11g

• Local OS jobs• run as any OS user

• Remote OS jobs• no Oracle database required

on node• run as any OS user

Agent

11g

ScheduleJobs

Execute remoteOS job as OS user X

Execute remote OS Job as OS user Z

Agent

Execute local OSjob A as OS user X &job B as OS user Y

• OLAP• Data Mining

Business Intelligence

See:-

Oracle Database 11g for Data Warehousing and Business Intelligence

An Oracle White Paper

July 2007

What is Oracle OLAP Option?

• A multidimensional calculation and aggregation ‘engine’ embedded in the database• multidimensional data types (Cubes and Dimensions)• OLAP API for cube definition and multidimensional query

and coming in Oracle Database 11g ….• transparent SQL interface to OLAP Cubes and dimensions• cube-organized materialized views• OLAP metadata in the Oracle Data Dictionary

Oracle OLAP 11g New Features in Oracle Database 11g

• Data accessibility: Enhanced SQL access to OLAP cubes• SQL Optimizer awareness of OLAP cubes• Cube Organized Materialized Views• System-maintained generation of cube and dimension

views

• Feature accessibility: Shift the focus from the skilled OLAP specialist to the database generalist• Cube and dimensions in the Oracle Data Dictionary• Cost-based aggregation• Dimensionally aware SQL-like calculation syntax

Cube-Organized Materialized Views

• Materialized view with data stored in an OLAP cube• Automatic query write of summary queries into the cube

for improving performance of summary queries• Materialized view managed refresh of cubes and

dimensions• Similar to materialized view on pre-built table; data is

stored in cube and MV is a meta data object

Cube-Organized Materialized Views

Cube MVCube OrganizedCube OrganizedMV representingMV representingsummary datasummary data

Tables

Detail data managed in Detail data managed in relational tablesrelational tables

SQL QuerySQL Query

Summary dataSummary datamanaged in cubesmanaged in cubes

DatabaseDatabase--managedmanagedrefreshrefresh

QueryQueryRewriteRewrite

• Detail data is stored in relational tables

• Application queries relational tables with SQL

• Summary data is managed by OLAP cube

• Access to summary data occurs via automatic query rewrite to the cube

• Application is unchanged, but updates and queries are faster

Query Rewrite to Cube

select a13.FAMILY_ID FAMILY_ID,a13.FAMILY_DSC FAMILY_DSC,a12.CALENDAR_YEAR_ID CALENDAR_YEAR_ID,a12.CALENDAR_YEAR_DSC CALENDAR_YEAR_DSC,sum(a11.SALES) WJXBFS1

from UNITS_FACT a11join TIME_DIM a12on (a11.MONTH_ID = a12.MONTH_ID)

join PRODUCT_DIM a13on (a11.ITEM_ID = a13.ITEM_ID)

group by a13.FAMILY_ID,a13.FAMILY_DSC,a12.CALENDAR_YEAR_ID,a12.CALENDAR_YEAR_DSC;

Select from relational tables

GROUP BY

SUM(SALES)

Query Rewrite to Cube

OPERATION---------------------------------------SELECT STATEMENTHASHHASH JOINHASH JOINCUBE SCAN CB$UNITS_CUBEVIEWHASHTABLE ACCESS PRODUCT_DIM

VIEWHASHTABLE ACCESS TIME_DIM

Cube Access

What is Oracle Data Mining?• In-Database Mining Engine

• wide range of algorithms• anomaly detection • attribute importance• association rules • clustering • classification & regression• nonnegative matrix factorization • structured & unstructured data (text mining)• BLAST (life sciences similarity search algorithm)

• Oracle Data Miner (GUI)• simplified, guided data mining

• Spreadsheet Add-In for Predictive Analytics• “1-click data mining” from a spreadsheet

• PL/SQL API & Java (JDM) API• develop advanced analytical applications

• Simplified development and deployment of models• supermodels: data preparation combined with mining model• tighter database integration – DMSYS schema eliminated; improved

sharing of objects, security, manageability• New algorithms: “General Linear Models”

• multivariate linear regression; logistic regression• basic tool for microeconomics analysis and business decision-making. Indicate

the impact of different attributes on a key performance indicator. “What is the impact on SUVs sales if the price of gas goes up by a $1.00?”

Oracle Data Mining 11gNew Features in Oracle Database 11g

Why In-Database Analytics ?

• Fewer moving parts : data remains in the database at all times … with appropriate access security control mechanisms

• Straightforward inclusion within interesting and arbitrarily complex queries

• Real-world scalability and performance• Fast scoring:

• 2.5 million records scored in 6 seconds on a single CPU system

• Real-time scoring: • 100 models on a single CPU: 0.085 seconds

Integration with Oracle BI EE ODM provides likelihood of fraud and other important questions.

Security

• User Management• Access Control• Data Protection• Monitoring

See:- Oracle Database 11g Release 1 Transparent Solutions for Security and Compliance


June 2007

Key Drivers for Data Security

• Sarbanes-Oxley (SOX), • Foreign Exchange Instruments and

Exchange Law (J-SOX)• EU Privacy Directives, CA SB 1386….• Payment Card Industry (PCI)• Adequate IT controls, COSO, COBIT• Separation of duty, Proof of compliance,

Risk Assessment and Monitoring

• Large percentage of threats go undetected• Outsourcing and off-shoring trend• Customers want to monitor insider/DBA

Regulatory Compliance

Insider Threats

Oracle Database Security Products

Core Platform Security

User Management• Oracle Identity Management• Enterprise User Security

Data Protection• Oracle Advanced Security • Oracle Secure Backup

Access Control• Oracle Database Vault• Oracle Label Security

Monitoring• Oracle Audit Vault• EM Configuration Pack

• User Management for Compliance• Centralized User Management • Consolidate database accounts with

shared database schemas• Centrally managed DBAs• Validated with Oracle Virtual Directory

• Enterprise Strong Authentication• Kerberos (MSFT, MIT)• PKI (x.509v3)• Password• SYSDBA Strong Authentication

• Database Enterprise Edition Feature• Requires Oracle Identity Management• Available since Oracle 8.1.6

Oracle Database 11gEnterprise User Security

Financial Database

Customer Database

HR Database

Oracle Identity Management

EUS EUS







Oracle Label SecurityManageability

• Policy based model• Multiple policies

supported• ACME, HR, Legal

• Policies are umbrellas applying to one or more tables, schemas, users

• Web based management

• Integrated with Oracle Identity Management







Oracle Advanced SecurityTablespace Encryption

• Define a new tablespace as ‘encrypted’ • cannot convert existing, un-encrypted tablespaces• however, content can be moved into encrypted tablespaces

• Always salted for higher security• Overcomes limitation of column-based TDE:

• supports indexes other than b-tree• supports foreign keys

• No additional management overhead• integrated into TDE key management, same wallet used as

for column based Transparent Data Encryption

• No storage overhead (!)

Oracle Advanced SecurityMaster key stored in HSM device

• Store the Master key in an external hardware device• Master key never leaves the device• Standard PKCS #11 API allows customers to choose from

a wide range of HSM vendors• Encryption and decryption done on the database server• Simplifies key management in distributed environments

(data guard, RAC)

Oracle Advanced SecuritySECUREFILE LOB Encryption

• SECUREFILE LOB encryption • All SECUREFILE LOBs in an encrypted column are

encrypted• In-line (in table) and out-of-line (in tablespace) are both

encrypted• BFILEs are not encrypted• Always salted for higher security







Oracle Database 11gSecurity Manageability

• Integrated with EM• Label Security• Virtual Private Database• Application Context• Enterprise Security

Manager• Transparent Data

Encryption

11g Database Manageability

• Memory Management• AWR Baselines• Automatic SQL Tuning• Advisors• Upgrade Considerations

See:- Oracle Database 11g:

Manageability Overview


August 2007

11g Database ManageabilityAutomatic Memory Management

Memory Management in 10g/11g

• SGA_TARGET

• Key shared memory components tuned automatically

• Single parameter controls all shared memory

10g Automatic Shared Memory Management (ASMM)

10g Automatic PGA Memory Management

• PGA_AGGREGATE_TARGET

• Various PGA sort areas tuned automatically

• Single parameter controls all PGA memory

• MEMORY_TARGET

• MEMORY_MAX_TARGET

• Improves memory utilization• Eases memory management

11g Automatic Memory Management

Automatic Memory Management in 11g

• Unifies system (SGA) and process (PGA) memory management

• Dynamic parameters for all database memory:MEMORY_TARGETMEMORY_MAX_TARGET

• Automatically adapts to workload changes

• Maximizes memory utilization• Available on:

• Linux• Windows• Solaris • HPUX• AIX

O/S MemoryO/S Memory

PGA

SGA

PGA

SGA

Memory Target Benefits

• Balances dynamic memory needs of SGA and PGA• Previously either or both could end up undersized• PGA can grow but not to the detriment of SGA performance• Best-effort memory transfers• Tunable PGA can help relieve 4031 from SGA

• Built-in safety nets:• Decides victim components• Paced memory donation avoids thrashing• Run-away workload cannot starve resources• Component-specific settings still honored

11g Database ManageabilityAWR Baseline and Adaptive Thresholds

Performance Monitoring and Baselines

• Performance Page shows certain metrics against 99th

percentile over selected baseline

• Adaptive Thresholds over select system metrics• Enabled by System Moving Window baseline

• AWR Compare Period Report• Use Baseline as one of the periods

Other Baselines Features

• Baseline Templates enable future Baseline creation• Pre-specify known interesting times, e.g. Thanksgiving

• System Moving Window Baseline• Always available• Defined by AWR retention or user specified • Adaptive thresholds computed over this baseline

Creating Single AWR Baseline

Quick Configure Adaptive Thresholds

Adaptive Thresholds

Adaptive threshold values follow baseline values

AWR Baseline

11g Database ManageabilityAutomatic SQL Tuning

Manual SQL Tuning Challenges

• Complex - requires expertise in several domains• SQL optimization: adjust the execution plan• Access design: provide fast data access• SQL design: use appropriate SQL constructs

• Time consuming• Each SQL statement is unique• Potentially large number of statements to tune

• Never ending task• SQL workload always evolving• Plan regressions can happen

SQL Tuning in Oracle Database 10g

Workload

SQL Tuning CandidatesSQL Tuning

Advisor

ADDM

AWR

one hour

GenerateRecommendations

DBA

Invoke Advisor

Implement

DBA

Some meaningful automation, but the DBA is still required

EvaluateRecommendations

DBA

SQL Tuning Automation in 11g

• Fully automatic SQL Tuning task• Runs automatically under Autotask framework:

• Maintenance Window, CPU resource controlled, on/off switch• Identifies, ranks and tunes candidate SQL• Leverages SQL Tuning Advisor

• Candidate SQL automatically chosen• Excluded: parallel queries, DML/DDL, recursive, ad-hoc (infrequent)

• Tests and (optionally) implements SQL profiles• Only implements significantly improved plans (3x) • Time budgeted to avoid run-away plans

It’s Automatic!

Choose Candidate

SQLone

week

Automatic SQL Tuning in Oracle Database 11g

Workload

SQL Tuning Candidates

Test SQL ProfilesImplementSQL Profiles

GenerateRecommendations

AWRDBA

View Reports / Control Process

Result Summary

Result Details

Result Recommendations

11g Database ManageabilitySQL Access Advisor

SQL Access Advisor: Partition Advice

Indexes Materializedviews

Materializedviews log

SQL Access Advisor

Hypothetical

SQL cache

Filter Options

STS

Complete Workload

Partitionedobjects

Hash partitions?

Interval partitions?

SQL Access Advisor Results

Access Advisor Recommendations

Editable implementation script

11g Database ManageabilityFault Diagnostic Automation

Oracle Database 11g R1 Fault Diagnostic Automation

Realistic Testingand

Automatic HealthChecks

Automatic DiagnosticWorkflow

IntelligentResolution

ProactivePatching

Goal: Reduce Time to Problem Resolution

Prevention Resolution

Diagnostic Solution Delivery

Automatic Diagnostic Workflow

DBA

CriticalError

AutomaticDiagnosticRepository

Knownbug?

DBA

EM Support Workbench:Apply patch / Data Repair

Yes

DBA

No

Alert DBATargeted health checks

Assisted SR fillingAuto incident creation

First failure capture

EM Support Workbench:Package incident info

Data Repair

1 2

3

4

Problems and Incidents

• Problems are fundamental code or configuration issues that can cause execution failures• They exist until they are corrected, e.g by patch• They are assigned a signature-based key value

• An incident is a single occurrence of a problem• They happen at point(s) in time and thus have timestamps• They induce diagnostic actions like dumps and traces• They are managed by ADR and packaged by IPS

Automatic Diagnostic Repository

diag

rdbms

DBName

SID

ADRBase

$ORACLE_HOME/log

DIAGNOSTIC_DEST

ADRHome

$ORACLE_BASE

ADRCIlog.xml alert_SID.log

V$DIAG_INFO

BACKGROUND_DUMP_DEST

USER_DUMP_DEST

CORE_DUMP_DEST

alert cdump (others)hmincpkg incident

metadata

incdir_1 incdir_n…

trace

Support Workbench

Important points about ADR

• Location of traces and dumps has changed• See V$DIAG_INFO for new locations• Symbolic links can provide work around for script

compatibility

• Format of alert.log is changed• New XML format more parse-able• Old style still available for backward compatibility

• Space management• Incident flood control minimizes number of traces• ADR auto-purging reduces disk footprint

Incident Packaging Service (IPS)

• IPS automates packaging of diagnostic data for given problem

• Creates zip files from incident package contents for sending to Oracle Support

• Intelligently correlates and includes dumps and traces from related incidents• Correlation rules based on time proximity, client id, error code

• Recommends further diagnostic actions for DBA • For example build SQL test case

• DBAs can explicitly add/edit or remove files before packaging• Including non-ADR managed files

EM Support Workbench Overview

• Primary customer-facing interface into IPS and ADR

• Task-oriented interface over problems and incidents• View problem and incident details• Create and manage IPS packages• Run health checks or generate additional diagnostic data• Run advisors to help resolve problems

• Linkages to Oracle Support• Generate and send incident package files using OCM• Create and track Metalink service requests from problem• Close problem SR once resolved

EM Support Workbench

Package details and IPS in EM

EM Support Workbench Quick Package

• Simple wizard workflow from problem through package and send to support• Single-problem package• Good for rapid first fault reporting

• Limitations• Package zip must be 2GB or less • Cannot scrub traces or customize package

Prevention: Health Monitoring

• Proactive Health Checks• E.g. Dictionary Validation

• Reactive Checks• Trigger on error via Intelligent dumping Rules• E.g. Checks blocks around corrupted block

• All Checks can be Activated on Demand• Incident Meter for EM

Intelligent Repair Advisors

• Data Recovery Advisor• Guided expert data recovery system using diagnostic data

and Health Check output

• SQL Test Case Builder• Automatically retrieves exact environment information from

ADR to build SQL test cases and replicate SQL issues

• SQL Repair Advisor• Analyzes failing SQL statements to isolate bug• May recommend SQL Patch as work around

Data Recovery Advisor

Data Recovery Advisor• Oracle Database tool that automatically diagnoses data failures,

presents repair options, and executes repairs at the user's request

• Determines failures based on symptoms• E.g. an “open failed” because datafiles f045.dbf and f003.dbf are missing• Failure Information recorded in diagnostic repository (ADR)• Flags problems before user discovers them, via automated health

monitoring• Intelligently determines recovery strategies

• Aggregates failures for efficient recovery• Presents only feasible recovery options• Indicates any data loss for each option

• Can automatically perform selected recovery steps

Reduces downtime by eliminating confusion

Data Recovery AdvisorEnterprise Manager Support

SQL Test Case Builder

SQL Test Case Builder

Business Requirement• Bug resolution

• Test case required for fast bug resolution• Not always easy to provide a test case

• What information should be provided?• How much data is need?

• Getting the test case to Oracle can be trickySolution• Oracle automatically creates a test case• Collects necessary information relating to a SQL incident• Collected data is packaged to be sent to Oracle• Collected data allows a developer to reproduce the problem

SQL Repair Advisor

SQL Repair Advisor

Business RequirementThe most common types of SQL problems - exception, performance regression etc., are hard to diagnoseA lot of time is spent trying to reproduce the problemIf a workaround is found it has to be applied to entire system

Solution• Advisor

• Investigates the incident locally• Automatically determines the root cause• Provides a workaround (SQL Patch) for just the effected SQL• If not, sends necessary diagnostic information to Oracle

SQL Repair Advisor Flow

DBA

SQLstatement Execute Statement

crashes

Generateincident in ADRautomatically

DBA runSQL Repair Advisor

DBA getsalerted

SQL

Rep

air A

dvis

orin

vest

igat

es

SQL patchgenerated

DBA acceptSQL patch

Statementexecutes

successfullyagain

SQL statementpatched

Execute

Trace files

DB Management Pack Enhancements

• Diagnostic Pack• ADDM for RAC• AWR Baselines and Adaptive Thresholds• Transportable AWR

• Tuning Pack• Automatic SQL Tuning Advisor• Partition Advisor• Real-time SQL Monitoring• SQL Plan Management and Plan Evolution

Upgrade Considerations

New Background Processes in 11g

ASMMaintains disk membership in ASM disk groupsGMON

ALL.The process archives historical rows for tracked tables into flashback data archives and manages the flashback archives. FBDA

ALL with any registered event

activity (optional non-fatal)

The Event Monitor Coordinator coordinates the event management and notification activity in the database which includes Streams Event Notifications, Continuous Query Notifications and Fast

Application Notifications. EMNC / e0xx

ALLPerforms diagnostic dumps and executes global oradebug commands.

DIAG

ALLResponsible for hang detection and deadlock resolution.

DIA0

ALLThe Resource Manager process is responsible for setting Resource Plans and other Resource Manager related tasks. DBRM

RAC

In an RAC environment, this per-instance process (Atomic Controlfile to Memory Service) is an agent which contributes to ensuring a distributed SGA memory update is either globally committed on success

or globally aborted in the event of a failure. ACMS

RAC onlyProvides transparent support for XA global transactions in a RAC environment. The databaseautotunes the number of these processes based on the workload of XA global transactions. GTX0-j

Description FoundProcess Name

New Background Processes in 11g

RAC onlyThe RAC Management Processes perform manageability tasks for RAC, e.g. creation of RAC related resources when new instances are added to the clusters RMSn

RAC onlyIn a RACenvironment, this process manages background slave process creation and communication on remote instances. These background slave processes perform tasks on behalf of a coordinating process

running in another instance. RMSN

ALLThe Virtual Keeper of TiMe? is responsible for providing a Wall-Clock time (updated every second) and Reference-Time Counter (updated every 20ms and available only when running at elevated priority). VKTM

ALL (optional non-fatal)

The space management coordinator process coordinates the execution of various space management related tasks, such as proactive space allocation and space reclamation. It dynamically spawns slave

processes (Wnnn) to implement the task. SMCO / Wnnn

ASM, spawned on demand.Marks ASM Allocation Units as stale following a missed write to an offline disk. MARK

ASM, spawned on demand.Performs proxy I/O to an ASM metafile when a disk becomes offline.KATE

Spawns Oracle processes.

Description

ALLPSP0

FoundProcess Name

Oracle Database 11g Release 1 Upgrade Paths

• Direct upgrade to 11g is supported from 9.2.0.4 or higher, 10.1.0.2 or higher, and 10.2.0.1 or higher.

• If you are not at one of these versions you need to perform a “double-hop” upgrade

• For example:• 7.3.4 -> 9.2.0.8 -> 11.1• 8.1.7.4->9.2.0.8->11.1

Choose an Upgrade Method

• Database Upgrade Assistant (DBUA)• Automated GUI tool that interactively steps the user through

the upgrade process and configures the database to run with Oracle Database 11g Release 1

• Manual Upgrade• Use SQL*Plus to perform any necessary actions to prepare

for the upgrade, run the upgrade scripts and analyze the upgrade results

• Export-Import• Use Data Pump or original Export/Import

• CREATE TABLE AS SQL statement

Oracle Database 11g Installation Changes

• Addition of new products to the install• SQL Developer• Movement of APEX from companion CD to main CD• Warehouse Builder (server-side pieces)• Oracle Configuration Management (OCM)• New Transparent Gateways

• Removal of certain products and features from the installation:• OEM Java Console• Oracle Data Mining Scoring Engine• Oracle Workflow• iSQL*Plus

Fine Grained Access Controlfor Utl_TCP and its cousins

Challenge• Oracle Database provides packaged APIs for PL/SQL

subprograms to access machines (specified by host and port) using bare TCP/IP and other protocols built on it (SMTP and HTTP)• Utl_TCP, Utl_SMTP, Utl_HTTP…• If you have Execute on the package, you can access ANY

host-port

Solution• an Access Control List (ACL) specifies a set of users and roles• you assign an ACL to a host and port range• you may need to explicitly grant this access in 11.1

Case Sensitive Password

• By default:• Default password profile is enabled• Account is locked after 10 failed login attempts

• In upgrade:• Passwords are case insensitive until changed• Passwords become case sensitive by ALTER USER

• On creation:• Passwords are case sensitive

• Review:• Scripts• Database links with stored passwords• Consider backward compatibility parameter,

SEC_CASE_SENSITIVE_LOGON

Log files changes

Automatic Diagnostic Repository• $ORACLE_BASE/diag

alert.log• xml format• $ORACLE_BASE/diag/rdbms/orcl/orcl/alert/log.xml• adrci> show alert –tail• $ORACLE_BASE/diag/rdbms/<dbname>/trace/alert_SID.log

Moving On

RBO• Still there, same behavior as 10.2, but even move

tools to ease the move• SQL performance analyzer• SQL plan management

Moving On

• Stored Outline SQL Plan Management• Ultrasearch Secure Enterprise Search• CTXPath Index XML Index• SYSDBA for ASM administration

SYSASM

No longer supported• Workflow BPEL

These deprecated features / privileges are still available for backward compatibility but the recommendation is to move on

Real Application Testing

• Real Application Testing• Enterprise Manager

Management Packs Enhancements

See:- Oracle Database 11g: Real Application

Testing & Manageability Overview


June 2007

What is Real Application Testing?

• New database option available with EE only• Includes two new features

• Database Replay (DB Replay)• SQL Performance Analyzer (SPA)

• SQL Tuning Set (STS) functionality is shared between Real Application Testing option and EM Tuning Pack

• Advanced functionality requires additional packs• Diagnostic Pack for DB Replay• Tuning Pack for SPA

• License• Requires licenses for both capture and replay system CPUs

Real Application Testing

RemediateRemediate

TestTestTest

DeployDeploy

• Value• Rapid technology adoption• Higher testing quality

• Business Benefit• Lower cost• Lower risk

Solution for the Agile Business

Change

Real Application TestingDatabase Replay

The Need for Database Replay

• Businesses want to adopt new technology that adds value• Extensive testing and validation is expensive in time and cost• Despite expensive testing success rate low

• Many issues go undetected • System availability and performance negatively impacted

• Cause of low success rate• Existing tools provide inadequate testing

• Simulate synthetic workload instead of replaying actual production workload

• Provide partial workflow coverage

Database Replay makes real-world testing possible

Database Replay

• Replay actual production database workload in test environment

• Identify, analyze and fix potential instabilities before making changes to production

• Capture Workload in Production• Capture full production workload with real load, timing &

concurrency characteristics• Move the captured workload to test system

• Replay Workload in Test• Make the desired changes in test system• Replay workload with full production characteristics• Honor commit ordering

• Analyze & Report• Errors• Data divergence • Performance divergence

Analysis & Reporting

Database Replay: Supported Changes

Changes Unsupported

Changes Supported•Database Upgrades, Patches

•Schema, Parameters•RAC nodes, Interconnect

•OS Platforms, OS Upgrades•CPU, Memory

•Storage•Etc.

ClientClient

…Client

Middle Tier

Storage

Recording of External Client

Requests

0

20

40

60

80

Tim

e T

aken

(D

ays)

Install & Setup UnderstandApplication

Usage

Identify KeyTransactions

GenerateWorkload

Run Test

DB Replay

LoadRunner2

205

4

80

24 20

DB Replay: 2 weeks

LoadRunner: 30 weeks

Total Testing Time

500

Comparison of LoadRunner & DB ReplayTesting e-Business Suite

To:From:

Why DB Replay?

Complete workflowsPartial workflows

Low riskHigh risk

AutomatedManual intensive

Production workloadsArtificial workloads

Days of developmentMonths of development

150Days

10Days

…

…

Database Replay WorkflowProduction (10.2.0.4) Test (11.1)

Capture Replay Analysis & ReportingProcess

Storage Storage

Mid-Tier

Replay DriverClients

Step 1: Workload Capture

File 1

File 2

File n

…

Production System • All external client requests

captured in binary files

• System background, internal activity excluded

• Minimal performance overhead for capture

• For RAC, shared and local file system supported

• Specify interesting time period for capture, e.g., peak workload, month-end processing, etc.

• Can capture on 10.2.0.4 and replay on 11g

File System

ClientClient

…Client

Middle Tier

Storage

Capture Options• Workload can be filtered to customize what is captured

• Filter Types• Inclusion Filters: Specifies which sessions should be captured• Exclusion Filters: Specifies which sessions should NOT be

captured• Filter Attributes: Workload capture can be filtered using any of the

following session attributes• User• Program• Module• Action• Service• Session ID

• Workload capture can be run on-demand or scheduled to run at later time

Step 2: Process Workload Files

File 1

File 2

File n

…

File 1

File 2

File n

…

Metadata

Capture Files Replay Files

Test System• Setup test system • Test DB is at same point in time

as before production capture

• Use RMAN to physically restore production db from backup

• Use Snapshot standby

• Use imp/exp, Data Pump, etc.

• Processing transforms captured data into replayable format

• Once processed, workload can be replayed many times

• For RAC copy all capture files to single location for processing

Step 3: Replay Workload

• Replays workload preserving timing, concurrency and dependencies of the capture system

• Replay Driver is a special client program that consumes processed workload and sends requests to the replay system

• Replay Driver consists of one or more clients. For workloads with high concurrency, it may be necessary to start multiple clients to drive workload

Test System

File 1

File 2

File n

…

Metadata

Replay Files

Replay Driver

Replay Options

• Synchronized Replay (Default)• Workload is replayed in full synchronized mode• Exact same concurrency and timing as production workload• Transaction commit order is honored• Ensures minimal data divergence

• Unsynchronized Replay• Workload can be replayed in unsynchronized mode • Useful for load/stress testing• High data divergence• Three (3) parameters provided to control degree of synchronization

• Think time synchronization• Connect (logon) time synchronization• Commit order synchronization

Replay Options

• Number of Replay Clients• Configurable by user• Client Calibration Advisor recommends number of replay

clients needed for specific workload• Replay clients are multithreaded clients that can drive multiple

workload sessions each

Analysis & Reporting

• Comprehensive reports are provided for analysis purposes • There (3) types of divergences are reported

• Data Divergence: Number of rows returned by each call are compared and divergences reported

• Error Divergence: For each call error divergence is reported• New: Error encountered during replay not seen during capture• Not Found: Error encountered during capture not seen during

replay• Mutated: Different error produced in replay than during capture

• Performance Divergence• Capture and Replay Report: Provides high-level performance

information• ADDM Report: Provides in-depth performance analysis• AWR, ASH Report: Facilitates comparative or skew analysis

Database Replay Summary Report

Performance Page – Database Replay

Top Activity Page: Database Replay



Current Restrictions

• Database Replay does not support the following features in the current release

• Direct path load, import/export • OCI based object navigation (ADTs) and REF binds • Streams, non-PL/SQL based AQ • Distributed txns, remote describe/commit operations • Flashback • Shared Server

Best Practices• Capture

• Provide adequate disk space for captured workload (binary files)• Database restart (Optional): Recommended to minimize divergence• For RAC, use shared file system

• Test System Setup • Ensure data in test is identical to production as of capture start time to

minimize data divergence during replay• Use RMAN backup/restore or Snapshot Standby to setup test system• For performance analysis test system capacity should be similar to

production• Reset system clock to same time as production if application logic involves

SYSDATE usage• Process Workload

• Processing workload has performance overhead and can possibly take a long time

• Process workload on test system instead of production• Replay

• Use Client Calibration Advisor to identify number of replay clients needed to replay workload properly

Real Application TestingSQL Performance Analyzer (SPA)

The Need for SQL Performance Analyzer (SPA)

• Businesses want systems that are performant and meet SLA’s

• SQL performance regressions are #1 cause of poor system performance

• Solution for proactively detecting all SQL regressions resulting from changes not available

• DBA’s use ineffective and time-consuming manual process to identify problems

SPA automates identification of all SQL performance regressions resulting from changes

To:

Why SQL Performance Analyzer?

Complete workload

Low risk

Automated analysis in minutes

Automated workload capture

Production workload

From:

Partial workload

High risk

Months of manual analysis

Manual workload capture

Synthetic workload

……

ClientClient

…Client

Capture SQL

• Test impact of change on SQL query performance• Capture SQL workload in production including statistics & bind

variables• Re-execute SQL queries in test environment• Analyze performance changes – improvements and regressions

Middle Tier

Storage

Oracle DB

Re-execute SQL Queries

Production Test

Use SQL Tuning Advisor to tune regression

SQL Performance Analyzer

• Enables identification of SQL performance regressions beforeend-users can be impacted

• SPA can help with any change that impacts SQL execution plan• DB upgrades• Optimizer statistics refresh• New indexes, Materialized Views, Partitions, etc.

• Automates SQL performance tracking of hundreds of thousands of SQL statements – impossible to do manually

• Captures SQL workload with low overhead

• Integrated with SQL Tuning Advisor and SQL Plan Baselines for regression remediation

SPA Benefits

…

SQL Performance Analyzer Workflow

Storage

Production (10.2)

Test (11.1)

Capture SQL

Transport SQL

Execute SQL Pre-change

Execute SQL Post-change

Compare Perf

Storage

Mid-Tier

Clients

Step 1: Capture SQL Workload

• SQL Tuning Set (STS) used to store SQL workload

• STS includes:• SQL Text• Bind variables• Execution plans• Execution statistics

• Incremental capture used to populate STS from cursor cache over a time period

• SQL tuning set’s filtering and ranking capabilities filters out undesirable SQL

• SQL workload captured in 10.2.0.1 and higher can be used for SPA tasks in 11gProduction Database

Cursor Cache

SQL Tuning Set

Incremental Capture

Step 2: Move SQL Workload to Test System

Production Database

Cursor Cache

SQL Tuning Set

Test Database

SQL Tuning Set

• Copy SQL tuning set to staging table (“pack”)• Transport staging table to test system (datapump, db link, etc.)• Copy SQL tuning set from staging table (“unpack”)

Export/Import

Step 3: Execute SQL Before Making Change

• Establishes SQL workload performance baseline

• SQL execution plan and statistics captured

• SQL executed serially (no concurrency)

• Each SQL executed only once

• DDL/DML skipped

• Option to do Explain Plan only analysis

SQL Tuning Set

Fetch Next SQL


Execution Plan &Statistics

Test Execute

Save Result

Before Change

Step 4: Execute SQL After Making Change

• Re-execute SQL after change• Gathers new SQL execution plans

and statistics

SQL Tuning Set


Before Change

Fetch Next SQL

Execution Plan &Statistics

Test Execute

Save Result

After Change

Completed

• Manually implement the planned change• Database upgrade, patches• Optimizer statistics refresh • Schema changes• Database parameter changes• Tuning actions, e.g., SQL Profile creation

Step 5: Compare & Analyze Performance

Before Change

Completed

After Change

Completed

Compare SQL Performance


Analysis Report

• Compare performance using different metrics, e.g.,

• Elapsed Time• CPU Time• Optimizer Cost• Buffer Gets

• SPA Report shows impact of change for each SQL

• Improved SQL• Regressed SQL• Unchanged SQL

• Fix regressed SQL using SQL Tuning Advisor or SQL Plan Baselines

SPA Report

SPA Report – Regressed SQL Statements

Best Practices

• Workload Capture• Use incremental STS capture• Peak or representative workloads can be more accurately captured

with this mechanism• Test system

• Run SPA on test system instead of production as analysis can be resource intensive

• Ensure test system has similar configuration and comparable optimizer statistics as production

• Performance comparison• Use several different metrics, e.g., elapsed time, CPU time, etc., to

compare pre- and post-change performance for reliable results• Regression remediation

• Use SQL Tuning Advisor and SQL Plan Baselines to fix regressions

Real Application Testing Summary

• Provides cost-effective and easy-to-use solution of assessing impact of change on production systems• Holistic and actual workload testing results in lower risk• Brings down testing cycle from months down to days• Lowers hardware costs by eliminating need for mid-tier and

application setup on test system• Maximizes ROI by leveraging Diagnostics and Tuning Pack to

remediate regressions

• With Real Application Testing businesses can• Stay competitive• Improve profitability• Be compliant

Q U E S T I O N SQ U E S T I O N SA N S W E R SA N S W E R S

Thank you for your attention !

43749305 Oracle Db 11g Readiness Technical Uk Sep07 2

Documents