40183 : Oracle XML DB Performance and Tuning Mark D. Drake Senior Product Manager Session id:

40183 :Oracle XML DB Performance and Tuning

Mark D. Drake

Senior Product Manager

Session id:

IntroductionIntroduction

Managing XML Content

XML Stored as XMLType SQL functions allow XPath expressions to

operate on XML content– extract(), extractValue(), updateXML(),

existsNode()

Two modes of operation– Functional Evaluation– Query-Rewrite

Functional Evaluation

Works with all XMLType columns and Tables. Parse the XML document and construct a

DOM Evaluate XPath expressions using DOM API Based on proven XDK ‘C’ XML Parser and

DOM Most appropriate to operations on small

numbers of small documents

XPath Re-write

Query and Update operations Requires Schema based XMLType Operates directly on the underlying Object

Store Relational database engine used to evaluate

XPath expression– XPath Expression translated into equivalent SQL– Eliminates need to parse XML and construct DOM

Allows efficient operations on large collections of XML Documents

CollectionsCollections

What is a collection

A Collection is a set of elements with the same name and similar structure

<LineItem ItemNumber="1"> <Description>Duel at Ichijoji Temple</Description> <Part Id="37429125526" UnitPrice="29.95"

Quantity="3"/</LineItem><LineItem ItemNumber="2"> <Description>The Red Shoes</Description> <Part Id="37429128220" UnitPrice="39.95"

Quantity="4"/></LineItem>

Defining a Collection

In XML Schema, Collections are defined using maxOccurs attribute

<xs:complexType name="LineItemsType“

<xs:sequence>

<xs:element name="LineItem“

type="LineItemType“

maxOccurs="unbounded"/>

</xs:sequence>

</xs:complexType>

Working with Collections

A Typical XML document contains one or more collections

Collections may be Nested– A collection may contain a collections

Query and Updating collections present the biggest challenge

Querying Collections

Sample Query on a CollectionGet the Reference and Item Number of each LineItem that contains a reference to Part “717951002372“

XPATH :='/PurchaseOrder/LineItems/LineItem[Part/@Id="717951002372"]';select extractValue(p.object_value,'/PurchaseOrder/Reference') extractValue(value(l),'/LineItem/@ItemNumber') from PURCHASEORDER p, table (xmlsequence(extract(p.object_value,XPATH))) l

Querying Collections

Use the extract() function to get the set of nodes that match an XPath expression

XPATH :='/PurchaseOrder/LineItems/LineItem[Part/@Id="717951002372"]';select extractValue(object_value,'/PurchaseOrder/Reference') extractValue(value(l),'/LineItem/@ItemNumber') from PURCHASEORDER p, table (xmlsequence(extract(p.object_value,XPATH))) l

Querying Collections

The extract function will be evaluated for each document in the table.

An XMLType will be generated from each document that contains a node matching the XPath expression.

The XML Type will be contain an XML Fragment – The Fragment will consist of the set of nodes that match

the specified XPATH expression.

Querying Collections

Use the xmlSequence function to generate a set of XMLType Objects from each fragment

XPATH :='/PurchaseOrder/LineItems/LineItem[Part/@Id="717951002372"]';select extractValue(object_value,'/PurchaseOrder/Reference') extractValue(value(l),'/LineItem/@ItemNumber') from PURCHASEORDER p, table (xmlsequence(extract(p.object_value,XPATH))) l

Querying Collections

xmlSequence will create a set of XMLType objects from the result of the extract.

Each set will contain one XMLType object for each top level node in the XML fragment.

Each XMLType will consist of a well-formed XML document.

Querying Collections

Use the TABLE function to convert the set of XMLType objects into a virtual table

XPATH :='/PurchaseOrder/LineItems/LineItem[Part/@Id="717951002372"]';select extractValue(p.object_value,'/PurchaseOrder/Reference') extractValue(value(l),'/LineItem/@ItemNumber') from PURCHASEORDER p, table (xmlsequence(extract(p.object_value,XPATH))) l

Join the virtual table with the base table to get required result set

ManagingCollectionsManaging

Collections

Collection Management

Four options for storing collections– BLOB Storage

SQL objects stored in a single BLOB Column

– Nested Table Storage SQL objects stored as rows in a Nested Table

– CLOB Storage XML text

– XMLType Table Storage XMLType objects stored in an XMLType Table

Why different storage models ?

Applications have different requirements– Querying within in a collection– Indexing within a collection– Updating within a collection

Choosing the correct storage model is vital

Collections and VARRAYS

VARRAY: SQL concept for managing collections

VARRAY can be collection of Scalars or collection of SQL Type

– AUTHOR_V VARRAY(n) of VARCHAR2(40)– LINE_ITEM_V VARRAY(n) of LINE_ITEM_T

In XML DB the members of a collection are mapped to a SQL VARRAY.

LOBLOB

LOB model

Default : No Annotation Required Collection stored as VARRAY of SQL Objects SQL Objects serialized into a single BLOB Indexing using CtxXPath index No partial updates

– update requires re-writing entire BLOB

Query re-write for XPath expressions

LOB : Objects<xs:complexType name="LineItemsType" xdb:SQLType="LINEITEMS_T"> <xs:sequence> <xs:element name="LineItem"

type="LineItemType" maxOccurs="unbounded" xdb:SQLName="LINEITEM" xdb:SQLCollType="LINEITEM_V"/>

</xs:sequence></xs:comlexType>

SQL> desc LINEITEMS_T LINEITEMS_T is NOT FINAL Name Null? Type ------------------- -------- ------- SYS_XDBPD$ XDB.XDB$RAW_LIST_T LINEITEM LINEITEM_V

LOB : StorageSQL> describe LINEITEM_V LINEITEM_V VARRAY(2147483647) OF LINEITEM_T LINEITEM_T is NOT FINAL Name Null? Type ------------------- -------- ------- SYS_XDBPD$ XDB.XDB$RAW_LIST_T ITEMNUMBER NUMBER(38) DESCRIPTION VARCHAR2(256 CHAR) PART PART_T

LOB Storage

ReferenceReference User IdUser Id …… LineItemLineItem

ABANDAABANDA ……ABANDA-20..

LINEITEM_V( LINEITEM_T(...), LINEITEM_T(...), LINEITEM_T(...), LINEITEM_T(...))

LOB : Query Re-Write

SELECT P.XMLDATA."Reference", L."ItemNumber" FROM PURCHASEORDER P, TABLE (P.XMLDATA."LineItems"."LineItem") L WHERE L."Part"."PartNumber" = '717951002372'

Evaluated using SQL to process VARRAY of LINEITEM objects

Entire VARRAY has to be loaded into Memory

NestedTablesNestedTables

Nested Table model

Annotation: xdb:storeVarrayAsTable=“true” Collection converted into a set of SQL objects Each SQL Object stored as a separate row in

a Nested Table. Nesting of Nested Tables for collections within

collections SQL Based Fragment access and update Index with B-Tree, Functional and CtxXPath

indexes

Nested Table : Objects<xs:complexType name="LineItemsType" xdb:SQLType="LINEITEMS_T"> <xs:sequence> <xs:element name="LineItem"

type="LineItemType" maxOccurs="unbounded" xdb:SQLName="LINEITEM" xdb:SQLCollType="LINEITEM_V"/>

</xs:sequence></xs:comlexType>

SQL> desc LINEITEMS_T LINEITEMS_T is NOT FINAL Name Null? Type ------------------- -------- ------- SYS_XDBPD$ XDB.XDB$RAW_LIST_T LINEITEM LINEITEM_V

Nested Table : ObjectsSQL> describe LINEITEM_V LINEITEM_V VARRAY(2147483647) OF LINEITEM_T LINEITEM_T is NOT FINAL Name Null? Type ------------------- -------- ------- SYS_XDBPD$ XDB.XDB$RAW_LIST_T ITEMNUMBER NUMBER(38) DESCRIPTION VARCHAR2(256 CHAR) PART PART_TSQL> describe LINEITEM_TABLE Name Null? Type ------------------- -------- ------- SYS_XDBPD$ XDB.XDB$RAW_LIST_T ITEMNUMBER NUMBER(38) DESCRIPTION VARCHAR2(256 CHAR) PART PART_T

Nested Table : Storage

ReferenceReference User IdUser Id …… LineItemLineItem

ABANDAABANDA ……ABANDA-20..

IDID ItemNumberItemNumber DescriptionDescription PartPart

Good Morn…Good Morn… ……1

Uriah Hee…Uriah Hee… ……21

SistersSisters ……31

The Prince…The Prince… ……41

1

1

Nested Table : Query Re-Write

SELECT P.XMLDATA."Reference", L.ITEMNUMBER FROM PURCHASEORDER P, LINEITEM_TABLE l WHERE L."Part."PartNumber"='717951002372'

AND L.NESTED_TABLE_ID = SETID(P.XMLDATA."LineItems"."LineItem“)

Evaluated using SQL to query contents of the Nested Table

Working with Nested Tables

xdb:storeVarrayAsTable=“true” is a “shotgun” approach

– Every collection stored as a Nested Table– Nested Tables have ‘unfriendly’ names

Simple cases– Rename tables created by Schema Registration

Complex cases – Create tables manually using CREATE TABLE

Renaming Nested Tables

Query USER_NESTED_TABLES to find the direct descendant of the Parent Table

SQL> select PARENT_TABLE_COLUMN, TABLE_NAME 2 from USER_NESTED_TABLES 3 where PARENT_TABLE_NAME = 'PURCHASEORDER' 4 /

PARENT_TABLE_COLUMN TABLE_NAME-------------------------------- -----------------------"XMLDATA"."LINEITEMS"."LINEITEM" SYS_NTDfLwYKWcRxmyssvmCvRMqw=="XMLDATA"."ACTIONS"."ACTION" SYS_NTqxF5epLrSniXaAWq5A4Uig==

Renaming Nested Tables

SQL> rename "SYS_NTDfLwYKWcRxmyssvmCvRMqw==" to LINEITEM_TABLE 2 /Table renamed.

SQL> rename "SYS_NTqxF5epLrSniXaAWq5A4Uig==" to ACTION_TABLE 2 /Table renamed.

SQL> select PARENT_TABLE_COLUMN, TABLE_NAME 2 from USER_NESTED_TABLES 3 where PARENT_TABLE_NAME = 'PURCHASEORDER' 4 /PARENT_TABLE_COLUMN TABLE_NAME--------------------------------- ----------------------"XMLDATA"."ACTIONS"."ACTION" ACTION_TABLE"XMLDATA"."LINEITEMS"."LINEITEM" LINEITEM_TABLE

Multi-Level Nested Tables

Collections within Collections result in Storage Models consisting of multiple levels of Nested Tables

Repeat the process using the Nested Table name as the PARENT_TABLE_NAME

Manual Table Creation

Can create Nested Tables as part of create Table

Need to know the name of the VARRAYs that will stored as Nested Tables

Allows flexibility, only use Nested Table Storage only where it adds value

Must use Nested Table storage with collections that contain an element stored as a CLOB.

Manual Table Creation Example

create table PURCHASEORDER of XMLTYPEXMLSCHEMA

"http://localhost:8080/home/SCOTT/xsd/purchaseOrder.xsd"ELEMENT "PurchaseOrder"varray "XMLDATA"."ACTIONS"."ACTION" store as table ACTION_TABLE ( (primary key (NESTED_TABLE_ID, ARRAY_INDEX)) organization index overflow )varray "XMLDATA"."LINEITEMS"."LINEITEM" store as table LINEITEM_TABLE ( (primary key (NESTED_TABLE_ID, ARRAY_INDEX)) organization index overflow )

CLOBCLOB

CLOB model

Annotation: xdb:SQLType=“CLOB” Collection stored as XML text

– Reduced complexity– Fast Ingestion and Retrieval of Collection – Lower memory and processing requirements

Index using CTXPATH index No Optimization of Fragment Level operations

– XPath based Functions evaluated via DOM API– Fetching Fragments requires Parsing– Updating Requires Parsing and Rewriting

CLOB : Objects

<xs:complexType name="LineItemsType" xdb:SQLType="LINEITEMS_T"> <xs:sequence> <xs:element name="LineItem"

type="LineItemType" maxOccurs="unbounded" xdb:SQLName="LINEITEM" xdb:SQLType="CLOB"/>

</xs:sequence></xs:comlexType>

SQL> describe LINEITEMS_T LINEITEMS_T is NOT FINAL Name Null? Type ------------------- -------- ------- SYS_XDBPD$ XDB.XDB$RAW_LIST_T LINEITEM CLOB

CLOB : Storage

ReferenceReference User IdUser Id …… LineItemLineItem

ABANDAABANDA ……ABANDA-20..

<LineItem ItemNumber="1"> <Description>Good … <Part Id="3742914 …</LineItem><LineItem ItemNumber="2“ <Description>Uriah.. <Part Id="6950030…</LineItem>

CLOB: Query Re-Write

SELECT P.XMLDATA."Reference", EXTRACTVALUE(value(l),'/LineItem/@ItemNumber') FROM PURCHASEORDER P, TABLE (

XMLSEQUENCE (

EXTRACT (

VALUE(P), '/PurchaseOrder/LineItems/LineItem/' || 'Part/@Id="717951002372"]' ) ) ) l

Evaluated using by Parsing CLOB and using DOM API

XMLTypeTable

XMLTypeTable

XMLType Table model

Annotation xdb:SQLInline=“false” Also known as ‘out-of-line’ storage Collection converted into a set of XMLType objects

stored in a separate XMLType table. Parent table contains a VARRAY of REF XMLTYPE

that points to rows in the child table Index using B-Tree, Functional and CtxXPath indexes SQL Based Fragment access and update. Must allow Schema Registration to create the Tables.

XMLType Table : Objects<xs:complexType name="LineItemsType" xdb:SQLType="LINEITEMS_T"> <xs:sequence> <xs:element name="LineItem"

type="LineItemType" maxOccurs="unbounded" xdb:SQLName="LINEITEM" xdb:SQLInline="false“ xdb:defaultTable="LINEITEM_TABLE"/>

</xs:sequence></xs:comlexType>

SQL> describe LINEITEMS_T LINEITEMS_T is NOT FINAL Name Null? Type ------------------- -------- ------- SYS_XDBPD$ XDB.XDB$RAW_LIST_T LINEITEM XDB.XDB$XMLTYPE_REF_LIST_T

XMLType Table : ObjectsSQL> describe XDB.XDB$XMLTYPE_REF_LIST_TXDB.XDB$XMLTYPE_REF_LIST_T VARRAY(2147483647)

OF REF XMLTYPE

SQL> describe LINEITEM_TABLEName Null? Type ------------------------- -------- ----------------------------TABLE of SYS.XMLTYPE

( XMLSchema "http://localhost:8080/home/SCOTT/xsd/purchaseOrder.xsd"

Element "LineItem“) STORAGE Object-relational TYPE "LINEITEM_T"

XMLType with Ref Table

Combine with xdb:storeVarrayAsTable=“true” to force VARRAY of REFS to be stored in a nested table

Each row in the nested table consists of SETID and a REF to a ROW in the child table

Scope the REFS in the child table to allow full optimization of queries

rename "SYS_NTDfLwYKWcRxmyssvmCvRMqw==" to LINEITEM_REF_TABLE

alter table LINEITEM_REF_TABLE add (scope for (COLUMN_VALUE) is LINEITEM_TABLE)

Scope REF of XMLType

select PARENT_TABLE_COLUMN, TABLE_NAME from USER_NESTED_TABLES where PARENT_TABLE_NAME = 'PURCHASEORDER‘/PARENT_TABLE_COLUMN TABLE_NAME-------------------------------- -----------------------"XMLDATA"."LINEITEMS"."LINEITEM" SYS_NTDfLwYKWcRxmyssvmCvRMqw==

rename "SYS_NTDfLwYKWcRxmyssvmCvRMqw==" to LINEITEM_REF_TABLE/alter table LINEITEM_REF_TABLE add (scope for (COLUMN_VALUE) is LINEITEM_TABLE)/create index LINEITEM_REF_INDEX on LINEITEM_REF_TABLE (COLUMN_VALUE)/

XMLType Table : Query Rewrite

SELECT P.XMLDATA."Reference", L.XMLDATA."Item" FROM PURCHASEORDER p, TABLE("P.XMLDATA."LineItems"."LineItem") R, LINEITEM_TABLE L WHERE L."Part"."PartNumber" = '717951002372' AND VALUE(R)= Ref(l)

Evaluated using SQL to query contents of the XMLType table

Relative FTP Insert performance

00.5

11.5

22.5

33.5

44.5

5

CLOB LOB NestedTable

XMLTypeTable

ScopedXMLType

Table

Un-Indexed Query Performance

0

2

4

6

8

10

12

14

CLOB LOB NestedTable

XMLTypeTable

ScopedXMLType

Table

IndexingCollections

IndexingCollections

Indexing Collections

Collections stored as LOB or CLOB can only be indexed using CtxXPath Index

Collections stored as Nested Tables or Out-of-Line tables can be index using B-Tree Indexes

– Indexes on collections are created on the nested table or Out-of-Line Table, not the Parent Table

Indexing with Nested Tables

Indexes on Nested Table are defined in terms of the SQL Type structure

Nested Tables contain a foreign key called NESTED_TABLE_ID

Consider adding NESTED_TABLE_ID to any secondary indexes

Nested Table: Secondary Index

create index iPartNumberIndex on LINEITEM_TABLE l(l.PART.PART_NUMBER, l.NESTED_TABLE_ID)

Query Plan showing use of Index

---------------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |---------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 662 | 17M| 89 (2)| 00:00:02 || 1 | NESTED LOOPS | | 662 | 17M| 89 (2)| 00:00:02 ||* 2 | INDEX UNIQUE SCAN | LINEITEM_TABLE_IOT | 82 | 370K| 6 (0)| 00:00:01 ||* 3 | INDEX RANGE SCAN | IPARTNUMBERINDEX | 33 | | 3 (34)| 00:00:01 || 4 | TABLE ACCESS BY INDEX ROWID| PURCHASEORDER | 8 | 176K| 2 (50)| 00:00:01 ||* 5 | INDEX UNIQUE SCAN | SYS_C004261 | 1 | | 1 (100)| 00:00:01 |---------------------------------------------------------------------------------------------------

Indexing with XMLType Tables

Indexes on Out-Of-Line XMLType tables are created using XPath syntax

Index is defined in terms of the extractValue() functions

Index is not a function index. XPath re-write allows the index to be converted into Object-Relational SQL.

XPath is from the root element of the Out-of-Line table, not the root of the parent document

When storing REFs as nested table remember to index the nested table

XMLType Table: Secondary Index

create index iPartNumberIndex on LINEITEM_TABLE l(extractValue(value(l),'/LineItem/Part/@Id'))create index LINEITEM_REF_INDEX on LINEITEM_REF_TABLE(COLUMN_VALUE)

Query Plan showing use of Index----------------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |----------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 4626 | 100M| 591 (2)| 00:00:08 || 1 | NESTED LOOPS | | 4626 | 100M| 591 (2)| 00:00:08 ||* 2 | HASH JOIN | | 571 | 136K| 14 (15)| 00:00:01 || 3 | TABLE ACCESS BY INDEX ROWID| LINEITEM_TABLE | 571 | 115K| 6 (17)| 00:00:01 ||* 4 | INDEX RANGE SCAN | IPARTNUMBERINDEX | 571 | | 2 (50)| 00:00:01 || 5 | INDEX FAST FULL SCAN | LINEITEM_REF_INDEX | 8168 | 295K| 8 (13)| 00:00:01 || 6 | TABLE ACCESS BY INDEX ROWID | PURCHASEORDER | 8 | 176K| 2 (50)| 00:00:01 ||* 7 | INDEX RANGE SCAN | SYS_C004315 | 8 | | 1 (100)| 00:00:01 |----------------------------------------------------------------------------------------------------

Indexed Query Performance

0

0.5

1

1.5

2

2.5

3

3.5

Nested Table XMLType Table Scoped XMLTypeTable

DOMFidelity

DOMFidelity

Dom Fidelity

Requires additional processing and storage overhead Specific meta data maintained for each instance

– Meta data managed at the ‘Type’ Level in the SYS_XDBPD$ attribute

– SYS_XDBPD$ is stored as a (in-line) LOB.

Do you need it ?

Can disable on a Type by Type basis using annotation xdb:maintainDOM=“false”

Not using Namespaces No Comments or Processing Instructions Do not care about empty Vs missing elements Elements in an All and Choice will be ordered

as per the Schema, not the instance Not worried about defaults Type does not allow Mixed text

Type defn with DOM Fidelity

desc LINEITEMS_T LINEITEMS_T is NOT FINAL Name Null? Type ------------------------------- -------- ------------------ SYS_XDBPD$ XDB.XDB$RAW_LIST_T LINEITEM LINEITEM_V

desc LINEITEM_V LINEITEM_V VARRAY(2147483647) OF LINEITEM_T LINEITEM_T is NOT FINAL Name Null? Type ------------------------------ -------- ------------------ SYS_XDBPD$ XDB.XDB$RAW_LIST_T ITEMNUMBER NUMBER(38) DESCRIPTION VARCHAR2(256 CHAR) PART PART_T

xdb:maintainDOM

<xs:complexType name="LineItemsType" xdb:SQLType="LINEITEMS_T" xdb:maintainDOM="false"> <xs:sequence> <xs:element name="LineItem" type="LineItemType" maxOccurs="unbounded" xdb:SQLName="LINEITEM" xdb:SQLCollType="LINEITEM_V"/> </xs:sequence></xs:complexType>

<xs:complexType name="LineItemType" xdb:SQLType="LINEITEM_T" xdb:maintainDOM="false"> <xs:sequence> <xs:element name="Description" type="DescriptionType“ xdb:SQLName="DESCRIPTION"/> <xs:element name="Part" type="PartType" xdb:SQLName="PART"/> </xs:sequence> <xs:attribute name="ItemNumber" type="xs:integer" xdb:SQLName="ITEMNUMBER" xdb:SQLType="NUMBER"/></xs:complexType>

Type defn without DOM Fidelity

desc LINEITEMS_T LINEITEMS_T is NOT FINAL Name Null? Type ------------------------------- -------- ------------------ LINEITEM LINEITEM_V

desc LINEITEM_V LINEITEM_V VARRAY(2147483647) OF LINEITEM_T LINEITEM_T is NOT FINAL Name Null? Type ------------------------------ -------- ------------------ ITEMNUMBER NUMBER(38) DESCRIPTION VARCHAR2(256 CHAR) PART PART_T

Dom Fidelity

00.10.20.30.4

0.50.60.70.80.9

1

Files DOM Fidelity No Fidelity

Insert TimeSize on Disk

HierarchicalTables

HierarchicalTables

Hierarchical Tables

XMLType Tables can be Hierarchically Enabled

– Table is linked to the Oracle XML DB repository– Each Row in the table corresponds to a

Resource (document) in the XML DB repository– Row Level Security defined by associating an

ACL with a resource– DML operations on the row reflected in the

resource (Last Modified Date, Delete)

Hierarchical Considerations

Tables created by Schema Registration are Hierarchically Enabled

– DML operations constrained by ACL– Table Level Triggers sync DML operations on

base-table with XDB repository

Cannot bypass ACL security by accessing content via SQL

Delete of a row in a table will delete the corresponding Resource

ACL Management

XML DB architected to support systems with thousands or millions of ACLs

ACL based security automatically enforced by calling the sys_checkAcl() function

ACL check is based on ACLOID and OID Dedicated ACL cache ensures that Oracle XML DB

can locate ACLs very efficiently ACLOID and OID are stored as hidden columns on an

XMLType table

ACL Checking

ACL checking can affect query performance– Need ACLOID and OID in addition to other

columns

ACL Based security can be disabled using dbms_xdbz.disable_hierarchy()

– Turns off ACL Evaluation and repository synchronization

In some cases disabling the hierarchy can lead to significant performance benefits

AQ&Q U E S T I O N SQ U E S T I O N S

A N S W E R SA N S W E R S