Chapter 10: XML

Chapter 10: XML

The world of XML

Context

• The dawn of database technology 70s

• A DBMS is a flexible store-recall system for digital information

• It provides permanent memory for structured information

Context

• Database Managements technology for administrative settings ‘completed’ in the early 80s

• Search for demanding application areas that could benefit from a database approach

– A sound datamodel to structure the information and maintain integrity rules

– A high level programming language model to manipulate the data

– Separation of concerns between modelling and manipulation, and physical storage and order of execution thanks to query optimizer technology

Context

• Demanding areas of research in DBMS core technology:– Office Information systems, e.g. document modelling

and workflow– CAD/CAM, e.g. how to manage the design of an

airplane or nucleur power plant– GIS, e.g. managing remote sensing information– WWW, e.g. how to integrate heterogenous sources– Agent-based systems, e.g. reactive systems– Multimedia, e.g. video storage/retrieval– Datamining, e.g. discovery of client profiles– Sensor networks, e.g. small footprint and energy-wise

computing

Context

• Demanding areas of research in DBMS core technology:

– Office Information systems, Extensible DBMS, blobs

– CAD/CAM, Object-oriented DBMS, geometry

– GIS, GIS DBMS, geometry and images

– Agent-based systems, Active DBMS, triggers

– Multimedia, MM DBMS, feature analysis

– Datamining, Datawarehouse systems, cube, association rules

– Sensor networks, P2P databases, ad-hoc networking

Context

• Application interaction with DBMS– Proprietary application programming interface,

shielding the hardware distinctions– Use readable interfaces to improve monitoring and

development• Example: in Monetdb the interaction is based on ascii text with

the first character indicative for the message type‘>’ prompt, await for next request‘!’ error occurred, rest is the message‘[‘ start of a tuple answer

– Language embedding to remove the impedance mismatch, i.e. avoid cost of transforming data

• Effectively failed in the OO world

Context

• Learning points database perspective, – Database system should not be concerned with the

user-interaction technology, ‘they should be blind and deaf’

– The strong requirements on schema, integrity rules and processing is a harness

– Interaction with applications should be self-descriptive as much as possible, because, you can not a priori know a complete schema

– Need for semi-structured databases

Semi-structured data

• Properties of semistructured databases:– The schema is not given in advance and may be implicit in the

data

– The schema is relatively large and changes frequently

– The schema is descriptive rather than prescriptive, integrity rules may be violated

– The data is not strongly typed, the values of attributes may be of different type

• Stanford Lore system is the prototypical first attempt to support semi-structured databases

Context

Accidentally, in the world of digital publishing there is a need for a simple datamodel to structure information

SMGL HTML XML XHTML

XPATH XQUERY XSLT

By the end 90s, the document world meets the database world

Introduction

• XML: Extensible Markup Language

• Defined by the WWW Consortium (W3C)

• Originally intended as a document markup language not a database language– Documents have tags giving extra information about sections of the

document• E.g. <title> XML </title> <slide> Introduction …</slide>

– Derived from SGML (Standard Generalized Markup Language), but simpler to use than SGML

– Extensible, unlike HTML

• Users can add new tags, and separately specify how the tag should be handled for display

XML Introduction (Cont.)

• The ability to specify new tags, and to create nested tag structures made XML a great way to exchange data, not just documents.– Much of the use of XML has been in data exchange applications, not as a

replacement for HTML

• Tags make data (relatively) self-documenting – E.g.

<bank> <account>

<account-number> A-101 </account-number> <branch-name> Downtown </branch-name> <balance> 500 </balance>

</account> <depositor>

<account-number> A-101 </account-number> <customer-name> Johnson </customer-name>

</depositor> </bank>

XML: Motivation

• Data interchange is critical in today’s networked world

– Examples:• Banking: funds transfer

• Order processing (especially inter-company orders)

• Scientific data

– Chemistry: ChemML, …

– Genetics: BSML (Bio-Sequence Markup Language), …

– Paper flow of information between organizations is being replaced by electronic flow of information

• Each application area has its own set of standards for representing information (W3C maintains ca 30 standards)

• XML has become the basis for all new generation data interchange formats

XML Motivation (Cont.)• Each XML based standard defines what are valid elements,

using

– XML type specification languages to specify the syntax• DTD (Document Type Descriptors)

• XML Schema

– Plus textual descriptions of the semantics

• XML allows new tags to be defined as required

– However, this may be constrained by DTDs

• A wide variety of tools is available for parsing, browsing and querying XML documents/data

Motivation for Nesting

• Nesting of data is useful in data transfer

– Example: elements representing customer-id, customer name, and address nested within an order element

• Nesting is not supported, or discouraged, in relational databases

– With multiple orders, customer name and address are stored redundantly

– normalization replaces nested structures in each order by foreign key into table storing customer name and address information

– Nesting is supported in object-relational databases and NF2

• But nesting is appropriate when transferring data

– External application does not have direct access to data referenced by a foreign key

Example of Nested Elements

<bank-1> <customer>

<customer-name> Hayes </customer-name> <customer-street> Main </customer-street> <customer-city> Harrison </customer-city> <account>

<account-number> A-102 </account-number> <branch-name> Perryridge </branch-name> <balance> 400 </balance>

</account> <account> … </account>

</customer> . .

</bank-1>

Structure of XML Data (Cont.)

• Mixture of text with sub-elements is legal in XML. – Example: <account>

This account is seldom used any more. <account-number> A-102</account-number> <branch-name> Perryridge</branch-name> <balance>400 </balance></account>

– Useful for document markup, but discouraged for data representation

Attributes

• Elements can have attributes– <account acct-type = “checking” >

<account-number> A-102 </account-number> <branch-name> Perryridge </branch-name> <balance> 400 </balance>

</account>

• Attributes are specified by name=value pairs inside the starting tag of an element

• An element may have several attributes, but each attribute name can only occur once

• <account acct-type = “checking” monthly-fee=“5”>

Attributes Vs. Subelements

• Distinction between subelement and attribute

– In the context of documents, attributes are part of markup, while subelement contents are part of the basic document contents

– In the context of data representation, the difference is unclear and may be confusing

• Same information can be represented in two ways

– <account account-number = “A-101”> …. </account>

– <account> <account-number>A-101</account-number> … </account>

– Suggestion: use attributes for identifiers of elements, and use subelements for contents

XML Document Schema

• Database schemas constrain what information can be stored, and the data types of stored values

• XML documents are not required to have an associated schema

• However, schemas are very important for XML data exchange– Otherwise, a site cannot automatically interpret data

received from another site• Two mechanisms for specifying XML schema

– Document Type Definition (DTD)• Widely used

– XML Schema • Newer, not yet widely used

Attribute Specification in DTD

• Attribute specification : for each attribute – Name– Type of attribute

• CDATA• ID (identifier) or IDREF (ID reference) or IDREFS (multiple IDREFs)

– more on this later

– Whether • mandatory (#REQUIRED)• has a default value (value), • or neither (#IMPLIED)

• Examples– <!ATTLIST account acct-type CDATA “checking”>– <!ATTLIST customer

customer-id ID # REQUIREDaccounts IDREFS # REQUIRED >

IDs and IDREFs

• An element can have at most one attribute of type ID

• The ID attribute value of each element in an XML document must be distinct

– Thus the ID attribute value is an object identifier

• An attribute of type IDREF must contain the ID value of an element in the same document

• An attribute of type IDREFS contains a set of (0 or more) ID values. Each ID value must contain the ID value of an element in the same document

Limitations of DTDs

• No typing of text elements and attributes

– All values are strings, no integers, reals, etc.

• Difficult to specify unordered sets of subelements

– Order is usually irrelevant in databases

– (A | B)* allows specification of an unordered set, but• Cannot ensure that each of A and B occurs only once

• IDs and IDREFs are untyped

– The owners attribute of an account may contain a reference to another account, which is meaningless

• owners attribute should ideally be constrained to refer to customer elements

XML Schema

• XML Schema is a more sophisticated schema language which addresses the drawbacks of DTDs. Supports

– Typing of values• E.g. integer, string, etc

• Also, constraints on min/max values

– User defined types

– Is itself specified in XML syntax, unlike DTDs• More standard representation, but verbose

– Is integrated with namespaces

– Many more features• List types, uniqueness and foreign key constraints, inheritance ..

• BUT: significantly more complicated than DTDs, not yet widely used.

Storage of XML Data

• XML data can be stored in

– Non-relational data stores• Flat files

– Natural for storing XML– But has all problems discussed in Chapter 1 (no

concurrency, no recovery, …)• XML database

– Database built specifically for storing XML data, supporting DOM model and declarative querying

– Currently no commercial-grade scaleable system

– Relational databases• Data must be translated into relational form• Advantage: mature database systems• Disadvantages: overhead of translating data and queries

Storing XML in Relational Databases

• Store as string– E.g. store each top level element as a string field of a tuple in a

database• Use a single relation to store all elements, or• Use a separate relation for each top-level element type

– E.g. account, customer, depositor– Indexing:

» Store values of subelements/attributes to be indexed, such as customer-name and account-number as extra fields of the relation, and build indices

» Oracle 9 supports function indices which use the result of a function as the key value. Here, the function should return the value of the required subelement/attribute

» SQL server 2005 same

Storing XML in Relational Databases

• Store as string– E.g. store each top level element as a string field of a tuple in a

database– Benefits:

• Can store any XML data even without DTD• As long as there are many top-level elements in a document, strings are

small compared to full document, allowing faster access to individual elements.

– Drawback: Need to parse strings to access values inside the elements; parsing is slow.

OEM model

• Semi structured and XML databases can be modelled as graph-problems

• Early prototypes directly supported the graph model as the physical implementation scheme. Querying the graph model was implemented using graph traversals

• XML without IDREFS can be modelled as trees

Storing XML as Relations (Cont.)

• Tree representation: model XML data as tree and store using relations nodes(id, type, label, value) child (child-id, parent-id)– Each element/attribute is given a unique identifier– Type indicates element/attribute– Label specifies the tag name of the element/name of attribute– Value is the text value of the element/attribute– The relation child notes the parent-child relationships in the tree

• Can add an extra attribute to child to record ordering of children

– Benefit: Can store any XML data, even without DTD– Drawbacks:

• Data is broken up into too many pieces, increasing space overheads• Even simple queries require a large number of joins, which can be

slow

Storing XML in Relations (Cont.)• Map to relations • If DTD of document is known, you can map data to relations

– Bottom-level elements and attributes are mapped to attributes of relations

– A relation is created for each element type• An id attribute to store a unique id for each element• all element attributes become relation attributes• All subelements that occur only once become attributes

– For text-valued subelements, store the text as attribute value– For complex subelements, store the id of the subelement

• Subelements that can occur multiple times represented in a separate table

– Similar to handling of multivalued attributes when converting ER diagrams to tables

– Benefits: • Efficient storage• Can translate XML queries into SQL, execute efficiently, and then

translate SQL results back to XML

Alternative mappings

• Mapping the structure– The Edge approach– The Attribute approach– The Universal Table approach– The Normalized Universal approach– The Dataguide approach

• Mapping values– Separate value tables– Inlining

• Shredding

Edge approach

• Use a single Edge table to capture the graph structure

Edge(source, ordinal, name, flag, target)

Flag: {value, reference}

Keys: {source, ordinal)

Index: source, {name,target}

Attribute approach

• Group all attributes with the same name into one table

Aname(source,ordinal,flag, target)

Key: {source,ordinal}

Index:{target}

Universal approach

• Use the Universal Table, all attributes are stored as columnsUniversal(source, ord-1,flag-1,target-1, …,ord-n,flag-n,target-n)

Key: source, index: target-i

Normalized Universal

• Same as Universal, but factor out the repeating values

Universal(source, ord-1,flag-1,target-1, …,ord-n,flag-n,target-n)

Overflow_n(source,ord, flag,target)

Key: source, and {source,ord}

Index: target-i

Mapping values

• Separate value tables

– Use V_type(vid, value) tables, eg. int(vid,val), str(vid,val),….

Mapping values

• Inlining

– As illustrated in previous mappings, inline the values in the structure relations

Shredding

• Try to recognize repeating structures and map them to separate tables

• Handle the remainder through any of the previous methods

Evaluation

• Some results reported by Florescu, Kossmann using a commercial DBMS on documents of 100K objects in 1999

• Database storage overhead:

Evaluation

• Some results reported by Florescu, Kossmann using a commercial DBMS on documents of 100K objects in 1999

• Bulk loading:

Evaluation

• Some results reported by Florescu, Kossmann using a commercial DBMS on documents of 100K objects in 1999

• Reconstruction:

The Data

Semistructured data instance = a large graph

The indexing problem

• The storage problem

– Store the graph in a relational DBMS

– Develop a new database storage structure

• The indexing problem:

– Input: large, irregular data graph

– Output: index structure for evaluating (regular) path expressions, e.g.

bib.paper.author.firstname

XSet: a simple index for XML

• Part of the Ninja project at Berkeley• Example XML data:

XSet: a simple index for XML

Each node = a hashtable

Each entry = list of pointers to data nodes (not shown)

XSet: Efficient query evaluation

• SELECT X FROM part.name X -yes• SELECT X FROM part.supplier.name X -yes• SELECT X FROM part.*.subpart.name X -maybe• SELECT X FROM *.supplier.name X -maybe

Will gain when index fits in memory

Region Algebras

• structured text = text with tags (like XML)

• data = sequence of characters [c1c2c3 …]

• region = interval in the text

– representation (x,y) = [cx,cx+1, … cy]

– example: <section> … </section>• region set = a set of regions

– example all <section> regions (may be nested)

• region algebra = operators on region set, s1 op s2s1 op s2

Region algebra: some operators

• s1 intersect s2 = {r | r s1, r s2}

• s1 included s2 = {r | rs1, r’ s2, r r’}

• s1 including s2 = {r | r s1, r’ s2, r r’}

• s1 parent s2 = {r | r s1, r’ s2, r is a parent of r’}

• s1 child s2 = {r | r s1, r’ s2, r is child of r’}

Examples:

<subpart> included <part>

<part> including <subpart>

Efficient computation of Region Algebra Operators

Example: s1 included s2

s1 = {(x1,x1'), (x2,x2'), …}

s2 = {(y1,y1'), (y2,y2'), …}

(i.e. assume each consists of disjoint regions)

Algorithm:

if xi < yj then i := i + 1

if xi' > yj' then j := j + 1

otherwise: print (xi,xi'), do i := i + 1

Can do in sub-linear time when one region is very small

From path expressions to region expressions

part.name name child (part child root)

part.supplier.name name child (supplier child (part child root))

*.supplier.name name child supplier

part.*.subpart.name name child (subpart included (part child root))

Region expressions correspond to simple XPath expressions

Storage structures for region algebras

• Every node is characterised by an integer pair (x,y)

• This means we have a 2-d space

• Any 2-d space data structure can be used

• If you use a (pre-order,post-order) numbering you get triangular filling of 2-d

(to be discussed later)

Alternative mappings

• Mapping the structure to the relational world

– The Edge approach

– The Attribute approach

– The Universal Table approach

– The Normalized Universal approach

– The Monet/XML approach

– The Dataguide approach

• Mapping values

– Separate value tables

– Inlining

• Shredding

Dataguide approach

• Developed in the context of Lore, Lorel (Stanford Univ)

• Predecessor of the Monet/XML model

• Observation:

– queries in the graph-representation take a limited form

– they are partial walks from the root to an object of interest

– this behaviour was stressed by the query language Lorel, i.e. an SQL-based query language based on processing regular expressions

SELECT X

FROM (Bib.*.author).(lastname|firstname).Abiteboul X

DataGuides

Definition

given a semistructured data instance DB, a DataGuide for DB is a graph G s.t.:

- every path in DB also occurs in G

- every path in G occurs in DB

- every path in G is unique

Dataguides

Example:

DataGuides

• Multiple DataGuides for the same data:

DataGuides

DefinitionLet w, w’ be two words (I.e word queries) and G a graph

w G w’ if w(G) = w’(G)

Definition

G is a strong dataguide for a database DB if G is the same as DB

Example:- G1 is a strong dataguide- G2 is not strong

person.project !DB dept.project

person.project !G2 dept.project

DataGuides

• Constructing the strong DataGuide G:

Nodes(G)={{root}}

Edges(G)=while changes do

choose s in Nodes(G), a in Labels

add s’={y|x in s, (x -a->y) in Edges(DB)} to Nodes(G)

add (x -a->y) to Edges(G)

• Use hash table for Nodes(G)

• This is precisely the powerset automaton construction.

DataGuides

• How large are the dataguides ?

– if DB is a tree, then size(G) <= size(DB)• why? answer: every node is in exactly one extent of G

• here: dataguide = XSet

– How many nodes does the strong dataguide have for this DB ? 20 nodes (least common

multiple of 4 and 5)

Dataguides usually fail on data with cyclic schemas, like: