2010 VLDB - TRAMP: Understanding the Behavior of Schema Mappings through Provenance

Introduction Approach Transformation Provenance Implementation Results Conclusions

TRAMP: Understanding the Behaviour of SchemaMappings through Provenance

Boris Glavic1 Gustavo Alonso2

Renee J. Miller1 Laura M. Haas3

1University of Toronto

2ETH Zurich

3IBM Almaden Research Center

VLDB 2010, September 16, 2010


Outline

1 Introduction

2 Approach

3 Transformation Provenance

4 Implementation

5 Results

6 Conclusions

TRAMP


What is TRAMP?

TRAMP: Transformation Mapping Provenance

Novel “holistic” approach to help users to understand schemamappings

Data ExchangeData Integration

Provides query language access for

Mapping scenariosProvenanceData

Slide 1 of 27 Boris Glavic TRAMP


Data Exchange

Problem Statement

Given a source and a target schema

How to map data from the source to the target?

General Approach

1 Find correspondences between schema elements

2 Generate schema mappings from correspondences andschema constraints

3 Generate implementing transformations

4 Execute transformations



Data Exchange Process: 1) Find Correspondences

Correspondence

Attribute X represents the same information as attribute Y

Generated by automatic matcher or user

Example

Employee Name Address

Address Id City Street

Person Name LivesAt Gender

Source Target



Data Exchange Process: 1) Find Correspondences

Correspondence

Attribute X represents the same information as attribute Y

Generated by automatic matcher or user

Example

Employee Name Address

Address Id City Street

Person Name LivesAt Gender

Source Target



Data Exchange Process: 2) Generate Schema Mappings

Schema Mapping

Declarative Constraints that model relationships betweenschemas (s-t tgd or source-to-target tuple-generatingdependencies)

Generated from correspondences and schema constraints

Example

For all employees with associated addresses exists a person withthe Name of the employee and the City of the address stored in theLivesAt attribute.




Schema Mapping



Example

M1 : ∀a, b, c , d : Employee(a, b) ∧ Address(b, c , d) ⇒ ∃f :person(a, c , f )




Schema Mapping



Example

M1 : ∀a, b, c , d : Employee(a, b) ∧ Address(b, c , d) ⇒ ∃f :person(a, c , f )M2 : ∀a, b : Employee(a, b) ⇒ ∃c , d : person(a, c , d)



Data Exchange Process: 3) Generate ImplementingTransformations

Implementing Transformation

Schema mappings do not specify full target instance

Need executable transformation

Generated from schema mappings (XQuery, SQL, . . . )

Many-to-Many (Mappings-Transformations)

Example

SELECT Name , city AS LivesAt , NULL AS Gender

FROM Employee e JOIN Address a ON (e.Address = a.Id)

UNION

SELECT Name , NULL AS LivesAt , NULL AS Gender

FROM Employee e;



Data Exchange Process: 4) Execute

SourceEmployee

Name AddressGerd 2

Nandy NULL

AddressId City Street1 Prag Krutz2 Aachen Pond

→

SELECT Name,city AS LivesAt,NULL AS GenderFROMEmployee eJOIN Address aON (e.Address = a.Id)UNIONSELECT Name,NULL AS LivesAt,NULL AS GenderFROM Employee e;

→Target

PersonName LivesAt GenderGerd Aachen NULLGerd NULL NULL

Nandy NULL NULL



Understanding and Debugging Schema Mappings

Complex error-prone process

Many sources of error:

Faulty source dataIncorrect correspondencesIncorrect schema mappingsIncorrect transformations

Hard to trace error source

How to help the user?

Provide information that aids in debugging

Allow for combination and filtering ⇒ Query language



Understanding and Debugging Schema Mappings

Complex error-prone process

Many sources of error:

Faulty source dataIncorrect correspondencesIncorrect schema mappingsIncorrect transformations

Hard to trace error source

How to help the user?

Provide information that aids in debugging

Allow for combination and filtering ⇒ Query language



Information of Interest

1 Where is data derived from?

2 Which mapping created which data?

3 Which part of a transformation created which data?

4 Mapping Scenario (Interrelationships)
























Where is Data Derived from?

ExamplePerson

Name LivesAt GenderGerd Aachen NULLGerd NULL NULL

Nandy NULL NULL

↑SELECT Name , City AS LivesAt , NULL AS Gender

FROM Employee e JOIN Address a ON (e.Address = a.Id)

UNION

SELECT Name , NULL AS LivesAt , NULL AS Gender

FROM Employee e;

↑Employee

Name AddressGerd 2

Nandy NULL

↑Address

Id City Street1 Prag Krutz2 Aachen Pond

↑Employee

Name AddressGerd 2

Nandy NULL



Data Provenance

Description

Which input tuples contributed to which output tuples?

ExamplePerson


Nandy NULL NULL

↑Employee

Name AddressGerd 2

Nandy NULL

↑Address


↑Employee

Name AddressGerd 2

Nandy NULL



Which Mapping Created which Data?

ExamplePerson


Nandy NULL NULL

↑

Mapping M1



Which Mapping Created which Data?

ExamplePerson


Nandy NULL NULL

↑

Mapping M2



Mapping Provenance

Description

Which mappings generated a target tuple?

ExamplePerson


Nandy NULL NULL

↑

Mapping M2



Which Transformation Part Created which Data?

Employee Address

1

1

0

1Employee

1

01

ExamplePerson


Nandy NULL NULL

↑

SELECT Name , city AS LivesAt ,

NULL AS Gender

FROM Employee e

JOIN Address a

ON (e.Address = a.Id)

UNION

SELECT Name , NULL AS LivesAt ,

NULL AS Gender

FROM Employee e;



Which Transformation Part Created which Data?

Employee Address

0

0

1

0Employee

1

10

ExamplePerson


Nandy NULL NULL

↑


NULL AS Gender

FROM Employee e

JOIN Address a


UNION


NULL AS Gender

FROM Employee e;



Transformation Provenance

Description

Which parts of atransformationcontributed to anoutput tuple?

ExamplePerson


Nandy NULL NULL

↑


NULL AS Gender

FROM Employee e

JOIN Address a


UNION


NULL AS Gender

FROM Employee e;



Outline

1 Introduction

2 Approach


4 Implementation

5 Results

6 Conclusions



Overview

Glue everything together

Store all mapping scenario elements and theirinter-relationships in a database

Relational: instance data, schemas, transformations (views)XML: Mappings, Correspondences

Provenance SQL extension

On-demand provenance computationRelational/XML representationPROVENANCE: Data provenance (relational)TRANSXML: Transformation provenance (XML)

One fits all query language: SQL

SQL: data and data-provenanceXSLT: queries and transformation provenance



Related Work

Understanding Schema Mappings

Approaches using provenance

SPIDER [Chiticariu, Tan, VLDB’06]MXQL [Velegrakis, Miller, Mylopoulos, ICDE’05]Orchestra [Karvounarakis, Ives, Tannen, SIGMOD’10]

Example based approaches

Clio Data Viewer [Yan, Miller, Haas, Fagin SIGMOD ’01]MUSE [Alexe, Chiticariu, Miller, Tan, ICDE ’08]



Related Work

Understanding Schema Mappings

Approaches using provenance

SPIDER [Chiticariu, Tan, VLDB’06]MXQL [Velegrakis, Miller, Mylopoulos, ICDE’05]Orchestra [Karvounarakis, Ives, Tannen, SIGMOD’10]

Supports\System SPIDER MXQL OrchestraData Provenance X * X

Mapping Provenance X X X

Transformation Provenance - - -

Querying Provenance - X X

Querying Mappings - - -



Contributions

Provenance

Data Provenance: Perm (ASPJ-Set queries with nestedsubqueries)

Mapping Provenance: In combination with transformations(ASPJ-Set)

Transformation Provenance: NEW (ASPJ-Set)

Querying

Single query language: Data + Provenance + MappingScenarios



System > Sum(components)

Advanced Examples

Which tuples where derived through mapping M1 or M2,without accessing source relation R, and are derived fromtuple x in Relation Y?

Are there tuples in the target relation R that have beenderived from the same input tuple?

Error Classification

Classification of error types in the paper

Foreach error type: example how to use TRAMP to debug



Outline

1 Introduction

2 Approach


4 Implementation

5 Results

6 Conclusions



Modelling Transformation Provenance

Representation

Transformation provenance of tuple t from query q:

(Set of) annotated algebra trees for q

Each node carries boolean annotation

1-Annotation: operator contributed0-Annotation: operator did not contribute

Example

Employee Address

1

1

0

1Employee

1

01Employee Address

0

0

1

0Employee

1

10



Modelling Transformation Provenance cont.

Definition

For tuple t from query q

Node for op carries 1-annotation iff:

Evaluating the subtree under op over the data provenance of tdoes not return the empty set

Intuition

Data provenance is necessary information to produce tNone of this information “reaches” the output of the subtreeunder op ⇒ This part of the query did not contribute

Relation to data provenance

AbstractionDifferent RepresentationData provenance of all query steps




Definition




Intuition







Definition




Intuition






Transformation Provenance Example

Employee Address

1

1

11

ExamplePerson

Name LivesAt GenderGerd Aachen NULL

Nandy NULL NULL

↑SELECT Name , city AS LivesAt ,

NULL AS Gender

FROM Employee e

LEFT JOIN Address a

ON (e.Address = a.Id);

↑Employee

Name AddressGerd 2

Nandy NULL

↑Address




Transformation Provenance Example

Employee Address

1

1

01

ExamplePerson

Name LivesAt GenderGerd Aachen NULL

Nandy NULL NULL

↑SELECT Name , city AS LivesAt ,

NULL AS Gender

FROM Employee e

LEFT JOIN Address a


↑Employee

Name AddressGerd 2

Nandy NULL

↑Address




Relational Representation

Data + Provenance Relation

Query result schema + transformation provenance attribute

Duplicate result tuple t

Add one annotated tree to each duplicate

XML representation of annotated tree

Example

<Query >

<Select >

<Attr name="Name"><Var>e.Name</Var></Attr>

...

<From><LeftJoin >

...

<NOT><Relation alias="a">Address </Relation ></NOT>

...



Outline

1 Introduction

2 Approach


4 Implementation

5 Results

6 Conclusions



System Overview

Based On Perm

Modified PostgreSQL server

Provenance language constructs implemented as queryrewrites

“Use SQL to compute the provenance of SQL”

Optimizer

Query Plan

ExecutionEngine

Executor

Query Resultsa prov_a prov_b

123 'hello' 2.45

445 'test' 1.333

TRAMP Module

Rewritten Query Tree

TRAMPModule

Postgres Parser

Query Tree

Postgres Analyser

Parser & Analyzer

SELECT PROVENANCE *

FROM ...

JDBC

User



Internal Bitset Representation

Annotated Algebra Trees only vary in annotation

⇒ Factor out the tree

⇒ Annotations = set of nodes with 1-annotation

⇒ Use a bit-vector

Compact representationSet union = bitwise-or



Transformation Provenance Computation

Query Rewrite

Rewrite query q into qT

qT propagates bitsets throughout the query (partial annotatedtrees)

Result construction function fXML: builds XML

No data provenance needed!

Algebraic rewrite rules (correctness proven)

Optimizations

Depending on operators some annotations are static

static = are independent of t

Static sets are generated beforehand



Rewrite Algorithm

1 Analyze query to

Enumerate the operators and attach singleton bitsetsDetermine static bit-sets

2 Apply rewrite rules

Recursively to each operator in the query

3 Add application of fXML



Rewrite Algorithm

1 Analyze query to







Rewrite Algorithm

1 Analyze query to







Algorithm Step 1

SELECT

Name ,

City AS LivesAt ,

NULL AS Gender

FROM

Employee e

LEFT JOIN

Address a


Employee Address

1000

0100

0010 0001

For this query

Projection + Left Join + Employee is fixed⇒ use one bit-set: 1110



Algorithm Step 1

SELECT

Name ,

City AS LivesAt ,

NULL AS Gender

FROM

Employee e

LEFT JOIN

Address a


Employee Address

1000

0100

0010 0001

For this query

Projection + Left Join + Employee is fixed⇒ use one bit-set: 1110



Algorithm Step 2

SELECT Name , City AS LivesAt , NULL AS Gender

bitor(

1110,

CASE

WHEN a.Id IS NULL THEN 0000

ELSE 0001

END) AS trans prov

FROM Employee e

LEFT JOIN Address a




Algorithm Step 2


bitor(

1110,

CASE


ELSE 0001

END) AS trans prov

FROM Employee e

LEFT JOIN Address a




Algorithm Step 2


bitor(

1110,

CASE


ELSE 0001

END) AS trans prov

FROM Employee e

LEFT JOIN Address a




Rewrite Result

Name LivesAt Gender trans provGerd Aachen NULL 1111

Nandy NULL NULL 1110



Algorithm Step 3


fXML(

bitor(

1110,

CASE


ELSE 0001

END)) AS trans_prov

FROM Employee e

LEFT JOIN Address a




Rewrite Result

Name LivesAt Gender trans provGerd Aachen NULL <Query><Select>...<From>...

Nandy NULL NULL <Query><Select>...<From>...

Gerd

<Query >

<Select >


...

<From><LeftJoin >

...

<Relation alias="a">Address </Relation >

...



Rewrite Result

Name LivesAt Gender trans provGerd Aachen NULL <Query><Select>...<From>...

Nandy NULL NULL <Query><Select>...<From>...

Nandy

<Query >

<Select >


...

<From><LeftJoin >

...

<NOT><Relation alias="a">Address </Relation ></NOT>

...



Outline

1 Introduction

2 Approach


4 Implementation

5 Results

6 Conclusions



Experimental Results

Based on Amalgam benchmark (publication data)

Execution times for implementing transformations with andwithout transformation provenance

1

1.2

1.4

1.6

1.8

2

2.2

2.4

2.6

2.8

1 5 10 50 100 500 1000

1

1.2

1.4

1.6

1.8

2

2.2

2.4

2.6

2.8

Rela

tive O

verh

ead

Instance Size (#publs in thousands)

W/O provenanceOnly Bitset

Transformation Prov



Outline

1 Introduction

2 Approach


4 Implementation

5 Results

6 Conclusions



Conclusion

TRAMP = holistic approach for understanding mappings

Perm approach for data provenance

Transformation provenance

Querying of

DataProvenanceMapping scenario information

Future Work

Integrate with data exchange/integration systems

Combine with example-based approaches like MUSE

Provide hints on how to change mappings to generate anexpected result [Tran, Chan SIGMOD ’10]


Questions

Questions

Perm/TRAMP Open Source Release

On Source Forge

http://permdbms.sourceforge.net/

TRAMP: Understanding the Behaviour of Schema Mappings through Provenance

Questions

Storing Mapping Scenarios

Storing Mapping Elements

<Correspondence >

<from>

<Table name="Employee">

<Column >name</Column >

</Table >

</from>

<to>

<Table name="Person">

<Column >name</Column >

</Table >

</to>

</Correspondence >


Questions

Storing Mapping Scenarios

Mapping-Transformation Interrelationships

Annotate parts of implementing transformations withcorresponding mappings.

SELECT ANNOT (’M1’) Name , city AS LivesAt ,

NULL AS Gender

FROM Employee e

JOIN Address a


UNION

SELECT ANNOT (’M2’) Name , NULL AS LivesAt ,

NULL AS Gender

FROM Employee e;


2010 VLDB - TRAMP: Understanding the Behavior of Schema Mappings through Provenance

Science

transformation schema

target schema

behaviour of schema

schema mappings xquery

boris glavic tramp

schema elements

source employee

schema constraints example