Containment of Conjunctive Queries on Annotated Relations TJ Green University of Pennsylvania

Containment of Conjunctive Queries on Annotated Relations

TJ Green University of Pennsylvania

Symposium on Database ProvenanceUniversity of Edinburgh

May 21, 2008

Provenance and Query Optimization

• Many kinds of semiring-based provenance annotations to choose from:– lineage– why-provenance– minimal witness why-provenance– provenance polynomials– ...

• These seem to keep track of more/less information

• A fundamental question: how does this affect query optimization?

2

Conjunctive Queries on K-Relations• Datalog-style syntax for conjunctive queries (CQs):

Q(x,y) :- R(x,z), R(z,y)• Semantics of applying the CQ to a K-relation R : D£D K:

Q(a,b) = z2D R(a,z)¢R(z,b)• # of repetitions of an atom in the body matters

• For unions of conjunctive quereis (UCQs) (equivalent to positive RA), sum over CQs:P(x,y) :- R(x,z), R(z,y) P(x,y) :- R(x,w), R(y,w)

• Semantics of UCQ applied to R ― a sum over CQs:P(a,b) = z2D R(a,z)¢R(z,b) + w2D R(a,w)¢R(b,w)

3

Choice of K Affects Query Optimization

K = N (bag semantics) differs from K = B (set semantics)e.g., the conjunctive queries Q1(x) :- R(x,y), R(x,z) Q2(u) :- R(u,v)

are set-equivalent, but not bag-equivalent

4

Conjunctive Queries (CQs)

Unions of Conjunctive Queries (UCQs)

Bag Semantics Containment (vN)

? (¦2p-hard)

[Chaudhuri&Vardi 93]undecidable [Ioannidis&Ramakrishnan 95]

Bag Semantics Equivalence (´N)

isomorphism () [CV 93]

?

Our Contributions

• We make a systematic study of query containment and query equivalence for various provenance models

• We show that K-containment and K-equivalence of CQs and UCQs are decidable for lineage, why-provenace, and the provenance polynomials N[X], as well as a new model, B[X]

• The decision procedures are based on interesting variations of containment mappings

• We analyze the complexity in each case

5

Our Contributions

• As a corollary of the decidability result for N[X]-equivalence of UCQs, we also fill in a gap in the chart for bag semantics:

6

Conjunctive Queries (CQs)

Unions of Conjunctive Queries (UCQs)

Bag Semantics Containment (vN)

? (¦2p-hard)

[Chaudhuri&Vardi 93]undecidable [Ioannidis&Ramakrishnan 95]

Bag Semantics Equivalence (´N)

isomorphism () [CV 93]

isomorphism ()

K-Containment for Queries

• For semiring K, define a ·K b , 9c . a + c = b. If ·K is a partial order, it is called the natural order, and K is said to be naturally-ordered

• B, N, lineage, why-provenance, B[X], and N[X] are all naturally-ordered

• We define K-containment using the natural order:

Q1 vK Q2 , 8I 8t Q1(I)(t) ·K Q2(I)(t)

Q1 ´K Q2 , 8I 8t Q1(I)(t) = Q2(I)(t) 7

A Hierarchy of Semiring Provenance (1)

• Provenance polynomials (N[X], +, ¢, 0, 1) – tracks calculations abstractly; most general

e.g., 2p2r + 3ps + ps3

• Drop coefficients to get (B[X], +, ¢, 0, 1)p2r + ps + ps3

• Drop exponents to get why-prov. (P(P(X)), [, d, ;, {;}){{p,r}, {p,s}}

• Flatten set-of-sets to get lineage (P(X), +, ¢, ?, ;){p,r,s}

• Drop, flatten, etc. correspond to surjective semiring homomorphisms

8

A Hierarchy of Semiring Provenance (2)

• Suppose h : K1 K2 is a semiring homomorphism. Then a ·K1

b implies h(a) ·K2 h(b). If h is also surjective,

then h(a) ·K2 h(b) implies a ·K1

b.

• Definition: K1 ¹ K2 means P vK2 Q implies P vK1

Q

• Proposition: for any positive KB ¹ K ¹ N[X](All those we consider are positive.) Moreover:

• Proposition (Provenance Hierarchy):B ¹ lineage ¹ Why-Prov. ¹ B[X] ¹ N[X]

9

Containment Mappings• A containment mapping from CQ Q to CQ P is a

function h : Vars(Q) Vars(P) such that– head of Q is mapped to head of P– every atom in body of Q is mapped to an atom in body

of P• Theorem [CM77]: For CQs P,Q we have P vB Q iff

there is a containment mapping from Q to P– e.g. Q1(x) :- R(x,y), R(x,z) Q2(u) :- R(u,v)– h which sends u x and v y is a containment

mapping• Checking for existence of containment mapping is

NP-complete10

Canonical Databases

• Take body of CQ, “freeze” into database instance [CM77], and tag each tuple with a “tuple id”

• We’ll denote by canK(Q) the canonical database for Q with abstract tags from K

• e.g., Q(w) :- R(u,v), R(v,w)

u v x1

v w x2

canN[X](Q) = canB[X](Q) = R

u v {x1}

v w {x2}canlin(Q) = R

u v {{x1}}

v w {{x2}}canwhy(Q) = R

11

Lineage-Containment of CQs• Covering set of containment mappings: for every

atom A in the body of P there is a containment mapping h : Q P with A in the image of h

• Theorem: For CQs P, Q the following are equivalent:1. P vlin Q2. P(canlin(P)) µlin Q(canlin(P))3. there is a covering set of containment mappings from Q

to P• Note: covering sets of containment mappings were

identified in [CV 93] as a necessary (but not sufficient) condition for bag-containment of CQs

12

Why-Containment of CQs• A containment mapping is onto if it induces a

surjection on atoms• Theorem: For CQs P, Q the following are

equivalent:1. P vwhy Q2. P(canwhy(P)) µwhy Q(canwhy (P))3. there is an onto containment mapping h : Q P

• Note: onto containment mappings were identified in [CV 93] as a sufficient (but not necessary) condition for bag-containment of CQs

13

B[X], N[X]-containment of CQs• A containment mapping is exact if it induces a

bijection on atoms• Theorem: For CQs P, Q and for K 2 {B[X], N[X]}

the following are equivalent1. P vK Q2. P(canK (P)) µK Q(canK (P))3. there is an exact containment mapping h : Q P

• Another way to think of exact containment mappings: by unifying variables in Q, you get a query isomorphic to P

14

So Far

• K-containment of CQs is decidable for all the provenance models in the hierarchy

• Next, we indicate which steps in the hierarchy are strict, and which collapse:

B Á lineage Á Why-Prov. Á B[X] ¼ N[X]

15

Separating the Models for v of CQs

• B Á lineage:Q1(x,y) :- R(x,y), R(x,z) Q2(x,y) :- R(x,y)Q1 vB Q2 but Q1 vlin Q2

• lineage Á why:Q1(x) :- R(x,y), R(x,z) Q2(x) :- R(x,y)Q1 vlin Q2 but Q1 vwhy Q2

• why Á B[X]:Q1(x,y) :- R(x,y) Q2(x,y) :- R(x,y), R(x,z)Q1 vwhy Q2 but Q1 vB[X] Q2

16

From Containment to Equivalence

• {Onto|exact} containment mappings in both directions implies CQs are isomorphic, so why-provenance, B[X], and N[X] collapse to:

P ´why Q , P ´B[X] Q , P ´N[X] Q , P Q

• In contrast, for lineage, having sets of covering containment mappings in both directions does not imply isomorphism (but still decidable)

17

From CQs to UCQs• For idempotent semirings (where + is

idempotent) this is easy. B, PosBool(B), lineage, why-provenance, and B[X] are idempotent; N[X] is not (omitted)

• Proposition [after SY80]: If K is idempotent, then for UCQs P, Q we have P vK Q iff for every CQ P in P there is a CQ Q in Q such that P vK Q

• Corollary: For idempotent K, the problems of checking K-equivalence of CQs and K-equivalence of UCQs are polynomially equivalent

18

N[X]- and Bag-Equivalence of UCQs

• As with CQs, N[X]-equivalence of UCQs turns out to be the same as isomorphism:Theorem: For UCQs P, Q, P ´N[X] Q iff P Q

• But, it turns out that N[X]-equivalence and N-equivalence of UCQs are intimately related:Theorem: for UCQs P, Q, P ´N[X] Q iff P ´N Q

Thus:Corollary: for UCQs P, Q P ´N Q iff P Q

19

• Theorem: checking for {covering set of|onto|exact} containment mappings is NP-complete

• Checking for query isomorphism: believed >P, <NP

Summary: Complexity Results

20

B PosBool(B) N Lineage Why-Pr. B[X] N[X]

CQs vK NP [CM 77]

NP[PODS 07]

? (¦2p-hard)

[CV 93]NP-ct NP-ct NP-ct NP-ct

´K NP ibid.

NPibid.

ibid.

NP-ct

UCQs vK NP [SY 80]

NPibid.

undec [IR 95]

NP-ct NP-ct NP-ct PSPACE

´K NP ibid.

NPibid.

NP-ct NP-ct NP-ct

Summary: Provenance Hierarchy

21

B PosB.(B) Lineage N Why-Pr. B[X] N[X]

CQs vK ¼ Á Á Á Á ¼

´K ¼ Á Á ¼ ¼ ¼

B PosB.(B) Lineage Why-Pr. B[X] N[X]

UCQs vK ¼ Á Á Á Á

´K ¼ Á Á Á Á

Related Work• Already mentioned– Set-cont. and equiv. of CQs [Chandra&Merlin 77]– Set-cont. and equiv. of UCQs [Sagiv&Yannakakis 80]– Bag-cont. of UCQs [Ioannidis&Ramakrishnan 95]– Bag-equiv. of CQs [Chaudhuri&Vardi 93]

• Containment of CQs with where-provenance [Tan 03]• Bag-set semantics [CV 93], combined semantics [Cohen 06]– For K-relations: support operator of [Geerts&Poggi 08]

generalizes duplicate elimination• Bag-containment of CQs [Jayram+ 06]

22

Future Work• Loose ends:– Lower bound for N[X]-containment of UCQs (we gave only

a PSPACE upper bound)– Generalize results for specific semirings to semirings with

certain properties?• Beyond UCQs: Datalog– is K-containment of Datalog programs the same as set-

containment when K is a distributive lattice?– is bag-equivalence/N[X]-equivalence undecidable for

Datalog?• Could semiring framework give any insight into bag-

containment of CQs?• Query optimization for annotated XML

23

24