1
TDD: Topics in Distributed Databases
Distributed Databases
Distributed database
Distributed query processing: joins and non-join queries
Updating distributed data
2
Distributed databases
Data is stored in several sites (nodes), geographically or
administratively across multiple systems
Each site is running an independent DBMS What do we get?
– Increased availability and reliability– Increased parallelism
Complications– Catalog management: distributed data independence and
distributed transaction atomicity– Query processing and optimization: replication and
fragmentation– Increased update costs, concurrency control: locking,
deadlock, commit protocol, recovery
Data centers
Architectures
33
4
Homogeneous vs heterogeneous systems
Homogeneous: identical DBMS, aware of each other, cooperate
Heterogeneous: different schemas/DBMS– Multidatabase system: uniform logical view of the data --
common schema– difficult, yet common: system is typically gradually
developed
network
DB
DBMS
local schema
DB
DBMS
local schema
DB
DBMS
local schema
network
global schema
query answer
5
Architectures
Client-server: client (user interface, front end), server (DBMS)– Client ships query to a server (query shipping)– All query processing at server
client
server serverserver
query answer
client-server
client
6
Architectures
Collaborating server: query can span several servers
serverserver
serverquery answer
collaborating-server
Middleware:– Coordinator: queries and transactions across servers
7
Warehouse architecture
data warehouse
client applications
integrator
monitor/wrapper monitor/wrapper monitor/wrapper
RDB OODB XML
8
Monitor/wrapper
A monitor/wrapper for each data source translation: translate an information source into a common
integrating model change detection: detect changes to the underlying data source
and propagate the changes to the integrator
– active databases (triggers: condition, event, action)
– logged sources: inspecting logs
– periodic polling, periodic dumps/snapshots Data cleaning:
– detect erroneous/incomplete information to ensure validity
– back flushing: return cleaned data to the source
9
Integrator
Receive change notifications from the wrapper/monitors and reflect the changes in the data warehouse.
Typically a rule-based engine: merging information (data fusion) handling references Data cleaning:
– removing redundancies and inconsistencies
– inserting default values
– blocking sources
10
When to use data warehouse
Problem: potential inconsistencies with the sources.
Commonly used for relatively “static” data
when clients require specific, predicable portion of the available information
when clients require high query performance but not necessarily the most recent state of the information
when clients want summarized/aggregated information such as historical information
Examples:
scientific data
historical enterprise data
caching frequently requested information
11
Data warehouse vs. materialized views
materialized view is over an individual structured database, while a warehouse is over a collection of heterogeneous, distributed data sources
materialized view typically has the same form as in the underlying database, while a warehouse stores highly integrated and summarized data
materialized view modifications occur within the same transaction updating its underlying database, while a warehouse may have to deal with independent sources:
– sources simply report changes
– sources may not have locking capability
– integrator is loosely coupled with the sources
12
Mediated system architecture
Virtual approach: data is not stored in the middle tier
client applications
Mediator
wrapper wrapper wrapper
RDB OODB XML
13
Lazy vs. eager approaches
Lazy approach (mediated systems): accept a query, determine the appropriate set of data sources,
generate sub-queries for each data source obtain results from the data sources, perform translation,
filtering and composing, and return the final answer
Eager approach (warehouses): information from each source that may be of interest is
extracted in advance, translated, filtered, merged with relevant sources, and stored in a repository
query is evaluated directly against the repository, without accessing the original information sources
14
Data warehouse vs. mediated systems
Efficiency
– response time: at the warehouse, queries can be answered efficiently without accessing original data sources. Advantageous when data sources are slow, expensive or periodically unavailable, or when translation, filtering and merging require significant processing
– space: warehousing consumes extra storage space Extensibility: warehouse consistency with the sources: warehouse data may become out
of date applicability:
– warehouses: for high query performance and static data– mediated systems: for information that changes rapidly
15
Distributed data storage -- replication
Fragments of a relation are replicated at several sites: R is
fragmented into R1, R2, R3 Why?
– Increase availability/reliability: if one site fails– Increase parallelism: faster query evaluation – Increase overhead on updates: consistency
Dynamic issues: synchronous vs. asynchronous
R1 R2 R3 R2
Site 1Site 2
Primary copy: e.g.,
Bank: an account at the site in which it was
opened
Airline: an flight at the site from which it originates
16
Distributed data storage -- fragmentation
A relation R may be fragmented or partitioned Horizontal Vertical: lossless join
Question: how to reconstruct the original R?
network
DB
DBMS
local schema
DB
DBMS
local schema
DB
DBMS
local schema
network
global schema
query answerEDIgrace003
NYCmary002
NYCjoe001
citynameeid fragmentation: determined by local ownership
NYCEDI
17
Transparency, independence
Distributed data transparency (independence):
– location (name) transparency
– fragmentation
– replication transparency (catalog management)
Transaction atomicity: across several sites
– All changes persist if the transaction commits
– None persists if the transaction aborts
Data independency and transaction atomicity are not supported
currently: the users have to be aware of where data is located
Distributed query processing: joins and non-join queries
1818
19
Distributed query processing and optimization
New challenges– Data transmission costs (network)– parallelism– Choice of replicas: lowest transmission cost– Fragmentation: to reconstruct the original relation
Query decomposition: query rewriting/unfolding
depending on how data is fragmented/replicated
network
DB
DBMS
local schema
DB
DBMS
local schema
DB
DBMS
local schema
network
global schema
query answer
sub-query sub-querysub-query
decomposition
20
Non-join queries
Schema: account(acc-num, branch-name, balance) Query Q: select * from account where branch-
name = `EDI” Storage: database DB is horizontally fragmented, based on
branch-name: NYC, Philly, EDI, … denoted by DB1, …, DBn DB = DB1 … DBn Processing:
– Rewrite Q into Q(DB1) … Q(DBn)– Q(DBi) is empty if branch-name <> EDI
• Q(DB1), where DB1 is the EDI branch– Q(DB1) = Q’(DB1)
• Q’: select * from account
21
Simple join processing – data shipping
R1 R2 R3
where Ri is at site i, S0 is the site where the query is issued Option 1: send copies of R1, R2, R3 to S0 and compute the
joins at S0 Option 2:
– Send R1 to S2, compute temp1 R1 R2 at S2– Send temp1 to S3, compute temp2 R3 temp1 at S3– Send temp2 to S0
Decision on strategies: The volume of data shipped The cost of transmitting a block Relative speed of processing at each site
22
Semijoin – reduce communication costs
R1 R2, where Ri is at site i, Compute temp1 (R1 R2) R1 at site 1
projection on join attributes only; assume R1 smaller Ship temp1 to site 2, instead of the entire relation of R1 Compute temp2 R2 temp1 at S2 Ship temp2 to site 1 compute result R1 temp2 at S1
Effectiveness If sufficiently small fraction of the relation of R2 contributes to
the join Additional computation cost may be higher than the savings in
communication costs
23
Bloomjoin – reduce communication costs
R1 R2, where Ri is at site i,
Compute a bit vector of size k by hashing (R1 R2) R1
– bit: set to 1 if some tuple hashes to it– smaller than the projection (constant size)
Ship the vector to site 2, instead of the entire relation of R1 Hash R2 using the same hashing function Ship to site 1 only those tuples of R2 that also hash to 1, temp1 compute result R1 temp1 at S1
Effectiveness Less communication costs: bit-vector vs projection The size of the reduction by hashing may be larger than that of
projectionQuestion: set difference?
24
exploring parallelism
Consider R1 R2 R3 R4, where Ri is at site i
temp1 R1 R2, by shipping R1 to site 2
temp2 R3 R4, by shipping R3 to site 4
result temp1 temp2 -- pipelining
Question: R1 R2 R3, using
Pipelined parallelism
Semi-join
both
parallel
25
Distributed query optimization
The volume of data shipped The cost of transmitting a block Relative speed of processing at each site Site selection: replication
Two-step optimization At compile time, generate a query plan – along the same lines
as centralized DBMS Every time before the query is executed, transform the plan and
carry out site selection (determine where the operators are to
be executed) – dynamic, just site selection
26
Practice: validation of functional dependencies
A functional dependency (FD) defined on schema R: X Y– For any instance D of R, D satisfies the FD if for any pair of
tuples t and t’, if t[X] = t’[X], then t[Y] = t’[Y]– Violations of the FD in D:
{ t | there exists t’ in D, such that t[X] = t’[X], but t[Y] t’[Y] }
Now suppose that D is fragmented and distributed
Develop an algorithm that given fragmented and distributed D and an FD, computes all the violations of the FD in D– semijoin– bloomjoin
Questions: what can we do if we are given a set of FDs to validate?
horizontally or vertically
Minimize data shipment
27
Practice: relational operators
Consider a relation R that is vertically partitioned and distributed across n sites
Develop an algorithm to implement
– A R,
– C R
by using – semijoin– bloomjoin
Column-oriented DBMS: store tables as sections of columns of data, rather than rows (tuples); good for, eg, certain aggregate queries
28
Practice: relational operators
Consider relations R1 and R2 that are horizontally partitioned and distributed across n sites
Develop an algorithm to implement R1 R1.A = R2.B R2 by using
– semijoin– bloomjoin
Question: is your algorithm parallel scalable? That is, the more processors are used, the faster it is
Updating distributed data
2929
30
Updating Distributed data
Fragmentation: an update may go across several sites– Local transaction– Global transaction
Replication: multiple copies of the same data -- consistency
network
DB
DBMS
local schema
DB
DBMS
local schema
DB
DBMS
local schema
network
global schema
query answer
updates
propagate
31
System structure
Local transaction manager: either local transaction or part of a
global transaction– Maintain a log for recovery– Concurrency control for transactions at the site
Transaction coordinator (not in centralized DBMS)– Start the execution of a transaction– Break a transaction into a number of sub-transactions and
distribute them to the appropriate sites– Coordinate the termination of the transaction
• Commit at all sites• Abort at all sites
32
Two-Phase Commit protocol (2PC): Phase 1
Transaction T; the transaction coordinator is at C
C
P3
P2
P1
prepare T
log(1) <prepare T>
(2) <ready T> <ready T>
(2) <abort T> <no T>
log
log
commit T
abort T
if all responses
are <ready>
if one of the responses is
<abort>
33
Two-Phase Commit protocol (2PC): Phase 2
Transaction T; the transaction coordinator is at C
C
P3
P2
P1
prepare T
log(3) <commit T>
(4) <ack T>
<ready T>(4) <ack T>
log
log
commit T
complete T
<commit T>
<commit T>
<ready T>
Similarly for abort
34
Comments on 2PC
Two rounds of communication: both initiated at the coordinator– Voting– Terminating
Any site can decide to abort a transaction Every message reflects a decision by the sender; The decision is recorded in the log to ensure the decision
survives any failure: transaction id– The log at each participating site: the id of the coordinator – The log at the coordinator: ids of the participating sites
35
Concurrency control
Single local manager: at a chosen site– Simple implementation and deadlock handling– Bottleneck– Vulnerability: if the site fails
Distributed lock manager: each site has one to update data item D at site j– Send request to the lock manager at site j– Request is either granted or delayed– Deadlock handling is hard
Major complication: replicated data
36
replication
Synchronous replication: all copies must be updated before the
transaction commits – data distribution transparent, consistent– expensive
Asynchronous replication: copies are periodically updated – Allow modifying transaction to commit before all copies
have been changed – users are aware of data distribution, consistency issues– Cheaper: current products follow this approach– Peer-to-peer replication (master copies)– Primary site replication (only the primary is updateable)
37
Synchronous replication
Majority approach -- voting: data item D replicated at n sites– A lock request is sent to more than one-half of the sites– D is locked if the majority vote yes, write n/2 + 1 copies– Each copy maintains a version number– Expensive
• 2(n/2 + 1) messages for lock• Read: at least n/2 + 1 copies to make sure it is current• Deadlock is more complicated: if only one copy is
locked
38
Synchronous replication (cont.)
Majority approach -- voting
Biased protocol: read-any write-all. – Shared lock (read): simply requests a lock on D at one site
that contains a copy of D– Exclusive lock (write): lock on all sites that contain a copy– Less overhead on read, expensive on write– Commonly used approach to synchronous replication
39
Synchronous replication -- exercise
A distributed system uses the majority (voting) approach to update data replicas. Suppose that a data item D is replicated at 4 different sites: S1, S2, S3, S4. What should be done if a site S wants to
Write D Read D
40
Synchronous replication -- answer
A distributed system uses majority the (voting) approach to update data replicas. Suppose that a data item D is replicated at 4 different sites: S1, S2, S3, S4. What should be done if a site S wants to
Write D– The site S sends a lock request to any 3 of S1, S2, S3, S4– The write operation is conducted if the lock is granted by the
lock manager of all the 3 sites; otherwise it is delayed until the lock can be granted
Read D– The site S reads copies of D from at least 3 sites– It picks the copy with the highest version number
41
Asynchronous replication
Primary site: choose exactly one copy, residing at a primary site– A lock request is sent to the primary site– Replicas at other sites may not be updated; they are
secondary to the primary copy – Simple to implement – Main issues:
• D becomes inaccessible if the primary site fails• Propagation of changes from the primary site to others
42
Asynchronous replication (cont.)
Primary site
Peer-to-peer: more than one of the copies can be a master– Change to a master copy must be propagated to others– Conflicts of changes to two copies have to be resolved– Best used when conflicts do not arise: e.g.,
• Each master site owns a distinct fragment• Updating rights owned by one master at a time
43
Distributed deadlock detection
Recall wait-for graph– Nodes: transactions– Edges: T1 T2 if T1 requests a resource being held by T2
Local wait-for graph: – Nodes: all transactions local or holding/requesting data item
local to the site– T1 T2 if T1 (at site 1) requests a resource being held by
T2 (at site 2)
Global wait-for graph: union of local wait-for graphs– Deadlock if it contains a cycle
T1
T5
T2 T2
T3 T3
T4
Site 1 Site 2 global
T1
T5
T2
T3
T4
44
False cycles
Due to communication delay Site 1: local wait-for graph has T1 T2 T2 releases the resource – deletion of the edge Site 2: T2 requests the resource again – addition of T2 T1 Cycle if insert T2 T1 arrives before removal of T1 T2
Centralized deadlock detection – deadlock detection coordinator Constructs/maintains global wait-for graph Detect cycles If it finds a cycle, chose a victim to be rolled backDistributed deadlock manager? More expensive
T1 T2 T1 T2Site 1
T1 T2globalSite 2
45
Summary and review
Homogeneous vs heterogeneous systems Replication and fragmentation. Pros and cons of replication.
How to reconstruct a fragmented relation (vertical, horizontal)? Simple join (data shipping), semijoin, bloomjoin
set-difference? Intersection? Aggregation? Transaction manager and coordinator. Responsibilities? Describe 2PC. Recovery: coordinator and participating sites Replication:
– majority, read-any write-all, – primary site, peer-to-peer
Local wait-for graph, global wait-for graph, deadlock detection,
deadlock handling
46
Reading list
MapReduce tutorial:
http://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html
Take a look at the following:
– Cassandra, http://en.wikipedia.org/wiki/Apache_Cassandra
– Clusterpoint, http://en.wikipedia.org/wiki/Clusterpoint
– Riak, http://en.wikipedia.org/wiki/Riak
Reading for the next week
47
1. Pregel: a system for large-scale graph processing
http://kowshik.github.io/JPregel/pregel_paper.pdf
2. Distributed GraphLab: A Framework for Machine Learning in the Cloud, http://vldb.org/pvldb/vol5/p716_yuchenglow_vldb2012.pdf
3. PowerGraph: Distributed Graph-Parallel Computation on Natural Graphs, http://select.cs.cmu.edu/publications/paperdir/osdi2012-gonzalez-low-gu-bickson-guestrin.pdf
4. GraphChi: Large-Scale Graph Computation on Just a PC, http://select.cs.cmu.edu/publications/paperdir/osdi2012-kyrola-blelloch-guestrin.pdf
5. Performance Guarantees for Distributed Reachability Queries, http://vldb.org/pvldb/vol5/p1304_wenfeifan_vldb2012.pdf
6. W. Fan, X. Wang, and Y. Wu. Distributed Graph Simulation: Impossibility and Possibility. VLDB 2014. (parallel model)
http://homepages.inf.ed.ac.uk/wenfei/papers/vldb14-impossibility.pdfPick two papers and write reviews