Sensor Data Managementand XML Data Management
Zachary G. IvesUniversity of Pennsylvania
CIS 650 – Implementing Data Management Systems
November 19, 2008
Administrivia
By next Tuesday, please email me with a status report on your project … We are well under a month from the
deadline!
For next time: Please read & review the TurboXPath paper
2
3
Sensor Networks: Target Platform
Most sensor network research argues for the Berkeley mote as a target platform: Mote: 4MHz, 8-bit CPU 128B RAM (original) 512B Flash memory (original) 40kbps radio, 100 ft range Sensors:
Light, temperature, microphone Accelerometer Magnetometer http://robotics.eecs.berkeley.edu/~pister/SmartDust/
4
Sensor Net Data Acquisition
• First: build routing tree
• Second: begin sensing and aggregation
5
Sensor Net Data Acquisition (Sum)
5 5 5 5
5
55
8
5
5
5
5
55
5
5 5
7
• First: build routing tree
• Second: begin sensing and aggregation (e.g., sum)
6
Sensor Net Data Acquisition (Sum)
5 5 5 5
5
55
8
5
5
5
5
55
5
5 5
10 15205
25
6055
20105
138 18
5
3023357
5
85
• First: build routing tree
• Second: begin sensing and aggregation (e.g., sum)
7
Sensor Network Research
Routing: need to aggregate and consolidate data in a power-efficient way Ad hoc routing – generate routing tree to base
station Generally need to merge computation with
routing Robustness: need to combine info from
many sensors to account for individual errors What aggregation functions make sense?
Languages: how do we express what we want to do with sensor networks? Many proposals here
8
A First Try: Tiny OS and nesC
TinyOS: a custom OS for sensor nets, written in nesC Assumes low-power CPU
Very limited concurrency support: events (signaled asynchronously) and tasks (cooperatively scheduled)
Applications built from “components” Basically, small objects without any local state
Various features in libraries that may or may not be included
interface Timer { command result_t start(char type,
uint32_t interval); command result_t stop(); event result_t fired();}
9
Drawbacks of this Approach
Need to write very low-level code for sensor net behavior
Only simple routing policies are built into TinyOS – some of the routing algorithms may have to be implemented by hand
Has required many follow-up papers to fill in some of the missing pieces, e.g., Hood (object tracking and state sharing), …
10
An Alternative
“Much” of the computation being done in sensor nets looks like what we were discussing with STREAM
Today’s sensor networks look a lot like databases, pre-Codd Custom “access paths” to get to data One-off custom-code
So why not look at mapping sensor network computation to SQL? Not very many joins here, but significant aggregation Now the challenge is in picking a distribution and routing
strategy that provides appropriate guarantees and minimizes power usage
11
TinyDB and TinySQL
Treat the entire sensor network as a universal relation Each type of sensor data is a column in a
global table
Tuples are created according to a sample interval (separated by epochs) (Implications of this model?)
SELECT nodeid, light, tempFROM sensorsSAMPLE INTERVAL 1s FOR 10s
12
Storage Points and Windows
Like Aurora, STREAM, can materialize portions of the data: CREATE STORAGE POINT recentlight SIZE 8
AS (SELECT nodeid, light FROM sensors SAMPLE INTERVAL 10s)
and we can use windowed aggregates: SELECT WINAVG(volume, 30s, 5s)
FROM sensorsSAMPLE INTERVAL 1s
13
Events
ON EVENT bird-detect(loc): SELECT AVG(light), AVG(temp), event.loc FROM sensors AS s WHERE dist(s.loc, event.loc) < 10m SAMPLE INTERVAL 2s FOR 30s
14
Power and TinyDB
Cost-based optimizer tries to find a query plan to yield lowest overall power consumption Different sensors have different power usage
Try to order sampling according to selectivity (sounds familiar?)
Assumption of uniform distribution of values over range Batching of queries (multi-query optimization)
Convert a series of events into a stream join with a table
Also need to consider where the query is processed…
15
Dissemination of Queries Based on semantic routing
tree idea SRT build request is flooded
first Node n gets to choose its
parent p, based on radio range from root
Parent knows its children Maintains an interval on
values for each child Forwards requests to
children as appropriate Maintenance:
If interval changes, child notifies its parent
If a node disappears, parent learns of this when it fails to get a response to a query
16
Query Processing
Mostly consists of sleeping! Wake briefly, sample, and compute operators,
then route onwards
Nodes are time synchronized Awake time is proportional to the
neighborhood size (why?)
Computation is based on partial state records Basically, each operation is a partial aggregate
value, plus the reading from the sensor
17
Load Shedding & Approximation
What if the router queue is overflowing? Need to prioritize tuples, drop the ones we don’t want FIFO vs. averaging the head of the queue vs. delta-
proportional weighting
Later work considers the question of using approximation for more power efficiency If sensors in one region change less frequently, can
sample less frequently (or fewer times) in that region If sensors change less frequently, can sample readings
that take less power but are correlated (e.g., battery voltage vs. temperature)
18
The Future of Sensor Nets?
TinySQL is a nice way of formulating the problem of query processing with motes View the sensor net as a universal relation Can define views to abstract some concepts, e.g., an
object being monitored
But: What about when we have multiple instances of an
object to be tracked? Correlations between objects? (Joins)
What if we have more complex data? More CPU power?
What if we want to reason about accuracy?
19
XML: A Format of Many Uses
XML has become the standard for data interchange, and for many document representations
Sometimes we’d like to store it… Collections of text documents, e.g., the Web, doc DBs
… How would we want to query those? IR/text queries, path queries, XQueries?
Interchanging data SOAP messages, RSS, XML streams Perhaps subsets of data from RDBMSs
Storing native, database-like XML data Caching Logging of XML messages
20
XML: Hierarchical Data and Its Challenges It’s not normalized…
It conceptually centers around some origin, meaning that navigation becomes central to querying and visualizing
Contrast with E-R diagrams How to store the hierarchy? Complex navigation may include going up, sideways in tree Updates, locking Optimization
Also, it’s ordered May restrict order of evaluation (or at least presentation) Makes updates more complex
Many of these issues aren’t unique to XML Semistructured databases, esp. with ordered collections,
were similar But our efforts in that area basically failed…
21
Two Ways of Thinking of XML Processing
XML databases (today) Hierarchical storage + locking (Natix, TIMBER,
BerkeleyDB, Tamino, …) Query optimization
“Streaming XML” (next time) RDBMS XML export Partitioning of computation between source and
mediator “Streaming XPath” engines
The difference is in storage (or lack thereof)
22
XML in a Database
Use a legacy RDBMS Shredding [Shanmugasundaram+99] and many others Path-based encodings [Cooper+01] Region-based encodings [Bruno+02][Chen+04] Order preservation in updates [Tatarinov+02], … What’s novel here? How does this relate to materialized
views and warehousing?
Native XML databases Hierarchical storage (Natix, TIMBER, BerkeleyDB,
Tamino, …) Updates and locking Query optimization (e.g., that on Galax)
23
Query Processing for XML
Why is optimization harder? Hierarchy means many more joins (conceptually)
“traverse”, “tree-match”, “x-scan”, “unnest”, “path”, … op Though typically parent-child relationships Often don’t have good measure of “fan-out” More ways of optimizing this
Order preservation limits processing in many ways Nested content ~ left outer join
Except that we need to cluster a collection with the parent Relationship with NF2 approach
Tags (don’t really add much complexity except in trying to encode efficiently)
Complex functions and recursion Few real DB systems implement these fully
Why is storage harder? That’s the focus of Natix, really
24
The Natix System
In contrast to many pieces of work on XML, focuses on the bottom layers, equivalent to System R’s RSS
Physical layout Indexing Locking/concurrency control Logging/recovery
25
Physical Layout What are our options in storing XML trees?
At some level, it’s all smoke-and-mirrors Need to map to “flat” byte sequences on disk
But several options: Shred completely, as in many RDBMS mappings
Each path may get its own contiguous set of pages e.g., vectorized XML [Buneman et al.]
An element may get its 1:1 children e.g., shared inlining [Shanmugasundaram+] and [Chen+]
All content may be in one table e.g., [Florescu/Kossmann] and most interval encoded XML
We may embed a few items on the same page and “overflow” the rest
How collections are often stored in ORDBMS We may try to cluster XML trees on the same page, as
“interpreted BLOBs” This is Natix’s approach (and also IBM’s DB2)
Pros and cons of these approaches?
26
Challenges of the Page-per-Tree Approach
How big of a tree? What happens if the XML overflows the tree?
Natix claims an adaptive approach to choosing the tree’s granularity Primarily based on balancing the tree, constraints
on children that must appear with a parent What other possibilities make sense?
Natix uses a B+ Tree-like scheme for achieving balance and splitting a tree across pages
27
Example
Split point in parent page
Note “proxy” nodes
28
That Was Simple – But What about Updates?
Clearly, insertions and deletions can affect things Deletion may ultimately require us to rebalance Ditto with insertion
But insertion also may make us run out of space – what to do? Their approach: add another page; ultimately may
need to split at multiple levels, as in B+ Tree
Others have studied this problem and used integer encoding schemes (plus B+ Trees) for the order
29
Does this Help?
According to general lore, yes The Natix experiments in this paper were
limited in their query and adaptivity loads But the IBM people say their approach, which
is similar, works significantly better than Oracle’s shredded approach
30
There’s More to Updates than the Pages
What about concurrency control and recovery?
We already have a notion of hierarchical locks, but they claim: If we want to support IDREF traversal, and
indexing directly to nodes, we need more What’s the idea behind SPP locking?
31
Logging
They claim ARIES needs some modifications – why?
Their changes: Need to make subtree updates more efficient – don’t
want to write a log entry for each subtree insertion Use (a copy of) the page itself as a means of tracking
what was inserted, then batch-apply to WAL “Annihilators”: if we undo a tree creation, then we
probably don’t need to worry about undoing later changes to that tree
A few minor tweaks to minimize undo/redo when only one transaction touches a page
32
Annihilators
33
Assessment
Native XML storage isn’t really all that different from other means of storage There are probably some good reasons to
make a few tweaks in locking Optimization remains harder
A real solution to materialized view creation would probably make RDBMSs come close to delivering the same performance, modulo locking