Semantic Streams: a Framework for Composable Semantic Interpretation of Sensor Data Kamin Whitehouse UC Berkeley EWSN, Feb 13, 2006 Joint with Feng Zhao and Jie Liu, Microsoft Research
Dec 30, 2015
Semantic Streams: a Framework for Composable Semantic
Interpretation of Sensor Data
Kamin WhitehouseUC Berkeley
EWSN, Feb 13, 2006
Joint with Feng Zhao and Jie Liu, Microsoft Research
fact
DataProcessing
DataProcessing
fact
fact
DataProcessing
Motivation
• Imaging that sensors cover the globe
• Can we automatically query for world facts?
• No: each fact is an interpretation of data
• Processing must consider– Sensor context– Sensor fusion– Data formats– Calibration– Signal transforms– Etc.
World Sensor
DB
fact
DataProcessing
fact
DataProcessing
Semantic Streams• Goal: to allow direct
querying of facts• Example query: “Is a
vehicle in the parking lot”• Instantiates a graph of
inference units to derive the desired fact
• Output is a stream of all vehicles detected
Fact: vehicle
SizeInference
VehicleInference
SpaceInference
Fact: empty space
Sensor.data=72Sensor.data=56Sensor.data=98obj.size=23 obj.size=23Sensor.data=49obj.type=vehicle
Usage Model
• Fixed Sensor infrastructure– Many simultaneous, unrelated users – Short-term queries repeatedly over long periods– Queries are often similar (though not identical)
• When programmer poses new query, system will use existing sensors and inference units, if possible.
• Otherwise, system may give actionable error messages, – need to add 2 more sensors in area XYZ– need to add new inference unit to the system
• Semantic values produced by infrastructure grows organically as it is used for new purposes
Complete System
MobileApplication
QueryQuery
ProcessorInference Graph
Embedding Engine
MSTMLExecution
Engine
Sensors
Planning time
User
OutputStream
Inference UnitInstances
Run time
Background: Prolog and FOL
• george, jane, carA• X, Y, Z, StudentA• female(jane), parent(X,Y)• parent(george,jane).
male(george).• father(X) :- parent(X,Y),male(X).• male(X).
> X = george• father(X).
> X = george• father(jane).
> false
• Constants• Variables• Predicates• Facts• Rules• Queries• Query processing
Markup Languagesensor(magnetometer,
[[60,0,0][70,10,10]]).
sensor(camera,
[[40,0,0][55,15,15]]).
sensor(breakBeam,
[[10,0,0][12,10,2]]).
inference( speedDetector,
needs(
sensor(magnetometer, R) ),
creates(
stream(X),
isa(X,object),
property(X,T,time),
property(X,R,region),
property(X,S,speed) ) ).
• Sensors are declared as logical facts– All regions are appx’d by
3D cubes
• Inference units are logical rules– Antecedents: the fact
streams that it needs– Consequents: the fact
streams that it creates– Spatial relationship
encoded in variable name
Stream Relationships
subregion( A, B )
inference(speedDetector,
needs(
sensor(magnetometer, R) ),
creates(
stream(X),
isa(X,object),
property(X,T,time),
property(X,R2,region),
subregion(R2,R),
property(X,S,speed) ) ).
• More sophisticated spatial relationships– Subregions
Sensor coverage (R)
Queried area (R2)
Stream Relationshipsinference( speedDetector,
needs(
sensor(breakBeam, R1),
sensor(breakBeam, R2),
sensor(breakBeam, R3),
subregion(R1,R),
subregion(R2,R),
subregion(R3,R),
\+ intersecttion(_,R1,R2),
\+ intersecttion(_,R1,R3),
\+ intersecttion(_,R2,R3) ),
creates(
stream(X),
isa(X,object),
property(X,T,time),
property(X,R,region) ) ).
• More sophisticated spatial relationships– Subregions– Non-intersection
Queried area
Sensor coverage
Stream Relationships
inference( vehicleDetector,
needs(
sensor(magnetometer, R),
stream(X),
isa(X,object),
property(X,S,speed) ),
creates(
stream(X),
isa(X,vehicle) ) ).
• More sophisticated spatial relationships– Subregions– Non-intersection
• Stream identity
Query processing
1. Choose first predicate in query
2. Search KB for sensors that match
3. Search output of inference units for streams that match
4. Add new input streams to query
5. Back to step 1
Region(A,[[10…]])isa(A,vehicle)
Stream(X),Isa(X,vehicle)
stream(X),property(X,speed)
Sensor(mag,…)
VehicleInference
stream(Y),property(Y,speed)
Sensor1 Sensor2 Sensor3
SpeedInference
Query processing
• Actionable Error Messages– When missing
inference unit– When missing
sensor
Region(A,[[10…]])isa(A,vehicle)
Stream(X),Isa(vehicle,X)
stream(X),property(X,speed)
Sensor(mag,…)
stream(Y),property(Y,speed)
Sensor1 Sensor2 Sensor3
VehicleInference
SpeedInference
Query processing
• When too many choices– Multiple graphs
provide logically equivalent results
Region(A,[[10…]])isa(A,vehicle)
Stream(X),Isa(vehicle,X)
stream(X),property(X,speed)
Sensor(mag,…)
stream(Y),property(Y,speed)
Sensor1 Sensor2 Sensor3
VehicleInference
SpeedInference
stream(Y),property(Y,speed)
Camera
SpeedInference
Quality of Service Constraints
subregion( A, B )
inference(speedDetector,
needs(
sensor(camera, R) ),
creates(
stream(X),
isa(X,object),
property(X,T,time),
property(X,R2,region),
property(X,S,speed),
property(X,C,confidence),
{C > 99},
property(X,L,latency),
{L < 450} ) ) .
• Inference unit may specify QoS of output– Confidence– Latency
Quality of Service Constraintssubregion( A, B )
inference(speedDetector,
needs(
sensor(breakBeam, R1) ),
sensor(breakBeam, R2) ),
sensor(breakBeam, R3) ),
creates(
stream(X),
isa(X,object),
property(X,T,time),
property(X,R2,region),
property(X,S,speed) ) ).
property(X,C,confidence),
{C > 80},
property(X,L,latency),
{L < 75} ) ) .
• Inference unit may specify QoS of output– Confidence– Latency
Quality of Service Constraints
subregion( A, B )
inference(vehicleDetector,
needs(
sensor(magnetometer, R1),
stream(X),
property(X,S,speed),
property(X,C2,confidence),
property(X,L2,latency),
creates(
stream(X),
isa(X,vehicle),
{C > 0.9 * C2},
{L = L2 + 125} ) ).
• Inference unit may specify QoS of output– Confidence– Latency
• May be function of input QoS
Query processing
• Query can declare QoS parameters to create ordering of inference graphs
Region(A,[[10…]])isa(A,vehicle)
Stream(X),Isa(vehicle,X)
stream(X),property(X,speed)
Sensor(mag,…)
stream(Y),property(Y,speed)
Sensor1 Sensor2 Sensor3
VehicleInference
SizeInference
stream(Y),property(Y,speed)
Camera
SizeInference
{C>70, L <300}
Example Infrastructure
Goal Application: Vehicle Detection
Query and Inference Graphisa(Y,histogram),property(Y,T,value),property(X,T,time),
isa(X,vehicle),region(X,R,[[10,10,0][40,30,12]]),Isa(Z,photo),property(Z,X,triggerStream)
VehicleInference
SpeedInference
DirectionInference
SizeInference
HistogramService
PhotoService
Break Beams Magnetometers Camera
Screenshot
Final Application
Camera image
Report (histogram)
Break beam plot
Micro Server Camera Server
Magnetometer plot
Conclusion
• Semantic Streams– Infers facts from data– Intended for sensor infrastructure– Can select inference graph based on QoS constraints
• Limitations– Must be a large supply of inference units– Stream operators are difficult to program
• Different programming language/execution model• Programmer must deal with temporal values, implement
appropriate buffering, etc.
Future Work: Streaming KB’s
Relational Database : Knowledge Base ::
Streaming Database : Streaming KB
SQL operators: Select, Join, MaxOperate on Tables
Logical rules: c(X) :- a(x),b(x).Operate on Facts
Infinite tables Infinite facts
Video: Vehicle Detection
Video: Human Filtering
Traffic Histogram