Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of Washington, Seattle, WA Funding for this research is provided in part by IISI and AFRL/IF
Dec 29, 2015
Making Sense of Sensors
Henry KautzDepartment of Computer Science & EngineeringUniversity of Washington, Seattle, WA
Funding for this research is provided in part by IISI and AFRL/IF
Making Sense of Sensors
or … Climbing the Data Interpretation Food-Chain
The Ubiquitous Future
Rapidly declining size and cost of sensing and networking technology makes it practical to rapidly deploy systems that monitor large environments in great detail– factories, airports, hospitals, homes– oceanic regions, cities, countryside
Problem: it is easier to collect data than make to sense of it!
Data Fusion
Traditional work in data-fusion attacks problem of recovering specific physical phenomena from the readings of homogeneous networks of noisy sensorsE.g.: given readings from underwater microphone array, determine the position of a submarine
Current Trends
Heterogeneous sensors– Instrumented environment: motion detectors, weight
detectors, video, audio, …– Instrumented personnel: smart badges, GPS phones,
metabolic sensors. …
Goal: high-level understanding– What actions are being performed?– What are the goals of the subjects?– Do we need to intervene?
Example: Security
System monitors activity in a post officeTracks common tasks performed by individuals– Mailing packages– Getting mail from PO boxes– Buying stamps
Alerts operator when abnormalities noted– Person leaves package on floor and exits– Loitering (but not waiting in line!)
Example: Guiding
Activity Compass: GPS system that– Learns daily patterns of travel – Understands walking, car, bus, bike– Integrates external information
• Real-time bus data
Predicts problems– Will user miss appointment?– Is user on the wrong bus?
Offer proactive help– E.g., suggest alternative travel plan
Triple-Use Technology
Plan-AwareComputing
Military
surveillanceaugmented cognition
CommercialSoftware
intelligent user interfaces
PatientCare
aging in placeassisted cognition
Key Issue
How to go from noisy and incomplete sensor measurements toA meaningful description of what a person is doing
• “Waiting to mail package”• “Trying to get home”
A decision by the system about whether or not to intervene … in a principled and scalable manner!
Data Interpretation Food Chain
Movement
Intentions
Behavior
Interventions
Model-Based Interpretation
General approach: build a probabilistic model of– Common user goals– Plans (complex behaviors) that achieve those goals
• Feasibility constraints • Temporal constraints• Failure (abnormality) modes
– How simple behaviors are sensed
Run model “backwards” to interpret sensed data
Million-Mile View
In principal we know how to estimate the state of the system under observation:
To make this practical, we must take advantage of the regular structure of the domain
1 1 1Bel( ) Pr( | ) Pr( | ) Bel( )t t t t t t tx z x x x x dx state at time t
observation at time t
system dynamics
Technical Foundations
Hidden Markov models– Mathematical framework for describing processes
with hidden state that must be inferred from observations
Hierarchical plan networks– Represents how a task can be broken down into
subtasksHierarchical hidden Markov models*– Key to climbing food-chain!
* Precisely speaking: factorial hierarchical hidden semi-Markov models
Video Door Sensor Motion
Location
Example
Enter PO
Wait in line
Let go package
Pay cashier
Exit PO
Mail Package
Video Door Sensor Motion
Location
Enter PO
Go to PO
boxes
Open PO box
Pick up mail
Exit PO
Retrieve Mail
Example
Video Door Sensor Motion
Location
Mail Package
PO Patron
Retrieve Mail
Outside PO
Example
Inexplicable Observations
Enter PO
Wait in line
Let go package
Pay cashier
Exit PO
Mail Package
Enter PO
Go to PO
boxes
Open PO box
Pick up mail
Exit PO
Retrieve Mail
Enter PO
Let gopackage
Exit PO
Absolute Timing Constraints
Mail Package active [9 am – 4 pm]
Enter PO
Retrieve Mail active [6 am – 8 pm]
Enter PO
Relative Timing Constraints
Go to PO
boxes
Open PO box
Retrieve Mail
Timeout
seconds
seconds
Forgot combo?Safecracking?
Summary
Commonsense knowledge base of “significant” behaviors– Hierarchically organized– Probabilistic at all levels– Many parallel ongoing activities possible– Absolute and relative timing constraints– Probabilities “tuned” by machine learning techniques for
individual users– Inexplicable observations and failure modes – points of
possible intervention
Interventions
Framework allows system to predict when an anomalous situation is likelyDifferent anomalies have different costs– Confused patron– Deliberate loitering– Planting bomb
Must avoid:
Deciding When to Intervene
(Horvitz 98)G = prediction that help is needed
Common Architecture
Activity Compass
Palm-based wireless GPS– No explicit programming – learns pattern of
transportation plans – Accesses user’s calendar, real-time bus information– Constantly tries to predict where user will go next, and
whether problems will arise– Proactive help:
• “Walk faster or you’ll miss the 9:15 bus!”• “Green St bus is late, suggest you take Elm St bus instead”
Substeps
Cleaning up GPS data– 3 meter accuracy– frequent signal loss– determine most likely path
Infer mode of transportationPredict when and where transitions in mode of travel will occurPredict points of possible failure
indoors
walk
bus
bikecar
Gathering Data
On Foot: Across Campus
By Bus: Across Seattle
Transition Prediction
Training Data:– 20,000 GPS readings gathered over 3 weeks
Inferring current mode– Input: current location, time, velocity– 98% accuracy (10 FCV)
Predicting next transition– Input: current mode, location, time, velocity– 97% accuracy (10 FCV)*
* Don is a very organized guy. Your accuracy may vary.
Predicting Transition Location
User Interface
Assisted Cognition
“Plan aware” systems to help people with cognitive disabilitiesNew project based at University of Washington – Computer Science & Engineering– UW Medical Center, ADRC– Collaborators: Intel, OGI, Elite Care
http://assistcog.cs.washington.edu/
Summary
Potential of widespread sensor networks just beginning to be tappedKey issue: interpreting data in terms of human behavior, plans, and goalsResearchers in data fusion, AI, and “ubicomp” coming together around a core set of representations and algorithms