Recognizing Human Activity from Sensor Data Henry Kautz University of Washington Computer Science & Engineering graduate students: Don Patterson, Lin Liao.

Recognizing Human Activity from Sensor Data

Recognizing Human Activity from Sensor Data

Henry Kautz

University of WashingtonComputer Science & Engineering

graduate students: Don Patterson, Lin Liao

CSE faculty: Dieter Fox, Gaetano Borriello

UW School of Medicine: Kurt Johnson

Intel Research: Matthai Philipose, Tanzeem Choudhury

Converging Trends…Converging Trends…

Pervasive sensing infrastructureGPS enabled phonesRFID tags on all consumer productsWireless motes

Breakthroughs in core artificial intelligenceAfter “AI boom” fizzled, basic science went on…Advances in algorithms for probabilistic reasoning and machine learning

Bayesian networksStochastic sampling

Last decade: 10 variables 1,000,000 variablesHealthcare crisis

Epidemic of Alzheimer’s Disease Deinstitutionalization of the cognitively disabledNationwide shortage of caretaking professionals

...An Opportunity...An Opportunity

Develop technology toSupport independent living by people with cognitive disabilities

At homeAt workThroughout the community

Improve health careLong term monitoring of activities of daily living (ADL’s)Intervention before a health crisis

The University of Washington Assisted

Cognition Project

The University of Washington Assisted

Cognition ProjectSynthesis of work in

Ubiquitous computingArtificial intelligenceHuman-computer interaction

ACCESSSupport use of public transit

CAREADL monitoring and assistance

This TalkThis TalkBuilding models of everyday plans and goals

From sensor dataBy mining textual descriptionBy engineering commonsense knowledge

Tracking and predicting a user’s behavior

Noisy and incomplete sensor dataRecognizing user errors

First steps toward proactive assistive technology

ACCESSAssisted Cognition in Community, Employment, & Support Settings

Supported by The National Institute on Disability &

Rehabilitation Research (NIDDR)The National Science Foundation (NSF)

ACCESSAssisted Cognition in Community, Employment, & Support Settings

Supported by The National Institute on Disability &

Rehabilitation Research (NIDDR)The National Science Foundation (NSF)

Learning & Reasoning About Transportation

Routines

TaskTask

Given a data stream from a wearable GPS unit...

Infer the user’s location and mode of transportation (foot, car, bus, bike, ...)Predict where user will goDetect novel behavior

User errors?Opportunities for learning?

Why Inference Is Not Trivial

Why Inference Is Not Trivial

People don’t have wheelsSystematic GPS error

We are not in the woodsDead and semi-dead zonesLots of multi-path propagationInside of vehiclesInside of buildings

Not just location trackingMode, Prediction, Novelty

GPS Receivers We UsedGPS Receivers We Used

Nokia 6600 Java Cell Phone with Bluetooth GPS

unit

GeoStats wearable GPS

logger

Geographic Information Systems

Geographic Information Systems

Bus routes and bus stopsData source: Metro GIS

Street mapData source: Census 2000

Tiger/line data

ArchitectureArchitecture

Learning Engine

Inference Engine

GIS

Database

Goals Paths Modes Errors

Probabilistic ReasoningProbabilistic Reasoning

Graphical model: Dynamic Bayesian network

Inference engine: Rao-Blackwellised particle filters

Learning engine: Expectation-Maximization (EM) algorithm

Graphical Model (Version 1)

Graphical Model (Version 1)

Transportation ModeVelocityLocation

BlockPosition along blockAt bus stop, parking lot, ...?

GPS Offset ErrorGPS signal

Rao-Blackwellised Particle Filtering

Rao-Blackwellised Particle Filtering

Inference: estimate current state distribution given all past readingsParticle filtering

Evolve approximation to state distribution using samples (particles)Supports multi-modal distributionsSupports discrete variables (e.g.: mode)

Rao-BlackwellisationEach particle includes a Kalman filter to represent distribution over positionsImproved accuracy with fewer particles

TrackingTracking

blue = foot

green = bus

red = car

LearningLearning

User model = DBN parametersTransitions between blocksTransitions between modes

Learning: Monte-Carlo EMUnlabeled data30 days of one user, logged at 2 second intervals (when outdoors)3-fold cross validation

ResultsResults

ModelMode Prediction

Accuracy

Decision Tree(supervised)

55%

Prior w/o bus info 60%

Prior with bus info 78%

Learned 84%

Pro

babili

ty o

f co

rrect

ly

pre

dic

ting t

he f

utu

rePro

babili

ty o

f co

rrect

ly

pre

dic

ting t

he f

utu

re

City BlocksCity Blocks

Prediction AccuracyPrediction Accuracy

How can we improve

predictive power?

Transportation Routines Transportation Routines

BA

Goalswork, home, friends, restaurant, doctor’s, ...

Trip segmentsHome to Bus stop A on FootBus stop A to Bus stop B on BusBus stop B to workplace on Foot

Work

“Learning & Inferring Transportation Routines”, Lin Liao, Dieter Fox, & Henry Kautz, AAAI-2004 Best Paper Award

Hierarchical ModelHierarchical Model

Transportation mode

x=<Location, Velocity>

GPS reading

Goal

Trip segment

xk-1

zk-1 zk

xk

mk-1 mk

tk-1 tk

gk-1 gk

Hierarchical LearningHierarchical Learning

Learn flat modelInfer goals

Locations where user is often motionlessInfer trip segment begin / end points

Locations with high mode transition probability

Infer trips segmentsHigh-probability single-mode block transition sequences between segment begin / end points

Perform hierarchical EM learning

Inferring GoalsInferring Goals

Inferring Trip SegmentsInferring Trip Segments

Going to work Going home

Correct goal and route predicted

100 blocks away

Novelty & Error DetectionNovelty & Error Detection

Approach: model-selectionRun several trackers in parallel

Tracker 1: learned hierarchical modelTracker 2: untrained flat modelTracker 3: learned model with clamped final goalEstimate the likelihood of each tracker given the observations

Detect User Errors

Untrained Trained Instantiated

Application:

Opportunity Knocks

Application:

Opportunity Knocks

Demonstration (by Don Patterson) at AAHA Future of Aging Services, Washington, DC, March, 2004

CARECognitive Assistance in Real-world

Environments

supported by the Intel Research Council

CARECognitive Assistance in Real-world

Environments

supported by the Intel Research Council

Learning & Inferring Activities of Daily Living

Research HypothesisResearch Hypothesis

Observation: activities of daily living involve the manipulation of many physical objects

Cooking, cleaning, eating, personal hygiene, exercise, hobbies, ...

Hypothesis: can recognize activities from a time-sequence of object “touches”

Such models are robust and easily learned or engineered

Sensing Object Manipulation

Sensing Object Manipulation

RFID: Radio-frequency ID tagsSmallSemi-passiveDurableCheap

Where Can We Put Tags?Where Can We Put Tags?

How Can We Sense Them?How Can We Sense Them?

coming... wall-mounted “sparkle reader”

Example Data StreamExample Data Stream

Making TeaMaking Tea

Building ModelsBuilding Models

Core ADL’s amenable to classic knowledge engineeringOpen-ended, fine-grained models: infer from natural language texts?

Perkowitz et al., “Mining Models of Human Activities from the Web”, WWW-2004

Experimental SetupExperimental Setup

Hand-built library of 14 ADL’s17 test subjectsEach asked to perform 12 of the ADL’sData not segmentedNo training on individual test subjects

Activity Prior Work

CAREAccuracy/Recall

Personal Appearance 92/92

Oral Hygiene 70/78

Toileting 73/73

Washing up 100/33

Appliance Use 100/75

Use of Heating 84/78

Care of clothes and linen 100/73

Making a snack 100/78

Making a drink 75/60

Use of phone 64/64

Leisure Activity 100/79

Infant Care 100/58

Medication Taking 100/93

Housework 100/82

95/84

General SolutionQuantitative Results

Point SolutionQuantitative Results

Point SolutionAnecdotal Results

General SolutionAnecdotal Results

Pervasive Computing, Oct-Dec 2004

Current DirectionsCurrent Directions

Affective & physiological stateagitated, calm, attentive, ...hungry, tired, dizzy, ...

Interactions between peopleHuman Social Dynamics

Principled human-computer interaction

Decision-theoretic control of interventions

Why Now?Why Now?

A goal of much work of AI in the 1970’s was to create programs that could understand the narrative of ordinary human experienceThis area pretty much disappeared

Missing probabilistic toolsSystems not able to experience worldLacked focus – “understand” to what end?

Today: tools, grounding, motivation

Challenge to Nanotechnology

Community

Challenge to Nanotechnology

CommunityCurrent sensors detect physical or physiological state: user mental state must be indirectly inferredTo what can extend can nanotechnology afford direct access to a person’s emotions and intentions?