Copyright © 2004 IEEE-USA Geriatric Care Working Group – June 2004 Howard D. Wactlar Carnegie Mellon University Pittsburgh, USA CareMedia Video and Sensor.

Copyright © 2004

IEEE-USA Geriatric Care Working Group – June 2004

Howard D. WactlarCarnegie Mellon University

Pittsburgh, USA

CareMediaVideo and Sensor Analysis for Geriatric Care

Copyright © 2004

Goal: Automated Behavior Analysis in the Nursing Home

We’re interested in automating detection of behavioral & psychological symptoms of dementia (BPSD).

• Monitoring and maintaining the quality of life

• With Western Psychiatric Institute & Clinic (WPIC-UPMC, U.of Pgh)

Ultimately, we seek to make automated, quantitative measurements to:

• Explore relationship of BPSD to environments in which they occur

• Evaluate symptoms longitudinally

• Determine the frequency of BPSD

• Develop a patient profile of responses of BPSD to pharmacological and non-pharmacological interventions

• >>>> Enable earlier intervention to sustain quality of life

Falling

Social InteractionIdentification

Kathy Jones

Aggression

Emotiontime

Circadian Analysis

Behavioral Analysis & Summarization

Information Access

Skilled Nursing Facility

Information Extraction & Identification

Activity & Environmental MonitoringLocation Biometrics SensorVideo

00011101

Pri

va

cy

Te

ch

no

log

y &

Po

lic

y

Doctor

Clinical

Researchers

Caregiver

Pharmaceutical

Companies

Hospitals

Public Health

Location Gesture

Time

Event People

SoundsMovement

Database

…

Environment

CareMedia Overview

Copyright © 2004

Copyright © 2004

Applications in the Nursing Home

• Clinical/Research

• Tracking patient behavior and incidents in long-term care facilities• e.g., disruptive vocalizations, falls• recording patient mobility and activity levels

• Correlating with time of day, location and environmental factors• Observing effects of drugs on individuals and groups

• Patient• Cognitive assist - reminding, alerting and summoning help

• Staff training• Analysis of video records of incidents used for training

• Management• Monitoring and documenting compliance

Copyright © 2004

CareMedia: What are the observables?

• Who? • Identify people across

cameras, days.

• What are they doing?• Wandering around

• Working on tasks

• Looking for things

• Eating, sleeping in public • How well did they do it?

• Quantify normal performance

• Detect/report anomalies

Query and Summary Reporting Daily patient summary distance traveled sleep behaviors triggers affect Location summaries use of space environmental triggers

Searchable video index by patient by time by event by location by behavior

Visual Processing Color Edge detection Motion detection Object detection Head/hands/torso det. Event detection Face ID Gait ID

Audio Processing Event detection Voice detection Voice ID Speech to text

Integration and Indexing Person ID Fall detector Behavior detection Interaction detection Interaction classification Aggressive Social

Hallway

rawvideo

& audio

rawvideo

& audio

Information Extraction and Reporting

Copyright © 2004

Copyright © 2004

Current research in activity recognition can loosely be grouped into a couple of categories:

1. kinematic/dynamic(Wren, Ma, Blake, Black, Freeman)

2. non-parametric, statistical(Davis, Liu, Lee, Yang, Cutler, Shi)

Interpreting Behavior

Copyright © 2004

Coarse Motion Measurement

Applying mean-shift analysis:

target detection

red indicates target

Informedia Digital Libraries

Copyright © 2004

Fine Motion with Directions

Applying optical flow analysis:

Copyright © 2004

Summarization Visualization

Large hr. of video

Measure Normal Activity, Detect What’s Not

Find pattern

Few important events

Copyright © 2004

Problem: Privacy Protection in Public Places

• Block the persons that are reluctant to be captured in the video

• ¼ of nursing home residents deny disclosure of their images

• Real-time automatic people tracking framework

• Detect foreground information; adapt for real-time background

• Multi-target, multi-assignment blob matching

• Apply mean shift algorithm to separate merged persons

Copyright © 2004

Problem: Monitoring in Private Spaces

• “Observe” and monitor activity without storing video• Maintain only feature vectors; classify in real-time

• Record event type, time of day, duration

• Detect changes in daily pattern of activity

• Example: Monitor bathroom/mirror activities• What: brushing teeth, combing hair, washing hands, washing

face

• How: small camera behind center of mirror, mono microphone, embedded computing

• Create summary

• how long, how often, chart by day

Copyright © 2004

Copyright © 2004

Future Opportunities

• Upgrade to hi-resolution cameras for fine motor detection• Measure tremors, facial expressions

• Instrument with distributed sensors for precision• Force sensors in chairs, beds, carpeting

• RFID in clothing, utensils

• Conduct large-scale testbeds for validation• Comprehensive instrumentation in multiple homes

• Move through lesser levels of care to expand market• From constrained skilled care environments to less structured

assisted and independent living

• >>>> Enable earlier detection and intervention

• Delaying nursing home entry by 1 month saves $1.2B/year

Copyright © 2004

Thank You

Questions?