Overview of Life Logging

Overview of Life Logging

September 4, 2008

Sung-Bae Cho

• Life logging

• Context-aware computing

• Sensory data for activity recognition

• Life logging with mobile devices

• Summary

Agenda

0

0.2

0.4

0.6

0.8

1

2002 2003 2004 2005 2006 2007

Dis

k ca

pac

ity

(TB

)

Source: Microsoft 2003

Advances of Digital Devices

Right now, it is affordable to buy 40 GB (in 2003)

In 3 years 1TB/year is affordable!

It is hard to fill a terabyte/year, but you can:Look at 9,800 pictures a day (300 KB JPEGs)

Read 2,900 documents a day (1MB files)

Listening to audio or view compressed video 24 hours/day (it takes more than 256 kb/s to fill a TB in a year)

Watch 1.5 Mb/s video 4 hours each day.

3/48

4/48

Digital Convergence

Properties of Mobile Devices

Multi-function Mobility

Large Memory Personal Device

Collect Various Log Data of

User’s Everyday Life

Advances of Mobile Devices

Everyday Life with Mobile Devices

I’m always I’m always with you!with you!

5/48

Available Information from Mobile Devices

Photo

Where is the place?

Who is in the media?

Video

How many people in the media?

User Created Contents

What is the activity?

Sensor Log

Bluetooth

Who is nearby?

Bio-sensor

Sleep? Activity Level?

GPS

Location

Vision Sensor

Contexts of environments

Audio

Noisy level?

User Data

Scheduler Personal Profile

Details on daily events Preference

Human Relationships

Demographical information

E-mail

Address Book

Usage Log

Call Log

DurationReceiver/Caller

Time

SMS Log

ContentsReceiver/Sender

Time

Application Usage

Mp3 Player

6/48

Life(Experience)

DigitalInformation

Life Record

Context DataAlways Bringing Contents Creation

Human Memorize Experience

Life Logging with Personal Digital Devices

Personal Digital Device

7/48

Necessity of Context-aware Computing

Integration of Various Typeof Information

User Modeling fromPersonal Information

Data Mining fromPersonal Information

Context-aware Computing

Personal Information Retrievalbased on Semantic Structure

Episodic Management ofPersonal Information

Summary ofEveryday Life

8/48

• Life logging

• Context-aware computing

• Sensory data for activity recognition

• Life logging with mobile devices

• Summary

Agenda

Definition of Context-aware Computing

• Context : Dey & Abowd (1999)

– Any information that can be used to characterize the situation of an entity, where an entity can be a person, place, physical or computational object

• Context-aware computing

– The use of context to provide task-relevant information and/or services to a user, wherever they may be

• Three important context-aware behaviors

– The presentation of information and services to a user

– The automatic execution of a service

– The tagging of context to information for later retrieval

• Importance of context

– The context of user is changed frequently and drastically in ubiquitous and mobile environment (Pascoe, et al., 1998)

10/48

Trends on Context Awareness

• TEA: EU project for enabling context awareness

– 1998 ~ 2000, with TeCo, Starlab, Omega and Nokia

– scenarios, technologies, market research and demonstrators

• DrWhatsOn concept project

– concept and user scenarios for context aware office PDA device

– main focus in usability and user interface design

• Earlier the focus was in sensor-based context recognition

– recognize and utilize information from user's activity and environment properties

• Now more directions and possibilities exist

– Context-aware computing

– Context-aware communication

– Context-aware services, connectivity, information retrieval, affective computing, etc.

TEA

TEA

DrWhatsOn

11/48

Why Context Awareness is Difficult?

• Context recognition is never 100% reliable

– Contexts are vague, dynamic, overlapping, and ill-defined

• Adaptive user interfaces are scary!

– May need an adaptive, learning SW

• Sensors & algorithms may need constant recalibration

– They may also be too CPU intensive

• Application development frameworks & support are missing

– Security, privacy concerns a lot

• Connectivity to environment

– Limited I/O through wireless links

– What short range connectivity technology should be used?

• Productization

– Where’s the business?

– What everyday problems it really solves?

– Where do we get sensors for sensor-based context recognition?

12/48

Some Technical Challenges

• Understanding structure and behavior of context information

– context are fuzzy, overlapping, and changing in time

• Lack of sensing methods for context information from various sources

• Lack of methods for fusing context information

• Lack of format of context information

• Sensor-based context recognition

– hard to obtain reliable data

– signal processing and recognition consume memory, energy and processing time

– results may be ambiguous

• Connectivity to environment and other devices

– Are profiles available; how about location information?

– Any privacy & security risks?

13/48

Why do we Need Context Awareness?

Maybe avoid disturbing people

at wrong times

Right information, right time, right place

Receding to background

(Calm computing)

New services

(Location-based services are the obvious ones)

Increased user

satisfaction

Better social acceptance

of technology

14/48

• Life logging

• Context-aware computing

• Sensory data for activity recognition

• Life logging with mobile devices

• Summary

Agenda

MIT Media LAB (1)

• Area: Visual Contextual Awareness in Wearable Computing (1998)

• Sensor: Vision

• Probabilistic object recognition

– Probabilistic dependence analysisbased on neighbor vector & object recognition

• O: object, M: measurement

– Task recognition with HMM

16/48

MIT Media LAB (2)

• Activity recognition based on accelerometer (2004)

– 20 activities

• Sensor: Accelerometer on diverse points of body

• Geometric analysis

– Mean, energy, frequency-domain entropy, and correlation

• Classification method

– Decision Tree C4.5, IBL, Naive Bayes

17/48

MIT Media LAB (3)the number of subjects collected under laboratory (L) or naturalistic (N) settings

18/48

MIT Media Lab (4)

• Aggregate confusion matrix for C4.5 classifier

– leave-one-subject-out validation for 20 subjects

19/48

eWatch Sensor Platform

• CMU Computer Science Lab in 2005– Defense Advanced Research Projects Agency (DARPA)

• Activity recognition, improving power consumption, location recognition

• Hardware

– LCD, LED, vibration motor, speaker,Bluetooth for wireless communication

– Li-Ion battery with a capacity of 700mAh

• Sensors

– a two-axis accelerometer (ADXL202; +/- 2g)

– Microphone, light & temperature sensors

• Method

– multi-class SVMs

– HMM based selective sampling

20/48

Classifier

• multi-class SVMs with Gaussian Radial Basis Function kernels

– frequency spectrum-based classification

– time-domain-based classification with SVM

• means, variances, square root of the uncentered second moment, the median absolute differences

21/48

University of Alberta

• National ICT Australia Project: University of Alberta, Canada

• Human activity & gesture recognition

• Sensor

– active, magnetic field, acoustic, laser, camera sensor

• Method

– Coupled hidden Markov model (CHMM)

• Extended HMM model for combining and utilizing concurrent information stream more effectively (By M. Brand et al., 1997)

– M. Brand, N. Oliver, and A. Pentland, “Coupled hidden Markov models for complex action recognition,” in IEEE Intl. Conf. Comp. Vis. Pat. Rec., pp. 994-999, 1997.

22/48

University of Bologna

• Micrel Lab: University of Bologna, Italy (2004)

• Research

– Construction of Ubiquitous environments

– Sensor-based gesture recognition

• Sensor

– Wireless MOCA (Motion capture with integrated accelerometers) sensor

• Accelerometer, gyroscope

• Small size, small power cosume

• Wireless operation

• Sticking on diverse points of human body

• Method

– Hidden Markov Model

23/48

Activity Recognition Summary

• Mostly focused on static classification for pose recognition (relatively easy)

USAMIT

FinlandNokia

JapanATR

Aus. Curtin Univ.

SwissETH

Zurich

BelgiumStarlab.

ChinaSJT Univ.

CanadaCMU

Static classification ○ ○ ○ ○ ○ ○ ○

Temporal classification

○ △

Accelerometer ○ ○ ○ ○ ○ ○ ○ ○

Vision sensor ○ ○

Instant behavior ○ △

Repeated behavior (Pose)

○ ○ ○ ○ ○ ○ ○

Geometric calculation

○ △ ○ △ △ △ ○ △

HMM △ ○ △

SOM ○

24/48

• Life logging

• Context-aware computing

• Sensory data for activity recognition

• Life logging with mobile devices

• Summary

Agenda

Context-aware Mobile Device Trends

• Increasing demands for the multi-functional and high-performance phone

• Prospects

– Increasing demands for Smart Phone

– Decreasing basic functional phone

• Enlarging role as a computing equipment

– Internet surfing

– External storage

• Increasing investment for adding value of phone

• Digital convergence with other functionality

– Mp3 player

– Personal Media Player

– Camera & Camcoder

26/48

Context-aware Phone on Media

The official newsletter of the institute for Complex Engineered Systems (CMU) Sep/Oct 2003

New Scientist, Nov 2004 MIT Sloan Management Review Fall 2004

27/48

SenSay (Sensing and Say)

• Carnegie Mellon University (2003)

• A context-aware mobile phone

– Adapting to dynamically changing environmental and physiological states

– Manipulating ringer volume, vibration, and phone alerts

– Providing remote callers with the ability to communicate the urgency of their calls

– making call suggestions to users when they are idle

– Providing the caller with feedback on the current status of the Sensay user

• Sensors

– accelerometers, light, and microphones

– mounted at various points on the body

28/48

SenSay (2)

• Context recognition by thresholding

29/48

SenSay (3)

• Context classification based on self-organizing map

30/48

MIT Reality Mining Group

• Utilizing Context application from the University of Helsinki (Raento et al., 2005)

• Capturing mobile phone usage patterns

– from one hundred people (MIT students)

– for an extended period of time

• Providing

– insight into both the users

– the ease of use of the device itself

• Method: Bayesian inference

31/48

Nokia

• Nokia 7650 (2002)

– Backlight time adjustment based on proximity & light sensor

– Speaker-phone mode change by proximity sensor

• Cancellation of speaker-phone mode near ear

• Nokia 6230, 6820, 7200

– Presence-enhanced chat service

• Presenting and sharing the user’s status

– SMS messaging based on other’s status

Light SensorProximity Sensor

32/48

Microsoft Research (1)

33/48

Microsoft Research (2)

• Alarm method for incoming call

– Quiet ringing

• Volume down with hand touching

– Acknowledging and ignoring calls

• Calling response: Tilting to body of phone on the pocket

• Notification stop: Holding phone with hand without movement on pocket

– Target device for notifications

• Device selection for alarm: Selecting the recent device if there are several devices of user

– Vibration notification

• Vibration mode change: Holding phone for a long time

34/48

Tilt Sensor & User Behavior

• Upper: forward/back tilt

• Below: left-right tilt

35/48

VTT Electronics (Finland)

• Supported by Nokia

• VTT Electronics – Advanced Interaction Systems – Context Awareness

• Fuzzy/Bayesian Approach

Summary of Context Recognition Procedure(From VTT Publications 511)

Backlight level & font size adjustment

36/48

VTT Technical Research Center of Finland

• Context representation Fuzzy logic

• Context reasoning Naive Bayes, Markov Chain

• Service Fuzzy Control

Context OntologyFuzzification

37/48

VTT Technical Research Center of Finland (2)

• Sensor based contexts

– Bottom: high-level contexts

38/48

VTT Technical Research Center of Finland (3)

• Audio based context

– Bottom: high-level context

39/48

VTT Technical Research Center of Finland (4)

• Naive Bayes based classification

40/48

TEA Project

• High-level context recognition

– Method: Rule & SOM http://www.teco.edu/tea

41/48

• Life logging

• Context-aware computing

• Sensory data for activity recognition

• Life logging with mobile devices

• Summary

Agenda

Summary

• Key components for context-aware applications (life logging)

– Sensor technology to acquire various contextual information

– Intelligence technology to model complex contexts

– Agent technology to provide seamless services

• Future requirements

– Context modeling and cognitive technologies to provide users with more advanced services

43/48

DrWhatsOn (Laerhoven, 2003)

• Concept and user scenarios for context aware office PDA device

• Main focus in usability and user interface design

– Information fusion: Sensor data, device status, personal preference, schedule

– Peripheral attention: supporting appropriate information at appropriate time under prepared conditions

• Scenario for a context sensitive phone of Nokia

– A day of a Finland student Dude

44/48

DrWhatsOn Scenario (1)

• [Nokia’s DrWhatsOn Concept Video, Urpo Tuomela, Nokia Research Laboratories]

– Red mark: Abstracted information from sensor

45/48

DrWhatsOn Scenario (2)

46/48

DrWhatsOn Scenario (3)

47/48

DrWhatsOn Scenario (4)

48/48

Homework #1: Due 9/9

• Survey the state-of-the-art research on the life logging at KIST, Nokia, Microsoft and MIT (10 min presentation per each institute)

– KIST

– Nokia

– Microsoft

– MIT

49/48