July 25, 2010 SensorKDD 2010 1 Activity Recognition Using Cell Phone Accelerometers Jennifer Kwapisz, Gary Weiss, Samuel Moore Department of Computer &

July 25, 2010 SensorKDD 2010 1

Activity Recognition Activity Recognition Using Using

Cell Phone Cell Phone AccelerometersAccelerometers

Jennifer Kwapisz, Gary Weiss, Samuel MooreJennifer Kwapisz, Gary Weiss, Samuel Moore

Department of Computer & Info. ScienceDepartment of Computer & Info. Science

Fordham UniversityFordham University

July 25, 2010 2SensorKDD 2010

We are Interested in We are Interested in WISDMWISDM

WISDM: WIreless Sensor Data MiningWISDM: WIreless Sensor Data Mining Powerful portable wireless devices are Powerful portable wireless devices are

becoming common and are filled with becoming common and are filled with sensors sensors

Smart phones: Android phones, iPhoneSmart phones: Android phones, iPhone Music players: iPod TouchMusic players: iPod Touch

Sensors on smart phones include:Sensors on smart phones include: Microphone, camera, light sensor, proximity Microphone, camera, light sensor, proximity

sensor, temperature sensor, GPS, compass, sensor, temperature sensor, GPS, compass, accelerometeraccelerometer


Accelerometer-Based Accelerometer-Based Activity RecognitionActivity Recognition

The ProblemThe Problem: use accelerometer : use accelerometer data to determine a user’s activitydata to determine a user’s activity

Activities include:Activities include: Walking and joggingWalking and jogging Sitting and standingSitting and standing Ascending and descending stairsAscending and descending stairs More activities to be added in future More activities to be added in future

workwork


Applications of Activity Applications of Activity RecognitionRecognition

Health ApplicationsHealth Applications Generate activity profile to monitor overall Generate activity profile to monitor overall

type and quantity of activitytype and quantity of activity Parents can use it to monitor their childrenParents can use it to monitor their children Can be used to monitor the elderlyCan be used to monitor the elderly

Make the device context-sensitiveMake the device context-sensitive Cell phone sends all calls to voice mail when Cell phone sends all calls to voice mail when

joggingjogging Adjust music based on the activityAdjust music based on the activity

Broadcast (Facebook) your every activityBroadcast (Facebook) your every activity


Our WISDM PlatformOur WISDM Platform

Platform based on Android cell phonesPlatform based on Android cell phones Android is Google’s open source mobile Android is Google’s open source mobile

computing OScomputing OS Easy to program, free, will have a large Easy to program, free, will have a large

market sharemarket share Unlike most other work on activity Unlike most other work on activity

recognition:recognition: No specialized equipmentNo specialized equipment Single device naturally placed on body (in Single device naturally placed on body (in

pocket)pocket)


Our WISDM PlatformOur WISDM Platform

Current research was conducted off-lineCurrent research was conducted off-line Data was collected and later analyzed off-lineData was collected and later analyzed off-line

In future our platform will operate in In future our platform will operate in real-timereal-time

In June we released real-time sensor data In June we released real-time sensor data collection app to Android marketplacecollection app to Android marketplace Currently collects accelerometer and GPS Currently collects accelerometer and GPS

datadata


AccelerometersAccelerometers

Included in most smart phones & other devicesIncluded in most smart phones & other devices All Android phones, iPhones, iPod Touches, etc.All Android phones, iPhones, iPod Touches, etc. Tri-axial accelerometers that measure 3 dimensionsTri-axial accelerometers that measure 3 dimensions

Initially included for screen rotation and Initially included for screen rotation and advanced game playadvanced game play


Examples of Raw DataExamples of Raw Data

Next few slides show data for one user Next few slides show data for one user over a few seconds for various activitiesover a few seconds for various activities

Cell phone is in user’s pocketCell phone is in user’s pocket Earth’s gravity is registered as Earth’s gravity is registered as

accelerationacceleration Acceleration values relative to axes of Acceleration values relative to axes of

the device, not Earththe device, not Earth In theory we can correct this given that we In theory we can correct this given that we

can determine orientation of the devicecan determine orientation of the device


StandingStanding


SittingSitting


WalkingWalking


JoggingJogging


Descending StairsDescending Stairs


Ascending StairsAscending Stairs


Data Collection Data Collection ProcedureProcedure

User’s move through a specific courseUser’s move through a specific course Perform various activities for specific timesPerform various activities for specific times Data collected using Android phonesData collected using Android phones Activities labeled using our Android appActivities labeled using our Android app

Data collection procedure approved by Data collection procedure approved by Fordham Institutional Review Board Fordham Institutional Review Board (IRB)(IRB)

Collected data from 29 usersCollected data from 29 users


Data PreprocessingData Preprocessing Need to convert time series data into Need to convert time series data into

examplesexamples Use a 10 second example duration (i.e., Use a 10 second example duration (i.e.,

window)window) 3 acceleration values every 50 ms (600 total 3 acceleration values every 50 ms (600 total

values)values) Generate 43 total featuresGenerate 43 total features

Ave. acceleration each axis (3)Ave. acceleration each axis (3) Standard deviation each axis (3)Standard deviation each axis (3) Binned/histogram distribution for each axis (30)Binned/histogram distribution for each axis (30) Time between peaks (3)Time between peaks (3) Ave. resultant acceleration (1)Ave. resultant acceleration (1)


Final Data SetFinal Data SetID Walk Jog Up Down Sit Stand Total 1 74 15 13 25 17 7 151 2 48 15 30 20 0 0 113 3 62 58 25 23 13 9 190 4 65 57 25 22 6 8 183 5 65 54 25 25 77 27 273 6 62 54 16 19 11 8 170 7 61 55 13 11 9 4 153 8 57 54 12 13 0 0 136 9 31 59 27 23 13 10 163

10 62 52 20 12 16 9 171 11 64 55 13 12 8 9 161 12 36 63 0 0 8 6 113 13 60 62 24 15 0 0 161 14 62 0 7 8 15 10 102 15 61 32 18 18 9 8 146 16 65 61 24 20 0 8 178 17 70 0 15 15 7 7 114 18 66 59 20 20 0 0 165 19 69 66 41 15 0 0 191 20 31 62 16 15 4 3 131 21 54 62 15 16 12 9 168 22 33 61 25 10 0 0 129 23 30 5 8 10 7 0 60 24 62 0 23 21 8 15 129 25 67 64 21 16 8 7 183 26 85 52 0 0 14 17 168 27 84 70 24 21 11 13 223 28 32 19 26 22 8 15 122 29 65 55 19 18 8 14 179

Sum 1683 1321 545 465 289 223 4526 % 37.2 29.2 12.0 10.2 6.4 5.0 100


Data Mining StepData Mining Step

Utilized three WEKA learning Utilized three WEKA learning methods methods Decision Tree (J48)Decision Tree (J48) Logistic RegressionLogistic Regression Neural NetworkNeural Network

Results reported using 10-fold cross Results reported using 10-fold cross validationvalidation


Summary ResultsSummary Results


J48 Confusion MatrixJ48 Confusion Matrix

Predicted Class

Walk Jog Up Down Sit Stand

Actual

Class

Walk 1513 14 72 82 2 0

Jog 16 1275 16 12 1 1

Up 88 23 323 107 2 2

Down 99 13 92 258 1 2

Sit 4 0 2 3 270 3

Stand 4 1 2 7 1 208


ConclusionsConclusions

Able to identify activities with good Able to identify activities with good accuracyaccuracy Hard to differentiate between Hard to differentiate between

ascending and descending stairs. To ascending and descending stairs. To limited degree also looks like walking.limited degree also looks like walking.

Can accomplish this with a cell phone Can accomplish this with a cell phone placed naturally in pocketplaced naturally in pocket

Accomplished with simple features and Accomplished with simple features and standard data mining methodsstandard data mining methods


Related WorkRelated Work At least a dozen papers on activity recognition At least a dozen papers on activity recognition

using multiple sensors, mainly accelerometersusing multiple sensors, mainly accelerometers Typically studies only 10-20 usersTypically studies only 10-20 users

Activity recognition also done via computer Activity recognition also done via computer visionvision

Actigraphy uses devices to study movementActigraphy uses devices to study movement Used by psychologists to study sleep disorders, ADDUsed by psychologists to study sleep disorders, ADD

A few recent efforts use cell phonesA few recent efforts use cell phones Yang (2009) used Nokia N95 and 4 usersYang (2009) used Nokia N95 and 4 users Brezmes (2009) used Nokia N95 with real-time Brezmes (2009) used Nokia N95 with real-time

recognitionrecognition One model per user (requires labeled data from each user)One model per user (requires labeled data from each user)


Future WorkFuture Work

Add more activities and usersAdd more activities and users Add more sophisticated featuresAdd more sophisticated features Try time-series based learning Try time-series based learning

methodsmethods Generate results in real timeGenerate results in real time Deploy higher level applications: Deploy higher level applications:

activity profileractivity profiler


Other WISDM ResearchOther WISDM Research

Cell Phone-Based Biometric identificationCell Phone-Based Biometric identification11

Same accelerometer data and same generated Same accelerometer data and same generated features but added 7 users (36 in total)features but added 7 users (36 in total)

If we group all of the test examples from one cell If we group all of the test examples from one cell phone and apply majority voting, achieve 100% phone and apply majority voting, achieve 100% accuracyaccuracy

Can be used for security or automatic Can be used for security or automatic personalizationpersonalization

Interested in GPS spatio-temporal data miningInterested in GPS spatio-temporal data mining

1 Kwapisz, Weiss, and Moore, Cell-Phone Based Biometric Identification, Proceedings of the IEEE 4th International Conference on Biometrics: Theory, Applications, and Systems (BTAS-10), September 2010.

July 25, 2010 SensorKDD 2010 25

Thank YouThank You

Questions?Questions?

July 25, 2010 SensorKDD 2010 1 Activity Recognition Using Cell Phone Accelerometers Jennifer Kwapisz, Gary Weiss, Samuel Moore Department of Computer &

Documents

accelerometer data

data preprocessingneed

data collection procedureusers

time series data

data mining steputilized

final data setsensorkdd

devicesall android phones

android marketplacecurrently