In Copyright - Non-Commercial Use Permitted Rights ...47928/eth-47928-02.pdfDiss. ETH No. 22650 Topic Models for Activity Discovery in Daily Life A thesis submitted to attain the degree

Research Collection

Doctoral Thesis

Topic Models for Activity Discovery in Daily Life

Author(s): Seiter, Julia S.

Publication Date: 2015

Permanent Link: https://doi.org/10.3929/ethz-a-010483640

Rights / License: In Copyright - Non-Commercial Use Permitted

This page was generated automatically upon download from the ETH Zurich Research Collection. For moreinformation please consult the Terms of use.

ETH Library

https://doi.org/10.3929/ethz-a-010483640

http://rightsstatements.org/page/InC-NC/1.0/

https://www.research-collection.ethz.ch

https://www.research-collection.ethz.ch/terms-of-use

Diss. ETH No. 22650

Topic Models for ActivityDiscovery in Daily Life

A thesis submitted to attain the degree of

DOCTOR OF SCIENCES of ETH ZURICH

(Dr. sc. ETH Zurich)

presented by

Julia Stephanie Seiter

Dipl. -Ing., Karlsruhe Institute of Technology

born on 31.08.1985

citizen of Germany

accepted on the recommendation of

Prof. Dr. Gerhard Tröster, examinerProf. Dr. Ben Kröse, co-examiner

2015

Julia SeiterTopic Models for Activity Discovery in Daily LifeDiss. ETH No. 22650

First edition 2015Published by ETH Zurich, Switzerland

Printed byReprozentrale ETH

Copyright c© 2015 by Julia Seiter

All rights reserved. No part of this publication may be reproduced,stored in a retrieval system, or transmitted, in any form or by anymeans, electronic, mechanical, photocopying, recording, or otherwise,without the prior permission of the author.

iii

Acknowledgments

I am sincerely thankful to my supervisor Prof. Gerhard Tröster forgiving me the opportunity to work at the Wearable Computing Laband for offering his support and guidance throughout my PhD. Sincerethanks also to Prof. Ben Kröse for co-examining my PhD thesis.

I would particularly like to thank Prof. Oliver Amft for four years ofproductive collaboration, numerous fruitful discussions and for sup-porting and advising me during my PhD.

A big thank you goes to all current and former members of theWearable Computing Lab for the great atmosphere: Alberto, Alwin,Amir, Andreas H., Andreas M., Bernd, Bert, Burcu, Catherine, Chris-tian, Christoph, Daniel, Franz, Fredy, Giovanni, Giuseppe, Kilian, Lars,Long-Van, Luisa, Martin K., Martin W., Mirco, Matija, Michael, Niko,Paul, Rolf, Ruth, Sebastian F., Simon, Sinziana, Thomas H., Thomas K.,Tobias, Ulf, and Zack. It was a pleasure to work with you, havinglunch and ice cream breaks and going to yoga with some of you. Spe-cial thanks go to my office mates Alwin, Christian, Martin and Michael- it was a great time. Thank you Ruth for all the help with the admin-istrative tasks, for all the nice chats and joining me for Kondi.

I also want to thank Corina Schuster-Amft and the whole researchgroup of Reha-Rheinfelden, the therapists of the day care center NTZand Adrian Derungs of the ACTLab for the collaboration and com-mitment throughout the experiments that were conducted within theinterdisciplinary European research project iCareNet.

Thanks also to all the members of the iCareNet project for theinspiring atmosphere during project meetings.

I would like to thank Wei-Chen Chiu and Mario Fritz from theMax-Planck Institute for Informatics for organizing a great researchvisit and for collaborating with me on a project.

Thanks go to Dr. Lucian Macrea of the Pain Research Unit of theUniversity Hospital Zurich for the collaboration in our pain study.

Finally, I want to thank the most important people to me - myfamily - for always being there and supporting me in every aspect.Especially Sebastian, thank you for your great support and all themotivation, for your interest and patience and that you were alwaysthere for me.

Contents

Abstract ix

Zusammenfassung xiii

1. Introduction 11.1 The nature of human physical activity . . . . . . . . . . 21.2 The need for activity monitoring in daily life . . . . . . 41.3 Activity discovery from sensor data . . . . . . . . . . . 51.4 Unsupervised methods for activity discovery . . . . . . 81.5 Probabilistic topic models . . . . . . . . . . . . . . . . . 91.6 Objectives of this thesis . . . . . . . . . . . . . . . . . . . 141.7 Thesis outline and paper list . . . . . . . . . . . . . . . . 161.8 Additional publications . . . . . . . . . . . . . . . . . . 19Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

2. Thesis summary 312.1 Framework for topic model based activity discovery

from sensor data. . . . . . . . . . . . . . . . . . . . . . . 322.2 Unsupervised methods for patient monitoring in daily

life . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342.3 Parametric topic models for activity discovery . . . . . 402.4 Robustness of topic models . . . . . . . . . . . . . . . . 472.5 Nonparametric topic models for joint segmentation and

activity discovery . . . . . . . . . . . . . . . . . . . . . . 522.6 Recommendations and guidelines . . . . . . . . . . . . 582.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . 592.8 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . 612.9 Outlook . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

3. Activity discovery in rehabilitation patients 673.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 693.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 713.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 783.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . 80

vi

3.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . 863.6 Acknowledgments . . . . . . . . . . . . . . . . . . . . . 86Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

4. Evaluating daily life activity changes after pain surgery 914.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 924.2 System implementation . . . . . . . . . . . . . . . . . . 944.3 Evaluation study . . . . . . . . . . . . . . . . . . . . . . 974.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 974.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . 1014.6 Acknowledgments . . . . . . . . . . . . . . . . . . . . . 102Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

5. Parametric topic models for activity discovery 1055.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 1065.2 Related work . . . . . . . . . . . . . . . . . . . . . . . . . 1085.3 Activity discovery using topic models . . . . . . . . . . 1105.4 Dataset properties . . . . . . . . . . . . . . . . . . . . . . 1155.5 Evaluation datasets . . . . . . . . . . . . . . . . . . . . . 1175.6 Evaluation methodology . . . . . . . . . . . . . . . . . . 1195.7 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1215.8 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . 1265.9 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . 1295.10 Acknowledgment . . . . . . . . . . . . . . . . . . . . . . 130Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131

6. Topic models: the influence of hyperparameters 1356.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 1366.2 Methodology . . . . . . . . . . . . . . . . . . . . . . . . 1376.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1426.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . 1456.5 Acknowledgments . . . . . . . . . . . . . . . . . . . . . 145Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147

7. Robustness of parametric topic models 1497.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 1507.2 Related work . . . . . . . . . . . . . . . . . . . . . . . . . 1527.3 Simulation of daily routines . . . . . . . . . . . . . . . . 1537.4 Topic modeling approach of daily routines . . . . . . . 1557.5 Analysis methodology . . . . . . . . . . . . . . . . . . . 1557.6 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159

vii

7.7 Conclusion and outlook . . . . . . . . . . . . . . . . . . 1647.8 Acknowledgments . . . . . . . . . . . . . . . . . . . . . 164Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165

8. Joint segmentation and nonparametric activity discovery 1678.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 1688.2 Related work . . . . . . . . . . . . . . . . . . . . . . . . . 1708.3 Joint segmentation and discovery approach . . . . . . . 1718.4 Discovery framework . . . . . . . . . . . . . . . . . . . . 1758.5 Evaluation methodology . . . . . . . . . . . . . . . . . . 1798.6 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1838.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . 1878.8 Conclusion and future work . . . . . . . . . . . . . . . . 1908.9 Acknowledgment . . . . . . . . . . . . . . . . . . . . . . 191Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193

Glossary 197

Curriculum Vitae 199

ix

Abstract

The ability to identify daily life activities and routines from sensordata can enable better patient care and assisted living, amongst otherthings. Discovery algorithms extract activity patterns in sensor datawithout using supervision: They cluster sensor data by grouping sen-sor data considered to be similar to other sensor data. The capacity todetect basic activity primitives such as walking, sitting, and lying usingsensor data is well-established. However, discovering complex activi-ties and routines such as having lunch, working in the kitchen, and hygieneis more challenging as they involve multiple activity primitives.

Topic models are probabilistic discovery algorithms and widely-applied for discovery tasks in text mining and vision. Yet, their appli-cation in activity discovery is less established. This thesis analyses theapplicability of topic models for the discovery of complex activitiesand routines using sensor data. It outlines the potential, challenges,and limitations of topic models when applied in activity discovery.

In this thesis, we propose an unsupervised hierarchical approachfor activity discovery: First, we introduce context words that relateto basic activity primitives, body postures and object usage, and thatare easily deducible from sensors attached to the body and objects.Subsequently, we apply topic models to extract characteristic activitypatterns, e.g., a lunch pattern from context words such as sitting, movearm, use spoon, etc.

This thesis makes four contributions to the field: (1) We show theapplicability of topic models for the discovery of daily life activities.(2) We introduce new parametric topic models and present guidelinesfor optimal model and parameter selection as related to dataset proper-ties and activity discovery accuracy. (3) We define preliminaries regard-ing dataset properties in order to achieve stability in the topic model’sdiscovery accuracy and outline the performance bounds. (4) We sug-gest a nonparametric topic model that performs joint segmentationand activity discovery to optimize the discovery process. Finally, thethesis provides practical guidelines for high performing and robusttopic model applications.

Activity monitoring in daily life e.g., during rehabilitation, providesimportant information to doctors and therapists. Our topic model ap-proach detected a set of six activity routines involving kitchen work,fitness, rest, motor training, cognitive training and socializing from previ-

x

ously detected context words with an average of 78% accuracy. In thestudy, 11 rehabilitation patients were monitored using wearable mo-tion sensors in a day care center for up to ten full days during a periodof 2−3 months. Context words contained basic activity primitives andbody postures that proved to be specific for activity routines.

The complexity of a dataset influences the discovery accuracy oftopic models. We define complexity by dataset properties such as thespecificity s of activities, i.e., the dissimilarity of activities regardingtheir composition from context words. Parametric topic models out-performed K-means clustering by up to 17% accuracy for datasets withhigh complexity (e.g., s = 0.39). In contrast, topic models did not out-perform basic K-means clustering for datasets with low complexity(e.g., s = 0.99).

Further, performance bounds of topic models are of interest forsuccessful application in activity discovery. Inaccurate context worddetectors critically influence the activity discovery accuracy of topicmodels. In this context, the topic model latent Dirichlet allocation (LDA)performed most robustly: For LDA discovery accuracy was robust upto an error rate of 80%, while the accuracy of K-means and two otherparametric topic models dropped at 10% for a complex dataset (s =0.39).

Parametric topic models require an a priori selection for the numberof activity patterns to be discovered. They also require the segment sizefor segmenting the context word sequence. The thesis shows that thesegment size and the number of selected activity patterns influencethe discovery accuracy. Empirically, we found that parametric topicmodels perform best when a segment size close to the mean durationof activities is chosen. However, variability between the duration ofdifferent activities results in instability regarding accuracy. Moreover,optimal topic model parameters are often unknown for new datasets.

To overcome parameter dependency and to optimize the discoveryprocess regarding accuracy and stability, this thesis introduces a non-parametric topic model. The approach estimates the optimal numberof activity patterns based on the structure of the data and adapts thesegment size dynamically. The nonparametric approach outperformedparametric LDA with optimal parameters by 5% accuracy when com-plex datasets were used (s = 0.39).

The thesis concludes that topic models and particularly nonpara-metric topic models with dynamic segmentation of sensor data areadequate for the discovery of complex daily life activities. Our prior

xi

detection of basic context words from sensor data such as activity prim-itives, body postures, and object usage proved to be highly practicalfor developing successful topic model applications.

xiii

Zusammenfassung

Die Erkennung von Alltagsaktivitäten und -routinen aus Sensordatenfindet zahlreiche Anwendungen wie z.B. in der Patientenpflege oder im Be-reich betreutes Wohnen. Unüberwachte Lernalgorithmen erkennen Aktivi-tätsmuster in den Sensordaten basierend auf deren Ähnlichkeitsstrukturen.Die Erkennung einfacher Aktivitätsprimitiven aus Sensordaten wie z.B. sitzen,gehen, liegen ist bereits etabliert. Alltagsaktivitäten und -routinen wie z.B.Mittagspause, Küchenarbeit und Hygiene sind dagegen komplex, da sie ausmehreren Aktivitätsprimitiven zusammengesetzt sind.

Topic Models sind Wahrscheinlichkeitsmodelle und zählen zu denunüberwachten Lernalgorithmen. Topic Models sind weit verbreitet in derText- und Bildverarbeitung, jedoch bisher nicht in der Aktivitätserkennung.Diese Doktorarbeit untersucht die Anwendbarkeit von Topic Models für dieunüberwachte Erkennung von komplexen Aktivitäten und Routinen. Dabeilegen wir Potential, Herausforderungen und Einschränkungen von Topic Mo-dels bezüglich einer Anwendung in der Aktivitätserkennung offen.

Wir entwickeln einen hierarchischen Ansatz, der unüberwacht Aktivitäts-muster in Sensordaten mit Hilfe von Topic Models entdeckt. Dazu führen wirsogenannte Kontextwörter ein, die aus einfachen Aktivitätsprimitiven, Kör-perhaltungen und Objektnutzung bestehen. Diese Kontextwörter werden imersten Schritt von Sensoren am Körper oder an Objekten detektiert. Nachfol-gend verwenden wir Topic Models um Aktivitätsmuster (z.B. das Muster fürMittagessen) aus Kontextwörtern wie z.B. sitzen, Arm bewegen, Glas benutzen zuextrahieren.

Diese Dissertation setzt vier Ziele um: (1) Wir zeigen die Anwendbarkeitvon Topic Models zur Erkennung von Alltagsaktivitäten in der Praxis. (2) Wirführen neue parametrische Topic Models für die Aktivitätserkennung ein undpräsentieren Empfehlungen zur Modell- und Parameterwahl bezogen aufDatensatzeigenschaften und die Aktivitätserkennungsrate. (3) Wir definierenVoraussetzungen in Bezug auf Datensatzeigenschaften für eine stabile Erken-nung von Aktivitäten und zeigen Anwendungsgrenzen von Topic Models auf.(4) Wir führen einen neuen, nichtparametrischen Topic Model Ansatz ein, derSegmentierung und Mustererkennung zur Optimierung der Aktivitätserken-nung gleichzeitig umsetzt. Schliesslich präsentiert die Dissertation zusammen-fassende Empfehlungen für eine leistungsfähige und robuste Anwendung vonTopic Models in der Praxis.

Aktivitätsmonitoring ambulanter Patienten im Alltag wie z.B. währendder Rehabilitation liefern wichtige Informationen für Ärzte und Therapeuten.Unser Topic Model Ansatz erwies sich als geeignet, sechs Alltagsroutinen(Küchenarbeit, Fitness, Ruhepause, Motortraining, kognitives Training, Sozialakti-vitäten) mit einer Genauigkeit von durchschnittlich 78% aus Aktivitätsprimi-

xiv

tiven (Arm bewegen, gehen) und Körperhaltungen (sitzen, Arm angewinkelt) zuerkennen. In der Studie beobachteten wir elf Rehabilitationspatienten in einemTageszentrum während bis zu zehn Tagen in einem Zeitraum von 2−3 Monatenmit Hilfe von tragbaren Bewegungssensoren. Die Alltagsroutinen erwiesensich als spezifisch in Bezug auf deren Zusammensetzung aus Kontextwörtern,die einfache Aktivitätsprimitiven und Körperhaltungen umfassten.

Die Eigenschaften und insbesondere die Komplexität eines Daten-satzes beeinflussen die Genauigkeit der Aktivitätserkennung verschiedenerparametrischer Topic Models. Komplexität definieren wir durch Daten-satzeigenschaften wie z.B. die Spezifizität s, d.h. wie verschieden die Aktiv-itäten bezüglich deren Zusammensetzung aus Kontextwörtern sind. Bei Daten-sätzen mit hoher Komplexität zeigten Topic Models einen Mehrwert gegenüberdem einfachen Clusterverfahren K-means: Auf komplexen Datensätzen (z.B.s = 0.39) erreichten Topic Models eine bis zu 17% höhere Erkennungsrate alsK-means. Bei Datensätzen mit geringer Komplexität dagegen (z.B. s = 0.99)erzielten Topic Models eine ähnliche Erkennungsrate wie K-means.

Kenntnis über die Einschränkungen von Topic Models sind eine Vor-aussetzung für deren erfolgreiche Anwendung in der Aktivitätserkennung.Fehlerhafte Kontextwortdetektoren wirken sich kritisch auf die Erken-nungsrate der Topic Models aus. In diesem Zusammenhang erwies sich LatentDirichlet allocation (LDA) am robustesten: Während K-means und zwei andereparametrische Topic Models bei komplexen Datensätzen (s = 0.39) nur bis zueiner Fehlerrate von 10% eine robuste Erkennungsrate lieferten, verhielt sichLDA bis zu einer Fehlerrate von 80% robust.

Parametrische Topic Models benötigen eine a priori Wahl der Anzahlzu erkennender Aktivitätsmuster, sowie eine feste Segmentierung der Kon-textwortsequenzen. Die Doktorarbeit zeigt, dass die Parameter Segmentlängeund Anzahl Aktivitätsmuster bei parametrischen Topic Models die Erken-nungsrate beeinflussen. Empirisch evaluierten wir, dass sich parametrischeTopic Models am genauesten erweisen, wenn die Segmentlänge nahe der durch-schnittlichen Dauer der Aktivitäten gewählt wird. Jedoch führen Unterschiedein der Dauer verschiedener Aktivitäten zu Instabilität in der Erkennungsrate.Ausserdem sind optimale Parameter für neue Datensätze in der Praxis oftunbekannt.

Um das Problem der Parameterabhängigkeit zu lösen und um den TopicModel Ansatz bezüglich Erkennungsrate und Stabilität zu optimieren, ent-wickeln wir im Rahmen dieser Arbeit einen nichtparametrischen Topic ModelAnsatz, der die optimale Anzahl Aktivitätsmuster basierend auf der Strukturder Daten schätzt. Des Weiteren passt der Ansatz die Segmentlänge währendder Mustererkennung auf die Daten an. Unser neuer Topic Model Ansatz führtezu einer 5% höheren Aktivitätserkennungsrate auf einem komplexen Daten-satz (s = 0.39) gegenüber dem parametrischen Topic Model LDA mit optimalerParameterwahl.

Das Fazit dieser Doktorarbeit ist, dass Topic Models und insbesondere nicht-

xv

parametrische Topic Models mit automatischer Segmentierung geeignet sind fürdie unüberwachte Aktivitätserkennung aus Sensordaten im Alltag. Besondersbewährt hat sich unser hierarchischer Ansatz, der vor der Topic Model Anwen-dung einfache Kontextwörter wie Aktivitätsprimitiven, Körperhaltungen undObjektbenutzung in den Sensordaten detektiert.

1Introduction

This chapter argues for the relevance of activity discovery in daily life, andreviews state-of-the-art approaches for unsupervised activity discovery fromcontext sensor data. Additionally, the objectives of the thesis are highlightedand an outline of the overall thesis is presented.

2 Chapter 1: Introduction

1.1 The nature of human physical activity

Human physical activity is driven by our personal motivation andgoals [1] and emerges in motions of the head, torso, and limbs.

Frequently, physical activities are categorized to differentiate be-tween their degree of abstraction, e.g., in the fields of activity theory [1],computer vision [2–4], and ubiquitous computing [5–9]. However, nodistinct, universal terminology or definitions for physical activity cat-egories exist. Therefore, the concrete interpretation of these terms mayvary and depend on the application.

In this thesis we group physical activities into five categories anduse the following terminology and definitions:

1. Gestures are the most elementary movements of a person’s torsoor limbs, e.g., raising an arm, flexing a leg.

2. Actions are composed of multiple gestures, for instance walking,waving and queuing.

3. Interactions can be actions or gestures where objects and/or otherpeople are involved. For example, grabbing a vacuum cleaner is aninteraction between a human and object while the act of discussingis an interaction between two humans.

4. High-level activities are composed of multiple gestures, actions,and interactions and may also include body postures such as(sitting, standing). For example, the high-level activity cleaning-upcomprises actions such as walking; postures, like standing; andinteractions for instance grabbing a vacuum cleaner.

5. Activity routines are a specific group of high-level activities thata human performs periodically for instance, daily or weekly ac-cording to individual habits. For example, the activity routinelunch is typically performed once a day, while commuting usuallyoccurs twice in a working day.

Figure 1.1 illustrates the dependencies between gestures, actions,interaction, high-level activities, and activity routines.

In literature, the hierarchical structure in human behavior and ac-tivities is often emphasized [1, 10, 11]. Similarly, we divide the fiveactivity categories into two hierarchical levels (see Figure 1.1): low-level activity primitives and high-level activity composites.

1.1. The nature of human physical activity 3

gest

ures

e.

g., a

rm s

tretc

h ac

tions

e.

g., w

alki

ng

inte

ract

ions

e.

g., g

rabb

ing

vacu

um c

lean

er

high

-leve

l act

iviti

es

e.g.

, cle

anin

g - up

activ

ity ro

utin

es

e.g.

, lun

ch

activ

ity c

ompo

site

s

activ

ity p

rimiti

ves

perio

dic

Figu

re1.

1:W

epa

rtit

ion

hum

anac

tivi

ties

into

5ca

tego

ries

depe

ndin

gon

thei

rde

gree

ofab

stra

ctio

n.G

estu

res

desc

ribe

elem

enta

rym

ovem

ents

for

exam

ple

arm

stre

tch,

acti

ons,

such

asw

alki

ng,a

reco

mpo

sed

ofm

ulti

ple

gest

ures

.Int

erac

tion

sde

fine

gest

ures

and

acti

ons

whe

reob

ject

sor

othe

rhum

ans

are

also

invo

lved

,for

inst

ance

grab

bing

ava

cuum

clea

ner,

the

act

ofdi

scus

sing

.Hig

h-le

vela

ctiv

itie

sar

eco

mpo

sed

ofse

vera

lges

ture

s,ac

tion

san

din

tera

ctio

ns,

like

clea

ning

-up.

Act

ivit

yro

utin

essp

ecif

yhi

gh-l

evel

acti

viti

esth

athu

man

spe

rfor

min

ape

riod

icm

anne

rsuc

has

havi

nglu

nch.

We

grou

phu

man

acti

viti

esin

two

hier

arch

ical

leve

lsde

pend

ing

onth

eir

mov

emen

tcom

plex

ity.

Act

ivit

yco

mpo

site

s,w

hich

are

high

-leve

lact

iviti

esan

dac

tivity

rout

ines

,are

com

pose

dof

low

-lev

elac

tivi

typr

imit

ives

like

gest

ures

,act

ions

and

inte

ract

ions

.


The lower level contains the activity primitives which are definedby gestures, actions, and interactions and typically last for seconds orminutes, such as walking.

In everyday life, humans execute a number of single activity prim-itives one after another or in parallel. A characteristic set of activityprimitives can be combined to form an activity composite at the toplevel. Such activity composites include high-level activities and acti-vity routines (see Figure 1.1) and typically last in the range of minutesor hours. For example, the activity routine lunch, which is an activitycomposite, could be composed of the activity primitives eating, cutting,drinking from a glass, sitting and may last for 30 minutes.

Depending on the desired abstraction level, human activity can bedescribed with different complexity levels; the activity composite orthe activity primitive level.

1.2 The need for activity monitoring in daily life

Human behavior is studied in many research fields including environ-mental psychology, behavioral science and social psychology [12–14].The behavior of a person is determined by external and personal fac-tors such as the environment, interactions with the environment andhuman activities. In particular, activity routines such as office work,having lunch, commuting, and fitness describe characteristic patternsin our lifestyle and are an important factor to specify behavior. Thus,understanding and inferring human activities and activity routines indaily life is an important step towards behavior modeling and is usefulfor a number of applications including personal behavior monitoring,elderly care, and remote patient care.

Monitoring activities in daily life could generate logbooks that tellpeople about their behavior and thus help them to remodel problemareas. Such logbooks may help to disclose, track, and analyze personallifestyles [15]. For example, revealing how often, for how long, andwhether activities such as hygiene, having lunch, resting, and shoppingfor groceries are performed might be particularly useful for the elderlyand patients [16–18].

Our aging society introduces several difficulties into elderly care,one of them being the increasing number of elderly people to monitoron a regular basis. The number of places in nursing homes is limited,and care is expensive. Furthermore, many elderly people prefer to livein their familiar environment for as long as possible. Therefore, smart

1.3. Activity discovery from sensor data 5

homes that are equipped with sensor technology are sometimes used tomonitor inhabitants [19–22] and alert caregivers and family membersin an emergency situation.

Existing systems detect emergency events such as falls and criti-cal vital signs [23–26]. Changes in duration and frequency of activityroutines such as rest, kitchen work and daily exercise may also be relatedto health. Detecting activity routines and changes within them couldbe a more robust indicator for long-term changes in a person’s healthcondition [17, 18, 27].

Similarly, remote patient care is useful for tracking a patient’s healthcondition after surgery or monitoring the rehabilitation process athome. Typically, doctors and therapists use clinical assessments suchas the Fugl-Meyer test that rates the degree of physical impairmentafter stroke [28, 29]. However, such clinical assessments can only beapplied during clinical consultations and do not provide insights intopatients’ behavior in daily life.

Daily questionnaires are often used to assess patients’ self-perceived activity/pain levels, degree of functional disability, and qual-ity of life. Yet, questionnaires are subjective and may not provide validmeasurements to clinicians and therapists [30–32]. Hence, additionalinformation about a patient’s daily activities outside the clinic (e.g.,lunch, rest, kitchen work) could support doctors and therapists in opti-mizing medical after-treatment and the rehabilitation process.

1.3 Activity discovery from sensor data

The increased availability of wearable and ambient sensors in our dailyenvironment provides a variety of sensor data that characterizes ourcontext. Context is any information that can be used to describe the sit-uation of a person, place or object [33]. Different sensor modalities existto measure context including body motion, e.g., accelerometers [34],location, for example GPS [35], interaction with objects, for instancemotion detectors [36], and sound, e.g., microphones [37]. Wearableand ambient sensors are valuable tools for inferring activities. Thus,machine learning algorithms are often applied to detect activities fromsensor data.

In supervised machine learning, activity recognition algorithmsare trained to identify characteristic patterns in sensor data that corre-spond to our activities in daily life [6, 9, 38–40]. Supervised machinelearning methods require accurate and comprehensive groundtruth


Commuting

Office work

Lunch

8:00 10:00 12:00 14:00 16:00 18:00 t

duration

Desk activityUsing phone

DrinkingEating

Queuing

Walking

Using bus

CafeteriaStreet

OfficeKeyboardDoorSpoon

Act

ivit

y co

mp

osi

tes

Co

nte

xt w

ord

s

Act

ivit

y p

rim

itiv

es

Loca

tio

nO

bje

ctA

ctiv

ity

rou

tin

es

10:00 12:00 14:00 16:00 18:00 t

.. .. . .

...

8:00

Activity topics1 2 3

Desk activityDrinkingEatingUsing busCafeteriaStreetOfficeKeyboardSpoon

Pro

ba

bili

ty

Topic modeling

a)

b)

Hierarchical abstraction of activity composites

12

3 13

0

1

Lunch Office workOffice work Com-muting

Com-muting

c)

Discovered activity patterns Occurrence probability of activity patterns over the course of the day

Figure 1.2: (a) Illustration of hierarchical abstraction we use for acti-vity composites discovery. The example shows three activity composites,i.e., activity routines and their composition from context words. (b,c) Il-lustration of TM concept for activity discovery from context words.(b) Discovered activity patterns, which we call activity topics, andtheir composition from context words. Dark gray shades in the matrixindicate high occurrence probability for a context word. (c) Inferred oc-currence probability of activity topics over the course of the day usingTMs. Occurrence probabilities show high correlations to groundtruthactivity routines lunch, office work, and commuting. For example, the oc-currence probability of activity topic 2 (pattern in (b), second column)is high during lunch time and likely involves context words eating,cafeteria, and spoon.

1.3. Activity discovery from sensor data 7

annotations of sensor data to build reliable activity recognition mo-dels. The activity groundtruth must often be annotated by the user.Such data annotations are time-consuming and distract users in dailylife [41]. Especially long-term annotations during weeks or monthsfor training algorithms for daily life activity recognition is laborious.Moreover, users tend to forget to annotate which often leads to in-complete and inaccurate data annotations [42, 43]. Thus, supervisedmachine learning methods seem to be impractical for activity monitor-ing in daily life.

Unsupervised machine learning methods, also referred to as dis-covery methods, automatically reveal activity patterns in sensor databased on the structure of the data [21, 41, 44–48]. In contrast to su-pervised methods, unsupervised methods do not require annotatedtraining data to create the model. Consequently, discovery appears tobe more suitable for daily life activity monitoring. Therefore, this thesisfocuses on unsupervised methods for activity discovery from sensordata.

The ability to detect activity primitives from sensor data, such aswalking, sitting is well-established [5, 34, 49]. However, discovering ac-tivity composites from sensor data is more challenging [7, 50]: Activitycomposites involve multiple activity primitives that humans also per-form concurrently. Furthermore, activity composites may vary in theircomposition of activity primitives over time (see Section 1.1).

Due to the hierarchical structure in the nature of human activities(see Figure 1.1), hierarchical approaches were frequently used to detectactivity composites from sensor data [6, 40, 51]. Activity compositeswere detected from previously recognized activity primitives such aswalking, eating [6, 40, 51]. Similarly, activity composite discovery in thisthesis is based on a hierarchical abstraction as depicted in Figure 1.2.

We introduced the group of context words to comprehensively de-scribe the context of a person. Such context words include activityprimitives (walking, eating), location (street, office) and object use infor-mation (cup, keyboard). The detection of such context words from sensordata is well-established [34, 49]. Subsequently, we applied discoverymethods to reveal characteristic patterns in the context words. Suchdiscovered patterns correspond to activity composites.

In this thesis, we focus on the discovery of high-level activities andactivity routines from context sensor data. Activity composites includehigh-level activities and activity routines such as those introduced in


Section 1.1. For simplification, we use the term activity to describe allactivity composites in the following sections.

1.4 Unsupervised methods for activity discovery

Unsupervised methods discover characteristic activity patterns auto-matically from the structure of the data without using data annotations.Among unsupervised methods for activity discovery two categoriesexist: deterministic and probabilistic approaches [52].

Deterministic methods include logic-based and clustering baseddiscovery approaches, in which activities are detected based on deter-ministic rules and measures [21, 41, 44, 48]. In contrast, probabilisticmethods such as Bayesian models [51, 53, 54] consider probabilities inactivity modeling and therefore allow for the capturing of uncertaintyin human behavior.

Within deterministic and probabilistic approaches, either his-togram based methods [47, 50, 51, 54] or sequence based meth-ods [21, 41, 44] are applied.

Sequential pattern mining approaches discover activities such aslunch based on characteristic sequential patterns in context words suchas walking - using toilet - washing hands. Aztiria et al. extracted behaviorin daily life from smart home sensor data using a sequential patternmining algorithm [21]. Clarkson et al. applied time-series clusteringand hierarchical HMMs to audio and video data for unsupervisedactivity inference [53]. Sequential pattern mining algorithms turnedout to be especially valuable when fixed recurring sequential patternsexisted in data [21, 41, 44]. Thus, recurring sequence patterns are a pre-liminary requirement for successful sequential pattern mining. Sincecontext words do not often emerge in a fixed sequence for any activity,sequential data mining is limited in its application.

Histogram based methods infer activities, like lunch, from the statis-tics of data [47, 50, 51, 54]. Thus, the sequence of data is not consideredin activity modeling. For example, Gu et al. used statistics over objectuse (filter, coffee) to discover activity patterns such as making coffee [47].Garcia-Ceja and Brena modeled activities, like shopping, as a prob-abilistic distribution of activity primitives that were represented asa histogram [50]. Similarly, probabilistic topic models are histogrambased and assume that the occurrence of human activity follows prob-abilistic distributions [55]. The group of topic models is discussed indetail in the next section.

1.5. Probabilistic topic models 9

1.5 Probabilistic topic models

1.5.1 Definition

Probabilistic topic models were originally introduced in text mining todiscover relevant topics in a documents such as economy, astronomy, andfinance [55]. In activity discovery, the topic model (TM) was applied todiscover characteristic activity patterns in sensor data [51].

Fig. 1.2(a) shows a subset of groundtruth activities during the day,these are commuting, lunch, office work, and their composition fromcontext words. For example, the activity commuting is composed of thecontext words using bus and street, the activity lunch contains eating andcafeteria. TMs are applied to discover such activity patterns in a set ofcontext words. We call these activity patterns activity topics. The conceptof TMs for activity discovery is illustrated in Fig. 1.2(b). ProbabilisticTMs model activity topics by assuming a distribution over a set ofdistributions:

• For each discovered activity topic z, the TM estimates a proba-bilistic context word distribution p(w|z). The distribution p(w|z)characterizes how likely a context word w occurs in the activitytopic z as depicted in the matrix of Fig. 1.2(b). For example, ac-tivity topic 2 likely contains the context words spoon, eating, andcafeteria but less likely street.

• The TM assumes an activity topic distribution p(z|s) to model theoccurrence probability of each activity topic z over the day. Thedistribution p(z|s) shows how active an activity topic is at a partic-ular time s during the day, as depicted in Fig. 1.2(c). For example,activity topic 2 is highly active during lunch time suggestingthat activity topic 2 models the activity lunch. Similarly, activitytopic 3 occurs in the morning and in the evening, as does theactivity commuting.

TMs are Bayesian models. Considering T activity topics, the prob-ability p(w|s) of context word w in the time segment s can be computedby:

p(w|s) =

T∑z=1

p(w|z)p(z|s). (1.1)

In practice, context words w in a time segment s can be observed as theyconsist of encoded sensor data (cafeteria, spoon). Thus, the probability


p(w|s) (see (1.1)) is deducible by calculating statistics of context words.In contrast, activity topics z and their distributions p(w|z) and p(z|s)(see (1.1)) are unknown and need to be estimated by the TM. The basicsteps we apply for TM based activity discovery from sensor data canbe found in Figure 1.3:

Context word encoding: To model activity topics and their distri-butions p(z|s) and p(w|z), a context word vocabulary is required. Thecontext word vocabulary contains context words for instance cafeteria,spoon which are likely to describe activities (lunch). Thus, we encodesensor data in a context word vocabulary (Figure 1.3).

Segmentation: TMs estimate activity topics (topic 2) from statisticsof context words (spoon, plate) during a particular time. Thus, we seg-ment the context word sequence, for example segments of 30 minutes,to form context word statistics p(w|s) for each segment (Figure 1.3).

Topic model: The TM decomposes the given probability p(w|s)(see (1.1)) to obtain the activity topic distribution p(z|s) (Fig. 1.2(c)) andthe context word distributions p(w|z) (Fig. 1.2 (b)). Activity topics corre-spond to the activity patterns in each segment. For example, activitytopic 1 corresponds to the activity office work as illustrated in Fig. 1.3.

In this thesis, we introduce and evaluate a number of BayesianTMs that use additional prior probabilities to adapt the distribu-tions to a concrete modeling problem. For example, latent Dirichletallocation (LDA) assumes that activity topics in a segment p(z|s) areDirichlet distributed [56]. Thus, LDA adds a Dirichlet prior p(θs|α) withthe Dirichlet hyperparameter α to the activity topic distribution p(z|θs).The context word distribution p(w|z) is assumed to be a Multinomialwith hyperparameter β resulting in p(w|z, β).

In topic modeling, the most likely activity topic distribution persegment, the topic-specific context word distributions and the hyper-parameters (e.g., for LDA α and β) are extracted in an optimizationprocess by maximizing the likelihood of the dataset. For example inLDA, the likelihoodLof S segments, each containing Ws context words,and considering T activity topics is described by:

L(α, β) =

S∏s=1

∫p(θs|α)

Ws∏n=1

T∑z=1

p(zsn|θs)p(wsn|zsn, β)

dθs (1.2)

This optimization problem is often solved by an expectation-maximization (EM) algorithm as detailed in [55, 57]. The selection of


Segments:

Activity topics:

Context word statistics:

TOPIC MODEL

CONTEXT WORD ENCODING

Context word sequence::

SEGMENTATION

Sensor data

a b c d e f

a: d:

e:

f: g: b:

a a e b a … d c c e c …

a a e b a … d c c e c …

a b c d e f …

topic 1 topic 2 office work lunch

…

c: Vocabulary

Figure 1.3: Approach for activity discovery from sensor data usingTMs: Sensor data is encoded in context words using a context vocab-ulary, for example using computer, mouse, spoon, plate, sitting, standing,walking. Then, the context word sequences are segmented and statisticsare formed. Based on context word statistics, the TM reveals charac-teristic activity topics (e.g., topic 2) that correspond to activities (e.g.,lunch).

priors (e.g., θ) and hyperparameters (α,β, etc.) varies for different TMsaccording to the underlying distributions as detailed in Section 2.3.

Generally, a distinction is made between parametric and nonpara-metric TMs. Parametric TMs assume that the number of activity topicsin a dataset is fixed and known prior to modeling, for instance 4 activitytopics to model the activities having breakfast, office work, having lunch,having dinner [56, 58, 59]. Depending on the person, a number of addi-tional activities are performed, such as fitness, shopping. As the numberof activity topics is not necessarily known prior to modeling, nonpara-metric TMs estimate the number of activity topics automatically fromthe structure of the data [60–62].


1.5.2 Applications of topic models

Text processing

TMs originate from text processing [55, 63]. Several TMs have been sug-gested to extract the topics (e.g., finance, economy) and their distributionin a document from the document’s words (asset, stock, economy, money,etc.). For example, the TM infers that 70% of the words in a documentdiscuss the topic finance and 30% the topic economics.

In LDA the topics that occur in a document were assumed to beDirichlet distributed [56]. Further, the LDA model was based on theassumptions that the words as well as the topics in a document occurindependently.

In practice, the topics economics and finance more likely occur to-gether in a document than economics and astronomy. Thus, the correlatedtopic model (CTM) modifies LDA by modeling correlations between top-ics in the topic distribution [58].

Wallach et al. introduced a parametric TM that considered the se-quence of two words in a document [64]. For example, the topic of theword toxic depends on the successive word; while toxic assets describethe topic finance, a toxic chemical refers to the topic chemistry.

The dynamic topic model (DTM) assumed that language changes overtime and therefore that the word distributions of a topic also change.For example, during a financial crisis the word toxic becomes morelikely to describe the topic finance. Thus, the DTM considers when thedocuments were written [59].

LDA, CTM and DTM are parametric TMs that assume a finite num-ber of topics. In contrast, nonparametric TMs model an infinite numberof topics and their distributions over words. The hierarchical Dirichletprocess (HDP) and extensions to HDP were suggested for extracting theoptimal number of topics based on the structure of the data [60–62].

Vision

TMs were also successfully applied for object and activity discoveryin image and video processing. Here, pixels were encoded in a color-codebook and the TM was applied to discover activities and objectpatterns based on the color values.

In vision, both parametric and nonparametric TMs were frequentlyapplied. Parametric latent Dirichlet allocation and extensions wereused to detect objects and categories from images [65–69]. The paramet-


ric CTM was deployed to infer human actions from video data [70, 71].Bettadapura et al. leveraged sequences of encoded pixels for activitydiscovery in video data [72].

Furthermore, nonparametric TMs were applied for object and acti-vity discovery in images and videos [73, 74].

Activity discovery

In activity discovery, parametric latent Dirichlet allocation (LDA) wasdeployed to reveal activity routines from activity primitives (sitting,walking) that were previously detected from acceleration sensor data[51, 75]. Similarly, LDA was applied to infer daily routine patterns us-ing location information [76, 77]. Castanedo applied LDA to infer roomoccupancy patterns in a smart home from motion detectors [78].

1.5.3 Potential and challenges of topic models in activity discovery

While a variety of TMs including parametric and nonparametric TMswere used for unsupervised discovery in text processing and vision,TMs were less established in activity discovery. Existing approacheswere based on LDA and showed promising results on a limited numberof datasets. Yet, the performance of TMs in comparison to other discov-ery methods, e.g., clustering, was unclear. Furthermore, extensions toLDA could leverage dependencies between activities or characteristiccontext word sequences similar to the methods applied in text pro-cessing and vision. TM based activity discovery requires context wordencoding and the segmentation of context word sequences. These stepsmight be challenging and, hence, require optimization within the ac-tivity discovery process (see Figure 1.3). Therefore, the applicationof TMs for activity discovery, including context word encoding andsegmentation as well as the potential and limitations of a variety ofparametric and nonparametric TMs, are investigated in this thesis.

Recently, TMs based on LDA have also been applied to infer be-havior patterns of the elderly [79] and activity patterns in healthy peo-ple [80–82]. This thesis investigates the feasibility of TMs for activitymonitoring in rehabilitation patients.

Moreover, recent research leveraged context word sequence infor-mation to increase activity discovery performance of TMs [54, 83].These approaches required sequence information to actually exist


among context words, limiting their application. We introduce asequence-based TM, the n− gram topic model (NTM), which overcomesthis limitation.

Rieping et al. introduced a LDA − Gaussian and a LDA − Poissonmodel where similar context words were assigned to the same activitytopic [79]. While their TM approach considered context word similar-ity in order to optimize a parametric activity discovery process, weinvestigated the use of semantic similarities between context words tooptimize nonparametric activity discovery.

1.6 Objectives of this thesis

This thesis aims to introduce and evaluate methods for unsupervisedactivity discovery in daily life using sensor data without data annota-tions. In particular, we focus on the group of TMs as LDA showedpromising results in activity discovery. To provide a better under-standing of high performing and robust TM applications in activitydiscovery, we assess their potential, challenges, robustness and limi-tations. In this thesis, we introduce a selection of promising TM ap-proaches and evaluate the TMs by using a number of publicly availabledatasets recorded in different scenarios. These datasets feature a va-riety of dataset properties, such as specificity of activities in terms ofcontext word composition. Furthermore, the application of unsuper-vised activity discovery methods in patient monitoring is investigated.In two patient studies we monitored patients during several weeks indaily life using wearable sensors. Specifically, the following objectiveswere addressed.

1.6.1 Unsupervised methods of patient monitoring in daily life

Outpatient monitoring in daily life, e.g., after surgery or during re-habilitation, often relies on subjective patient feedback obtained fromquestionnaires. In this thesis, we introduce and evaluate unsupervisedmethods and wearable sensor systems to automatically infer activityinformation in daily life.

In a first study, we show that TMs can be used to discover activities(e.g., fitness) in hemiparetic rehabilitation patients through motionsensor data. Consequently, 11 hemiparetic rehabilitation patients weremonitored with wearable motion sensors for up to 10 days in a day carerehabilitation center. Distinct context words that are specific in their

1.6. Objectives of this thesis 15

description of activities are crucial for successful TM based discoveryfrom sensor data. Thus, we introduce and compare the performance oftwo context word vocabularies, that are used to retrieve context wordsfrom sensor data without data annotations.

In a second study, we show a proof of concept that a smartphonesystem and unsupervised data analysis techniques can detect a changein patient activity after pain intervention. We measured the potential ofa smartphone system combined with unsupervised methods to serveas outcome measure after a pain therapy.

1.6.2 Parametric topic models for activity discovery

In activity discovery, the parametric LDA was successfully applied.LDA was established based on a number of assumptions, e.g., inde-pendence between context words and independence between activi-ties. Thus, this work introduces and investigates alternative TMs thatrelax LDA assumptions for activity discovery. Further, we compare theactivity discovery performance of TMs in relation to basic clustering.

For parametric TMs, the segment size for the segmentation of thecontext word streams and the number of activity topics (Section 1.5)has to be specified prior to modeling. Moreover, we expect that datasetproperties influence activity discovery. Hence, we investigated the in-fluence of TM parameter selection and dataset properties on discov-ery performance using three datasets with varying dataset properties.Eventually, we established recommendations for TM and parameterselection related to dataset properties.

1.6.3 Robustness of topic models

TMs yielded promising discovery accuracy in activity discovery, how-ever their robustness and limitations were not evaluated. In practice,experiment design and the selection of the discovery method influ-ences TM stability and performance.

This thesis provides an in depth analysis of TM performance stabil-ity and performance bounds, a requirement for successful TM appli-cations in activity discovery. We point out stability issues in discoveryaccuracy regarding dataset properties. For TM based activity discovery,context words need to be detected from sensor data. Here, the impactof noisy context word detectors is evaluated to assess the performancebounds of different TMs.


1.6.4 Nonparametric topic models for joint segmentation and activitydiscovery

This thesis reveals that the a priori selection of TM parameters segmentsize and activity topic count critically influence the accuracy and stabil-ity of the activity discovery process. Nonparametric TMs can estimatethe number of topics automatically from the structure of the data andwere successfully applied in text mining [60, 62] and vision [73, 84].

This work introduces a joint segmentation and activity discoveryapproach based on nonparametric TMs and segmentation priors. Thesegmentation prior considers semantic and temporal information ofcontext words. For example, the activity lunch would likely contain thecontext words spoon and plate, whereas the activity office work wouldlikely contain the word computer. Therefore, we expect the semanticrepresentation of the context word spoon and plate to be more similarthan spoon and computer.

Our approach combines the segmentation step in Figure 1.3 withthe activity topic discovery step in order to jointly optimize the seg-mentation and the activity discovery process. We show that our non-parametric approach performs optimal topic count estimations andsegmentation automatically based on the structure of the data.

1.7 Thesis outline and paper list

This thesis comprises of seven scientific publications (Chapters 2-8).Chapter 2 includes a summary of the thesis contributions and Chap-ters 3-8 address the objectives introduced above. Figure 1.4 illustratesthe thesis objectives as detailed in Section 1.6 along with the thesiscontributions.

The publications are arranged according to the objectives of thethesis. Table 1.1 lists the chapter organization and the publications in-cluded. Chapter 2 summarizes the contributions and conclusions of thethesis and provides an outlook on future research challenges. Chapters3 and 4 evaluate unsupervised methods for daily life patient monitor-ing in two studies: (a) TMs are investigated for activity discovery inrehabilitation patients (Chapter 3). (b) We use a smartphone systemand unsupervised data analysis techniques for change detection inthe activity of pain patients after surgery (Chapter 4). Chapters 5 and6 present the influence of TM parameters on discovery performancefor parametric TMs. While Chapter 5 evaluates in addition the im-

1.7. Thesis outline and paper list 17

Thesis objectives

Activity discovery in rehabilitation patients

Chapter 3

Patient monitoring using unsupervised methods

Detecting change in pain patient activity

Chapter 4

Model and parameter selection

Chapter 5

Parametric topic models: Model and parameter selection

Hyperparameter selection Chapter 6

Stability requirements Chapter 7

Topic model robustness and performance bounds

Performance bounds Chapter 5, 8

Data-driven segmentation and discovery

Chapter 8

Nonparametric topic models: Joint segmentation and discovery

Thesis summary Achievements of the thesis Chapter 2

Figure 1.4: Outline of the chapters in this thesis according to the objec-tives summarized in Section 1.6. The corresponding publications aredetailed in Table 1.1.

pact of dataset properties on discovery performance for three differentTMs, Chapter 6 focuses on the effect of TM hyperparameters on dis-covery accuracy. We evaluate TM robustness in Chapter 7 regardingTM stability requirements and TM performance bounds for parametric(Chapters 5) and nonparametric TMs (Chapter 8). We introduce a jointsegmentation and activity discovery approach using nonparametricTMs and a data-driven segmentation in Chapter 8.


Chapter Publication

2 Topic models for activity discovery and daily life monitor-ing.J. Seiter, O. Amft, G. TrösterContext-Aware Systems - Fundamentals and Applications, in press, Springer,2015.

3 Daily life activity routine discovery in hemiparetic rehabil-itation patients using topic models.J. Seiter, A. Derungs, C. Schuster-Amft, O. Amft, G. TrösterMethods of Information in Medicine, pp. 248-255, Schattauer, 2015.

4 Evaluating daily life activity using smartphones as noveloutcome measure for surgical pain therapy.J. Seiter, L. Macrea, O. Amft, S. Feese, B. Arnrich, K. Maurer, G. Trösterin Proceedings of the 8th International Conference on Body Area Networks(BodyNets), pp. 153-156, ICST, 2013.

5 Discovery of activity composites using topic models: Ananalysis of unsupervised methods.J. Seiter, O. Amft, M. Rossi, G. TrösterPervasive and Mobile Computing, vol. 15, no. 1, pp. 215-227, Elsevier, 2014.

6 Activity routine discovery in stroke rehabilitation patientswithout data annotation.J. Seiter, A. Derungs, C. Schuster-Amft, O. Amft, G. Trösterin Proceedings of the 8th International Conference on Pervasive ComputingTechnologies for Healthcare (PervasiveHealth), pp. 270-273, ICST, 2014.

7 Assessing topic models: how to obtain robustness?J. Seiter, O. Amft, G. Trösterin Proceedings of the First Workshop on Recent Advances in Behavior Predictionand Pro-Active Pervasive Computing (Pervasive), pp. 1-12, 2012.

8 Joint segmentation and activity discovery using semanticand temporal priors.J. Seiter, W. Chiu, M. Fritz, O. Amft, G. Trösterin Proceedings of the International Conference on Pervasive Computing andCommunications (PerCom), pp. 71-78, IEEE, 2015.

Table 1.1: Publications considered in this thesis (Chapters 2 to 8).

1.8. Additional publications 19

1.8 Additional publications

In addition to the publications presented in this thesis the followingpublications have been authored and co-authored:

• M. Hardegger, B. Ledergerber, S. Mutter, C. Vogt, J. Seiter, A.Calatroni and G. Tröster. Sensor technology for ice hockey andskating. In Proceedings of the 12th Annual Body Sensor NetworksConference, IEEE, 2015

• C. Strohrmann, J. Seiter and G. Tröster. Feedback provision onrunning technique with a smartphone. Journal of Ubiquitous Sys-tems & Pervasive Networks, vol. 5, no. 1, pp. 25-31, IASKS, 2014.

• A. Muaremi, A. Bexheti, F. Gravenhorst, J. Seiter, S. Feese, B.Arnrich and G. Tröster. Understanding aspects of pilgrimage us-ing social networks derived from smartphones. In Pervasive andMobile Computing, vol. 15, no. 1, pp. 166-180, Elsevier, 2014.

• A. Muaremi, F. Gravenhorst, J. Seiter, A. Bexheti, B. Arnrich andG. Tröster. Merging inhomogeneous proximity sensor systemsfor social network analysis. In Mobile and Ubiquitous Systems:Computing, Networking, and Services (MobiQuitous), pp. 181-194,Springer, 2014.

• J. Seiter, L. Macrea, S. Feese, O. Amft, B. Arnrich, K. Maurer andG. Tröster. Activity monitoring in daily life as an outcome mea-sure for surgical pain relief intervention using smartphones. InProceedings of the 17th Annual International Symposium on WearableComputers, pp. 127-128, ACM, 2013.

• C. Strohrmann, J. Seiter, Y. Llorca and G. Tröster. Can smart-phones help with running technique? In Procedia Computer Sci-ence, vol. 19, no. 1, pp. 902-907, Elsevier, 2013.

• T. Bennett and J. Seiter Getting dressed in tech. In XRDS: Cross-roads, The ACM Magazine for Students), vol. 20, no. 2, pp. 9-9,ACM, 2013.

• A. Muaremi, J. Seiter, A. Bexheti and G. Tröster. Monitor and un-derstand pilgrims: data collection using smartphones and wear-able devices. In Proceedings of the Conference on Pervasive and Ubiq-uitous Computing Adjunct Publication, pp. 679-688, ACM, 2013.


• A. Muaremi, J. Seiter, F. Gravenhorst, A. Bexheti, B. Arnrich andG. Tröster. Monitor pilgrims: prayer activity recognition usingwearable sensors. In Proceedings of the 8th International Conferenceon Body Area Networks, pp. 161-164, ICST, 2013.

• M. Rossi, J. Seiter, O. Amft, S. Buchmeier and G. Tröster. Room-Sense: an indoor positioning system for smartphones using ac-tive sound probing. In Proceedings of the 4th Augmented HumanInternational Conference, pp. 89-95, ACM, 2013.

Bibliography

[1] B. A. Nardi, Context and consciousness: activity theory and human-computer interaction. Mit Press, 1996.

[2] S. Park and J. K. Aggarwal, “A hierarchical bayesian network forevent recognition of human actions and interactions,” Multimediasystems, vol. 10, no. 2, pp. 164–179, 2004.

[3] M. S. Ryoo and J. K. Aggarwal, “Recognition of composite humanactivities through context-free grammar based representation,” inIEEE Computer Society Conference on Computer Vision and PatternRecognition, vol. 2, pp. 1709–1718, IEEE, 2006.

[4] P. Turaga, R. Chellappa, V. S. Subrahmanian, and O. Udrea, “Ma-chine recognition of human activities: A survey,” IEEE Transac-tions on Circuits and Systems for Video Technology, vol. 18, no. 11,pp. 1473–1488, 2008.

[5] A. Mannini and A. M. Sabatini, “Machine learning methods forclassifying human physical activity from on-body accelerome-ters,” Sensors, vol. 10, no. 2, pp. 1154–1175, 2010.

[6] O. Amft, C. Lombriser, T. Stiefmeier, and G. Tröster, “Recognitionof user activity sequences using distributed event detection,” inSmart Sensing and Context, pp. 126–141, Springer, 2007.

[7] E. Kim, S. Helal, and D. Cook, “Human activity recognition andpattern discovery,” Pervasive Computing, vol. 9, no. 1, pp. 48–53,2010.

[8] D. Roggen, A. Calatroni, M. Rossi, T. Holleczek, K. Forster,G. Troster, P. Lukowicz, D. Bannach, G. Pirkl, A. Ferscha, et al.,“Collecting complex activity datasets in highly rich networkedsensor environments,” in Seventh International Conference on Net-worked Sensing Systems, pp. 233–240, IEEE, 2010.

[9] T. L. van Kasteren, G. Englebienne, and B. J. Kröse, “Hierarchi-cal activity recognition using automatically clustered actions,” inAmbient Intelligence, pp. 82–91, Springer, 2011.


[10] R. Dawkins, Hierarchical organisation: a candidate principle for ethol-ogy. Cambridge U Press, 1976.

[11] H. Hendriks-Jansen, Catching ourselves in the act: Situated activity,interactive emergence, evolution, and human thought. MIT Press, 1996.

[12] D. Stokols and I. Altman, Handbook of environmental psychology,vol. 2. Wiley New York, 1987.

[13] H. A. Simon, “Theories of decision-making in economics and be-havioral science,” The American Economic Review, vol. 1, pp. 253–283, 1959.

[14] R. C. Gardner, “Social psychology and second language learn-ing: The role of attitudes and motivation social psychology oflanguage, vol 4,” Annals of Behavioral Medicine, vol. 24, no. 1, 2002.

[15] M. Ogawa and T. Togawa, “Monitoring daily activities and behav-iors at home by using brief sensors,” in 1st Annual InternationalConference On Microtechnologies in Medicine and Biology, pp. 611–614, IEEE, 2000.

[16] T. S. Barger et al., “Health-status monitoring through analysisof behavioral patterns,” IEEE Transactions on Systems, Man andCybernetics, Part A: Systems and Humans, vol. 35, no. 1, pp. 22–27,2005.

[17] S. Katz, “Assessing self-maintenance: activities of daily living,mobility, and instrumental activities of daily living.,” Journal ofthe American Geriatrics Society, vol. 31, pp. 721–727, 1983.

[18] W. A. Rogers, B. Meyer, N. Walker, and A. D. Fisk, “Functional lim-itations to daily living tasks in the aged: A focus group analysis,”Human Factors: The Journal of the Human Factors and ErgonomicsSociety, vol. 40, no. 1, pp. 111–125, 1998.

[19] M. Chan, D. Estève, C. Escriba, and E. Campo, “A review of smarthomes - present state and future challenges,” Computer methodsand programs in biomedicine, vol. 91, no. 1, pp. 55–81, 2008.

[20] T. van Kasteren, G. Englebienne, and B. J. Kröse, “Activity recog-nition using semi-markov models on real world smart homedatasets,” Journal of Ambient Intelligence and Smart Environments,vol. 2, no. 3, pp. 311–325, 2010.

23

[21] A. Aztiria, J. C. Augusto, R. Basagoiti, A. Izaguirre, and D. J. Cook,“Discovering frequent user–environment interactions in intelli-gent environments,” Personal and Ubiquitous Computing, vol. 16,pp. 91–103, 2012.

[22] M. E. Pollack, “Intelligent technology for an aging population: Theuse of ai to assist elders with cognitive impairment,” AI Magazine,vol. 26, no. 2, p. 9, 2005.

[23] J. Chen, K. Kwong, D. Chang, J. Luk, and R. Bajcsy, “Wearable sen-sors for reliable fall detection,” in 27th Annual International Con-ference of the Engineering in Medicine and Biology Society, pp. 3551–3554, IEEE, 2006.

[24] M. Marschollek, A. Rehwald, K. Wolf, M. Gietzelt, G. Nemitz,H. Meyer Zu Schwabedissen, and R. Haux, “Sensor-based fall riskassessment-an expert’to go’,” Methods of Information in Medicine,vol. 50, no. 5, pp. 420–426, 2011.

[25] R. Fensli, E. Gunnarson, and T. Gundersen, “A wearable ecg-recording system for continuous arrhythmia monitoring in a wire-less tele-home-care situation,” in Proceedings of the 18th IEEE Sym-posium on Computer-Based Medical Systems, pp. 407–412, June 2005.

[26] S. Bakkes, R. Morsch, and B. Krose, “Telemonitoring for indepen-dently living elderly: Inventory of needs amp; requirements,” in5th International Conference on Pervasive Computing Technologies forHealthcare, pp. 152–159, May 2011.

[27] S. Robben, M. Boot, M. Kanis, and B. Kröse, “Identifying and visu-alizing relevant deviations in longitudinal sensor patterns for careprofessionals,” in Proceedings of the 7th International Conference onPervasive Computing Technologies for Healthcare, pp. 416–419, ICST(Institute for Computer Sciences, Social-Informatics and Telecom-munications Engineering), 2013.

[28] A. R. Fugl-Meyer, L. Jääskö, I. Leyman, S. Olsson, and S. Steglind,“The post-stroke hemiplegic patient. 1. a method for evaluationof physical performance.,” Scandinavian journal of rehabilitationmedicine, vol. 7, no. 1, pp. 13–31, 1974.

[29] C. Gowland et al., “Measuring physical impairment and disabilitywith the chedoke-mcmaster stroke assessment.,” Stroke, vol. 24,pp. 58–63, 1993.


[30] K. Liu, E. O’Brien, J. M. Guralnik, M. H. Criqui, G. J. Martin,P. Greenland, and M. M. McDermott, “Measuring physical acti-vity in peripheral arterial disease: a comparison of two physicalactivity questionnaires with an accelerometer,” Angiology, vol. 51,no. 2, pp. 91–100, 2000.

[31] D. C. Currow, A. P. Abernethy, and M. J. Johnson, “Activity as ameasure of symptom control,” Journal of Pain and Symptom Man-agement, vol. 44, pp. e1–e2, 2012.

[32] R. H. Dworkin, D. C. Turk, J. T. Farrar, J. A. Haythornthwaite, M. P.Jensen, N. P. Katz, R. D. Kerns, G. Stucki, R. R. Allen, N. Bellamy,et al., “Core outcome measures for chronic pain clinical trials:Immpact recommendations,” Pain, vol. 113, no. 1, pp. 9–19, 2005.

[33] A. K. Dey, “Understanding and using context,” Personal and Ubiq-uitous Computing, vol. 5, no. 1, pp. 4–7, 2001.

[34] L. Bao and S. S. Intille, “Activity recognition from user-annotatedacceleration data,” in Pervasive Computing, pp. 1–17, Springer,2004.

[35] A. Schmidt, M. Beigl, and H.-W. Gellersen, “There is more to con-text than location,” Computers & Graphics, vol. 23, no. 6, pp. 893–901, 1999.

[36] M. Philipose, K. P. Fishkin, M. Perkowitz, D. J. Patterson, D. Fox,H. Kautz, and D. Hahnel, “Inferring activities from interactionswith objects,” Pervasive Computing, vol. 3, no. 4, pp. 50–57, 2004.

[37] A. J. Eronen, V. T. Peltonen, J. T. Tuomi, A. P. Klapuri, S. Fager-lund, T. Sorsa, G. Lorho, and J. Huopaniemi, “Audio-based con-text recognition,” IEEE Transactions on Audio, Speech, and LanguageProcessing, vol. 14, no. 1, pp. 321–329, 2006.

[38] E. M. Tapia, S. S. Intille, and K. Larson, “Activity recognition inthe home using simple and ubiquitous sensors,” in Proceedings ofPervasive, pp. 158–175, Springer, 2004.

[39] Y.-S. Lee and S.-B. Cho, “Human activity inference using hierar-chical bayesian network in mobile contexts,” in Neural Informa-tion Processing (B.-L. Lu, L. Zhang, and J. Kwok, eds.), vol. 7062of Lecture Notes in Computer Science, pp. 38–45, Springer Berlin /Heidelberg, 2011.

25

[40] N. Oliver, A. Garg, and E. Horvitz, “Layered representations forlearning and inferring office activity from multiple sensory chan-nels,” Computer Vision and Image Understanding, vol. 96, no. 2,pp. 163–180, 2004.

[41] P. Rashidi, D. J. Cook, L. B. Holder, and M. Schmitter-Edgecombe,“Discovering activities to recognize and track in a smart envi-ronment,” IEEE Transactions on Knowledge and Data Engineering,vol. 23, no. 4, pp. 527–539, 2011.

[42] M. Csikszentmihalyi and R. Larson, “Validity and reliability of theexperience-sampling method,” The Journal of Nervous and MentalDisease, vol. 175, no. 9, pp. 526–536, 1987.

[43] S. Szewcyzk, K. Dwan, B. Minor, B. Swedlove, and D. Cook, “An-notating smart environment sensor data for activity learning,”Technology and Health Care, vol. 17, no. 3, pp. 161–169, 2009.

[44] R. Ali, M. ElHelw, L. Atallah, B. Lo, and G. Yang, “Pattern miningfor routine behaviour discovery in pervasive healthcare environ-ments,” in International Conference on Information Technology andApplications in Biomedicine., pp. 241–244, IEEE, 2008.

[45] R. Hamid, S. Maddi, A. Johnson, A. Bobick, I. Essa, and C. Is-bell, “Unsupervised activity discovery and characterization fromevent-streams,” in Proceedings of the 21st Conference on Uncertaintyin Artificial Intelligence, 2005.

[46] R. Hamid, S. Maddi, A. Johnson, A. Bobick, I. Essa, and C. Isbell,“A novel sequence representation for unsupervised analysis ofhuman activities,” Artificial Intelligence, vol. 173, no. 14, pp. 1221–1244, 2009.

[47] T. Gu, S. Chen, X. Tao, and J. Lu, “An unsupervised approach toactivity recognition and segmentation based on object-use finger-prints,” Data & Knowledge Engineering, vol. 69, no. 6, pp. 533–544,2010.

[48] J. Begole, J. Tang, and R. Hill, “Rhythm modeling, visualizationsand applications,” in Proceedings of the 16th annual ACM Sympo-sium on User Interface Software and Technology, pp. 11–20, ACM,2003.


[49] T. Brezmes, J.-L. Gorricho, and J. Cotrina, “Activity recognitionfrom accelerometer data on a mobile phone,” in Distributed com-puting, artificial intelligence, bioinformatics, soft computing, and am-bient assisted living, pp. 796–799, Springer, 2009.

[50] E. Garcia-Ceja and R. Brena, “Long-term activity recognition fromaccelerometer data,” Procedia Technology, vol. 7, no. 0, pp. 248 – 256,2013. 3rd Iberoamerican Conference on Electronics Engineeringand Computer Science, {CIIECC} 2013.

[51] T. Huynh, M. Fritz, and B. Schiele, “Discovery of activity pat-terns using topic models,” in Proceedings of the 10th internationalconference on Ubiquitous computing, pp. 10–19, ACM, 2008.

[52] C. M. Bishop et al., Pattern recognition and machine learning, vol. 4.Springer New York, 2006.

[53] B. Clarkson and A. Pentland, “Unsupervised clustering of ambu-latory audio and video,” in Proceedings of the International Confer-ence on Acoustics, Speech, and Signal Processing, vol. 6, pp. 3037–3040, IEEE, 1999.

[54] K. Farrahi and D. Gatica-Perez, “Extracting mobile behavioralpatterns with the distant n-gram topic model,” in 16th InternationalSymposium on Wearable Computers, pp. 1–8, 2012.

[55] D. M. Blei, “Probabilistic topic models,” Communications of theACM, vol. 55, no. 4, pp. 77–84, 2012.

[56] D. Blei, A. Ng, and M. Jordan, “Latent dirichlet allocation,” TheJournal of Machine Learning Research, vol. 3, pp. 993–1022, 2003.

[57] M. Steyvers and T. Griffiths, “Probabilistic topic models,” Hand-book of Latent Semantic Analysis, vol. 427, pp. 424–440, 2007.

[58] D. Blei and J. Lafferty, “Correlated topic models,” Advances inNeural Information Processing Systems, vol. 18, pp. 147–154, 2006.

[59] D. M. Blei and J. D. Lafferty, “Dynamic topic models,” in Pro-ceedings of the 23rd International Conference on Machine Learning,pp. 113–120, ACM, 2006.

[60] D. M. Blei, T. L. Griffiths, and M. I. Jordan, “The nested chineserestaurant process and bayesian nonparametric inference of topichierarchies,” Journal of the ACM, vol. 57, no. 2, p. 7, 2010.

27

[61] Y. W. Teh, M. I. Jordan, M. J. Beal, and D. M. Blei, “Hierarchicaldirichlet processes,” Journal of the Aamerican Statistical Association,vol. 101, pp. 1566–1581, 2006.

[62] D. Griffiths and M. Tenenbaum, “Hierarchical topic models andthe nested chinese restaurant process,” Advances in Neural Infor-mation Processing Systems, vol. 16, p. 17, 2004.

[63] D. M. Blei and J. Lafferty, “Topic models,” Text Mining: Classifica-tion, Clustering, and Applications, vol. 10, pp. 71–94, 2009.

[64] H. M. Wallach, “Topic modeling: beyond bag-of-words,” in Pro-ceedings of the 23rd International Conference on Machine learning,pp. 977–984, ACM, 2006.

[65] X. Wang and E. Grimson, “Spatial latent dirichlet allocation,” inAdvances in Neural Information Processing Systems, pp. 1577–1584,2008.

[66] J. Sivic, B. C. Russell, A. A. Efros, A. Zisserman, and W. T. Freeman,“Discovering objects and their location in images,” in Tenth IEEEInternational Conference on Computer Vision, vol. 1, pp. 370–377,IEEE, 2005.

[67] B. C. Russell, W. T. Freeman, A. A. Efros, J. Sivic, and A. Zisser-man, “Using multiple segmentations to discover objects and theirextent in image collections,” in IEEE Computer Society Conferenceon Computer Vision and Pattern Recognition, vol. 2, pp. 1605–1614,IEEE, 2006.

[68] L. Cao and L. Fei-Fei, “Spatially coherent latent topic model forconcurrent segmentation and classification of objects and scenes,”in 11th International Conference on Computer Vision, pp. 1–8, IEEE,2007.

[69] J. C. Niebles, H. Wang, and L. Fei-Fei, “Unsupervised learning ofhuman action categories using spatial-temporal words,” Interna-tional Journal of Computer Vision, vol. 79, no. 3, pp. 299–318, 2008.

[70] H.-b. Tu, L.-m. Xia, and Z.-w. Wang, “The complex action recog-nition via the correlated topic model,” The Scientific World Journal,vol. 2014, p. 10, 2014.


[71] Y. Wang and G. Mori, “Human action recognition by semilatenttopic models,” IEEE Transactions on Pattern Analysis and MachineIntelligence, vol. 31, no. 10, pp. 1762–1774, 2009.

[72] V. Bettadapura, G. Schindler, T. Plötz, and I. Essa, “Augmentingbag-of-words: data-driven discovery of temporal and structuralinformation for activity recognition,” in IEEE Conference on Com-puter Vision and Pattern Recognition, pp. 2619–2626, IEEE, 2013.

[73] E. B. Sudderth, A. Torralba, W. T. Freeman, and A. S. Willsky,“Describing visual scenes using transformed objects and parts,”International Journal of Computer Vision, vol. 77, pp. 291–330, 2008.

[74] X. Wang, K. T. Ma, G.-W. Ng, and W. E. L. Grimson, “Trajectoryanalysis and semantic region modeling using nonparametric hier-archical bayesian models,” International Journal of Computer Vision,vol. 95, no. 3, pp. 287–312, 2011.

[75] P. Casale, O. Pujol, P. Radeva, and J. Vitrià, “A first approachto activity recognition using topic models.,” in CCIA, pp. 74–82,2009.

[76] K. Farrahi and D. Gatica-Perez, “Discovering human routinesfrom cell phone data with topic models,” in Proceedings of the 12thIEEE International Symposium on Wearable Computers, pp. 29–32,2008.

[77] L. Ferrari and M. Mamei, “Discovering daily routines from googlelatitude with topic models,” in IEEE International Conference onPervasive Computing and Communications Workshops, pp. 432–437,IEEE, 2011.

[78] F. Castanedo, H. Aghajan, and R. Kleihorst, “Modeling and dis-covering occupancy patterns in sensor networks using latentdirichlet allocation,” in Foundations on Natural and Artificial Com-putation, pp. 481–490, Springer, 2011.

[79] K. Rieping, G. Englebienne, and B. Kröse, “Behavior analysis of el-derly using topic models,” Pervasive and Mobile Computing, vol. 15,no. 0, pp. 181–199, 2014.

[80] F. Castanedo, D. L. de Ipiña, H. K. Aghajan, and R. Kleihorst,“Learning routines over long-term sensor data using topic mo-dels,” Expert Systems, vol. 31, no. 4, pp. 365–377, 2014.

29

[81] S. Hasan and S. V. Ukkusuri, “Urban activity pattern classificationusing topic models from online geo-location data,” TransportationResearch Part C: Emerging Technologies, vol. 44, no. 0, pp. 363 – 381,2014.

[82] Z. Huang, W. Dong, L. Ji, C. Gan, X. Lu, and H. Duan, “Discoveryof clinical pathway patterns from event logs using probabilistictopic models,” Journal of Biomedical Informatics, vol. 47, no. 0, pp. 39– 57, 2014.

[83] B. Chikhaoui, S. Wang, and H. Pigot, “Adr-splda: Activity discov-ery and recognition by combining sequential patterns and latentdirichlet allocation,” Pervasive and Mobile Computing, vol. 8, no. 6,pp. 845 – 862, 2012. Special Issue on Pervasive Healthcare.

[84] D. Kuettel, M. D. Breitenstein, L. Van Gool, and V. Ferrari, “What’sgoing on? discovering spatio-temporal dependencies in dynamicscenes,” in CVPR, pp. 1951–1958, IEEE, 2010.

2Thesis summary

This chapter summarizes the most important achievements of this thesis.First, we introduce our fully unsupervised framework for TM based activitydiscovery from sensor data. The thesis further evaluates the potential, robust-ness and limitations of topic models regarding their application in activitydiscovery from context sensor data. Finally, the limitations of the work andan outlook on future research directions are discussed.

Detailed descriptions of the achievements are available in Chapters 3 - 8as referenced in this summary.

This chapter is based on the following publication: Topic models foractivity discovery and daily life monitoring

Julia Seiter, Oliver Amft and Gerhard Tröster

Context-Aware Systems: Methods and Applications, Springer 2015

32 Chapter 2: Thesis summary

2.1 Framework for topic model based activity discoveryfrom sensor data.

Fig. 2.1 depicts our framework for TM based activity discovery fromcontext sensor data. Our approach can be applied to a variety of con-tinuous sensor measurements recorded from e.g., IMUs attached to thebody and objects and motion detectors attached to rooms and furni-ture.

The framework contains the following steps:

1. Context word detection: Sensor data is processed to feature se-quences and encoded in context words based on a predefinedcontext vocabulary. The context vocabulary contains a finitenumber of valid context words (spoon, walking, etc.).

2. Segmentation: After encoding, context word sequences are seg-mented at a particular segment size DS. The predefined segmentsize should be selected large enough to capture context wordsthat are characteristic for activities (e.g., lunch).

3. Pattern discovery: Statistics of context words are formed for eachsegment as input for the TM. TMs discover activity topics (acti-vity patterns) and their occurrence probability in the segments(e.g., activity topics 1, 2 are discovered as illustrated in Fig. 2.1).

4. Mapping: To compare the discovery performance of differentTMs, we map activity topics (e.g., activity topic 2 in Fig. 2.1)and activities (e.g., lunch) using the activity groundtruth annota-tions. As evaluation measure we analyze the activity discoveryaccuracy.

As we target fully unsupervised activity discovery, we do not usedata annotations at any of the activity discovery steps, which are con-text word detection, segmentation, and pattern discovery (Fig. 2.1).

Topic models basically are clustering algorithms, where each ac-tivity topic corresponds to a data cluster. While parametric TMs re-quire the selection of the number of activity topics (clusters) T prior tomodeling, nonparametric TMs estimate the number of activity topics(clusters) automatically from the data.

2.1. Framework for topic model based activity discovery from sensor data. 33

Mapping

MAPPING

Activity topics

Activities

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Pattern discovery

T activity topics

M activities

Context word statistics per segment

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

ACTIVITY TOPIC DISCOVERY

Parametric topic models OR

Nonparametric topic models

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

CONTEXT WORD ENCODING

FEATURE EXTRACTION

Context word detection

Context word sequence

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Sensor data

Segment size DS SEGMENTATION Segmentation

- - - - - - - - - - - - - - - - - - -- - - - - - - - - - - - - - - - - - - - - - - - - - - -

Sensor data features a: d: e: f: g:

b:

a a e b a d c c e c …

a a e b a d c c e c … DS

a b c d e f g …

topic 1 topic 2 K=2

office work lunch

M=2

c: Context vocabulary

time

p

time

p

Figure 2.1: Framework for activity discovery from sensor data se-quences using TMs: Extracted sensor data features are encoded in con-text words using a context vocabulary (using computer, mouse, spoon,plate, sitting, etc.). Context word sequences are segmented with seg-ment size DS and context word statistics are derived for each timesegment. Subsequently, T activity topics and their occurrence proba-bility p are discovered by applying nonparametric or parametric TMs.Parametric TMs require the selection of T a priori, nonparametric TMsestimate T from the data. For performance evaluation, we map T acti-vity topics to M activities. Exemplary, the activity discovery process isillustrated for M = 2 activities (office work, lunch) using T = 2 activitytopics and DS = 5 context words.


2.2 Unsupervised methods for patient monitoring indaily life

We conducted two patient monitoring studies using wearable sensortechnology and unsupervised methods for activity discovery and ac-tivity change detection in daily life. Both, activity discovery and thedetection of changes in human behavior can provide important in-formation to doctors, therapists, and nurses that is useful for remotepatient and elderly care.

The first study investigated whether TMs can be used for activitydiscovery from wearable sensor data in the daily life of hemipareticrehabilitation patients (Chapter 3).

The second study focused on the detection of a change in activityfeatures after pain intervention. In daily life, we monitored patientsusing a smartphone based system (Chapter 4).

2.2.1 Topic model based activity discovery in rehabilitation patients

Activity monitoring can provide valuable insights into abilities andhabits of patients in their daily life, and thus, serve as feedback on therehabilitation process to therapists and patients. For example, changesin duration and frequency of daily activity routines such as rest, kitchenwork, and exercise reveal the rehabilitation progress. In Chapter 3, weinvestigated whether a TM can be used to discover daily activities (e.g.,motor training) from motion sensor data without any data annotations.

To evaluate our TM based activity discovery approach we mon-itored 11 patients after stroke and brain tumor extirpation in a re-habilitation day care center during up to ten full days spread over1 − 2 months. Sensors (IMUs) were attached to both wrists and thenon-affected thigh as illustrated in Figure 2.2. Patients suffered fromlimited movement ability at one side of the body (affected side).

In total, we recorded 621h of data during 99 recording days. A set ofsix activities rest, motor training, cognitive training, medical fitness, kitchenwork, and eating/leisure was inferred from 3 motion sensors.

2.2. Unsupervised methods for patient monitoring in daily life 35

S1

S2 S3

S4

S5

S6

Figu

re2.

2:Se

nsor

setu

pfo

rTM

base

dac

tivi

tydi

scov

ery

inhe

mi-

pare

tic

reha

bilit

atio

npa

tien

ts.I

nth

eev

alua

tion

,we

used

the

acce

lera

tion

sign

also

fthr

eese

nsor

sdue

tore

dun-

danc

y:Se

nsor

sS2

,S5,

and

the

non-

affec

ted

thig

hse

nsor

S3or

S6(C

hap-

ter

3).

Rul

e-ba

sed

cont

extw

ord

voca

bula

ryEx

trem

ity

and

body

mot

ion

(1)m

otio

nbo

thar

ms,

(2)m

otio

naff

ecte

dar

m,

(3)m

otio

nno

n-aff

ecte

dar

m,

(4)n

oar

mm

otio

n,(5

)mot

ion

non-

affec

ted

leg,

(6)n

ole

gm

otio

n,pe

akin

low

freq

uenc

y/hi

ghfr

e-qu

ency

(LF/

HF)

band

:(7)

both

arm

s,(8

)aff

ecte

dar

m,(

9)no

n-aff

ecte

dar

m,(

10)n

one,

peak

LF/H

Fba

nd:(

11)n

on-a

ffec

ted

leg,

(12)

none

,(13

)bod

ym

otio

n,(1

4)no

body

mot

ion

Extr

emit

yan

dbo

dypo

stur

e(1

5)St

and,

(16)

sit,

wri

stor

ient

atio

naff

ecte

dar

m:(

17)h

oriz

on-

tal,

(18)

vert

ical

,(19

)-(2

0)no

n-aff

ecte

dar

man

alog

ue,a

ffec

ted

fore

arm

orie

ntat

ion:

(21)

dow

n,(2

2)up

,(2

3)ho

rizo

ntal

,(2

4)-

(26)

non-

affec

ted

arm

anal

ogue

(Cha

pter

3).

Tabl

e2.

1:R

ule-

base

dco

ntex

tvoc

abul

ary

wit

hN

r=

26co

ntex

twor

dsfo

rTM

base

dac

tivi

tydi

scov

ery.

Con

text

wor

dsco

nsis

ted

ofac

tivi

typr

imit

ives

desc

ribi

ngbo

dyan

dex

trem

ity

post

ures

and

mov

emen

t.C

onte

xtw

ords

wer

ede

tect

edfr

omac

cele

rati

onsi

gnal

sba

sed

ona

seto

frul

es(s

eeC

hapt

er3)

.


Our TM approach was based on the activity discovery frameworkintroduced in Figure 2.1 and the following implementations: We cal-culated P = 24 statistical time-domain and frequency features from the3-axis acceleration signals of three sensors. Data features were encodedin context words (body movement and posture) using a context vocab-ulary. Subsequently, encoded context word sequences were segmented.The parametric TM latent Dirichlet allocation (LDA) was applied for ac-tivity topic discovery from context word statistics. We used a k−nearestneighbor algorithm (kNN) for mapping T activity topics (e.g., topic 1)and M activities (e.g., motor training) for performance evaluation.

We compared two context vocabularies for unsupervised contextword encoding (see Chapter 3 for details):

1. The clustering-based vocabulary, where we used K−means clus-tering to partition the data features in K = Ncl clusters [1]. Thus,context words were defined by feature clusters and extractedfully data-driven.

2. The rule-based vocabulary, where context words consisted ofactivity primitives that included body and extremity posturesand movement as listed in Table 2.1. Compared to the clustering-based approach, the rule-based approach included expert knowl-edge to define the vocabulary and detect context words fromsensor data. Context words were assumed to be characteristicin representing activities based on previous patient observationsleading to Nr = 26 selected context words.

We evaluated the activity discovery accuracy of our TM approachas depicted in Figure 2.3. Results were cross-validated in a leave-one-day-out scheme per-patient. We used T = 2M activity topics and asegment size DS = 20 min according to parameter selection guidelinesestablished in Chapter 5. The most important findings were:

• The TM and the rule-based context vocabulary was adequatefor activity discovery in rehabilitation patients in this study:accuracies between 67% and 86% for up to six activities wereachieved (Fig. 2.3). We observed, that activities with high repeti-tion count and long durations (e.g., eating/leisure, medical fitness)tended to show higher accuracies compared to short activitieswith few repetition count (e.g., cognitive training, kitchen work) asdetailed in Chapter 5.


1 2 3 4 5 6 7 8 9 10 11

10

20

30

40

50

60

70

80

90

Acc

urac

y [%

]

Patient ID

1 2 3 4 5 6 7 8 9 10 11

2

4

6

8

# va

lid a

ctiv

ities

cluster−based rule−based mean cluster−based mean rule−based # valid activities

Figure 2.3: Activity discovery accuracies using LDA (T = 2M, DS = 20min) with a clustering-based (optimal at Ncl = 10 context words) anda rule-based context word vocabulary (Nr = 26 context words). Thenumber of valid activities M is indicated per patient. On average, therule-based approach outperformed the clustering-based approachesby 10% (Chapter 3).

• The rule-based approach outperformed the clustering-based ap-proach with optimal Ncl = 10 by 10% accuracy on average (76%versus 66%). Higher accuracy for the rule-based vocabulary sug-gested that activities showed characteristic patterns in body andextremity postures and movement. Contrary, the patterns in dataclusters extracted with the clustering-based vocabulary were lesscharacteristic. Thus, considering expert knowledge when defin-ing the context word vocabulary (here, body motion and pos-ture) can be useful to increase context words specificity. We usethe term specificity to asses how characteristic context words arefor activities (see Section 2.3).

• The rule-based context vocabulary retrieved context words in-dependent of the patient. Patients showed high variability ine.g., age (54 ± 13 years), abilities (4 wheelchair users and 7 nonwheelchair users), side of the body affected by hemiparesis, andtherapy schedule. These variety influenced type and executionof daily activities (e.g., exercise). Despite patient variety the rule-based context vocabulary was descriptive for all patients (67%and 86% activity discovery accuracy).

Differences in discovery accuracy between individual patients(Fig. 2.3) was an indicator for differences in the specificity of the con-


t }

9 daysData recording Intervention

a)

} a)

b) Experimental setupFollow-upBaseline

patient with back pain

neuro-stimulator

electrode

Figure 2.4: (a) Schedule for data recording covering patient with painbefore intervention (baseline) and after intervention with fully im-planted neuro-stimulator system (follow-up). (b) Experimental setupfor smartphone-based patient monitoring (Chapter 4).

text vocabulary. The influence of dataset properties including activityspecificity on the discovery performance of TMs is discussed in Sec-tion 2.3.

2.2.2 A smartphone based system to detect changes in patient activityafter pain intervention

Detecting changes in patient behavior outside the clinic e.g., aftersurgery provides important information to doctors and patients andcontributes to optimal patient treatment. The neuro-stimulator inter-vention is an invasive therapy and applied to patients who heavilysuffer from chronic back pain. The therapy procedure is illustratedin Figure 2.4 (a): A neuro-stimulator is implanted inside the body tostimulate the nerves in the back with electric pulses and thereforerelease pain. Doctors are interested in the outcome after full implanta-tion (follow-up) to provide optimal after-treatment to patients.

In chapter 4, we investigated whether a smartphone system pro-vided meaningful measures to detect a change in patient activity af-ter pain intervention. In a case study, two pain patients were moni-tored in their daily life before (baseline) and several weeks after inter-vention (follow-up) when the wound pain of the surgery had disap-


peared. Patients carried a smartphone that logged acceleration, GPS,and barometer data during eight days in baseline and another eightdays in follow-up (see Fig. 2.4). Patients further answered a question-naire stating self-estimated pain and activity level every evening. Datawas uploaded to a webserver daily.

We used unsupervised methods (K-means clustering, rule-baseddetectors) to extract a set of activity features from acceleration, barom-eter, and GPS data as detailed in Chapter 4. Features included e.g.the physical activity level deduced from the acceleration signal en-ergy [2], step frequency (cadence), instances of stair climbing per day,and number of location clusters visited daily (e.g., home).

In this evaluation, we used a statistical test (2-tailed t-test, signifi-cance level p < 0.095) to detect significant activity feature changes afterintervention compared to baseline. Thus, this approach was based onthe statistical analysis of basic activity features and did not use activitymodeling such as realized by TMs.

The results of our study confirmed that smartphone based activitymonitoring has the potential to provide objective intervention outcometo clinicians while not obstructing patients in their daily life activities.Further, the approach was unsupervised and no annotated data wasrequired. Fig. 2.5 depicts the results, and the following list summarizesthe most relevant findings:

• Measured activity levels of patients increased in follow-up (+20%patient 1, +10% patient 2) while patients perceived a decrease inpain level compared to baseline (-40% patient 1, -12% patient 2).

• Questionnaire assessed activity levels showed no consistency toactivity measurements and turned out to be highly subjectiveconfirming the need for objective outcome measures.

• Both patients increased the activity level at home significantly infollow-up (averaged +55% compared to baseline). Contrary, theactivity level away from home, e.g., during work did not increasesignificantly. We interpret this finding to the effect that in case ofperceived pain patients were less active in home environment.In contrast, at work patients felt committed to be active despiteperceived pain.


1 2 3 4 5 6 7 8 9 mean0

0.250.5

0.751.0 Patient 1 baseline

1 2 3 4 5 6 7 8 9 mean06121824

1 2 3 4 5 6 7 8 9 mean0

0.250.5

0.751.0

Patient 1 follow−up

AL

(nor

med

)

1 2 3 4 5 6 7 8 9 mean06121824

ALq

, PLq

AL ALq PLq

1 2 3 4 5 6 7 8 9 mean0

0.250.5

0.751.0 Patient 2 baseline

1 2 3 4 5 6 7 8 9 mean06121824

1 2 3 4 5 6 7 8 9 mean0

0.250.5

0.751.0 Patient 2 follow−up

A

L (n

orm

ed)

Recording day1 2 3 4 5 6 7 8 9 mean

06121824

ALq

, PLq

Figure 2.5: Measured daily activity level (AL) using a smartphone sys-tem in comparison to activity (ALq) and pain level (PLq) assessed fromquestionnaires. The AL is shown before (baseline) and after surgicalpain intervention (follow-up) for two patients (normalized to the maxi-mal occurring AL per patient). The AL measured in follow-up increasedfor both patients compared to baseline while the pain level decreased.Questionnaire assessed activity levels showed no consistency to acti-vity measurements (Chapter 4).

2.3 Parametric topic models for activity discovery

This section provides an overview of parametric TMs for activity dis-covery (Chapter 5): First, dataset properties and their influence onTM activity discovery are presented. Then, three parametric TMs areintroduced and compared regarding activity discovery performance.Furthermore, we exhibit the effect of TM parameter selection on theactivity discovery accuracy.

2.3.1 Dataset properties

In Chapter 5, we discuss the influence of dataset complexity on TMs’activity discovery performance. Therefore, we introduced the follow-ing dataset properties to describe dataset complexity:

2.3. Parametric topic models for activity discovery 41

1. The context word rate r = 1 − NN describes the ratio of the mean

number of context words N that occur per activity and the totalnumber of context words N in the vocabulary. The smaller rthe more similar is the set of context words between differentactivities.

2. The activity specificity s = 1 −∑N

i=1 Oi

N·M is the similarity betweenM activities when described by N context words. Oi specifiesthe number of activities that context word i occurs in. The moresimilar the context word mixture for different activities, the lowerthe specificity s, and the more challenging is activity discoverygiven the context word vocabulary.

3. The context word sequence similarity q = 1 −∑M

m=1 Lm

M reflects theaveraged repetitiveness of context word sequences for activities.Lm ∈ [0, 1] is the averaged Levenshtein distance between activityinstances of activity m. The lower q, the less recurring contextword sequence patterns exist for activities. For example, q = 1indicates that the context word sequence of all activity instancesis identical per activity.

4. The activity-instance ratio a = 1 − MR

is the ratio of the activitycount M and the averaged number of available repetitions peractivity R (activity instances). The lower a, the less amount ofdata, and the less activity instances are available to build theactivity discovery model.

Dataset properties are detailed in Chapter 5 and were normed to therange [0, 1] to express maximal dataset complexity at the lower bound(→ 0).

We considered three datasets that provided groundtruth labelsof context words and activities and assessed the complexity of eachdataset. The Dart dataset contains 11 car manufacturing activities with42 activity primitives (context words) [3]. The Ubicomp′08 dataset in-cludes 4 activity routines and 34 activity primitives (context words) [4].The Opportunity dataset consists of 5 activities in daily living and 40context words that are composed of activity primitives and object us-age [5].

Figure 2.6 depicts that for DART, complexity was lowest asmeasured by the dataset properties, followed by Ubicomp′08, and


Dataset Averaged con-text word rate r

Activityspecificity s

Context word sequencesimilarity q

Activity-instance ratio a

DART 0.87 0.99 1.0 0.45Ubicomp’08 0.25 0.79 0.36 0.43Opportunity 0.58 0.39 0.19 0

1

s

ar

q

0 1

0

01

s

ar

q

0 1

0

1 01G

s

ar

q

0

0

01

1

0

Ubicomp’08Dart Opportunity

1 1 11

00

high complexitylow complexity

Figure 2.6: (top) Dataset properties for DART, Ubicomp′08, andOpportunity and (bottom) corresponding dataset complexity plots. Thesmaller the area of the quadrangle in the complexity plot, the less chal-lenging is activity discovery under the activity/context word abstrac-tion (Chapter 5).

Opportunity. Opportunity turned out to be the most challenging datasetregarding activity discovery between the three datasets.

2.3.2 Performance comparison of parametric topic models

In Chapter 5, we introduce the parametric TMs NTM and CTM andevaluate the influence of dataset properties on parametric TM per-formance: As LDA does not consider the context word sequence (e.g.,eating follows cutting food) when modeling activities (lunch), we formu-lated the NTM. The NTM extracts relevant n-grams in context word se-quences by applying a recursive permutation test [7]. The NTM workssimilar as LDA except that the NTM extended the context word vocab-ulary by relevant n-grams of context words.

To capture possible co-occurrence of activities, e.g., co-occurringactivities within daily behavior, we further introduced the correlatedtopic model (CTM) to activity discovery. In LDA, the activity topic dis-tribution is assumed to be drawn from a Dirichlet distribution (seeSection 1.5). The CTM adds dependencies between activity topics


0 0.5 1 1.5 2404550556065707580859095

100

Aac

cura

cy [%

]

DartKmeans LDA NTM CTM

0 15 30 45 60 75 90 105

Ubicomp‘08

Segment size DS [min]0 1 2 3 4 5

Opportunity

µ±σ µ±σ µ±σ

0 10 20 30 40404550556065707580859095

100

Dart

Acc

urac

y [%

]

0 5 10 15

Ubicomp‘08

Number of Activity Topics K0 10 20 30

Opportunity

M MMN NN

a)

b)

Figure 2.7: Average class-specific accuracies and standard deviationcross-validated for (a) the datasets DART (T = 22 activity topics),Ubicomp′08 (T = 10), and Opportunity (T = 10) at varying segmentssizes DS. Optimal segment size settings per dataset indicated therange of the weighted mean duration of all activities µ ± σ. (b) Ac-curacies for DART (DS = 0.7 min), Ubicomp′08 (DS = 30 min), andOpportunity (DS = 2.5 min). Activity topic count T was varied betweenM (number of activities) and N (number of context words). A k−nearestneighbor (kNN) algorithm was used to map activity topics and activ-ities. Activity topic and activity mapping was the more obvious thecloser T was selected towards the lower bound M [6].


by replacing the Dirichlet distribution in LDA by a normal distribu-tion N(µ,Σ). The hyperparameter Σ contains covariance matrices thatmodel the correlation between activity topics, µ are the means of theactivity topic distributions.

We evaluated the activity discovery accuracy of LDA, NTM, CTM,and a baseline using K − means clustering. Results are depicted inFig. 2.7, that emerges the following findings:

• Discovery performance of all discovery methods was highestfor the DART dataset (∼ 95%), followed by Ubicomp′08 (∼ 80%),and Opportunity (∼ 60%) as illustrated in Figure 2.7. Obviousperformance differences between the three datasets were likelycaused by the different dataset complexities that showed thesame relation (see Figure 2.6).

• In our analysis, parametric TMs (LDA, NTM, CTM) provided bet-ter activity discovery performance compared to basic K −meansclustering. However, the actual gain of a TM compared to basicclustering depended on the dataset properties:

– For datasets with high activity specificity such as the Dartdataset (s = 0.99), TMs did not gain accuracy compared tobasic clustering (Fig. 2.7). Dart provided obvious activitystructure as overlap barely existed between context wordsof the 11 activities (s = 0.99).

– TMs clearly enhanced performance for datasets with littlespecificity: For Ubicomp′08, the accuracy gain of TMs com-pared to clustering yielded about 8% (s = 0.79) and forOpportunity ∼17% (s = 0.39) at the optimal segment size.

• The best TM model choice regarding activity discovery accuracydepended on the dataset properties:

– The accuracy gain of considering context word sequences asrealized in the NTM was limited. For datasets with perfectsequence similarity (Dart: q = 1) or little sequence similar-ity (Opportunity: q = 0.19), NTM did not enhance discoveryperformance. Contrary, we measured a 5% gain for NTMversus LDA in the Ubicomp′08 dataset with moderate se-quence similarity (q = 0.36). The NTM found valid n-gramsfor Ubicomp′08 e.g., walking - using toilet - washing hands thatenhanced activity discovery accuracy (Fig. 2.7). However,


Context word sequence

Groundtruth

Segmentationwith fixedDS=7

Optimalsegmentation

a)

b)

c)

activities

Segmentationwith fixedDS=3

DS=3

DS=7

context word

1:office work 2:lunch 3:rest

DS=7DS=7

DS=6DS=7

1:office work

DS=3 DS=3 DS=3 DS=3 DS=3 DS=3

DS=4

Contextvocabulary computer: mouse: spoon: plate: sitting: walking:

activity instance

Figure 2.8: Segmentation of the context word sequence for activity dis-covery: (a) Segmentation with fixed segment size DS=7 context words.(b) Segmentation with fixed segment size DS=3 context words. (c) Vari-able segment sizes where each segment includes exactly one activityinstance. Contrary to (a,b), (c) provides maximal specific context wordstatistics for activity discovery due to variable segment sizes.

n-grams for Opportunity were not characteristic resulting inreduced accuracy compared to LDA due to reduced speci-ficity of the context word vocabulary.

– Generally, the CTM did not enhance performance comparedto LDA. In order to benefit from activity correlations, corre-lated activity topics (e.g., topic of commuting and office work)would need to occur frequently in the same time segment.Such a occurrence only applied for large segment sizes (e.g.,Ubicomp′08: DS > 45 min). In this investigation, such largesegment sizes would exceed the optimal segment size rangeregarding activity discovery performance (see Section 2.3.3).

2.3.3 Topic model parameter selection

Latent Dirichlet allocation (LDA) was frequently applied for activitydiscovery from sensor data using optimal TM parameters [4, 8, 9].However, the selection of optimal TM parameters regarding activitydiscovery accuracy for a particular dataset is challenging. When using


parametric TMs for activity discovery there are two major parametersthat influence discovery performance: the number of activity topics Tand the segment size DS for segmenting the context word sequences asillustrated in Fig. 2.8.

Optimally, the segment size would equal the duration of every ac-tivity instance as illustrated in Fig. 2.8(c). Then, context word statisticswould be maximal specific for the activities. In practice, the selectionof DS is challenging as the duration of activity instances is typicallynot given a-priori. Further, the duration varies between activities andtherefore, complicates the selection of an optimal segment size DS.For example, a segment size of DS=7 would optimally capture contextword statistics of the activity office work (see Fig. 2.8(a)). However, theactivity rest ideally requires DS = 4. Using DS = 7, context words of ac-tivities rest and office work would be combined in one segment and leadto overlapped statistics for rest (Fig. 2.8(a)). Contrary, a small segmentsize (e.g. DS=3) would provide incomplete context word statistics forthe activity office work, as illustrated in Fig. 2.8(b).

Similarly, the activity topic count T influences activity discovery.For intuitive mapping of activity topics and activities, the number ofselected activity topics T should equal the number of valid activities M.

Chapter 5 investigates the optimal choice of DS and T for maximaldiscovery accuracy empirically using three datasets (Dart, Ubicomp′08,Opportunity) with different dataset properties. From our results weconclude the following:

• A close to optimal DS can be chosen around the weighted meanduration and standard deviationµ±σ across all activity instancesas obvious in Figure 2.7. The weighted mean considers µ withsmaller σ as more important than µ with larger σ.

• Performance accuracy increased with increasing number of top-ics T across all datasets up to a saturation point at around T = 2M.Often, an interpretation of activity topics is required, e.g., inFig. 2.1 topic 1 corresponds to office work: The higher the numberof topics (T > M), the less apparent was the activity topic andactivity mapping. When selecting T → N, the discovery methodbecame redundant. Given a context vocabulary with N contextwords complexity was not reduced as N context words were justtransformed to T = N activity topics. Contrary, datasets may con-tain variability in activity composition and thus require T > Mactivity topics to represent a single activity by several activity

2.4. Robustness of topic models 47

topics. As tradeoff between topic-activity mapping ability andactivity representation, T should be selected close to M but allowT > M to cover activity variability.

Besides TM parameters DS and T, there exist TM parameters thatmodel the underlying probabilistic distributions, the so called hy-perparameters. Hyperparameters are optimized within a variationalexpectation-maximization (EM) algorithm that maximizes the likeli-hood across all segments (see Section 1.5). While hyperparameterswere initialized randomly, an initialization of the Dirichlet parameter αfor LDA is of interest as local optima of the optimization problem mayexist.

While α→ 1 favored all activity topics in a segment equally, small α(α→ 0) rather privileged one particular activity topic for one segment.In Chapter 6, we evaluate the influence of varying initialization valuesαon LDA based activity discovery accuracy. We used an activity datasetof 3 rehabilitation patients including 5 activities and 36 context wordsfor the evaluation:

Results showed that the Dirichlet parameter α did not influencediscovery performance as no local optima of the EM existed. However,if local optima existed, choosing α→ 0 would emerge clearly activatedactivity topics and thus make topic-activity mapping more obvious.

2.4 Robustness of topic models

TM based activity discovery showed promising discovery perfor-mances on a variety of datasets [4, 6, 10]. For practical applications,in addition TM robustness regarding activity discovery performanceis of interest. Thus, we assess TM robustness in Chapters 5 and 8.

2.4.1 Topic model performance stability

In Chapter 7, we outline that key dataset properties influenced TM sta-bility regarding activity discovery accuracy. Such dataset propertiesincluded average duration of activity instances, amount of data avail-able for activity topic discovery and specificity of activities regardingcontext words (see dataset properties in Section 2.3).

We implemented a hierarchical simulation model based on HiddenMarkov Models (HMMs) and Markov Chains (MCs) to generate datasetswith varying dataset properties as illustrated in Fig. 2.9. We sampled


Number of days

Sequence of activity states

Activity sequence

Context words

HMM

Cont

ext w

ordsMC 1 MC 5MC 2

Act

ivit

y se

quen

ceCo

ntex

t wor

dsSe

quen

ce o

fac

tiviti

es

dinner

commutinglunch o�ce work

Figure 2.9: Simulation model to generate synthetic activity sequences(commuting, office work, lunch, etc.) and context words (walking, etc). TheHMM sampled sequences of activity states based on 5 activities. Foreach activity state, the activity duration was sampled from Normaldistributions N(µ, σ). For each activity instance, context words weresampled from the corresponding Markov Chain (MC). We modeled oneMC with a set of 34 context words for each of the 5 activities.

0 50 100 15040

60

80

100

mean activity duration [min]

0 50 100 1500

10

20

30

mean activity duration [min]

std

[%]

office workUbicomp‘08dinnerUbicomp‘08

dinnercommutinglunchoffice workUbicomp‘08

5 10 15 20 2540

60

80

100

amount of data [days]

accu

racy

[%]

5 10 15 20 250

10

20

30

accu

racy

[%]


std

[%]

a) b)

Figure 2.10: TM discovery accuracy and stability (standard deviationstd) over 20 TM runs (LDA) for the Ubicomp′08 dataset [4] and simulateddata with segment size DS = 30 min and K = 10 activity topics: (a) Thestability of four activities increased with increasing amount of dataavailable for TM based discovery. (b) The activities dinner and officework became the more stable the longer the activity durations wereselected using a fixed segment size [11].


the sequence of activity instances from HMMs, the duration of an acti-vity instance from a Normal distribution N(µ, σ), and the set of contextwords per activity instance from MCs. Initial parameters for the simu-lation model including number of activities, context word count, µ, σ,HMM and MC transition and emission matrices were extracted fromthe UbiComp′08 dataset [4]. Thus, the simulated datasets contained 5activities (lunch, office work, etc.) and 34 context words (walking).

We evaluated the influence of each dataset property on the stabilityof the activity discovery accuracy separately. To influence dataset prop-erties, we tuned the corresponding parameter of the simulation modelas detailed in Chapter 7. A standard deviation std < 5% in activitydiscovery accuracy across 20 LDA runs was considered as stable dis-covery of activity topics. We assumed performance variations std < 5%to be random, e.g., caused by the initialization of the model. In contrast,high accuracy variations (std > 5%) were assumed to emerge from thediscovery of instable activity topics across repeated LDA calculations.

The results confirmed that dataset properties influenced perfor-mance stability:

• The more data to infer activity topics, the higher was the acti-vity discovery accuracy and TM stability as obvious from Fig-ure 2.10(a).

• While a short activity duration (DS < 50min) at fixed segmentsize DS caused TM instability, the model became the more stablethe larger the segment size was selected (see Figure 2.10(b)). Asoutlined in Section 2.3 the optimal size DS regarding activitydiscovery accuracy should be selected around the mean activitydurationµ. However, given variations in the duration of activitiesa segmentation with fixed segment size DS resulted in stabilityissues for short activities (µ < DS).

• The specificity of activities influenced stability. We found thatwith decreasing activity specificity regarding context words theTM became less stable (Chapter 5).

• While dataset properties influenced TM stability there existedcommon requirements that ensured stable TM performance. Inthe considered conditions, 14 recording days and an averagedactivity duration beyond 50 min at a segment size of DS = 30 minappeared essential.


2.4.2 Topic model performance bounds under noise

TMs discover activity topics (e.g., topic of lunch) from discrete con-text words (sitting, spoon) that are previously detected from sensordata (Fig. 2.1). However, context word detectors naturally suffer fromdetector noise i.e. detection errors such as deletions and insertions.

To illustrate performance bounds of TMs and K-means clustering,we investigated the influence of deletion and insertion detector er-rors on activity discovery accuracy (Chapter 5). In the analysis, weadded synthetically generated noise to context word groundtruth an-notations. Annotations were assumed to be perfect context word de-tections (100% accuracy).

We evaluated the influence of detector noise on discovery accuracyin three datasets: Dart, Ubicomp′08, and Opportunity as introduced inSection 2.3.2. The following results were retrieved:

• LDA outperformed K −means clustering regarding noise robust-ness across all three datasets (see Fig. 2.11).

• The NTM turned out to be more sensitive to noise than LDA (seeFig. 2.11). Changes in the context word sequence due to noiseprevented robust finding of n-grams.

• Although the CTM yielded similar discovery performance asLDA using perfectly detected context word input, the CTMwas less robust against noise due to higher model complex-ity (Fig. 2.11).

• Among investigated parametric TMs, LDA was the most robustmethod against noisy context word input. Insertions did not in-fluence LDA performance for datasets with high or moderateactivity specificity (Dart: s = 0.99 and Ubicomp′08: s = 0.79). Con-trary, discovery performance of LDA declined for Opportunitydue to the low activity specificity of this dataset (s = 0.39). Thus,in our analysis deletions and insertions decreased performancefor datasets with low activity specificity (Opportunity).

Further, we analyzed noise robustness of a nonparametric TM withjoint segmentation and activity discovery as detailed in Chapter 8.Our nonparametric TM approach performed topic count estimationdata-driven (see Fig. 2.1). Further, context word segmentation was per-formed data-driven leading to varying segment sizes (Fig 2.8). Thus,


Deletions [%] Insertions [%]

Leas

t [%

]

Mos

t [%

] Dart kmeans

10080 40 20 10 0 10 20 40 80100

803020100

10203080


Leas

t [%

]

Mos

t [%

] Ubicomp08 kmeans

10080 40 20 10 0 10 20 40 80100

803020100

10203080


Leas

t [%

]

Mos

t [%

] Opportunity kmeans

10080 40 20 10 0 10 20 40 80100

803020100

10203080


Leas

t [%

]

Mos

t [%

] Dart LDA

10080 40 20 10 0 10 20 40 80100

803020100

10203080


Leas

t [%

]

Mos

t [%

] Dart NTM

10080 40 20 10 0 10 20 40 80100

803020100

10203080


Leas

t [%

]

Mos

t [%

] Dart CTM

10080 40 20 10 0 10 20 40 80100

803020100

10203080


Leas

t [%

]

Mos

t [%

] Ubicomp08 LDA

10080 40 20 10 0 10 20 40 80100

803020100

10203080


Leas

t [%

]

Mos

t [%

] Ubicomp08 NTM

10080 40 20 10 0 10 20 40 80100

803020100

10203080


Leas

t [%

]

Mos

t [%

] Ubicomp08 CTM

10080 40 20 10 0 10 20 40 80100

803020100

10203080


Leas

t [%

]

Mos

t [%

] Opportunity LDA

10080 40 20 10 0 10 20 40 80100

803020100

10203080


Leas

t [%

]

Mos

t [%

] Opportunity NTM

10080 40 20 10 0 10 20 40 80100

803020100

10203080


Leas

t [%

]

Mos

t [%

] Opportunity CTM

10080 40 20 10 0 10 20 40 80100

803020100

10203080

-4%

-8%

-6%

-2%

0%

<-10%

Delta

Figure 2.11: We analyzed activity discovery accuracies of perfect (accp)and noisy (accn) context word detectors (see Chapter 5). This figureshows the relative activity discovery performance Delta =

accn−accp

accpus-

ing K −means clustering, LDA, NTM, and CTM approaches for DART,Ubicomp′08, and Opportunity datasets. We evaluated context word dele-tions and insertions for the most and least relevant context words inthe datasets. Relevance was ranked using the context word occurrenceratio. LDA showed the most robust performance among considereddiscovery approaches.

activity discovery performance of our nonparametric TM approachwas independent of activity topic count K and segment size DS. In or-der to perform data-driven segmentation, we defined a segmentationprior probability to likely cluster context words of the same activityinstance.

We assessed the following performance bounds of our nonpara-metric TM using the Opportunity dataset (Chapter 8):

• Discovery accuracy was sensitive to context word insertions(from 20% insertions) but robust against deletion noise (up to


60% deletions). The nonparametric TM performed data-drivencontext word segmentation and activity discovery jointly by in-troducing a segmentation prior. Thus, discovery accuracy wasinfluenced by the segmentation prior: Discovery accuracy re-sults suggest that the segmentation prior showed good qualitydespite deletions (up to 60%). However, the segmentation priorshowed poor quality for more than 20% insertions.

For parametric TMs, LDA was the most robust method againstnoisy context word input. Noise sensitivity was mainly influenced bydataset properties (Chapter 5). The noise sensitivity of our nonpara-metric TM approach was additionally influenced by the goodness ofthe segmentation prior under noise along with dataset properties. Yet,discovery performance of our nonparametric TM approach did notshow parameter dependency and outperformed parametric LDA asdescribed in the next section.

2.5 Nonparametric topic models for joint segmentationand activity discovery

TM parameters number of topics and segment size influenced the dis-covery performance and stability of parametric TMs as pointed out inSections 2.3 and 2.4. Although we assessed parameter selection guide-lines empirically in Section 2.3, they cannot guarantee the optimumfor every new dataset. While nonparametric TMs were applied to esti-mate the optimal number of activity topics from the data, the activitydiscovery performance remained sensitive to the selected segment sizeDS [12].

TMs estimate the activity topics from context word statistics persegment. Only DS = d can provide comprehensive context word statis-tics for an activity instance with duration d, as illustrated in Fig. 2.8(c)(e.g., DS = d = 6 for the activity lunch and DS = d = 4 for rest). Segment-ing the context word sequences with fixed segment size cannot handlevariations in activity instance durations that naturally occur: Segmentsizes DS , d leads to incomplete (Fig. 2.8(b), first segment) or over-lapped statistics with context words from other activities (Fig. 2.8(a),third segment) and thus hamper activity topic modeling.

To solve the segmentation problem, we introduced a nonparamet-ric TM approach as detailed in Chapter 8. Our TM approach was based

2.5. Nonparametric topic models for joint segmentation and activity discovery 53

Cont

ext

w

ord

se

quen

ce:

Gro

undt

ruth

ac

tiviti

es:

1:offi

ce w

ork

2:lu

nch

3:re

st1:

Segm

enta

tion

prio

rs p

:

Clus

ters

:

Stati

stics

:

p=0.

6p=

0.9

p=0.

1p=

0.9

p=0.

6p=

1p=

0.6

p=0.

6p=

0.9

p=0.

9p=

0.9

p=1

p=1

p=1

Segm

enta

tion

prio

r ext

ract

ion

Cont

ext w

ord

dete

ctio

n

Segm

enta

tion

&

disc

over

y:

ddCR

P

Activ

ity to

pics

:to

pic

1to

pic

2to

pic

3

Sens

or d

ata

co

mpu

ter:

mou

se:

spoo

n:pl

ate:

sitting

:w

alki

ng:

Cont

ext v

ocab

ular

y:co

uch:

Map

ping

Activ

ities

:o�

ce w

ork

lunc

hre

st

p=0.

1p=

0.6

Figu

re2.

12:

Illu

stra

tion

ofou

rjo

int

segm

enta

tion

and

acti

vity

disc

over

yap

proa

chba

sed

onth

edi

stan

cede

pend

ent

Chi

nese

rest

aura

ntpr

oces

s(d

dCR

P)(C

hapt

er8)

:Se

nsor

data

isen

code

din

cont

ext

wor

dsus

ing

aru

le-b

ased

cont

ext

voca

bula

ry.T

hese

gmen

tati

onpr

ior

cons

ider

edse

man

tic

and

tem

pora

lrel

atio

nsbe

twee

nco

ntex

twor

dsto

likel

ycl

uste

rco

ntex

twor

dsof

the

sam

eac

tivi

ty.O

urdd

CR

Pap

proa

chus

edth

ese

gmen

tati

onpr

ior

for

cont

ext

wor

dcl

uste

ring

.Sub

sequ

entl

y,co

ntex

tw

ord

stat

isti

csw

ere

form

edpe

rcl

uste

ran

ddd

CR

Pin

ferr

edac

tivi

tyto

pics

.To

asse

ssth

epe

rfor

man

ce,a

ctiv

ity

topi

csw

ere

map

ped

togr

ound

trut

hac

tivi

ties

.


on the distance dependent Chinese restaurant process (ddCRP) [13] . Basi-cally, ddCRP is a clustering algorithm where the number of clusters(activity topics) is estimated automatically from the data. Further, weused ddCRP to perform a data-driven segmentation of context wordsequences by introducing a segmentation prior. Thus, our ddCRP ap-proach combined the segmentation step and the pattern discovery stepin Fig. 2.1.

The principle of our joint segmentation and discovery approach isillustrated in Figure 2.12. Sensor data was encoded in context wordsusing a rule-based context word vocabulary. In order to perform seg-mentation and activity discovery jointly, ddCRP considered a segmen-tation prior in the clustering step. The segmentation prior indicatedthe probability how likely context words of the context word sequencewere assigned to the same cluster (Fig. 2.12). The ddCRP clusteredcontext words and inferred an activity topic for each cluster based oncontext word statistics per cluster. Optimally, clusters contained allcontext words of an activity instance to sample meaningful activity

Relaxing

Co�eetime

Earlymorning

Clean-up

Sandwichtime

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Relaxing Co�eetime

Earlymorning

Clean-up

Sandwichtime

Figure 2.13: Semantic distances between activity instances of 5 ac-tivities in the Opportunity dataset regarding context words. Contextwords used within the same activity (relaxing) measured smaller se-mantic distances compared to context words of different activities re-laxing and clean-up. Semantic distances were assessed from word2vecrepresentations of context words and considered in the segmentationprior (Chapter 8).


1 2 3 4 540

50

60

70

80

90

segment size DS [min]

norm

aliz

ed a

ccur

acy

[%]

ddCRP(Dt,Ds) ddCRP(Dt) CRP LDA

5 10 15 2040

50

60

70

80

90

number of topics Tno

rmal

ized

acc

urac

y [%

]

Figure 2.14: Activity discovery performance for the Opportunitydataset [5] using TMs (Chapter 8): Our nonparametric ddCRP approachincluding temporal and semantic segmentation priors ddCRP(Dt,Ds),ddCRP with only temporal segmentation prior ddCRP(Dt), nonpara-metric CRP dependent on the segment size DS and parametric LDAdependent on segment size DS and topic count T. Our ddCRP approachperformed segmentation data-driven, estimated the optimal numberof topics automatically (T = 7), and outperformed considered para-metric and nonparametric TMs even at optimal parameter settings(LDA: DS = 2.5, T = 7, CRP: DS = 2, T = 7).

topics (Fig. 2.12), similar to segments with variable segment size inFig. 2.8(c). Finally, activity topics were mapped to activities to evaluatethe activity discovery accuracy.

The segmentation prior was based on the idea that e.g., contextwords computer and mouse were semantically more similar than com-puter and plate, as computer and mouse likely belonged to the activityoffice work, whereas plate rather belonged to activity lunch. The seg-mentation prior considered semantic and temporal distances of con-text words and therefore encouraged clustering of context words thatbelonged to the same activity (Fig. 2.12). Semantic vector representa-tions of each context word were extracted unsupervised from Wikipediaarticles by applying the word2vec algorithm [14]. We used these seman-tic vector representations of context words to calculate their semanticdistances and the segmentation prior (Chapter 8).


To evaluate the approach we used the Opportunity dataset intro-duced in Section 2.3.2 that contains M = 5 activities. Our contextword vocabulary contained 25 context words including usage of 20objects (plate, drawer, etc.), a no object used context word, and 4 activityprimitives (sitting, walking). We detected context words from acceler-ation signals. Acceleration sensors attached to leg, back, and objectswere considered.

As the Opportunity dataset only contains sparse object usage data,we performed a data-driven pre-segmentation of the context wordsequences before formulating the segmentation prior (Chapter 8). Fur-ther, our ddCRP based approach worked hierarchical: (a) local clus-tering of activity topics in each data recording (b) global clustering togroup activity topics across all data recordings as e.g., each day (datarecording) the activity lunch is performed. A detailed description ofour ddCRP approach is available in Chapter 8.

We compared the activity discovery performance of our joint seg-mentation and discovery approach (ddCRP) to the parametric TM LDA.While ddCRP performed segmentation and activity count estimationdata-driven, LDA required a selection of activity topic count T andsegment size DS prior to modeling.

Further, we compared our ddCRP approach to the same approachbut based on the Chinese restaurant process (CRP) [15] instead of ddCRP.Both, CRP and ddCRP are nonparametric TMs and estimate the num-ber of clusters (activity topics) data-driven. However, CRP did notperform context word segmentation as no segmentation prior for con-text word clustering are considered in CRP. Hence, in CRP a precedingcontext word segmentation with fixed segment sizes DS was requiredas illustrated in Fig. 2.1.

From our evaluations we retained the following results:

• The results confirmed the assumption that context words whichoccur in the same activity are semantically similar and have closeword2vec representations: Semantic distances of context wordswere small between instances of the same activity, e.g., relax-ing but high for independent activities, e.g., relaxing and clean-up (Fig. 2.13). We extracted semantic representations of contextwords from Wikipedia using word2vec. Thus, the activities andtheir context words were semantically represented in Wikipediaarticles.


• Our joint segmentation and discovery approach (ddCRP) withsemantic and temporal prior outperformed parametric LDA by5% activity discovery accuracy even with optimal TM parametersT and DS for LDA (Fig. 2.14).

• Our approach (ddCRP) further outperformed the nonparametricTM (CRP) at optimal parameters by 5% activity discovery accu-racy as illustrated in Fig. 2.14. The nonparametric CRP estimatedthe topic count T data-driven. The topic count T increased con-siderably with decreasing segment sizes (T > 20 thus, T >> Mfor DS < 2 min). As T should be selected close to M due to theability of mapping activity topics and activities (see Section 2.3for details), small segment sizes DS < 2 min were not considered.

• Semantic similarities of context words turned out to be usefulfor a data-driven segmentation: The performance gain of ddCRPwith semantic and temporal priors was 18% compared to ddCRPwith temporal prior only (Fig. 2.14).

Our ddCRP approach worked fully unsupervised and did not re-quire annotated data at any stage. To estimate the segmentation prior,our ddCRP required additional computational effort (applying theword2vec algorithm) compared to parametric LDA and nonparametricCRP. Yet, due to the combination of a segmentation prior and ddCRP,segmentation of context word sequences was performed data-drivenand thus adjusted to the data. Further, the nonparametric ddCRP se-lected the optimal number of activity topics automatically. Our ddCRPapproach outperformed parametric and nonparametric TMs (even atoptimal parameter DS and T) and is thus an adequate technique foractivity discovery.


2.6 Recommendations and guidelines

This section provides practical recommendations towards robust andaccurate activity discovery from context sensor data using TMs.

• TM selection: Parametric and nonparametric TMs proved tobe suitable for activity discovery in a number of applications.Clearly, our joint segmentation and activity discovery approach(ddCRP) should be considered in the first place as it outperformedparametric and nonparametric TMs in performance and stabil-ity. To reduce the computational effort for model estimation theparametric LDA can be preferred over ddCRP as no segmentationprior is required in LDA. LDA yielded reasonable performancewhen parameter selection guidelines and stability requirementswere considered.

• Parameter selection for parametric TMs: When using parametricTMs the segment size DS should be selected around the weightedmean duration of activities to maximize discovery performance.Expected mean could be estimated from available datasets withsimilar activities. Setting the number of expected activity topicsT is a tradeoff between representing all available activity topicsand the ability of mapping T activity topics to M activities. Foractivity discovery, T should be selected close to the activity countM and allow T > M to represent activities with variability in theircontext word composition.

• Parametric TM selection: For complex datasets with high acti-vity specificity and low detector noise, basic clustering appears tobe sufficient. Considering the higher model complexity and noisesensitivity of CTM compared to LDA, LDA should be clearly pre-ferred over CTM as no performance difference was found. Simi-larly, the NTM showed only small performance gain over LDA,but higher noise sensitivity. Yet, for datasets with very little de-tector noise and moderate sequence similarity the NTM can beapplied to increase performance. Generally, if detector noise isunknown and/or dataset properties are unknown, LDA seems tobe the best option among parametric TMs to yield high discoveryperformance and robustness against noise.

• Stability requirements: Dataset properties including amount ofdata, specificity, and duration of activities highly influence TM

2.7. Conclusion 59

stability. Stability requirements as introduced in Sec. 2.3 shouldbe approximated based on datasets with similar properties. Fur-ther, considering stability requirements already during the de-sign of the experiment is highly recommended, e.g., adaptingdata recording and context vocabulary definition to influenceamount of data and activity specificity. Parametric TMs thatsegment the context word sequence at fixed segment sizes can-not ensure stable activity discovery performance when activitiesshow variability in duration. Nonparametric TMs and a jointdata-driven segmentation should be applied towards stable TMperformance.

• Context vocabulary and context word detectors: The contextvocabulary should be defined to maximize activity specificity re-garding context word composition e.g., by adding expert knowl-edge to define a rule-based vocabulary. Existing vocabularies anddetection rules could be adapted for different sets of activities.Depending on the dataset properties and the segmentation prior,context word detectors should be either tuned for: (a) high pre-cision when using our nonparametric ddCRP approach with seg-mentation prior, (b) high recall when using parametric LDA anddatasets with reasonable specificity (DART, Ubicomp′08), (c) highrecall and precision when using parametric LDA and datasetswith low specificity (Opportunity).

2.7 Conclusion

This thesis focused on unsupervised methods i.e. TMs and their appli-cation in activity discovery from context sensor data. We introducedtwo applications in patient monitoring : (a) activity discovery in hemi-paretic rehabilitation patients and (b) a smartphone based patient mon-itoring system to assess activity changes as outcome measure afterpain surgery. We further discussed the parametric TMs LDA, CTM,and NTM and outlined challenges regarding stable discovery perfor-mance. Moreover, the influence of model and parameter selection ondiscovery performance was investigated in relation to dataset prop-erties. Parametric TMs yielded reasonable discovery performances ona number of applications using optimal parameters segment size andtopic count. To overcome parameter dependency and towards optimalperforming and stable activity discovery, a data-driven approach for


joint segmentation and activity discovery was introduced and inves-tigated. Finally, we provided practical guidelines towards high per-forming and stable TM applications in activity discovery.

Our results showed that a smartphone system and unsuperviseddata analysis techniques were adequate as outcome measure after painsurgery. Especially the smartphone’s location information contributedto detect significant activity increases of pain patients after interven-tion. However, depending on the application basic activity featuresextracted from smartphone sensor data might be not sufficient to un-derstand the patients behavior and possible behavior changes. A sec-ond study showed that the TM LDA can be applied to reveal recurrentactivity patterns in patients from body-worn sensor data without su-pervised methods or data annotations. Up to 6 activities of 11 rehabil-itation patients were discovered from rule-based context words withaveraged 76% accuracy using TMs. The rule-based context vocabularyincluded expert knowledge to derive body and extremity postures andmovements. Adding expert knowledge increased the specificity of therule-based context vocabulary, and thus, outperformed the clustering-based vocabulary by 10%. We conclude that parametric LDA and ageneric rule-based context vocabulary were suitable for patient moni-toring despite high variability between patients’ behavior.

Parametric TMs outperformed basic clustering especially fordatasets with high complexity, e.g., TMs were 17% more accurate thanK − means clustering for the Opportunity dataset. While for datasetswith high specificity (Dart, s = 0.99) basic clustering appeared suffi-cient regarding performance, the TM LDA was less sensitive to contextword detector noise compared to clustering. Considering context wordsequences and activity correlations only yielded limited performancegains and highly depended on the dataset porperties. LDA turnedout to be most robust against detecor noise and the best performingTM across three datasets with different complexity. Yet, TM param-eters highly influenced discovery performance. Parametric TMs per-formed best when the segment size was selected around the weightedmean duration of activities and the topic count close to the number ofgroundtruth activities.

However, optimal parameter selection requires knowledge on thedataset properties that are often unknown in practice. To overcomeparameter dependency we introduced the nonparametric ddCRP thatperformed joint segmentation and activity discovery with data-drivensegmentation and topic count estimation. Our ddCRP method out-

2.8. Limitations 61

performed the parametric TM LDA with optimal parameters by 5%accuracy. Further, due to joint segmentation and discovery the ddCRPmodel outperformed the nonparametric model CRP with optimal seg-ment size by 5% at comparable topic count. To enable data-drivensegmentation a segmentation prior based on semantic and temporalfeatures was used, where particularly semantic similarities betweencontext words supported data-driven segmentation. We conclude thatnonparametric TMs with data-driven segmentation is an adequatetechnique towards high performing and robust activity discovery.

Overall, unsupervised methods and in particular TMs were suit-able for activity discovery from context sensor data. While nonpara-metric TMs with joint segmentation and discovery would be clearlythe best choice regarding discovery performance and robustness, alsoparametric LDA could be applied at less model complexity but reason-able performance when parameter selection and stability guidelinesare met.

2.8 Limitations

The thesis focused on unsupervised methods for activity discoveryand their application in daily life patient monitoring. In particular,challenges and possibilities of TMs and their application in activitydiscovery from context sensor data were highlighted. As a completeexamination of the research topic is beyond the scope of this thesis thefollowing limitations of the presented findings should be mentioned:

• The recording of activity datasets in daily life environment ischallenging. Hence, presented findings are based on a limitednumber of recorded and public available datasets and may notexactly generalize to every new dataset.

• Several unsupervised methods for activity discovery exist. As anevaluation of the introduced TM approaches against all existingdiscovery methods is beyond feasibility only a limited numberof discovery methods could be considered.

• Evaluating activity discovery methods requires the interpreta-tion of discovered activity patterns and therefore the mapping ofK activity topics and M daily life activities. We used supervisedmapping methods that may bias discovery performance results.


The same mapping method was used to compare different dis-covery methods. However, the influence of supervised mappingmethods (e.g., kNN) on the overall discovery performance wasnot analyzed.

• The investigation assessed the influence of each dataset prop-erty on TM stability separately. To analyze the impact of a givendataset property the corresponding simulation model parame-ter was tuned while all other parameters were fixed at initialvalues. Simultaneous tuning of different dataset properties to in-vestigate interdependencies between dataset properties was notconsidered.

• Our joint segmentation and activity discovery approach (ddCRP)required semantic meaning of context words to formulate thesegmentation prior. The word2vec algorithm was applied to ex-tract semantic representations of context words from generic textcorpora. Alternative segmentation priors for context vocabular-ies without semantic meaning were not investigated.

2.9 Outlook

This thesis disclosed challenges in TM based activity discovery fromcontext sensor data and introduced new TM approaches and practi-cal guidelines towards stable and high performing activity discovery.Nevertheless, activity discovery and TMs are a broad field where fur-ther research is necessary that may address the following challenges:

• Generalization: Applying topic methods to newly recorded orsimulated datasets could provide further generalization of pre-sented findings. Further, comparing the performance of TMs toother unsupervised methods e.g., sequential data mining or otherhistogram-based methods could be of interest.

• Mapping: For performance evaluation and interpretation of dis-covered activity topics semi-supervised extensions of TMs couldbe introduced. For example, a semi-supervised TM could con-sider the groundtruth activity label for a subset of segments to es-timate the model parameters. Such a semi-supervised TM couldbe similar to the supervised TM applied in text mining [16] thatconsiders labels for all segments. Semi-supervised TMs could

2.9. Outlook 63

give insights on how the degree of supervision influences acti-vity recognition performance.

• Stability: The influence of interdependencies between datasetproperties on TM stability could be investigated. For this pur-pose, an activity and context word simulation model that allowssimultaneous tuning of different dataset properties could be im-plemented.

• Segmentation prior: In future research, generalizable segmenta-tion priors that are applicable to any context vocabulary couldbe introduced. In particular, the word2vec algorithm could beadapted for other corpora e.g., containing data clusters or anykind of symbols that can be extracted from sensor data.

• Nonparametric TMs: The composition of activities in dailylife may change due to personal circumstances e.g. rehabilita-tion progress. The applicability of nonparametric TMs to assesschanging behavior patterns over time e.g. distribution changesand newly arising activities could be assessed in future work.

• Applications: TM based activity monitoring could be investi-gated for other applications within and beyond patient moni-toring. For example, using TMs for remote patient care with anadapted activity set and adapted rules for a rule-based contextvocabulary could be investigated. For patient monitoring, thedetection of a change in activity behavior due to a change in e.g.,health condition or rehabilitation progress could be analyzed ina long-term study ranging over several months.


Bibliography

[1] S. Z. Selim and M. A. Ismail, “K-means-type algorithms: a gener-alized convergence theorem and characterization of local optimal-ity,” IEEE Transactions on Pattern Analysis and Machine Intelligence,vol. 1, no. 1, pp. 81–87, 1984.

[2] C. V. Bouten, K. R. Westerterp, M. Verduin, and J. JANSSEN,“Assessment of energy expenditure for physical activity using atriaxial accelerometer,” Age (yr), vol. 23, no. 1.8, pp. 21–27, 1994.




[6] J. Seiter, O. Amft, M. Rossi, and G. Tröster, “Discovery of acti-vity composites using topic models: An analysis of unsupervisedmethods,” Pervasive and Mobile Computing, vol. 15, no. 1, pp. 215– 227, 2014.

[7] D. M. Blei and J. D. Lafferty, “Visualizing topics with multi-wordexpressions,” 2009.

[8] K. Farrahi and D. Gatica-Perez, “Discovering routines from large-scale human locations using probabilistic topic models,” ACMTransactions on Intelligent Systems and Technology, vol. 2, no. 1,pp. 1–27, 2011.

[9] J. Seiter, A. Derungs, C. Schuster-Amft, O. Amft, and G. Tröster,“Activity routine discovery in stroke rehabilitation patients with-


out data annotation,” in Proceedings of the 8th International Confer-ence on Pervasive Computing Technologies for Healthcare, pp. 270–273,2014.

[10] J. Seiter, A. Derungs, C. Schuster-Amft, O. Amft, and G. Tröster,“Daily life activity routine discovery in hemiparetic rehabilitationpatients using topic models (in press),” Methods of Information inMedicine, no. 0, 2015.

[11] J. Seiter, O. Amft, and G. Tröster, “Assessing topic models: howto obtain robustness?,” in First Workshop on Recent Advances inBehavior Prediction and Pro-Active Pervasive Computing, Pervasive,2012.

[12] F.-T. Sun, Y.-T. Yeh, H.-T. Cheng, C. Kuo, and M. Griss, “Non-parametric discovery of human routines from sensor data,” inIEEE International Conference on Pervasive Computing and Commu-nications (PerCom), pp. 11–19, IEEE, 2014.

[13] D. M. Blei and P. I. Frazier, “Distance dependent chinese restau-rant processes,” The Journal of Machine Learning Research, vol. 12,pp. 2461–2488, 2011.

[14] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Dis-tributed representations of words and phrases and their compo-sitionality,” in Advances in Neural Information Processing Systems,pp. 3111–3119, 2013.


[16] D. M. Blei and J. D. McAuliffe, “Supervised topic models.,” inNeural Information Processing Systems, vol. 7, pp. 121–128, IEEE,2007.

3Activity discovery in

rehabilitation patients

Julia Seiter, Adrian Derungs, Corina Schuster-Amft, Oliver Amft and Ger-hard Tröster

Full publication title: Daily life activity routine discovery inhemiparetic rehabilitation patients using topic models.

Methods of Information in Medicine, 2015

DOI: http://dx.doi.org/10.3414/ME14-01-0082

68 Chapter 3: Activity discovery in rehabilitation patients

Abstract

Background: Monitoring natural behavior and activity routines of hemi-paretic rehabilitation patients across the day can provide valuable progressinformation for therapists and patients and contribute to an optimized re-habilitation process. In particular, continuous patient monitoring could addtype, frequency and duration of daily life activity routines and hence com-plement standard clinical scores that are assessed for particular tasks only.Machine learning methods have been applied to infer activity routines fromsensor data. However, supervised methods require activity annotations tobuild recognition models and thus require extensive patient supervision. Dis-covery methods, including topic models could provide patient routine infor-mation and deal with variability in activity and movement performance acrosspatients. Topic models have been used to discover characteristic activity rou-tine patterns of healthy individuals using activity primitives recognized fromsupervised sensor data. Yet, the applicability of topic models for hemipareticrehabilitation patients and techniques to derive activity primitives withoutsupervision needs to be addressed.

Objectives: We investigate, (1) whether a topic model-based activity rou-tine discovery framework can infer activity routines of rehabilitation patientsfrom wearable motion sensor data. (2) We compare the performance of our topicmodel-based activity routine discovery using rule-based and clustering-basedactivity vocabulary.

Methods: We analyze the activity routine discovery in a dataset recordedwith eleven hemiparetic rehabilitation patients during up to ten full record-ing days per individual in an ambulatory daycare rehabilitation center usingwearable motion sensors attached to both wrists and the non-affected thigh.We introduce and compare rule-based and clustering-based activity vocab-ulary to process statistical and frequency acceleration features to activitywords. Activity words were used for activity routine pattern discovery usingtopic models based on Latent Dirichlet Allocation. Discovered activity routinepatterns were then mapped to six categorized activity routines.

Results: Using the rule-based approach, activity routines could be dis-covered with an average accuracy of 76% across all patients. The rule-basedapproach outperformed clustering by 10% and showed less confusions forpredicted activity routines.

Conclusion: Topic models are suitable to discover daily life activityroutines in hemiparetic rehabilitation patients without trained classifiers andactivity annotations. Activity routines show characteristic patterns regardingactivity primitives including body and extremity postures and movement.A patient-independent rule set can be derived. Including expert knowledge

3.1. Introduction 69

supports successful activity routine discovery over completely data-drivenclustering.

3.1 Introduction

3.1.1 Scientific background

Wearable sensors and signal processing methods have been success-fully applied in activity and movement analysis of rehabilitation pa-tients including gait and fall risk detection [1, 2], tele-rehabilitationsystems [3] and stroke patient monitoring [4–6]. For example, motorfunction scores of stroke patients can be derived from sensor mea-surements of predefined movement tasks [4, 7]. Moreover, wearableaccelerometers were used to assess upper-limb activity levels of af-fected and non-affected extremities [5, 6, 8, 9] to automate classic clin-ical motor assessments [10–12]. While functional assessments usingwearable sensors can provide quantifications of movement and mo-tor performance, they typically require supervised environments andlack information on the patient’s daily activity and lifestyle. Varioussupervised machine learning methods have been proposed to contin-uously recognize basic daily activity primitives in healthy individuals,including sleeping, walking, and many others [13–17]. By contrast,unsupervised activity discovery methods were applied in healthy vol-unteers to retrieve activity routines that are temporally coarse andrecurring, but do not require explicit supervision [18–21]. Besides spe-cific individual movements and motor assessments, activity routines,such as lunch, kitchen work, and rest can characterize motor capabilitiesand lifestyle. In particular, activity routine information can providevaluable insights into possibilities and habits of stroke patients, andthus serve as feedback on rehabilitation progress to therapists and pa-tients. For example, changes in duration and frequency of daily activityroutines such as rest, kitchen work and daily exercise (e.g. go for a walk)could provide insights on the rehabilitation progress.

Activity routines are often composed of activity primitives thathave finer temporal granularity and are suitable for direct recognitionfrom the sensor data. For example, Barger et al. used probabilistic mix-ture models to infer behavior patterns in daily life from senors eventclusters in a smart home [21]. Huynh et al. applied a probabilistic topicmodel to discover activity routines, such as lunch and office work fromrecognized activity primitives and clustered sensor data [18]. Topic


models originated from text processing to reveal underlying themesacross documents using the documents’ word statistics [22]. For acti-vity discovery, themes correspond to activity routines and documentsto sensor data slices, from which activity words, i.e. activity primi-tives are extracted. While discovery techniques aim at deriving acti-vity routines without supervision, the activity primitives were oftenobtained using supervised recognition. Huynh at al. applied an ac-tivity classifier to obtain a defined set of activity primitives as wordvocabulary for the topic modeling process, including walking freely anddesk activities [18]. In patient monitoring, however, continuous, poten-tially patient-specific supervision, as required for activity classifiers isimpractical. Yet, Huynh at al. reported that the activity word vocab-ulary obtained using basic data clustering yielded less accurate acti-vity routine predictions compared to the vocabulary using an activityclassifier [18]. Furthermore, activity discovery in hemiparetic strokepatients needs to deal with larger between-subject variability in acti-vity execution than in healthy volunteers, as activities are performedaccording to individual capabilities. In a pre-study, we investigatedactivity routine discovery, using a topic model vocabulary based onpredefined detection rules for postures and movements in three strokepatients [23]. In this work, we provide a comparison of topic modelperformance between clustering-based and rule-based vocabularies, aswell as including additional hemiparetic patients and activity routinesis of interest.

3.1.2 Study objectives

We investigated whether topic models can be used in rehabilitationpatient monitoring during regular, unscripted patient activities in arehabilitation day-care center. Our objective was to analyze fully non-supervised routine discovery, hence compare activity word vocabular-ies that avoid legacy activity classification. We evaluated a rule-basedactivity vocabulary including body and extremity postures that maybe characteristic for a patient’s activity routines, and thus could out-perform the clustering-based vocabulary. Our rule-based vocabularywas defined prior to applying it with the patients’ sensor data, thus ispatient-independent.

The objectives of this work are: (1) We introduce an activity rou-tine discovery framework based on topic models to infer activity rou-tines of rehabilitation patients using wearable movement sensor data.

3.2. Methods 71

(2) We compare the performance of our topic model-based activityroutine discovery using rule-based and clustering-based activity vo-cabulary. We evaluate our approach in a dataset recorded with elevenhemiparetic rehabilitation patients during their stay at a rehabilitationdaycare center.

3.2 Methods

3.2.1 Study participants

We implemented a study to investigate whether activity routines ofrehabilitation patients with motor function impairment can be dis-covered from sensor data using topic models. Inclusion criteria were:hemiparetic rehabilitation patients after stroke or brain tumor extir-pation with upper and/or lower extremity motor function impairmentincluding wheelchair users. Participants visited the ambulatory reha-bilitation daycare center of Reha Rheinfelden, Switzerland, were olderthan 18 years, and signed an informed consent form. Exclusion criteriawere: further motor function impairment due to additional neurolog-ical diseases other than stroke and brain tumor. The Swiss cantonalEthics committee Aargau approved the study. Eleven (six male, fivefemale) hemiparetic patients between 34-75 years (56±13), eight afterstroke and three after brain tumor extirpation were included. Fourpatients were wheelchair users.

3.2.2 Study design

Patients visited the ambulatory daycare center during 2 − 3 days perweek and were monitored during their presence in the daycare cen-ter (up to eight hours/day) for up to ten full days in total, spread over1 − 2 months. To obtain a daily activity reference log as basis for ourmethods’ performance analysis, we categorized patient activities dur-ing the day into six activity routines. Categories were established inagreement with therapists before starting the recording study from apreliminary observational analysis and the therapy schedule to reflectthe patients’ daily life structure. The therapy schedule included cogni-tive training, medical fitness and motor training. We considered additionalactivity routines that occurred regularly during therapy breaks includ-ing rest, kitchen work and eating/leisure. Eating and leisure containedpredominant periods of active interaction with other people and were


thus combined to a single activity routine. For patient recordings, thedaily activity reference log was extracted from the patient’s individualtherapy schedule of the day. Activity routines during therapy breakswere manually added to the log by the study examiner, who followedthe patient during the recording day. Activities that did not match aroutine were assigned to the null class. The following activity routineswere considered in this work:

• Eating/leisure: eating, active interaction with other persons, play-ing table games.

• Cognitive training: cognitive tasks using e.g. a computer, exercisesheets, puzzles.

• Medical fitness: intense physical exercises.

• Kitchen work: all kind of household and cooking activities.

• Motor training: all kind of motor function exercises.

• Rest: resting phases in e.g. bed, deck chair.

3.2.3 Data acquisition and measurements

Shimmer3 sensors [24] were used, providing three inertial sensormodalities. Sensor nodes were configured to log the 3-axis accelera-tion (range: ± 4g) at a frequency of 50 Hz to the sensors’ internal SDcard. In the morning of each recording day, after patients arrived tothe daycare center, six Shimmer3 sensors (LxWxH = 51mm x 34mmx 14mm) were attached to wrists, upper arms, and thighs using vel-cro straps as illustrated in Fig. 3.1. Sensors were temporarily removedduring water and massage therapies. Due to redundancy in the mea-sured data, we only used acceleration signals of three sensors for ouranalysis: right and left wrist sensors (in Fig. 3.1: S2, S5) and the non-affected thigh sensor (in Fig. 3.1: S3 or S6 , selected for each patientindividually).

3.2.4 Activity routine discovery framework

We deployed a topic model discovery framework to extract activityroutine patterns of the patients from wearable sensor data as illustratedin Fig. 3.1. Sensor data features are pre-processed into activity wordsand used as an activity vocabulary for the topic model algorithm. We

3.2. Methods 73

considered two fully unsupervised approaches to derive an activity vo-cabulary: (1) Clustering, where activity words form groups (clusters)of similar feature space patterns obtained from sensor data. (2) Rules,where activity primitives aim to provide postures and body motion,including sit and affected arm motion, that are characteristic of the acti-vity routines. The topic model is then applied to derive activity topicsfrom the activity words. Subsequently, activity topics are mapped backto activity routines using a k-nearest-neighbor (kNN) classifier for ourperformance analysis.

Features

Different time-domain and frequency-domain features were extractedfrom the sensor data to describe motion and postures. Meanµ and vari-ance σ2 were derived from acceleration signals for 1s windows for thenon-affected thigh and both wrist sensors. Frequency features includedpower in the low (LF: 0.2−2.5 Hz) and high frequency band (HF: > 2.5Hz) of the 3-axis acceleration signals (x,y,z) computed for 5s windows.In total, P = 24 features were used.

Clustering-based activity vocabulary

We used K-means clustering to partition n data points represented bytheir feature vectors x j in K = Ncl clusters [25]. Clusters were formed byminimizing the Euclidean distance of each data feature vector x j andthe cluster centroid µi for all clusters Si=1...Ncl : min

∑Ncli=1

∑x j∈Si

∥∥∥x j − µi

∥∥∥.We analyzed hard and soft clustering. For hard clustering, each datapoint was assigned to the closest cluster, resulting in a sequential ac-tivity word stream. For soft clusters, instead of assigning data featurevectors to a single cluster, cluster weights were calculated for each clus-ter according to the distance of the data point and each cluster centroidas suggested in [18]. We varied the number of clusters NCl within theinterval [5, 40], thus providing activity word vocabularies of varyingsize.

Rule-based activity vocabulary

Activity primitives were calculated from sensor data features by ap-plying a rule set as detailed in Table 3.1. For each sensor and feature,thresholds were used to obtain binary features F resulting in, e.g.,


PA

TIE

NT

MO

NIT

OR

ING

S1

S2

S3

S4

S5

S6

FE

AT

UR

E

EX

TR

AC

TIO

N

AC

TIV

ITY

TO

PIC

DIS

CO

VE

RY

AC

TIV

ITY

VO

CA

BU

LA

RY

RU

LE

-B

AS

ED

Activ

ity w

ord

s = a

ctiv

ity p

rimitiv

es

N=

Nr =

26

word

s

TO

PIC

MO

DE

L

LD

A

CLU

ST

ER

-B

AS

ED

Activ

ity w

ord

s = c

luste

rs

N=

Ncl =

10

word

s

N

Da

ta

fea

ture

s

Activ

ity

top

ics

L

P

Activ

ity

rou

tine

s

M

To

pics =

rou

tine

pa

ttern

s

S

en

so

r

da

ta

Activ

ity

wo

rds

P=

24

sta

tistic

al &

freq

uen

cy

fea

ture

s

MA

PP

ING

kN

N

OR

L to

pic

s

Figure3.1:Sensorsetup

andactivity

routinediscovery

framew

orkconsidered

forunsupervisedpatientroutine

discoveryand

toassess

differentactivityvocabulary

derivationm

ethods.Sensordata

isprocessed

intoP

datafeatures.D

atafeatures

areprocessed

toobtain

anactivity

word

vocabularyofsize

Nusing

clustering-basedor

rule-basedapproaches.T

hetopic

modeldiscovers

Lactivity

topics.Foranalysisand

quantification,theactivity

topicsare

mapped

toM

activityroutines

usinga

kNN

classifier.Due

toredundancy

inthe

measured

data,we

usedacceleration

signalsofthree

sensorsin

ouranalysis:Sensors

S2,S5,andS3

orS6,selected

foreach

patientindividually.

3.2. Methods 75

movement F = 1 or no movement F = 0. Subsequently, activity prim-itives were derived by applying logic equations to binary features F,e.g. for a left-sided hemiparetic patient, as illustrated in Table 3.1: mo-tion both arms: affected wrist (in Fig. 3.1: S5) movement FS5 = 1 ANDnon-affected wrist (in Fig. 3.1: S2) movement FS2 = 1. Every activityprimitive was either true or false, while several activity primitivesmight be true simultaneously, e.g. affected arm movement and sit. Acti-vity primitives were calculated from sensor data streams resulting insequential activity word streams. We defined a vocabulary of activityprimitives that we assumed to be characteristic in representing activityroutines based on previous patient observations. Activity primitiveswere specified to be patient-independent (e.g. affected arm movementinstead of right arm movement). Specifically, we included motion of af-fected and non-affected extremities, body postures (sit, stand), wristand forearm orientation towards the horizontal plane, and extremitymovements in low (LF: 0.2−2.5 Hz) and high frequency band (HF:> 2.5Hz). Binary features and basic logic equations were defined to extractactivity primitives from sensor data based on sensor position and theobservation of sensor data and the corresponding activity primitives.In total, we considered Nr = 26 activity primitives describing bodyand extremity postures and movements.

Activity routine pattern discovery

We used a probabilistic topic model based on Latent Dirichlet Alloca-tion (LDA) [22] that retrieves recurring activity routine patterns fromN activity words for each time segment s of a day (e.g. 20 min). Inthis work, activity words are either clusters of the clustering-basedvocabulary (N = Ncl) or activity primitives of the rule-based vocabu-lary (N = Nr).

Topic models operate based on a number of probabilistic as-sumptions. Each activity topic z has a fixed probability density func-tion (PDF) (multinomial Mult(β)), defined as the distribution over ac-tivity words w1,2,...N. The probability of an activity word w dependson p(w|z, β). For each time segment s, there is a PDF θs over activitytopics defined, denoting the probability p(z|θs) of each activity topicz in time segment s. The activity topic distribution θs of each seg-ment s is derived from a Dirichlet density distribution Dir(α) withp(θs|α). Thus, the probability of word w in segment s is given byp(w|s) = p(w|z, β) · p(z|θs) · p(θs|α). Building activity word histograms


Table3.1:R

ule-basedactivity

vocabularyw

ithN

r=

26activity

primitives.Feature

thresholdsto

obtainbinary

featuresF

S2,3,5and

logicequations

foreach

activityprim

itiveare

shown.Statistical

time-dom

ain(µ,σ

2)and

frequency-domain

features(LF,

HF

bands)of

the3-axis

accelerationsignals

accxyzof

bothw

ristsand

non-affected

thighw

ereused.T

hetable

listsfeatures

fora

left-sidehem

ipareticpatientw

ithsensor

positionsS2,S5

andS3

(seeFig.3.1).

Rule-B

asedA

ctivityV

ocabularyLogic

EquationsExtrem

ityand

bodym

otion(1)m

otionboth

arms,(2)m

otionaffected

arm,

(1):FS2∧

FS5

(2):FS2∧

FS5 ,(3):F

S2∧

FS5 ,(4):F

S2∧

FS5 ;

(3)motion

non-affectedarm

,(4)noarm

motion

F=

1:σ

2(||accxyz ||)≥

0.05(5)m

otionnon-affected

leg,(6)noleg

motion

(5):FS3 ,(6):F

S3 ;F=

1:σ

2(||accxyz ||)≥

0.05Peak

inLF

/HF

band:(7)botharm

s,(7):F

S2∧

FS5

(8):FS2∧

FS5 ,(9):F

S2∧

FS5 ,(10):F

S2∧

FS5

;(8)affected

arm,(9)non-affected

arm,(10)none

F=

1:m

ax(LF,HF)>

5Peak

LF/H

Fband:

(11)non-affectedleg,

(12)none(11):F

S3(12):F

S3 ;F=

1:m

ax(LF,HF)>

5

(13)bodym

otion,(14)nobody

motion

(13):FS2∨

FS3∨

FS5 ,(14):F

S2∧

FS3∧

FS5 ;F=

1:σ

2(||accxyz ||)≥0.05

Extremity

andbody

posture(15)stand,(16)sit

(15):FS3

(16):FS3 ;F

=1

:µ(‖accy‖)>µ(‖accz ‖)

Wristorientation

affectedarm

:(17)horizontal,(17):F

S5(18):F

S5(19):F

S2(20):F

S2 ;(18)vertical,(19)-(20)non-affected

armanalogue

F=

1:µ(‖accz ‖)

>µ(‖accx

‖)

Affected

forearmorientation:(21)dow

n,(22)up,(21):F

1S5 (22):F2S5 (23):F

1S5∧

F1S5 (24):F

1S2 (25):F2S2 (26):F

1S2∧

F1S2 ;

(23)horizontal,(24)-(26)non-affected

armana-

logueF

1=

1|atan2(acc

y ,||accxz ||)<

60◦,

F2=

1:

atan2(accy ,||accxz ||)

≥

120◦

3.2. Methods 77

for each segment s provides the probabilities p(w|s) entering the topicmodel. The topic model fits parameters α, β by maximizing the likeli-hood across all words and segments. Applying the topic model revealsthe occurrence ratio γs of L activity topics in each segment s. Detailson the LDA topic model, parameter fitting, and activity topic inferencecan be found in [22].

As M activity routines might be composed of several activity topics,typically L > M. To map L activity topics and M activity routines,we applied a kNN classifier [26]. The kNN uses γs as feature vector.Based on the minimal Euclidean distance in the feature space for eachtesting sample the activity routine of the closest training sample isassigned. We selected L = 2M topics and a topic model segment sizeof DS = 20min with 90% overlap for our analysis following parameterselection guidelines in [27]. For overlapping segments due to slidingwindows, we applied the Borda Count ranking method [28]. Details ontopic model based activity routine discovery from activity words areavailable in [27].

3.2.5 Evaluation methodology

We evaluated the activity routine discovery procedure in a per-patientleave-one-day-out cross-validation. Activity words (clusters or acti-vity primitives) and activity routine topics were estimated withoutusing supervised information. Activity words were not intended to beevaluated directly as no groundtruth labels were available. Instead,we evaluated activity words by assessing the activity routine discov-ery performance of the corresponding clustering-based and rule-basedapproaches. To ensure sufficient data amounts, we considered activityroutines that occurred more than three times for a patient. Thus, thenumber of relevant activity routines M was specific for each patient.Invalid activity routines were assigned to the null class. The kNN clas-sifier was trained using training data with relevant activity routines,testing was done on the full dataset (relevant activity routines and nullclass) for the left-out day. As evaluation measure, we used the aver-aged class-specific accuracy across all relevant activity routines. Dueto random initialization of K-means clustering and LDA topic model,we derived mean accuracies across five repeated clustering and fiverepeated topic model runs.


3.3 Results

3.3.1 Patient activity routine dataset

In total, we collected 621 h of data during 99 recording days. Fig. 3.2lists recording times, activity routine repetitions and duration accord-ing to the obtained reference log. Recording times varied due to thepatients’ individual therapy schedules and duration of outpatient re-habilitation treatment. E.g., ID4 visited the daycare center during fourweeks whereas ID1 for 2.5 months. For all patients 376 h of relevantactivity routine reference logs were collected. The type and number ofrelevant activity routines varied across patients, e.g. ID2 counted fourdifferent activity routine types while ID9 counted six. Data amountvaried for different activity routines. Eating/leisure was the most fre-quent routine and counted up to 60 repetitions, while, e.g., medicalfitness occurred up to 15 times and was considered as relevant routinefor eight patients only.

3.3.2 Activity routine discovery

Our rule-based approach yielded accuracies between 67% and 86%for up to six relevant activity routines. Fig. 3.3 illustrates the per-patient routine discovery performance. The clustering-based approachshowed less accurate results and larger accuracy variance across pa-tients with 56%-83%. Averaged across all patients, the rule-based ap-proach outperformed the clustering-based approach by 10% (76% ver-sus 66%). The rule-based activity vocabulary covered Nr = 26 activitywords (activity primitives). For the clustering-based vocabulary weused Ncl = 10 activity words (clusters), which provided the best rou-tine discovery performance in the range between [5, 40] clusters, asdepicted in Fig. 3.4 a). Contrary to activity primitives, clusters do notrepresent a particular movement or posture but are groups of datawith similar features. While the per-patient evaluation varied highlyfor different cluster counts, no substantial influence on mean accuracywas found for larger cluster counts above the minimum number ofsufficient clusters (Ncl > 5). On average, soft-clustering did not im-prove accuracies yet the standard deviation across repeated clusteringruns was lower compared to the hard clustering approach (Fig. 3.4 b)).Fig. 3.5 b) shows confusion matrices, indicating that the relevant ac-tivity routines were separable using the rule-based approach. For theclustering-based approach, the matrices show more confusions, e.g.,

3.3. Results 79

12

34

56

78

91

01

10

20

40

repetition count

ea

tin

g/l

eis

ure

cog

nit

ive

tra

inin

g

me

dic

al !

tne

ss

kitc

he

n w

ork

mo

tor

tra

inin

g

rest

bo

un

da

ry

12

34

56

78

91

01

10

20

40

60

80

pa

tie

nt

IDaverage duration [min]

12

34

56

78

91

01

10

10

20

30

40

50

60

70

80

pa

tie

nt

ID

data recorded [h]

02

46

81

01

2246810

12

14

16

#recording days

rele

va

nt

rou

tin

es

nu

ll c

lass

# d

ays

a)

b)

Figu

re3.

2:D

atas

etst

atis

tics

.(a)

Rec

ordi

ngti

mes

and

num

ber

ofre

cord

ing

days

for

each

pati

ent

incl

udin

gto

tally

621

hof

data

wit

h37

6h

ofre

leva

ntac

tivi

tyro

utin

ere

fere

nce

logs

duri

ng99

days

.R

ecor

ding

tim

esva

riab

ility

acro

sspa

tien

tsw

asdu

eto

indi

vidu

alth

erap

ysc

hedu

les.

(b)

Top:

pati

ent-

spec

ific

acti

vity

rout

ine

repe

titi

onco

unt

from

refe

renc

elo

g.Bo

ttom

:ave

rage

dura

tion

ofre

leva

ntac

tivi

tyro

utin

espe

rpa

tien

tfr

omre

fere

nce

log

(rou

tine

sth

atoc

curr

edm

ore

than

thre

eti

mes

).


for ID1 and ID11 cognitive training was mismatched with eating/leisureand for ID6 and ID9 kitchen work with motor training. We observed,that activity routines with high repetition count and long durations,e.g. eating/leisure Fig 5b), tended to show higher accuracies comparedto short routines with few repetition count (e.g. kitchen work, cogni-tive training). Further, the activity routine cognitive training (e.g. ID9)showed decreased performance compared to kitchen work despitesimilar amount of data. The reasons might be high variability in theroutine cognitive training, i.e. once tasks were done at a computer, thenext time on paper.

3.4 Discussion

Our activity discovery approach aimed to provide an activity routinelogbook for rehabilitation patients in their daily life. Our approach didnot focus on recognizing particular exercises nor inferring functionalscores. Instead, we consider activity routine discovery in stroke pa-tients essential to obtain objective statistics regarding type, frequency,and duration of natural and recurring activity routines that a patientperforms according to individual habits. Activity routine discoverycould provide complementary information to clinical scores, whichare assessed on predefined tasks and in a clinical environment only.

Generating activity words from the clustering-based and rule-based vocabularies and discovering activity routines were done with-out supervision, hence without trained classifiers and activity anno-tations. In our analysis, the rule-based approach outperformed data-driven clustering by 10%. We interpret this improved performance asan advantage of a rule set to capture characteristic activity patterns,including particular postures and movements, to describe activity rou-tines. Rule sets can be designed without detailed information of thedataset, hence conform to an unsupervised discovery approach, ratherthan the classification techniques used in previous works.

Activity routines and execution varied from day to day and be-tween patients, according to the patients’ individual possibilities andhabits. For example motor training covered all kind of motor exer-cises from distinct periodic arm rotations to playing tennis; similarly,wheelchair users occasionally left the wheelchair during motor train-ing. Even though these variations existed in our study data, we foundthat the patient-independent rule-based activity vocabulary was char-acteristic for activity routines of all patients. Our routine discovery

3.4. Discussion 81

12

34

56

78

91

01

1

10

20

30

40

50

60

70

80

90

accuracy [%]

pa

tie

nt

ID

12

34

56

78

91

01

1

246810

12

14

16

18

# relevant routines

ha

rd c

lust

ers

soft

clu

ste

rs

rule

−b

ase

d

me

an

ha

rd c

lust

ers

me

an

so

ft c

lust

ers

me

an

ru

le−

ba

sed

# r

ele

va

nt

rou

tin

es

Figu

re3.

3:A

ccur

acie

sfo

rpe

r-pa

tien

tac

tivi

tyro

utin

edi

scov

ery

runs

usin

gto

pic

mod

els.

Act

ivit

yw

ord

vo-

cabu

lari

esw

ere

obta

ined

usin

gha

rdan

dso

ftcl

uste

ring

(Ncl

=10

)and

the

rule

-bas

edap

proa

ch(N

r=

26).

On

aver

age,

the

rule

-bas

edap

proa

chou

tper

form

edth

ecl

uste

ring

-bas

edap

proa

ches

by10

%.I

nad

diti

on,r

elev

ant

rout

ines

are

indi

cate

dpe

rpa

tien

t(×

).


12

34

56

78

91

01

10 2 4 6 8

accuracy std. dev. [%]

pa

tien

t ID

ha

rd clu

sters

soft clu

sters

b)

51

01

52

02

53

03

54

04

0

60

80

10

0

nu

mb

er o

f cluste

rs

accuracy [%]

ha

rd clu

sterin

g

me

an

ind

ivid

ua

l pa

tien

ts

a)

Figure3.4:(a)Per-patientactivity

routinediscovery

accuracyfor

hardclustering

with

varyingactivity

vocabu-lary

sizeobtained

byvarying

thenum

berofclusters

Ncl .M

eanaccuracy

acrosspatients

indicateno

substantialinfluence

ofaccuracy

oncluster

count.(b)A

ccuracystandard

deviationσ

acrossrepeated

clusteringruns

(asdescribed

inevaluation

methodology)w

ashigher

forhard

clusteringcom

paredto

softclustering.

3.4. Discussion 83

eating/leisure

cognitive training

medical �tness

kitchen work

motor training

rest

ID1

ea

tin

g/l

eis

ure

cog

nit

ive

tra

inin

gm

ed

ica

l �tn

ess

kitc

he

n w

ork

mo

tor

tra

inin

gre

st

ID2

ID3

ID4

ID5

ID6

ID7

ea

tin

g/l

eis

ure

cog

nit

ive

tra

inin

gm

ed

ica

l �tn

ess

kitc

he

n w

ork

mo

tor

tra

inin

gre

st

ID8

ID9

ID1

0ID

11

050

10

0

050

10

0

eating/leisure

cognitive training

medical �tness

kitchen work

motor training

rest

eating/leisure

cognitive training

medical �tness

kitchen work

motor training

rest

eating/leisure

cognitive training

medical �tness

kitchen work

motor training

rest

eating/leisure

cognitive training

medical �tness

kitchen work

motor training

rest

eating/leisure

cognitive training

medical �tness

kitchen work

motor training

rest

ID1

ea

tin

g/l

eis

ure

cog

nit

ive

tra

inin

gm

ed

ica

l �tn

ess

kitc

he

n w

ork

mo

tor

tra

inin

gre

st

ID2

ID3

ID4

ID5

ID6

ID7

ea

tin

g/l

eis

ure

cog

nit

ive

tra

inin

gm

ed

ica

l �tn

ess

kitc

he

n w

ork

mo

tor

tra

inin

gre

st

ID8

ID9

ID1

0ID

11

050

10

0

050

10

0

a)

b)

Figu

re3.

5:C

onfu

sion

mat

rice

ssh

owin

gac

tual

(in

row

s)an

dto

pic

mod

elpr

edic

ted

(in

colu

mns

)ac

tivi

tyro

utin

es.(

a)H

ard

clus

teri

ng-b

ased

acti

vity

voca

bula

ry.(

b)R

ule-

base

dac

tivi

tyvo

cabu

lary

.Hat

ched

rout

ines

did

notr

each

the

occu

rren

ceco

untt

obe

com

ere

leva

ntfo

rth

epa

rtic

ular

pati

ent.

The

rule

-bas

edap

proa

chsh

owed

cons

iste

ntan

dbe

tter

perf

orm

ance

acro

ssal

lact

ivit

yro

utin

esco

mpa

red

toth

ecl

uste

ring

-bas

edap

proa

ch.


approach using topic models was applied patient-specific. Given thatour routine discovery is an unsupervised method, a patient-specificanalysis does not have disadvantages for practical application. Vary-ing accuracies between patients (67%-87%) suggest that our rule-basedvocabulary was descriptive for, e.g., ID02 (accuracy: 87%) and less use-ful for ID11 (accuracy: 67%). The clustering-based vocabulary showedto be less characteristic and discriminative compared to rule-based ac-tivity primitives for particular activity routines. While soft-clusteringis less sensitive to cluster boundaries due to soft cluster assignments,it could not outperform hard clustering in our evaluation. However,soft-clustering turned out to be more robust across repeated clusteringruns. Furthermore, we showed that a setup with three sensors attachedto wrists and non-affected thigh is less obtrusive for patients and re-duced accuracy by only 2% compared to the configuration with sixsensors.

The rehabilitation progress of patients could affect the distribu-tions over the activity vocabulary of activity routines. We monitoredpatients during 1-2 months and did not observe gradual movementchanges that affected the topic model discovery. Considerable move-ment changes, e.g. a wheelchair user becoming wheelchair indepen-dent might indeed modify word distributions for particular routines,e.g. kitchen work. Distribution changes could be addressed by re-estimating activity word distributions using the topic model. In futurework, the robustness of the topic model for rehabilitation applica-tions should be evaluated in a longitudinal study beyond two months.Moreover, nonparametric topic models [29] that estimate the numberof topics automatically from data could be investigated to deal withactivity word distribution changes and newly arising activity routinesdue to rehabilitation progress.

Our analysis showed that discovery performance tended to in-crease with increasing number of activity routine repetitions and rou-tine duration. However, at the same time less characteristic activityword compositions for an activity routine may result in more discov-ery errors. Thus, in practice the definition of an activity vocabularyis crucial to select activity words that are discriminative for activityroutines. While the rule-based activity vocabulary as a whole turnedout to be characteristic for activity routines we did not analyze thecharacteristic value of single activity primitives per activity routine.The specificity of single activity primitives depends on the dataset [27]and could be investigated in future work using several rehabilitation

3.4. Discussion 85

datasets. Selecting an optimal vocabulary size a priori is challenging.Yet, not the vocabulary size but rather the specificity of activity wordsfor a particular activity routine influences topic model performance. Inour investigation, the clustering-based vocabulary and the rule-basedvocabulary showed different specificity. Thus, the optimal vocabularysize regarding discovery performance differed between the clustering-based (Ncl = 10) and rule-based (Nr = 26) vocabularies. In general,topic models might be sensitive to overfitting when using a high num-ber of topics M. Selecting the number of topics M close to L activityroutines helps preventing overfitting. We used M = 2L topics as sug-gested in [27].

Topic models were shown to outperform basic clustering of sensordata for activity routine discovery [27]. The rule-based approach inour present work inferred 76% accurate activity routine predictionsfor 61% of the recorded data. 39% of the recorded data was assignedto the null class, hence no performance could be measured. The nullclass contained short interruptions between activity routines and activ-ities that occurred sparsely (e.g. transitions between therapies, gardenwork) that could not be assigned to any activity routine. Thus, indi-vidual, sparse and short routines were not considered since statisticsof these routines could not be derived. Nevertheless, our rule-basedapproach predicted activity routines with most similar body posturesand movements. To robustly infer routines, topic model-based dis-covery requires that sufficient activity routine repetitions and routineduration (in our analysis more than three repetitions) are available.To monitor physical training and recovery progress, we expect that apatient’s regular behavior and lifestyle is primarily relevant, exactlyhere are topic models particularly useful.

In this study, we monitored eleven ambulatory rehabilitation pa-tients in a daycare center, where patients followed an individual ther-apy schedule combined with time spent on their own during therapybreaks or scheduled self-determined activity. The daycare scenarioprovided us with access to a comprehensive activity routine refer-ence, necessary to evaluate and compare discovery performance. Weacknowledge that habits and activity routines in the daycare centermay differ from typical activity routines at home. However, we expectthat the rule-based vocabulary is transferable to routines in the homeenvironment. In the future, rule sets could be adjusted to fit particularpatient groups or routine analysis needs, prior to any data recordings.Our results indicate that activity routine discovery using topic mo-


dels and the rule-set activity vocabulary could be adapted for a homemonitoring situation in the future.

3.5 Conclusion

We analyzed whether a set of six daily life activity routines of hemi-paretic rehabilitation patients can be discovered from wearable sensorsusing topic models. Our results indicate that topic models are suitablefor patient monitoring as activity routine patterns were successfullydiscovered from sensor data without the need of trained classifiersand supervised pattern models. Our rule-based approach to derive anactivity word vocabulary outperformed a clustering-based baseline by10% and yielded 76% accuracy on average across patients. We con-cluded that activity routines show characteristic patterns that can becaptured in activity primitives, including body and extremity posturesand movement. A generic, patient-independent detection rule set wasadequate for successful activity routine discovery.

3.6 Acknowledgments

We are grateful to the participating patients and supporting therapistsfrom the daycare center of Reha Rheinfelden and their photographprovision. This work was supported by EU Marie Curie NetworkiCareNet, grant number 264738.

Bibliography

[1] M. Gietzelt, K. Wolf, M. Kohlmann, M. Marschollek, and R. Haux,“Measurement of accelerometry-based gait parameters in peoplewith and without dementia in the field. a technical feasibilitystudy.,” Methods of information in medicine, vol. 52, no. 4, pp. 319–325, 2013.

[2] M. Marschollek, A. Rehwald, K. Wolf, M. Gietzelt, G. Nemitz,H. Meyer Zu Schwabedissen, and R. Haux, “Sensor-based fall riskassessment-an expert’to go’,” Methods of Information in Medicine,vol. 50, no. 5, pp. 420–426, 2011.

[3] D. Antón, A. Goñi, and A. Illarramendi, “Exercise recognition forkinect-based telerehabilitation (in press),” Methods of Informationin Medicine, vol. 53, 2014.

[4] S. Del Din, S. Patel, C. Cobelli, and P. Bonato, “Estimating fugl-meyer clinical scores in stroke survivors using wearable sen-sors,” in Annual International Conference of the IEEE Engineeringin Medicine and Biology Society, pp. 5839–5842, IEEE, 2011.

[5] E. Haeuber, M. Shaughnessy, L. W. Forrester, K. L. Coleman,and R. F. Macko, “Accelerometer monitoring of home-andcommunity-based ambulatory activity after stroke,” Archives ofPhysical Medicine and Rehabilitation, vol. 85, no. 12, pp. 1997–2001,2004.

[6] G. Uswatte, W. L. Foo, H. Olmstead, K. Lopez, A. Holand, andL. B. Simms, “Ambulatory monitoring of arm movement usingaccelerometry: an objective measure of upper-extremity rehabili-tation in persons with chronic stroke,” Archives of Physical Medicineand Rehabilitation, vol. 86, no. 7, pp. 1498–1501, 2005.

[7] E. Wade, A. R. Parnandi, and M. J. Mataric, “Automated adminis-tration of the wolf motor function test for post-stroke assessment,”in 4th International Conference on Pervasive Computing Technologiesfor Healthcare, pp. 1–7, IEEE, 2010.


[8] S. C. van der Pas, J. A. Verbunt, D. E. Breukelaar, R. van Woerden,and H. A. Seelen, “Assessment of arm activity using triaxial ac-celerometry in patients with a stroke,” Archives of Physical Medicineand Rehabilitation, vol. 92, no. 9, pp. 1437–1442, 2011.

[9] M. de Niet, J. B. Bussmann, G. M. Ribbers, and H. J. Stam,“The stroke upper-limb activity monitor: its sensitivity to mea-sure hemiplegic upper-limb activity during daily life,” Archives ofPhysical Medicine and Rehabilitation, vol. 88, no. 9, pp. 1121–1126,2007.



[12] E. Taub, N. Miller, T. Novack, E. Cook 3rd, W. Fleming, C. Nepo-muceno, J. Connell, and J. Crago, “Technique to improve chronicmotor deficit after stroke.,” Archives of Physical Medicine and Reha-bilitation, vol. 74, no. 4, pp. 347–354, 1993.

[13] L. Bao and S. S. Intille, “Activity recognition from user-annotatedacceleration data,” in Pervasive Computing, pp. 1–17, Springer,2004.

[14] U. Blanke and B. Schiele, “Daily routine recognition through ac-tivity spotting,” in Location and Context Awareness, pp. 192–206,Springer, 2009.

[15] E. M. Tapia, S. S. Intille, and K. Larson, “Activity recognition inthe home using simple and ubiquitous sensors,” in Proceedings ofPervasive, pp. 158–175, Springer, 2004.

[16] A. Mannini and A. M. Sabatini, “Machine learning methods forclassifying human physical activity from on-body accelerome-ters,” Sensors, vol. 10, no. 2, pp. 1154–1175, 2010.

[17] J. Parkka, M. Ermes, P. Korpipaa, J. Mantyjarvi, J. Peltola, andI. Korhonen, “Activity classification using realistic data from

89

wearable sensors,” IEEE Transactions on Information Technology inBiomedicine, vol. 10, no. 1, pp. 119–128, 2006.






[23] J. Seiter, A. Derungs, C. Schuster-Amft, O. Amft, and G. Tröster,“Activity routine discovery in stroke rehabilitation patients with-out data annotation,” in Proceedings of the 8th International Confer-ence on Pervasive Computing Technologies for Healthcare, pp. 270–273,2014.

[24] A. Burns, B. R. Greene, M. J. McGrath, T. J. OShea, B. Kuris, S. M.Ayer, F. Stroiescu, and V. Cionca, “Shimmer–a wireless sensorplatform for noninvasive biomedical research,” Sensors Journal,IEEE, vol. 10, no. 9, pp. 1527–1534, 2010.

[25] S. Z. Selim and M. A. Ismail, “K-means-type algorithms: a gener-alized convergence theorem and characterization of local optimal-ity,” IEEE Transactions on Pattern Analysis and Machine Intelligence,vol. 1, no. 1, pp. 81–87, 1984.

[26] P. Cunningham and S. J. Delany, “k-nearest neighbour classifiers,”Multiple Classifier Systems, vol. 1, pp. 1–17, 2007.



[28] T. K. Ho, J. J. Hull, and S. N. Srihari, “Decision combination inmultiple classifier systems,” Pattern Analysis and Machine Intelli-gence, IEEE Transactions on, vol. 16, no. 1, pp. 66–75, 1994.


4Evaluating daily life

activity changes afterpain surgery

Julia Seiter, Lucian Macrea, Oliver Amft, Sebastian Feese, Bert Arnrich,Konrad Maurer and Gerhard Tröster

Full publication title: Evaluating daily life activity using smartphonesas novel outcome measure for surgical pain therapy.

Proceedings of the 8th International Conference on Body AreaNetworks, 2013

DOI: 10.4108/icst.bodynets.2013.253635

92 Chapter 4: Evaluating daily life activity changes after pain surgery

Abstract

In this paper we investigate the potential of a smartphone to measure pa-tients’ changes in physical activity before and after a surgical pain reliefintervention. Providing an objective intervention outcome measure to clini-cians could enhance subjective assessments from patient questionnaires andcontribute to optimal patient treatment. Thus, we show a proof of concept forour smartphone system providing physical activity from acceleration, barom-eter and location data to infer meaningful activity features that measure theintervention’s outcome. In a case study, we monitored two patients carryingthe smartphone 9 days before and another 9 days after a surgical interven-tion. Results indicate significant activity changes after intervention while thepain level decreased. Particularly physical activity in the home environmentincreased significantly for both patients where an averaged 98% increase inwalking and a more than 150% gain in fast cadence was measured. Question-naire assessed activity levels showed no meaningful correlations to activitymeasurements and turned out to be highly subjective.

4.1 Introduction

Chronic pain is a widespread and serious disease in our society.Around 20% of the world’s adult population suffer from chronicpain [1]. Chronic pain occurs at different severity levels ranging fromloss in quality of life (QoL) to bed-bound situations. While mild formsare often treated with medication, seriously affected pain patients needsurgery. The University Hospital Zurich is applying an interventionalpain therapy for patients seriously suffering from back and leg pain(Figure 4.1). A neuro-stimulator is implanted inside the body to stim-ulate nerves with electric pulses and therefore release pain. In a testphase, electrodes are implanted near the backbone to investigate suc-cess of the therapy (Figure 4.1b). The correct position of the electrodesis crucial to obtain pain-relieving stimulations. In case of at least 40%pain relief in daily life during the test phase, the neuro-stimulator isfully implanted inside the body (Figure 4.1c).

The neuro-stimulator intervention is an invasive therapy and pa-tients who undergo the implanting procedure are rare but heavilysuffering from pain. To prevent therapy failure and unnecessary riskfor patients an outcome measure during the test phase is crucial. Fur-thermore, doctors are highly interested in the outcome after full im-plantation (follow-up) to provide optimal after-treatment to patients. It


a) c)b)Baseline Test-phase Follow-up

t}

9 days Data recording

}

Interventiond)

Figure 4.1: a) Patient with pain during baseline, b) in test phase, withtest electrodes implanted but external neuro-stimulator, c) in follow-upwith fully implanted system (after intervention), d) schedule for datarecording.is assumed that a release in pain results in a change in physical activity.Currently, outcome in test and follow-up phase is assessed using feed-back from patients such as activity/pain diaries, functional disabilityand QoL questionnaires [2]. However, feedback from questionnairesis very subjective and therefore doctors look for an optimal outcomemeasure using non intrusive objective methods [3].

In pain patient monitoring the physical activity level is frequentlyassessed from accelerometers. Kop et al. [4] monitored the physicalactivity of Fibromyalgia pain patients using wrist-worn actigraphs,Ferriolli [5] investigated physical activity of cancer pain patients atdifferent stages using an accelerometer worn at the thigh. Both stud-ies reported a significant difference in physical activity of severe painpatients compared to healthy controls. Assessing the patient’s activityfrom smartphones would not require further sensors such as actigraphsand even allow modalities beyond accelerometry thus leading to morecomprehensive activity information. Smartphone applications such asthe HealthMate1 track and visualize sleep and physical activity. Theapplication BeWell [6] provides feedback to the user in terms of phys-ical, social, and mental well-being based on physical activity, ambientsound and sleep.

However, to our knowledge there are no studies available inves-tigating objective measures to compare pain patients’ activity before

1http://www.withings.com/en/app/healthmate


and after surgical intervention. To ensure optimal treatment, an ob-jective quantification of the patients’ physical activity changes afterintervention is of great interest to doctors. Thus, in a case study weanalyzed the potential of the smartphone to measure changes in phys-ical activity of two pain patients undergoing a pain relief intervention.In our previous work [7], we explored the pain reduction effect in afirst patient case and analyzed activity feature changes in responseto the intervention. We extend the investigation in this present workto two patient cases and provide a detailed analysis on activity fea-ture changes to measure the outcome of the therapy. Furthermore,correlations between the patients’ self-assessed physical activity andsmartphone measurements as well as between pain level and activityare analyzed. The contributions in this paper are threefold:

1. We investigate activity changes of pain patients before and aftera neuro-stimulator intervention.

2. We show a proof of concept for using a smartphone system toassess activity as outcome measure for a pain relief interventionincluding modalities beyond accelerometry.

3. We present and analyze the relevance of various features to re-veal changes in activity before and after intervention as noveloutcome measure for surgical pain therapy.

4.2 System implementation4.2.1 Smartphone system

An overview of the smartphone based patient monitoring system weused is depicted in Figure 4.2. The Android application is based on theframework introduced in [8] and featured logging of raw data from allmodalities, upload of data to a webserver and real-time visualizationon a remote computer.

To ensure a data logging runtime across the whole day we usedacceleration, barometer and GPS signals for inferring meaningful ac-tivity information combining physical activity with location. Acceler-ation and barometer data were resampled to a frequency of 30 Hz, forenergy savings GPS updates were logged every 3 minutes and datawas exclusively uploaded when the smartphone was plugged-in. Us-ing the Sony Xperia Active smartphone a total battery life of around 17hrs was achieved when logging data continuously. Sampling and datalogging was started automatically when the charging cable was un-plugged in the morning. Plugging-in the cable in the evening stopped

4.2. System implementation 95

DA)ata analysisData analysis

Patient

Database Doctor

Data logger

Webserver

a) b)

Figure 4.2: a) Patient monitoring system: Data is logged on a smart-phone, uploaded to a webserver and stored in a database, enablingdata visualization to doctors for analysis, b) experimental setup.

data sampling, enabled data upload and triggered a questionnaire,assessing the time the patient spent away from home as well as thedaily physical activity and pain level. The smartphone had only to becharged during night.

4.2.2 Features and data analysis

We considered the daily recording time (Trec), the ratio of walking(Walk), the time the patient walked at a certain speed (no cadence,low, medium, high), the number of steps (Ns) and the instances ofclimbing up stairs (Nst) to be indicators for the physical activity level ofa patient. Furthermore, we assumed the total energy of the accelerationsignals per day (AL) and the ratio patients spent in each intensity class(low, medium and high) to reveal activity intensity patterns acrossdays. Changes of activity levels before and after intervention mightdepend on the location of the patient, e.g. when being away fromhome, activities might be more constrained due to commitments atwork, whereas at home they might be more unlimited. Therefore, allfeatures mentioned here were calculated for periods spent at home,away from home and overall (away+home). Furthermore, the numberof location clusters a patient visited daily (Nc) and the number oftransitions in between different clusters (Nct) might reveal new lifestyle habits after intervention.

All features were calculated offline from the three axis accelerationsignal accx,y,z, the filtered and derivated barometer signal δp and theGPS data as defined in Table 4.1. Thresholds were extracted from thefeature validation dataset described in the next section. Features were


calculated per day, features in home environment were normed to thedaily time spent home, features away respectively. On all features weperformed a two-sample 1-tailed Student’s t-test to reveal substantialchanges in features before and after intervention. The two sample t-testperforms a t-test of the null hypothesis, that data samples before andafter intervention structure from normal distributions with equal meanagainst the alternative hypothesis that the mean before intervention issmaller (left-tailed) or greater (right-tailed). Normality of data distri-bution was verified by the Shapiro-Wilk test. The level of significanceadopted was p=0.095 (weakly significant). To reveal correlations be-tween activity features and the patients’ questionnaire indications wecalculated the Pearson correlation coefficient ρx/y between two pairedsequences of N data samples xi and yi, i = 1, 2, ..N.

Table 4.1: Feature abbreviations, their definition and calculation todescribe physical activity.

Feature DescriptionTrec Total recording time.Nc Number of location clusters visited; kmeans clustering on GPS

data, fusion of cluster centers closer than 10m.Nct Number of transitions in between different location clusters.

AL IAA∆t =∫ ∆t

0rect(accx)+rect(accy)+rect(accz) based on [9];

∆t=Trec, rect()=rectified signal.I Intensity. Number of intervals i within thresholds: min <

IAA∆ti ≤max; ∀i | ∆ti ∈ Trec (sliding window), ∆ti = 30s.

Ilow 0< IAA∆ti ≤30Imed 30< IAA∆ti ≤110.Ihigh IAA∆ti >110.

C Cadence. Number of steps S within thresholds: min<S≤max.Step detection:(acc2

x+acc2y+acc2

z)12>0.3 g

s

nC S<20 stepsmin

Clow 20<S≤60 stepsmin

Cmed 60<S≤90 stepsmin

Chigh S>90 stepsmin

Ns Number of steps.Walk Clow + Cmed + Chigh.Nst Number of instances climbing stairs; increased if 4.8 < δp ≤

8.4 mmin

4.3. Evaluation study 97

4.3 Evaluation study

We monitored a 48 year old female and a 45 year old male patient,both undergoing a neuro-stimulator implantation (as detailed above).Patients had been suffering from intense leg and low back pain formore than 10 years. Interaction with patients took always place inclose collaboration with doctors. Before the study, we informed thepatients about the smartphone system and its usage. Furthermore, avalidation dataset including 2 min of each lying, sitting, walking andclimbing stairs was recorded.

During data recording patients were asked to carry the smartphonein a belt bag attached to the waist as shown in Figure 4.2b), wheneverthey were out of bed or water. Remote data access via webserver al-lowed us to monitor data collection and revealed daily measurementdurations, whether the smartphone was carried at the body (from theacceleration signal’s standard deviation) as well as the detection ofoperating errors by the user. During medical consultations patientsreceived feedback regarding data collection. As depicted in Figure4.1d) we monitored the patients for 9 consecutive days in baseline(before the intervention) and another 9 days during follow-up. Ev-ery evening patients completed a questionnaire on the smartphone.During medical consultations subsequently to baseline and follow-uppatients completed an additional questionnaire. The questionnaire as-sessed whether the smartphone was worn during the whole day andcarried in the bag-belt and whether any problems occurred operatingthe system during data recording.

4.4 Results

In the following, we first verify the definition and thresholds of physi-cal activity features. Subsequently, we compare activity level measure-ments to the patients’ self-assessments and investigate correlationsbetween pain and physical activity followed by a feature relevanceanalysis pre and post intervention.

4.4.1 Feature validation

Applying feature calculation to the validation datasets of both patients,overall, we achieved 100% accuracy for the instance detection of climb-ing stairs (Nst) and 98% for walking (Walk). For stationary classes (ly-


ing, sitting) 100% was classified to be part of the low intensity class(Ilow) and cadence class nC. For movement classes (walking, climbingstairs) 100% was detected to be part of intensity class Imed, the assign-ment to the cadence classes followed a ratio of nC:Clow:Cmed:Chigh= 2:8:34:3. Thus, stationary classes were covered by low intensity andcadence (Ilow, nC), movement classes mainly by medium intensity andcadence (Imed, Cmed). High cadence/intensity classes (Chigh, Ihigh)did not occur in the validation dataset but could cover more intensemovements. As features described the validation dataset reasonablewe assumed feature calculation to be accurate.

4.4.2 Self-assessment versus measurement

In total 474 hrs of data were recorded during 16 days for patient 1 and18 days for patient 2. Patient 2 did not complete one questionnaireand one recording day was missed because of wrong system operationby patient 2. On average patients used the application 14.0 hrs/day.Figure 4.3 depicts measured and questionnaire assessed activity lev-els. For patient 1 we measured a weak correlation of questionnaireassessed and measured activity (ρAL/ALq = 0.44). The qualitative trend,i.e. increases or decreases in AL and ALq compared to the previous day,are consistent for measurements and self-estimation, except for day 8in baseline and day 5 in follow-up (increase in activity measured butself-estimated decrease). However, absolute values were indicated dif-ferently in comparison to measurements: e.g. day 1 and 4 in follow-upshow the same perceived activity level but day 4 measures 2.5 times theactivity level of day 1. Contrary to patient 1, data of patient 2 indicatedno correlation (ρAL/ALq = 0.02) between the physical activity level inthe questionnaire (ALq) and the activity measurements (AL). In base-line patient 2 indicated a 27% increase in activity (ALq) between day1 and day 2 but an inverse trend (-43%) was measured (AL). Further-more, average questionnaire activity levels (ALq) in follow-up were75% lower than in baseline whereas measurements (AL) increased by10% in the same time. Either the patient perceived activity levels inbaseline differently than in follow-up or had difficulties in consistentlyranking activity levels (ALq) on the scale.

4.4.3 Pain level versus physical activity level

A decrease in pain (PLq) was expected to result in an increase inphysical activity (AL). For patient 1, measurements confirmed this

4.4. Results 99

assumption: An average 12% decrease in pain level (PLq) resulted in20% increase in physical activity (AL) (Figure 4.3). For patient 2, weobserved a similar trend: A mean 40% decrease in pain level (PLq) infollow-up resulted in a 10% increase in physical activity (AL). However,neither for patient 1 nor for patient 2 meaningful correlations (ρ fi/PLq >0.8) or anti-correlations (ρ fi/PLq < −0.8) were found between pain leveland features fi. All features showed only weak correlationsρ fi/PLq < 0.5.

4.4.4 Feature changes

Figure 4.4 depicts mean percentage changes of features in follow-up,averaged for both patients. We measured decreases for all featuresdescribing inactivity (e.g. nC, Ilow). Increases were observed for acti-

1 2 3 4 5 6 7 8 9 mean0

0.25

0.5

0.75

1.0 Baseline Patient 1

1 2 3 4 5 6 7 8 9 mean0

6

12

18

24

1 2 3 4 5 6 7 8 9 mean0

0.25

0.5

0.75

1.0 Follow−up Patient 1

AL/

max

(AL)

1 2 3 4 5 6 7 8 9 mean0

6

12

18

24

ALq

, PLq

AL home AL away ALq PLq

1 2 3 4 5 6 7 8 9 mean0

0.25

0.5

0.75

1.0 Baseline Patient 2

1 2 3 4 5 6 7 8 9 mean0

6

12

18

24

1 2 3 4 5 6 7 8 9 mean0

0.25

0.5

0.75

1.0 Follow−up Patient 2

AL/

max

(AL)

Day1 2 3 4 5 6 7 8 9 mean

0

6

12

18

24

ALq

, PLq

Figure 4.3: Comparison of measurements and questionnaire: daily ac-tivity level (AL) measured at home and away, activity (ALq) and painlevel (PLq) from questionnaires in baseline and follow-up for bothpatients.


0

50

100

>150

(fB

L−

f FU

) / f

BL

[%]

AL

Ns

nC

Clo

w

Cm

ed

Chi

gh

Wal

k

Ilow

Imed

Ihig

h

Tre

c

Nst Nc

Nct

Overall Away Home significant [p<0.095]

Figure 4.4: Mean percentage changes of features in follow-up fFU to-wards the same feature in baseline fBL at home, away from home andoverall (away + home). Values are averaged over both subjects’ featurechanges. Changes showing statistical significance for both patients arehighlighted.

vity intense features (AL, Ns, Cmed, Chigh, Ihigh, Walk, Nst) in homeenvironment, away from home and overall, except for the activitylevel away (AL away). Generally, activity increases in home environ-ment were much higher (e.g. AL: +54%) compared to away (-15%)and overall features (+15%). Considering these gains and the patients’statements always having worn the smartphone being out of bed orwater, we measured increased physical activity in follow-up.

Figure 4.4 points out averaged feature changes after interventionbut does not provide an insight in the ratio of different walking speeds(nC:Clow:Cmed:Chigh) for each patient. The ratio of all cadence classesis illustrated in Figure 4.5a) and b) for patients 1 and 2. For patient1, all movement cadence classes (Clow, Cmed, Chigh) increased infollow up in each scenario, home, away and overall. Not only thetotal amount of walking increased in follow-up but most of the timewhen walking, fast walking (Chigh) was performed when being awayfrom home while in baseline slow and medium walking (Clow, Cmed)

4.5. Conclusion 101

were predominant. For patient 2, the ratio of walking (all cadencespeeds Clow+Cmed+Chigh) increased at home, but did not changewhen patient 2 was away after intervention. Away, the increase infast walking (Chigh) compensated the decrease in low speed walking(Clow).

We found 8 feature changes being significant for both patients ashighlighted in figure 4.4: The activity level home (Al home), numberof steps walked at home (Ns home), all cadence classes (nC home,Clow home, Cmed home, Chigh home) and the low intensity classImed home. Thus, feature analysis particularly pointed out increasedphysical activity in home environment.

4.5 Conclusion

In this paper, we investigated the usage of a smartphone for monitor-ing physical activity in daily life to infer meaningful activity featuresas novel outcome measure after a pain relief intervention. In a case

BL FU BL FU BL FU60

65

70

75

80

85

90

95

100

Per

cent

age

[%]

nC Clow Cmed Chigh

BL FU BL FU BL FU60

65

70

75

80

85

90

95

100

Per

cent

age

[%]

Patient 1 Patient 2

Overall OverallHome Home AwayAway

Figure 4.5: Mean percentages of time spent in each cadence class nC,Clow, Cmed and Chigh in baseline (BL) and follow-up (FU). Percent-ages are in % per day for BL and FU overall, in % of time spent homefor BL and FU home and in % of time spent away for BL and FU away.


study, two pain patients were monitored in their daily life during twophases of the pain therapy, baseline and follow-up. While the patients’perceived pain levels decreased in follow-up, we found an increase(20% patient 1, 10% patient 2) in the physical activity level. However,no strong correlations between activity features and pain level werefound on a daily basis. Thus, a decrease in pain resulted into an in-crease in physical activity during different phases, but not on the dailylevel. Furthermore, questionnaire assessed activity levels turned outto be highly subjective and showed no strong consistency to activitymeasurements as also reported in [10]. We conclude that activity ques-tionnaires may not provide valid measurements to clinicians.

Looking at the patients’ life styles, the release in pain did not resultin more diverse location visits as the number of location clusters did notchange. However, for both patients, we found statistically significantincreases in activity after intervention. In this study, the smartphone’slocation information contributed strongly to provide meaningful ac-tivity measures. Only features describing physical activity at homeincreased significantly for both patients, such as the number of stepswalked at home (+128%), while changes in features overall and awayfrom home were smaller for both patients and only significant for pa-tient 1. We conclude that smartphone based activity monitoring has thepotential to provide objective intervention outcome to clinicians whilenot obstructing patients in their daily life activities. In the next step, weplan to validate our approach in a more extensive study. Furthermore,assessing social activities using sound could provide meaningful painrelief indicators.

4.6 Acknowledgments

This work was supported by the EU Marie Curie Network iCareNetunder grant number 264738.

Bibliography

[1] H. Breivik, B. Collett, V. Ventafridda, R. Cohen, and D. Gallacher,“Survey of chronic pain in europe: prevalence, impact on dailylife, and treatment,” European journal of pain, vol. 10, no. 4, pp. 287–287, 2006.

[2] R. H. Dworkin, D. C. Turk, J. T. Farrar, J. A. Haythornthwaite, M. P.Jensen, N. P. Katz, R. D. Kerns, G. Stucki, R. R. Allen, N. Bellamy,et al., “Core outcome measures for chronic pain clinical trials:Immpact recommendations,” Pain, vol. 113, no. 1, pp. 9–19, 2005.

[3] D. C. Currow, A. P. Abernethy, and M. J. Johnson, “Activity as ameasure of symptom control,” Journal of Pain and Symptom Man-agement, vol. 44, pp. e1–e2, 2012.

[4] W. J. Kop, A. Lyden, A. A. Berlin, K. Ambrose, C. Olsen, R. H.Gracely, D. A. Williams, and D. J. Clauw, “Ambulatory monitoringof physical activity and symptoms in fibromyalgia and chronicfatigue syndrome,” Arthritis & Rheumatism, vol. 52, no. 1, pp. 296–303, 2005.

[5] E. Ferriolli, R. J. Skipworth, P. Hendry, A. Scott, J. Stensteth, M. Da-hele, L. Wall, C. Greig, M. Fallon, F. Strasser, et al., “Physical ac-tivity monitoring: a responsive and meaningful patient-centeredoutcome for surgery, chemotherapy, or radiotherapy?,” Journal ofpain and symptom management, vol. 43, pp. 1025–1035, 2012.

[6] N. D. Lane, M. Mohammod, M. Lin, X. Yang, H. Lu, S. Ali, A. Do-ryab, E. Berke, T. Choudhury, and A. T. Campbell, “Bewell: Asmartphone application to monitor, model and promote wellbe-ing,” in 5th International ICST Conference on Pervasive ComputingTechnologies for Healthcare, pp. 23–26, 2011.

[7] J. Seiter, L. Macrea, S. Feese, O. Amft, B. Arnrich, K. Maurer,and G. Tröster, “Activity monitoring in daily life as an outcomemeasure for surgical pain relief intervention using smartphones,”in Proceedings of the International Symposium on Wearable Computers,pp. 127–128, 2013.


[8] S. Feese, B. Arnrich, G. Troster, M. Burtscher, B. Meyer, andK. Jonas, “Coenofire: monitoring performance indicators of fire-fighters in real-world missions using smartphones,” in Proceedingsof the ACM International Joint Conference on Pervasive and UbiquitousComputing, pp. 83–92, ACM, 2013.

[9] C. V. Bouten, K. R. Westerterp, M. Verduin, and J. JANSSEN,“Assessment of energy expenditure for physical activity using atriaxial accelerometer,” Age (yr), vol. 23, no. 1.8, pp. 21–27, 1994.

[10] K. Liu, E. O’Brien, J. M. Guralnik, M. H. Criqui, G. J. Martin,P. Greenland, and M. M. McDermott, “Measuring physical acti-vity in peripheral arterial disease: a comparison of two physicalactivity questionnaires with an accelerometer,” Angiology, vol. 51,no. 2, pp. 91–100, 2000.

5Parametric topic

models for activitydiscovery

Julia Seiter, Oliver Amft, Mirco Rossi and Gerhard Tröster

Full publication title: Discovery of activity composites using topicmodels: an analysis of unsupervised methods.

Pervasive and Mobile Computing, 2014

DOI: 10.1016/j.pmcj.2014.05.007

106 Chapter 5: Parametric topic models for activity discovery

Abstract

In this work we investigate unsupervised activity discovery approaches us-ing three topic model (TM) approaches, based on Latent Dirichlet Alloca-tion (LDA), n-gram TM (NTM), and correlated TM (CTM). While LDAstructures activity primitives, NTM adds primitive sequence information, andCTM exploits co-occurring topics. We use an activity composite/primitive ab-straction and analyze three public datasets with different properties that affectthe discovery, including primitive rate, activity composite specificity, primi-tive sequence similarity, and composite-instance ratio. We compare the acti-vity composite discovery performance among the TM approaches and againsta baseline using k-means clustering. We provide guidelines for method andoptimal TM parameter selection, depending on data properties and activityprimitive noise. Results indicate that TMs can outperform k-means clusteringup to 17%, when composite specificity is low. LDA-based TMs showed higherrobustness against noise compared to other TMs and k-means.

5.1 Introduction

Discovering activity composites in ubiquitous sensor data could pro-vide insights into individual behavior with broad applications stretch-ing from assisted living to medical diagnosis. Unsupervised discoveryapproaches enable users and analysts to detect and describe structuresin activity sensor data without requiring annotations and supervisedpattern learning.

A hierarchical abstraction has often been considered to partitionhuman behavior into activity primitives, which can be recognized fromon-body and ambient sensor data, and more abstract activity composites.Typically, activity primitives have a fine temporal granularity and mustbe suitable for recognition from sensor measurements. Subsequently,activity primitives could be composed into activity composites usingdiscovery methods. Figure 5.1 exemplarily illustrates the hierarchicalabstraction. Some approaches towards activity discovery from ubiq-uitous sensor data have been proposed (see Sec. 5.2 for more details),however the required activity data properties and algorithm configura-tions are not established. In addition, datasets vary widely dependingon the application, e.g. regarding activity primitive composition andcomposite specificity.

TMs are probabilistic graphical models and find their origin inthe text processing community. TMs were initially used to discover


Commuting

Office work

Lunch

8:00 10:00 12:00 14:00 16:00 18:00 t

Desk activity

Attending a meeting

Discussing at whiteboard

Eating

Queuing

Walking

Using bus

Act

ivit

y co

mp

osi

teA

ctiv

ity

pri

mit

ive

s

Figure 5.1: Example of a hierarchical activity abstraction that is con-sidered for activity discovery. In this example, three activity compositescould be discovered based on various activity primitives. In this work,we analyze the performance of unsupervised discovery methods basedon topic models using real-world datasets with different properties.

hidden topics from a corpus of documents, each containing a bag-of-words from a predefined vocabulary [1]. In activity discovery, wordscorrespond to activity primitives and topics to activity composites. Anappropriate configuration of TMs depending on the data properties isessential to obtain meaningful composite discovery results. In partic-ular, the selection of TM parameters, including primitive segment sizeand number of activity topics, has large impact on the discovery per-formance. The most frequently applied TM based on Latent DirichletAllocation (LDA) uses activity primitive histograms, e.g. in [2]. Thesehistograms represent time-independent statistics of primitives, thusdo not consider primitive sequences as they often occur in activity andbehavior data. The sequence of activity primitives might provide im-portant information on the data structure and could enhance discoveryperformance. Although discovery approaches exist that incorporate se-quence information (e.g. [3], [4], [5]), the benefits of using sequenceinformation are not yet established. Similarly, activity composites mayco-occur, which could be captured using correlated TMs (CTM) [6].

In this paper, we investigate three topic modeling approaches basedon LDA, the n-gram TM NTM and CTM, in three public activitydatasets with different activity data properties that affect the topicmodeling. The paper provides the following contributions:

1. We introduce LDA, NTM, and CTM approaches for activity dis-


covery and compare performances in three datasets to a baselinemethod using k-means data clustering. Based on the results, weprovide recommendations on optimal TM parameter choices.For this investigation, we consider three publically availabledatasets.

2. We investigate four essential dataset properties, including theactivity primitive rate, composite specificity, primitive sequencesimilarity, and composite-instance ratio, which affect the TM op-eration to derive guidelines for TM parameter choice to achieveoptimal discovery performance.

3. We analyze the effects of imperfect activity primitive recognitionon discovery performance. Here, we consider primitive insertionand deletion errors to illustrate TM performance bounds.

In our previous work, we investigated LDA-based TMs and simu-lated activity data with varying data properties [7]. In this work, weextend the discovery analysis to include NTM, CTM, and a baselinemethod for comparisons on three structurally different datasets. Wealso refine the dataset properties considered to provide guidelines inparameter and method selection.

5.2 Related work

In activity discovery, histogram based methods are frequently used toextract structural patterns. Gu et al. extracted characteristic object usefingerprints applying web-mining and discovered contrast patternsfor each activity using emerging patterns [8] . Each activity’s finger-print consisted of a histogram over object usage. Begole et al. applieda rhythm model to visualize daily rhythms from computer usage byclustering patterns of computer activity [9]. Besides the approaches us-ing clustering-based methods, probabilistic models were also appliedfor activity discovery. Barger et al. used probabilistic mixture modelsto infer behavior patterns in daily life from clusters that were formedfrom occurrence statistics of senors events in a smart home [10].TMs have been frequently applied to discover human activities andbody postures from video data and image features such as [11–15].Applications of TMs in activity discovery from wearable sensors areless frequent. Farrahi et al. inferred daily routines from proximity [16]and mobile phone data [17] using TMs. Huynh et al. discovered daily

5.2. Related work 109

routine patterns from activity primitives by applying a TM [2] to apersonal monitoring dataset obtained over several regular days. Sub-sequently, identified topics were mapped to daily life routines. In thiswork, we follow a similar approach for TM-based activity discovery.However, we aim at an in-depth analysis of TMs when being adaptedfor activity discovery across different datasets, investigate optimal pa-rameter choices, and study TMs under primitive recognition noise withthe aim to guide method and parameter selection.

Sequential information has been considered to infer activities ofdaily living. Aztiria et al. applied a sequential pattern mining algo-rithm and defined a descriptive language to infer user behavior indaily life from smart home sensor data [3]. Rashidi et al. used a discon-tinuous varied order sequence mining algorithm to extract sequencesin smart home sensor data [4]. Subsequently, they applied clusteringon extracted sequence patterns to discover ADLs. Ali et al. used theCloset+ algorithm for sequence mining and created an activity treeat different granularities to discover daily routines from physical acti-vity levels [5]. Clarkson et al. applied time-series clustering to audioand video data [18]. Subsequently, hierarchical HMMs were used forunsupervised activity inference. Hamid et al. extracted n-grams fromevent streams and used a histogram based approach to reveal acti-vity patterns by solving a graph-theoretical problem [19]. Farrahi etal. introduced a distant N-gram TM to discover topics that entirelydepend on sequences of locations and revealed days of similar loca-tion sequences using mobile phone location data [20]. Hamid et al.introduced n-grams to represent activities in terms of their local eventsequences in video data [14]. Subsequently, they used histograms ofn-grams to discover the various categories of human behavior.

The variety of sequence-based approaches suggests that sequenceinformation is valuable for activity discovery. However, the ap-proaches mentioned above require that sequential information actuallyexists among activity primitives. However, depending on the behav-ioral data, activity primitives or sensor events might not necessarilyshow a fixed sequence information for all activities and instances. Bet-tadapura et al. included sequence information from video data in theirdiscovery approach by extending event histograms with n-grams ex-tracted from events and event durations [15]. Similar to their workwe enrich activity primitive histograms with sequential informationbased on n-grams. While [15] introduce a new approach for n-gramencoding and extraction based on local and global temporal structure


combined with a k-means clustering for activity discovery we inves-tigated the advantage of integrating sequential information to TMs.We introduce NTMs, as a new TM approach to extract n-grams fromactivity primitives and combine sequential and non-sequential infor-mation for activity discovery.

The text processing and vision community adopted a variety ofpromising TM approaches. The CTM was successfully used to considercorrelations of co-occurring topics in a document [6]. Tu et al. used theCTM to infer human actions from video data [21]. Wang and Mori [22]introduced a semi-latent CTM for human activity recognition in videosequences. In this work, we discuss, why the correlation characteristicmight be useful for activity discovery from activity primitive sequencesand investigate discovery performance of the CTM in comparison tothe standard LDA-based TM and others.

5.3 Activity discovery using topic models

In this chapter we describe the topic modeling approach for activitydiscovery and introduce three types of TMs, LDA, NTM, and CTM.The general framework for processing activity primitives, performingdiscovery, and mapping topics to activity composites is detailed.

5.3.1 Activity discovery framework

An overview of our approach for activity composite discovery fromactivity primitives is illustrated in Figure 5.2. As basis for the discoverytask, we consider a vocabulary of activity primitives of fixed size Nthat can be recognized from sensor data, using activity classification orspotting methods. Since recognizers may work in parallel, we considerX channels of continuously operating activity primitive generators. Inthe first step, for each channel {1...X}, time segments s of segment sizeDS containing activity primitives are formed using sliding windows.Subsequently, primitive histograms Hxs are extracted for each channelx and time segment s. A combined histogram HXs across all channelsX is generated: HXs =

∑Xx=1 Hxs . The total set of S primitive histograms

HX is used as input for the activity composite discovery. The discoverymethods then produce a topic activation matrix Γ of size (S × K) forevery time segment s as detailed below for each TM. For k-means,we used K clusters and set them according to TM’s topic count K.The matrix Γ weighs the activation of K activity topics in each time

5.3. Activity discovery using topic models 111

Mapping kNN

Topic activations Γ

Activity composites

X channels

1

X

. . .

. . .

{1,2,…S} segments containing sequences of detected activity primitives

(S x N)

(S x K) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

(X x S)

Discovery

K activity topics

M activity composites

S histograms HX over N activity primitives Activity Primitives {1,2,...Ps}

Segment 1

Segment 2 Segment 3

Segment 1 Segment 2 Segment 3 Segment S

. . .

. . .

. . .

. . .

ds

Segment S

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

... ... ... ...

... ... ... ...

(S x M)

Discovery method:

LDA or NTM or CTM or baseline

Figure 5.2: Framework for activity composite discovery. X parallelrecognizers (channels) continuously provide activity primitives, whichare segmented with segment size DS per channel. Primitive histogramsare derived for each time segment. Subsequently, K activity topics areinferred from primitive histograms by applying one of the discoverymethods LDA, NTM, CTM or baseline. K activity topics are finallymapped to M activity composites using a kNN classifier.

segment {1...S}. Finally, for evaluation a k-nearest neighbor algorithmwas used to map activity topics and activity composites as detailed inSection 5.6.2. In the following, we present the basic ideas of TMs. Amore detailed description of the TMs learning process is beyond thescope of this paper. The reader is referred to the references mentioned.

5.3.2 Latent Dirichlet allocation (LDA)

LDA is a three-level hierarchical Bayesian model (see Figure 5.3) andcan be applied to infer activity topic activations from a corpus of seg-ments, each containing discrete activity primitives. LDA is based on


the following principles:

1. Each activity primitive pn n ∈ {1...Ps} in a time segment s isassigned to one topic zn. Each topic z, where z ∈ 1...K has afixed PDF, defined as the distribution over activity primitives pii ∈ 1, 2, ...N. The PDF is assumed as multinomial Mult(βzj, ...βzN),with βzj defining the probability of each activity primitive j ∈{1...N} in topic z. The probability of primitive pn depends onp(pn|zn, β).

2. For each time segment s, there is a PDF θs over activity top-ics defined, denoting the probability p(zn|θs) of topic zn for anyprimitive pn n ∈ {1...Ps} in time segment s. The activity topic dis-tribution θs of each segment s ∈ {1...S} is derived from a Dirichletdensity distribution Dir(α) with p(θs|α).

When applying topic models in activity discovery, hyperparametersα, β, θs and activity topics zn are hidden and need to be inferred fromobserved activity primitives pn in the time segments s. TM parametersare optimized within a variational expectation-maximization (EM) al-gorithm that maximizes the likelihood ` of all time segments s ∈ {1...S}where each segment s includes Ps activity primitives:

`(α, β) =

S∏s=1

∫p(θs|α)

Ps∏n=1

K∑z=1

p(zsn|θs)p(psn|zsn, β)

dθs (5.1)

We used the implementation of [23] where hyperparameters are opti-mized iteratively within the EM algorithm based on observed activityprimitives pn in a corpus of time segments S. As initialization we usedα = 0.01 as suggested by [2]. Other TM parameters were initializedrandomly.

Applying LDA provides a (S×K) topic activation matrix Γ for eachsegment based on the estimated TM parameters. The rows of Γ consistof the K-dimensional activity topic proportions θs per time segment s.More detailed information on LDA can be found in the work of Blei etal. [23].

5.3.3 N-gram topic model (NTM)

Activity primitive sequences might be characteristic for an activitycomposite and thus provide useful information for the discovery. Here,

5.3. Activity discovery using topic models 113

1...S1...Pα

z p

θ

1...S1...Pz p

η

µ

Σ

ββ

LDA and NTM CTM

s s

Figure 5.3: Graphical models for LDA, NTM and CTM [24]. Unshadednodes denote hidden model parameters, shaded nodes correspondto observed primitives p, rectangulars illustrate replications for allprimitives p ∈ {1...Ps} in a time segment s ∈ {1...S}. Edges indicatedependencies between two parameters. The LDA and NTM modelvary in the activity primitive vocabulary.

we introduce the NTM, an extended version of the LDA-based TM con-sidering both, primitive histograms and activity primitive sequences.We expanded the primitive vocabulary with L n-grams, thus resultingin a primitive vocabulary of size N′ = N + L. N-grams were extractedfrom the primitive sequences separately for each channel x ∈ {1...X},after filtering replications of activity primitives.

In order to find meaningful n-grams, we extracted multi-activityprimitive sequences from a corpus of time segments [25]: LDA with Kactivity topics is applied to infer topic assignments zsn for each activityprimitive pn ∈ 1...Ps in segment s ∈ 1...S. Pairs v of activity primitives pand topics z were obtained according to:

v = (p1, z1), (p2, z2), (p3, z3), ..., (pPs , zPs ). (5.2)

Subsequently, a recursive permutation test [26] was run to identifynew n-grams from v until no further significant n-grams were added.The permutation test provides a score that measures the degree of de-pendency between activity primitives. N-grams showing a significantscore (p-value) are considered relevant. We considered only 2 and 3-grams, where each n-gram should occur at least once in each instanceof an activity.

The n-grams found in the primitive streams of each channel x wereupsampled to match the primitive frequency f and added as addi-tional channel x resulting in X′ = X + X2gram + X3gram = 3X channels.X′ included unigram primitive channels X, 2-gram channels X2gramand 3-gram channels X3gram. Subsequently, we created histogramsHX′s =

∑X′x=1 Hxs on the extended primitive vocabulary N′ = N + L as


detailed in Sec. 5.3.1. Activity primitive histograms Hxs were normedaccording to Hxs = Hxs/2|xs ∈ X2gram and Hxs = Hxs/3|xs ∈ X3gram toprevent prioritized weighing of L n-grams in the fused histograms HX′scompared to the N unigrams. Subsequently, LDA was applied to theprimitive histogram set HX′ .

5.3.4 Correlated topic model (CTM)

LDA and NTM lack the ability to capture correlations between differ-ent activity topics in a time segment. The CTM (see Fig. 5.3) couldovercome the assumption that topics occur independently in a timesegment by replacing the Dirichlet distribution for topics in LDA witha normal distribution. Hence, we investigate the potential of the CTM,which follows the principle listed below:

1. Equivalent to the procedure for LDA, each activity primitivepn n ∈ {1...Ps} in a time segment s is assigned to one topic zn withfixed multinomial PDF Mult(βzj, ...βzN), where βzj is the probabil-ity of each activity primitive j ∈ {1...N} in topic z.

2. For each time segment s, a PDF ηs describing the probability oftopic p(zn|ηs) of topic zn for primitive pn in time segment s. Theactivity topic distribution ηs is described by a normal distributionN(µ,Σ) with p(ηs|µ,Σ) for each segment s ∈ {1...S}.

Hyperparameters µ,Σ, β are estimated by applying an variational EM-algorithm to maximize the lower bound of the likelihood ` of all timesegments s ∈ {1...S} where each segment s includes Ps activity primi-tives:

`(µ,Σ, β) =

S∏s=1

∫p(ηs|µ,Σ)

Ps∏n=1

∑zK

n =1

p(zsn|ηs)p(psn|zsn, β)dηs. (5.3)

We used the implementation of [6] where hyperparameter optimiza-tion µ,Σ, β is done iteratively within the variational EM using randominitial values. Applying CTM segments s ∈ S provides a (S × K) topicactivation matrix Γ, which specifies the activity topic proportions foreach segment s and each topic k ∈ {1...K}. The CTM approach is de-scribed in detail in [6].

5.4. Dataset properties 115

5.3.5 Optimal segment size selection

Following the TM description above, the optimal setting for the seg-ment size DS depends on the targeted activity composite durations. Toestimate optimal DS , we introduce wi to weigh the mean duration µiand the duration’s standard deviation σi of an activity composite i:

wi =(1 − σi

µi)∑M

i=1 (1 − σiµi

)(5.4)

The optimal segment size would be the duration of every activity com-posite instance. To consider typical variations of composite durations,the weighted mean duration µw =

∑Mi=1 wiµi considers µi with smaller

σi as more important than µi with larger σi. Analog, the weightedstandard deviation can be denoted by σw =

∑Mi=1 wiσi.

5.4 Dataset properties

In this section, we introduce four essential properties that can be usedto describe the dataset complexity that may affect TM-based activitydiscovery.

5.4.1 Averaged primitive rate

The averaged primitive rate r is the ratio of the mean number of activityprimitives Ni per composite i normed to the total number of activityprimitives N:

r = 1 −1M

∑Mi=1 Ni

N. (5.5)

Since activity primitives are needed to infer activity composites, rprovides an intuitive quantification of the composite inference chal-lenge. Typical value ranges are 0 < r ≤ 1. E.g., for r→ 0, the inferencemust be made from the same set of activity primitives for each com-posite. Nevertheless, r does not reveal whether primitives specificallydescribe an activity composite.

5.4.2 Activity composite specificity

The activity composite specificity s denotes the similarity among acti-vity composites, when described by activity primitives. We derive s as


follows:

s = 1 −∑M

i=1(i − 1) · oi

N · (M − 1). (5.6)

The variable oi describes the number of activity primitives thatbelong to i different activity composites. The activity composite speci-ficity is normed to a range [0, 1]. The more similar composites are,the lower is the specificity s. A dataset, where all activity primitivesuniquely occur within one composite attains the maximum specificity,hence s = 1.

5.4.3 Activity primitive sequence similarity

To quantify repetitiveness of activity primitive sequences for an acti-vity composite m, m ∈ [1...M], we calculate the Levenshtein distanceLm(pi, p j) of all combinations of activity primitive sequences pi and p jwith i, j belonging to the Ym activity instances of the same activitycomposite m. The averaged Levenshtein distance Lm of the activitycomposite m is defined as:

Lm =

∑Ymi, j;i< j Lm(pi, p j)

Ym·(Ym−1)2

. (5.7)

Lm(pi, p j) is normalized to [0, 1] by Lm(pi, p j) =Lm(pi,p j)

maxlength(qi,q j). The over-

all sequence similarity q of a dataset is calculated from the averagedLevenshtein distances over all activity composites M:

q = 1 −∑M

m=1 Lm

M. (5.8)

A dataset would obtain perfect sequence similarity for all compos-ites, hence q = 1, if e.g., data recordings were scripted using a fixedaction protocol. In contrast, q = 0 indicates that nonrecurring activityprimitive sequence exists for all activity composites.

5.4.4 Composite-instance ratio

The composite-instance ratio a denotes the ratio of activity compositecount M and number of repetitions nm per activity composite. For un-balanced datasets that have varying activity composite instance counts,

5.5. Evaluation datasets 117

the lowest nm was considered:

a = 1 −M

minMm=1(nm)

where M ≤ nm. (5.9)

The composite-instance ratio can be used to describe the relativedata available to infer activity composites. If a = 0, the same numberof activity composites as composite instances are available. Typically,far more instances per composite should be available than compositesM, thus a > 0.

5.5 Evaluation datasets

We consider three publically available datasets to evaluate the dis-covery methods: DART, Ubicomp’08, and Opportunity. The datasetsshow distinct differences for the properties introduced in Sec. 5.4. Forall datasets, ground truth information at activity primitive and activitycomposite levels was available, which was used to assess the discoverymethods in this work. Table 1 lists all activity composites and activityprimitives for all datasets.

5.5.1 DART dataset

The DART dataset [27] contains ∼ 4.8 hours of data at a frequency of50 Hz from two participants. Each participant performed 10 iterationsof car assembly task sequences, which we consider as activity compos-ites. In total, there are 11 activity composites and 42 activity primitivesannotated. For unlabeled data, we introduced an unlabeled activitycomposite and an unlabeled activity primitive label.

5.5.2 Ubicomp’08 dataset

The Ubicomp’08 dataset [2] contains 84 hours of data annotations fromseven days for one test subject. The dataset is provided at a frequencyof 2.5 Hz. Data was annotated at two levels. Four daily routines plusan unlabeled routine were available and we considered as activitycomposites. In addition, 34 activity primitives plus an unlabeled classwere annotated.


Dataset

Activity

composites

Activity

primitives

DA

RT

[27]m

ountfront

door,m

ountback

door,test

frontdoor,

testback

door,testtrunk,m

ountbrake

light,test

hood,hood

rod,mount

water

tank,mountbar,m

ountlight(11)

openhood1,teeter

hood,closehood,fetch

rod,openhood2,installrod,fetch

tank,hand

screwtank,

pickupdriver4tank,

usedriver4tank,

returndriver4tank,

fetchbar,

handscrew

bar,pickup

driver4bar,use

driver4bar,return

driver4bar,fetch

light,installlight,pickupdriver

light,usedriver

light,returndriver

light,fetchfrontdoor,hang

infrontdoor,hand

screwfrontdoor,fetch

backdoor,hang

inback

door,handscrew

backdoor,pickup

rattle,userattle,return

rattle,openfrontdoor,

teeterfrontdoor,close

frontdoor,openback

door,teeterback

door,closeback

door,open

trunk,teetertrunk,closetrunk,fetchbrakelight,insertbrakelight,hand

screwbrake

light(42)U

bicomp’08

[2]com

muting,offi

cework,

lunchroutine,

dinneractivities

andunla-

beled(4)

drivingbike,driving

car,brushingteeth,personalhygiene,kneeling

,running,sit-ting

havinga

coffee,havingbreakfast,having

dinner,havinglunch,sitting

talkingon

phone,usingthe

toilet,sittingdesk

activities,standingtalking,standing

hav-ing

acoffee,queuing

inline,standing

talkingon

phone,standingusing

thetoilet,

walking,w

alkingw

hilecarryingsom

ething,walking

freely,washing

dishes,pickingup

mensa

food,lyingusing

computer,w

ipingthe

whiteboard,discussing

atwhite-

board,kneelingm

akingfire

forbarbecue,fanning

barbecue,washing

hands,settingthe

table,watching

movie,m

akingcoffee,attending

apresentation,preparing

foodand

unlabeled(34)

Opportunity

[28]relaxing,

coffeetim

e,early

morning,cleanup,

sandwich

time

(5)

stand,walk,sit,lie,unlock,stir,lock,close,reach,open,sip,clean,bite,cut,spread,

release,move,bottle,salam

i,bread,sugar,dishwasher,sw

itch,milk,draw

er3(low

er),spoon,knife

cheese,frawer2

(middle),table,glass,cheese,chair,door1,door2,plate,

drawer1

(top),fridge,cup,knifesalam

i,lazychair(40)

Table5.1:D

atasetsused

inthis

work

toassess

discoveryperform

anceof

differentTM

s.Activity

composites

andactivity

primitives

arelisted

with

thetotalinstance

numbers

inparentheses.

5.6. Evaluation methodology 119

5.5.3 Opportunity dataset

The opportunity dataset [28] consists of around 30 hours of data an-notations from 5 runs for 4 different participants. The participantsperformed activities of daily living (ADL) and were recorded usingvarious sensors at a frequency of 30 Hz. Data was annotated at 3layers, high-level activities, medium-layer activities, including armmovements, and low-level activities, comprising left and right handmovements and object usage. In addition, locomotion was annotated.In this work, we considered the high-level activities as activity com-posites and activity primitives in 5 parallel channels: locomotion (4instances), right and left arm movements (13), right and left hand ob-ject use (23 instances). In total, the dataset lists 5 activity compositesand 40 activity primitives. We introduced an unlabeled class for both,activity composites and activity primitives for time intervals, whereno annotations were given.

5.6 Evaluation methodology

5.6.1 TM parameter selection

In order to investigate TM and k-means performances, we analyzed theeffect of segment size DS and the number of topics K. For each dataset,the segment size DS was varied to cover at least the minimum averageactivity composite duration µmin and the maximum µmax. The numberof topics K was varied around the number of activity primitives and thenumber of activity composites in each dataset, respectively. For all sub-sequent analyses, we set the parameters as follows: for the Ubicomp’08,DS = 30 min and K = 10 according to [2], for DART, DS = 0.7 min andK = 22, and for Opportunity K = 10 and DS = 2.5 min.

5.6.2 TM performance estimation

To obtain stable TM estimations, three training runs were performedfor each parameter setting and the run yielding the highest modellikelihood was selected for evaluation. Primitives that occur in almostevery segment or rarely in any should be removed as those primi-tives contain no additional information [29]. Thus, the ”unlabeled”primitive and activity primitives occurring less than 0.5% of all primi-tives (without the unlabeled primitive) were disregarded. Activationsfor empty segments were set to zero in the topic activation matrix Γ.


We applied Borda Count ranking [30] to topic activations of all over-lapping time segments that covered the same actual time due to thesliding windowing. The resulting total topic activation vectors wereupsampled to match the ground truth frequency. Typically, the numberof activity topics K is higher than the number of activity compositesM as suggested by [2] because of variability of activity composites interms of activity primitive composition. In order to compare the per-formance of different discovery approaches, we used topic activationsΓ as feature matrix to train a kNN classifier mapping K activity top-ics to M activity composites. We used the topic activation matrix Γestobtained from TM estimation and the corresponding activity compos-ite groundtruth to train a kNN classifier (k=1). We then classified thetopic activation matrix Γin f that resulted from topic model inferencefor unseen segments using the trained kNN. Classification was basedon the minimal Euclidean Distance d(Γin f ,Γest).

5.6.3 Evaluation strategy

The three datasets considered in this work contained data from differ-ent participant numbers and had different activity composite instancecounts. We adjusted our evaluation strategy to subsequently compareTM performances across the different datasets: for DART, we useda 10-fold cross-validation of all participants’ repetitions since activi-ties were scripted and activity primitives are independent of the indi-vidual participant. For Ubicomp’08, we applied a participant-specificleave-one-day-out cross-validation. For Opportunity, we evaluated aparticipant-specific leave-one-iteration out cross-validation with fiveiterations and averaged classification performances across all partici-pants. The unlabeled activity composite data was included for TM andk-means model estimations and inference, but left out for kNN modelestimation. To measure the probabilistic deviations due to randominternal model initializations, we repeated each TM calculation fivetimes. The averaged class-specific accuracy and standard deviation ofthe five runs were analyzed. We performed a McNemar χ2-square test(with Yate’s continuity correction) to reveal significant accuracy differ-ences between discovery methods. Due to space constraints we onlyconsidered two tests for each dataset. Test 1 was used to assess sig-nificance of performance results between the best performing TM andbaseline accuracy. Test 2 was used analog to compare the best perform-ing TM (NTM or CTM) and the standard TM LDA. Both McNemar tests

5.7. Results 121

were performed for the empirically best parameters (DS,K) for TMson each dataset. For the McNemar test 1 we took false positives fromthe best performing TM which were at the same time true positives forbaseline into account and inverse (accordingly for test 2). The level ofsignificance was set to p = 0.05 which equals χ2 > 3.84.

5.6.4 Analysis of primitive recognition noise

We evaluated the effect of primitive recognition noise on discoveryperformance of baseline and TMs. For this purpose, we varied dele-tion and insertion errors of activity primitives between 0 and 100%.E.g., 100% deletions would indicate a total failure of an activity prim-itive recognizer. In addition, we evaluated the noise effect separatelyfor most relevant and the least relevant activity primitives of eachdataset. To determine primitive relevance, we ranked activity primi-tives in a dataset: activity primitives typically occur in one or moreactivity composites. The higher the number of different activity com-posites that an activity primitives occurs in, the higher the primitiverelevance. Ties (primitives occurring in the same number of compos-ites) were resolved by ranking the primitive higher that occurred morefrequently in the dataset. For this analysis we used the empirically bestTM parameters (DS,K).

5.7 Results

5.7.1 Dataset properties analysis

Figure 5.4 lists the dataset properties for DART, Ubicomp’08, and Op-portunity datasets following the definitions in Sec. 5.4. The averagedprimitive rate was low for Ubicomp’08 (r = 0.25), followed by Oppor-tunity (r = 0.58) and DART (r = 0.87). Furthermore, the Opportunitydataset showed the lowest activity composite specificity s = 0.39, in-dicating that the activity composites have similar activity primitivecontent. In contrast, DART resulted in s = 0.99. Moreover, DART isthe only dataset with high primitive sequence similarity (q = 1). Over-all, the dataset complexity plots indicate that Opportunity is the mostchallenging dataset for discovery, followed by Ubicomp’08. DART pro-vides conveniently separable activity composites.

122 Chapter 5: Parametric topic models for activity discoveryDataset Averaged

primitive rateActivity com-posite speci-ficity s

Activity prim-itive sequencesimilarity q

Composite-instance ratio a

DART 1-5.27/42=0.87 0.99 1.0 0.45Ubicomp’08 1-9.75/13=0.25 0.79 0.36 0.43Opportunity 1-15.4/37=0.58 0.39 0.19 0

1

s

ar

q

0 1

0

01

s

ar

q

0 1

0

1 01G

s

ar

q

0

0

01

1

0

Ubicomp’08Dart Opportunity

1 1 11

00

Figure 5.4: Top panel: dataset properties for DART, Ubicomp’08 andOpportunity. Bottom panel: dataset complexity plots, illustrating theactivity discovery complexity using dataset properties. The larger thearea of the quadrangle in the complexity plot, the more challenging isthe activity discovery under the composite/primitive abstraction.

5.7.2 Selection of segment size and number of activity topics

Figure 5.5 shows the TM performance while varying segment sizesDS for all datasets. All TM approaches showed an accuracy maximumclose to the weighted mean and standard deviation µw ± σw of thedatasets’ activity composite durations. For very large DS, performancedropped for all datasets.

Figure 5.6 shows the TM performance while varying the numberof activity topics. All TM approaches show a similar trend on: per-formances increase with K until saturation. We assume a saturationpoint Ks, such that variability in accuracy for K > Ks was less than 2%.For DART, saturation occurred at Ks ≈ 22, for Opportunity at Ks ≈ 7.Both results are close to the lower bound M. In contrast, Ubicomp’08showed saturation at Ks ≈ 13 only, matching the number of activityprimitives N in the Ubicomp’08 dataset.

5.7.3 Performance of TMs compared to baseline

For DART, the baseline k-means clustering outperformed the LDA ap-proach for low topic count. Although DART showed high primitive

5.7. Results 123

0 0.5 1 1.5 2404550556065707580859095

100

Aac

cura

cy [%

]

DartKmeans LDA NTM CTM

0 15 30 45 60 75 90 105

Ubicomp‘08

Segment size DS [min]0 1 2 3 4 5

Opportunity

µmin µmin µmin µmax µmax µmax

µw±σw µw±σw µw±σw

Figure 5.5: Average class-specific accuracies and standard deviation forDART (K=22), Ubicomp’08 (K=10), and Opportunity (K=10) datasets,while varying segments sizes DS between µmin and µmax of M activitycomposites. The weighted mean duration of all activity compositesµw ± σw indicates optimal parameter settings per dataset.

sequence similarity (q = 1), the NTM approach did not enhance per-formance compared to LDA. At empirical-optimal TM parameters thebest performing TM was CTM and the baseline showed similar ac-curacy and no statistically significant difference was found (χ2 = 1.7),while LDA performed significantly worse than CTM (χ2 = 10.5). How-ever, for large segment sizes (DS > 1 min) the baseline performancedropped and the TMs gained up to 10% in accuracy.

All TM approaches outperformed the baseline in the Ubicomp’08and Opportunity datasets. Especially NTM showed a 8% statisticallysignificant performance increase on Ubicomp’08 for empirical-optimalsegment sizes (χ2 = 4.6) and even outperformed the standard TM LDA(by 6%, χ2 = 12.3). Here, the primitive sequence similarity contributedvaluable information to structuring activity composites. Compared toLDA, the CTM showed increased performance for larger segment sizeson Ubicomp’08.

For Opportunity, 17% were gained statistically significantly by LDAand CTM compared to the baseline given an empirical-optimal seg-ment size setting (χ2 = 35.8). However, the CTM performed similar toLDA (χ2 = 0.4). NTM performed about 10% worse than LDA. For largersegment sizes (DS > 1 min), the baseline did not discover the shortest


0 10 20 30 40404550556065707580859095

100

Dart

Acc

urac

y [%

]Kmeans LDA NTM CTM

0 5 10 15

Ubicomp‘08

Number of Activity Topics K0 10 20 30

Opportunity

M MMN NN

Figure 5.6: Average class-specific accuracies and standard deviationfor DART (DS = 0.7 min), Ubicomp’08 (DS = 30 min) and Opportu-nity (DS = 2.5 min) datasets, while varying numbers of activity topics Kbetween M (number of activity composites, lower bound) and N (num-ber of activity primitives). The closer K is set towards the lower boundM, the more likely activity topics will resemble directly to activitycomposites.

activity composite relaxing that had a mean duration of µ = 0.6 min.

5.7.4 Effect of activity primitive noise on discovery

Figure 5.7 depicts the noise-related performance decrease of baseline,all TM approaches and all datasets. In all datasets, the k-means clus-tering was more sensitive to noise than LDA. Especially for inser-tions, accuracy of k-means decreased already at small activity primi-tive noise (Opportunity: 10%, DART: 20%) regardless of the share ofactivity primitives affected (most important). LDA showed to be re-sistant against deletion errors: accuracy decreased only for deletionsabove 80% for Ubicomp’08 and Opportunity, and above 40% for DART.DART and Ubicomp’08 datasets were robust against insertion noisecompared to Opportunity, where LDA accuracy dropped starting from20% insertions for high percentages of activity primitives affected.

Compared to LDA, NTM is more sensitive to noise. This resultcould indicate that the primitive sequence was disturbed by deletionsand insertions and thus no meaningful n-grams were found. However,for DART, noise did not influence NTM accuracy. CTM showed to be

5.7. Results 125


Leas

t [%

]

Mos

t [%

] Dart kmeans

10080 40 20 10 0 10 20 40 80100

803020100

10203080


Leas

t [%

]

Mos

t [%

] Ubicomp08 kmeans

10080 40 20 10 0 10 20 40 80100

803020100

10203080


Leas

t [%

]

Mos

t [%

] Opportunity kmeans

10080 40 20 10 0 10 20 40 80100

803020100

10203080


Leas

t [%

]

Mos

t [%

] Dart LDA

10080 40 20 10 0 10 20 40 80100

803020100

10203080


Leas

t [%

]

Mos

t [%

] Dart NTM

10080 40 20 10 0 10 20 40 80100

803020100

10203080


Leas

t [%

]

Mos

t [%

] Dart CTM

10080 40 20 10 0 10 20 40 80100

803020100

10203080


Leas

t [%

]

Mos

t [%

] Ubicomp08 LDA

10080 40 20 10 0 10 20 40 80100

803020100

10203080


Leas

t [%

]

Mos

t [%

] Ubicomp08 NTM

10080 40 20 10 0 10 20 40 80100

803020100

10203080


Leas

t [%

]

Mos

t [%

] Ubicomp08 CTM

10080 40 20 10 0 10 20 40 80100

803020100

10203080

Deletions [%] Insertions [%]Le

ast [

%]

M

ost [

%] Opportunity LDA

10080 40 20 10 0 10 20 40 80100

803020100

10203080


Leas

t [%

]

Mos

t [%

] Opportunity NTM

10080 40 20 10 0 10 20 40 80100

803020100

10203080


Leas

t [%

]

Mos

t [%

] Opportunity CTM

10080 40 20 10 0 10 20 40 80100

803020100

10203080

-4%

-8%

-6%

-2%

0%

<-10%

Figure 5.7: Relative activity discovery performance under primitiverecognition noise using k-means clustering, LDA, NTM, and CTMapproaches for DART, Ubicomp’08, and Opportunity datasets. Activityprimitive deletions and insertions were analyzed separately for themost and least relevant activity primitives in the datasets. LDA showedmost robust performance.


more sensitive to noise compared to LDA on all datasets, especially fornoise on the least relevant activity primitives in Opportunity.

5.8 Discussion

Results showed that for all datasets the TM approaches could out-perform our baseline k-means clustering for large segment sizes. Thiseffect is probably due to more diverse activity primitive histogramswith increasing DS. The discovery performance of all methods washighest for the DART dataset (∼ 95%), followed by Ubicomp’08 (∼80%) and Opportunity (∼ 65%). We attribute the performance differ-ences between the three datasets to the different dataset complexitiesthat showed the same relation. For DART, complexity was lowest asmeasured by the dataset properties (see Fig. 5.4), followed by Ubi-comp’08 and Opportunity.

5.8.1 Selecting optimal TM parameters

The segment size DS is an important parameter to be set when apply-ing TMs in activity discovery. Following the hypothesis that an activitycomposite should be specific in the representation by activity primi-tives, an optimal segment size would be the duration of every activitycomposite instance. However, an a-priori knowledge of the activitycomposite duration is typically not given in activity discovery. The re-sults in this work across all datasets indicate that the empirical-optimalDS could be chosen around the weighted mean duration µw ± σw ofactivity composites. When applying activity discovery to unknowndata this value needs to be estimated. Expected mean and varianceof activity composite durations could be approximated from availabledatasets with similar properties. Clearly, future TM frameworks wouldbenefit from an extension to set DS dynamically based on data.

The number of activity topics K depended on the diversity of pat-terns available in the dataset. A low K, close to the number of activitycomposites M, would reveal more general topics. When K > M, i.e. Kapproaches the number of activity primitives N would find more spe-cific activity topics. Clearly, K = M would be an ideal choice. The higherthe number of topics, the less apparent is the activity topic to activitycomposite mapping. If K → N, the discovery method transforms acti-vity primitive histograms only and becomes redundant at K = N. Theaccuracy saturation points Ks showed Ks > M for all datasets, suggest-

5.8. Discussion 127

ing that the targeted number of activity composites may not representthe dataset structure. For example, unlabeled data that includes non-categorized activity patterns may require additional activity topics. Asreal-life datasets typically contain unlabeled data and the compositedistribution is not known a priori, we used activity primitives withinunlabeled composites for TM and k-means topic discovery. However,the kNN based mapping of topics and known activity composites wasapplied excluding the unlabeled class. Unlabeled data in our datasetsmay have influenced the parameter analyses. We analyzed statistics ofthe unlabeled data in the datasets to provide convenient comparisonfor subsequent investigations. Specifically, we considered the ratio ofunlabeled data vs. total data size (Dart: 0.15, Ubicomp’08: 0.26, Oppor-tunity 0.18). To assess the occurrence frequency of unlabeled data inthe dataset, we derived the ratio of unlabeled data transitions vs. totalnumber of label transitions: 0.15 for Dart, 0.49 for Ubicomp’08 and 0.53for Opportunity.

For the Ubicomp’08 dataset, we found Ks = N, thus one could con-clude that the activity composites targeted for this dataset may notrepresent the data structure sufficiently. Rather, the four activity com-posites appear too broad compared to the variety of topics discoveredin the dataset. In general, setting K is a tradeoff between represent-ing all available activity composites in a dataset, and the discoverymethod’s ability to map activity topics to activity composites. For ac-tivity discovery we recommend to select K close to the number ofexpected activity composites M, but allow K > M to cover unknowntopics.

The guidelines for optimal TM parameters in our work focusedon choosing DS and K. Our results provide formal and empirical pa-rameter bounds but may not exactly match the optimum values, asparameters highly depended on the dataset itself. Due to the limitedavailability of datasets that provide annotations at two granularitylevels only three datasets could be investigated.

5.8.2 Selecting a TM approach

Our results showed that TMs can provide better activity discoveryperformance compared to k-means clustering. However, the actualgain of a TM compared to clustering and the best TM choice dependson the dataset and its properties. For DART, the TM approaches in thiswork did not outperform baseline k-means clustering. The activity


composite specificity was found to be high and almost no overlapbetween activity primitives was derived (s = 0.99). In this dataset,the structure of activities is obvious from the primitives and thereforeTMs cannot attain advantages. However, the conditions change if morerealistic primitive recognition noise is added (see further below). Fordatasets with lower activity composite specificity, TMs could clearlyenhance discovery performance. The less specific activity compositesare, the higher are the discovery performance gains. For Ubicomp’08,the gain was about 8% (s = 0.79) and for Opportunity, ∼17% (s = 0.39)at the optimal segment size. Thus, TMs may only bring advantages forcomplex datasets in terms of their dataset properties.

In this paper, we introduced and analyzed the NTM to leverageactivity primitives sequence similarity for activity discovery. Resultsindicated that for a dataset with perfect primitive sequence similar-ity (DART) the NTM did not enhance discovery performance. Similarobservations were made with the same dataset earlier when investi-gating composite recognition [27]. We assume that the high compositespecificity in DART could yield the maximum discovery performance,hence sequence information does not provide further gains. For theUbicomp’08 dataset, sequence information helped to increase perfor-mance by 5% compared to LDA. The NTM found valid n-grams, as e.g.walking - using toilet - washing hands. On the contrary, for Opportunitythe NTM performed worse than LDA.

The CTM considered correlations of different activity topics ap-pearing in the same time segment. Generally, the CTM did not enhanceperformance compared to LDA. In order to reveal correlations, corre-lated activity topics would need to occur frequently in the same timesegment, which most probably did not. Only for large segment sizesin Ubicomp’08, the CTM showed increased discovery performance. Athigher segment sizes the probability for co-occurring topics in a timesegment increases and thus benefit the CTM.

5.8.3 Performance under noise

Our results revealed that LDA was the most robust TM under noisyprimitive input. Overall, LDA showed better performance comparedto k-means clustering (see Fig. 5.7). The NTM was more sensitive tonoise as the activity primitive sequence similarity was disturbed andthus no meaningful n-grams could be found to enhance accuracy (seeFig. 5.7). Filtering of activity primitive sequences might reduce the

5.9. Conclusion 129

impact of noise on the sequence similarity. While the CTM showedsimilar or even better discovery performance compared to the LDAunder ideal input, the CTM was less robust against noise. This result isprobably due to additional estimation complexity in the CTM method.

For LDA, insertions did not affect performance for DART and Ubi-comp’08 datasets. This result suggest that there is a robust underlyingactivity composite structure, denoted by the activity primitives. How-ever, discovery performance declined for Opportunity, most probablydue to the low activity composites specificity of this dataset (s = 0.39).These results suggest that for datasets with high or moderate com-posite specificity (Dart: s = 0.99 and Ubicomp’08: s = 0.79 ) LDA ismore robust against false positives than false negatives. Subsequently,activity primitive recognizers should be tuned for high recall, whileprecision could be reduced without negative effects on the discoveryperformance. In contrast, for datasets with low composite specificity(Opportunity), primitive recognizers should provide high precisionand high recall to ensure discovery performance.

The noise investigations were focused on type 1 (insertions) andtype 2 (deletions) errors, the basic statistical error types as they occurfor primitive spotting and detectors. For primitive classifiers also sub-stitutions and timing errors are likely. However, timing errors wouldonly influence results if the timing shift was across an analysis frame ofsegment size DS. However, DS is typically large to capture multiple ac-tivity primitives, such that timing errors may only have minor effect onthe overall performance. Substitutions are defined by insertions anddeletions at the same time. The probability for substitutions amongdifferent classes and thus the noise model itself highly depends on theapplied classifier. To provide a classifier independent noise model withlittle complexity we focused on insertions and deletions. An empiricallower bound for the substitution error could be approximated fromthe insertion and deletion errors presented in our analyses.

5.9 Conclusion

In this work, we investigate three topic modeling approaches based onLDA, NTM, CTM, in three public activity datasets to obtain guidelinesfor TM parameter selection depending on the dataset properties. TheNTM was introduced in this work to incorporate frequently occurringprimitive sequence similarity information into the discovery process.The CTM was introduced to deal with co-occurring topics. We com-


pared the TM approaches among themselves and to a k-means cluster-ing to support model selection under ideal and noisy primitive inputconditions. Results suggest, that a segment size within the weightedmean and standard deviation of activity composite durations providesclose-to-optimal settings across all datasets. The number of activitytopics should be selected close to the number of expected activity com-posites to maximize the discovery method’s benefits. Choosing highersettings for the activity topics results in more redundancy in the TMoutput.

Our results showed that TMs can provide better discovery perfor-mance compared to a k-means clustering, with performance increasesup to 17% in complex dataset. However, for datasets with obvious acti-vity composite structures, k-means clustering was sufficient. Primitivesequence similarity information showed to be an important additionto TM-based discovery for complex datasets.

The CTM yielded similar performance as LDA for ideal primitiveinputs. However, the CTM was more sensitive to noise compared toLDA. The NTM showed high sensitivity to noise, deletions and in-sertions disturbed the primitive sequence similarity. Yet, filtering ofactivity primitive sequence streams could help to overcome noise sen-sitivity. Overall, the LDA approach performed best on all datasetsunder both, ideal and noisy activity primitive input.

5.10 Acknowledgment


Bibliography

[1] D. M. Blei, “Probabilistic topic models,” Communications of theACM, vol. 55, no. 4, pp. 77–84, 2012.



[4] P. Rashidi, D. J. Cook, L. B. Holder, and M. Schmitter-Edgecombe,“Discovering activities to recognize and track in a smart envi-ronment,” IEEE Transactions on Knowledge and Data Engineering,vol. 23, no. 4, pp. 527–539, 2011.

[5] R. Ali, M. ElHelw, L. Atallah, B. Lo, and G. Yang, “Pattern miningfor routine behaviour discovery in pervasive healthcare environ-ments,” in International Conference on Information Technology andApplications in Biomedicine., pp. 241–244, IEEE, 2008.

[6] D. Blei and J. Lafferty, “Correlated topic models,” Advances inNeural Information Processing Systems, vol. 18, pp. 147–154, 2006.

[7] J. Seiter, O. Amft, and G. Tröster, “Assessing topic models: howto obtain robustness?,” in First Workshop on Recent Advances inBehavior Prediction and Pro-Active Pervasive Computing, Pervasive,2012.





[11] X. Wang, X. Ma, and E. Grimson, “Unsupervised activity per-ception by hierarchical bayesian models,” in IEEE Conference onComputer Vision and Pattern Recognition, pp. 1–8, IEEE, 2007.

[12] A. Bissacco, M.-H. Yang, and S. Soatto, “Detecting humans viatheir pose,” Advances in Neural Information Processing Systems,vol. 19, pp. 169–176, 2007.


[14] R. Hamid, S. Maddi, A. Johnson, A. Bobick, I. Essa, and C. Isbell,“A novel sequence representation for unsupervised analysis ofhuman activities,” Artificial Intelligence, vol. 173, no. 14, pp. 1221–1244, 2009.

[15] V. Bettadapura, G. Schindler, T. Plötz, and I. Essa, “Augmentingbag-of-words: data-driven discovery of temporal and structuralinformation for activity recognition,” in IEEE Conference on Com-puter Vision and Pattern Recognition, pp. 2619–2626, IEEE, 2013.

[16] K. Farrahi and D. Gatica-Perez, “Discovering human routinesfrom cell phone data with topic models,” in Proceedings of the 12thIEEE International Symposium on Wearable Computers, pp. 29–32,2008.


[18] B. Clarkson and A. Pentland, “Unsupervised clustering of ambu-latory audio and video,” in Proceedings of the International Confer-ence on Acoustics, Speech, and Signal Processing, vol. 6, pp. 3037–3040, IEEE, 1999.

133

[19] R. Hamid, S. Maddi, A. Johnson, A. Bobick, I. Essa, and C. Is-bell, “Unsupervised activity discovery and characterization fromevent-streams,” in Proceedings of the 21st Conference on Uncertaintyin Artificial Intelligence, 2005.

[20] K. Farrahi and D. Gatica-Perez, “Extracting mobile behavioralpatterns with the distant n-gram topic model,” in 16th InternationalSymposium on Wearable Computers, pp. 1–8, 2012.

[21] H.-b. Tu, L.-m. Xia, and Z.-w. Wang, “The complex action recog-nition via the correlated topic model,” The Scientific World Journal,vol. 2014, p. 10, 2014.

[22] Y. Wang and G. Mori, “Human action recognition by semilatenttopic models,” IEEE Transactions on Pattern Analysis and MachineIntelligence, vol. 31, no. 10, pp. 1762–1774, 2009.


[24] D. M. Blei and J. Lafferty, “Topic models,” Text Mining: Classifica-tion, Clustering, and Applications, vol. 10, pp. 71–94, 2009.

[25] D. M. Blei and J. D. Lafferty, “Visualizing topics with multi-wordexpressions,” 2009.

[26] E. J. Pitman, “Significance tests which may be applied to samplesfrom any populations: Iii. the analysis of variance test,” Biometrika,pp. 322–335, 1938.



[29] D. M. Blei and J. D. McAuliffe, “Supervised topic models.,” inNeural Information Processing Systems, vol. 7, pp. 121–128, IEEE,2007.



6Topic models: the

influence ofhyperparameters

Julia Seiter, Adrian Derungs, Corina Schuster-Amft, Oliver Amft and Ger-hard Tröster

Full publication title: activity routine discovery in strokerehabilitation patients without data annotation.

Proceedings of the 8th International Conference on PervasiveComputing Technologies for Healthcare, 2014

DOI: 10.4108/icst.pervasivehealth.2014.255275

136 Chapter 6: Topic models: the influence of hyperparameters

Abstract

In this work, we investigated whether activity routines of stroke rehabilitationpatients can be discovered from body-worn motion sensor data and withoutdata annotation using topic modeling. Information about the activity routinesperformed by stroke patients during daily life could add valuable informationto personal therapy goals. As topic model input, we used a set of activityprimitives derived from upper and lower extremity motion sensor data. Wemonitored three stroke patients during their daily life in a day care center for8 days each within 3 weeks. We achieved up to 88% accuracy for activityroutine discovery for subject-dependent evaluations. Our discovery approachseems suitable for activity routine discovery in rehabilitation patients.

6.1 Introduction

Stroke is considered to be the most leading cause of disability in theworld [1]. To assess the physical impairment of stroke patients ther-apists use motor function tests, as the Fugl-Meyer and the Chedoke-McMaster Stroke Assessment [2, 3]. However, these clinical assess-ments can only be applied during therapy and do not provide infor-mation on patients’ activities and routines in daily life. Several ap-proaches exist to implement objective activity measurements in dailylife [4–6]. Patel et al. estimated the total Functional Ability Scale scoreof patients from acceleration data [4]. Uswatte et al. used accelerom-eters at the wrists to infer the activity of the impaired arm [6]. Whilethese approaches provide quantitative scores and activity measures,they do not yield insight in the type of activities and daily routinesthat patients performed. Nevertheless, monitoring patients’ behaviorand activity routines outside therapy could add valuable informationon describing patient lifestyle and thus defining individual therapygoals.

Activity discovery has been proposed to identify daily routines,without previously trained classifiers [7, 8]. However, activity routinediscovery needs to deal with variations in routine performance fromday to day and between persons. A commonly considered approach toactivity routine discovery is to compose routines from activity prim-itives. Activity primitives have a fine temporal granularity and canbe recognized from on-body sensor data. Huynh et al. applied topicmodels to discover activity routines, such as lunch and office work fromactivity primitives [9]. They used activity primitives including desk acti-

6.2. Methodology 137

vity and having lunch, which were classified using a previously trainedNaïve Bayes model. These activity primitives were subsequently con-sidered as input for the topic model. Since the activity primitives re-quired a trained classifier, annotations of actual activity performanceswould be needed during the classifier training. In contrast, we inves-tigate whether the activity routines of rehabilitation patients could bediscovered from activity primitives without the need of trained classi-fiers. Our approach is based on person-independent body posture andactivity features measured at the extremities.

In this work, we investigate a topic model-based activity routinediscovery in rehabilitation patients using wearable motion sensors.Our approach does not require annotations for activity primitives androutines. In particular, the contributions of this paper are the follow-ing: (1) we show that activity routines can be discovered from a setof rule-based, person-independent activity primitives that do not re-quire trained classifiers and activity annotations. (2) We analyzed theinfluence of the key topic model parameters document size, number ofactivity topics, and the hyperparameter α, on the activity routine dis-covery performance. (3) We evaluated our approach with three strokepatients who were recorded in a day care center using wearable motionsensors during 8 days within 3 weeks.

6.2 Methodology

For activity routine discovery, we used a layered approach as illus-trated in Figure 6.1a. Activity primitives were derived from sensordata according to a set of rules. Subsequently, the activity primitiveswere used as input for topic model based activity routine discovery.Activity topic activations (probabilities) were then mapped to distinctactivity routines. The study recording process and the different layersof the discovery approach are detailed in the following.

6.2.1 Study recordings

In our monitoring study, we included three male stroke patients, aged47-57 years. The patients regularly visited the day care center of theReha Rheinfelden rehabilitation center in Switzerland. Patients sufferedfrom hemispheric stroke resulting in either left or right upper andlower extremity activity impairment. Two of three patients primarilyused a wheelchair but were capable of short distance walking. Patients


Table6.1:

Activity

primitives

(1)-(36)detected

atthe

low-level

activityprim

itivedetector:

Thetable

shows

featurew

indowsizes/steps,thresholds

ofbinaryfeatures

FSi inferred

fromthe

3-axisacceleration

signalaccxyzand

quaternions(q

Si ,qS

j )forsensors

Si,j|∈{S1,S2..,S6

}asw

ellasdetection

logicfor

eachactivity

primitive.

Activity

Primitives

Window

Binary

FeatureD

etectionLogic

(1)activityboth

arms,(2)activity

affectedarm

,1s/1s

F=

1|σ

2(||accxyz||)≥

0.05,(1):F

S2 FS5

(2):FS2 F

S5 ,(3)activity

non-affectedarm

,(4)noarm

activity(3):F

S2 FS5 ,(4):F

S2 FS5

(5)activityboth

legs,(6)activityaffected

leg,1s/1s

F=

1|σ

2(||accxyz||)≥

0.05,(5):F

S3 FS6

(6):FS3 F

S6 ,(7)activity

non-affectedleg,(8)no

legactivity

(7):FS3 F

S6 ,(8):FS3 F

S6

Peakin

lowfrequency

band:(9)botharm

s,5s/1s

F=

1|m

ax{FFT

(accxyz )}∈[0.2,2.5]H

z(9):F

S2 FS5

(10):FS2 F

S5 ,(10)affected

arm,

(11)non-affectedarm

,(12)none

(11):FS2 F

S5 ,(12):FS2 F

S5

Peakin

lowfrequency

band:(13)bothlegs,

5s/1sF

=1

|max{FFT

(accxyz )}∈[0.2,2.5]H

z(13):F

S3 FS6

(14):FS3 F

S6 ,(14)affected

leg,(15)non-affectedleg,(16)none

(15):FS3 F

S6 ,(16):FS3 F

S6(17)body

movem

ent,120s/1s

F=

1|σ

2(||accxyz||)≥

0.05(17):F

S2∨

FS3∨

FS5∨

FS6 ,

(18)nobody

movem

ent(18):F

S2 FS3 F

S5 FS6

(19)stand,(20)sit1s/1s

F=

1|µ(‖accy

‖)>µ(‖accz

‖),(19):F

S3 FS6

(20):FS3∨

FS6

Wristorientation

affectedarm

(21)horizontal,1s/1s

F=

1|µ(‖accz

‖)>µ(‖accx

‖)(21):F

S5(22):F

S5

(22)vertical,(23)-(24)non-affectedarm

analogue(23):F

S2(24):F

S2

Arm

postureaffected

arm(25)adducted,

1s/1sF

1=

1|arccos( ⟨q

Si ,qS

j ⟩)<

60◦

(25):F1S4,S5

(26):F2S4,S5

(26)90◦angle,(27)streched,

F2

=1

|60◦≤

arccos( ⟨qSi ,q

Sj ⟩)<

120◦

(27):F3S4,S5

(28):F1S1,S2

(28)-(30)non-affectedarm

analogueF

3=

1|arccos( ⟨q

Si ,qS

j ⟩)≥

120◦

(29):F2S1,S2

(30):F3S1,S2

Lower

armorientation

affectedarm

1s/1sF

1=

1|atan2(accy ,

||accxz||)<

60◦

(31):F1S5

(32):F2S5

(33):F3S5

(31)down,(32)horizontal,(33)up,

F2

=1

|60◦≤

atan2(accy ,||accxz

||)<

120◦

(34):F1S2

(35):F2S2

(36):F3S2

(34)-(36)non-affectedarm

analogueF

3=

1|atan2(accy ,

||accxz||)≥

120◦


arrived in the day care center in the morning, followed their dailytherapy schedule including lunch and resting phases and went homein the evening. In the morning, 6 Shimmer3 motion sensors were at-tached to each wrist, upper arm and upper leg as illustrated in Figure6.1b and logged acceleration, gyroscope and magnetometer data at 50Hz. Sensors were carried during the whole day (except for temporaryremoval during water therapy) and detached before patients left theday care center in the evening. The study was approved by the localcantonal Ethics committee.

For recording days, activity routine annotations were extractedfrom the individual daily therapy schedule by assigning each ther-apy to one of the activity routines. Activity routines included cognitivetraining (covering training exercises on a computer or working sheet)socializing (active interaction with other people) motor training (ther-apies that involve physical motor function training exercises) medi-

Figure 6.1: a) Overview of activity routine discovery approach: Activityprimitives are detected from sensor data S1−S6. Subsequently activitytopics are discovered from activity primitives followed by kNN basedmapping of activity topics to activity routines, b) sensor setup withaffected arm left.


Table 6.2: Number of activity routine repetitions recorded for eachpatient in 8 days within 3 weeks.

Patient 1 Patient 2 Patient 3 Total

Cognitive train-ing

3 1 5 9

Socializing 34 33 35 102Motor training 13 20 14 47Medical fitness 2 8 7 17Rest/sleep 1 6 4 11

cal fitness (intense physical training exercises) and rest/sleeping phases.Rest/sleeping phases were not specified in the therapy schedule butperformed during breaks in the day care center. Thus, time and dura-tion of rest/sleeping were added to the therapy schedule (hand written)by therapists and the study examinators. In total, we collected 137hours of data, of which 100 hours were annotated. For each patient8 days within 3 weeks were recorded. Depending on patients’ per-sonal therapy schedule, only a subset of activity routines might beperformed regularly. Table 6.2 shows the routines and number of rep-etitions recorded for each patient.

6.2.2 Activity primitive detector

In total, we derived 36 activity primitives from upper arm and lowerarms and thigh-worn sensors. Activity primitives were described bybinary decisions on particular arm and body postures as well as armand leg movement as detailed in Table 6.1. We considered arm andleg activity including affected an non-affected extremities to be indica-tors for the physical activity of patients and thus relevant for activityroutine discovery. We further distinguished between activity in thelow frequency band (0.2− 2.5Hz) and outside the low frequency band.We also assumed body postures like sitting and standing and arm pos-tures such as the angle between upper and lower arm, wrist orientationand orientation of the lower arm towards the horizontal plane to beinformative for activity routine discovery in rehabilitation scenarios.

6.2.3 Activity topic discovery and mapping

We applied the Latent Dirichlet Allocation topic model (LDA) for acti-vity topic discovery as suggested by [9]. In activity routine discovery


the topic model is used to discover K hidden activity topics in a corpusof segments. Each segment covers histograms over activity primitivesequences of a time segment DS of a day. LDA assumes that for eachsegment s there is an activity topic distributions θs which is derivedfrom a Dirichlet distribution Dir(α) with the hyperparameter α. Whilea high α value should favor all activity topics in a segment equally,small α would privilege one particular activity topic for one segment.When applying LDA to a corpus of segments a K-dimensional activitytopic activation vector γs is inferred from the activity primitive his-togram of each segment s based on the estimated distribution θs. Thenormalized γs describes the estimated occurrence ratio of each activitytopic in a segment s. More detailed information on LDA can be foundin [10]. As suggested by [9] the number of activity topics K in the datacould be higher than the number of activity routines M. In order toassess and compare discovery accuracy across patients we applied akNN classifier for mapping activity topics to activity routines usingthe topic activation vector γs as feature vector.

6.2.4 Implementation

To evaluate the topic model we used the LDA implementation of [10],which includes an iterative optimization for topic model parametersα, θ regarding model likelihood. The initial hyperparameter α was setto α = 0.01 as suggested by [9]. Activity primitive segments wereformed of segment size DS with a segment step DW = 0.1 · DS.We applied the Borda Count ranking method to the topic activa-tions γs of all segments s covering the same DW time slot. We in-vestigated subject-dependent leave-one-day-out cross-validation andsubject-adapted leave-one-day-out cross-validation. The evaluationsinvolved topic model and kNN model estimation on all patients’ dataexcept for the left-out day. For the evaluation analysis we only eval-uated activity routines counting at least 3 repetitions in the datasetresulting in 3, 4, 5 and 5 activity routines for patient 1, 2, 3 and subject-adapted analysis (Table 6.2). For each topic model estimation we per-formed 3 iterations and chose the one with the highest likelihood. Asevaluation measure we used the averaged class-specific accuracy ofactivity routine predictions on activity routine annotations. Because ofrandom topic model internal parameter initialization we investigatedmean and standard deviation across 5 independent topic model runs.We further investigated the influence of the hyperparameter alpha


on the discovery performance. Thus, we evaluated fixed alpha valuesbeyond the suggested default setting 50/K [11].

6.3 Results

6.3.1 Activity routine discovery

Figure 6.2 shows the averaged class-specific accuracies for the subject-dependent and subject-adapted evaluation for different segment sizesDS and number of activity topics K. Using our layered discovery ap-proach the activity routines of rehabilitation patients were discoveredwith up to 88% accuracy for patients 1 and 2 and about 75% for pa-tient 3. The evaluation included activity routines as specified in Section6.2.4. With increasing number of topics, accuracies even increased for

10 20 30 40 5030

40

50

60

70

80

90

100

Segment size DS [min]

Acc

urac

y [%

]

5 10 1530

40

50

60

70

80

90

100

Number of activity topics K

Patient 1 Patient 2 Patient 3 All Patients

Figure 6.2: Averaged class specific accuracies and standard deviationfor varying segment size DS and number of activity topics K for subject-dependent (patients 1, 2 and 3) and subject-adapted (all patients) eval-uation.

6.3. Results 143

Cog

nitiv

e tra

inin

g

Soc

ialis

ing

Mot

or tr

aini

ng

Med

ical

fitn

ess

Res

t/Sle

epin

g0

.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Patie

nt 1

Patie

nt 2

Patie

nt 3

All

patie

nts

Cogn

itive

train

ing

Socia

lisin

g Mot

or tr

aini

ng Socia

lisin

g

Mot

or tr

aini

ng

Med

ical f

itnes

s

Rest

/Sle

epin

g

Cogn

itive

train

ing

Socia

lisin

g

Mot

or tr

aini

ng

Med

ical f

itnes

s

Rest

/Sle

epin

g

Cogn

itive

train

ing

Socia

lisin

g

Mot

or tr

aini

ng

Med

ical f

itnes

s

Rest

/Sle

epin

g

pred

icte

d co

ntex

t cat

egor

y

actual context category

35

79

1113

150.

70

−1−2

−3

0

0.51

Num

ber o

f top

ics K

log10

(α)

accuracy

a)b)

Figu

re6.

3:a)

Con

fusi

onm

atri

ces

for

subj

ect-

depe

nden

t(p

atie

nts

1,2,

3)an

dsu

bjec

t-ad

apte

d(a

llpa

tien

ts)

eval

uati

onsh

owin

gpa

tien

tspe

cific

acti

vity

rout

ine

clas

ses.

Act

ivit

yro

utin

esar

ecl

earl

yse

para

ble

from

acti

vity

prim

itiv

esba

sed

onbo

dypo

stur

ean

dex

trem

ity

acti

vity

,b)a

vera

ged

clas

ssp

ecifi

cac

cura

cies

ford

iffer

entt

opic

mod

elhy

perp

aram

eter

sα

and

num

ber

ofto

pics

K.

The

hype

rpar

amet

erdo

esno

tin

fluen

ceth

edi

scov

ery

accu

racy

ofth

eap

proa

chas

vari

atio

nsar

esm

alla

ndw

ithi

nth

est

anda

rdde

viat

ion

of5%

.


all patients. For all patients, the highest accuracies could be achieved ata segment size of DS = 20 min. At comparable parameters for subject-dependent models (segment size of DS = 20 min and K = 2M) weachieved 78 % accuracy for patient 1 (K = 6) and 3 (K = 10) and 88% for patient 2 (K = 8). Patient 2 was not using a wheelchair andthe recordings showed larger variability in activity primitives acrossthe activity routines, which could explain the overall higher accuracyobtained for this patient. We even yielded 71 % accuracy for subject-adapted evaluation including all activity routine data (also routineswith less than 3 repetitions per patient, Table 6.2).

Figure 6.3b shows discovery performances across different α valuesexemplary for patient 3. The parameter rarely has an influence on theaccuracy as variations are small and within the accuracy’s standarddeviation (Figure 6.2). This trend is similar for all 3 patients. Thus,routine topic activations γ inferred from activity primitives seem to bediscriminative, independent of α. Having discriminative routine topicactivations the kNN (used for mapping and performance evaluation)yields high accuracies. However, when targeting activity routine dis-covery favored and thus clear activated activity topics (small α values)per segment could make routine topic and activity routine mappingmore evident. The investigate this hypothesis an alternative to kNNfor performance evaluation should be analyzed in future work.

Fig. 6.3a depicts the confusion matrices for subject-dependent and -adapted evaluations. Using the layered discovery approach all activityroutine patterns were clearly separable for each patient. Furthermore,few confusions for the subject-adapted analysis suggest activity rou-tine patterns to be similar across patients. The 6% decrease in averagedaccuracy compared to subject-dependent analysis (Figure 6.2) resultsfrom increasing confusions for cognitive training and socialising. Rea-sons might be highly patient dependent execution of the activity rou-tine cognitive training and few activity routine repetitions (total= 9,Table 6.2). Overall high accuracies show that it is possible to discoveractivity routines of stroke patients from sensor data using topic model-ing. The topic model discovered meaningful activity routine patternsfrom activity primitive derived from upper and lower extremity acti-vity and body and arm postures.

6.4. Conclusion 145

6.4 Conclusion

In this paper, we investigated activity routine discovery of rehabili-tation patients from sensor data using topic modeling. We achievedaccuracies between 78% and 88% for all subject-dependent evalua-tions when considering activity routines including socialising, cognitivetraining, medical fitness, motor training and rest/sleeping phases. Theseresults indicate that our approach can be suitable for the discovery ofactivity routines that rehabilitation patients perform during a day inthe day care center. The topic model parameter investigation showedthat with increasing number of activity topics, accuracy increased forall subject. While the optimal accuracy was found for a segment sizeof DS = 20min the hyperparameter α did not influence accuracy. Highdiscovery accuracies suggests that the activity routines show charac-teristic patterns regarding the derived set of activity primitives whichwas based on arm and leg movement as well as arm and body pos-tures for both, affected and non-affected body side. Thus, activity rou-tine disocvery does not require complex activity primitives detectedfrom trained classifiers. Furthermore, the subject-adapted investiga-tion showed, that activity routine patterns seem to be highly similiaracross all three stroke patients. In future work, we plan to validate ourapproach by extending the study at the day care center and expandthe evaluations to a higher number of patients.

6.5 Acknowledgments

This work was supported by the EU Marie Curie Network iCareNet,grant number 264738. We are grateful to the participating patients andtherapists from the day care center.


Bibliography

[1] W. H. Organization, The world health report 2003: shaping the future.World Health Organization, 2003.



[4] S. Patel, R. Hughes, T. Hester, J. Stein, M. Akay, J. G. Dy, andP. Bonato, “A novel approach to monitor rehabilitation outcomesin stroke survivors using wearable technology,” Proceedings of theIEEE, vol. 98, no. 3, pp. 450–461, 2010.

[5] E. Haeuber, M. Shaughnessy, L. W. Forrester, K. L. Coleman,and R. F. Macko, “Accelerometer monitoring of home-andcommunity-based ambulatory activity after stroke,” Archives ofPhysical Medicine and Rehabilitation, vol. 85, no. 12, pp. 1997–2001,2004.

[6] G. Uswatte, W. L. Foo, H. Olmstead, K. Lopez, A. Holand, andL. B. Simms, “Ambulatory monitoring of arm movement usingaccelerometry: an objective measure of upper-extremity rehabili-tation in persons with chronic stroke,” Archives of Physical Medicineand Rehabilitation, vol. 86, no. 7, pp. 1498–1501, 2005.






[11] M. Steyvers and T. Griffiths, “Probabilistic topic models,” Hand-book of Latent Semantic Analysis, vol. 427, pp. 424–440, 2007.

7Robustness of

parametric topicmodels

Julia Seiter, Oliver Amft and Gerhard Tröster

Full publication title: Assessing topic models: how to obtainrobustness?

First Workshop on recent advances in behavior prediction andpro-active pervasive computing, Pervasive, 2012.

150 Chapter 7: Robustness of parametric topic models

Abstract

In this work we investigate the influence of varying daily activity datasetcharacteristics on topic model performance stability for daily routine discov-ery. For this purpose, we denote a set of key dataset properties that influencethe experimental design regarding recording, as well as data pre-processingsteps. Using generated daily activity datasets, we identified optimal topicmodel stability for particular dataset properties. Results indicated that topicmodel routine duration should exceed document size by a factor of more thantwo. Recording durations of more than 9 days were required for a set of fourroutines and activity primitive overlap may not exceed 5%.

7.1 Introduction

The discovery of complex daily routines from sensor data is relevantfor a variety of applications stretching from medical diagnosis to in-dependent living. Wearable sensors can provide information on thestructure and routines in daily life, including complex routines such ashygiene, lunch and dinner. As daily life activities are very subject depen-dent and vary regarding duration and individual activities involved,discovering structures in daily activities is a challenging research prob-lem. A commonly considered concept is to partition daily activity intoabstraction levels, where regular daily routine structures can be com-posed of activity primitive sets. The latter typically has finer temporalgranularity and - at the lowest level - must be suitable for recognitionfrom sensors. Figure 7.1 illustrates this concept, where daily routinestructures form a composition of different activity primitives.

Several approaches exist towards complex activity recognition anddiscovery that could describe daily routines. For instance, Huynh etal. [1] used probabilistic topic models to reveal specific activity pat-terns from a number of primitives, which were mapped to complexdaily routines, such as office work or commuting. As primitives, Huynhet al. used activities, including queuing in a line and sitting/desk activi-ties. Using topic models seems a very promising approach to discoverstructures in daily activities. Depending on the application, however,routines vary highly in e.g. duration and primitive composition. It isyet not clear, under which dataset conditions topic models can performrobustly. To design future experimental evaluations and topic modelsystem designs, it is thus critical to identify key properties, includ-


Commuting

Office work

Lunch

8:00 10:00 12:00 14:00 16:00 18:00 t

duration

Having a meeting

Talking on the phone

Discussing at whiteboard

Picking up mensa food

Queuing in line

Walking freely

Driving car

Routines

Prim

itives

Figure 7.1: Illustration of occurrence and duration of three daily rou-tines and their composition out of primitives derived from the Ubi-Comp’08 dataset [1]. The example visualizes the common assumptionto abstract daily activities. Our approach incorporates this conceptfor generating datasets with different properties and evaluating topicmodel performance stability.

ing training dataset duration, primitive specification, etc. that couldinfluence performance.

In this work, we investigated the topic model stability by varyingselected dataset properties. As an exhaustive evaluation of potentiallyinfluential properties is beyond feasibility, we focused on a set of keyelements that influence the dataset recordings, including duration ofroutines, amount of training data, and specificity of routines. We con-sidered that these properties profoundly influence data needs andnumber and granularity of primitives for obtaining robust discoveryresults. In order to evaluate dataset properties, we implemented a dailyactivity simulation model. The simulation model allowed us to gener-ate datasets with different characteristics. We based our investigationon the UbiComp’08 dataset presented in Huynh et al. [1].

In order to validate our daily activity simulation model and thetopic model implementation, we used the UbiComp’08 dataset to(1) compare the performance reported by Huynh et al. in [1] to ours,and (2) confirm that the dataset creation can replicate the results in [1].Subsequently, we investigated the topic model performance stabilityusing a framework of simulation-based dataset generation and stabil-ity measurement. As common stability criterion we used the standarddeviation of the routine prediction accuracy across multiple gener-


ated datasets. From the simulation results, requirements for a systemregarding sensor modalities and data pre-processing could be derived.

The paper is structured as follows: first, we review related initia-tives in complex activity recognition and topic models in Section 7.2.We then describe the daily activity simulation model and its formali-ties in Section 7.3, followed by fundamentals of the topic model frame-work for daily routine discovery in Section 7.4. We formally introducethe dataset properties considered for daily activity simulations in Sec-tion 7.5 and describe the analysis implementation. Sections 7.6 and 7.7present results and the conclusion of this investigation.

7.2 Related work

Hierarchical activity recognition. For complex activity recognitionhierarchical models have been frequently used. Olivier at al. [2] rec-ognized office activities using multiple HMM layers. In their work,video, audio and computer work was processed, and activities at dif-ferent granularity levels were recognized. Complex activities, such asgiving a presentation were inferred at the top layer. Lee at al. [3] pre-sented a framework to infer activities from a variety of contextual datain the mobile setting using hierarchical Bayesian networks. Amft etal. [4] inferred composite activities from wearable and environmentalsensors in a two layered model. Huynh et al. [1] classified low-levelactivity data such as walking freely or standing using Naive Bayes. In asecond layer, they derived activity patterns from a probabilistic topicmodel. These activity patterns were matched to daily routines, such asoffice work and lunch. They achieved an averaged recall of 86.1% andprecision of 67.2% on a set of four specific routines.

Topic models. Besides the approach of Huynh et al. [1] who focusedon activity recognition to discover routines, e.g. Farrahi et al. [5] ap-plied topic models in an unsupervised manner. They introduced aframework to discover daily routines from location and proximitydata using topic models.

Topic models have been successfully applied for activity recog-nition in video frames. For example, in [6] and [7] human actioncategories were recognized from complex video streams using topicmodels. While topic models are common in many application fieldsbesides text processing, we have not found investigations on topic

7.3. Simulation of daily routines 153

models and its input demands, focusing on model stability. For dailyroutine discovery, robustly performing topic models could be appliedin various applications to reveal the daily activity structure.

7.3 Simulation of daily routines

In this section, we introduce the daily activity simulation model, usedto generate daily activity datasets. With the simulated daily activitieswe subsequently investigated the topic model stability.

For the daily activity simulation in this work, we assumed routinesto be composed of several activity primitives of finer temporal gran-ularity. For instance, the routine office work would consists of severalprimitives, such as making a phone call, walking freely or having a meet-ing (see Fig. 1.2). Due to this hierarchical structure in daily activities,we defined a three layered simulation model for sampling daily rou-tines and its primitives. The top layer defines the sequence of routines,the intermediate layer describes their duration. The primitives of eachroutine are derived from a lower layer model. An illustration of oursimulation model is provided in Figure 7.2.

Top layer. The top layer consists of an HMM, which describes the setof consecutive routines during a considered number of days. Routinesare chosen from k routines ri; i ∈ 1, 2, ..., k. An HMM with n states x j

and k observations e j = ri is used to model the routine sequence. Indaily life, the same routine ri may occur several times during a daywith varying durations. Thus, a routine may be represented by severalHMM states (n ≥ k) in this model layer. The HMM is described by ann × n state transition matrix T and an n × k emission matrix E. Whensampling data, the outputs of the top layer are a z-dimensional statevector ~x and an emission vector ~e containing the sequence of z statesand z assigned routines.

The sequence of routines is very specific during the day. Highfluctuations between different routines may thus not represent a re-alistic sequence. As an HMM is based on a probabilistic process, weobserved that the HMM typically shows more frequent alternationsbetween routines even when trained with a realistic set of routine se-quences. To obtain realistic routine representations, we only used theHMM to sample sequences containing state transitions in differentstates (x j → xh; j , h).


Consecution of routines – HMM

Duration of routines – N(µ,σ² )

MCr1 MCr2 MCrk

Routines

...

Number of routines

Primitives

x,e

e,d

p,l

(T1) (T2) (Tk)

(T,E)

Figure 7.2: Daily activity simulation model for sampling of daily rou-tines and its activity primitives. The top layer estimates a sequencevector of routines~e and states ~x. In the intermediate layer, the durationvector ~d is derived from normal distributions N j(µ j, σ2

j ) assigned to theHMM states in ~x. The lower layer applies a Markov chain MCri for eachroutine ri to sample primitives. The number of primitives is defined by~d. The outputs of all MCs form the primitive vector ~p and the routinevector~l. Model parameters are denoted in grey.

Intermediate layer. The duration d j of each HMM state x j is estimatedfrom its corresponding normal distribution N j(µ j, σ2

j ), j ∈ 1, 2, ...,n. Thesampling output of the intermediate layer contains a duration vector~d describing the duration of each routine in a sequence denoted by theemission vector ~e.

Lower layer. In this work, the occurrence of primitives for a particularroutine is of interest. The lower layer consists of k independent Markovchains MCri describing the sequence of primitives for each of the kroutines ri individually. Each state in the MCs represents one of the mactivity primitives. When sampling primitives, for each of the routinesri in~e, the corresponding Markov chain MCri is selected. The number ofprimitives sampled from the Markov chains is denoted by the duration

7.4. Topic modeling approach of daily routines 155

vector ~d. As output, the primitive label vector~p is formed by the orderedoutput of all the z Markov chain calls belonging to the sequence of zroutines in ~e. Additionally, a routine label vector~l is emitted. We use am×m primitive transition matrix Ti, i ∈ 1, 2, ..., k for each Markov chainMCri .

7.4 Topic modeling approach of daily routines

For the daily routine discovery, we used a similar topic model frame-work as reported by Huynh et al. [1]. Topic models find their originin the text processing community and are used to discover kT hiddentopics in a corpus of documents filled with words from an alphabet. Inthis work we applied the Latent Dirichlet Allocation (LDA) algorithm,which assumes distributions of topics over documents to be derivedfrom a Dirichlet distribution. When applying LDA on a corpus of docu-ments, the algorithm infers for each document d a kT-dimensional topicactivation vector gamma ~γd from the bag-of-words in d. The normal-ized vector ~γd describes the estimated occurrence ratio of each topic ind. More detailed information on LDA can be found in [8].

Here, we used the topic model to infer routine patterns from prim-itives. Routines correspond to topics, words are formed by primitives.Documents cover a time slice of a day, containing all the activity prim-itives in that time slice. Structuring a day into subsequent time slicesis equal to structuring it in subsequent documents. The inputs for thetopic model are the primitive histograms of the documents. The topicmodel then reveals patterns in the primitives and infers a topic acti-vation vector ~γd for every document. The number of topics does notnecessarily match the number of routines. Therefore, we use a super-ordinated kNN classifier for mapping topics to routines. The topicactivation vector ~γd is used as feature vector for the kNN. Both thetopic model and the kNN are trained by a subset of the considereddaily activity data.

7.5 Analysis methodology

The topic model performance stability could be affected by variousdataset properties. We describe in this section the properties consid-ered in this work and our overall evaluation strategy. In the evaluation,we firstly validated the daily activity simulation model against the Ubi-


Comp’08 dataset presented in Huynh et al. [1]. Subsequently, we usedthe simulation model to generate datasets with explicit properties andanalyzed the topic model stability. The UbiComp’08 dataset was con-sidered as basis of the evaluation. Our simulation approach could begeneralized to other datasets that include a two-layer data hierarchy ofprimitives and routines by inferring the simulation model parametersas shown below.

7.5.1 Dataset properties considered in the daily activity simulation

In order to determine requirements for a stable topic model perfor-mance, we considered the following dataset properties in our dailyactivity simulation: duration of routines, amount of data and speci-ficity of routines. The dataset properties are detailed in this section.Each property was individually investigated to avoid co-occurring ef-fects.

Duration of routines and amount of data.

The duration of a specific routine ri was changed by varying the meansµ j of the normals N j(µ j, σ2

j ) in the simulation model. This was donefor all states x j showing an emission e j = ri. Given the varied meanµ∗j, a corresponding standard deviation σ∗j was adapted according toσ∗j = σ jµ∗j/µ j. We varied the number of simulated recording days toinvestigate the amount of data.

Specificity of routines.

We investigated the similarity of different routines to gain insight intohow specific routines need to be for stable topic model performance.As measure for the similarity of two routines ri, r j the overlap oi j oftheir primitive histograms hi and h j was derived according to:

oi j = 1 −m∑

s=1

|his − h j

s|/2 . (7.1)

The parameter his is the occurrence ratio of primitive s in routine i.

The specificity of all routines in a dataset is described by the overlapototal, which is the mean over all pairwise routine overlaps oi j

|i , j, j >i. The transition matrices Ti from the lower layer of our simulation

7.5. Analysis methodology 157

model (see Fig. 7.2) were used as tuning parameters when samplingdata. In order to derive less specific routines, the new transition matrixTi∗ is a combination of the original Ti and the transition matrices T j

of the other routines r j: j ∈ 1, .., k; i , j. We derive the new transitionmatrices Ti

∗ by:

Ti∗ = (1 − (k − 1)p)Ti +

k∑j=1; j,i

pT j . (7.2)

The tuning parameter p ∈ [0, 1m ] was used in our analysis. For p = 1

m ,all routines share an identical transition matrix and therefore wouldshow an equal primitive histogram (see Fig. 7.3). For p = 0, Ti

∗ and Ti

are identical.Two routines are highly specific when they do not share the same

activated primitives in their histograms. Therefore, we define the tran-sition matrices Ti

spec for each routine ri. Tispec was derived from Ti by

copying a subset of state transition matrix components tihj ∈ Ti for se-

lected primitives h, j ∈ 1, 2, ...,m. The other components tirs; rs , hj in

Tispec were set to zero. The matrix Ti

spec was then normalized to a rowsum of 1. Primitives h, j were chosen such that different routines do notshare any primitives. Figure 7.4 provides an illustration of this setting.We derived the transition matrices Ti

∗ according to (p ∈ [0, 1]):

Ti∗ = (1 − p)Ti + pTi

spec . (7.3)

For Eq. 7.3 and p = 1, different routines do not show overlap intheir primitive histograms. In our evaluation, more specific routineswere obtained by adapting Ti according to Eq. 7.3, whereas applyingof Eq. 7.2 resulted in less specific routines compared to the basis (Ubi-Comp’08 dataset).

7.5.2 Implementation

Simulation of data.

All dataset properties used in our simulation model were sampledfrom the UbiComp’08 dataset (Huynh et al. [1]). This setup formedthe basis (number of days, k, T,E, µk,σk, m, Ti) for all simulations in thiswork.


0.5

h1

0.5

h2

primitives

Figure 7.3: Histograms h1 and h2 ofroutines 1 and 2 showing an equalmixture of all Ti’s (100% primitiveoverlap).

0.5

h1

0.5

h2

primitives

Figure 7.4: Histograms h1 and h2 ofroutines 1 and 2 showing T1

spec andT2

spec, with no overlap in primitives.

The UbiComp’08 dataset covers seven days without sleepingphases. It contains 4 routine labels dinner, commuting, lunch and of-fice work, and a null class. In total, 24 primitive labels are available asuser annotations at a frequency of f = 2.5Hz, including activities suchas using the toilet, preparing food, and sitting/desk activities.

To analyze routine duration ri, the means µ j with a correspond-ing emission e j = ri and µ j > 5 min were swept in the interval[10 min, 150 min]. For the amount of data, we varied the number ofsampling days in [3 days, 25 days]. To analyze routine specificity, wevaried the tuning parameter p as described above.

Daily routine discovery.

To evaluate the topic model we used the LDA implementation accord-ing to [9]. All topic model parameters were set corresponding to [1].Documents were formed over a duration of 30 min, shifted by 2.5 min,the number of topics kT was set to 10.

To obtain stable topic model estimation results, three training runswere performed, choosing the one with the highest likelihood. Primi-tives that occurred in a single day only were not considered, as well asthe primitive unlabeled. We applied the Borda Count ranking method tothe topic activations ~γd of all documents d covering the same 2.5 mintime slot. The resulting total topic activation vectors showed a resolu-tion of 2.5 min. After upsampling to the ground truth frequency, topicactivations were used as input for the kNN classifier.

7.6. Results 159

Evaluation.

We used a leave-one-day-out scheme in our analysis. For data samplesexceeding 11 days, we applied a 10-fold-cross validation. The nullclass was used for training the topic model and the kNN, but left outfor evaluation. To compensate for the probabilistic data acquisitionvia sampling, we repeated for each dataset property analysis both, thedataset simulation and topic model calculations 20 times. The averagedaccuracy and standard deviation of the 20 runs were analyzed. Inthe evaluation we considered a standard deviation of less than 5%as stable performance. Performance variations below this standarddeviation could be considered random, e.g. caused by the initializationof models.

7.6 Results

Firstly, we validated our daily activity simulation model against theperformance reported by Huynh et al. [1]. In the subsequent sections,we investigated selected dataset properties regarding the topic modelstability for daily routine discovery.

7.6.1 Validation of the daily activity simulation model

When using the UbiComp’08 dataset [1], our topic model achieved arecognition performance as reported in Table 7.1. For the evaluationapproach in this work, it is sufficient to consider the class-specificaccuracies. In order to compare against Huynh et al., we show theprecision here as well. Compared to Huynh et al., our topic modelresults showed higher precision and recall since we used labels andnot classified primitives as topic model input.

In order to validate our simulation model we compared the recog-nition performance of our topic model implementation using the Ubi-Comp’08 dataset [1] against a simulated dataset sampled from the Ubi-Comp’08 dataset. On the simulated data, we achieved similar recog-nition performance compared to using the dataset directly, except forthe lunch routine. For lunch, performance results of the simulated datawere lower. We assume that the difference for this routine was dueto an inadequate sampling of the primitive transition matrix Ti in thelower layer of our simulation model. We attributed the low perfor-mance variations of other routines to the random topic model initial-


Table7.1:C

omparison

ofperformance

regardingdaily

routinediscovery.The

performancesforourtopic

model

implem

entationand

asim

ulateddataset

sampled

fromthe

actualUbiC

omp’08

in[1]

were

compared

againstthe

resultsof[1].

Routine

Huynh

[1]O

urtopic

model

Simulated

datarecall

precisionrecall/accuracy

precisionrecall/accuracy

precisiondinner

40.275.5

73.671.8

74,871,8

comm

uting51.8

85.582.9

90.686,3

85,1lunch

83.387.0

86.791.6

75,476,2

offi

cew

ork93.7

96.494.5

96.291,0

94,7m

ean67.2

86.184.4

87.681.9

82.0

7.6. Results 161

ization. Comparing the routine sequence in a day and the histogramof primitives per routine we found that the UbiComp’08 dataset andthe simulated dataset showed high similarity. Overall, structure andperformances of simulated and actual dataset correspond well. Conse-quently, we considered the simulation model as capable of generatingrealistic datasets and suitable to analyze specific dataset properties.

7.6.2 Influence of routine duration

Figure 7.5 shows that with increasing document length performanceand stability of the topic model for the best and the lowest performingroutines of the dataset (office work, dinner) increase. This effect dependsmarginally on the total amount of data available for each routine: din-ner occurred once a day, while office work occurred three times a day,and therefore comprises three times the data amount of dinner. Never-theless, both routines show the same trend in accuracy and standarddeviation for sufficiently long routines.

However for short durations below 50 min, larger data amountslead to higher stability. Too short routines relative to the documentlength are highly instable, such as seen for dinner. Following ourassumption of topic model stability (std<5%), both routines require80 min duration. Hence, routine duration must considerably exceedthe document length (30 min).

7.6.3 Influence of the amount of training data

Amount of data particularly influences model stability, if few daysof data (below 5 days) are available. Figure 7.6 shows the effects onperformance. Office work and commuting are already stable (std<5%)at 9 days of data, while the total occurrence times of office work was61.0 hours and of commuting 6.4 hours.

Although lunch (8.4 hours) had more data compared to commutingduring 9 days, stable results were obtained for ∼ 14 days only (totalingto 13 hours of data). This result indicates that the total amount of datadoes not have an unique impact on stability. There are other param-eters, such as specificity of the routine influencing it. Nevertheless,the number of recording days turns out to be one crucial tuning pa-rameter for stable topic models. Dinner does not become stable in theconsidered interval at all. The analysis shows that, in order to ensurestability, data acquisition should be performed for ∼ 14 days. More


0 50 100 15040

60

80

100

mean of duration [min]

accu

racy

[%]

office work dataset Huynh dinner dataset Huynh

0 50 100 1500

5

10

15

20

25

mean of duration [min]

std

[%]

Figure 7.5: Recognition accuracy and stability over 20 runs for dinnerand office work using simulated data. Both routines become more stablefor longer routine durations.

4 6 8 10 12 14 16 18 20 22 2460

80

100


accu

racy

[%]

dinner commuting lunch office work dataset Huynh

4 6 8 10 12 14 16 18 20 22 240

10

20

30


std

[%]

Figure 7.6: Recognition accuracy and stability over 20 runs for simu-lated data conditioned on the number of recorded days for four rou-tines.

7.6. Results 163

0 5 10 15 20 25 30 35 40 45 5040

60

80

100

overlap of histograms of different routines [%]

accu

racy

[%]

simulated datasetdataset Huynh

0 5 10 15 20 25 30 35 40 45 500

5

10

15

std

[%]

overlap of histograms of different routines [%]

Figure 7.7: Recognition accuracy and stability over 20 runs for simu-lated data conditioned on the specificity of routines. With increasinghistogram overlap of different routines, the topic model becomes lessstable.

data can yield performance benefits for low frequent routines such asdinner.

7.6.4 Influence of the specificity of routines

The specificity of routines highly influences stability and performanceof the topic model, as Figure 7.7 illustrates. With increasing overlap,routines become more similar and the topic model yields less stableresults. Overlap highly depends on how specific routines are. For a20% overall overlap in the dataset, commuting and dinner show a verylow routine-to-routine overlap (4.4%), whereas dinner and office workappear similar in terms of their primitives (routine-to-routine overlapwas 37%). This implies that the choice of primitives is highly connectedto performance and stability. Thus, when targeting stability under thegiven topic modeling approach, the overlap may not exceed 5%.


7.7 Conclusion and outlook

In order to analyze key dataset properties that could provide stabletopic model performance, we investigated the duration of routines,amount of data and specificity of routines in this work. The choice ofthese dataset properties appears essential for the experiment and datarecording design, when targeting daily routine discovery.

The validation of our daily activity simulation model confirmedthat performances closely resembling those in previously publishedwork can be achieved. Subsequently, we created datasets of differentcharacteristics by varying the selected dataset properties. Our inves-tigations showed that specific requirements exist that would ensurestable topic model performance. In particular, routine durations needto be considerably longer than the document size and 14 recordingdays appeared essential in the considered conditions. Furthermore, wefound that the primitive histogram overlap of different routines highlycorresponds to topic model stability. Thus, a bound on the primitiveoverlap can be given to support the design of lower recognition layers.

Using our results, a suitable primitive set regarding number andgranularity could be defined. The potential performance of the selectedprimitive set could be predicted by deriving the histogram overlap ofroutines. The preliminaries found in this paper can be used towards astable topic model application in activity recognition.

7.8 Acknowledgments


Bibliography

[1] T. Huynh, M. Fritz, and B. Schiele, “Discovery of activity patternsusing topic models,” in Proceedings of the 10th international conferenceon Ubiquitous computing, pp. 10–19, ACM, 2008.

[2] N. Oliver, A. Garg, and E. Horvitz, “Layered representations forlearning and inferring office activity from multiple sensory chan-nels,” Computer Vision and Image Understanding, vol. 96, no. 2,pp. 163–180, 2004.

[3] Y.-S. Lee and S.-B. Cho, “Human activity inference using hierar-chical bayesian network in mobile contexts,” in Neural InformationProcessing (B.-L. Lu, L. Zhang, and J. Kwok, eds.), vol. 7062 of LectureNotes in Computer Science, pp. 38–45, Springer Berlin / Heidelberg,2011.


[5] K. Farrahi and D. Gatica-Perez, “Discovering routines from large-scale human locations using probabilistic topic models,” ACMTransactions on Intelligent Systems and Technology, vol. 2, no. 1, pp. 1–27, 2011.


[7] X. Wang and E. Grimson, “Spatial latent dirichlet allocation,” inAdvances in Neural Information Processing Systems, pp. 1577–1584,2008.


[9] D. Blei, “Implementation of lda athttp://www.cs.princeton.edu/ blei/lda-c.”

8Joint segmentationand nonparametric

activity discovery

Julia Seiter, Wei-Chen Chiu, Mario Fritz, Oliver Amft and Gerhard Tröster

Full publication title: Joint segmentation and activity discovery usingsemantic and temporal priors.

IEEE International Conference on Pervasive Computing andCommunications (PerCom), 2015

DOI: 10.1109/PERCOM.2015.7146511

168 Chapter 8: Joint segmentation and nonparametric activity discovery

Abstract

We introduce a hierarchical nonparametric topic modeling approach to inferactivity routines from context sensor data streams based on a distance de-pendent Chinese restaurant process (ddCRP). Our approach does not requirelabeled data at any stage. Neither does our approach depend on time-invariantsliding windows to sample context word statistics. Our activity discovery ap-proach builds on the idea that context words occurring within one activityare semantically similar, whereas context words of different activities are lesssimilar. Context word streams are segmented into supersamples and then se-mantic and temporal features are obtained to construct a segmentation priorthat relates supersamples via its context words. Our hierarchical model usesthe segmentation prior and ddCRP to group supersamples and the Chineserestaurant process (CRP) to discover activities. We evaluate our approachusing the Opportunity dataset that contains activities of daily living. Be-sides being nonparametric, our ddCRP based model outperforms both, classicparametric latent Dirichlet allocation (LDA) and the nonparametric Chineserestaurant franchise (CRF). We conclude that ddCRP+CRP is an adequateapproach for fully unsupervised activity discovery from context sensor data.

8.1 Introduction

Discovery of daily activities and routines from ubiquitous sensor dataprovides insights into individual behavior without prior model learn-ing, which is relevant for assisted living, remote patient care, and re-lated applications [1, 2]. A common approach to assess human behavioris to partition activity routines into abstract levels. For example, officework and lunch can be decomposed into context symbols, typically ofshorter temporal duration. These may include activity primitives (sit,walk), locations (home), or object use (computer, spoon). Context symbolscould be detected from the continuous acquired data of on-body andambient sensors, where frequently supervised classification or dataclustering were used [3]. Activity routine discovery requires methodsfor analyzing context symbol patterns, where often parametric topicmodels were applied, such as latent Dirichlet allocation (LDA) [3, 4].Topic models originate from text mining and aim at discovering hid-den themes from word statistics in documents. For parametric topicmodels it is assumed that one document contains a mixture of a fi-nite number of topics and that each topic is described as probabilisticdistribution over words from a predefined vocabulary.


In activity discovery, words correspond to context symbols andtopics correspond to activities, which we call context words and ac-tivity topics respectively. Typically, documents are obtained using atemporal segmentation of the continuous context word stream with apredefined segment size that is large enough to capture context wordstatistics. Subsequently, discovery results per segment are retrieved.With frequently used segment sizes of 30 min [3, 5], activity transi-tions and activities with variations in duration may not be accuratelyidentified. Moreover, parametric topic models, such as LDA, requireto set the number of topics. Selecting topic model parameters, includ-ing segment size and number of topics, impacts activity discoveryperformance and highly depends on dataset properties that may beunknown [6]. Recently, Bayesian nonparametric topic models wereproposed for activity discovery to overcome the dependency on a pre-defined topic count [5, 7]. However, existing nonparametric modelsalso depend on a fixed segment size.

In this paper, we introduce a novel hierarchical topic model ap-proach that does not depend on manually selecting parameters seg-ment size and number of topics. Instead, segmentation and topic countestimation is performed based on the data and jointly with the acti-vity topic discovery. We propose a framework that includes contextword extraction and activity discovery. Context words are obtainedfrom sensor data without statistical classifier training and thus do notrequire activity annotations. We introduce a segmentation prior con-sidering semantic and temporal information and use the nonparamet-ric distance dependent Chinese restaurant process (ddCRP) to groupcontext words that belong to one activity. For example, segmentationof activity lunch would contain context words such as spoon and plate,whereas activity office work may contain computer. Thus, our semanticrelationship representation of spoon is “closer” to plate than to computer.

The contributions in this paper are threefold: (1) We introduce ajoint segmentation and activity discovery approach that is indepen-dent of the number of topics and the segment size. Here, we combinethe nonparametric ddCRP and Chinese restaurant process (CRP) hier-archically and formulate a segmentation prior that considers seman-tic and temporal features of context words. Semantic representationswere extracted from a corpus of Wikipedia articles. (2) We show thatour approach outperforms the parametric LDA and the nonparamet-ric Chinese restaurant franchise (CRF) on the Opportunity dataset thatcontains multi-modal sensor data [8]. (3) We demonstrate the increased


robustness and performance of ddCRP+CRP compared to other meth-ods regarding activity discovery from context word annotations, actualcontext word detections from raw data, and synthetic noise.

8.2 Related work

Several attempts towards activity discovery from sensor data weremade. Gu et al. extracted characteristic object use fingerprints applyingweb-mining and discovered contrast patterns for each activity usingemerging patterns [9]. Begole et al. applied data clustering to extractand visualize human’s daily rhythms from computer activity [10]. Asclustering-based methods cannot capture uncertainty in the structureof human activities, frequently probabilistic models have been applied.Barger et al. used probabilistic mixture models to infer daily life be-havior patterns from clusters of sensor events in a smart home [11].Probabilistic topic models have been applied to extract activity routinesfrom mobile phone data [4, 12] and activity primitives [3]. However,all of these topic model approaches are parametric and assume a fixedmodel complexity. Thus, discovery performance critically depends onthe number of topics specified. In contrast, our approach is nonpara-metric, thus estimates optimal topic count from the data structure.

Nonparametric models were recently applied for activity discov-ery. The hierarchical Dirichlet process HDP-HMM was used for ab-normal activity detection [13] and activity discovery from smartphonesensor data [14]. Similarly, Nguyen et al. used HDP to discover la-tent activity topics from acceleration and proximity data [7]. Sun et al.used HDP to discover patterns of high-level activities [5] from dataclusters. While nonparametric topic models estimate an optimal num-ber of topics based on the data, their discovery performance remainssensitive to selecting proper segment sizes. The topic model-based dis-covery frequently used time-invariant segmentation, such as slidingwindows [3, 5]. Yet, time-invariant segmentation fails to handle tran-sitions, variations in activity duration, and short activities accurately.Our approach no longer requires selecting a segment size, but per-forms segmentation dynamically based on the data by introducing asegmentation prior to group context words of the same activity.

Nonparametric topic models were successfully applied to inferthemes in text documents [15, 16]. In text mining, segmentation oftext is not needed as text is naturally segmented in documents. How-ever, using nonparametric topic models for object discovery in images

8.3. Joint segmentation and discovery approach 171

and videos [17, 18] or activity discovery from sensor data [5] requires aproper segmentation. Recently, the distance dependent Chinese restau-rant process (ddCRP) has been suggested to consider segmentationpriors for nonparametric object discovery as well as joint video seg-mentation and inference of object appearance models in images andvideos [19, 20]. Chiu and Fritz defined a video segmentation prior togroup pixels of coherent motion using ddCRP and an infinite mixturemodel to extract global object classes based on CRP [20]. We introducea similar approach for segmentation and nonparametric activity dis-covery from multi-modal sensor data. While in [20] the segmentationprior was based on spatio-temporal and motion similarities betweenpixel groups, we introduce semantic and temporal features to relatecontext words.

8.3 Joint segmentation and discovery approach

Time-invariant sliding windows cannot adequately handle variationsin activity duration. Figure 8.1 illustrates a segmentation problem ofvariable durations in activity discovery with examples: Large time-invariant windows, e.g. a segment size of DS=7, capture context wordstatistics of activity 1 exactly (see Fig. 8.1(a)). However, context wordstatistics for activity 3 would be incomplete, as the context word win-dows of activity 2 and 1 overlap. Contrary, a small segment size (e.g.DS=3) does not provide distinct context word statistics for activity1:lunch, as illustrated in Fig. 8.1(b).

We introduce a joint segmentation and discovery approach as de-picted in Fig. 8.2 to solve the segmentation problem. The first stageextracts data from multi-modal sensor sources into context words e.g.,sit, spoon moved. The context word extraction relies on basic logic func-tions, thus avoiding supervised statistical learning and classification.Subsequently, we introduce a data-driven segmentation based on statechanges in context words to obtain supersamples (see Fig. 8.1(c)). Su-persamples represent short temporal segments of context words withvariable size. As state changes in context words may occur within ac-tivities, supersamples may not comprehensively capture context wordstatistics that represent a particular activity, e.g., activity 1:lunch inFig. 8.1(c). Therefore, supersamples will be grouped according to se-mantic and temporal context word relations.

We assumed that an activity includes semantically similar con-text words, whereas the semantic relation of context words between


g g e eg g o o eh f h h o o

x x x y y y y z z v v z z x x x x x y y x x

e g e e g e e

e g e e g e e h f h h o o e g g e e Context wordchannels

Groundtruth

Time-invariantwindowsDS=7

Data-drivensupersamplesegmentation

a)

b)

c)

activities

Time-invariantwindowsDS=3

Segmentation of supersamplesto groups d)

g g o o

e g e e g e e h f h h o o e g g e eg g o o



DS=3

DS=7

supersamplegroup

context word x

variable DS

i)ii)

super-sample




x x x x x x x x y y y y z z v v z z y y x x

1:lunch 2:office work 3:rest 1:lunch

Figure 8.1: Illustration of segmentation methods for activity discovery.Exemplary, 3 activities are shown and segmentations for 2 context wordchannels {i, ii} with the context word vocabulary {e, f , g, h, v, x, y, z, o}.(a) Time-invariant windowing with segment size DS=7. (b) Time-invariant windowing with segment size DS=3. (c) Data-driven su-persamples segmentation. A new supersample is formed each time acontext change occurs in channel (i). (d) supersample groups are seg-mented by ddCRP with segmentation prior. Whereas in (a) and (b)windows intersect activities, (c,d) perform segmentation according todata.

8.3. Joint segmentation and discovery approach 173

TOP

IC M

OD

EL

SU

PE

RS

AM

PLE

SE

GM

EN

TATI

ON

CO

NTE

XT

WO

RD

EN

CO

DIN

G

FE

ATU

RE

EX

TRA

CTI

ON

CO

NTE

XT W

OR

D E

XTR

AC

TIO

N

JOIN

T SE

GM

ENTA

TIO

N A

ND

AC

TIVI

TY D

ISC

OVE

RY

D

ata

feat

ures

C

onte

xtw

ords

Sup

ersa

mpl

es

Sen

sor

data

SE

GM

EN

TATI

ON

PR

IOR

EX

TRA

CTI

ON

S

egm

enta

tion

prio

r

PER

FOR

MA

NC

E EV

ALU

ATI

ON

A

ctiv

ityto

pics

MA

PP

ING

EV

ALU

ATI

ON

A

ctiv

ities

Gro

undt

ruth

D

isco

very

Per

form

ance

ddC

RP

CR

P

Figu

re8.

2:A

ctiv

ity

disc

over

yfr

amew

ork:

Sens

orda

tais

proc

esse

dan

den

code

din

cont

ext

wor

dsus

ing

apr

edefi

ned

cont

ext

voca

bula

ry.O

urap

proa

chjo

intl

yse

gmen

tsco

ntex

tw

ords

and

perf

orm

sac

tivi

tydi

scov

-er

y.In

itia

lly,a

data

-dri

ven

segm

enta

tion

tran

sfor

ms

cont

ext

wor

dst

ream

sin

tosu

pers

ampl

esas

inpu

tfo

ra

hier

arch

ical

,non

para

met

ric

topi

cm

odel

.We

use

sem

anti

can

dte

mpo

ralp

rior

sto

grou

psu

pers

ampl

esof

the

sam

eac

tivi

tyw

ith

ddC

RP.

The

CR

Ppr

oces

sis

then

used

toin

fer

acti

vity

topi

csfr

omco

ntex

twor

dst

atis

tics

ofsu

pers

ampl

egr

oups

.To

eval

uate

perf

orm

ance

,act

ivit

yto

pics

are

map

ped

toa

seto

fact

ivit

ies.


different activities is lower. For example, context words x and y inFig. 8.1 may correspond to plate and spoon. Then, x:plate and y:spoon aresemantically more similar than x:plate and z:computer. To group super-samples that belong to the same activity (Fig. 8.1(d)), we introduce asegmentation prior that considers semantic and temporal relationshipsof supersamples based on the context words in each supersample. Wededuced semantic distances between context words based on word2vecrepresentations that were extracted from a corpus of Wikipedia arti-cles [21]. For example, activity 1:lunch contains a supersample i=1 withcontext words {x,x,x,e,g,e} and a supersample i=2 with {y,y,y,y,e,g,e,e}(Fig. 8.1(c)). The third supersample i=3 belongs to the activity officework and includes context words {z,z,h,f }. We expect higher prior prob-ability to group supersamples 1 and 2 than supersamples 1 and 3 as thesemantic and temporal distance of x:plate (supersample 1) and y:spoon(supersample 2) should be smaller than between x:plate (supersample1) and z:computer (supersample 3). Contrary to the example here, dis-tances for all pairs of context words were considered in the prior (seeSec. 8.4.4 for details).

We then apply a hierarchical, nonparametric topic model for acti-vity topic discovery using ddCRP and CRP as depicted in Fig. 8.3(c):ddCRP and CRP are clustering algorithms where the number of clus-ters is not given a priori but estimated from the data. In the local layer,supersamples are clustered into groups as illustrated in Fig. 8.1(d) andFig. 8.3(c) using ddCRP. Grouping by ddCRP depends on the seg-mentation prior: In our example, supersamples i=1 and i=2 belong toactivity 1:lunch and have high prior probability to be grouped contraryto supersamples i=1 and i=3 that belong to different activities (seeFig. 8.1(c)). We expect supersample groups to provide comprehensivecontext word statistics describing activities (see Fig. 8.1(d)). Individualdata recordings of a dataset likely contain the same activities. Thus,the global layer combines supersample groups that belong to the sameactivity by CRP to an activity topic group e.g. q=1:lunch (see Fig. 8.3(c)).For each activity topic group, the context word distribution is sampledfrom context word statistics of all assigned supersample groups suchthat the likelihood of the data is maximized. Retrieved activity topicswere mapped to activities and discovery performance was analyzed(see Fig. 8.2).

8.4. Discovery framework 175

8.4 Discovery framework

The complete discovery framework is illustrated in Fig. 8.2.

8.4.1 Context word extraction

The context vocabulary covers X context words {e, f , g...} that are ex-tracted from body worn and ambient sensor data. First, features areextracted from raw sensor data (see Fig. 8.2). Each statistical featurefrom sensor data is transformed to a binary feature F using thresholds.Binary features are subsequently included in logic functions to obtaincontext words (see Sec. 8.5.2, Tab. 8.1 for an example). Parallel operat-ing context word detectors (e.g. mode of locomotion, object usage) resultin several context word channels. Each context word channel provideseither an active context word or a null class symbol, when no contextword is active.

8.4.2 Segmenting context words into supersamples

We use a data-driven segmentation for context word streams that resultin variable sized segments, referred as supersamples similar to super-pixels in vision [20]. New supersamples are formed each time a contextstate change occurs (see Fig. 8.1(c) for illustration). As there may beseveral parallel context channels from different sensors sources, weuse the channel that includes the least sparse context word sequence.We consider that supersamples will typically have shorter temporalduration than activities and subsequently need to be grouped. We usea joint segmentation and activity discovery approach, as describedbelow.

8.4.3 Chinese restaurant process (CRP) and distance dependent Chi-nese restaurant process (ddCRP)

The Chinese restaurant process (CRP) is based on a Dirichlet pro-cess DP(α,Go) with base distribution Go and concentration parameterα [16]. As metaphor it can be described by a Chinese restaurant withan infinite number of tables k and a menu of dishes φ as depictedin Fig. 8.3(a). Each table serves one dish φk. N customers enter therestaurant sequentially and randomly sit at a table. The probabilitythat customer i is assigned to an existing table k ∈ 1...K depends on the


1 2

3 4

5 6

1 2

3 4

5

6

1 2

3 4

5 6

1 2

4

5 6

CRP ddCRP

φ1

φ3

φ2

φ

1φ

3φ

2

d13 =0

d

34 =0

d22 =0

d56 =0

k=1 n

1 =3 k=2 n

=2 k=3 n

3 =1

1

23

45

6

1

2

3

4

5

6

supersample groups k

l=1

4

12

3

56

2

3

5 4

6 1

12

34

56

1

2

3

φ

kl=11 =Ψ

q=1

data recording l=1 (restaurant)

…

Ψq=1

a)b)

c)

supersamples i1

customers i

customers i

tables k

tables k

φ21

=Ψ2

φ31

=Ψ3

supersamples i2


l=2

supersamples iL


l=L

φ12

=Ψ2

φ22

=Ψ3

φ32

=Ψ1

φ2L

=Ψ3

φ1L

=Ψ2

φ32

φ3L

φ11

φ21

φ12

φ2L

φ31

φ22

Ψq=2

Ψq=3

activity topic groups q(global tables)

2

(customers)

(customers)

(customers)

data recording l=2(restaurant)

data recording l=L(restaurant)

(local tables)(local tables)

(local tables)

CRP

activity qq=1: lunch

q=2: o�ce w

orkq=3: rest

local layer global layer

Chinese restaurant metaphor

3

5

6

4

φ3L

=Ψ1

φ1L

Activity discovery approach

ddCRPddCRP

ddCRP

Figure8.3:Illustration

ofdiscovery

processes.(a)C

hineserestaurant

process(C

RP):custom

ersare

assignedto

tablesk

accordingto

Eq.(8.1).

Custom

erssitting

atthe

same

tablehave

thesam

edish

φk .

(b)D

istancedependent

Chinese

restaurantprocess

(ddCR

P),introducingcustom

erdependencies:custom

ersiand

jwith

small

distancesd

ijare

likelyto

sitat

thesam

etable

kas

definedby

Eq.(8.2).

(c)hierarchical

discoveryfram

ework

ddCR

P+

CR

P.Restaurants

correspondto

datarecordings,custom

ersto

supersamples,tables

areindividualsupersam

plegroups,and

dishesareactivity

topics.Inthe

locallayer,supersamplesfrom

recordingslare

combined

tosupersam

plegroups

byddC

RP

(seeSection

8.3).Eachlocalsupersam

plegroup

kl belongs

toa

localactivity

topicφ

kl .Inthe

globallayer,local

supersample

groupsfrom

allLrecordings

areassigned

toglobalactivity

topicgroups

qusing

CR

Pand

shareglobalactivity

topicΨ

q .Localactivitytopics

φkl inheritthe

globalactivitytopic

Ψq .For

example,supersam

plesthat

areassigned

tosupersam

plegroups

k1 =

1,k2 =

3and

kL =

3belong

toactivity

q=1:lunch.

8.4. Discovery framework 177

number of customers nk already sitting at table k. The customer opensup a new table proportional to the parameter α:

p(zi = k|z1:i, α) ∝

nk k ≤ Kα k > K

, (8.1)

where zi is the table assignment of customer i. In case a new table isopened up, the dish φk at the new table is sampled from Go. In CRP,the table assignment is independent of previously entered customers.However, it might be more likely that e.g. customers, who enter therestaurant in close temporal relation sit at the same table. Thus, ddCRPintroduces customer dependencies [22], see Fig. 8.3(b). Customers i arelinked to other customers j based on their dependency di j. Linked cus-tomers share the same table k. In ddCRP, the probability that customeri is linked to customer j is inverse proportional to their distance di j,whereas customer i sits alone proportional to α:

p(ci = j|D, f , α) ∝

f (di j) j , iα j = i

, (8.2)

where ci is the customers assignment, f (d) denotes the decay functionand D the set of all distances between customers. For activity discovery,restaurants correspond to data recordings, customers to supersamples,tables to supersample groups, and dishes to activity topics.

8.4.4 Segmentation priors for activity discovery

In this work, the word2vec algorithm was used to extract vector rep-resentations of words, where the word vectors capture semantic re-lationships between words [21]. Word2vec is based on a continuousSkip-gram model that infers word vector representations unsuper-vised from a corpus of articles. Initially, the algorithm constructs aword2vec vocabulary of size W from the text corpus and then deducesvector representations based on neural networks. Finally, each wordin the word2vec vocabulary is represented by the semantic relationshipto W other words leading to a 1 ×W word vector for each word. Weused a word2vec vocabulary of dimension W = 1000 to extract wordvector representations from a corpus of Wikipedia articles (available athttps://code.google.com/p/word2vec/ ). Context words represented a subsetof the word2vec vocabulary (X << W) and were manually mapped to


relevant word vectors vx for X context words by searching the labelsof X context words in the word2vec vocabulary.

We used the word2vec-based semantic as well as temporal distancesbetween supersamples to form a segmentation prior over supersam-ples for ddCRP that likely groups supersamples belonging to the sameactivity (see Fig. 8.2). To semantically represent a context word x, weused word vector vx. To semantically represent supersample i, we cal-culated the mean word vector vi across all Xi unique context words x insupersample i: vi = 1

Xi

∑Xix∈Xi

vx. The semantic distance dsi j of supersam-

ples i and j is the Euclidean distance dsi j = d(vi, v j) of their mean word

vectors. The temporal distance dti j counts the number of supersamples

between supersample i and j. Considering our segmentation prior oversupersamples, we modified Eq. (8.2) for supersample assignment ci:

p(ci = j|D, f , α) ∝

f t(dti j) f s(ds

i j) j , i

α j = i. (8.3)

The distance measure D and decay function f for ddCRP are com-posed of a temporal distance measure and decay function (Dt, f t) anda semantic distance measure and decay function (Ds, f s). The windowdecay function f t(dt) = [dt < A] assigns direct linkage probabilitiesfor supersamples that are at most distance A apart. For the semanticdistance ds, we use an exponential decay function f s(ds) = exp(− ds

B ) thatdecreases linkage probability with increasing semantic distance. B isthe width parameter.

8.4.5 Joint segmentation and activity discovery (ddCRP+CRP)

Our activity discovery approach uses ddCRP in the local layer andCRP in the global layer as illustrated in Fig. 8.3(c). The ddCRP+CRPapproach can be interpreted as follows: There is a set of L data record-ings (restaurants) with a shared set of global activity topics Ψ (globaldishes) across all recordings (restaurants). For each recording l, su-persamples il and jl with small semantic and temporal distances di jlare likely grouped to the same supersample group kl (local table). Forexample, linked supersamples in Fig. 8.3(c) (bottom) are assigned tothe same supersample group. Each supersample group kl of all datarecordings l is assigned to one global activity topic Ψq (global dish) byCRP with the activity topic group q (global table) (see Fig. 8.3(c), top).The local activity topic φkl (local dish) in recording l inherits the global


activity topic Ψq (global dish) from the activity topic group q wherekl is assigned to, e.g. Fig. 8.3(c) Ψ1 = φ11 = φ3L = φ32. Thus, multiplesupersample groups kl in multiple data recordings l can belong to thesame activity topic Ψq.

The generative process ddCRP+CRP is described by:

1. Each supersample il in recording l draws supersample assign-ment cil with supersample group kli from ddCRP(D,f,α).

2. Each supersample group kl in recording l draws a global activitytopic group assignment qkl from CRP(γ).

3. Each global activity topic group q draws activity topic Ψq fromG0.

4. For each supersample il in recording l, context word statistics uilare drawn from ηq, where ηq is a multinomial distribution andqkli

= q.

Given the observed context word statistics ui for supersample i,the likelihood that ui is sampled from the global activity topic q isp(ui|Ψq) = ηq(ui). We used Gibbs sampling to infer the probabilitiesp(ui|Ψq) and thus the most likely activity topic assignment q for eachsupersample i as detailed in [20].

8.5 Evaluation methodology

The evaluation strategy is illustrated in Figure 8.4. We compared per-formance of our nonparametric ddCRP+CRP approach with data-driven supersamples segmentation and joint segmentation and ac-tivity discovery to the parametric LDA-based topic model with time-invariant segmentation (segment size DS) and predefined activity topiccount T. We further compared ddCRP+CRP to LDA with supersamplessegmentation and predefined T and to the nonparametric model CRFwith data-derived T, but time-invariant segmentation DS. CRF [23]is a hierarchical method as well. However, CRF uses CRP in the local


Time-invariantsegmentation

LDA

CRF

# topics data-driven

segmentation

ddCRP+CRP# topics, segmentation

data-driven

T topics

Contextwords

Segmentation prior

Supersamples

Prior

Windows

Activitytopics

Supersample

Data-driven segment size

Time-invariant segment size

Predefined #topics

Predefined #topics

LDA

T topics

Data-driven #topics

Data-driven #topics

Figure 8.4: Illustration of the evaluation strategy to assess and com-pare performance of our joint segmentation and activity discoveryapproach with other variants. We compare performance against LDAwith time-invariant segmentation and predefined T, LDA with super-samples segmentation and predefined T, and CRF with time-invariantsegmentation.


layer instead of ddCRP and thus does not consider segmentation priorsto segment supersample groups. All evaluations were performed perstudy participant and present average results across all participantsand 10 topic model runs. For LDA, we varied DS within [1, 5] minwith empirically optimal T = 10 as well as varied T within [5, 20] atDS = 2.5, as suggested in [6]. We evaluated discovery performance us-ing context word annotations that can be seen as perfect context worddetectors, as well as from encoded sensor data using the context vo-cabulary. We further investigated sensitivity to context word detectornoise by adding equally distributed noise to context word annotations.

8.5.1 Dataset

To evaluate our approach, we used the Opportunity dataset that con-sists of ∼ 30 hours of activities of daily living (ADL) recorded at 30 Hz,including annotations for 5 recordings from 4 participants [8]. ADLsincluded relaxing, coffee time, early morning, cleanup, sandwich time plus ahigh-level null class, in total 120 instances. The dataset further providesannotations for mode of locomotion (4 labels) and object usage (20 la-bels), plus a low-level null class. We considered ADLs as activities,mode of locomotion and object usage corresponded to context wordsresulting in 25 individual words. To infer context words from sensordata, we used the 3-axis acceleration signals accx,y,z of the right upperleg sensor SL and the back-worn sensor SB. We included 3-axis accel-eration sensor data of sensors SOi (i = 1...15) attached to 15 objects:salami, bread, sugar, bottle, milk, spoon, knife cheese, glass, cheese, door1, door2, plate,cup, knife salami, lazy chair. We used binary signals b of reed switches SOiattached to 5 objects (i = 16...20) including fridge, top drawer, middle drawer,lower drawer, dishwasher.

8.5.2 Framework implementation

We extracted a context vocabulary with X = 25 context words fromthe sensor data as detailed in Table 8.1. Context words included modeof locomotion and object use (Oi) resulting in 21 parallel context wordchannels (20 object channels, 1 channel for mode of locomotion). Super-samples segmentation from the context word stream was performedusing mode of locomotion as context state information, which is theleast sparse context word channel of the Opportunity dataset. We usedall 20 context word channels with object information to calculate the


Table 8.1: Summary of context vocabulary (25 context words). Featuresµ, σ2 of acceleration signal accx,y,z and binary signals b of switches weretransformed to binary features F using thresholds. Logic equationswere applied to obtain context words from binary features F of leg SL,back SB, and object sensors SOi.

Context Vocabulary Logic EquationsMode of Locomotion(1)walk, (2)lie, (3)sit, (4)stand (1): F1

SL, (2): F2SB ∧ F1

SB ∧ F1SL,

(3): F3SL ∧ F1

SL ∧ F2SB,

(4): F3SL ∧ F1

SL ∧ F2SB;

F1 = 1 : σ2(||accxyz||) ≥ 10000F2 = 1 : µ(||accyz||) ≥ µ(||accx||)F3 = 1 : µ(||accz||) ≥ µ(||accxy||)

Object Usage(4+i)motion Oi, i = 1...15 (4+i): F1

SOi

(25)null class (25): F1SOi

(15+i)motion Oi, i = 16...20 (15+i): F4SOi

; F4 = 1 : b = 1

(25)null class (25): F4SOi

8.6. Results 183

semantic distance dsi j between supersamples i and j. For ddCRP+CRP,

we used the implementation of [20] with width parameters B = 0.1for f s and A = 3 for f t, and hyperparameters α = 50, γ = 1 andη = 1. For CRF, we used hyperparameters α = 1, γ = 1 and η = 1. ForLDA we used the implementation of [24] with α = 1 and T topics. Fortime-invariant segmentation, we used sliding windows of size DS andsegment step 0.1∗DS and applied Borda Count ranking to overlappingsegments [25].

8.5.3 Performance estimation

To assess activity discovery performance we mapped discovered ac-tivity topics to activities by assigning the most frequent activity perpredicted activity topic using the groundtruth. Null class data was in-cluded for topic discovery, but removed in the performance analysis.As performance measure we used class-normalized accuracy across all5 activities and the Rand index RI.

8.6 Results

8.6.1 Semantic relationships within and between activities

Figure 8.5 shows that semantic distances Ds of context word vectorswere small among instances of the same activity, e.g. early morning.Confirming our approach to use semantic priors, independent activi-ties showed high distances, e.g. early morning and coffee time. Detectorerrors may have decreased within-activity similarity of detected con-text words compared to context labels, e.g. coffee time. Nevertheless,we also observed a reverse trend, where context word annotations ap-peared to be imperfect and incomplete compared to detections, e.g. forrelaxing, clean up and sandwich time.

8.6.2 Activity discovery from context word labels

ddCRP+CRP versus LDA

Our ddCRP+CRP approach yielded 83% accuracy and Rand index RI =0.83, clearly outperforming LDA as depicted in Fig. 8.6. LDA usingtime-invariant segmentation showed a peak in accuracy and Randindex for DS = 2.5 min and T = 10 topics.


510

1520

2530

51015202530

Relaxing

Relaxing

Co�ee

time

Co�ee

time

Earlym

orning

Earlym

orning

Clean-up

Clean-up

Sandwich

time

Sandwich

time

510

1520

2530

510152025300 0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

RelaxingCo�

eetim

eEarly

morning

Clean-up

Sandwich

time

Semantic D

istances - Detected Context W

ordsSem

antic Distances - Context W

ord Annotations

a)b)

Figure8.5:Illustration

ofsem

anticdistances

Ds

between

activityinstances

of5

activitiesin

theO

pportunitydatasetfor

(a)contextword

labelsand

(b)contextword

detectionsfrom

sensordata.Sem

anticdistances

were

calculatedfrom

contextw

ordvector

representationsand

averagedacross

all4

subjects.Thegraph

indicatesthatcontextw

ordsused

within

thesam

eactivity

haveclose

semantic

relation.

8.6. Results 185

ddCRP+CRP versus CRF

ddCRP+CRP outperformed nonparametric CRF with optimal segmentsize by 10% in accuracy and by RI = 0.02. With decreasing segmentsizes accuracy of CRF increased up to 73%. The Rand index showed apeak at DS = 2.5 min with RI = 0.81.

Temporal and semantic priors

We assessed the benefit of temporal and semantic priors. ddCRP+CRPwith semantic prior increased accuracy by 6% compared to dd-CRP+CRP with only temporal prior. The performance of ddCRP+CRPwith temporal prior was close to the performance of LDA and CRFwith optimal parameters.

8.6.3 Sensitivity to context word noise

Figure 8.7 shows that ddCRP+CRP was robust against deletion noisewith up to 60% deletions and 20% insertion noise, outperforming LDAat optimal parameter settings. In practice, uniformly distributed noiseacross context word detectors is unlikely to occur. Besides deletionsand insertions also timing and substitution errors may hamper dis-covery. The noise analysis may thus rather illustrate boundaries of ourddCRP+CRP approach: ddCRP+CRP performance depends on thesegmentation prior. For uniformly distributed insertion noise, ddCRPlikely grouped supersamples of different activities in the local layerleading to less distinct context word statistics of supersample groupsat the global CRP layer. Contrary, ddCRP+CRP was less affected bydeletions, as they only reduced priors grouping supersamples of thesame activity. In contrast, LDA estimates activity topics exclusivelyfrom context word statistics in time-invariant segments. Evenly dis-tributed noise offsets all context word statistics and therefore barelychanges the context word structure in a segment. The sensitivity ofddCRP+CRP for insertions and robustness against deletions suggeststuning context detectors for high precision.

8.6.4 Activity discovery from sensor data

For activity discovery from detected context words using our con-text word extraction approach, all methods showed decreased per-formance compared to discovery from annotations. Figure 8.8 shows


1 2 3 4 540

50

60

70

80

90


norm

aliz

ed a

ccur

acy

[%]

ddCRP(Dt,Ds)+CRP ddCRP(Dt)+CRP CRF* LDA&supersamples LDA*

1 2 3 4 50.65

0.7

0.75

0.8


Ran

d in

dex

5 10 15 2040

50

60

70

80

90

number of topics T

norm

aliz

ed a

ccur

acy

[%]

5 10 15 200.65

0.7

0.75

0.8

number of topics T

Ran

d in

dex

Figure 8.6: Averaged normalized accuracy and Rand index for discov-ering activities from context word labels. Results are shown for dd-CRP+CRP with temporal (Dt) and semantic (Ds) segmentation priors,CRF, and LDA. ddCRP+CRP outperformed nonparametric CRF andparametric LDA. (*) We varied segmentation window and number oftopics for CRF and LDA-based methods when parameter dependencywas present.

8.7. Discussion 187

100 90 80 70 60 50 40 30 20 10 0 10 20 30 40 50 60 70 80 90 10040

50

60

70

80

90no

rmal

ized

acc

urac

y [%

]ddCRP+CRP LDA

100 90 80 70 60 50 40 30 20 10 0 10 20 30 40 50 60 70 80 90 1000.6

0.65

0.7

0.75

0.8

noise [%]

Ran

d in

dex

insertionsdeletions

Figure 8.7: Influence of evenly distributed noise over context worddetectors on discovery performance. ddCRP+CRP was robust againstcontext word deletions up to 60%, but showed sensitivity to inser-tions. Our approach outperformed LDA between 60% deletion and20% insertion noise.

that our ddCRP+CRP model outperformed LDA with time-invariantsegmentation and optimal parameters (T = 7, DS = 2.5) by 4.5% accu-racy and ∆RI = 0.1. For LDA, optimal activity topic count decreasedfor detected context words, compared to the discovery from annota-tions (T = 7 vs. T = 10). Moreover, optimal segment size changed (DS =3.5 vs. DS = 2.5). Our ddCRP+CRP model automatically selected asmaller number of activity topics for activity discovery from detectedcontext words compared to context word labels (T = 7 vs. T = 15).ddCRP+CRP with just temporal segmentation prior performed with60% accuracy worse than ddCRP(Dt,Ds)+CRP (78%). CRF with opti-mal segment size DS = 2 min yielded the same accuracy and slightlysmaller Rand index ∆RI = −0.1 as ddCRP+CRP, but more activitytopics ∆T = 5.

8.7 Discussion

By introducing a framework for joint segmentation and activity dis-covery in this work, the time-invariant segmentation and parametersused in previous works towards unsupervised activity discovery wereremoved. Our ddCRP+CRP approach performed supersamples seg-


1 2 3 4 540

50

60

70

80

90


norm

aliz

ed a

ccur

acy

[%]

ddCRP(Dt,Ds)+CRP ddCRP(Dt)+CRP CRF* LDA&supersamples LDA*

1 2 3 4 50.65

0.7

0.75

0.8


Ran

d in

dex

5 10 15 2040

50

60

70

80

90

number of topics T

norm

aliz

ed a

ccur

acy

[%]

5 10 15 200.65

0.7

0.75

0.8

number of topics T

Ran

d in

dex

Figure 8.8: Performance of activity discovery from context word de-tections for ddCRP+CRP including temporal Dt and semantic Ds seg-mentation priors, CRF and LDA. Our nonparametric ddCRP+CRP ap-proach outperformed parametric LDA and nonparametric CRF at theiroptimal parameter settings. (*) We varied segmentation window andnumber of topics for CRF and LDA-based methods when parameterdependency was present.

8.7. Discussion 189

mentation and activity discovery simultaneously and outperformedthe parametric LDA as well as nonparametric CRF, both using time-invariant segmentation. In future work, the performance of motifsearch or other data-driven segmentation approaches for discretizeddata separately or in combination with LDA, ddCRP+CRP or otherdiscovery methods could be compared to our ddCRP+CRP approach.

We used basic logic functions to extract context words, thus avoid-ing statistical classifier learning. Our data-driven context word seg-mentation generated supersamples that were typically shorter thanactivities. Hence, individual supersamples did not capture distinctcontext word statistics to describe activities and may explain the poorperformance of LDA using supersamples segmentation. Using a tem-poral segmentation prior for ddCRP+CRP increased accuracy overthe LDA-based approach, but performed less accurate compared toddCRP+CRP using a temporal and semantic segmentation prior. Fordiscovery from context labels, ddCRP+CRP outperformed CRF by 10%in accuracy. For discovery from detected context words, both methodsshowed similar peak accuracy. However, CRF yielded 5 additionalactivity topics compared to ddCRP+CRP (CRF: T = 12 at DS = 2,ddCRP: T = 7). For an intuitive mapping, T in the range of M was de-sirable. Thus, results were shown for T < 20 activity topics to describethe M = 5 activities. CRF obtained T > 20 for DS < 2 min.

While the nonparametric model ddCRP+CRP and CRF infer op-timal activity topic count T, hyperparameters α, γ determine the ex-pectation over T. There are no established strategies to select α, γ.However, if a range for T̂ ≈ T ± 5 is estimated, ddCRP+CRP and CRFcan automatically choose an optimal T. In our work, we used the samehyperparameter setting for discovery using labels and detected con-text words. In our tests, ddCRP(Dt,Ds)+CRP showed similar discoveryperformance even when hyperparameters were varied, indicating ro-bustness of the method. Omitting semantic priors, i.e. ddCRP(Dt)+CRPshowed lower robustness to hyperparameter variation that may ex-plain the performance difference between labels and detected contextwords.

We segmented context words into supersamples and used contextstate changes to determine supersample bounds. In this work, we onlyconsidered context changes in one selected context word channel. Statechanges could similarly be estimated by combining several contextword channels to create a virtual context state. Selecting and construct-ing a segmentation source still requires expert knowledge about the


targeted discovery objective and context word processing. Similarly,constructing logic functions to derive context words requires knowl-edge about the sensor modalities and discovery goals. Nevertheless,we consider that such logic functions could be cataloged according tosensor type and scenario, thus become reusable for similar discoveryapplications without parametric adjustments.

Due to the limited availability of datasets exhibiting a hierarchicalannotation structure we evaluated our method with the Opportunitydataset. In future work, other datasets could be analyzed to verifyscalability of the method. Our approach required context words withsemantic meaning, as we used word2vec to formulate a segmentationprior. Nevertheless, word2vec is flexible and could be applied to a dif-ferent corpus, e.g. containing data clusters or other symbols extractedfrom sensor data. In our approach context words corresponded to asmall subset of the word2vec word vocabulary and we manually ex-tracted context word vectors from the word vocabulary. Instead, stringmatching could be applied in future to automate the mapping. In ourevaluation, all context words could be mapped to a word vocabularyof W = 1000. It is nevertheless simple to increase the vocabulary, ifcorresponding words could not be found or to search synonyms usinglanguage processing algorithms. We used a generic text corpus fromWikipedia to extract the word2vec vocabulary. Domain specific text cor-pora might yield even more relevant word coverage and context wordvectors.

8.8 Conclusion and future work

We introduced a novel non-parametric topic model approach for jointsegmentation and activity discovery from sensor data that is indepen-dent from topic model parameters, such as segment size and number oftopics. We segmented context words into supersamples using contextstate and formulated a segmentation prior with semantic and temporalinformation to group supersamples that belong to individual activitiesusing ddCRP and CRP. With this method, segmentation is adjusted tothe underlying data. Evaluation results show that our approach canoutperform classical parametric LDA and non-parametric CRF evenat optimal parameter settings. We concluded that combining segmen-tation and non-parametric activity discovery by using a segmentationprior and ddCRP+CRP is an adequate technique for activity discov-

8.9. Acknowledgment 191

ery. In future work, we like to adapt the segmentation prior to datasetswith different sensor modalities and discovery objectives.

8.9 Acknowledgment

We thank Zeynep Akata for providing the word2vec vector representa-tions extracted from Wikipedia articles. This work was supported bythe EU Marie Curie Network iCareNet under grant number 264738.


Bibliography

[1] J. Seiter, A. Derungs, C. Schuster-Amft, O. Amft, and G. Tröster,“Activity routine discovery in stroke rehabilitation patients with-out data annotation,” in Proceedings of the 8th International Confer-ence on Pervasive Computing Technologies for Healthcare, pp. 270–273,2014.




[5] F.-T. Sun, Y.-T. Yeh, H.-T. Cheng, C. Kuo, and M. Griss, “Non-parametric discovery of human routines from sensor data,” inIEEE International Conference on Pervasive Computing and Commu-nications (PerCom), pp. 11–19, IEEE, 2014.


[7] T. Nguyen, “Bayesian nonparametric extraction of hidden con-texts from pervasive honest signals,” in PERCOM Workshops,pp. 168–170, 2014.






[12] J. Zheng and L. M. Ni, “An unsupervised framework for sens-ing individual and cluster behavior patterns from human mobiledata,” in Proceedings of the ACM Conference on Ubiquitous Comput-ing, pp. 153–162, 2012.

[13] D. H. Hu, X.-X. Zhang, J. Yin, V. W. Zheng, and Q. Yang, “Abnor-mal activity recognition based on hdp-hmm models.,” in IJCAI,pp. 1715–1720, 2009.

[14] Y. Zhu, Y. Arase, X. Xie, and Q. Yang, “Bayesian nonparamet-ric modeling of user activities,” in Proceedings of the InternationalWorkshop on Trajectory Data Mining and Analysis, pp. 1–4, ACM,2011.

[15] D. Griffiths and M. Tenenbaum, “Hierarchical topic models andthe nested chinese restaurant process,” Advances in Neural Infor-mation Processing Systems, vol. 16, p. 17, 2004.


[17] D. Kuettel, M. D. Breitenstein, L. Van Gool, and V. Ferrari, “What’sgoing on? discovering spatio-temporal dependencies in dynamicscenes,” in CVPR, pp. 1951–1958, IEEE, 2010.

[18] E. B. Sudderth, A. Torralba, W. T. Freeman, and A. S. Willsky,“Describing visual scenes using transformed objects and parts,”International Journal of Computer Vision, vol. 77, pp. 291–330, 2008.

195

[19] S. Ghosh, A. B. Ungureanu, E. B. Sudderth, and D. M. Blei, “Spatialdistance dependent chinese restaurant processes for image seg-mentation,” in Advances in Neural Information Processing Systems,pp. 1476–1484, 2011.

[20] W.-C. Chiu and M. Fritz, “Multi-class video co-segmentation witha generative multi-video model,” in CVPR, pp. 321–328, IEEE,2013.

[21] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Dis-tributed representations of words and phrases and their compo-sitionality,” in Advances in Neural Information Processing Systems,pp. 3111–3119, 2013.

[22] D. M. Blei and P. I. Frazier, “Distance dependent chinese restau-rant processes,” The Journal of Machine Learning Research, vol. 12,pp. 2461–2488, 2011.




Glossary

Notation Descriptionactivities We use the term activities as abbreviation of the

term activity composites. 7, 8, 32, 34activity composites Abstract activities in an activity hierarchy in-

cluding high-level activities and activity rou-tines that are composed of several activity prim-itives. 2, 6, 7

activity primitives Activities in an activity hierarchy with little de-gree of abstraction including gestures, actionsand interactions with objects or persons. 2, 3, 7,36, 41, 56

activity routines Activity routines are activity composites thathumans perform periodically according to per-sonal habits, e.g., lunch. 7, 34, 41

activity topics Characteristic activity patterns discovered incontext words using topic models. 6, 32, 54

AL Activity level measured from body-worn sen-sors. 40

ALq Activity level assessed from questionnaires. 40

context Any information that describes the situation ofa person, e.g., location, activity, object usage. 5

context words Context words describe the context detectedfrom sensor data and include activity primi-tives, location and object use. 7, 32, 36, 49, 54

CRP Chinese restaurant process. 55–57CTM Correlated topic model. 12, 42, 45, 58, 59

ddCRP Distance dependent Chinese restaurant pro-cess. 53, 54, 56, 57, 59, 60

DS Segment size to partition the context word se-quence using windowed segmentation. 32, 33,36, 43, 45, 46, 48, 49, 52, 57

DTM Dynamic topic model. 12

198

Notation DescriptionEM Expectation-maximization algorithm. 11, 47

GPS Global positioning system. 5

HF High frequency band. 35HMM Hidden Markov model. 8, 47

IMU Inertial measurement unit. 32, 34

K-means K-means clustering algorithm. 36, 44kNN The k-nearest neighbor algorithm. 36, 62

LDA Topic model based on latent Dirichlet alloca-tion. 10, 12, 13, 36, 42, 48, 56, 58, 59

LF Low frequency band. 35

M Number of groundtruth activities. 33, 36, 46MC Markov chain. 47

NTM The n-gram topic model. 14, 42, 44, 58, 59

PLq Pain level assessed from questionnaires. 40

stability Stable activity topic discovery measured by thestandard deviation of the activity discovery ac-curacy across multiple TM runs. 15, 49, 52

T Number of activity topics for the TM. 32, 33, 36,46, 58

TM Topic model. 9, 32, 34, 40, 47, 52

Curriculum Vitae

Personal Information

Julia Seiter

Born 31 August 1985, in Pforzheim, GermanyCitizen of Germany

Education

2011–2015 Doctoral studies (Dr. sc. ETH) in Information Technology andElectrical Engineering, ETH Zurich, Switzerland.

2005–2011 Dipl.-Ing. in Electrical Engineering and Information TechnologyKarlsruhe Institute of Technology (KIT), Germany.

1996–2005 Abitur, Lise-Meitner Gymnasium, Königsbach, Germany.

Work Experience

2011–2015 Research assistant, Electronics Laboratory,ETH Zurich, Switzerland.

2014–2014 Research visit, Max Planck Institute for Informatics (MPI),Saarbrücken, Germany.

2009–2010 Research student, Karlsruhe Institute of Technology (KIT),Karlsruhe, Germany.

2008–2009 Internship, Continental Automotive InstrumentsMalaysia Sdn Bhd, Penang, Malaysia.

2007–2008 Student teaching assistant,Karlsruhe Institute of Technology (KIT), Germany.

2005–2005 Internship, Kunzmann Maschinenbau GmbHNöttingen, Germany.