Techniques and tools for field-based early-stage study and iteration of Ubicomp applications 1 Scott Carter EECS Department University of California, Berkeley.

Techniques and tools for field-based early-stage studyand iteration of Ubicomp applications

1

Scott CarterEECS DepartmentUniversity of California, [email protected]

CommitteeJennifer Mankoff (chair), Anind Dey, John Canny, Peter Lyman

2

what are ubicomp apps? examples…social proximity jabberwocky (paulos ’04)

location urban tapestry (lane ‘05)

activity

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

resource queries transit services tour guide (abowd ’97)

3

capture and access field data health (alivetech ’05) reminder apps


gaming cysmn (blast theory ’01)

location,proximity-based

what are ubicomp apps? examples…

4


ubicomp evaluation is hard (Carter ’05)

5


ubicomp evaluation is hard

rapid prototyping

6



rapid prototyping

scale

7



rapid prototyping

scale

unobtrusiveness

8



rapid prototyping

scale

unobtrusiveness

data sparsity

9


thesis

Leverage pre-existing infrastructures and the increasing power of mobile devices to address the challenges of early stage field evaluations of ubicomp applications.

10


why early stage and field?

early stage

field

>> potential to iterate>> could answer some research questions

>> realism especially important in ubicomp >> Sharp & Rehman ‘05: “Real-world deployment is the only way”>> Kjeldskov: might be able to recreate field in a lab

11


approach: WoZ

researcher designs app behavior in momento and provides instructions for wizards

wizards use momento to simulate app behavior

12


approach: report methods

researcher designs experiment, configures momento or reporter

researcher runs experiment, monitoring tool

13


limitations: when challenges do not applydata easy to get (no sparsity)

infrastructure already available (no scale issues)

target users early, early adopters (no unobtrusiveness)

tools off-the-shelf (no prototyping)

14


most relevant related work

rapid prototyping

Topiary (Li ’04)

scale

unobtrusiveness

data sparsity

rapid prototypingContextESM (Intille ’03)

scale

unobtrusiveness

data sparsity

WoZ Report methods

SUEDE (Klemmer ‘00)

DART (Dow ‘05)

Doorman (Makela ‘01)

Crossweaver (Sinha ‘03)

Active Capture (Chang ‘05)

other work

iESP (Intel ’03)

Mobile probes (Hulkko, ‘04)

other work

15


outline

ubicomp challenges case study

report methods

momento

validation plans

timeline

summary

16


ubicomp challenges case study: dr. bluesocial mobile game

get points by collecting cures, healing the sick

17


ubicomp challenges case study: dr. blue

18


ubicomp challenges case study: dr. blue

19


case study: dr. blue: rapid prototypingmajor difficulty: building a mobile application

communicationsensordisplay

also difficult: game engine

20


case study: dr. blue: scale

everyone needs a device and recording method

21


case study: dr. blue: unobtrusivenessnew device and interface

interruptions: evaluation and application behavior

22


case study: dr. blue: sparsity

encounters may be limited

23


outline


report methods

momento

validation plans

timeline

summary

24

momento

WoZ simulation using SMS/MMS infrastructure


25

momento

WoZ simulation using SMS/MMS infrastructure

… and J2ME on the mobile device


26

momento

input and output


27

momento: building dr. blue


28



29



30



31



32

momento: limitations

SMS/MMS, wizard latency


24-hour wizards?

synchronous and streaming data

gui creation (damask – Lin ’02)

momento: summary


scale

unobtrusivenessdata sparsity

rapid prototypingany device supporting sms/mms

esm feedback on critical events

easy to change application behavior

integrate context

more control

participants

rules to take advantage of external services

wizards

familiar devices, but interruptions

34


outline


report methods

momento

validation plans

timeline

summary

35

report methods

diary study

experience-sampling method (esm)


36

diary study issues


unobtrusiveness

data sparsity

scale

neglect due to burden of study

generic responses

no feedback

data re-entry, transcribing

37

diary study technique (Carter & Mankoff ’05)

capture

annotate

upload

feedback

elicitation


38

diary study technique: reporter


39

reporter


scale


rapid prototypingany digital capture device

rapid feedback cycle

-

familiar devices

no interruptions

no transcribing

responses tied to captured artifacts feedback not in situ

40

esm: momento


scheduled or context events

automatic question formatting

41

esm: momento


scale


many devices

responses in situ

rapid prototypingknowledge feedback to application behavior

redirect questions

participants

integrate context

familiar devices, but interruptions

42


outline


report methods

momento

validation plans

timeline

summary

43


validation: diary study pipeline

testbed: diary study at uc berkeley

interviews with researchers

compare to traditional diary studies (literature)

unobtrusiveness

data sparsity

scale

success?

fewer instances of neglect due to burden of study

more specific recall, fewer generic responses

reduce time to data analysis (no data re-entry)

44


validation: momento esm study

sensor data for awareness app

location, activity data

interviews with wizards, participants

unobtrusiveness

data sparsity

scale

success?

minimal neglect due to burden of study

useful feedback on critical events

wizards able to manage momento queries and complete other work

45


validation: momento scale study

implement dr. blue

wide deployment (>~30)

feedback on critical events, retainment

unobtrusiveness

data sparsity

scale

success?

minimal neglect due to burden of study

useful feedback on critical events

wizards able to manage momento queries and complete other work

rapid prototypingiteration time shorter than similar social mobile apps

46


contributions

a tool (momento) that improves evaluations by simplifying prototyping, making them easier to scale, reducing obtrusiveness, and facilitating data collection

a diary study technique that is easier to scale, less obtrusive, and yields better quality data than previous diary study techniques

identification of major challenges in ubicompevaluation: rapid prototyping, scale, unobtrusiveness, and data sparsity

validation of momento and diary study technique

47


timeline

48


timeline

49


timeline: diary study

50


timeline: momento

51


summary

Leverage pre-existing infrastructures and the increasing power of mobile devices to address the challenges of field evaluations of Ubicomp applications

Wizard-of-Oz and report methods for early stage, field experiments

Ubicomp evaluation challenges: rapid prototyping, scale, unobtrusiveness, data sparsity

52


acknowledgements

Scott Lederer, Christopher Beckmann, Jonathan Hey, Jeffrey Heer, Tara Matthews, Scott Klemmer, James Lin, Tye Rattenbury, Alan Newberger, David Nguyen, Judd Antin, Joe Hall, Nathan Good, Siobhan Hadley

Prof. Robert Kraut, Prof. Susan Fussell, Prof. Jason Hong, Laura Dabbish, Joe Tullio, Jeffrey Nichols, James Fogarty

Techniques and tools for field-based early-stage studyand iteration of Ubicomp applications

53

Scott CarterEECS DepartmentUniversity of California, [email protected]

CommitteeJennifer Mankoff (chair), Anind Dey, John Canny, Peter Lyman

54


other slides

55


worldwide txt stats

number of computer users: about 1 bil by 2006 (IDC)email users: 700 mil by 2006 (Radicati Group)

about 2 bil mobile subscribers by 2006 (In-Stat/MDR)70% use sms (text.it) (number much lower in us)

56


scope

formative and early stages

field methods

distributed, asynchronous activities

>> easier to iterate, could answer some research questions

>> realism especially important in ubicomp >> Sharp & Rehman ‘05: “Real-world deployment is the only way…”>> Kjeldskov: might be able to recreate field in a lab

>> distributed harder, concentrate on asynchronous

broad user base>> minimize programming

57


rapid prototyping

scale

unobtrusiveness

data sparsity

ambiguity

>> simplify interface iteration, reduce coding, reduce burden of reinstallations

>> minimizing infrastructure deployment and time, encourage uptake

>> use familiar devices and lower distraction costs

>> capture and encourage feedback from critical events

>> allow people to control error handling

thesis: addressing challenges

58


approach: wizard of oz

Kelley (1984)

59



Kelley (1984)

60



61



diary

experience sampling

62



diary

experience sampling>> researchers send questions

63



diary

experience sampling

>> participants report on events

>> researchers send questions

64



65

diary studies

feedback or elicitation

participants report on events

form completion (feedback)participant controlled capture (elicitation)


66

advances in recording devices

smaller

part of everyday infrastructure

easier to transfer files

annotation

new, more frequent uses for media


67

findings

+ photos best for who and where recognition

audio elicitation

tangible objects

raw location information

- poor recognition

+/- clandestine captures

- poor recall

+ attitudes and beliefs

- poor sequencing and recall

+ peripheral information often crucial


68

findings

tangible objects and attitudes and beliefs


69

findingsperipheral information often crucial


70

momento

desktop


71

applications


72

applications

interruptibility updates (social, activity)

Momento: How busy are you right now? 1 not busy at all, 5 is really busy.

…

P: Who is most available?

Momento: Joe


73

applications

txt adventure (social, gaming)

Momento: You're in a forest. There is a trail to the N and a cabin to the W. Which way would you like to go? N/W

P: N

Momento: You open the cabin but there was a troll and he squashed you with his fist. Game over. Bad luck


74

applications

near by (location, resource)


75

applications

idea tagger (capture and access)


76

diary study or esm?


77

diary study or esm?


>> auto-driven: participant-sensed events>> creativity, attitudes, beliefs, breadth

diary study

78

diary study or esm?


>> auto-driven: participant-sensed events>> creativity, attitudes, beliefs, breadth

diary study

>> researcher-driven: more control>> time use, depth

esm

79


validation: interface support

augment current application

add functional feature

success?

demonstrate it works

no deployment

novel approach

80



81



82



83


other slides

When Participants Do the Capturing

84

Carter & Mankoff

diary studies and related methods

beeper studies

When Participants Do the Capturing

85

Carter & Mankoff

diary studies and related methods

cultural probes

86



87



rapid prototyping

scale

unobtrusiveness

data sparsity

ambiguity

88


case study: dr. blue: ambiguity

89


ubicomp evaluation challenges

rapid prototyping

scale

unobtrusiveness

data sparsity

ambiguity

Metrics: adoption, awareness, distraction, invisibility, (Scholtz and Consolvo)

90


ubicomp challenges case study: hebb

91


case study: hebb: rapid prototyping

92


case study: hebb: rapid prototyping>> simplify interface iteration, reduce coding, reduce burden of reinstallations

93


case study: hebb: scale

94


case study: hebb: scale>> minimize infrastructure deployment and time, encourage uptake

95


case study: hebb: unobtrusiveness

96


case study: hebb: unobtrusiveness>> use familiar devices and lower distraction costs

97


case study: hebb: sparsity

98


case study: hebb: sparsity

>> capture and encourage feedback from critical events

99


case study: hebb: ambiguity

100


case study: hebb: ambiguity

>> allow people to control error handling

101


ubiquitous computing (weiser)

102


ubiquitous computing

off-the-desktop interfaces

Techniques and tools for field-based early-stage study and iteration of Ubicomp applications 1 Scott Carter EECS Department University of California, Berkeley.

Documents