Two Studies of Opportunistic Programming: Interleaving Web Foraging, Learning, and Writing Code

Stanford HCI Group Adobe Advanced Technology Labs

Two Studies ofOpportunistic Programming: Interleaving Web Foraging, Learning, and Writing Code

Joel Brandtwith Philip Guo (S), Joel Leweinstein (S), Mira Dontcheva (A), Scott

Klemmer (S),

http://hci.stanford.edu/opportunistic

2

“good grief, I don’t even

remember the syntax for

forms!”

3

4

5

6

7

8

9

10

11

12

13

14

15

The Web has fundamentally changed the cost structure

of information access.

Enables an opportunistic

approach to programming

16

People program opportunistically to prototype, ideate, and discover

Emphasize speed and ease of development over code robustness

and maintainability

17

http://flickr.com/photos/royalsapien/2387707860/

Designers…

18

http://flickr.com/photos/niosh/2492023651/

Scientists…

19

http://www.flickr.com/photos/samthor/3198975900/

Makers…

20

By 2012:13 million who program as part of their job…

…but only 3 million who are

professional programmers

[Scaffidi 2005]

21

How do programmers use online resources?

22

Related WorkWeb Search for Programmers

[Stylos 06, Hoffmann 07]

Code Cloning in Software Engineering[Kim 04, Ducasse 99]

Learning Barriers of Programming[Ko 04]

Learning on the Web[Torrey 09]

23

Study 1: Prototyping a

Web-based chat room

Study 2: Log Analysis of Programmers’

Web Queries

24

Study 1: Web Chat Room 20 participants (9 Graduate, 11

Undergrad) “basic knowledge” of PHP, JS, and

HTML 2.5 hours each use any resources 3 researchers observed,

one asked questions to prompt think aloud

METHOD

25

Chat Room Specifications

1. Set username

2. Post messages

3. Update without page reload

4. Messages have timestamp

5. Limited chat history

METHOD

26

27

Most Subjects Met All Specs

� � � ��

user

nam

epo

st m

essa

geAJ

AX u

pdat

etim

esta

mp

hist

ory

� Specification Met

� Specification Attempted

subj

ects

RESULTS

28

Frequent Web Usesu

bjec

ts

time (minutes)

RESULTS

29

Frequent Web Usesu

bjec

ts

time (minutes)

RESULTS

30

Frequent Web Usesu

bjec

ts

time (minutes)

RESULTS

31

Frequent Web Usesu

bjec

ts

time (minutes)

RESULTS

32

The Web plays multiple roles

RESULTS

leng

th (s

econ

ds)

session (sorted by length)

33


RESULTS

leng

th (s

econ

ds)


34


RESULTS

leng

th (s

econ

ds)


35

Three Intentions Behind Web UseLearning

just-in-time acquisition of new skills

Clarification connecting high-level knowledge to implementation details

Reminder offloading memory to external resources

RESULTS

36

Just-in-Time Learning-by-DoingCopy and modify code before reading prose

RESULTS

37

“There’s some stuff in this code that I don’t really know

what it’s doing, but I’ll just try it and

see what happens.”

38


Not concerned with deep learning

RESULTS

39

“I think it’s less expensive for me to just take the first code I find and see how helpful it is at … a very high level … as

opposed to just reading all these

descriptions and text”

40


Not concerned with deep learning

Learn new terminology from result snippets

RESULTS

41

Example Query

Terms

TimeNumber of

Refinements

Number of Result Clicks

Types of pages visited

Lines of code

copied

42

Learning

Example Query

Terms

TimeNumber of

Refinements



Lines of code

copied

43

Learning

Example Query ajax tutorial

Terms

TimeNumber of

Refinements



Lines of code

copied

44

Learning


Terms English

TimeNumber of

Refinements



Lines of code

copied

45

Learning


Terms English

Time 10s of minutes

Number of Refinement

sNumber of

Result Clicks


Lines of code

copied

46

Learning


Terms English

Time 10s of minutes


s2-3



Lines of code

copied

47

Learning


Terms English

Time 10s of minutes


s2-3


5+Types of

pages visited

Lines of code

copied

48

Learning


Terms English

Time 10s of minutes


s2-3


5+Types of

pages visited

tutorials, how-to articles

Lines of code

copied

49

Learning


Terms English

Time 10s of minutes


s2-3


5+Types of

pages visited


Lines of code

copieddozens

50

Clarification of Existing Knowledge

Web search enables translations and “language analogies”

Know exactly what code does when they see it

Copied code often not immediately tested

RESULTS

51

Learning


Terms English

Time 10s of minutes


s2-3


5+Types of

pages visited


Lines of code

copieddozens

52

Learning Clarification


Terms English

Time 10s of minutes


s2-3


5+Types of

pages visited


Lines of code

copieddozens

53


Example Query ajax tutorial javascript

thread

Terms English

Time 10s of minutes


s2-3


5+Types of

pages visited


Lines of code

copieddozens

54



thread

Terms English mix of code and English

Time 10s of minutes


s2-3


5+Types of

pages visited


Lines of code

copieddozens

55



thread


Time 10s of minutes

about 1 minute


s2-3


5+Types of

pages visited


Lines of code

copieddozens

56



thread


Time 10s of minutes

about 1 minute


s2-3 0-1


5+Types of

pages visited


Lines of code

copieddozens

57



thread


Time 10s of minutes

about 1 minute


s2-3 0-1


5+ 1-2Types of

pages visited


Lines of code

copieddozens

58



thread


Time 10s of minutes

about 1 minute


s2-3 0-1


5+ 1-2Types of

pages visited


API docs, blogs, forums

Lines of code

copieddozens

59



thread


Time 10s of minutes

about 1 minute


s2-3 0-1


5+ 1-2Types of

pages visited



Lines of code

copieddozens several

60

Reminding about forgotten details Explicitly choose not to learn

things Search serves as “Web-

enabled auto-complete”

RESULTS

61

62

63

Reminding about forgotten details Explicitly choose not to learn

things Search serves as “Web-

enabled auto-complete” Despite utility, can break flow

RESULTS

64



thread


Time 10s of minutes

about 1 minute


s2-3 0-1


5+ 1-2Types of

pages visited



Lines of code


65

Learning Clarification Reminder


thread


Time 10s of minutes

about 1 minute


s2-3 0-1


5+ 1-2Types of

pages visited



Lines of code


66



threadfetch_array

php


Time 10s of minutes

about 1 minute


s2-3 0-1


5+ 1-2Types of

pages visited



Lines of code


67



threadfetch_array

php

Terms English mix of code and English mostly code

Time 10s of minutes

about 1 minute


s2-3 0-1


5+ 1-2Types of

pages visited



Lines of code


68



threadfetch_array

php


Time 10s of minutes

about 1 minute

less than 1 minute


s2-3 0-1


5+ 1-2Types of

pages visited



Lines of code


69



threadfetch_array

php


Time 10s of minutes

about 1 minute

less than 1 minute


s2-3 0-1 0


5+ 1-2Types of

pages visited



Lines of code


70



threadfetch_array

php


Time 10s of minutes

about 1 minute

less than 1 minute


s2-3 0-1 0


5+ 1-2 0-1Types of

pages visited



Lines of code


71



threadfetch_array

php


Time 10s of minutes

about 1 minute

less than 1 minute


s2-3 0-1 0


5+ 1-2 0-1Types of

pages visited



API docs,result

snippetsLines of

code copied

dozens several

72



threadfetch_array

php


Time 10s of minutes

about 1 minute

less than 1 minute


s2-3 0-1 0


5+ 1-2 0-1Types of

pages visited



API docs,result

snippetsLines of

code copied

dozens several none to several

73

How do these results generalize?

74

Study 2: Query Log Analysis Adobe Community Help logs from July

2008 ~25,000 programmers performing

101,289 queries about Adobe Flex Framework

Grouped queries into 65,955 search sessions

Analyzed three properties of sessions: Format of query strings Process of query refinement Types of Web pages visited

Hand-coded 300 sessions for intent

METHOD

75

Query format predicts types of pages visited

76


Page type code only English + code English only

Adobe APIs 38% 16% 10%Adobe tutorials 31% 33% 39%

blogs 15% 22% 19%forums 3% 7% 6%

unclassified 13% 22% 27%All inter-row differences significant at p < 0.001

RESULTS

77




blogs 15% 22% 19%forums 3% 7% 6%

unclassified 13% 22% 27%

RESULTS

All inter-row differences significant at p < 0.001

78




blogs 15% 22% 19%forums 3% 7% 6%


RESULTS


79




blogs 15% 22% 19%forums 3% 7% 6%


RESULTS


80

Reminder sessions are typically code-only queries

81

Query format indicative of intention

query format reminding learning

code only 56% 21%English + code 10% 29%

English only 34% 50%

RESULTS


82

Programmers rarely need to refine queries

83

Programmers refine rarely

RESULTS

num

ber

of q

uerie

s

query number in session

84

…and they start out with either words or code

RESULTS

perc

ent o

f que

ries

query number in session

85

Five Key Insightsfor Tool Design

86

Just-in-time learning is common

Create tutorials that teach “how the code works”

87

Copied code often not immediately tested

Demarcate copied code,highlight what is executed

88

Web search often used as a “translator”

Improve search ranking with synonyms,

perform searches automatically

89

Programmers choose not to learn syntax

Make Web-enabled auto-complete

90

Query refinement is often predictable

Automatically augment queries

91

Preview of Ongoing Work

BlueprintEnabling Example-Centric Programming

92

Stanford HCI Group Adobe Advanced Technology Labs

http://hci.stanford.edu/opportunistic

95

96



threadfetch_array

php


Time 10s of minutes

about 1 minute

less than 1 minute


s2-3 0-1 0


5+ 1-2 0-1Types of

pages visited



API docs,result

snippetsLines of

code copied

dozens several none to several

Two Studies of Opportunistic Programming: Interleaving Web Foraging, Learning, and Writing Code

Documents

code robustness

type of programming

web search

interleaving web foraging

short blocks of code

user interfaces17 http

experiments18 http

webbased chat room