Direct Manipulation Programming Systemspeople.cs.uchicago.edu/~rchugh/static/talks/feb-2017-sns.pptx.pdf · February 20, 2017 Direct Manipulation Programming Systems Ravi Chugh Brian

February 20, 2017

Direct Manipulation Programming Systems

Ravi Chugh

Brian Hempel Jacob Albers

Grace Lu Justin Lubin

Mitch Spradlin

[Wikipedia]

(d)

(a) (b) (c)

(e) (f)Task:

Q: What Would You Choose? Programming Language

(e.g. JavaScript, Processing) Direct Manipulation System (e.g. Illustrator, PowerPoint)

2

(d)

(a) (b) (c)

(e) (f)Task:

Q: What Would You Choose? Programming Language

(e.g. JavaScript, Processing) ✗ Direct Manipulation System (e.g. Illustrator, PowerPoint) ✗

3

Imagine Using a Direct Manipulation System…

4


5


6


7


8


9


10


11

✗


12


13


14


15


16


17

✗


18


19

Imagine Using a Programming System…

circle1 = ...

20

"Draw"


circle1 = ... line1 = ... line2 = ...

21

"Draw"


circle1 = ... line1 = ... line2 = ...

circle1 = ... pt1 = left(circle1) line1 = ... line2 = ...

22

"Relate"


circle1 = ... pt1 = left(circle1) line1 = ... line2 = ...

circle1 = ... pt1 = left(circle1) pt2 = right(circle1) line1 = ... line2 = ...

23

"Relate"


circle1 = ... pt1 = left(circle1) pt2 = right(circle1) line1 = ... pt3 = top(circle1) pt4 = bottom(circle1) line2 = ...

24

"Relate"


circle1 = ... pt1 = left(circle1) pt2 = right(circle1) line1 = ... pt3 = top(circle1) pt4 = bottom(circle1) line2 = ... rectArray = ...

25

"Relate"


circle1 = ... pt1 = left(circle1) pt2 = right(circle1) line1 = ... pt3 = top(circle1) pt4 = bottom(circle1) line2 = ... rectArray = ...

function ferrisWheel ( numSpokes , spokeLen , rotAngle , sizeCar , radiusCenter , cx , cy ) { ... } ferrisWheel(4, ...);

26

"Group"



27

"Tweak"



28

"Tweak"

29 29



"Tweak"



30

"Tweak"



31

"Tweak"



32

"Tweak"


33


34

��Programming

Language

��Direct

Manipulation

2D Graphics

Presentations

Documents

Spreadsheets

Web Apps

35

��Programming

Language

��Direct

Manipulation

2D Graphics

Presentations

Documents

Spreadsheets

Web Apps

General-Purpose��Languages

Domain-Specific��Languages or��

Libraries Domain-Specific��

GUI Tools

36

��Programming

Language

��Direct

Manipulation

2D Graphics

Presentations

Documents

Spreadsheets

Web Apps

Direct Manipulation Programming

???

???

???

???

???

Sketch-n-Sketch

37

Our Approach

Reuse effort across domains.

Enable experts and library writers to extend built-ins.

Smooth spectrum between "Experts" and "End Users".

λ-calculus + Direct��Manipulation

38

Sketch-n-Sketch Demo

39

Sketch-n-Sketch Demo

Programming with:��Less Keyboard. ��More Mouse.

40

Semi-Automated Programming by��

Manipulating��Output

41

Draw

Relate

Group

Tweak PLDI 2016

UIST 2016

42

Relate Group Tweak Draw

43

Tweak Draw Relate Group

e' ⇓t 'v'

44


e' ⇓t 'v' e' ⇓t 'v' v' = t Constraint on Original e:

v' = t Constraint on Updated e':

e' ⇓t 'v'

⇝

45


e' ⇓t 'v' v' = t Constraint on Original e:

v' = t Constraint on Updated e':

e' ⇓t 'v'

⇝

46


e' ⇓t 'v' t ::= x��t :|= t1 + t2 ��t :|= t1 * t2 t :|= sin t t :|= pow t1 t2 t :|= ...

Dataflow-Only Traces

47


let a = 1 let b = 2 in …

⇓a a 1'

⇓a+b a+b 3'

⇓a+b*b a+b*b 5'

⇓a (\x.x) a 1'

⇓a true ? a : b 1'

48


155 = x0 + 2 * sep

Change x0 or sep? Or both? Or more?

Change exactly one constant. Heuristics to automatically choose.

Annotations to freeze (n!) or thaw (n?).

110 = x0 + 2 * sep ⇝

49


Can solve for x if x only occurs once.

5*sin(x) = n

x*(y+y2) = n

x2 = n

✓

✓

✓

50


Good enough for 80% of equations.

Can solve for x if x only occurs once.

5*sin(x) = n

x*(y+y2) = n x*(x+y2) = n

x2 = n x*x = n

✗ ✓

✗ ✓

✓

51

Draw Relate Group Tweak

(x1,y1)

(x3,y3)

(x2,y2)

"Raw"��Shape

52


(x1,y1)

(x1+2,y1+8) (x1+10,y1+10)

"Anchored"��Shape

53


(x1,y1)

(x2,y2)

(x1+0.20*(x2-‐x1) ,y1+0.80*(y2-‐y1))

"Bounded"��Shape

54


(def rawPoly (let [x1 y1] ... (let [x2 y2] ... (let [x3 y3] ... ... ))))

(def anchoredPoly (let [x1 y1] ... (let [dx2 dy2] ... (let [dx3 dy3] ... ... ))))

(def boundedPoly (let [x1 y1] ... (let [x2 y2] ... (let [px3 py3] ... ... ))))

⇓ ⇓ ⇓

Representation affects grouping operations.

55

Group Draw Relate Tweak

"Anchored"��Group

"Bounded"��Group

56

Group Draw Relate



Tweak

57

Group Draw Relate



Tweak

58

Group Draw Relate



Tweak

59

Relate Draw Group Tweak

(def rect1 (let x 52 ... )) (def line2 (let x1 50 ... )) (def line3 ... )

60



SELECT DIG FILL

CLEAN

Features in Canvas New Hole in Code Hole with Relationship New Code

61



UNSELECTED Features

62



UNSELECTED Features SELECTED Features

63


(def rect1_x 52) (def line2_x1 50) (def [rect1_x' line2_x1'] [rect1_x line2_x1]) (def rect1 (let x rect1_x' ... )) (def line2 (let x1 line2_x1' ... )) (def line3 ... )

LIFT Constants and��DIG New Hole FILL Hole w/�� Desired Relationship

64


(def rect1_x 52) (def line2_x1 50) (def [rect1_x' line2_x1'] [rect1_x rect1_x]) (def rect1 (let x rect1_x' ... )) (def line2 (let x1 line2_x1' ... )) (def line3 ... )

CLEAN Dead Vars

65


(def rect1_x 52) (def [rect1_x' line2_x1'] [rect1_x rect1_x]) (def rect1 (let x rect1_x' ... )) (def line2 (let x1 line2_x1' ... )) (def line3 ... )

CLEAN Redundant Vars

66


(def rect1_x 52) (def rect1 (let x rect1_x ... )) (def line2 (let x1 rect1_x ... )) (def line3 ... )

67



Automatic solver��for Make Equal.

68


Which Constants to Abstract?

Heuristic: Unfrozen and Named

(def foo (let [a b] [1 2!] [a b 3]))

foo

69


Which Constants to Abstract?

Heuristic: Unfrozen and Named

(def foo (let [a b] [1 2!] [a b 3]))

foo

(def foo (\a (let [b] [2!] [a b 3])))

(foo 1)

70


Sketch-n-Sketch Programming in 2015

Less Keyboard. ��More Mouse.

71


Sketch-n-Sketch

72


Sketch-n-Sketch

e3

⇓

e1

⇓

e2

⇓

73

Relate Draw Tweak

Sketch-n-Sketch

e3

⇓

e1

⇓

e2

⇓

Group

Less Reliance��

on Syntactic Structure

Sketch-n-Sketch ++

; Top-‐Level Defs (def rect1 ...) (def line2 ...) (def line3 ...) ; Main Expression [ rect1 ... ]

74


Sketch-n-Sketch Sketch-n-Sketch ++

; Top-‐Level Defs def rect1 = ... def line2 = ... def line3 = ... ; Main Expression (blobs [ rect1 ... ])

(def group1 (def rect1 ...) (def line2 ...) (def line3 ...) [ rect1 ... ] ) group1

75


Sketch-n-Sketch

Less Reliance��

on Syntactic Structure

Sketch-n-Sketch ++

(def polygon7_bot (+ (+ (* 0.5!... (def k3105 (/ (-‐ (+ (-‐ polygon6... (def polygon7_top (-‐ (* 0.5! (+... (def [polygon5_right k3038] [(-‐... (def k3061 (/ (-‐ (+ polygon5_ri... (def polygon6_bot (-‐ (+ (-‐ poly... (def k3063 (/ (-‐ (+ polygon6_bo... (def polygon5_top (-‐ polygon6_t... (def k3103 (/ (-‐ (+ (-‐ polygon5... (def [k3041 polygon5_bot] [(-‐ p... (def k3134 (/ (-‐ (+ k3041 helpe... (def k3141 (/ (-‐ (+ k3038 helpe...

76



Smarter��Algebraic��Constraint��

Solver

77



78



Smarter��Algebraic��Constraint��

Solver

79



Additional��Interaction��to Resolve��User Intent

80



Expressive,��General Program Synthesis

Limited, Syntactic Program Updates

81


82


Prototype; Repair; Refactor; Repeat

Idea

DM

2D Graphics

Presentations

Documents

Spreadsheets

Web Apps

???

???

???

???

PL

83


Idea

General��Purpose PL

+

General��Program��Synthesis

Domain Specific UI

+

Domain Specific��Program��Synthesis

+

Sketch-n-Sketch

DM

2D Graphics

Presentations

Documents

Spreadsheets

Web Apps

???

???

???

???

PL

Domain Specific UI

+


84


Idea


+


+

λ Sketch-n-Sketch

85

λ-Calculus + Direct Manipulation of Output

1 : Mouse-Edit Output

86

λ-Calculus + Direct Manipulation of Code


2 : Mouse-Edit Code

Programming with��Less Keyboard.

Programming with��More Mouse.

87

88

Related Work

DM + Code

Dynamic Drawing��Apparatus

Code + DM

Wang et al. [FSE 2012]��McDirmid's Demos��

Sketch-n-Sketch

Constraints

Sketchpad��ThingLab ��

Juno-2

Program Synthesis��(Sketching, PBE)

Program Repair

Text + Structure Editors

Barista��

Drag-n-Drop Refactor Sketch-n-Sketch++

89

http://ravichugh.github.io/sketch-‐n-‐sketch/

VIDEOS

PAPERS

DEMO

CODE

PLDI 2016

UIST 2016

~20,000 LOC Elm

90

λ-Calculus + Direct Manipulation of Code and Output

0 : Text-Edit Code


2 : Mouse-Edit Code

91

EXTRA SLIDES

92

Draw

Relate

Group

Tweak

Sketch-n-Sketch

DM

2D Graphics

Presentations

Documents

Spreadsheets

Web Apps

???

???

???

???

PL

93


Idea


+


Domain Specific UI

+


+

Sketch-n-Sketch

Programs ???

DM

2D Graphics

Presentations

Documents

Spreadsheets

Web Apps

???

???

???

???

PL

94


Idea


+


Domain Specific UI

+


+

Sketch-n-Sketch

DM Text Interactive General Purpose IDE

95

Reuse effort across domains.

Enable experts and library writers��to extend built-ins.

Smooth spectrum between��"Experts" and "End Users".

λ-Calculus + Direct Manipulation

96

Text editing is great, but…

Many mundane, low-level edits.

Especially with generated code.

λ-Calculus + Direct Manipulation of Code

97

Idea: Selectable AST Nodes

(def foo (\a (let [b] [2!] [a b 3])))

(foo 1)

(def foo (let [a b] [1 2!] [a b 3]))

foo

98


(def foo (\a c (let [b] [2!] [a b c])))

(foo 1 3)

(def foo (let [a b] [1 2!] [a b 3]))

foo

99



100



101




102

Idea: Draggable AST Nodes

(def rect1_x 52)

(def line3 ... )

(def line2 (let x1 rect1_x ... ))

(def rect1 (let x rect1_x ... ))

103


(def rect1_x 52)

(def line3 ... )



104


(def rect1_x 52)

(def line3 (let x1 200 ... )



105


(def line3_x1 200)

(def line3 (let x1 line3_x1 ... )



(def rect1_x 52) Copy-Paste,��Scope Manipulation,��Name Generation, Substitution,��Formatting, …

106

(def [x1 y1] [0 0]) (def [x2 y2] [(+ x1 10) (+ y1 10)]) (def [x3 y3] [(+ x1 2) (+ y1 8)])

Idea: Auto Reformatting

107

Idea: Auto Reformatting

(def [x1 y1] [0 0]) (def x2 (+ x1 10) (def y2 (+ y1 10) (def x3 (+ x1 2) (def y3 (+ y1 8)

Whitespace,��Line Breaks,��Single vs. Multi Defs,��Alignment,��Style Conventions, …

108

END