Paper Digital Applications - Stanford University · 11/14/2006  · Anthropologists at Intel Digital Home Design Students at Stanford. 26 Augmenting Walls and Tables. 27 Poster-Sized

stanford hci group

Paper + Digital Applicationsfor mobility and collaboration

Ron B Yeh14 November 2006

UNIFOR

2

Augmenting Notebooks

3

Augmenting Walls and Tables

4

Designing & Building

Paper Interactions

5

BackgroundResearch

6

Fluid Interaction with Paper

7

Qa

Interviews and Observations

8

Los Tuxtlas Tropical Rainforest

9

We also observed that…

The process of capturing data is easy, but…

…the data are not.

Accessing

Transforming

Sharing

10

Clear Difference in Advantages

Paper Notebooks [Robust, ∞ “Battery”, …]

Computers [Search, Storage, Sharing, …]

11

Digitizing Pen Technology

12

Digitizing Pen Technology

13

Augmenting Notebooks

14

15

Notes + Photos associated by time

notes @ 4:43pm photo @ 4:44pm

Automatic Association

16

Notes + Photos associated by Inked Gesture (Hotspot Gestures)

Manual Association

Video #0Hotspot Linking

18

ButterflyNet Browser

19

20

21

22

Video #1Browser & Spreadsheet

24

Browse using the Physical Notebook

Navigate by Pen

25

Longitudinal Evaluation

Used By:Field Biologists at JRBP/StanfordNASA Researchers(Death Valley, Alaska Toolik Field Station)

Anthropologists at Intel Digital HomeDesign Students at Stanford

26

Augmenting Walls and Tables

27

Poster-Sized Displays

The Office of the Future

28

29

Download Photographs

Photo Wall

Video #2Photo Wall

31

Video #3Network Monitoring

33Search for Photographs

Map-Based Queries

Video #4Twistr Game

35

Designing & Building

Paper Interactions

36

Reduce, Recycle, Reuse (R3)A Paper Applications Toolkithttp://hci.stanford.edu/paper/

37

Goals for the Toolkit

Lower the threshold for building this class of pen-based paper + digital user interfaces.

Provide a high enough ceiling that talented designers/developers can do interesting research and build useful systems.

38

Pipeline for building a Paper + Digital Application

Design and Print the Paper User Interface

Architect the Application Logic and Connect the Sheets to Pens and Other Devices

Write custom GUIs and Event Handlers

39

Designing and Printing

Create the Paper Interface in Illustrator (or some other tool) and Export to PDF

Add Active Regions through Java Code or the Interactive Designer

Export to Pattern-Augmented PDF

Print to Laser Printer or Wide-format Inkjet

40

41

42

43

Event HandlersAdd behavior to your Paper UI by attaching code

that will be run whenever a person clicks on, drags on, writes on one of your active regions.

toBeClickedRegion.addEventHandler(new ClickHandler() {

public void clicked(PenEvent e) {

< do something >

}

}

}

44

Building Blocks

Sheets, Regions, HandlersDevices, PensApplicationsToolkit

45

Powerpoint Advancer Example

46

Turn Pen Events into Keystrokespublic static void main(String[] args) {Application app = new Application("Advancer");Sheet s = new Sheet(new Inches(8.5), new Inches(11));

Region r = new Region("Button", 1, 1, 4, 2);r.addEventHandler(getMarkHandler());s.addRegion(r);app.addSheet(s, new File("data/Grid/App.patternInfo.xml"));

app.addPen(new Pen("Primary Pen"));PaperToolkit toolkit = new PaperToolkit();toolkit.startApplication(app);

}

47

Create an Applicationpublic static void main(String[] args) {Application app = new Application("Advancer");Sheet s = new Sheet(new Inches(8.5), new Inches(11));

Region r = new Region("Button", 1, 1, 4, 2);r.addEventHandler(getMarkHandler());s.addRegion(r);app.addSheet(s, new File("data/Grid/App.patternInfo.xml"));

app.addPen(new Pen("Primary Pen"));PaperToolkit toolkit = new PaperToolkit();toolkit.startApplication(app);

}

48

Add one Sheetpublic static void main(String[] args) {Application app = new Application("Advancer");Sheet s = new Sheet(new Inches(8.5), new Inches(11));

Region r = new Region("Button", 1, 1, 4, 2);r.addEventHandler(getMarkHandler());s.addRegion(r);app.addSheet(s, new File("data/Grid/App.patternInfo.xml"));

app.addPen(new Pen("Primary Pen"));PaperToolkit toolkit = new PaperToolkit();toolkit.startApplication(app);

}

49

One Patterned Regionpublic static void main(String[] args) {Application app = new Application("Advancer");Sheet s = new Sheet(new Inches(8.5), new Inches(11));

Region r = new Region("Button", 1, 1, 4, 2);r.addEventHandler(getMarkHandler());s.addRegion(r);app.addSheet(s, new File("data/Grid/App.patternInfo.xml"));

app.addPen(new Pen("Primary Pen"));PaperToolkit toolkit = new PaperToolkit();toolkit.startApplication(app);

}

50

Use the Marking Gesture Handlerpublic static void main(String[] args) {Application app = new Application("Advancer");Sheet s = new Sheet(new Inches(8.5), new Inches(11));

Region r = new Region("Button", 1, 1, 4, 2);r.addEventHandler(getMarkHandler());s.addRegion(r);app.addSheet(s, new File("data/Grid/App.patternInfo.xml"));

app.addPen(new Pen("Primary Pen"));PaperToolkit toolkit = new PaperToolkit();toolkit.startApplication(app);

}

51

Add the Region & Sheetpublic static void main(String[] args) {Application app = new Application("Advancer");Sheet s = new Sheet(new Inches(8.5), new Inches(11));

Region r = new Region("Button", 1, 1, 4, 2);r.addEventHandler(getMarkHandler());s.addRegion(r);app.addSheet(s, new File("data/Grid/App.patternInfo.xml"));

app.addPen(new Pen("Primary Pen"));PaperToolkit toolkit = new PaperToolkit();toolkit.startApplication(app);

}

52

We need at least 1 Penpublic static void main(String[] args) {Application app = new Application("Advancer");Sheet s = new Sheet(new Inches(8.5), new Inches(11));

Region r = new Region("Button", 1, 1, 4, 2);r.addEventHandler(getMarkHandler());s.addRegion(r);app.addSheet(s, new File("data/Grid/App.patternInfo.xml"));

app.addPen(new Pen("Primary Pen"));PaperToolkit toolkit = new PaperToolkit();toolkit.startApplication(app);

}

53

Ask the Toolkit to Start the Apppublic static void main(String[] args) {Application app = new Application("Advancer");Sheet s = new Sheet(new Inches(8.5), new Inches(11));

Region r = new Region("Button", 1, 1, 4, 2);r.addEventHandler(getMarkHandler());s.addRegion(r);app.addSheet(s, new File("data/Grid/App.patternInfo.xml"));

app.addPen(new Pen("Primary Pen"));PaperToolkit toolkit = new PaperToolkit();toolkit.startApplication(app);

}

54

Marking Gesture Handler?private static EventHandler getMarkHandler() {return new GestureHandler() {public void handleMark(PenEvent e, GestureDirection dir) {switch (dir) {default:RobotAction robot = new RobotAction();robot.keyType(KeyEvent.VK_RIGHT);robot.invoke();break;

}}

};}

55

The Event includes the Mark’s Direction

private static EventHandler getMarkHandler() {return new GestureHandler() {public void handleMark(PenEvent e, GestureDirection dir) {switch (dir) {default:RobotAction robot = new RobotAction();robot.keyType(KeyEvent.VK_RIGHT);robot.invoke();break;

}}

};}

56

To Key Strokeprivate static EventHandler getMarkHandler() {return new GestureHandler() {public void handleMark(PenEvent e, GestureDirection dir) {switch (dir) {default:RobotAction robot = new RobotAction();robot.keyType(KeyEvent.VK_RIGHT);robot.invoke();break;

}}

};}

57

Other Directionsprivate static EventHandler getMarkHandler() {return new GestureHandler() {public void handleMark(PenEvent e, GestureDirection dir) {switch (dir) {case NE: // do somethingbreak;

case E: // do something differentbreak;

default:RobotAction robot = new RobotAction();robot.keyType(KeyEvent.VK_RIGHT);robot.invoke();break;

}} };}

58

Swing Example

59

R3 Example

Video #5Other Paper Applications

61

AudioGuide ExampleWe started out with an idea of making a

mobile audio interface for biologists, and decided that a tourist map would be a good first step.

62

The Map

63

Map Adobe Illustrator

64

Add Active Regions to the PDF

In this case, we used the R3 Acrobat Designer.You may alternatively choose the pure

programmatic route, by writing Java code to define the active regions.

public class AudioGuide extends Application {< lots of stuff >

}

65

Add Active Regions to the PDFpublic class AudioGuide extends Application {public static void main(String[] args) {AudioGuide guide = new AudioGuide();PaperToolkit p = new PaperToolkit();p.useApplicationManager(true);p.loadApplication(guide);

}< lots of stuff >

}

66

Add Active Regions to the PDFpublic class AudioGuide extends Application {public static void main(String[] args) {AudioGuide guide = new AudioGuide();PaperToolkit p = new PaperToolkit();p.useApplicationManager(true);p.loadApplication(guide);

}< lots of stuff >

}

67

68

Conclusions

Audience for the R3 Paper Toolkit isHCI Practitioners like yourself!

Open Source SoftwareCan evolve based on your feedback

http://hci.stanford.edu/paper

stanford hci group

Questions???

ronyeh@cs.stanford.edu