Lecturing with Digital Ink Richard Anderson University of Washington.

Lecturing with Digital Ink

Richard AndersonUniversity of Washington

Lessons learned from the Classroom Presenter project

Classroom Pedagogy Teaching with ink

HCI Ink based presentation

Multimedia Analysis of lecture artifacts

Classroom Presenter

Integration of slides and digital ink using Tablet PC

Key ideas: Ink overlay on images Distributed application

Many other systems also support ink and slides

Classroom Presenter as a distributed application Designed as

distributed application for distance learning

Enables many scenarios

Mobility Walking and talking

Sharing materials with students

Note taking Classroom interaction

Student submissions

Deployments

Estimated use in at least 100 courses Wide use inside of computer science Push for adoption outside of CS Lecture archives from UW

Professional Master’s Program Several hundred hours of recorded audio,

video, and ink.

Distance Learning Classes

“Typical ink usage”

“Typical ink usage”

Planning for ink usage

Ink use in presentation

Cognitive load Only limited attention available for

computer while lecturing Linkage with speech

Close tie between ink and speech

Cognitive load Limited feature use

Even color change unusual User interface must be simple (and

robust) Cannot give feedback to user Many actions appear to minimize

mental effort Color change only for contrast Reliance on screen erase

Understanding Attentional Marks Properties

Brief, simple markings Occur with speech Augment meaning of speech Ad hoc form

Is there a linguistic context in which to understand these marks?

Spontaneous Hand Gestures

Spontaneous Hand gestures [McNeill]: are synchronous w/speech are co-expressive w/speech lack standard of form

Attentional marks share these properties.

Gesture Types: Iconic

Gesture Types: Deictic & Cohesive

Analysis of digital ink Understand ink usage Motivation: inform development of

ink based applications Archiving

Search, Summarization, Transcription Lecture based

Improved rendering, note taking, accessibility

Ink classification Textual Diagrammatic Attentional

% of strokes % of episodes

B C B+C B C B+C

Attentional 49 53 51 77 74 76

Diagram 9 7 8 8 8 8

Writing 41 38 40 14 16 15

Other 1 2 1 2 2 2

Coding of six hours of lecture

Goals

Understand usage “in the wild” Cannot expect lecturers to modify

behavior Determine opportunities for

automatic analysis Identify challenges

Methodology

Study of recorded classes Best data set: Professional

Master’s Program Distance courses Audio, Video, Ink archives HCI, Compilers, Programming

Languages, AI, Transaction Processing

Attentional ink Problem – content

matching Identify slide content

referred to by ink Study

Implement basic algorithms to match attention marks to slide content

Compare results with human coders

Attentional ink

Determine the lecturer’s intent:

Determine level to parse the content

Attentional ink Challenges

Recognition of attentional ink on text

Difficult example:

Handwriting

How well does handwriting recognition work on “typical” instructor writing? Domain has many challenges

Recognition Study Studied isolated

words/phrases written on slides

Removed non-textual ink

Fed through the Microsoft Handwriting Recognizer

No training

Recognition Examples The Good:

The Bad:

The Ugly:

Handwriting Reco Results

Exact Alternate

Close None

Prof. A

16 (88%) 1 (6%) 0 (0%) 1 (6%)

Prof. B

146 (59%)

26 (10%) 6 (2%) 71 (29%)

Prof. C

18 (42%) 5 (11%) 1 (3%) 19 (44%)

Prof. D

262 (61%)

45 (11%) 9 (2%)111

(26%)

Prof. E

408 (79%)

46 (9%) 2 <(1%) 58 (11%)

Total 850 (68%)

123 (10%)

18 (1%)260

(21%)

Joint Writing and Speech Recognition Can we use handwriting recognition

with speech recognition together to improve accuracy?

Co-expression of ink and speech Are written words spoken as well?

Can speech disambiguate handwriting?

Can handwriting disambiguate speech?

Examples Difficult for Speech and Ink Recognition

Difficult Written Abbreviations

Speech/Ink Used to Disambiguate Ink/Speech

Experiment Examined instances of isolated word

writing Selected word writing episodes at random

but uniformly from the various instructors Generated transcripts manually from the

audio Checked whether the instructor spoke the

exact word written Measured the time between the written

and spoken word

Speech/Text Co-occurrence Results

Exact Approx None Simul 0-2s > 2s

A 1 (100%) 0 (0%) 0 (0%) 1 (100%) 0 (0%) 0 (0%)

B 9 (75%) 3 (25%) 0 (0%) 12 (100%) 0 (0%) 0 (0%)

C 9 (82%) 2 (18%) 0 (0%) 10 (91%) 1 (9%) 0 (0%)

D 12 (86%) 2 (14%) 0 (0%) 10 (71%) 4 (29%) 0 (0%)

E 9 (56%) 7 (44%) 0 (0%) 7 (44%) 4 (25%) 5 (31%)

Total 40 (74%) 14 (26%) 0 (0%) 40 (74%) 9 (17%) 5 (9%)

Activity Recognition Identifying slide corrections

Example Results

Diagrammatic ink

How do instructors use diagrams Basic legibility Observed behaviors

Diagram phasing Locality of expression

Typical diagram Basic, irregular

shapes Difficult labels Attentional ink

More examples

Zipf diagram

Stroke order

Diagram phasing

More phasing

Top arrows: “Not there”

Separate wins indicated together

Locality in diagrams

Summary

Pedagogy with ink How is ink used in conjunction with

content and speech to express information

Presentation with ink Low attention task

Analysis of ink usage Extracting meaning from archived

lectures

Resources

cs.washington.edu/education/dl/presenter/ Software Downloads Papers

Contact info Richard Anderson,

[email protected] Ruth Anderson, [email protected] Craig Prince, [email protected]