McSig Chi 2008

McSig - A Multimodal Collaborative Handwriting

Trainer for Visually-Impaired People

Beryl Plimmer, Rachel BlagojevicUniversity of AucklandAndrew Crossan, Stephen BrewsterUniversity of Glasgow

www.cs.auckland.ac.nz/~beryl www.dcs.gla.ac.uk/~stephen

Outline

Motivation Why is writing important for a visually impaired person Why is a difficult thing to train

How can we train users to draw a letter shape One solution – haptic trajectory playback How good are people at recalling a trajectory they’ve been

dragged through

McSig Iterative Design Evaluation

Conclusions

2

Why do Visually Impaired People Need to Write ?

Signature Difficult without visual

feedback Important for Job Applications,

Legal documents etc. One participant described her

signature as ‘resembling the meanderings of an inebriated fly’

More general problem with spatial data Presenting Creating

3

Accessible Computing

Magnifiers Need some sight

Screen Readers Synthesized speech

reads text from the screen

Not good for non-text information

Dynamic Braille Non-text or spatial data

is very difficult to present non-visually Maps, charts, graphs,

diagrams, web pages

4

Force Feedback Haptic – Relating to the sense of touch

Force feedback Tactile feedback

Force feedback Forces can be exerted through motors Can be arranged to form 3D objects Can be used to directly influence the movements of the

user We use a PHANTOM OMNI from SensAble

Provides 6 dof in, 3 dof out Produces rich feedback suitable for many general

purpose applications Max 3 Newton force so the user can overpower the

device5

Force Feedback as an Accessibility Aid Non-visual communication channel Can directly affect the user’s exploration

Potential for collaborative applications Research question

Can trajectory playback techniques be used to communicate shape and trajectory information to visually impaired people ?

6

Initial Studies Learning abstract trajectories through haptic

guidance drag a user through a set of pre-defined trajectories Initially significantly harder for visually impaired users &

congenitally blind participants found the task harder still But combining haptic guidance with a sonification of the

trajectory (pitch shift & panning a sinusoidal tone) significantly improved learning

7

Audio Pan

Pitch ShiftDrawing

Area

McSig: Multimodal, collaborative handwriting and signature training tool Teacher and student work

synchronously on shared representation sat next to each other Haptic guidance and audio feedback Teacher guides the student to learn

letter shapes using words and actions Student holds PHANTOM Omni

pen, teacher uses a stylus & touchscreen Teacher can move student’s pen

around the character shape Audio feedback as shape is drawn

Left/right movements: pan, up/down movements: pitch

8

McSig – system design Simulate standard school learning scenario Teacher can choose between three modes

Collaborative Playback mode: Student holds PHANTOM Omni pen student dragged through shape as it is drawn by teacher

Free Stencil mode: Teacher draws letter which is used as a virtual stencil which

thestudent can then explore Can reduce constraining forces as student gets more

experienced

Free drawing mode Student moves Omni pen freely within area. Movements echoed on teacher’s screen

9

First evaluation

1 blind adult Feedback key to learning

User tried to feel what they had drawn on the paper

Added Dutch drawing board – shape is raised on the paper

Can be felt with other hand

Stencil didn’t work very well

10

Usability testing 4 visually impaired adults (3 blind, 1 partially sighted)

Familiarization Letter set ‘o,c,a,d,e’ Started with playback mode and then moved on to stencil Finally drew the letter unsupported

Stencil mode hard to use Strengthened forces for shapes to give clearer path Still didn’t work very well

Audio feedback useful to some, teacher descriptions most useful Speech feedback error prone and reduced confidence of users Omni pen difficult to hold, plus pressing button whilst drawing

tricky Users not used to holding pens Gave some pen training before main study

11

Evaluation Could McSig improve handwriting performance?

Task designed with teachers Some children almost no handwriting skills, some have

good skills 5 characters chosen after discussion with teachers

o, c, a, d, e Participants

8 children 11-17 years old, read Braille, no other major disabilities

3 partially sighted, 5 blind 4 stage study

Familiarization with McSig, then for each letter: Pre-test McSig training Post-test

12

Familiarization Participants could feel setup,

PHANTOM, mat, PC Spatial orientation Drew circle, horizontal and vertical

lines Practised with the pen

Visually impaired people don’t commonly use pens

13

14

McSig: Pre-Test Training and Post-Test Pre-test: participants asked to draw each letter as

best they could Some unable to draw one or more of them

Training: teacher showed participant how to draw letter in Playback mode Experimenter wrote shape on screen, child felt it with

PHANTOM and scored line on tactile sheet Synchronous audio/haptic/tactile feedback Number of repeats based on child’s confidence

Post-test: participants draw character in free draw mode If participant could not draw it we trained and tested

again Time-out after 20 mins

Stopped earlier if all letters done

15

Results – partially-sighted children

Participants All could read enlarged print All had deteriorating sight but had learned to write

when sight was better Did not write now as sight too bad

Familiarized very quickly, could all do circle, horizontal and vertical lines, no problem

One participant did all of our letters in the pre-test

Both of the others started ‘d’ in wrong place

Pre Post16

Results – partially-sighted children

All letter correct except One did a normal ‘e’ in mirror image

Participants had eyes close to drawing surface but did not feel drawing surface with non-dominant hand Wanted to use their sight

All trained quickly and did all letters correctly in post-test Completed within 20 mins

Politely interested but not captivated

17

Results – blind children

Participants 5 totally blind One lost her sight at 3 years, others blind from

birth Familiarization took much longer

Pressure on pen – too much/too little Interacted with drawing space very differently

Non-dominant hand for orientation in space All but one could draw circle and lines Before and after examples

18

Before and after

19

20

Participant who had sight until age 3

21

Discussion - Technology

Training Student independent

Input - teacher Stylus pen Verbalization

Output Visualization on teacher’s

display Phantom Force feedback Sound pan and pitch Tactile trace Recognition to voice

output

Input Phantom pen position

Output Visualization on teacher’s

display

Sound pan and pitch Tactile trace Recognition to voice output

22

Discussion

Results suggest that McSig could help children to learn Especially blind children

Why didn’t stencil mode work? No tactile representation of the letter for non-

dominant hand Therefore an inconsistent interface

23

Future Work Cursive handwriting and signatures

Support move from single letters to cursive A signature can be created and then practised to

keep it consistent over time A self-teaching tool Wider context

Could be used in any application where the teacher wants to guide student Geometry, 2D and 3D shapes Charts and graphs

24

Conclusions Hard for visually-impaired people to learn to

handwrite Signatures difficult to learn and keep consistent Required for important aspects of life

McSig: a collaborative tool that allows a teacher to guide a student to handwrite letter shapes Dynamic haptic and audio feedback

Can improve handwriting in 20 minute session All blind students learned at least 2 new letters Enjoyed the experience

Now working on longer-term study to see how learning develops over time

25

McSig - A Multimodal Collaborative Handwriting

Trainer for Visually-Impaired People

Beryl Plimmer, Rachel Blagojevic

University of Auckland

Andrew Crossan, Stephen Brewster

University of Glasgow

www.cs.auckland.ac.nz/~beryl

www.dcs.gla.ac.uk/~stephen

Thanks to our participants, John Williamson and Malcolm Hall at Glasgow University and our sponsors University of Auckland and the EU FP6 MICOLE Project