Scott MacKenzie York University Gesture Recognition CSE4443 – Mobile User Interfaces © Scott MacKenzie
Scott MacKenzieYork University
Gesture Recognition
CSE4443 – Mobile User Interfaces
© Scott MacKenzie
Map
• Gestures are everywhere• Android gestures• Unistrokes / Graffiti recognition• Android (revisited)
2© Scott MacKenzie
Map
• Gestures are everywhere• Android gestures• Unistrokes / Graffiti recognition• Android (revisited)
3© Scott MacKenzie
in life
ASL (American Sign Language)
© Scott MacKenzie
Copy Editing
© Scott MacKenzie
Traffic Control
© Scott MacKenzie
Piazza Venezia
© Scott MacKenzie
Semaphore Signaling
© Scott MacKenzie
N U J V
© Scott MacKenzie
Map
• Gestures are everywhere• Android gestures• Unistrokes / Graffiti• Android (revisited)
10© Scott MacKenzie
in computing
Lightpen on CRT1
1 Sutherland, I. E. (1963). Sketchpad: A man‐machine graphical communication system. Proceedings of the AFIPS Spring Joint Computer Conference, 329‐346. New York: ACM.
© Scott MacKenzie
Finger(s) on CRT1
1 Herot, C. F., & Weinzapfel, G. (1978). One‐point touch input of vector information for computer displays. Proceedings of SIGGRAPH 1978, 210‐216. New York: ACM.
© Scott MacKenzie
Pointing1
1 Bolt, R. (1980). Put‐that‐there: Voice and gesture at the graphics interface. Computer Graphics, 14(3), 262‐270.
© Scott MacKenzie
Finger Gestures1
1 Minsky, M. R. (1984). Manipulating simulated objects with real‐world gestures using a force and position sensitive screen. Proceedings of SIGGRAPH 1984, 195‐203. New York: ACM.
© Scott MacKenzie
Virtual Devices and Templates1
1 Buxton, W., Hill, R., & Rowley, P. (1985). Issues and techniques in touch‐sensitive tablet input.Proceedings of SIGGRAPH '85, 215‐224. New York: ACM.
© Scott MacKenzie
Hand Gestures1
1 Zimmerman, T. G., Lanier, J., Blanchard, C., Bryson, S., & Harvill, Y. (1987). A hand gesture interface device. Proceedings of CHI+GI 1987, 189‐192.
© Scott MacKenzie
Pie Menus1
1 Callahan, J., Hopkins, D., Weiser, M., & Shneiderman, B. (1988). An empirical comparison of pie vs. linear menus. Proceedings of CHI '88, 95‐100. New York: ACM.
© Scott MacKenzie
Pen‐based Computing (early 1990s)
GRiDPad (Grid Systems)
MessagePad 2100(Apple)
325 Point (Fujitsu)
Pilot (Palm)
© Scott MacKenzie
© Scott MacKenzie 19
EO Personal Communicator
1993
Marking Menus1
1 Kurtenbach, G., & Buxton, W. (1993). The limits of expert performance using hierarchic marking menus. Proceedings of the INTERCHI '93, 482‐487. New York: ACM.
MeatBreadFruit & Veg Staples
MiscJunk
Groceries
Auto House
Clothing
© Scott MacKenzie
Marking Menus1
1 Kurtenbach, G., & Buxton, W. (1993). The limits of expert performance using hierarchic marking menus. Proceedings of the INTERCHI '93, 482‐487. New York: ACM.
© Scott MacKenzie
GEdit1
1 Kurtenbach, G., & Buxton, W. (1991). GEdit: A testbed for editing by contiguous gesture. SIGCHI Bulletin, 23(2), 22‐26.
© Scott MacKenzie
Today
© Scott MacKenzie
© Scott MacKenzie 24
Gestures for text entry…
Unistrokes1
1 Goldberg, D., & Richardson, C. (1993). Touch‐typing with a stylus. Proceedings INTERCHI '93, 80‐87. New York: ACM.
© Scott MacKenzie
Graffiti
96% accuracy after five minutes1
1 MacKenzie, I. S., & Zhang, S. (1997). The immediate usability of Graffiti. Proceedings of Graphics Interface '97, pp. 129‐137. Toronto: Canadian Information Processing Society.
© Scott MacKenzie
1995
Graffiti Research
1 Enns, N., & MacKenzie, I. S. (1998). Touch‐based remote control devices. Ext Abstracts of CHI '98, 229‐230. New York: ACM.2 MacKenzie, I. S., Chen, J., & Oniszczak, A. (2006). Unipad: Single‐stroke text entry with language‐based acceleration. Proceedings of NordiCHI 2006, 78‐85. New York: ACM.3 Tinwala, H., & MacKenzie, I. S. (2009). Eyes‐free text entry on a touchscreen phone. Proceedings of TIC‐STH 2009, 83‐88. New York: IEEE.
1 2
3
© Scott MacKenzie
RollPad1
1 Oniszczak, A., & MacKenzie, I. S. (2004). A comparison of two input methods for keypads on mobile devices. Proceedings of NordiCHI 2004, 101‐104. New York: ACM..
© Scott MacKenzie
T‐Cube1
1 Venolia, D., & Neiberg, F. (1994). T‐Cube: A fast, self‐disclosing pen‐based alphabet. Proceedings of CHI '94, 265‐270. New York: ACM. © Scott MacKenzie
QuikWriting1
1 Perlin, K. (1998). Quikwriting: Continuous stylus‐based text entry. Proceedings of UIST '98, 215‐216. New York: ACM. © Scott MacKenzie
Cirrin1
1 Mankoff, J., & Abowd, G. A. (1998). Cirrin: A word‐level unistroke keyboard for pen input, Proceedings of UIST ‘98 (pp. 213‐214): New York: ACM.
© Scott MacKenzie
Edge Keyboard1
1 Wobbrock, J. O., Myers, B. A., & Hudson, S. E. (2003). Exploring edge‐based input techniques for handheld text entry. Proceedings of ICDCS 2003, 280‐282. New York: IEEE.
© Scott MacKenzie
MDITIM1
1 Isokoski, P., & Raisamo, R. (2000). Device independent text input: A rationale and an example. Proc of the Working Conference on Advanced Visual Interfaces ‐ AVI 2000, 76‐83. New York: ACM.
Four strokes
• North
• South
• East
• West
c
E
S
W
© Scott MacKenzie
EdgeWrite1
1 Wobbrock, J. O., Myers, B. A., & Kembel, J. A. (2003). EdgeWrite: A stylus‐based text entry method designed for high accuracy and stability of motion. Proc UIST 2003, 61‐70. New York: ACM.
© Scott MacKenzie
Shape Writing1
1 Zhai, S., & Kristensson, P.‐O. (2007). Learning shape writing by game playing. Ext Abstracts of CHI 2007, 1971‐1976. New York: ACM. © Scott MacKenzie
Shape Writing1
1 Zhai, S., & Kristensson, P.‐O. (2007). Learning shape writing by game playing. Ext Abstracts of CHI 2007, 1971‐1976. New York: ACM. © Scott MacKenzie
Map
• Gestures are everywhere• Android gestures• Unistrokes / Graffiti recognition• Android (revisited)
37© Scott MacKenzie
Android Gestures (Review)
Touch Longpress
Swipe Drag Doubletouch
Pinchopen
Pinchclosed
http://developer.android.com/design/patterns/gestures.html
1‐finger Gestures 2‐finger Gestures
38© Scott MacKenzie
GestureDetector.SimpleOnGestureListener“a convenience class”
1‐finger Gestures
39© Scott MacKenzie
ScaleGestureDetector.SimpleOnScaleGestureListener“a convenience class”
ScaleGestureDetector
40© Scott MacKenzie
2‐finger Gestures
Demo_Scale
41© Scott MacKenzie
(see PaintPanel class in Demo_Scale)
Map
• Gestures are everywhere• Android gestures• Unistrokes / Graffiti recognition• Android (revisited)
42© Scott MacKenzie
Background: Handwriting Recognition
• Advantages– considerable existing skill– good for annotating, sketching, etc.– handwriting is natural
• Disadvantages– handwriting recognition is hard!– in early products, goods delivered did not live up
to promises (next slide)
43© Scott MacKenzie
44© Scott MacKenzie
Unistrokes
• What is a Unistroke?– A single stylus or finger gesture– Begins on touch– Ends on lift– Each “symbol” is one gesture
• Benefits– Recognition is simple– Gestures made on top of one another– Heads‐up typing– Eyes‐free typing
45© Scott MacKenzie
Unistrokes vs. Graffiti
• Unistrokes– Invented by David Goldberg, Xerox PARC – First presented at INTERCHI ’93 a
– Patent held by Xerox (issued Jan. 1997)• Graffiti
– An “example” of Unistrokes– Commercial product– 3COM PalmPilot, HP Omnigo, etc.
a Goldberg, D., & Richardson, C. (1993). Touch‐typing with a stylus. Proceedings of the INTERCHI '93 Conference on Human Factors in Computing Systems, 80‐87. New York: ACM. 46
© Scott MacKenzie
Palm Pilot
Graffiti strokes are made here
Recognized results appear here
47© Scott MacKenzie
1997
Goldberg’s Unistrokes
A B C D E F G H I J K L M
N O P Q R S T U V W X Y Z
Goal: Fast for experts
48© Scott MacKenzie
Palm's Graffiti
A B C D E F G H I J K L M
N O P Q R S T U V W X Y Z
Goal: Easy for novices
49© Scott MacKenzie
Demo_Graffiti
© Scott MacKenzie 50
Steps for Gesture Recognition
0. Input points for a gesture (aka stroke)1. Normalize the points to fit in a unit bounding
box2. Look for straight‐line strokes (N, S, E, W, NE,
SE, SW, NW)3. Compute stroke features4. Use stroke features to look for a match in the
dictionary
51© Scott MacKenzie
© Scott MacKenzie 52
Step #0(input points for stroke)
private ArrayList gesture;...public boolean onTouchEvent(MotionEvent me){
...int x = Math.round(me.getX());int y = Math.round(me.getY());gesture.add(new Point(x, y));
...
String raw = u.recognize(gesture);
GraffitiPanel.java
An instance of the Unistroke class
Step #1(normalize points to fit in a unit bounding box)
© Scott MacKenzie 53
012
3(0,0) (1,0)
(0,1) (1,1)
© Scott MacKenzie 54
Unistrokes.java
Step #2a(look for straight‐line strokes)
© Scott MacKenzie 55
Unistrokes.java
© Scott MacKenzie 56
Unistrokes.java
Step #2b(look for straight‐line strokes)
Step #3a(compute stroke features)
• Cumulative motion– cumulative xmotion (e.g., A = 1, Z = 3)– cumulative ymotion (e.g., A = 2, Z = 1)
57© Scott MacKenzie
Step #3a(compute stroke features)
© Scott MacKenzie 58
Unistrokes.java
Steps #3b(compute stroke features)
• Quadrants– 1st quadrant– 2nd quadrant– 2nd last quadrant– last quadrant
59© Scott MacKenzie
Unistrokes.java
Steps #3b(compute stroke features)
© Scott MacKenzie 60
Step #3c(compute stroke features)
© Scott MacKenzie 61
• Start/stop directions– start x direction (+ve, ‐ve)– start y direction (+ve, ‐ve)– stop x direction (+ve, ‐ve)– stop y direction (+ve, ‐ve)
Note:X 0 = left 1 = rightY 0 = down 1 = up
Step #3c(compute stroke features)
© Scott MacKenzie 62
Unistrokes.java
Unistrokes.java
Step #4(use stroke features to look for match in dictionary)
© Scott MacKenzie 63
Dictionary Entries (1 of 2)
• Each entry contains…– symbol (the ‘output’)– 1st quadrant– 2nd quadrant– 2nd last quadrant– last quadrant– min cumulative x– max cumulative x
0, 1, 2, 3, don’t care
Need min and max because the user’s strokes are “sloppy”
64© Scott MacKenzie
Dictionary Entries (2 of 2)
– min cumulative y– max cumulative y– start xmotion – start ymotion– stop xmotion– stop ymotion
• Note: can have multiple entries for each symbol (e.g. cw “0”, ccw “0”)
Need min and max because the user’s strokes are “sloppy”
+ve, ‐ve, don’t care
65© Scott MacKenzie
Dictionary Example (“B”)
Symbol B1st quad 22nd quad 3n‐1 quad 1n quad 2kx min 2.2kx max 4ky min 1.5ky max 2.5start x don’t carestart y +stop x ‐stop y don’t care
0
12
3
66© Scott MacKenzie
Gesture Set (Graffiti)
67© Scott MacKenzie
Can you imagine what the dictionary entry would look like for each of these gestures?
Graffiti Dictionary
© Scott MacKenzie 68
Unistrokes.java
etc.
Map
• Gestures are everywhere• Android gestures• Unistrokes / Graffiti recognition• Android (revisited)
69© Scott MacKenzie
Graffiti for Android
70© Scott MacKenzie
DemoGraffiti
71© Scott MacKenzie
Graffiti(experiment software)
See also…MacKenzie, I. S., & Castellucci, S. J. (2012). Reducing visual demand for gestural text input on touchscreen devices. Extended Abstracts CHI 2012, pp. 2585‐2590. New York: ACM. 72
© Scott MacKenzie
Graffiti vs. Soft Keyboard
• Frame Model of Visual Attention
PointFrame
TargetFrame
SurfaceFrame
EnvironmentFrame
SoftKeyboard Graffiti
73© Scott MacKenzie
Graffiti IME
74© Scott MacKenzie
© Scott MacKenzie 75
Other possibilities…
Air Gestures
© Scott MacKenzie 76
(Graffiti)
Shape Writing
77© Scott MacKenzie
Thank You
78© Scott MacKenzie