Page 1
A LOOK AT AVATAR TECHNOLOGY AND SIGNING MEDIUMS
Guest Lecture for Language technology for accessibility, Institute of Computational Linguistics, University of Zurich
Robert SmithSchool of Informatics and Engineering
Institute of Technology, BlanchardstownIrish Sign Language Special Interest Group
Computational and Functional Linguistics Group
Page 2
The “why” question
• Why do we need sign language?• 1st language of a nations deaf community
• Why do we need signing video or avatars?• Cannot “write” sign language like we do with spoken languages• Records• Information (TV, web, classroom, kiosk, etc…)
• Can’t Deaf people read subtitles, pamphlets etc.?• Yes but the average reading age of Deaf school leavers is comparable to
that of an 8-9 year old hearing child (Conrad, 1979)• Sign language for most Deaf people, is their 1st language
• Lip reading?• Really? Can you do it?
10/21/14 Robert Smith 2014 2
Page 3
Signing mediums
•When the signer is present•Live signing
•When the signer is not present•Signing video•Traditional animation•Motion capture signing•Synthesised signing
10/21/14 Robert Smith 2014 3
Page 4
The problems with video
10/21/14 Robert Smith 2014 4
Page 5
Signing avatars
•Signing video
•Traditional animation•Motion capture signing•Synthesised signing
10/21/14 Robert Smith 2014 5
These methods require knowledge of animation
The reminder of this lecture is dedicated to understanding avatar animation
Page 6
Origin of AvatarsAvatar (Original, Hindu term), manifestation or
appearance of a Hindu deity
Avatar (computing term), the graphical representation of a user
Avatar (Sign Language term), a 3D animated character used to deliver Sign Language utterances
Robert Smith 201410/21/14 6
Page 7
ANIMATION BASICS
10/21/14 Robert Smith 2014 7
Page 8
Frame on movie reel
Robert Smith 201410/21/14 8
Page 9
10/21/14 Robert Smith 2014 9
Zoopraxiscope
Page 10
Animated ball
Animation Frame
Robert Smith 201410/21/14 10
Page 11
2D animation
Robert Smith 201410/21/14 11
Page 12
2D Evolution, Pencil-to-Pencil
Robert Smith 201410/21/14 12
Page 13
3D ANIMATION
10/21/14 Robert Smith 2014 13
Page 14
2D - 3D animation
1994, and the first completely computer-generated animated movie was Toy Story
Robert Smith 201410/21/14 14
Page 15
Concept drawing
Body shape, sizes, colours, textures etc…
Robert Smith 201410/21/14 15
Page 16
Analogy (Papier-mâché)
1. Structure 2. Mesh3. Texture4. Colour
10/21/14 Robert Smith 2014 16
Page 17
Analogy (Papier-mâché)
1. Structure 2. Mesh3. Texture4. Colour
10/21/14 Robert Smith 2014 17
We will look at this last
Page 18
Analogy (Papier-mâché)
1. Structure 2. Mesh3. Texture4. Colour
10/21/14 Robert Smith 2014 18
This 1st
Page 19
Body Shape - 3D polygon mesh
Robert Smith 201410/21/14 19
Page 20
Mesh structure
Robert Smith 201410/21/14 20
Page 21
Number of polygons is important
Robert Smith 201410/21/14 21
Page 22
Adding skin detail
Normal maps (skin pores etc)
Specular and diffuse lighting mapsBump maps / texture maps
Robert Smith 201410/21/14 22
Page 23
Bump/Specular/Normal maps on a 3D mesh
Robert Smith 201410/21/14 23
Page 24
Example
10/21/14 Robert Smith 2014 24
Page 25
Analogy (Papier-mâché)
1. Structure 2. Mesh3. Texture4. Colour
10/21/14 Robert Smith 2014 25
Page 26
Structure/Movement (armatures)
Robert Smith 201410/21/14 26
Page 27
Body rigging
Robert Smith 201410/21/14 27
Page 28
Basic facial rigging
Robert Smith 201410/21/14 28
Page 29
Robert Smith 201410/21/14 29
Page 30
Robert Smith 201410/21/14 30
Page 31
3D Rendering
Robert Smith 201410/21/14 31
Page 32
Hardware vs time
10/21/14 Robert Smith 2014 32
Page 33
Real-time rendering
3D Rendering - examples
Robert Smith 201410/21/14 33
Page 34
3D animation methods
Up to now I’ve described how 3D characters are created…now lets talk about how we can use them…
1. ‘Traditional’ Animation using animators
2. Motion capture (mocap)
3. Full synthesis
Robert Smith 201410/21/14 34
Page 35
3D animation methods
Up to now I’ve described how 3D characters are created…now lets talk about how we can use them…
1. ‘Traditional’ Animation using animators
2. Motion capture (mocap)
3. Full synthesis
Robert Smith 201410/21/14 35
Page 36
Using Animators
• 3 year production
• Teams of artists and animators
• 3500 Sun Microsystems processors, compared with 1400 for Toy Story 2 and only 200 for Toy Story
Robert Smith 201410/21/14 36
Page 37
Signed Language Example
Robert Smith 201410/21/14 37
Page 38
3D animation methods
Up to now I’ve described how 3D characters are created…now lets talk about how we can use them…
1. ‘Traditional’ Animation using animators
2. Motion capture (mocap)
3. Full synthesis
Robert Smith 201410/21/14 38
Page 39
Motion captureThe most commonly used technique for sign language animation on the web is motion capture
Robert Smith 201410/21/14 39
Page 40
Motion capture setup
Robert Smith 201410/21/14 40
Page 41
Facial emotion on these projects was good
Motion/Performance capture - examples
Robert Smith 201410/21/14 41
Page 42
“Avatar” movie• 190 inferred cameras + Dozens of HD cameras
Robert Smith 201410/21/14 42
Page 43
Facial mapping
Robert Smith 201410/21/14 43
Page 44
360˚ HD photo sphere
Robert Smith 201410/21/14 44
Page 45
Capturing emotion
Robert Smith 201410/21/14 45
Page 46
MoCap drawbacks
• Expensive equipment.
• Requires labour-intensive calibration of the equipment.
• …and post-editing of the captured data.
• Good for the movie industry• Realism, faster production, cheaper
• Bad for Sign Language• Finite data set, Signed Exact English
Robert Smith 201410/21/14 46
Page 47
3D animation methods
Up to now I’ve described how 3D characters are created…now lets talk about how we can use them…
1. ‘Traditional’ Animation using animators
2. Motion capture (mocap)
3. Full synthesis
Robert Smith 201410/21/14 47
Page 48
SL Synthesis
• Sign synthesis is not new (The concept is over 10 years old).
• Less costly to implement.
• Has the flexibility of a writing system.
• Real-time rendering.
Robert Smith 201410/21/14 48
Page 49
SL Synthesis2005-2006, ASL project,
University of Pennsylvania
2006-2007, The South African SL Project,
University of Stellenbosch
2004-2007, The Spanish SL project,
University of Madrid
Robert Smith 201410/21/14 49
Page 50
Issues with synthesis
• Focus is often narrowed i.e.• Machine Translation• Specific task• Specific linguistic attribute
• Low processing requirements for rendering• Mesh polygons• Armature bones• Texture/Specular/Normal mapping
• Little work to date on NMF
Robert Smith 201410/21/14 50
Page 51
State-of-the-art system ( )
HamNoSys-to-SiGML
AnimGenAnimation data
SiG
ML
HamNoSys
<sigml> <hns_sign gloss="$PROD:Morning"> …al> </hns_sign></sigml>
Robert Smith 201410/21/14 51
Page 52
SummaryMoCap
Synthesis
Concept Mesh Skin Rigging Render
HamNoSysAdd animation
parametersMachine readable
Render
Concept Mesh Skin Rigging Render
Capture Motion Data
SearchData
Machine readable
Assemble utterance
Add animation parameters Render
Robert Smith 201410/21/14 52
Page 53
Summary
•When the signer is present•Live signing
•When the signer is not present•Signing video•Traditional animation•Motion capture signing•Synthesised signing
10/21/14 Robert Smith 2014 53
Page 54
Questions
?
Robert Smith 201410/21/14 54