IsVR: Interactive Systems and Virtual Reality Research Group 2005 Virtual HAND: A Virtual Reality Interface Dr Manolya Kavakli Senior Lecturer Department.

isVR: Interactive Systems and Virtual Reality Research Group 2005

Virtual HAND: A Virtual Reality Interface

Dr Manolya KavakliSenior Lecturer

Department of Computing

Macquarie University

Sydney, Australia

25 October 2005

E6B Room 136


Goal• discuss the current interfaces for

– sketch and hand motion recognition

• report the outcomes of our research studies with• 3 hypotheses• demonstrate a method to construct an interactive 3D model

– from a combination of sketches drawn in 3D space – using the user's hands as dynamic input devices.

• address the issues for further developments.

Problem

• Gap• In CAD tools to support the transition • between the conceptual design and detailed design process

• Reason• insufficient structural information available to develop CAD tools• CAD tools have paid scant attention to

– how designers draw – how cognitive tasks and the progression of conceptual design affect

drawing behavior

• Solution:• to define the requirements for CAD systems capable of supporting

sketching in idea generation• to build systems for sketch analysis, sketch recognition, cognitive

structuring, 2D & 3D transformation, etc.

Standard input devices do not closely mimic natural motions of hands such as drawing and sketching

• interactive computer graphics systems – (Sutherland, 1965, Negroponte, 1973)

• object recognition and scene segmentation – (Guzman, 1971, Waltz, 1975, Marr, 1977,1982, Kosslyn, 1994)

• sketch/diagram recognition as an image – (eg., FABEL project, Schaaf, 1994)

• exploiting properties of drawing production – (Scrivener et al., 1993, Gross, 1994)

• the nature of human visual processing/ – the part-subpart structure of objects – (Bartram, 1974, Kanade, 1981, Biederman, 1987)

• the cognitive processes and the acts of attending to visuo-spatial features– (Suwa, Gero and Purcell, 1998)

Previous attempts


Project background

• NATO FellowshipNATO Fellowship (1996)

Design Research Center, Derby University, UK– An AI Application for the Transformation of a 2D Sketch to a 3D

Geometric Model– Project Report:Project Report: The NATO Science Fellowship Program for Post

Doctoral Studies, NATO area code: 4301, NATO list code: 51/B96/TU

• Postdoctoral Fellowship (1998-2000)

Key Centre of Design Computing and Cognition, Sydney University


Hypothesis I

• If the hands of an artist could speak, • they would tell us more about the nature of the creative task they are engaged in, • thus we would be able to simulate the creative processes.


Assumption: Ideas are not images – but the result of a synthesissynthesis occurring between

perceptual dataperceptual data and abstract ways of conceptualising or understandingunderstanding such data

Stage I:

• is sketching behavior structured or not?– structurestructure is the interrelation of partsinterrelation of parts as dominated by the

general charactergeneral character of the whole.– it is essential to parse objectsparse objects to sort out what an object was

before recognition.

• Perception as a constructive process– When a stimulus is encountered, – the system generates perceptual information relevant to it (stimulus

encoding), – and this information makes contact with perceptual structures stored in

memory– that serve to identify the nature of the stimulus

(recognition/identification).

Experimental room

• Sessions– Free Sketching I– Free Sketching II– Part by part Sketching– Design– Overtracing– Interpretation

• Subjects– 2nd year Applied Arts Students

ObjectsObjects

Chairs

Kavakli et al, 1998, Structure in Idea Sketching Behaviour,Design Studies, 19, pp.485-517

FOR MORE INFO...


Analysis


Part by part drawing

• Object held in memory/imagery are externalised in sketches part by part in terms of volumetric primitives (529/607)

• Part by part drawing occurs in absence of direct instruction to draw part by part• Participants performed least well when instructed

– conscious effort to draw part by part interfered in someway with the externalisation process

– the object model is more difficult to remember and recall than the others


Non part by part drawing

• Visual layering– Viewer centered– Splitting by self occluding contour

• Functional layering

–Clarifying remembrance of part relationships

–Resolving parts

–Resolving relationships between parts

• Overtracing


Hypothesis II

• If a complex system could decode the sign language and reasoning of the artist's hands motions and thoughts,

• it would be possible to support creativity by stimulating the right set of cognitive actions with the interface.


Stage II: Protocol Analysis

– Protocol analysis methods (Dorst and Dijkhuis, 1995) :• the process-oriented approach • the content-oriented approach

– Retrospective protocol analysis method • based on the content-oriented approach

– A coding scheme (Suwa et al., 1998a) to systematically code cognitive actions of designers from video/audio protocols. • Modified version of the coding scheme developed by Suwa and Tversky (1997)

– Analysis of the cognitive processes of novice and expert designers• novice: 2nd year student of architecture • expert: a practising architect with more than 25 years experience


Segmentation

– A cognitive segment consists of cognitive actions that appear to occur simultaneously.

– SEGMENT 1: Got access somehow as a spine…– SEGMENT 2: Maybe but no..We've got this crossed. The big museum

space is going to have some dedicated shot to it. – I'm not going to cross that, cannot break that. That's interesting about this..

thinking in hindsight. – So I've got the masses..Coming is the service here, with the little line

through..Maybe I can catch the Front. No! I didn't ever realize when I was drawing it. that I had drawn it quite so heavy. No because I don't want this mass to be broken. I don't want to carry service to the middle of the museum.

– SEGMENT 3: so I am going to have to segment this a little bit. Something has to be here and something back here. And I am not going to bisect the main space.


Coding


Cognitive Actions I• D-actions: drawing actions

– Dc: create a new depiction– Drf: revise an old depiction– Dts: trace over the sketch– Dtd: trace over the sketch on a different

sheet– Dsy: depict a symbol– Dwo: write words

• M-actions: moves – Moa: motion over an area – Mod: motion over a depiction – Mrf: move attending to relations / features – Ma: move a sketch against the sheet

beneath – Mut: motion to use tools – Mge: hand gestures

G1: goals to introduce new functions

•G1.1: based on the initial requirements

•G1.2: directed by the use of explicit knowledge or past cases

•G1.3: extended from a previous goal

•G1.4: not supported by knowledge, given requirements or a previous goal

G2: goals to resolve problematic conflicts

G3: goals to apply introduced functions or arrangements in the current context

G4: repeated goals


Cognitive Actions II• Perceptual actions

• Psg: discover a space as a ground• Posg: discover an old space as a ground• Pfn: attend to the feature of a new depiction• Pof: attend to an old feature of a depiction• Pfp: discover a new feature of a new depiction• Prn: create or attend to a new relation• Prp: discover a spatial or organizational relation• Por: mention or revisit a relation

• Functional actions• Fn: associate a new depiction, feature or relation with a new function• Frei: reinterpretation of a function• Fnp: conceiving of a new meaning independent of depictions• Fo: continuing or revisited thought of a function• Fop: revisited thought independent of depictions• Fi: implementation of a previous concept in a new setting


General Differences• The expert's cognitive activity and productivity

– three times as much as the novice's (Kavakli et al., 1999). • Pages: 13 (expert) / 4 (novice)• Alternatives: 7 (expert) / 2 (novice)

– expert's productivity is nearly 3.5 times as much as the novice’s

• Actions: 2916 (expert) / 1027 (novice)– expert's design protocol is 2.8 times as rich as the novice's

• Segments: 348 (expert) / 122 (novice)– 2.8 times as many segments in the expert designer's session as in the novice's – In both protocols, each segment includes 8 cognitive actions on average.

• The novice's performance in certain types of tasks – (such as discovery of implicit spaces) – proportionally higher than the expert's.


Coexistence of cognitive actions

• Although there is no clear evidence for causality among cognitive actions, there is evidence for the coexistence of the cognitive actions (Finke et al., 1992, Kosslyn, 1994, Suwa et al, 1999, Kavakli and Gero, 2000).

• If the cognitive activities slow down at some stage, this may be because of not only one activity, but also other activities having different roles that occur together (Kosslyn, 1995).

expert's cognitive actions/pages

050

100150200250300350400450

P1

P2

P3

P4

P5

P6

P7

P8

P9

P10

P11

P12

P13

pages

actio

ns

Physical

drawing

looking

moves

Perceptual

Functional

Goals

TOTAL

novice's cognitive actions/pages

0

50

100

150

200

250

300

350

P1 P2 P3 P4

pages

ac

tio

ns

Physical

drawing

looking

moves

Perceptual

Functional

Goals

TOTAL


Table A Correlation coefficients of cognitive actions across design depictions (Dc)Expert Novice

Drf 0.03 0.34Dts 0.58 0.98Dtd 0.25 -0.75Dsy 0.35 0.74Dwo 0.32 0.75L 0.81 0.99Psg -0.17 0.71Posg 0.27 0.64Pfn 0.45 0.66Pfp 0.15 0.90Pof 0.53 -0.27Prp 0.74 0.98Prn 0.70 0.28Por 0.57 0.92Fn 0.75 0.86Frei 0.20 0.21Fo 0.83 0.51Fnp 0.31 0.60Fop 0.68 0.21Fi 0.24 0.26G1-1 0.45 -0.29G1-2 0.67 0.73G1-3 0.44 0.21G1-4 0.14 0.85G2 0.34 0.38G3 0.21 0.71G4 0.19 0.58Ma 0.31 -0.29Mod 0.07 0.60Moa 0.69 0.89

•Concurrent actions:

•Primary concurrent actions:

–the cognitive actions that directly correlate with depicting drawings.

•Secondary concurrent actions:

–the cognitive actions that highly correlate with the primary actions.

Table 5. Correlation coefficients of cognitive actions in pagesexpert-page Drawing Looking Perceptual Functional Goals MovesDrawing 1.000Looking 0.864 1.000Perceptual 0.998 0.909 1.000Functional 0.998 0.951 0.998 1.000Goals 0.995 0.829 0.996 0.996 1.000Moves 0.975 0.635 0.968 0.978 0.975 1.000

novice-page Drawing Looking Perceptual Functional Goals MovesDrawing 1.000Looking 0.968 1.000Perceptual 0.786 0.898 1.000Functional 0.744 0.828 0.670 1.000Goals 0.655 0.806 0.981 0.617 1.000Moves 0.951 0.862 0.680 0.504 0.529 1.000


(constant-4)

• Strong correlations in both design protocols:– between depicting drawings (Dc) and– looking actions (L),– discovery of a relation (Prp), – association of a new depiction with a function (Fn).

– (4+2) in the expert's design protocol:– creation of a new relation (Prn) – revisited thought of a function (Fo) – there are weak correlations in these categories in the

novice's design protocol.


• (4+11) in the novice's protocol:– overtracing (Dts), – writing (Dwo), – depicting symbols (Dsy), – discovery of a space as a ground (Psg), – discovery of a new feature of a new depiction (Pfp), – mention of a relation (Por), – motion over an area (Moa), – goals directed by the use of explicit knowledge or past cases (G1-2), – goals not supported by knowledge, requirements or previous goals (G1-4), – and goals to apply previously introduced functions in the current context (G3). – Tracing over the sketch on a different sheet is also strongly negative correlated with

depicting drawings (Dc) for the novice.


Table 4. Primary Concurrent Actions Correlated with Depicting Drawings (Dc)

RootAction Code

Primar y ActionCode:Novice

Primar y ActionCode: Expert

Dc L LDts ~DtsPor ~PorPrp Prp

Prn~Fo FoFn FnMoa ~MoaG1-2 ~G1-2DwoDsy-DtdPsgPfpG1-4G3

( ) positive strong correla tion(-) negative strong correlation(~) substantial correlation

Expert's concurrent actions

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9 10 11 12 13

pages

acti

on

s

Dc

Dts

L

Prp

Prn

Por

Fn

Fo

G1-2

Moa

Novice's concurrent actions

0

10

20

30

40

50

60

70

1 2 3 4

pages

ac

tio

ns

Dc

Dts

Dtd

Dsy

Dwo

L

Psg

Pfp

Prp

Por

Fn

Fo

G1-2

G1-4

G3

Moa


Table 7. Secondary Concurrent Actions Correlated with Depicting Drawings (Dc)ActionCode

Novice Expert Novice'sSecondary Concurrent Actions

Expert's SecondaryConcurrent Actions

L + + Dc, Dts, -Dtd, Dwo, Psg, Posg, Pfp, Prp, Por, Fn, G1-2, G1-4, G3, Moa

Dc, Prp, Por, Fo

Dts + ~ Dc, Pfn, -Prn, Fi, G1-1, Ma DtdPor + ~ Dc, Dts, -Dtd, Dwo, L, Posg, Prp, Fo, G1.2, G1.4,

G2, G3L, Prp, Fo

Prp + + Dc, Dts, -Dtd, Dwo, L, Psg, Posg, Pfp, Por, Fn, G1-2,G1-4, G3, Moa

Dc, L, Pof, Por, Fo

Prn 0 + DcFo ~ + -Dtd, Pfn, Por, Frei, Fop, G1-3, G1-4, G2, G3 Dc, L, Prp, PorFn + + Dc, Dsy, L, Psg, Pfp, Prp, -Pof DcMoa + ~ Dc, Dts, Dsy, L, Psg, Pfp, Prp, Fn, Fnp, Mod Dc, Fn, Fop, G1-2G1-2 + ~ Dc, Dts, Dwo, L, Psg, Posg, Prp, Prn, Por, -G1.1,

G1.4, G4, -MaMoa

Dwo + 0 Dc, Dts, L, Posg, Prp, Prn, Por, G1-2, G1-4, G2, G3Dsy + 0 Dc, Psg, Pfp, -Pof, Fn, Fnp, Mod, MoaDtd - 0 -Dc, -Dts, -L, -Pfn, -Prp, -Por, -Fo, -Fi, -G1-4, -G3Psg + 0 Dc, Dts, Dsy, L, Pfp, Prp, Fn, Fnp, -G1.1, G1-2, G4,-

Ma, Mod, MoaPfp + 0 Dc, Dts, Dsy, L, Psg, Fo, Fi, G3G1-4 + 0 Dc, Dts, -Dtd, Dwo, L, Posg, Prp, Por, Fo, G1-2, G2,

G3G3 + 0 Dc, Dts, -Dtd, Dwo, L, Posg, Pfn, Prp, Por, Frei, Fo,

Fop, G1-3, G1-4, G2

(+) positive strong correlation(-) negative strong correlation

(~) substantial correlation(0) weak/no correlation


Miller’s theory & Cognitive Performance

• Concurrent actions at Primary level:– Novice/Expert– 14/5= 2.8

• the human short term memory can manage to process • 7+/-2 actions at one time (Miller, 1956).

• Expert’s protocol: – Cognitive actions= 2.8 x– Cognitive segments= 2.8 x


Hypothesis III• If sketching in 2D is a problem, • sketching in 3D may offer a solution.

– Imagine that just by drawing a room you could find yourself fully immersed in that space, or by shaping a virtual sculpture with your hands, you could model and touch its 3D physical model.

• 3D Sketchpad!• Sketching in 3D using a Virtual Reality System:

– VR hardware maps natural behavior onto digital streams

– VR software provides tools for construction of, management of, and interaction with

digital environments surrounding a user.


Macquarie University Start Up Grant 2004Macquarie University Safety Net Grant 2005

3D SKETCHPADCan we use our hands as dynamic input devices for

sketching?Aim: – to develop a prototype for a Virtual Reality (VR) interface – to recognise simple hand gestures and – build an interactive 3D model of a sketch drawn by the user. – The VR system provides a “3D Sketchpad" and – the designer has the benefit of a stereo image.

Stage III: 3D Sketchpad &Virtual Hand


VIRTUAL HAND• The proposed VR system provides the user with the ability of

using his/her hands as dynamic input devices for interactive 3D sketching within virtual reality (VR).

• The project describes a novel user interface in which a designer defines the contour of a sketch by controlling a pointer using a pair of data gloves in 3D space.

• Both the data gloves and the pointer incorporate 3D position sensors so that drawing primitives entered are recreated in real time on a head mounted display worn by the user.


• a medium composed of interactive computer simulations

– sense the participant's position and

– replace or augment the feedback to one or more senses

– giving the feeling of • being immersed or • being present in the simulation.

Virtual Reality


Characteristics of VR• Immersion / Visualization:

• Immersion is maintained at least in one sensory modality (vision)

– The computer generating visual, auditory or other sensual outputs to the user of a world within the computer.

• Interaction:– The user can interact with this virtual

world, directly manipulating objects in it.


1989 – Mattel designs Power Glove for Nintendo.– combined with Sega 3D glasses.

a handtracking device based on a glove. can track motion of the glove in 3D, finger position, and has a set of buttons/switches on the top of the wrist. has two modes "hires" and "lores“: In "hires" mode, the PG reports the position in threespace, the orientation, and configuration of fingers. In lores mode the glove reports position on the hand on the x and y axis and the buttons.

Virtual Hand Systems


Virtual Hand Hardware

Head Mounted Display (HMD) Data Glove

Tracking Device


VPL Data gloves

• Lightweight Lycra• A position/orientation tracker• A set of sheathed fiberoptic cables that run along

the back up each finger• Sensors on the back of finger-joints

– to detect the finger flex and extension

• Data of two joints are measured for each finger– (lower knuckle and middle knuckle)

• 10 joints specify the configuration of the hand


Cyber gloves

• 22 sensors: – 3 bend sensors and one

abduction sensor per finger– Thumb and pinkie cross-over– Wrist pitch and yaw


Cyber Grasp

• Intricate exoskeleton of aluminium

• Tactile feedback– Suppose the special

glove could not only sense joint angles, but also had actuators that could push back at you.

– The actuators could present the illusion of hard objects at particular locations.


Inertial

– Uses instruments that can detect and measure change in gyroscopic forces (acceleration and inclination).

– Generally inexpensive. – Doesn't require a reference point.

• need to set one arbitrarily.

– Can lose accuracy over time. – Works well in combination with other tracking systems.

•Measure position and/or orientation of a sensor:

–Position (3 axis): X,Y, Z

–Orientation/Rotation:

•Roll, Pitch, Yaw


Index Finger Point Open Hand

Sketch & GestureRecognition

draw and erase


Sketch Recognition

• Sketch Recognition– To match models of curvilinear configurations to hand

drawn sketches– approximation algorithms in the production of real-time

interactive computer graphics providing key parameters that carry information about the design concept.


• Magic Touch – a natural user interface (Pederson, 2000)

• VRShoe – Virtual Manufacturing Environment Group and the Institute of Industrial Technologies (ITIA-CNR) and Automation

(Sacco et al., 2002) • MagicMirror

– Augmented Reality System called by ITIA-CNR (Vigano et al., 2004) • Virtual DesignWorks

– Virtual Engineering Centre of Queens University, Belfast developed an Interface called (Liu et al., 2003)

• MOVE ON – Styling approach introduced to design automobiles using hand gestures

(Hummels et al., 1997)


• Virtual paint-brush – to recognise hand sketches – Research group in Hong Kong University of Science and

Technology (Chu et al., 2004) • Multi-stroke sketch recognition environment

– to draw UML class diagrams– Hammond et al. (2002)

• 3D Immersive Virtual Sculpting • Kuester et al, 2000 • head tracking + electro magnetic tracking• Stereo projection of 3D models onto a 2x1.5m area • like Immersive Workbench and Immersadesk• SGI Onyx2 rendering engine


Gesture Recognition• Classify movements and configurations of the hand in

different categories– Parametric information for that gesture can be extracted

from the way it was performed– An action in the virtual world can be executed.

• Two main portions:– Posture recognition (classification of the finger

configurations)– Path recognition (classification of the path)

• Gesture: the path of the hand while the hand fingers remain stable in a recognised posture.

– Template Matching• Includes 2 procedure: template matching & classification

– Collect samples for each class of gesture– Classification is performed by evaluating a function which measures the similarity

between the input data and templates – Set a similarity threshold: if below, the input data is rejected

» 30% of the total flexion range of each sensor» Templates tends to overlap in recognition space, if postures are above 10

– NN• Murakami, 1991 – data gloves, finger alphabet• Fel, 1990 – gesture to speech

– 92% success on the recognition of 203 signs based on 66 hand shapes

• Beale, 1992 - data gloves, American Sign language • Vaanenen, 1993, GIVEN – 5 time steps, 16 sensed data from Data gloves and

space tracker


– Feature based • Rubine, 1991- Stroke recognition

– Feature vector of 13 dimensions » Cos, sin of the gesture angle, length, angle of the bounding box, distance between 1st and last

points» Classification rate 100%, digits 98.5%, letters 97.1%

• Watson, 1995- Approximate Spline curve fitting • Starner, 1995 - American Sign language

– HMM:Highest probability of generating the data stream

• Liang, 1996 – Taiwanese Sign language

– Camera-based• Weissmann et al. (1999)

– Symbol based• Baudel, 1993 – data gloves controlling a hypertext program

– Kinematics based • Wellner, 1991, Digital Desk Calculator–

– through motion detection, 7 frames/sec

• Maes, 1995, ALIVE II– full body gestures


Stages of Virtual Hand System

• Creation of a simple 3D hand model using OpenGL:– Analysis of VR tools – System design of VRI– Integration of OpenGL 3D hand model and VR tools using

C++ – Simulation of hand gestures using the OpenGL hand– Implementation of simple sketch recognition algorithm


OpenGL interface• Creation of a simple 3D hand model

void Hand::create(void){

glPushMatrix();glPushMatrix();

this->drawPalm();glPopMatrix();glPushMatrix();

glPushMatrix();glScalef(1.0, 1.0, 0.6);glTranslatef(-2.5, 0.0, 0.0);

this->drawFinger(THUMB_FINGER); glPopMatrix();.........

this->drawFinger(INDEX_FINGER);.........

this->drawFinger(MIDDLE_FINGER);.........

this->drawFinger(RING_FINGER);.........

this->drawFinger(LITTLE_FINGER);glPopMatrix();

glPopMatrix();}

Wrist

Palm

Thumb

Index

Middle

Ring

Little


OpenGL interface II• Creation of a simple 3D hand model

void Hand::drawFinger(int finger){

glPushMatrix();glRotatef(-fingers[finger].knuckleJoint, 1.0, 0.0, 0.0);glPushMatrix();

glTranslatef(0.0, 0.0, 0.75);glutSolidCube(1.0);

glPopMatrix();glPushMatrix();

glRotatef(-fingers[finger].firstJoint, 1.0, 0.0, 0.0);glPushMatrix();

glTranslatef(0.0, 0.0, 1.25); // 0.75 + 0.5glutSolidCube(1.0);

glPopMatrix();glPushMatrix();

glRotatef(-fingers[finger].lastJoint, 1.0, 0.0, 0.0);glPushMatrix();

glTranslatef(0.0, 0.0, 1.0); // 0.75 + 0.5glutSolidCube(1.0);

glPopMatrix();glPopMatrix();

glPopMatrix();glPopMatrix();

}

Finger

1st Section

2nd Section

3nd Section


VR Hand Stage I• Creation of a simple 3D hand model

using OpenGL interface

Screen shot of OpenGL hand


VR Hand Stage II

• Integration of OpenGL 3D hand model and VR tools using C++

3D Sketching Application

Libraries for VR Hardware

Windows OS (Win32 C Compiler)

Standard C Libraries

OpenGL Interface

Flow of Control


VR Hand Stage III• System design of VRI

UML Class Diagram of VRI

GLUTApp

display()Tracker

update()

HandTracker

new()update()

HeadTracker

new()update()

GloveTracker

new()getGesture()

Hand

new()

0..1

1

0..1

1

This only testing purposes using keyboard

Circle

Tracks the position of

Recognises hand gestures and orientation

3D hand model realising the hand gestures

Line Polygon

Tracks the head orientation

Object3D

new()

Point3D

new()

0..*

2

0..*

2

1

0..*

1

0..*0..*

3..n

0..*

3..n

Sphere

0..*

11

0..*

BSR converts skeches to 3D objects and stores them

MainApp KeyboardTracker

Sketch

new()addPoint()

11

BasicSketchRecogniser

new()recog()

converts_to

Object3DStore

new()addObject()display()

0..*10..*1

SketchStore

new()addSketch()convertAll()display()

0..*

1

0..*

1

stores


VR Hand Stage IV• Simulation of hand using the OpenGL

hand model and a data glove

Open Hand Index Finger Point


VR Hand Stage V• Simulation of hand using the OpenGL

hand model and a data glove


Research Results• Hand gesture recognition

– 25=32 possible combinations of gestures– 5W:1 sensor per finger vs 16W:3 sensors– Orientation trackers

• Switch tracking the motion of the hand in 3D• Zoom in and out using mouse or keyboard• Need motion trackers: SpacePad

Gesture Sketching Task

Flexure value x (0 ≤ x ≥ 1)

ID Name Thumb Index Middle Ring Little

0 Fist Stop ≤ 0.1 ≤ 0.1 ≤ 0.1 ≤ 0.1 ≤ 0.1

1 Index Finger Point

Draw ≤ 0.1 ≥ 0.9 ≤ 0.1 ≤ 0.1 ≤ 0.1

2 Open Hand Erase ≥ 0.9 ≥ 0.9 ≥ 0.9 ≥ 0.9 ≥ 0.9

Gesture Definition Table


Conclusion

• 3D Sketchpad requires both technical and conceptual developments to allow natural interactions with Virtual Environments– Explore sketching in 3D as a language of conceptual design

process

– Use design cognition to build an interactive complex system

• Hand gesture interface: – object selection, action selection, action modifiers, rhythm of

interaction, syntax of hand gestures, semantics of pause, comma and retraction.

• Multi-modal interfaces:– Inclusion of speech in gesture analysis may simplify gesture

segmentation and feature detection.

• http://www.comp.mq.edu.au/~isvr

http://www.comp.mq.edu.au/~isvr


Macquarie University Safety Net Grant 2005Face Detection through interactive sketching in virtual reality (Kavakli) • Aim: developing and testing the 3D sketchpad prototype for the face

detection of criminals in a Police database using sketches. • the grammar of sketches: facial features. • Thus, the proposed VR system will allow a police officer to sketch the face of

a suspect from the verbal descriptions of eye-witnesses and generate the 3D model of the suspect to be displayed on a head mounted display worn by the officer. – The model produced will be projected to a computer screen or a digital wall to get

the eye-witness’ comments on the facial features of the suspect throughout sketching.

– The completed model of the suspect will be fed to a criminal database to find a match.

Future plans

http://inventory.universityarchives.com/browse.asp?sn=37592-001&show=True&thumbnails=True&range=&fullimage=3


• Face Detection and Reconstruction Using Forensic Arts and Virtual Reality Technology (Kavakli & Watters) 2006-2009– Some of the proposed post-September11 uses of facial identification

systems in immigration and airport security have been welcomed without clearly evaluating their effectiveness and without weighing the potential harms involved with its use.

• Aim: to improve the accuracy rate in face identification with different facial expressions and features. – Combining 2D and 3D face data and finding the primitives of a generic

face, we will develop novel face identification techniques based on visual cognition and test their performance with the implementation of an intelligent virtual reality interface in the detection of criminals based on forensic evidence.


Division of Information & Communication Sciences

Department of Computing Interactive Systems &Virtual Reality Research Group

• Virtual Reality LAB – Virtual Reality – Interactive Systems – Computer Games– Computer Graphics

29 active members including:

12 Full-time staff members

8 Full-time Postdocs and Postgrads

4 Part-time research associates

5 Honours students


Our Research & Infrastructure Grants

• 2005-2007 Australian Research Council Discovery Grant (Richards, Kavakli, Dras) Risk Management Using Agent Based Systems, Macquarie University ($410K)• 2005 Australian Research Council Linkage International Fellowship (Kavakli, Pelachaud, Szilas) Interactive Drama Engine in Virtual Reality , Macquarie University ($71K)• 2003-2005 Australian Research Council Linkage Grant (Kavakli, Bossomaier, Cooper) Cognitive Modeling of Computer Game Pidgins ($75K)• 2005 Macquarie University Research Infrastructure Block Grant (Kavakli, Watters, Richards, Burke, Szilas, Leslie) Virtual Reality Engine($58K)• ICS Major Equipment Grant ($90K)

Virtual Reality Lab• 2005 Safety Net Grant, Macquarie University (Kavakli)

Face Detection Through Sketching in Virtual Reality ($19K)• 2004 ICS Research Startup Grant (Kavakli)

Interactive Sketching in Virtual Reality, Macquarie University ($6K)


User Interface Design Cognition

Artificial Intelligence

Visualisation & Computer Graphics Knowledge Acquisition

& Structuring

Virtual Reality

Education

Virtual Environments Computer Games Design

Our Research Areas


ReferencesSacco, M., Vigano, G., Paris, I., “Virtual reality and CAD/CAM systems applied to custom shoe manufacture on a mass market basis”, in proceedings of the World Congress on Mass Customization and Personalization(MCPC01), Hong Kong, September 2001.

Vigano, G., Mottura, S., Greci, L., Sacco, M., Boer, C., “Virtual reality as a support tool in the shoe life cycle”, International Journal of Computer Integrated Manufacturing, October-November 2004, vol. 17, no. 7, pp. 653-660.

Liu, x., Hinds, B., McCartney, J., Dodds, G., ”Virtual DesignWorks - Designing 3D CAD models via touch interaction”, International Mechanical Engineering Congress and RD&D Expo, Washington, November 2003.

Pederson, T., “Human Hands as a Link between Physical and Virtual”, in proceedings of the Conference on Designing Augmented Reality Systems (DARE2000), Helsinore, Denmark, April 2000.

Hummels, C., Smets, G., Overbeeke, C., “An Intuitive Two-handed Gestural Interface for Computer Supported Product Design”, In proceeding of the International Gesture Workshop on Gesture and Sign Language in Human-Computer Interaction, September 1997.

Hummels, C., Paalder, A., Overbeeke, C., Stappers, P., Smets, G., “Two-handed Gesture-based Car Styling in a Virtual Environment”, In proceeding of the 30th International Symposium on Automotive Technology and Automation (ISATA), Mechatronic, 1997.

Chu, N., Tai, C., “Real-Time Painting with an Expressive Virtual Chinese Brush”, IEEE Computer Graphics and Applications, September/October 2004, pp. 76-85.

Bricken, W., ”Virtual Reality: Directions of Growth”, Notes from the SIGGHRAPH ’90 Panel, 1990.

Weissmann, J., Salomon, R., “Gesture Recognition for Virtual Reality Applications Using Data Gloves and Neural Networks”, In proceedings of the IEEE International Joint Conference on Neural Networks, July 10-16, 1999.

Hammond, T., Davis, R., “A Geometrical Sketch Recognition System for UML Class Diagrams”, In proceedings of the AAAI Spring Symposium on Sketch Understanding, pp. 59-66, 2002.

Davis, R., Adler, A., Alvarado, C., Hammond, T., Hitchcock, R., Sezgin, T., Veselova, O., “Designs for the future”. MIT Artificial Intelligence Laboratory Annual Abstract, September 2002.

IsVR: Interactive Systems and Virtual Reality Research Group 2005 Virtual HAND: A Virtual Reality Interface Dr Manolya Kavakli Senior Lecturer Department.

Documents

d sketch

design studies

sketch recognition

d transformation

d space

d model

design research center

object recognition