Occlusion based Interaction Methods Tangible Augmented ...gun-a-lee.appspot.com/pub/vrcai2004-ppt.pdf · Occlusion based Interaction for Tangible AR Environments Simple, easy to use,

VR Lab HITLabNZPOSTECH Univ. of Canterbury

Occlusion based Interaction Methodsfor

Tangible Augmented Reality Environments

Gun A. Leeα Mark Billinghurstβ Gerard J. Kimα

α Virtual Reality Laboratory, Pohang University of Science and Technology

β Human Interface Technology Laboratory New Zealand,University of Canterbury

[email protected]

VRCAI 2004

VR LabHITLabNZ

2VRCAI 2004

Introduction

■ Tangible Augmented Reality[Kato 2001]

■ Each virtual object registered to physical object

■ Users manipulate virtual objects by manipulating corresponding physical objects

VR LabHITLabNZ

3VRCAI 2004

Introduction

■ Tangible AR Applications

▲MagicBook

ARGroove ►

▼ VOMAR

◄ Tiles

VR LabHITLabNZ

4VRCAI 2004

Introduction

■ Tangible Augmented Reality■ Easy natural 3D interactions

■ Full 6DOF 3D Direct manipulation

■ 1/2D interactions?■ Buttons?■ Sliders?■ Menus?■ Keypads?

VR LabHITLabNZ

5VRCAI 2004

Related Works

■ 2D Input methods in AR■ Screen stabilized fashion

■ Tinmith [Piekarski 2002]

■ Magic Ring [Dias 2003]

■ World stabilized fashion■ Windows on World■ ARGUI [Geiger 2003]

■ Hand gesture mouse [McDonald 2003]

■ Used indirect input methods (mouse, getsture, etc) rather than direct touch

VR LabHITLabNZ

6VRCAI 2004

Related Works

■ Large scale touch sensor■ SmartSkin [Rekimoto 2002]

■ Touch implies occlusion■ Commercial products using

occlusions as inputs■ Virtual Keyboard, Canesta■ Parapara Paradise, KONAMI

VR LabHITLabNZ

7VRCAI 2004

Occlusion based Interaction

■ 2D Interaction using pointers■ Pointer centered view

■ Tracking the pointers■ Cost depends on number of pointers + interaction

points■ Interaction point centered view

■ Sensing on interaction points■ Cost depends on number of interaction points

■ In TAR environments, interaction points (props) are trackedrather than pointers (hands).

VR LabHITLabNZ

8VRCAI 2004

Occlusion based Interaction

■ Occlusion Detection in TAR■ Tracking a set of visual markers with

predefined spatial relationship■ Provides robustness■ Invisible marker positions are inferred from

the visible ones

■ Occlusion = Invisible + Within the view■ Two methods for detecting occlusions

■ Boundary Marker Method■ Estimated Marker Projection Method

VR LabHITLabNZ

9VRCAI 2004

Occlusion based Interaction

■ Boundary Marker Method

VR LabHITLabNZ

10VRCAI 2004

Occlusion based Interaction

■ Estimated Marker Projection Method

VR LabHITLabNZ

11VRCAI 2004

Occlusion based Interaction

■ Boundary Marker Method■ Easy and straight forward, reliable

implementation■ Lesser computation

■ Estimated Marker Projection Method■ No need of boundary markers■ More computation required■ Estimation errors due to lens distortion

■ Over estimating the size of markers helps

VR LabHITLabNZ

12VRCAI 2004

Occlusion based Interaction

■ Interaction Design■ Time out constraints

■ Point & hold -> Explicit command■ Preventing accidental commands■ Repeating the command (common in

keyboards)

VR LabHITLabNZ

13VRCAI 2004

Occlusion based Interaction

■ Interaction Design (cont’d)

■ Sub-marker level measurement■ Two consecutive markers occluded

-> The pointer is in-between them■ Relatively lesser markers needed

VR LabHITLabNZ

14VRCAI 2004

Occlusion based Interaction

■ Interaction Design (cont’d)

■ Tip point marker detection■ 2D grid of markers■ Top-left one from the

occluded marker blob■ Other heuristics applicable

– e.g. select top-right one when the blob is skewed to the right

VR LabHITLabNZ

15VRCAI 2004

Implementation

■ Desktop■ Custom built PC with Windows XP■ Athlon 1.5GHz / 512MB■ NVIDIA GeForce4 MX

■ Mobile■ Apple Macintosh iBook with Mac OS X■ G3 900MHz / 640MB■ ATI Radeon 7500

VR LabHITLabNZ

16VRCAI 2004

Implementation

■ ARToolKit (2.65DS / 2.61Mac)■ Logitech Notebook PRO USB Camera

■ 320x240 @ 30Hz■ i-Visor HMD

■ Video image stretched to fit 800x600■ OpenGL drawn in full resolution

■ Performance: 19~30fps■ 5fps with 35 markers on mobile

configuration (low computation power)

VR LabHITLabNZ

17VRCAI 2004

Applications

■ Buttons

VR LabHITLabNZ

18VRCAI 2004

Applications

■ Sliders

VR LabHITLabNZ

19VRCAI 2004

Applications

■ Menu bars

VR LabHITLabNZ

20VRCAI 2004

Applications

■ Ball-pushing Game

VR LabHITLabNZ

21VRCAI 2004

Applications

■ Calculator

VR LabHITLabNZ

22VRCAI 2004

Applications

■ Board games

VR LabHITLabNZ

23VRCAI 2004

Discussion

■ Informal user study■ 6 subjects■ 7 scale questionnaire

■ Easy to learn: 5.6 (stddev=0.55)■ Easy to use: 5 (stddev=0.7)

■ Two 2D game applications demonstrated on public■ Over 100 peoples tried and gave

positive feedbacks

VR LabHITLabNZ

24VRCAI 2004

Discussion

■ The interaction appeared natural■ Using bare hands■ Direct manipulation (pointing)

■ Occlusion of users’ hands by virtual objects were reported unnatural ■ Partially solved by making the interface

semitransparent when occluded■ Easily applicable to mobile and/or

wearable systems

VR LabHITLabNZ

25VRCAI 2004

Conclusion & Future Works

■ Occlusion based Interaction for Tangible AR Environments■ Simple, easy to use, and natural 2D interaction■ Low development and computational cost

■ Future Works■ Try other type of visual markers or visual

tracking methods■ Add passive tactile feedbacks