Visual Cues For The Instructed Arrangement of Physical Objects Using Spatial Augmented Reality (SAR) Jessica Tsimeris Supervisor: Bruce Thomas Wearable.

Visual Cues For The Instructed Arrangement of Physical Objects Using Spatial Augmented Reality (SAR)Jessica TsimerisSupervisor: Bruce ThomasWearable Computer LabUniversity of South Australia

What is Augmented Reality?

•Properties of Augmented Reality:▫Blends real and virtual, in a real

environment▫Real time interactive▫Registered in 3D [thus, the real and virtual

objects are aligned]Azuma et al (2001)

What is Augmented Reality?

•Typically requires Head Mounted Devices (HMDs).

•Differs from Virtual Reality because:▫Virtual Reality suppresses the real world▫Augmented Reality enhances the real world

Raskar et al (2005)

What is Spatial Augmented Reality?•Branch of AR

•The paradigm was introduced in 1998 by Raskar et al

•Enhances the real world with virtual objects which are integrated into the real world.▫Projectors▫Flat panel displays▫etc

Advantages of SAR

•Users don’t have to wear a head mounted device or hold a device

•Allows the object to be rendered at the real world location▫Better for the human eye

Advantages of SAR

•Multiple user interaction can be achieved without multiple HMDs or mobile devices

•The visual fidelity of a physical object is unchanged, only the augmentations are rendered with lesser quality.

Advantages of SAR

•Allows for larger, brighter, higher resolution virtual objects

•Therefore allowing:▫Increased integration into the real world▫Increased immersion▫Improved user interaction

•Not restricted to a low resolution▫Mobile devices

Previous Work

•Shaderlamps, projection of textures onto scaled buildings and other real objects

Raskar et al (1999)

Previous Work

•The BUILD-IT system, a virtual object is manipulated via physical object manipulation

Fjeld (1999)

Previous Work• Virtual Object Manipulation in Virtual Worlds

▫Multimodal – speech and gestures, Irawati et al (2006)

▫Kato et al (2000) addressed a similar problem, but not multimodal

• Virtual Object Manipulation in Real Worlds▫Arrange virtual objects, surround user in a

sphere, Webster et al (1996)▫Wang et al (2007) addressed a similar problem,

but not limited to a sphere

Research Question

•What are the appropriate Spatial Augmented Reality visual cues to instruct a user on how to arrange physical objects?

Example Use

•A stagehand has to arrange sets in a theatre. They would like to rearrange the current sets and include new sets▫The system could tell him what objects to

move and the order in which to move them

Methodology

•To arrange the physical objects, the following instructions will be performed:▫Rotate▫Translate (move)

Methodology – Translation Instruction

•A circle is projected at the centre of the current location and another is displayed at the centre of the destination location

•A line between the circles shows a path from the current position to the destination

Methodology – Translation Instruction

•The projection updates as the user moves the physical object

•The object is successfully translated when the circles overlap

Methodology – Translation Instruction

•Dotted shape is the destination position•Solid shape is the current position of the

object

Methodology – Translation Instruction

•Dotted shape is the destination position•Solid shape is the current position of the

object

Methodology – Translation Instruction

•Dotted shape is the destination position•Solid shape is the current position of the

object

Methodology – Rotation Instruction

•A line from the centre of the object is displayed to indicate the current rotation, and another line is displayed to indicate the destination rotation

•A circle is projected at the end of each line

Methodology – Rotation Instruction

•The projection updates as the user rotates the physical object

•The object is successfully rotated when the circles overlap

Methodology – Rotation Instruction

•Dotted shape is the destination position•Solid shape is the current position of the

object

Methodology – Rotation Instruction

•Dotted shape is the destination position•Solid shape is the current position of the

object

Methodology – Rotation Instruction

•Dotted shape is the destination position•Solid shape is the current position of the

object

Methodology – Visual Cues

•Most used the translation and rotation instructions as described

Methodology – Visual Cues

•Most used the translation and rotation instructions as described

Methodology – Visual Cues

•Most used the translation and rotation instructions as described

Methodology – Visual Cues

•Most used the translation and rotation instructions as described

Implementation

•Camera•Projector•Tracking

▫ARToolKitPlus•SAR Module

▫OpenGL/C++

Implementation

•Two Dimensional Environment▫To arrange 2D representations of the

physical objects from a top-down view.▫Arrangement can then be performed on the

corresponding physical objects using SAR cues. Can be used for remote object arrangement

Implementation

•Object Arrangement Process▫Revisit incorrect

instructions.

Implementation

Implementation

Implementation

Implementation

Implementation

User Study

•Nine visual cues to arrange objects▫One without using SAR▫Eight visual cue variations

User Study• Task Completion Time• Accuracy• User Opinion (survey)

1 2 3 4 5 6 7 8 90

5000

10000

15000

20000

25000

30000

35000

40000

45000

50000

Task Number

Tim

e (

mil

lise

conds)

Results

•The cue that didn’t use SAR was the second slowest

Results

•The cue that didn’t use SAR had the least accurate x axis arrangements

1 2 3 4 5 6 7 8 90

5

10

15

20

25

30

35

Task Number

Avera

ge x

Axis

Vari

ati

on (

unit

s)

1 2 3 4 5 6 7 8 90

5

10

15

20

25

30

35

Task Number

Avera

ge y

Axis

Vari

ati

on (

unit

s)Results

•The cue that didn’t use SAR had the least accurate y axis arrangements

1 2 3 4 5 6 7 8 90

1

2

3

4

5

6

7

8

Task Number

Avera

ge R

ota

tion V

ari

ati

on (

degre

es)

Results

•The cue that didn’t use SAR had the least accurate rotation

Results

•95% of participants preferred Task 2 (Translation First) over the manual arrangement

Preference for Task 1 (No SAR)Preference for Task 2 (Translation First)

Preference for the Wireframe Cues (Task 6 and Task 7)Preference for the Square Cues (Task 8 and Task 9)

Results

•75% of participants preferred visual cues that utilised a square projection over visual cues that utilised a wireframe projection

Results

•Visual cues that displayed both rotation and translation at the same time were most preferable.

2 3 4 6 7 8 90

2

4

6

8

10

12

Preferences for the Visual Cue as Part of a Combi-nation

Preference for the Visual Cue

Task Number

Nu

mb

er

of

Pre

f-ere

nces

Conclusion

•SAR visual cues have been developed that are:▫Faster than a manual arrangement

technique▫More accurate for x axis translation▫More accurate for y axis translation▫More accurate for rotation▫Preferred over a manual arrangement

technique

Improved Visual Cue

•Development of a new visual cue that took the user study results into account, particularly:▫Square Cues were quicker▫One of the Square Cues was the most

accurate cue▫Participants preferred seeing both

translation and rotation at the same time More intuitive

▫Participants ranked the square cues highly Legible, less cluttered

Improved Visual Cue

•Developed from results of the user study

Improved Visual Cue

•Developed from results of the user study

Improved Visual Cue

•Developed from results of the user study

Improved Visual Cue

•Developed from results of the user study

Improved Visual Cue

•Developed from results of the user study

Improved Visual Cue

•Developed from results of the user study

Improved Visual Cue

•Developed from results of the user study

Improved Visual Cue

•Developed from results of the user study

Improved Visual Cue

•Developed from results of the user study

Improved Visual Cue

•Developed from results of the user study

Future Work

•The order in which the objects will be arranged

•Better tracking•Formal evaluation of the new visual cue•Multiple cameras and projectors

▫Testing scalability

References• Azuma, R 1997, ‘A Survey of Augmented Reality’, Presence: Teleoperators & Virtual

Environments, vol. 6, no. 4, pp. 355-385.• Bimber, O & Raskar, R 2005, Spatial Augmented Reality: Merging Real and Virtual

Worlds, AK Peters, Ltd, Wellesley, MA.• Fjeld, M, Voorhorst, F, Bichsel, M, Lauche, M, Rauterberg, M & Krueger, H 1999,

'Exploring Brick-Based Navigation and Composition in an Augmented Reality', Handheld and Ubiquitous Computing, vol. 1707, pp. 102-116.

• Irawati, S, Green, S, Billinghurst, M, Duenser, A & Ko, H 2006, '"Move the Couch Where?": Developing an Augmented Reality Multimodal Interface', Symposium on Mixed and Augmented Reality, Proceedings of the 5th IEEE and ACM International Symposium on Mixed and Augmented Reality, pp. 183-186.

• Kato, H, Billinghurst, M, Poupyrev, K, Imamoto, K & Tachibana, K 2000, 'Virtual Object Manipulation on a Table-Top AR Environment', Human Interface, pp. 275-278.

• Raskar, R & Low, K-L 2001, Interacting With Spatially Augmented Reality, ACM, Camps Bay, Cape Town, South Africa, pp. 101-108.

• Wang, X & Gong, Y 2007, Augmented Virtuality Space: Enriching Virtual Design Environments with Reality, Brisbane, Australia.

• Webster, A, Feiner, S, MacIntyre, B, Massie, W & Krueger, T 1996, Augmented Reality in Architectural Construction, Inspection, and Renovation.

Questions?