ARIS Augmented Reality Image Synthesis through illumination reconstruction and its integration in interactive and shared mobile AR-systems for E-(motion)-commerce.

ARISAugmented Reality Image Synthesis through

illumination reconstruction and its integration in interactive and shared mobile AR-systems for E-

(motion)-commerce applications

9th E3 Concertation Meeting, Brussels, September 10th, 2002

Mr. Ioannis Koufakis, INTRACOM S.A.

[email protected]

Introduction

• Goals of ARIS:– Provide new technologies for seamless

integration of virtual objects in an augmented environment.

– Develop new visualization and interaction paradigms for novel collaborative AR-applications.

• Two application scenarios will be developed:– Interactive desktop system.– Mobile AR-unit,that could be used on-site.

Introduction

Introduction

• Collaboration– Between participants on-site– With a remote consultant (interior designer)

in the store

• Realism enhancements– Physically correct placement of furniture

(scaling, occlusion/collision detection etc.)– Modeling of lighting effects and correct

visualization of virtual objects within the real environment (shading, illumination etc.)

Consortium

House Market S.A.(IKEA)A7

Athens Technology Center

P6

University of BristolP5

University of Manchester

P4

Inria-LoriaP3

Intracom S.AP2

Fraunhofer InstituteC1

C - Coordinator P - Principal ContractorA - Assistant Contractor

Contract Information

• Contract No.: IST-2000-28707• Duration: 36 months• Commencement: Oct. 2001• Current Stage: 12th month• Budget: ~4.9M Euro ( ~2,9M

funded by EU)• Web: aris-ist.intranet.gr

Geometry and Illumination• Geometry Reconstruction

– Scene and camera reconstruction, both from still images and videosequences.

– Real-time camera tracking.• Illumination Reconstruction

– Reconstructing illumination and material properties from a sequence of images and 3D scene models.

– Camera calibration:computation of camera response function.

Combined Lighting Simulation

• Algorithms for lighting simulation and image synthesis techniques for integration of virtual and real environments.

Perceptual Evaluation• Development of a psychophysical

framework measuring the perceptual realism in mixed/augmented reality scenes

• Tone mapping operators– Comparison of different tone mapping

operators for both low and high dynamic range images.

– Development of perceptual tone mapping operators.

Application Scenarios• Specification of realistic application scenarios in

an interior design application environment.• User Requirements definition spanned two

phases:– Initial phase, in order to guide research activities

• Questionnaires on ARIS web site were filled by visitors of the site

• Use case diagrams defined in UML notation

– Final phase (due now), in order to capture the work situation in which the system will be deployed

• Special attention to HCI part• A users’ workshop was organized internally in INTRACOM to

evaluate the usability and acceptance of various tools.

Users Workshop

Application Scenarios• Survey of related systems and

applications (deliverable D5.2)• Results of the previous two were used

to define the application scenarios and provide an initial specification of the system.

Application System• 3D e-commerce server: repository of

furniture models along with e-commerce functionality.

• Database was designed and implemented in Oracle 9i (ICOM,ATC).

• First approach of AR mobile unit was designed (ZGDV)– Marker based approach, for placement of

furniture items within a scene– Active X component for visualization (AR

browser), displaying VRML and live video.

• Wireless network, based on IEEE 802.11b.

Architecture Overview Interactive (web-based) application.

Participants on-site

3D e-Commerce Server

Authorization module

Presentation Component

Communication

component

Collaboration Gateway

Repository

DIM

3D models of furniture, textures etc.

User profiles

DIM

Wearable Computer

(coordinator)

Tracking

Visualization

Communication

Collaboration

Wearable Computer

(participant)

Tracking

Visualization

Communication

Broadband modem

Access Point

Broadband Network

Local Server3D Geometry Reconstruction module

Illumination Reconstruction module

3D Geometry Reconstruction module

Illumination Reconstruction module

Collaboration Module

Demos• Image synthesis using illumination

reconstruction• AR browser• Interaction with Joystick• Interaction with Voice Recognition

Problems so far• User Interaction required for calibration and

3D reconstruction is too much (esp. for one of the techniques)Vision based techniques should be investigated to

automate the process.

• Illumination reconstruction with the light probe is a strict procedure and requires knowledge of digital cameras and photographyAutomating detection of light sources should be

pursued.

Problems so far• Extension of those techniques to mobile AR-

unit is not easy, especially due to limitations of mobile devices(small display, lack of keyboard, not enough computing power,etc.) Introduction of local server on-site, where an

expert could operate the geometry and illumination modules.

New interaction paradigms e.g voice,gesture recognition

• VRML not capable of handling illumination data and perform rendering of shadows.

Future Work

• Finalization of the system architecture and design.

• Investigate on how to minimize /improve user interaction.

• Collaboration paradigms: – Platforms to be used: NetMeeting, VoIP,

Whiteboards, chat, Shared 3D Worlds etc.

• Investigate how wearable machines could be exploited.

• X-3D standardization effort.

Cross-project collaboration• Exchange of experiences, especially for

the mobile AR unit, with other projects – LifePlus– Archeoguide– Arvika (funded by German Ministry of

Education and Research)• Interaction paradigms within 3D worlds

for mixed/augmented environments– Clustering