A Multi-agent Approach for the Integration of the Graphical and Intelligent Components of a Virtual Environment Rui Prada [email protected] INESC-ID.

A Multi-agent Approach for the A Multi-agent Approach for the Integration of the Graphical and Integration of the Graphical and Intelligent Components of a Intelligent Components of a Virtual EnvironmentVirtual Environment

Rui [email protected] – Technical University of Lisbon, Portugal

Introduction

Report on ELVIS Project achievements Combine efforts from two different research

communities Artificial Intelligence Computer Graphics

Open Source Framework for easy development of Intelligent Virtual Environments populated with Synthetic Characters (AgentLib)

Motivation (1)

Entertainment and Education applications often make use of virtual realities (environments)

Often inhabited by Synthetic Characters Interact with the user However the interactions should be believable in

order to give a good interaction experience to the user

Two different dimensions: Visual believability Behaviour believability

Motivation (2) Visual believability

Believable appearance Natural body and facial animations (gestures) Believable interaction with the virtual world Emotional expression

Behaviour believability Action planning Knowledge representation Synthetic emotions Social skills Nature language

Motivation (3)

Both dimensions are important But they have very distinct problems,

development approaches and tools Additionally, the integration of the intelligent and

the visual components is often not clean Dependencies between the intelligent and the visual

components Problems for reusability

Motivation (4)

To build a framework that supports the development of virtual environments and synthetic characters

And present a library with ready to use components

We need to have reusable components Goal: Create a well defined interface between

the graphic and the intelligent components

The Framework (1)

Inspiration from the Model-Controller-View paradigm Create an abstract layer between the two different kind

of components Interconnect the intelligent and visual component

without creating dependencies between them

AbstractModel

IntelligentComponents

VisualComponents

Behaviour Layer Presentation LayerAbstract Layer

The Framework (2)

The presentation layer that contains the components that present the simulated world to the user (e.g. 3D visualization and animation)

The abstract layer that contains an abstract representation of the simulated world and the synthetic characters (e.g. an abstract model)

The behaviour layer that contains the components that implement the intelligent behaviour of the synthetic characters. (e.g. planning algorithms)

Abstract Model (1)

Based on the Entity-Relationship knowledge representation scheme Contains the abstract representation of the world's

entities, their properties and relations

Entity Property1 N

Relation

N

N

N1

Entities block A, block B, and block

C

Properties colour(A, blue) colour(B, green) colour(C, yellow)

Relations OnTopOf(A, B) OnTopOf(Table, A) OnTopOf(Table, C)

Abstract Model (2)

Table

A

B

C

Abstract Model (3)

Agent Based Approach Entity Hierarchy Agent

Behaviours

Entity Property

Relation

1 N

N

N

N1

1

N

Agent

Behaviour

1

Behaviours

Define the abilities of the agents in the world Change the abstract model

E.g. set new properties or relations, add or remove entities

Execution cycle Start procedure

Arguments

Stop procedure Termination test

Update Cycle

Every Entity (including the agents) has two update methods Start Update Finished Update

Two methods to guarantee that the update process is independent from the order of update

One entity updates its children Calls the update methods of its child entities Calls the update methods of its behaviours

There is one Root Entity for each world

Events

Changes in the abstract model and the execution of behaviours generate events Property Changes Events Relation Changes Events Behaviour Execution Events (start, stop) Add/remove Entity Event Add/remove Behaviour Event User defined events (raised by the behaviours)

Components register into the Entities that they want to observe

Interface with the Intelligent Components

● Agents can be controlled by Intelligent components external to the Abstract Model1.These components register into the entities that they

want to monitor

2.Request the execution of actions in the world through the execution of behaviours

(a)Start a behaviour

(b)Stop a behaviour

3.Receive Perceptions from the World through the Event mechanism

View (1)

Establishes the interface with the presentation layer

Encapsulates the representation engines and devices 2D animated world 3D virtual reality Augmented reality

View (2)

View Represents the device that displays the world to the

user (e.g. the drawing window) Characterized by several Attributes (e.g. window size) Composed by several Layers

Layer x,y coordinates Width, height Depth level Attributes

View Attributes

Layers

1 N

N

N1

1

View (3)

Layers are placeholders to attach Layer Elements These Elements define the contents of the layers

and the representation object that they display Bitmap 3D Camera Text Area View Attributes

Layers

1 N

N

N1

1LayerElement

11

1

N

View (4)View

Layer (Bitmap)

Layer (Text Field)

Layer (Button)

View (5) Scenes

Are characterized by several Attributes Contain several Scene Elements Attached to a Layer through Layer Elements (e.g. a

Camera)

Scene Elements Attributes Feature Points

Where other Scene Elements may be attached

Plug-ins Manipulate the Scene Elements

Examples of Scene Elements: 3D Model, Light Source

View (6)

View Attributes

Layers

1 N

N

N1

1LayerElement

11

1

N

SceneScene

ElementN1

Plug-inN1

FeaturePoints

N

1

1

N

N

1

1

1

View (7)

Plug-ins Started Stopped Examples: Animations, Path Interpolators, Sound

Sources

Observers Registered in the Scene Elements Receive events

Collision events User Input

Keyboards events

The Whole Picture The Intelligent Components interact with the entities

defined on the Abstract Model Receive perception through events Act through agents' behaviours

The behaviours may change the Abstract Model and the View Elements For example: one may define a behaviour that plays an

animation on a 3D character and then, when it finishes, changes the a property in the Abstract Model

The View interface defines and abstract language that is implemented and realized by the Components in the presentation layer (e.g. The 3D Graphics Engine)

Test CaseExample: FearNot! episode

Abstract Model

User

Luke

John

Books

View / Interaction Manager

Stage ManagerControl

Luke’s Mind

Control

John’s Mind

3D Luke

3D John

3D Books

3D Classroom

User Interface

Test Case (2)

Video

Conclusions (1)

Multi-agent technology may is used to define an abstract layer that promotes the interdependency of the intelligent and visual components of a virtual environment

This independence allows the components to be reused and therefore to build a toolkit for rapid development of virtual environments, namely with the presence of synthetic characters

Conclusions (2)

Using an abstract common model is one way to share research efforts

The Abstract Model was used to establish the interface of the Intelligent Components (e.g. planning algorithms) that implement the behaviour of some synthetic characters in some of our projects: e.g. the VICTEC project

The View abstraction is currently being implemented and will be tested with a simple example

Questions?

[email protected] The AgentLib is on SourceForge

sourceforge.net/projects/agent-sg/

Elvis Workshop in Lisbon 6th to 8th of April