MediaHub: An Intelligent MultiMedia Distributed Platform Hub

PGNET, Liverpool JMU, June 2005

MediaHub: An Intelligent MultiMedia

Distributed Platform Hub Glenn Campbell, Tom Lunney, Paul Mc Kevitt

School of Computing and Intelligent Systems Faculty of Engineering

University of Ulster, Magee CampusNorthland Road, Derry

{Campbell-g8, TF.Lunney, P.McKevitt} @ulster.ac.uk


Outline Goals and objectives Key research problems Distributed Processing Distributed Platforms Architecture of MediaHub Tools and future development


Project goals

The primary objectives of this research are to:

Interpret/generate semantic representations of multimodal input/output

Perform decision-making (fusion and synchronisation) over

multimodal data Implement MediaHub, a multimodal platform hub


Project objectives

Focus on following research questions:

Will MediaHub use frames for semantic representation or XML or one of its derivatives?

How will MediaHub communicate with various elements of a platform?

Will MediaHub constitute a blackboard or non-blackboard model?

What mechanism will be implemented for decision-making within MediaHub?


Key research problems Semantic Representation

Represent language and vision Frames or XML?

Semantic Storage Blackboard model? Non-blackboard model?

Decision-making Fusion and synchronisation AI technique


Frames

(CHAMELEON)

[MODULEINPUT: inputINTENTION: intention-typeTIME: timestamp] [SPEECH-RECOGNISERUTTERANCE:(Point to

Hanne’s office)INTENTION: instruction!TIME: timestamp] [GESTUREGESTURE: coordinates (3,

2)INTENTION: pointingTIME: timestamp]

XML (M3L, SmartKom)<presentationTask> <presentationGoal> <inform> <informFocus> <RealizationType>list </RealizationType>

</informFocus> </inform> <abstractPresentationContent><discourseTopic> <goal>epg_browse</goal> </discourseTopic><informationSearch id="dim24"><tvProgram id="dim23"> <broadcast><timeDeictic id="dim16">now</timeDeictic> <between>2003-03-20T19:42:32 2003-03-20T22:00:00</between> <channel><channel id="dim13"/> </channel> </broadcast></tvProgram></informationSearch> <result> <event><pieceOfInformation> <tvProgram id="ap_3"><broadcast> <beginTime>2003-03-20T19:50:00</beginTime> <endTime>2003-03-20T19:55:00</endTime> <avMedium> <title>Today’s Stock News</title></avMedium> <channel>ARD</channel></broadcast>…….. </event> </result></presentationGoal> </presentationTask>

Semantic representation


Semantic storage Blackboard or Non-blackboard?

High coupling – Blackboard? Low coupling - distributed architecture?

Communication Via central blackboard? Message passing between modules?


Decision-making (fusion & synchronisation)

Rule-based

Potential for Other AI techniques

Fuzzy Logic Neural Networks Genetic Algorithms Bayesian Networks (CPNs)


Distributed processing

PVM (Parallel Virtual Machine) (Sunderam 1990, Fink et al. 1995) ICE (Amtrup 1995) DACS (Fink et al. 1995, 1996) Open Agent Architecture (OAA) (Cheyer et al. 1998, OAA 2004) JATLite (Kristensen 2001, Jeon et al. 2000) JavaSpaces (Freeman 2004) CORBA (Vinoski 1993)


Intelligent Multimedia Distributed Platforms

Blackboard Model:

Ymir (Thórisson 1999)

CHAMELEON (Brøndsted et al. 1998, 2001)

Smartkom (Bühler et al. 2002, Wahlster et al. 2001, SmartKom 2004)

DARBS (Nolle et al. 2001)

DARPA Galaxy Communicator (Bayer et al. 2001)

Psyclone (Psyclone 2004)

Spoken Image/SONAS (Ó Nualláin et al. 1994, Ó Nualláin & Smith 1994, Kelleher et al. 2000)


Intelligent Multimedia Distributed Platforms

Non-blackboard Model:

WAXHOLM (Carlson et al. 1996)

AESOPWORLD (Okada 1996)

COLLAGEN (Rich et al. 1997)

INTERACT (Waibel et al. 1996)

Oxygen (Oxygen 2004)

EMBASSI (Kirste 2001, EMBASSI 2004)

MIAMM (MIAMM 2004)


CHAMELEON

Language & vision integration system consists of ten modules, mostly programmed in C and

C++ DACS communication system used for communication Blackboard stores semantic representations produced

by other modules Communication between modules achieved by

exchanging semantic representations between themselves or blackboard

Semantic representation in form of input, output and integration frames


Architecture of CHAMELEON


SmartKom

User adaptive interface for human-computer interaction

Mobile Public Home/Office

Facilitates speech, gestures and facial expression input

XML-based mark-up language, M3L, used for semantic representation

Distributed multiple blackboard model


Architecture of SmartKom


Dialogue Manager Acts as a blackboard module Facilitates communication between other modules Synchronisation

Semantic Representation Database Provides semantic representation of language and

vision data

Decision Making Module AI technique for a unique form of decision-making

Bayesian Networks (CPNs) Neural Networks, Genetic Algorithms, Fuzzy Logic

Project proposal


Architecture of MediaHub


Comparison of Intelligent MultiMedia Platforms


Software Analysis Main Programming Language

Java C++

Semantic Representation XML XHTML + Voice SMIL RDF Schema MPEG-7

Decision Making HUGIN (Bayesian Networks) (Hugin 2004) FuzzyJ Toolkit (Fuzzy Logic) (NRC 2004)


Project Schedule


Conclusion An intelligent multimodal distributed platform hub called MediaHub

will be developed

MediaHub will interpret and generate semantic representations of multimodal input and output

MediaHub will perform fusion and synchronisation of language and vision data

Unique contribution of MediaHub is to provide a new method of decision making

MediaHub will be tested within an existing multimodal platform (e.g. CONFUCIUS)

MediaHub: An Intelligent MultiMedia Distributed Platform Hub

Documents