CS 586 – Distributed Multimedia Information Management

CS 586 – Distributed Multimedia Information Management

Prof. Dennis McLeod

About the paper . . .

Towards Ontology-Driven Discourse: From Semantic Graphs to Multimedia PresentationsIn Proceedings of the 2nd International Semantic Web Conference, 2003

By: Joost Geurts, Stefano Bocconi, Jacco van Ossenbruggen, and Lynda Hardman

Presented by: Gabriela Gutierrez, February 11, 2004

Overview

• Introduction

• Example Scenario: Rembrandt

• Process 1: From Semantic Graph to Structured Progression

• Process 2: From Structured Progression to Multimedia Presentation

• Conclusion

Introduction

• Traditionally– Applying Semantic Web technology to multimedia

information systems focuses on using annotations and ontologies to improve retrieval process

– Presentation of data is “detail” best left to CSS or XSLT style sheets

• In this paper– Claim that importance and complexity of effective

presentation design is grossly underestimated – Concentration on improving the presentation of the

retrieval results

Introduction

• Human professional designers must understand:– Underlying semantics of the client’s

information– Most effective order, grouping and priorities for

structuring this information– Most effective means of using the chosen

medium to convey the information

Introduction

• Information presentation design is a knowledge-driven process. It requires:– Sufficient knowledge about domain– Knowledge on ordering, grouping and

prioritizing information– Knowledge about media design

• Selection of most appropriate medium• Understanding of medium characteristics in order to

choose an effective means to achieve the communication goal

Introduction

• Problem: – Professional designers can only design data-driven web

sites if the underlying data, its semantics and target audience are relatively homogeneous.

– Variety of data sources, semantic relations, output devices, and user profiles forces content providers to adopt one-size-fits-all approach.

• Automation is needed in order to make the presentation of information knowledge-driven.

Introduction

• Assumptions:– Multimedia items are properly annotated– Annotations represent domain relations in a semantic

graph (e.g. RDF)– Graph has associated Domain ontology– There is a Discourse ontology containing information

about different document genres and building blocks for creating documents for each genre

– There is a Design ontology containing media design knowledge

Introduction

Example Scenario: Rembrandt

• Web query: “life and work of Rembrandt”• User-selected type of structured progression:

disc:Biography• User-selected output medium: non-interactive

multimedia presentation• Semantic graph = retrieval component’s results +

domain ontology semantics relations• Structured progression = typical facts (name,

DOB,…) + career facts + personal life info

Process 1: From Semantic Graph to Structured Progression

• CSS and XSLT operate purely on the XML level of RDF’s serialization syntax w/o any understanding or support for semantics of RDF data model

• Transformation process needs access to knowledge on RDF Schema level– For querying underlying domain ontology– For access to its own operating knowledge

Process 1: From Semantic Graph to Structured Progression

• Several transformations prototyped in Java and Prolog environments

• Direct access to a Sesame RDF Schema-based repository

• Can use any query language supported by Sesame (RQL, RDQL, SeRQL) to gain direct access on the RDF instance level and the RDF Schema Level

• Transformation process uses (declarative) domain and discourse-specific knowledge, while (procedural) transformation code remains generic

Process 1: From Semantic Graph to Structured Progression

• Transformation code uses RQL query to retrieve classes that Rembrandt instance belongs to . . . dom:Artist

• Discourse ontology defines instance of disc:ArtistBiography that has disc:Subject property with value dom:Artist

• Structured progressions have a disc:narrativeUnits property that specifies the disc:NarrativeUnits that can be used to construct it (e.g. disc:PersonalData, disc:PrivateLife and disc:Career)

Process 1: From Semantic Graph to Structured Progression

• Narrative Units have associated rules used to select matching content

• Example: disc:PrivateLife– Rules to select information about family relations from semantic

graph– Graph includes relation dom:isMarried between Rembrandt and

Saskia_Uylenburgh– Rule #3 in following table can use domain relation to select Saskia

in the disc:Role of disc:Spouse

• Rules can be applied recursively– Rule #3 specifies that PrivateLife is the narrative unit that can be

used for a subsequent nested story line– Process continues until no more rules can be applied or a rule

specifies that no further expansion should happen

Process 1: From Semantic Graph to Structured Progression

• After all rules have been applied:– Biography w/ 3 narrative units– disc:PersonalData (Rembrandt in role of

disc:MainCharacter)– disc:Career (Chiaroscuro in role of

disc:Technique)– Disc:PrivateLife (Saskia_Uylenburgh in role of

disc:Spouse)

Process 2: From Structured Progression to Multimedia Presentation

• Two-step process:1. Structured progression transformed into

Document Structure– Decisions on output medium (e.g. text, interactive

hypermedia, passive multimedia)

2. Document Structure transformed into a tree of formatting objects

– Detailed layout and formatting decisions (e.g. timing of presentation, transition effects)

• Advantage:– Mapping discourse-specific narrative units to more general

document elements allows for more commonly applicable formatting rules (e.g. disc:PrivateLife can be mapped to document section element, relying on common formatting rules for section-level elements)

• Disadvantage:– There is always a level that can no longer be specified in

terms of document structure (e.g. a figure w/ too much detail)– Solution: detailed structures copied directly into document

structure in step 1 in order to define specific rules in step 2 to deal w/ formatting directly

Process 2: From Structured Progression to Multimedia Presentation

From Structured Progression to Document Structure

Conveying Document Structure• Transforming a document structure into presentation

constructs uses Cuypers library– Uses constraint solving techniques to verify that a presentation

construct conforms to delivery-context constraints (e.g. screen size)– Allows alternative formatting specification if constraints are

violated

• A rule that transforms a document structure into presentation construct has 2 discourse parameters:– disc:NarrativeType– disc:Role– Parameters allow system to adapt formatting of presentation to

convey message more effectively

Conveying Discourse Semantics Directly

• Depending on their function, we need to define formatting for different media types– Rembrandt self-portrait (disc:Portrait in

disc:PersonalData vs. disc:Painting illustrating Chiarocuro)

• Awareness of impact of different media modalities• Fall-back rules

– Image not identified as either disc:Portrait or disc:Painting should be applied generic formatting for images since mm:Painting and mm:Portrait are subclasses of mm:Image

Conclusion

• Only short presentations have been generated to date, based on restricted domain ontology

• Focus has been on single discourse structure (biography) and single document structure (multimedia presentation)

• Additional research required to scale the system to more realistic scenarios

• Under investigation: how knowledge about the user interacts w/ discourse and design knowledge used in current prototype