SAMT09 - Web of Data Tutorial - Part 2

1 SAMT 2009 – Tutorial Web of Data in the Context of Multimedia (WoDMM) Graz, Austria - 2 Dec 2009

Web of Data in the Context of Multimedia Part 2: Linked Multimedia

Bernhard Haslhofer, Bernhard Schandl, Andreas Langegger,

Wolfgang Halb, Tobias Bürger


Agenda   Multimedia Interlinking: Concept + Vision

  Multimedia Annotations

  Enriching Personal Media Collections


Multimedia Interlinking   Concept and Vision of a Web of Video Data

  Applying the Linked Data principles to multimedia content

  W3C Video on the Web activity   Fragment Identification

  Media Annotation


Vision of a Web of Video Data   Aim: Make video an integral part of the Web in which

users can seamlessly consume and create videos

  Requirements:   Deep Linking: timed URIs

  Metadata: Time-aligned annotations & tags

  Search-ability: Searching media using text

  Dynamic composition of video mashups

  Discoverability: Hyperlinking from inside media to other Web resources

  Share-ability: Integrated handling of metadata with the content   Personalisation: Dynamic composition of media web content

cf. Silvia Pfeiffer “Architecture of a Video Web - Experience with Annodex” W3C Video on the Web workshop, 13.12.2008


Why?

Source: Cisco's Global Consumer Internet Traffic Forecast, Cisco, 2009


Example 1: Where is Danny?


Example 2: Who is Jack?   Region-based annotation in Flickr

  Cannot be taken outside of Flickr

  Can only be accessed via the Flickr API


Example 3: Complex Query 1.  show me photos of actors participating in the

SHINING movie visiting a country in Europe   LinkedMovieDB: list of actors participating in SHINING-: [L-

ACT]   Geonames: list European countries -: [L-EC]   Google: list photos taken in a country of [L-EC] -: [L-ECP]   Google: in [L-ECP] find regions that depict members of [L-

ACT] -: result

2.  give me a summary of all scenes from arbitraty videos where actors talk to a film producer


Linked Data: Current State of Practice

  Linked Data currently mostly applied for textual resources (cf. LOD cloud)

  If any media resources are interlinked then on a global level (cf. BBC)

  Some communities demand for methods to link on a more fine grained level (cineasts, biologists, etc.) [Schroeter and Hunter, 2007]


Problems   How to address and retrieve parts of multimedia

content (e.g., a region of an image, sequence of a video)

  How to describe parts of multimedia content in the Web of Data?

cf. M. Hausenblas, R. Troncy, T. Bürger, and Y. Raimond “How to apply Linked Data principles to multimedia fragments” Proceedings of LDOW 2009, 2009.


Interlinking Multimedia (iM) 1.  Apply linked data principles for fine-grained

identification and description of multimedia fragments -> W3C Media Fragments WG

2.  Describe multimedia content -> W3C Media Annotations WG

3.  Deploy legacy multimedia metadata formats on the Web of Data -> ramm.x

4.  Develop specialized interlinking methods for multimedia

  User-contributed, game-based interlinking

  Collaborative interlinking, cf. MetaVidWiki

  (Semi-)automatic methods: Multimedia analysis


Media Fragments WG Goal

Provide URI-based mechanisms for uniquely identifying fragments for media objects on the Web, such as video, audio, and images.

Credits: Slides about Media Fragments WG by the Media Fragments WG.


Media Fragments: Use Case and Requirements

  Document: http://www.w3.org/2008/WebVideo/Fragments/wiki/Use_Cases_&_Requirements_Draft   Link to, Display, Bookmark, Annotate, Search, etc. media

fragments

  Type of fragments: four dimensions considered   time ([point?] interval), space ([point?] rectangle), track and id

(named fragment)

  Protocols covered: HTTP(S), RTSP


Technological Survey   Media Fragments URI already present on the Web ... but with

very different syntax and behaviour   Google Video: (#50m16s)

  YouTube: (#t=1m45s)

  Archive.org powered by TemporalURI: (?t=74.5 )

  Videosurf: (?t=140&e=184)

  Microsoft IIS 7.0: (Fragments(video=20000000))

  Other time-clipping and spatial clipping methods:   MPEG-21: time, space, track, id (mp(/CD/track/~time(%27npt

%27,%270%27,%2760%27))

  SVG: space (svgView(14.64,15.73,146.98,147.48))

  SMIL: time (clipBegin, clipEnd), space (panZoom, area), media markers

  MPEG-7: time, space, modality


Media Fragments 1.0   Syntax

  http://www.example.com/movie.mov#t=12.33,21.16

  http://www.example.com/movie.mov#t=smpte-25:00:12:33:06,00:21:16:00

  http://www.example.com/movie.mov#xywh=20,20,40,40

  http://www.example.com/movie.mov#xywh=pixel:20,20,40,40

  http://www.example.com/movie.mov#track='audio1'

  http://www.example.com/movie.mov#id='the%20kiss%20scene'

  http://www.example.com/movie.mov#track='audio2'&t=12.33,21.16


Media Annotation WG: Purpose


Media Annotations WG: Core Ontology

  Goal: to develop a simple lingua franca


Bringing Legacy Metadata to the Web of Data: ramm.x

  Problem: How to make existing multimedia metadata in legacy formats (e.g., MPEG-7) available on the Semantic Web?

  Aim: To deploy existing legacy metadata in HTML documents and by that integrate external descriptions in a RICO by reference.

  Ramm.X uses RDFa, formalisations of metadata standards, and the Ramm.X vocabulary to deploy multimedia metadata.


Interlinking Methods 1.  Automatic Interlinking

2.  Emergent Interlinking

3.  User Contributed Interlinking

4.  Game based Interlinking

Time

Quality

x UCI

# Annotations

x AI

EI

GBI x

x


References   Michael Hausenblas, Raphael Troncy, Tobias Bürger, and Yves

Raimond "Interlinking Multimedia: How to Apply Linked Data Principles to Multimedia Fragments" In: Proceedings of Linked Data on the Web (LDOW2009), co-located with the 18th International World Wide Web Conference (WWW2009), Madrid, Spain, 2009

  WonSuk Lee, Tobias Bürger, Felix Sasaki, Véronique Malaisé, Florian Stegmaier, and Joakim Söderberg (eds.) "Ontology for Media Resource 1.0" W3C Media Annotation Working Group, W3C Working Draft 18 June 2009, 2009.

  Raphaël Troncy, Erik Mannens, Michael Hausenblas, Jack Jansen, Yves Lafon, Silvia Pfeiffer, and Davy van Deursen "Media Fragments URI 1.0" W3C Working Draft 18 November 2009, 2009.

  Silvia Pfeiffer “Architecture of a Video Web - Experience with Annodex” W3C Video on the Web workshop, 13.12.2008, 2008.


What’ll come next   Multimedia annotations: The LEMO framework

  Enriching personal media collections


Web of Data in the Context of Multimedia Part 1: Linked Open Data: Vision, Concepts and Technologies

Bernhard Haslhofer, Bernhard Schandl, Andreas Langegger,

Wolfgang Halb, Tobias Bürger


Agenda   Introduction

  Producing Linked Data

  Existing Data Sets

  Linking Data

  Consuming Linked Data

  Multimedia Interlinking




Many interpretations   An annotation is…

  … a remark, explanation, or interpretation added to the original document

  … a means to make implicit structures explicit

  … a datum created by a third party that can take the form of a written note, a symbol, a drawing, or a multimedia clip

  ...

  There are different interpretation of the term Annotation in different domains



Many solutions


Many solutions


What is LEMO?   A Web-based, multimedia annotation framework

  Consisting of   A semantic backend-end storage

  Front-ends for image, audio, and video annotations

  Online Demo at: http://dme.arcs.ac.at/annotation/econnect-annotation-showcase.html


Driving requirements   Objects to be annotated are multimedia

  Need a uniform data model, no content-type specific solution

  Annotations often address specific content parts (fragements)   Need a uniform fragment identification approach

  Shift towards the Web   Do no lock annotation data in closed data stores but…

  Expose annotation data as Linked Data on the Web (following the Linked Data Principles)


Driving Requirements   Reuse of existing standards

  (Semantic) Web Standards   URI, HTTP, RDF, RDF/S, OWL

  RESTful Web Service Design

  Annotea data model & protocol

  Xpointer, MPEG21, SVG, etc. for fragment identification


Why not use Annotea as it is?   Annotea is for the annotation of Web pages; LEMO

should also support annotation of embedded content items

  Annotea doesn’t assign datatypes to its properties; LEMO uses OWL’s (data)typing features

  Annotea doesn’t support multimedia fragment identification; LEMO uses MPEG-21 (multimedia) fragment identification


A Uniform Annotation Model   annotation profiles; cf. application profiles for

metadata interoperability

  an Annotea-based core model

  with content- and annotation-type specific model extensions

  implemented as light-weight software components (add-ons)


A Uniform Annotation Model


A Uniform Annotation Model


Uniform Fragment Identification   introduce a fragment element with URI range

  follow MPEG-21 media pointer scheme

  an annotation can define multiple fragments

  URI-encoding of fragments   minimal consensus

  content-type specific extensions


Uniform Fragment Identification


Uniform Fragment Identification


Annotations as Web Resources


Annotations as Web Resources


References   The LEMO Annotation Framework (paper)

  http://www.cs.univie.ac.at/upload//550/papers/lemo_framework_final.pdf

  The LEMO Annotation Framework (code)   http://europeanalabs.eu

  Open Annotation Collaboration   http://www.openannotation.org/


Agenda   Introduction

  Producing Linked Data

  Existing Data Sets

  Linking Data

  Consuming Linked Data

  Multimedia Interlinking




Enriching Personal Media Collections


Goal

  to raise problem awareness for personal semantic multimedia

  to introduce the basic concepts of semantic personal information management

  to introduce recent research and tools from the semantic desktop community

  to indicate overlaps and potential for cross-fertilization


The Web

  The World Wide Web is designed for humans, not machines.

  Automatic information processing is hard.

  But at least, there is a common technology infrastructure (URIs, HTTP, HTML, browsers, etc.).

  This is a solid basis to build upon: the Semantic Web and Linked Data reuse and extend the human-centric Web.


The Desktop

 On your desktop, this problem is even worse:

 No unified identification of data items.

 No standardized, simple communication protocol.

 No agreed, open representation format for data.

 No means to store and retrieve semantic annotations.


How does your desktop look like?

  files

  emails

  documents

  contacts

 media

  ...

  at most: fulltext search ... which doesn’t work well for multimedia data


… organize meeting in Belfast …

Invitation E-Mail

Time Schedule

Belfast Weather Meeting Agenda Attendee Dirk


How are data organized?

 Hierarchical structures (aka data silos) for files, emails, bookmarks, applications, ...

  Separate organization metaphors for contacts, calendars, communication, ...

 Multimedia data: media-dependent metadata (look at your music player or your photo manager), locked away in files or in application-specific data bases.

  The walled personal information garden ...



Personal Multimedia

  For the organization of multimedia data, metadata is key.

 How to manage your personal metadata?


The Semantic Desktop

  The vision of the Semantic Desktop is to create an infrastructure for the management of meaningful metadata about resources that covers the entire personal sphere of information.

  In fact, the Semantic Desktop is your personal Semantic Web.


How?

  assign globally unique, immutable URIs to all resources.

  collect / store / index resource descriptions and relations between resources.

  provide uniform access to these descriptions.

  integrate these descriptions into applications and make use of them.

  social semantic desktop: share and communicate personal resources.


Why?

  The Semantic Desktop reflects the user’s Personal Information Model — i.e., the user’s view on their data.

  This allows tools to deliver information when it is actually needed.


… organize meeting in Belfast …

Invitation E-Mail

Time Schedule

Belfast Weather Meeting Agenda Attendee Dirk

Topic

Management CID

Project


What can the Semantic Desktop deliver?

 Contextualized, semantic search   find information more efficiently

  Similarity metrics and recommendations  show the user what they already have in their

data space

 Data analysis and information extractions  understand and interpret what is there


What can the Semantic Desktop deliver?

  Semantic Tagging   relate things and concepts

  Semantic Personal Wikis  useful notetaking and semistructured resource

description

 User Interfaces   faceted search, drag+drop annotations, etc.


Recent Research Projects

 NEPOMUK (EU FP6) [Gro07]  has (amongst others) defined a reference

architecture and ontologies for the representation of personal information

 reference implementation (desktop service), based on Java/Eclipse

 goal: infrastructure for existing OS

 integration in KDE 4

 raised significant public awareness (2 times slashdotted)


Commercial Break

 Results from NEPOMUK and other projects are commercialized, have a look at www.gnowsis.com


Recent Research Projects

 Haystack [Kar07]  has presented a new metaphor for interacting with data

 dynamically assembled user interfaces

 dynamically assembled workflows and data conversions

 basically an OS modeled in RDF


Many more

 DeepaMehta

 DBin

 Chandler

  Semex

  iMeMex

  IRIS

  X-COSIM

  ...


Semantic Desktop Infrastructure

  SemDAV [Sch09]  combines aspects from the Semantic Web and

file systems into a unified data model

 provides storage and query mechanism for semantic files

 provides virtual file system access for file-based applications

 provides an API for “semantic” applications

 provides file browser-like user interaction metaphors




 TripFS [Sch09a]  exposes a file system as Linked Data

 assigns immutable URIs to files and traces file system modifications

 extracts metadata from files and exposes them as RDF

 provides a SPARQL endpoint over the file system

 links files to external data sources (e.g., MusicBrainz, ACM, ...)




 Aperture [Sau06]  a generic framework for extraction of

metadata from digital resources

 provides extractors, crawlers, monitors, ... for different kinds of data sources (e.g., file system, email, web pages, web services, ...)

 can be extended and customized

 open-source



 Nepomuk-KDE  core components from NEPOMUK

(resource identification, triple store, ontologies) have been integrated in KDE 4 and are now part of the standard distribution.

 first implemented application: tags, comments and ratings for files

 much more to follow (e.g., Gwenview image viewer, Bangarang media player,...)








Summary

  The Semantic Desktop is highly relevant for the management of personal media collections — metadata is key!

  It is there! – just use it.

 Questions? [email protected]


References & Credits

  [Gro07] Tudor Groza et al., The NEPOMUK Project – On the Way to the Social Semantic Desktop. Proc. I-Semantics 2007

  [Kar07] David Karger, Haystack: Per-User Information Environments Based on Semistructured Data, in: Beyond the Desktop Metaphor, MIT Press, 2007.

  [Sch09] Bernhard Schandl et al., The Sile Model — A Semantic File System Infrastructure for the Desktop. Proc. ESWC 2009


References & Credits

  [Sch09] Bernhard Schandl, TripFS: Exposing File Systems as Linked Data. Triplification Challenge 2009.

  [Sau06] Leo Sauermann et al., Semantic Desktop 2.0: The Gnowsis Experience. Proc. ISWC 2006

SAMT09 - Web of Data Tutorial - Part 2

Technology

tutorial web of data

web of video data aim

web workshop

multimedia analysis

theweb of data

video web experience

linked data principles

agenda multimedia interlinking