Event Web

1ICIG Xian Sept 21, 2009 © Ramesh Jain

EventWeb

Ramesh Jainand

Several Collaborators

Contact: [email protected]


Today

• Data, Experience, and Knowledge

• Semantic Gap• Bridging the Semantic Gap

• EventWeb• Applications:

– EMME: Experiential Media Management Environment– E2E: Environment to Environment Connection


Name 3 most popular companies that emerged in the last 3 years.

• YouTube • Facebook• Twitter


Twitter’s Growth

This is a growth pattern I have never seen in terms of speed - not in the nearly 25 years I've been watching this industry. --- John Battelle


Name 3 most popular Internet concepts in the last 3 years.

• Social Networks• Micro-blogging (Ambient Awareness)• Tags

News: None of these come from GYM or Academia.


Interesting!!!

• Social Networks

• Microblogging• Tags

• YouTube

• Facebook• Twitter

What does this tell us?


Messages for Research Community

• New media: Text based media is not enough.

• Experiences: People want to experience and share experiences – with minimal latency.

• Socialize: Family and friends remain a strong influence in all facets of life.– Family and friends are closer to each other today than

ever!!!

People want:


ExperiencesLife =Life =Life =Life =

EventsEventsEventsEvents

+


Recording experiences

• Visual• Aural• Tactile• (not worry about smell and taste for the

next few years)• Text• Log of activities




The Challenge

Connecting


Bits and Bytes

Alphanumeric Characters

Lists, Arrays, Documents, Images …

Transformations


Semantic Gap

The semantic gap is the lack of coincidence between the information that one can extract from the visual data and the interpretation that the same data have for a user in a given situation. A linguistic description is almost always contextual, whereas an image may live by itself.

Content-Based Image Retrieval at the End of the Ear ly YearsFound in: IEEE Transactions on Pattern Analysis and Machine IntelligenceArnold Smeulders , et. al., December 2000


Semantic Gap


History: Year 1991

– We had Internet.– Lots of computers were connected to each

other.

– Computers had files on them.– We had GOPHER and other FTP

mechanisms.


Tim Berners-Lee thought:

• Suppose all the information stored on computers everywhere were linked.

• Suppose I could program my computer to create a space in which anything could be linked to anything.

Others – including Bush -- had

that idea earlier but the

technology was not ready.


That resulted in the Web

• DocumentWeb– Each node is a ‘Page’ or a document.

– Pages are linked through explicit referential links


Today we have

• WWW that has revolutionized our life in many ways.

• Sensor networks– Billions of sensor getting connected

• Ambitious projects– Planetary Skin by Cisco and NASA

– Smart Planet by IBM


Can we take the Web beyond the Cyberspace?

• Can things in real world be connected to other things?

• Does this even make sense?


Connectors• My 5 Senses are connectors between

‘me’ and the world.• We use our sensors (vision, audio, …) to

experience the world.• Sensors could be the interface between

the Cyberspace and the Real World.• Sensors are placed for ‘detecting events’.

– How do you decide what sensors to put at any place?

– Would you put a sensor if nothing interesting ever happens at a place?


Modeling the World

• Data (Semantic Web)• Objects (Search Companies, …)• Relationships and Events


Evolution of ‘Thought’

• Objects -- popular in the West.• Relationships and Events – popular in the

East.• Objects and Events – seems to be the new

trend.

• The Web has reemphasized the importance of every object and event being connected to others -- East Meets West.


Continuing the Evolution of the Web

• Consider a Web in which each node – Is an event

– Has informational as well as experiential data– Is connected to other nodes using

• Referential links• Structural links• Relational links• Causal links

– Explicit links can be created by anybody

• This EventWeb is connected to other Webs.


Different types and Granularities• Conferences

– Days• Sessions

– Talks» Purpose of the talk

• Wedding• An Earthquake• The Big Bang• 9/11• 26/11• Formation of Google• Attending WWW• Me

– My Birth, – Being here, and – Dying in 100 years.


Event Representation


Bits and Bytes

Alphanumeric, Pixel, Characters

Text, Images, Audio, Video, Tactile…

Transformations


Tim

e1- dimensional Space

Events


Tim


EventWeb


Multimedia Storytelling• Collect information about events

– Select relevant events

– For each event, select appropriate information– In right media

• Stories are sequence of coherent events.– Stories/Novels

– Drama– Movies

Present right event information using right media in right order.


Tim


Multimedia Storytelling

Text

Video

Photo


Stages in EventWeb• Cyberspace: Every event is ‘created’

in cyberspace and exists there.• Physical and Cyberspace: Events in

real space are also linked. We have Sentient EventWeb linked with the Cyber Web.


Experiential Media ManagementEnvironment

• Event-based• Should be able to deal with ‘multimedia’

– Photos– Audio– Video– Text– Information and data– …

• Searching based on events and media.• Storytelling

First Photos then other media.


ContenxtContent Context

• Contenxt = Content + Context

• Context is as powerful, possibly more, as content in understanding audio-visual information


Modern Cameras• Are more than ‘Camera Obscura’: They capture an

event.• Many sensors capture scene context and store it

along with intensity values.

• EXIF data is all metadata related to the Event.

Exposure TimeAperture DiameterFlashMetering ModeISO RatingsFocal Length

TimeLocation (soon)Face


Photos can be tagged using only EXIF

We will also use content features and LSCOM Concepts – will soon start using them.

Information from calendars and other sources will be introduced soon.


EMME Event Cycle

Event Base

Event Presentation/ Navigation

Event Grouping, Linking, Assimilation

Atomic Event Entry

EXIFEXIFEXIFEXIF

Features

Tags/

Context


Using Context/Models to Build the EventWeb

• Folder structure• Calendar• Social Network• EXIF Data• Event Ontology• Personal annotations

• Photostream Segmentation• Event Detection from photos


Photo Stream Segmentation

Definition: given a photo stream }{ ipP =

Segment

Event 1 Event 2 Event 3


EMME Event Cycle

Event Base



Atomic Event Entry

EXIF

Features

Tags/

Context

Photo

stream

Segment.

Event

Ontology

User

Annot-

ations

Minimize Manual Work


EMME Event Cycle

Event Base



Atomic Event Entry

EXIF

Features

Tags/

Context

Photo

stream

Segment.Event

Ontology

User

Annot-

ations

Story

Telling

Search

Explore


Using EMME• Searching for photo

– ISWC2008 • Creating Albums:

– Professional – Family– Tourism

• Telling stories– What did I do in Karlsruhe

• Scenario: In December 2008, I have 20,000 pictures taken in 2008. How do I (semi-automatically) select 25 to send to– My mother– The uncle that I hate– My personal friend– My professional friend– …

Version 0.1

is ready


Personal-Photo-EventWeb


Singapore – Outdoor --People


People-No Face - Outdoor


Singapore-Outdoor-Night


Indoor no faces


Indoor People


Indoor Portraits


E-mail can also be parsed.


Photos Summarization


Original Data Set


Photo-Summarization using content


Photo-Summarization using Faces


Event IngestorEvent Ingestor

Atomic Events

Event Mash-up InterfaceEvent Mash-up Interface

Composite Events

EXIF Extractor Text Parser Calendar Parser

Event DigestorEvent DigestorClustering Annotation Event Ontology




Sentient EventWeb

• Use sensors to create events.– This conference

• Infrastructure to create and link all such events.– Index

– Search– Present

• Capture events and put them on the Web.

Imagine each sensor tweeting.


Sentient EventWeb

• Use sensors to create events.– This Seminar

• Infrastructure to create and link all such events.– Index

– Search– Present

• Archive events and put them on the Web.


First Version: E2E

• Person to Person• Device to Device

– Telephone– Video Conferencing

• Environment to Environment (E2E)

Communication Modes:


Environment

• Fixed objects• Active Objects• Events and Activities

• Sensor to capture Objects and Events– Cameras, microphones, pressure, RFID, …

• Displays and Actuators– Projectors, Displays, Speakers, …


Key Idea

• Put enough sensors and devices to make mediation disappear.

• System detects events and objects and sends appropriate data streams and displays/renders incoming streams on appropriate devices

• Bring back Person to Person communication


Vision

• First build Communication Nodes• Next build Web Nodes• Build Sentient EventWeb


E2E communication: Vision

Sentient Information

System

Sentient Information

System

Environment 1 Environment 2

WebWeb

Device to Device communication


Network/Transmiss

ion

Environment Model

Environment Server

Situation based

controller

Actuator / Presentation

Model

MMDB

Sensors

Actuators / Presentation

Devices

Physical Environment

EventBase

Environment: Node Architecture


Event detection and organization


Env. 1

Env. 4

JSM 2

JSM 1

JSM 1

Env. 5Env. 3

Env. 2Joint SM

Shared Visualization Spaces for Environment to Environment Communication , in Workshop on Media, Arts, Science and Technology (MAST 09), 2009.


Telemedicine ApplicationTelemedicine ApplicationTelemedicine ApplicationTelemedicine Application


Conclusion

• New models for Events and Objects are required for bridging the semantic gap.

• Semantics emerges due to relationships captured in Web.

• EventWeb facilitates emerging applications dealing with multimedia data.


Thanks for your time and attention.

For questions: [email protected]