Cross-Media Information Systems - Quo Vadis?

2 December 2005

Cross-Media Information Systems - Quo Vadis?

Prof. Beat Signer

http://www.beatsigner.com

Web & Information Systems Engineering Lab (WISE)

Department of Computer Science

Vrije Universiteit Brussel

http://www.beatsigner.com/

01001000100 0101001100010010100010010 0100100010010001000100001001000 100010001100001001000110010101010110 1100010010100010010010010100010101011001 101001100001100 101010100 00101000100 110101010101 00100 01010101000 1001 1111111111 111 0000000010 0 11111111 0101001011 010101010 111010101 001010010110

Fluid Cross-Media Information Spaces

Prof. Dr. Beat Signer Interactive Paper, Cross-Media

Information Architectures

Dr. Bruno Dumas Multimodal Interaction, Cross-

Media Information Architectures

Lode Hoste Multimodal Interaction, Midas

and Mudra Framework

Brecht De Rooms Implicit Context-aware Human-

Computer Interaction, HCII

CISA

Human-Machine &

Human-Information

Interaction

Information

Systems &

Management

Information

Visualisation

& Navigation

WEB & INFORMATION SYSTEMS ENGINEERING

CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA)

Sandra Trullemans Personal Cross-Media

Information Management

Reinout Roels MindXpres Next Generation

Presentation Tool

Ahmed A.O. Tayeh Open Cross-Media Authoring,

Fluid Document Formats

WEB & INFORMATION SYSTEMS ENGINEERING

CISA

Human-Machine &

Human-Information

Interaction

Information

Systems &

Management

Information

Visualisation

& Navigation

CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA)

Beat Signer - Department of Computer Science - [email protected]

November 2, 2012

PARC User Interface (Mid 70s)

Desktop metaphor

WIMP interaction

"filing cabinets" with

hierarchical folders

Paper paradigm

"What You See Is What

You Get" (WYSIWYG)

what about richer

document formats that

existed at that time?

Xerox Star 8010 [http://www.digibarn.com/collections/systems/xerox-8010/]


November 2, 2012

Digital Documents as a Paper Simulator?

Vannevar Bush

Ted Nelson

Most people don't understand the logic of the concept: "What You See Is What You Get" is based on printing the document out ("get" means "get WHEN YOU PRINT IT OUT"). And that means a metaphysical shift: a document can only consist of what can be printed! [...] No overlays [...] – PAPER UNDER GLASS.

When data of any sort are placed in storage, they are filed alphabetically or numerically, and information is found (when it is) by tracing it down from subclass to subclass. It can be in only one place, unless duplicates are used [...] The human mind does not work that way. It operates by association.

As We May Think, Atlantic Monthly, July 1945

Geeks Bearing Gifts: How the Computer World Got This Way, Mindful Press 2009


November 2, 2012

What is Wrong with Digital Documents?

Existing document formats are based on the simulation

of paper affordances on desktop computers

How to manage mixed-media "documents" in open and

fluid cross-media information spaces? on the data level

- context-sensitive adaptation

- cross-media transclusion

on the visualisation and navigation level

- zoomable user interfaces

on the cross-media interaction level

- fluid multimodal cross-media interfaces

Remediation of the "paper simulation" approach WYSIWYG is only one out of many options!


November 2, 2012

RSL Metamodel and iServer


November 2, 2012

Links

Users

Layers Plug-ins

Structure

Active

Content


November 2, 2012

Bridging the Paper-Digital Divide


November 2, 2012

Interactive Paper Applications

Enhanced Reading

interactive paper maps

interactive festival brochures

educational materials

auction brochures

scientific publications

…

Enhanced Writing

photo album

researcher's notebook

mammography annotation

query by sketching

…

Paper-based Interfaces

PaperPoint

document proof-editing

digital libraries

video analysis tool

educational games

Art Installations

Lost Cosmonaut

Generosa Enterprise

Interactive Tabletops

photo browser

collaborative sketch tool


November 2, 2012

Interactive Festival Guide (EdFest)


November 2, 2012

The Lost Cosmonaut

Writing as a collaborative

act of memory and story-

telling

Reading, writing and

narrative as an act of

making sense

Handwriting as a tool for

human-computer

interaction

In collaboration with Axel Vogelsang,

Artists in Labs Programme


November 2, 2012

ArtVis


November 2, 2012

ArtVis ...


November 2, 2012

ArtVis ...

Advanced visualisation

techniques in combination

with a TUI

explore Web Gallery of Art

faceted browsing

Phidgets-based TUI

RFID-tagged physical

objects

Three main components

to explore, analyse and

browse the information


November 2, 2012

RSL-based Associative File System

signer

My Pictures

Vancouver ER 2010

Granville Island

Vancouver Harbour

Public Market

Keynote J. Thorp

Olga and Lamia

Dinner ER 2010

multiple classification

structural link

navigational link

My Videos

Vancouver Aquarium

ER 2010 Keynote

cross-media association

My Presentations

ER 2010 WISE 2009

cross-media

transclusion

single slide

transclusion

(via OOXML

RSL selector)


November 2, 2012

RSL-based Associative File System ...


November 2, 2012

What is Wrong with Slideware?

Simulation of physical slides limited space due to the slide concept

linear navigation from slide to slide

difficult to reuse content and embed rich media types

MindXpres addresses these issues unlimited canvas with zoomable user interface

non-linear navigation and associative linking

content-based approach with automatic visualisation


November 2, 2012

MindXpres Presentation Tool

HTML5-based extensible

core with three types of

JavaScript plug-ins

components

- source code, graph, video, ...

containers

- slide, image gallery, ...

structures

- grid, graph, clustered, ...

Presentation as RSL-

based overlay structure

enables navigation across

different presentations


November 2, 2012

MindXpres Presentation Tool ...


November 2, 2012

MindXpres Presentation Tool ...

Functional MindXpres prototype

Ongoing work visual authoring tool

- alternative to XML-based authoring

rich media plug-ins

- videos with subtitles, annotations etc.

- statistical data with alternative visualisations

- live sensor data

- ...

audience-driven presentations

collaborative annotations and information sharing

multimodal interaction (e.g. integration of PaperPoint concepts)

import of existing slideware formats


November 2, 2012

Fluid Multimodal Cross-Media Interfaces

Investigation of mobile and distributed document formats Mobile Cross-Media Applications (MobiCraNT)

Passive or active information units (crosslets) processing and storage of sensor data

binding of specific multimodal interactions to resources

Implicit context-aware human-computer interaction (HCII)

Personal Cross-Media Information Management


November 2, 2012

Midas/Mudra Framework declarative definition of multimodal

and multi-touch interactions

rule-based language approach

rapid protoyping and application development

iGesture Workbench create and test gesture sets and algorithms

different modalities: digital pen, Wii remote, …

open source (www.igesture.org)

SpeeG multimodal speech- and body gesture-based

text input

real-time correction of recognised voice input

Multimodal and Multi-Touch Interaction


November 2, 2012

Open Cross-Media Linking

Four existing classes of cross-media link systems

A: single source and single target type

B: single source and multiple target types

C: multiple source and single target type

D: multiple source and multiple target types

Our proposed open cross-media link approach covers the whole space

# source types1 2 3 4 5 ...

1

2

3

4

5

...

# target types

B1

B2

D2

open cross-media

linking

D1

A1... An C1 C2 C3


November 2, 2012

Open Cross-Media Authoring

Cross-Media

Link Service

Client Application

Cross-Media

Browser & Editor

Visual Plug-ins

Resource Plug-in Repository

Data Plug-ins

Visual Plug-ins Data Plug-insVisual Plug-ins


November 2, 2012

Open Cross-Media Authoring ...

Integration of a new resource type includes development of data and visual plug-ins

registration with the resource plug-in repository

Cross-media browser loads plug-ins on demand internal or external (third party application) visualisation

Third party applications delegate the visualisation of a

link target to the cross-media browser similar to "integration" of third-party applications in web browsers

Non-monolithic link model and authoring tool resource-specific plug-ins


November 2, 2012

Open Cross-Media Browser and Editor

Cross-media browser & editor Client application

showEntity(), getSelectedEntity()

showEntity()

C

R

U

D

R

S

L

Menu

Source Window Target Window


November 2, 2012

Visual Plug-ins

Cross-media browser deals only with RSL-specific data

but not with any media-specific details create/delete links (CRUD), user management, …

Definition of link source and target anchors (resources or

selectors) handled by visual plug-ins separation of concerns

Two basic purposes rendering of specific resource types (resources and selectors)

creation and deletion of resources and selectors

Well-defined visual plug-in API


November 2, 2012

Conclusions

Time to question existing document formats which are

based on the simulation of paper on desktop computers

RSL-based representation of open and fluid cross-media

information spaces interactive paper solutions

associative file system

MindXpres presentation tool

crosslet-based fluid multimodal interfaces

open cross-media information browser and editor

Interlinked structural components rather than a

document-centric representation of information data no longer "owned" by applications


November 2, 2012

References

Beat Signer, Fundamental Concepts for

Interactive Paper and Cross-Media Information

Spaces, ISBN 978-3-8370-2713-6, May 2008

Vannevar Bush, As We May Think, Atlanic Monthly,

July 1945

Beat Signer and Moira C. Norrie, As We May Link: A

General Metamodel for Hypermedia Systems,

Proceedings of ER 2007, Auckland, New Zealand,

November 2007


November 2, 2012

References ...

Beat Signer, What is Wrong with Digital

Documents? A Conceptual Model for Structural Cross-

Media Content Composition and Reuse, Proceedings of

ER 2010, Vancouver, Canada, November 2010

Moira C. Norrie, Beat Signer, Michael Grossniklaus, Rudi

Belotti, Corsin Decurtins and Nadir Weibel, Context-

Aware Platform for Mobile Data Management, Journal on

Wireless Networks (WINET), Vol. 13, No. 6, Springer,

December 2007

ArtVis http://wise.vub.ac.be/content/artvis-exploring-information-through-advanced-visualisation-techniques


November 2, 2012

References ...

Axel Vogelsang and Beat Signer, The Lost

Cosmonaut: An Interactive Narrative Environment on

Basis of Digitally Enhanced Paper, Proceedings of

International Conference on Virtual Storytelling,

Strasbourg, France, December 2005

Christophe Scholiers, Lode Hoste, Beat Signer and

Wolfgang De Meuter, Midas: A Declarative Multi-Touch

Interaction Framework, Proceedings of TEI 2011,

Funchal, Portugal, January 2011


November 2, 2012

References ...

Lode Hoste, Bruno Dumas and Beat Signer,

Mudra: A Unified Multimodal Interaction Framework,

Proceedings of ICMI 2011, Alicante, Spain, November

2011

Beat Signer, Ueli Kurmann and Moira C. Norrie,

iGesture: A General Gesture Recognition Framework,

Proceedings of ICDAR 2007, Curitiba, Brazil, September

2007

Lode Hoste, Bruno Dumas and Beat Signer, SpeeG: A

Multimodal Speech- and Gesture-based Text Input

Solution, Proceedings of AVI 2012, Capri Island, Italy,

May 2012


November 2, 2012

References ...

Beat Signer and Moira C. Norrie, A Model and

Architecture for Open Cross-Media Annotation and Link

Services, Information Systems 36(3), Elsevier, May 2011

Cross-Media Information Systems - Quo Vadis?

Science

media information

mindxpres

fluid multimodal

visual plug

media authoring

machine amp

web amp

multiple source