EMM (Embedded Media Marker)

Embedded Media Markers: Marks on Paper that Signify Associated Media

Qiong Liu, Chunyuan Liao, Lynn Wilcox, Anthony Dunnigan, Bee Liew

FX Palo Alto Laboratory, Palo Alto, CA, USA

Motivation & Vision• Motivation: Combine advantages of

paper and cell phones

• Vision: Use Embedded Media Markers (EMMs) to link multimedia to paper patch •EMMs are meaningful marks on paper•Use visual features inside EMM to identify document patch

Slide 2

User takes a picture of a document patch

Multimedia and digital content displayed on cell phone

What is EMM• Embedded Media Markers (EMM) are optical-filter-like overlaid marks printed on paper documents.

• Printed marks (EMMs) that improve usability and recognition accuracy

• Computationally efficient features (FIT) that identify document patches

Slide 3

EMM-signified document patch

EMM-signified document location

Media type (video)

Slide 4

EMM Patch Identification•Patch identification is based on image local feature (FIT) • Similar to SIFT features. More efficient.• Robust to scaling, rotation and partial occlusion

•FIT features work on a broad range of document types• Western/Asian text, images, photos

EMM Operation Flowchart

Take a snapshotof EMM

Match feature points against the document database

Compute feature points

Find the matched document patch

Retrieve the linked videoPlay the video on

the phone

• Marker-based Technologies• Recognizable Visual Markers (e.g.

Barcode, QR-Code)• Nearly Invisible Fiduciary Markers (e.g.

DataGlyphs, Anoto)

• Content-based Technologies• Text-words-based Features (e.g. BWC)• Image-based Features (e.g. SIFT, FIT)

• EMM is a hybrid approach• Provide users multimedia cues and

interaction guidance• Save computation and

improve performance• Minimize the interference with

document content

Document Identification Methods

Barcode QR-code

Data Glyphs

Anoto Pattern

BWC

EMM

SIFT

EMM Design Guidelines• User Friendly – guide

users for media finding and query.

• Document Friendly –minimize changes to document content and format.

• Machine Friendly – save computation resources for document identification process.Major Entities

EMM Components’ Functions

• Feature boundary mark – framing feature capture

• Media type mark

• Anchor point – associate media with an accurate location.

• Graphical effects – attract attention and help separate content and marker.

feature-boundary-circle

media-type-icon

Anchor point

Top View of a 3-level EMM Model

•Semi-automatically arrange EMM based on feature distribution• Criterion 1 – Minimize author’s effort.

Author only selects an EMM anchor point.

• Criterion 2 – Minimize machine resources used for patch query. Find a small feature boundary, and index the small number of keypoints inside the feature boundary.

• Criterion 3 – Minimize the EMM interference to document. Minimize the number of keypoints overlaid by an EMM.

EMM Authoring Tool

Parameters for EMM Placement

•Anchor point location (m,n) – provided by a user

•Feature-boundary-circle center (X,Y)

•Feature-boundary-circle radius R

•Media-type-circle center (x,y)

Parameter Optimization• EMM boundary optimization• Find the center (X,Y) and the minimum radius R such that

• The number of keypoints in the circle exceeds the threshold to ensure patch identification accuracy

• The user selected anchor point is contained in the boundary

• Media type icon placement• Select the media-type-icon center (x,y) so that the icon covers the minimum number of keypoints

•Counting the number of keypoints in a circle is time consuming • O(2N). N is the number of

keypoints in a page.•Speedup

• Inscribed square to estimate the number of keypoint in a circle

• Use the cumulative feature-point histogram

• IABCD =IC-IB-ID+IA

•Optimization of media type circle is similar

Parameter Optimization Speedup

A B

CD

A

C

B

D

IPP

Boundary Circle

Video

Slide 13http://www.youtube.com/watch?v=oub-2b6wQIY

EMM Evaluation in Real Life

• 2188 EMMs in the database• At randomly selected 2188 document locations of2188

pages• 110 EMM Overlaid Page Printouts

• 90 1-up + 20 4-up on letter-size paper• 110 Queries

• Either in an office or open patio w/natural lighting conditions

The Missed One• Include many less

distinctive keypoints

• Correctly identified after position adjustment

Slide 15

Related Work

Ricoh “HotPaper”Link personal media to text.Use BWC (Brick Wall Coding)Only works on Western textACM Multimedia 08

University of Oldenberg “Bookmarkr”Link photobook picture to digital photo.Use SIFT featuresACM Multimedia 08

Document ID Technology Comparison

Marker-based Technologies Content-based Technologies Hybrid Technology

TypeVisual Markers

Fiduciary Markers

Text-based Features

Image-based Features

EMMs

ExampleBar Code, QR Code, SP Code

Dataglyph, Anoto, MIST, MIG

Ricoh BWC SIFT, FITMarker overlay plus FIT

Capture Guidance

Good (no indication of media type)

NoneFair (Scan to find patches with media)

None

Excellent (shows existence, accurate association location, and type of media)

MachineReadability

Excellent ExcellentFair (depends on content)

Fair (depends on content)

Good (optimized for content)

DocumentFidelity

Poor (visually obtrusive marks need extra space)

Fair (slight modifications of document appearance)

Excellent ExcellentGood (nearly transparent and attractive markers)

Technology Comparison - cont

Marker-based TechnologiesContent-based Technologies

Hybrid Technology

Type Visual MarkersFiduciary Markers

Text-based Features

Image-based Features

EMMs

ExampleBar Code, QR Code, SP Code

Dataglyph, Anoto, MIST, MIG

Ricoh BWC SIFT, FITMarker overlay plus FIT

Document Restrictions

None NoneYes (Western Text only)

Some (depends on content)

Some (depends on content)

Marking Resolution

Low (Page)High (Patch, Pixel-level)

Medium (Block of ~100 words)

High (Patch, Pixel)

High (Patch, Pixel)

Scan/print Restrictions

None* (Regular printer and camera)

Yes (High-res printer and scanner)

None (Regular printer and camera)

None (Regular printer and camera)

None (Regular printer and camera)

Accuracy Control

Full Full None None Partial

* SP-code requires high-res printer and scanner

Summary

• EMMs are nearly transparent iconic marks printed on paper document to link various multimedia to specific document locations

• EMMs are robust to scaling, rotation and partial occlusion, and applicable to a broad range of document types

• Unlike barcodes, EMMs do not change document layouts and are less visually obtrusive

• Unlike other feature-based approaches, EMMs explicitly signify the media-augmented document regions and guide user interaction

• EMM can improve the efficiency of computation

Thank You