Top Banner
UIST 2004 Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon Kim Steven M. Seitz Maneesh Agrawala University of Washington Microsoft Research
95

UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

Dec 21, 2015

Download

Documents

Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Video-Based Document Tracking:

Unifying Your Physical and Electronic Desktops

Jiwon Kim Steven M. Seitz Maneesh Agrawala

University of Washington Microsoft Research

Page 2: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Motivation

Page 3: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Unifying physical andelectronic desktops

Record video of paper on physical desktop

Video camera

Desktop

Page 4: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Unifying physical andelectronic desktops

Record video of paper on physical desktop– Tracking

Video camera

Desktop

Page 5: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Unifying physical andelectronic desktops

Record video of paper on physical desktop– Tracking– Recognition

Video camera

Desktop

Page 6: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Unifying physical andelectronic desktops

Record video of paper on physical desktop– Tracking– Recognition– Linking

Video camera

Desktop

Page 7: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

ApplicationsVideo camera

Desktop

Page 8: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Applications

• Find lost documentVideo camera

Desktop

Page 9: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Applications

• Find lost document• Browse remote desk

Video camera

Desktop

Page 10: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Applications

• Find lost document• Browse remote desk• Find electronic

version

Video camera

Desktop

Page 11: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Applications

• Find lost document• Browse remote desk• Find electronic

version• History-based

queries

Video camera

Desktop

Page 12: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Example Input Video

Page 13: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Demo – Remote Desktop

Page 14: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Related Work

• Interactive desktops

DigitalDesk [Wellner 93]

Page 15: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Related Work

• Interactive desktops

Self-Organizing Desk[Rus et al. 97]DigitalDesk [Wellner 93]

Page 16: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Related Work

• Interactive desktops• Augmented paper

PADD [Guimbretière 03]

Page 17: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Related Work

• Interactive desktops• Augmented paper

PADD [Guimbretière 03] CyberCode[Rekimoto et al. 00]

Page 18: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Related Work

• Interactive desktops• Alternative media• Object tracking & recognition

SIFT [Lowe 04]

Page 19: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

System OverviewVideo camera

DeskUser

Computer

Page 20: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

System Overview

Video of desk

Page 21: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

System Overview

Video of desk Images from PDF

Page 22: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

System Overview

Video of desk Images from PDF

Track & recognize

Page 23: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

System Overview

Video of desk Images from PDF

Track & recognize

T T+1

Desk Desk

Internal representation

Page 24: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

System Overview

Video of desk Images from PDF

Track & recognize

T T+1

Desk Desk

Internal representation

Scene Graph

Page 25: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

System Overview

Video of desk Images from PDF

Track & recognize

T T+1

Desk Desk

Internal representation

Where is my W-2?

Page 26: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

System Overview

Video of desk Images from PDF

Track & recognize

T T+1

Desk Desk

Internal representation

Where is my W-2?

Answer

Page 27: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

System Overview

Video of desk Images from PDF

Track & recognize

T T+1

Desk Desk

Internal representation

Where is my W-2?

Page 28: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Tracking & Recognition

… …

Page 29: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Tracking & RecognitionEvent

… …

Page 30: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Event Types

Move

before after

Page 31: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Event Types

Move

Entry

before after

Page 32: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Event Types

Move

Entry

Exit

before after

Page 33: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Tracking & Recognition

Event

… …

Desk

Page 34: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Tracking & Recognition

… …

Event

… …

Desk Desk

Page 35: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Tracking & Recognition

… …

… …

Desk Desk

Event

tut-article.pdf

sanders01.pdf

objectspaces.pdf kidd94.pdf

lowe04sift.pdf

Page 36: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Assumptions

• Document– Corresponding electronic copy exists– No duplicates of same document

Page 37: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Assumptions

• Document– Corresponding electronic copy exists– No duplicates of same document

• Motion– 3 event types: move/entry/exit– One document at a time– Only topmost document can move

Page 38: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Non-Assumptions

• Desk need not be initially empty

Page 39: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Non-Assumptions

• Desk need not be initially empty• Stacks may overlap

Page 40: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Algorithm OverviewInput

Frames… …

Page 41: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Algorithm OverviewInput

Frames… …

Event Detection

before after

Page 42: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Algorithm OverviewInput

Frames… …

Event Detection

Event Interpretation

“A document moved from (x1,y1) to (x2,y2)”

before after

Page 43: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Algorithm OverviewInput

Frames… …

Event Detection

Event Interpretation

“A document moved from (x1,y1) to (x2,y2)”

Document Recognition

before after

File1.pdf

File2.pdf

File3.pdf

Page 44: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Algorithm OverviewInput

Frames… …

Event Detection

Event Interpretation

“A document moved from (x1,y1) to (x2,y2)”

Document Recognition

before after

File1.pdf

File2.pdf

File3.pdf

Scene Graph Update

Desk Desk

Page 45: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Algorithm OverviewInput

Frames… …

Event Detection

Event Interpretation

“A document moved from (x1,y1) to (x2,y2)”

Document Recognition

before after

File1.pdf

File2.pdf

File3.pdf

Scene Graph Update

Desk Desk

Page 46: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Event Detection

… …

Page 47: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Event Detection

time

Frame differences

… …

Page 48: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Event Detection

time

Frame differences

… …

Page 49: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Event Detection

time

Frame differences

… …

Page 50: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Event Detection

time

Frame differences

… …

Page 51: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Event Detection

time

Frame differences

… …

Page 52: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Event Detection

time

Threshold

Event Frames

time

… …

Motion Frames

Image motion

Page 53: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Event Detection

before after

Motion Frames

… …

Page 54: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Algorithm OverviewInput

Frames… …

Event Detection

Event Interpretation

“A document moved from (x1,y1) to (x2,y2)”

Document Recognition

before after

File1.pdf

File2.pdf

File3.pdf

Scene Graph Update

Desk Desk

Page 55: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Event Interpretation

Move

Entry

Exit

before after

Page 56: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Event Interpretation

Move

Entry

Exit

1. Move vs. Entry/Exit

before after

Page 57: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Event Interpretation

Move

Entry

Exit

2. Entry vs. Exit

before after

Page 58: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Event Interpretation

Move

Entry

Exit

1. Move vs. Entry/Exit

before after

Page 59: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Move vs. Entry/Exit

before after

Page 60: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Move vs. Entry/Exit

before after

Page 61: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Move vs. Entry/Exit

before after

Page 62: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Move vs. Entry/Exit

before after

Page 63: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Event Interpretation

• Use SIFT [Lowe 04]

– Rotation- and scale-invariant– Highly distinctive (128-bit vector)

Page 64: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Move vs. Entry/Exit

before after

Page 65: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Move vs. Entry/Exit

before after

Page 66: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Move vs. Entry/Exit

before after

Page 67: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Move vs. Entry/Exit

before after

Page 68: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Move vs. Entry/Exit

before after

Page 69: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Move vs. Entry/Exit

before after

Page 70: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Move vs. Entry/Exit

before after

Motion: (x,y,θ)

Page 71: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Algorithm OverviewInput

Frames… …

Event Detection

Event Interpretation

“A document moved from (x1,y1) to (x2,y2)”

Document Recognition

before after

File1.pdf

File2.pdf

File3.pdf

Scene Graph Update

Desk Desk

Page 72: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Document Recognition

File1.pdf File2.pdf File3.pdf File4.pdf File5.pdf File6.pdf

• Match against PDF image database

Page 73: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Document Recognition• Performance analysis

– Tested 20 pages against database of 162 pages

Page 74: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Document Recognition• Performance analysis

– Tested 20 pages against database of 162 pages

– ~200x300 pixels per document for reliable match

Document Resolution

Recognition Rate

Page 75: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Document Recognition• Performance analysis

– Tested 20 pages against database of 162 pages

– ~200x300 pixels per document for reliable match

Document Resolution

Recognition Rate

300

0.9

Page 76: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Algorithm OverviewInput

Frames… …

Event Detection

Event Interpretation

“A document moved from (x1,y1) to (x2,y2)”

Document Recognition

before after

File1.pdf

File2.pdf

File3.pdf

Scene Graph Update

Desk Desk

Page 77: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Scene Graph Update

before after

Motion: (x,y,θ)

Desk

Page 78: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Scene Graph Update

before after

Motion: (x,y,θ)

Desk

Page 79: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Scene Graph Update

before after

Motion: (x,y,θ)

Desk Desk

Page 80: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Results

• Input video– ~40 minutes– 1024x768 @ 15 fps– 22 documents, 49 events

• Running time– Video processed offline– No optimization– A few hours for entire video

Page 81: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Demo – Paper tracking

Page 82: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Photo Sorting Example

Page 83: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Photo Sorting Example

Page 84: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Demo – Photo Sorting

Page 85: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Future Work

• Enhance realism• More applications

Page 86: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Future Work

• Enhance realism– Handle more realistic desktops

Page 87: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Moving a stack of documents

Page 88: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Documents with no electronic versions

Page 89: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Future Work

• Enhance realism– Handle more realistic desktops– Real-time performance

Page 90: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Future Work

• More applications– Support other document tasks

• E.g., attach reminder, cluster documents

Page 91: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Future Work

• More applications– Support other document tasks

• E.g., attach reminder, cluster documents

– Beyond documents

Page 92: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Future Work

• More applications– Support other document tasks

• E.g., attach reminder, cluster documents

– Beyond documents

Page 93: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Future Work

• More applications– Support other document tasks

• E.g., attach reminder, cluster documents

– Beyond documents

Page 94: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Future Work

• More applications– Support other document tasks

• E.g., attach reminder, cluster documents

– Beyond documents

Page 95: UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University.

UIST 2004 Kim, Seitz, Agrawala

Acknowledgments

• NSF• Intel Corp.• Li Zhang