Analysis, Indexing and Visualization of Presentation Videos
Analysis, Indexing and Visualization of Presentation Videos
Analysis, Indexing and Visualization of Presentation Videos
Analysis, Indexing and Visualization of Presentation Videos
Mic
he
le M
erl
er
em
ail
: m
me
rle
r@cs
.co
lum
bia
.ed
u
C
om
pu
ter
Sci
en
ce D
ep
art
me
nt,
Co
lum
bia
Un
ive
rsit
y
Mo
tiv
ati
on
& D
om
ain
De
scri
pti
on
Mic
he
le M
erl
er
em
ail
: m
me
rle
r@cs
.co
lum
bia
.ed
u
C
om
pu
ter
Sci
en
ce D
ep
art
me
nt,
Co
lum
bia
Un
ive
rsit
y
Mo
tiv
ati
on
& D
om
ain
De
scri
pti
on
Do
ma
in c
ha
lle
ng
es:
“W
ILD
” !
Ma
ny
v
ide
os
are
alr
ea
dy
arc
hiv
ed
Low
qu
ali
tyLa
ck o
f S
tru
ctu
reA
qu
ick
ly i
ncr
ea
sin
g q
ua
nti
ty o
f p
rese
nta
tio
n v
ide
os
is
GO
AL
: H
elp
use
rs e
ffic
ien
tly
M
oti
va
tio
n &
Do
ma
in D
esc
rip
tio
nD
om
ain
ch
all
en
ge
s: “
WIL
D”
! •
Lack
of
ad
dit
ion
al
•N
ot
reco
rde
d b
y
•U
nco
nst
rain
ed
ca
me
ra
alr
ea
dy
arc
hiv
ed
Low
qu
ali
tyLa
ck o
f S
tru
ctu
re
Vid
eo
s o
f p
rese
nta
tio
ns
are
to
ols
no
wa
da
ys
A q
uic
kly
in
cre
asi
ng
qu
an
tity
of
pre
sen
tati
on
vid
eo
s is
pu
bli
cly a
vail
ab
le a
nd
re
trie
vab
le o
n t
he
we
bG
OA
L :
He
lp u
sers
eff
icie
ntl
y
an
d e
ffe
ctiv
ely
acc
ess
•
Lack
of
ad
dit
ion
al
sou
rce
s o
f
•N
ot
reco
rde
d b
y
pro
fess
ion
al
•U
nco
nst
rain
ed
ca
me
ra
mo
vem
en
ts
Vid
eo
s o
f p
rese
nta
tio
ns
are
to
ols
no
wa
da
ys
em
plo
yed
in a
la
rge
va
rie
ty o
f sy
ste
ms
an
d e
ffe
ctiv
ely
acc
ess
(ed
uca
tio
na
l) in
form
ati
on
sou
rce
s o
f
info
rma
tio
n
(e.g
. e
lect
ron
ic
pro
fess
ion
al
cam
era
me
n
mo
vem
en
ts
•S
lid
es
Tru
nca
tio
n�
Dis
tan
ce o
r E
-le
arn
ing
�C
on
fere
nce
pro
cee
din
gs
(ed
uca
tio
na
l) in
form
ati
on
(e.g
. e
lect
ron
ic
cop
ies
of
slid
es)
•Li
gh
t ca
nn
ot
be
use
d a
s cl
ue
•S
lid
es
Tru
nca
tio
n
•C
om
pre
ssio
n�
Co
nfe
ren
ce p
roce
ed
ing
s
�S
tud
en
t p
rese
nta
tio
ns
resu
lts
1-2
0 o
f 1
,16
0
use
d a
s cl
ue
•N
ot
ed
ite
d
•C
om
pre
ssio
n
•S
tan
da
rd p
roce
ssin
g
�S
tud
en
t p
rese
nta
tio
ns
�C
orp
ora
te t
alk
s6
59
eve
nts
9
K a
uth
ors
12
K le
ctu
res
1
4K
vid
eo
sre
sult
s 1
-20
of
1,1
60
•N
ot
ed
ite
d•
Sta
nd
ard
pro
cess
ing
do
es
no
t a
pp
ly
�C
orp
ora
te t
alk
s1
2K
lect
ure
s 1
4K
vid
eo
s
3.
Gra
ph
ics
Ind
ex
Ge
ne
rati
on
GO
AL
: E
nsu
re e
nd
use
rs s
ati
sfa
ctio
nw
ith
ho
w t
he
1
. U
ser
Pre
ferr
ed
Fa
ce I
nd
exe
s3
. G
rap
hic
s In
de
x G
en
era
tio
nG
OA
L:
En
sure
en
du
sers
sa
tisf
act
ion
wit
h h
ow
th
e
info
rma
tio
n e
xtra
cte
d f
rom
th
e v
ide
os
is p
rese
nte
d
Re
sult
s
1.
Use
r P
refe
rre
d F
ace
In
de
xes
3.
Gra
ph
ics
Ind
ex
Ge
ne
rati
on
Ex
pe
rim
en
tal
Se
tup
Re
sult
s
�1
57
5 A
ma
zon
Me
cha
nic
al
Turk
HIT
s
(15
sp
ea
kers
x 3
ord
eri
ng
x 3
5 u
niq
ue
wo
rke
rs)
�M
ost
pe
op
le p
refe
r H
ea
d &
Sh
ou
lde
r F
RO
NTA
Lv
iew
�3
5%
of
vote
s w
en
t to
Le
ft a
nd
Rig
ht
¾ H
ea
d &
Sh
ou
lde
r!P
rop
ose
d S
olu
tio
n(1
5 s
pe
ake
rs x
3 o
rde
rin
g x
35
un
iqu
e w
ork
ers
)�
35
% o
f vo
tes
we
nt
to L
eft
an
d R
igh
t ¾
He
ad
& S
ho
uld
er!
Co
nfi
rms
resu
lts
of
psy
cho
log
ica
l st
ud
ies
on
in
fere
nce
of
Pro
po
sed
So
luti
on
�LB
P H
isto
gra
m +
Co
lor
His
tog
ram
Co
nfi
rms
resu
lts
of
psy
cho
log
ica
l st
ud
ies
on
in
fere
nce
of
he
ad
3D
in
form
ati
on
fro
m 3
/4 v
iew
of
face
[B
urk
e V
R0
7]
Ind
ex
pre
sen
tati
on
vid
eo
s b
ase
d o
n f
ou
r m
ajo
r cu
es:
�LB
P H
isto
gra
m +
Co
lor
His
tog
ram
�O
nli
ne
Clu
ste
rin
g (
vis
ua
l + t
em
po
ral)
wit
h a
vg.
Lin
kag
e
Ind
ex
pre
sen
tati
on
vid
eo
s b
ase
d o
n f
ou
r m
ajo
r cu
es:
�Te
xt (
+a
ud
io t
ran
scri
pts
)�
Gra
ph
ics
�O
nli
ne
Clu
ste
rin
g (
vis
ua
l + t
em
po
ral)
wit
h a
vg.
Lin
kag
e
�N
orm
. C
ross
Co
rre
lati
on
fo
r Te
mp
late
Ma
tch
ing
�
Text
(+
au
dio
tra
nsc
rip
ts)
�S
pe
ake
r fa
ces
�G
rap
hic
s
�M
osa
ics
5.0
),
(
11
),
(1
2
C k
ji
cx
CC
xS
j
+=
∑ =χ
ixregion
It h
as
be
tte
r il
lum
ina
tio
n
It h
as
be
tte
r re
solu
tio
n
I ca
n s
ee
/te
ll m
ore
ab
ou
t th
e w
ho
le a
pp
ea
ran
ce o
f th
e p
ers
on
I ca
n s
ee
be
tte
r th
e e
ye
s a
nd
exp
ress
ion
of
the
pe
rso
n
�M
osa
ics
()
() )
(4.0
)(
4.0
),
(
5.0
),
(1
2
jj
ji
kjk
ij
CT
tC
SC
xT
cx
C
−+
−+
=
+∑ =
βα
χj
Ccluster
I ca
n s
ee
be
tte
r th
e e
ye
s a
nd
exp
ress
ion
of
the
pe
rso
n
I p
refe
r th
is p
ose
of
a p
ers
on
in
ge
ne
ral
I p
ick
ed
th
e b
est
ou
t o
f a
bu
nch
of
ba
d p
ictu
res
No
ne
of
the
ab
ov
e(p
lea
se e
xpla
in y
ou
r re
aso
n w
ith
a f
ew
wo
rds
in t
he
bo
x b
elo
w)
BA
CK
-EN
D
()
()
jj
ji
),
(
),
(j
ij
iC
xT
Cx
S<>
BA
CK
-EN
D)
,(
),
(j
ij
iC
xT
Cx
S<>
2.
Au
tom
ati
c G
en
era
tio
n o
f S
pe
ak
ers
Fa
ce I
nd
exe
sU
ser
Pre
ferr
ed
Te
xtu
al
Ind
ex
Gra
ph
ics
Ind
ex
4.
Tex
tua
l In
de
x G
en
era
tio
n2
. A
uto
ma
tic
Ge
ne
rati
on
of
Sp
ea
ke
rs F
ace
In
de
xes
Use
r P
refe
rre
d
Face
In
de
xes
Text
ua
l In
de
x
Ge
ne
rati
on
Gra
ph
ics
Ind
ex
Ge
ne
rati
on
4.
Tex
tua
l In
de
x G
en
era
tio
n1
34
Ed
ge
s C
on
ne
cte
d
Ge
om
etr
ic +
Ed
ge
Lo
cal A
da
pti
ve O
tsu
Fa
ce I
nd
exe
sG
en
era
tio
nG
en
era
tio
n
Se
lect
ion
ba
sed
on
3 q
ua
lity
me
asu
res
�V
iola
Jo
ne
s d
ete
cto
rFa
ce
LoG
ed
ge
sE
dg
es
Co
nn
ect
ed
Co
mp
on
en
ts
Ge
om
etr
ic +
Ed
ge
De
nsi
ty C
on
stra
ints
Loca
l Ad
ap
tive
Ots
u
(LA
O)
Bin
ari
zati
on
Tess
era
ctO
CR
Sp
ea
ker
Face
S
em
an
tic
Sh
ot
25
1.
Re
solu
tio
n
Se
lect
ion
ba
sed
on
3 q
ua
lity
me
asu
res
�V
iola
Jo
ne
s d
ete
cto
r
�C
olo
r sk
in f
ilte
rFa
ce
De
tect
ion
Co
mp
on
en
tsD
en
sity
Co
nst
rain
ts(L
AO
) B
ina
riza
tio
nTe
sse
ract
OC
R
Co
mp
lete
d T
ask
sS
pe
ake
r Fa
ce
Ind
ex
Ge
ne
rati
on
Se
ma
nti
c S
ho
t
Re
pre
sen
tati
on
25
hw
×1
. R
eso
luti
on
Siz
e o
f th
e f
ace
re
gio
n
�C
olo
r sk
in f
ilte
rD
ete
ctio
nR
csca
rch
Inte
rvie
w w
ith
Cli
en
t
{ i
site
d P
roje
ct S
pa
ce
Co
mp
lete
d T
ask
s
Ind
ex
Ge
ne
rati
on
Re
pre
sen
tati
on
hw
×S
ize
of
the
fa
ce r
eg
ion
Face
Se
ed
s
Qia
nt
Ho
use
Re
sid
en
t A
sso
cia
tio
n M
ee
tin
g
�1
ho
ur
an
d 4
5 m
inu
tes
of
vid
eo
, 8
stu
de
nt
pre
sen
tati
on
s
Face
Se
ed
s
Co
mp
lete
d T
ask
s
�1
ho
ur
an
d 4
5 m
inu
tes
of
vid
eo
, 8
stu
de
nt
pre
sen
tati
on
s
�1
3 s
lid
es
pe
r p
rese
nta
tio
n (
ave
rag
e)
O tf
�M
ILTr
ack
(pre
dic
tio
n):
Face
Re
sea
rch
Inte
rvie
w
Cli
en
tA
fte
r vo
cab
ula
ry
corr
ect
ion
LAO + Tesseract
�1
3 s
lid
es
pe
r p
rese
nta
tio
n (
ave
rag
e)
VA
ST
MM
Bro
wse
r [1
]6
tf
P tf
�M
ILTr
ack
(pre
dic
tio
n):
�V
iola
Jo
ne
s d
ete
cto
r(o
bse
rva
tio
n):
Face
In
terv
iew
C
lie
nt
Pro
ject
Sp
ace
Ho
use
Re
sid
en
t A
sso
cia
tio
n M
ee
tin
g
corr
ect
ion
Tesseract
8000
Recognized
LAO + Tesseract
VA
ST
MM
Bro
wse
r [1
]6
tf
2.
Po
se
�V
iola
Jo
ne
s d
ete
cto
r(o
bse
rva
tio
n):
�S
imp
lifi
ed
Ka
lma
nfi
lte
r:Tra
ckin
gH
ou
se R
esi
de
nt
Ass
oci
ati
on
Me
eti
ng
Tesseract
6000
Recognized Characters
FR
ON
T-E
ND
O t
P tt
ff
f)
1(α
α−
+←
�Le
ft a
nd
rig
ht
¾ p
ose
cla
ssif
iers
Ed
ge
his
tog
ram
de
scri
pto
r
2.
Po
se�
Sim
pli
fie
d K
alm
an
filt
er:
4000
6000
Number Recognized Characters
FR
ON
T-E
ND
tt
tf
ff
)1(
αα
−+
←
15
38
3
15
38
5
15
38
7
15
37
1
15
40
9
15
35
5
�E
dg
e h
isto
gra
m d
esc
rip
tor
�S
VM
R
BF
ke
rne
lFa
ce T
rack
s
2000
4000
Number Characters
6.
Fin
al
Bro
wse
r in
terf
ace
15
38
3
15
38
5
15
38
7
15
37
1
15
40
9
15
35
5
�S
VM
R
BF
ke
rne
l
�Fa
ceTr
ace
rd
ata
set
Face
Tra
cks
2000
Number
6.
Fin
al
Bro
wse
r in
terf
ace
Tra
inin
g S
et
(le
ft ¾
, fr
on
t,
rig
ht
¾)
~1
0K
im
ag
es
Home
Search
Explore Collections
Visual Search
Login
or sign up
0
Recognition Method
Tra
inin
g S
et
(le
ft ¾
, fr
on
t,
rig
ht
¾)
~1
0K
im
ag
es
Test
Se
t ~
12
K i
ma
ge
s
Av
era
ge
Te
st A
ccu
racy
81
.5%
�S
ele
ctio
n o
f fa
ces
to m
atc
h
Se
arc
h
Home
Search
Explore Collections
Visual Search
Login
or sign up
Nu
mb
er
N
um
be
r P
reci
sio
nR
eca
ll
Recognition Method
Av
era
ge
Te
st A
ccu
racy
81
.5%
�S
ele
ctio
n o
f fa
ces
to m
atc
h
�LB
P d
esc
rip
tor
+ S
q.
L2
dis
tan
ceTra
cks
Se
arc
h
People Index
Graphics Index
Search Tips
Phone + P05 + G08
Tag
Nu
mb
er
GT
Wo
rds
Nu
mb
er
Re
c. W
ord
sP
reci
sio
nR
eca
ll3
Sk
in R
ati
oM
atc
hin
gPhone + P05 + G08
Tag
People Index
Graphics Index
22
76
11
26
0.4
95
0.6
65
Un
iqu
e S
pe
ake
rs
area
skinPixels
skinRatio
#=
>
185
.1
RU
niq
ue
Sp
ea
kers
Face
Tra
cks
5.
Se
ma
nti
c S
ho
t R
ep
rese
nta
tio
nE
nh
an
ced
Fe
atu
re B
ase
d M
osa
ic
area
skinRatio
=
>
>
⇔=
107
.0
185
.1
skin
Pixel
RBGR
Face
Tra
cks
5.
Se
ma
nti
c S
ho
t R
ep
rese
nta
tio
nE
nh
an
ced
Fe
atu
re B
ase
d M
osa
ic
>>+
+⇔
=
112
.0
107
.0
)(
skin
Pixel
2
RG
BG
R
RB
Face
In
de
x
Resolution 10 secs
>
++
112
.0
)(
2B
GR
RG
�S
ele
ct“b
est
fa
ces”
to
pre
sen
t t
o e
nd
use
r
Face
In
de
x
Ge
ne
rati
on
�P
TZ
Est
ima
tio
n
�S
IFT
+ R
AN
SA
C o
n
min max
Resolution 10 secs
Click on an icon to find the graphic in the video
pre
sen
t t
o e
nd
use
rG
en
era
tio
n�
SIF
T +
R
AN
SA
C o
n
key
fra
me
sskinRatio
wresolution
wpose
wQ
⋅+
⋅+
⋅=
32
1
35
0O
verl
ay R
eco
gn
ize
d T
ext
min max
Tagline
Click on an icon to find the graphic in the video
skinRatio
wresolution
wpose
wQ
⋅+
⋅+
⋅=
32
1
Av
era
ge
Tra
ck M
atc
hin
g T
ime
(se
cs)
33
5
30
0
35
0O
verl
ay R
eco
gn
ize
d T
ext
Tagline Frames
Test
on
3
33
5T
rack
Ma
tch
ing
Fa
ce S
ele
ctio
n
Left
/rig
ht3
4 E
xtra
ctio
n
20
0
25
0P
rob
lem
Sta
tem
en
t
•E
colo
gic
al
Imp
act
Te
xt1
Te
xt3
Te
xt4
Te
xt5
Te
xt7
Te
xt8
Te
xt1
0
Frames
51
ou
t o
f 5
8 w
ith
He
ad
& s
ho
uld
er,
¾ p
rofi
le v
iew
Test
on
3
stu
de
nt
Left
/rig
ht3
4 E
xtra
ctio
n
Sk
in-R
es
Ext
ract
ion
K-M
ea
ns
Co
mp
uta
tio
n
15
0
20
0•
Wa
ste
go
es
to L
an
dfi
lls
•E
ne
rgy
So
urc
e
•C
ost
Eff
icie
ncy
•W
ast
e D
isp
osa
l B
ill
Te
xt1
Te
xt3
Te
xt4
Te
xt5
Te
xt7
Te
xt8
Te
xt1
0
Text
Problem
Phone
51
ou
t o
f 5
8 w
ith
He
ad
& s
ho
uld
er,
¾ p
rofi
le v
iew
stu
de
nt
pre
sen
ta-
K-M
ea
ns
Co
mp
uta
tio
n
50
10
0
15
0•
Ele
ctri
cal
Bil
l
•M
s W
ilso
n i
s lo
ok
ing
fo
r a
n e
co-f
rie
nd
ly,
cost
eff
icie
nt,
an
d e
asy
to
use
pro
du
ct t
ha
t
wil
l co
nve
rt h
er
soli
d w
ast
e i
nto
usa
ble
en
erg
y
�S
eg
me
nt
vid
eo
in
to s
em
an
tica
lly
dis
tin
ct s
ho
ts b
ase
d
on
sli
de
s
People
pre
sen
ta-
tio
nv
ide
os,
0
50
en
erg
y
En
ha
nce
Gra
ph
ics
on
sli
de
s
�C
ha
ng
es
in t
ext
use
d t
o a
sse
ss s
lid
e c
ha
ng
es
Graphics
45
min
ute
s
ea
ch
20
1
9
K-M
ea
ns(
10
0)
sele
ct (
10
0)
min
-min
0
12
3E
nh
an
ce G
rap
hic
s�
Ch
an
ge
s in
te
xt u
sed
to
ass
ess
sli
de
ch
an
ge
s
Graphics
ea
chK
-Me
an
s(1
00
)se
lect
(1
00
)m
in-m
in