Transcript
Humans & Machinescollaborating on vision
Pietro PeronaCalifornia Institute of Technology
NSF Workshop - Frontiers in VisionCambridge, 23 Aug 2011
Friday, August 26, 2011
“Collaborative vision’’ ?
Pietro PeronaCalifornia Institute of Technology
NSF Workshop - Frontiers in VisionCambridge, 23 Aug 2011
Friday, August 26, 2011
Objectives
• Sketch new area of research
• Sampler of initial work
• Drawing lessons
• Brainstorm: potential, way forward
Friday, August 26, 2011
Plan
• Define area (10’)
• Presentations (50’): Perona, Geman, Grauman, Berg, Belongie
• Discussion (15’)
Friday, August 26, 2011
Definition
Friday, August 26, 2011
6
Friday, August 26, 2011
?
6
Friday, August 26, 2011
7
Friday, August 26, 2011
Friday, August 26, 2011
Friday, August 26, 2011
Friday, August 26, 2011
9
Friday, August 26, 2011
Lessons:
• Visual queries
• Easy for humans
• Difficult for machines
• Much information is available on line
• Pictures are digital dark matter
• Experts not providing visual knowledge
10
Friday, August 26, 2011
Unsupervised learning?
[Fergus et al., CVPR03] 11
Friday, August 26, 2011
Unsupervised learning?
[Fergus et al., CVPR03] 11
Friday, August 26, 2011
12
Friday, August 26, 2011
Friday, August 26, 2011
Throat
Friday, August 26, 2011
Throat
Friday, August 26, 2011
Visual knowledge
Task-oriented (practitioners)Categorical (experts) 14
Friday, August 26, 2011
Annotators Automata
ExpertsShared
knowledgeUsers
World
Querie
s
Answ
ers
Education
Models
ObservationObserv
ation
Science,expertise
Imageannotations
Machine visionscientists15
Friday, August 26, 2011
Annotators Automata
ExpertsShared
knowledgeUsers
World
Querie
s
Answ
ers
Education
Models
ObservationObserv
ation
Science,expertise
Imageannotations
Machine visionscientists15
Friday, August 26, 2011
Annotators Automata
ExpertsShared
knowledgeUsers
World
Querie
s
Answ
ers
Education
Models
ObservationObserv
ation
Science,expertise
Imageannotations
Machine visionscientists15
Friday, August 26, 2011
Annotators Automata
ExpertsShared
knowledgeUsers
World
Querie
s
Answ
ers
Education
Models
ObservationObserv
ation
Science,expertise
Imageannotations
Machine visionscientists15
Friday, August 26, 2011
Some progress...
Friday, August 26, 2011
Waterbirds
Mallard American Black Duck
Canada Goose Red Necked Grebe Clutter
DUCKS
Friday, August 26, 2011
x1i
x2i
xi = (x1i , x
2i )
p(xi | zi = 1)
p(xi | zi = 0)
Multidimensional signals and annotators
Friday, August 26, 2011
x1i
x2i
xi = (x1i , x
2i )
p(xi | zi = 1)
p(xi | zi = 0)
Multidimensional signals and annotators
Friday, August 26, 2011
x1i
x2i
xi = (x1i , x
2i )
p(xi | zi = 1)
p(xi | zi = 0)
Multidimensional signals and annotators
wj = (w1j , w
2j )
τj
Friday, August 26, 2011
lijxi
N
M
ij
σj
yij
θz
zi
Ji
βwj τj
γα
images
annotators
labels |Lij |
Full model
[Welinder et al., NIPS2010]Friday, August 26, 2011
Is there a duck in the image?
x1
i
x2
i
Friday, August 26, 2011
Is there a duck in the image?
x1
i
x2
i
Friday, August 26, 2011
Is there a duck in the image?
x1
i
x2
i
Friday, August 26, 2011
Is there a duck in the image?
x1
i
x2
i
Friday, August 26, 2011
Is there a duck in the image?
x1
i
x2
i
Friday, August 26, 2011
Is there a duck in the image?
x1
i
x2
i
Friday, August 26, 2011
Is there a duck in the image?
x1
i
x2
i
Friday, August 26, 2011
Concluding...
Friday, August 26, 2011
Aut
omat
ion
Performance
100%
100%
100%
0%
Collaborative vision
Friday, August 26, 2011
Aut
omat
ion
Performance
100%
100%
100%
0%
Collaborative vision
Friday, August 26, 2011
Aut
omat
ion
Performance
100%
100%
100%
0%
Collaborative vision
Friday, August 26, 2011
Aut
omat
ion
Performance
100%
100%
100%
0%
Collaborative vision
Friday, August 26, 2011
Aut
omat
ion
Performance
100%
100%
100%
0%
Collaborative vision
+ApplicationsTraining data
-ComplexityCost
Friday, August 26, 2011
Annotators Automata
ExpertsShared
knowledgeUsers
World
Querie
s
Answ
ers
Education
Models
ObservationObserv
ation
Science,expertise
Imageannotations
Machine visionscientists24
Friday, August 26, 2011
New research directions• Incremental learning
• Models of human vision, decision, attention
• Systems composed of machines and humans
• Performance bounds (humans, machines)
• Representations (human-machine-friendly)
• Extracting visual knowledge from experts
Friday, August 26, 2011
top related