Lecture 1 - Fei-Fei Li Lecture 1: Introduc.on to “Computer Vision” Professor FeiFei Li Stanford Vision Lab 24Sep12 1
Lecture 1 - !!!
Fei-Fei Li!
Lecture 1: Introduc.on to “Computer Vision”
Professor Fei-‐Fei Li Stanford Vision Lab
24-‐Sep-‐12 1
Lecture 1 - !!!
Fei-Fei Li!
Welcome to CS231a: Computer Vision
Slid
e ad
apte
d fr
om S
vetl
ana
Laze
bnik
24-‐Sep-‐12 2
Lecture 1 - !!!
Fei-Fei Li!
Today’s agenda
• Introduc.on to computer vision • Course overview
24-‐Sep-‐12 3
Lecture 1 - !!!
Fei-Fei Li!
Quiz?
24-‐Sep-‐12 4
Lecture 1 - !!!
Fei-Fei Li!
What about this?
24-‐Sep-‐12 5
Lecture 1 - !!!
Fei-Fei Li!
Image (or video) Sensing device Interpreting device Interpretations
garden, spring, bridge, water, trees, flower, green, etc.
What is (computer) vision?
24-‐Sep-‐12 6
Lecture 1 - !!!
Fei-Fei Li!
What is it related to?
Computer Vision
Neuroscience
Machine learning
Speech
Informa.on retrieval
Maths
Computer Science
Biology
Engineering
Physics
Robo.cs Cogni.ve sciences
Psychology
graphics,algorithms, system,theory,…
Image processing
24-‐Sep-‐12 7
Lecture 1 - !!!
Fei-Fei Li!
The goal of computer vision • To bridge the gap between pixels and “meaning”
What we see What a computer sees Sou
rce:
S. N
aras
imha
n
24-‐Sep-‐12 8
Lecture 1 - !!!
Fei-Fei Li!
Image (or video) Sensing device Interpreting device Interpretations
garden, spring, bridge, water, trees, flower, green, etc.
What is (computer) vision?
24-‐Sep-‐12 9
Lecture 1 - !!!
Fei-Fei Li!
1981: Nobel Prize in medicine
Hubel & Wiesel
24-‐Sep-‐12 10
Lecture 1 - !!!
Fei-Fei Li!
Potter, Biederman, etc. 1970s
Human vision is superbly efficient
24-‐Sep-‐12 11
Lecture 1 - !!!
Fei-Fei Li!
Thorpe, et al. Nature, 1996
24-‐Sep-‐12 12
Lecture 1 - !!!
Fei-Fei Li!
Thorpe, et al. Nature, 1996
150 ms !!
24-‐Sep-‐12 13
Lecture 1 - !!!
Fei-Fei Li!
Change blindess
Rensink, O’regan, Simon, etc.
24-‐Sep-‐12 14
Lecture 1 - !!!
Fei-Fei Li!
Rensink, O’regan, Simon, etc.
Change blindess
24-‐Sep-‐12 15
Lecture 1 - !!!
Fei-Fei Li!
segmenta.on
24-‐Sep-‐12 16
Lecture 1 - !!!
Fei-Fei Li!
Percep.on
24-‐Sep-‐12 17
Lecture 1 - !!!
Fei-Fei Li! 24-‐Sep-‐12 18
Lecture 1 - !!!
Fei-Fei Li! 24-‐Sep-‐12 19
Lecture 1 - !!!
Fei-Fei Li!
Image (or video) Sensing device Interpreting device Interpretations
garden, spring, bridge, water, trees, flower, green, etc.
What is (computer) vision?
24-‐Sep-‐12 20
Lecture 1 - !!!
Fei-Fei Li!
The goal of computer vision • To bridge the gap between pixels and “meaning”
What we see What a computer sees Sou
rce:
S. N
aras
imha
n
24-‐Sep-‐12 21
Lecture 1 - !!!
Fei-Fei Li!
Origins of computer vision: an MIT undergraduate summer project
L. G. Roberts, Machine Percep,on of Three Dimensional Solids, Ph.D. thesis, MIT Department of Electrical Engineering, 1963.
24-‐Sep-‐12 22
Lecture 1 - !!!
Fei-Fei Li!
What kind of informa.on can we extract from an image?
• Metric 3D informa.on • Seman.c informa.on
24-‐Sep-‐12 23
Lecture 1 - !!!
Fei-Fei Li!
Vision as measurement device Real-time stereo Structure from motion
NASA Mars Rover
Pollefeys et al.
Reconstruction from Internet photo collections
Goesele et al.
24-‐Sep-‐12 24
Lecture 1 - !!!
Fei-Fei Li!
Vision as a source of semantic information sky
water
Ferris wheel
amusement park
Cedar Point
12 E
tree
tree
tree
carousel deck
people waiting in line
ride
ride ride
umbrellas
pedestrians
maxair
bench
tree
Lake Erie
people sitting on ride
Objects Activities Scenes Locations Text / writing Faces Gestures Motions Emotions…
The Wicked Twister
Slid
e cr
edit
: Kr
iste
n G
raum
an
24-‐Sep-‐12 25
Lecture 1 - !!!
Fei-Fei Li!
Why study computer vision?
Personal photo albums
Surveillance and security
Movies, news, sports
Medical and scientific images
• Vision is useful: Images and video are everywhere!
24-‐Sep-‐12 26
Lecture 1 - !!!
Fei-Fei Li!
Why study computer vision? • Vision is useful • Vision is interes.ng • Vision is difficult
– Half of primate cerebral cortex is devoted to visual processing
– Achieving human-‐level visual percep.on is probably “AI-‐complete”
24-‐Sep-‐12 27
Lecture 1 - !!!
Fei-Fei Li!
Why is computer vision difficult?
24-‐Sep-‐12 28
Lecture 1 - !!!
Fei-Fei Li!
Challenges: viewpoint variation
Michelangelo 1475-1564
slide credit: Fei-Fei, Fergus & Torralba
24-‐Sep-‐12 29
Lecture 1 - !!!
Fei-Fei Li!
Challenges: illumination
image credit: J. Koenderink
24-‐Sep-‐12 30
Lecture 1 - !!!
Fei-Fei Li!
Challenges: scale
slid
e cr
edit:
Fei
-Fei
, Fer
gus
& T
orra
lba
24-‐Sep-‐12 31
Lecture 1 - !!!
Fei-Fei Li!
Challenges: deformation
Xu, Beihong 1943 slide credit: Fei-Fei, Fergus & Torralba
24-‐Sep-‐12 32
Lecture 1 - !!!
Fei-Fei Li!
Challenges: occlusion
Magritte, 1957
slide credit: Fei-Fei, Fergus & Torralba 24-‐Sep-‐12 33
Lecture 1 - !!!
Fei-Fei Li!
Challenges: background clutter
slid
e cr
edit:
Sve
tlana
Laz
ebni
k
24-‐Sep-‐12 34
Lecture 1 - !!!
Fei-Fei Li!
Challenges: Motion
slid
e cr
edit:
Sve
tlana
Laz
ebni
k
24-‐Sep-‐12 35
Lecture 1 - !!!
Fei-Fei Li!
Challenges: object intra-‐class varia.on
slid
e cr
edit:
Fei
-Fei
, Fer
gus
& T
orra
lba
24-‐Sep-‐12 36
Lecture 1 - !!!
Fei-Fei Li!
Challenges: local ambiguity
slid
e cr
edit:
Fei
-Fei
, Fer
gus
& T
orra
lba
24-‐Sep-‐12 37
Lecture 1 - !!!
Fei-Fei Li!
Challenges or opportuni.es? • Images are confusing, but they also reveal the structure of the world through numerous cues
• Our job is to interpret the cues!
Imag
e so
urce
: J. K
oend
erin
k
24-‐Sep-‐12 38
Lecture 1 - !!!
Fei-Fei Li!
Depth cues: Linear perspec.ve
slid
e cr
edit:
Sve
tlana
Laz
ebni
k
24-‐Sep-‐12 39
Lecture 1 - !!!
Fei-Fei Li!
Depth cues: Aerial perspec.ve
slid
e cr
edit:
Sve
tlana
Laz
ebni
k
24-‐Sep-‐12 40
Lecture 1 - !!!
Fei-Fei Li!
Depth ordering cues: Occlusion
Sou
rce:
J. K
oend
erin
k
24-‐Sep-‐12 41
Lecture 1 - !!!
Fei-Fei Li!
Shape cues: Texture gradient
slid
e cr
edit:
Sve
tlana
Laz
ebni
k
24-‐Sep-‐12 42
Lecture 1 - !!!
Fei-Fei Li!
Shape and ligh.ng cues: Shading
Sou
rce:
J. K
oend
erin
k
24-‐Sep-‐12 43
Lecture 1 - !!!
Fei-Fei Li!
Posi.on and ligh.ng cues: Cast shadows
Sou
rce:
J. K
oend
erin
k
24-‐Sep-‐12 44
Lecture 1 - !!!
Fei-Fei Li!
Grouping cues: Similarity (color, texture, proximity)
slid
e cr
edit:
Sve
tlana
Laz
ebni
k
24-‐Sep-‐12 45
Lecture 1 - !!!
Fei-Fei Li!
Grouping cues: “Common fate”
Imag
e cr
edit:
Arth
us-B
ertra
nd (v
ia F
. Dur
and)
24-‐Sep-‐12 46
Lecture 1 - !!!
Fei-Fei Li!
Bogom line • Percep.on is an inherently ambiguous problem
– Many different 3D scenes could have given rise to a par.cular 2D picture
24-‐Sep-‐12 47
Lecture 1 - !!!
Fei-Fei Li!
Bogom line • Percep.on is an inherently ambiguous problem
– Many different 3D scenes could have given rise to a par.cular 2D picture
• Possible solu.ons – Bring in more constraints (more images) – Use prior knowledge about the structure of the world
• Need a combina.on of different methods 24-‐Sep-‐12 48
Lecture 1 - !!!
Fei-Fei Li!
Computer Vision in the Real World
24-‐Sep-‐12 49
Lecture 1 - !!!
Fei-Fei Li!
Special effects: shape and mo.on capture
Sour
ce: S
. Sei
tz
24-‐Sep-‐12 50
Lecture 1 - !!!
Fei-Fei Li!
3D urban modeling
Bing maps, Google Streetview Source: S. Seitz
24-‐Sep-‐12 51
Lecture 1 - !!!
Fei-Fei Li!
3D urban modeling: Microsoj Photosynth
hgp://labs.live.com/photosynth/ Source: S. Seitz
24-‐Sep-‐12 52
Lecture 1 - !!!
Fei-Fei Li!
Face detec.on
• Many new digital cameras now detect faces – Canon, Sony, Fuji, …
Source: S. Seitz
24-‐Sep-‐12 53
Lecture 1 - !!!
Fei-Fei Li!
Smile detec.on
Sony Cyber-shot® T70 Digital Still Camera Source: S. Seitz
24-‐Sep-‐12 54
Lecture 1 - !!!
Fei-Fei Li!
Face recogni.on: Apple iPhoto sojware
hgp://www.apple.com/ilife/iphoto/
24-‐Sep-‐12 55
Lecture 1 - !!!
Fei-Fei Li!
Biometrics
How the Afghan Girl was Iden.fied by Her Iris Pagerns
Source: S. Seitz
24-‐Sep-‐12 56
Lecture 1 - !!!
Fei-Fei Li!
Biometrics
Fingerprint scanners on many new laptops, other devices
Face recogni.on systems now beginning to appear more widely hgp://www.sensiblevision.com/ Source: S. Seitz
24-‐Sep-‐12 57
Lecture 1 - !!!
Fei-Fei Li!
Op.cal character recogni.on (OCR)
Digit recognition, AT&T labs
Technology to convert scanned docs to text • If you have a scanner, it probably came with OCR sojware
License plate readers http://en.wikipedia.org/wiki/Automatic_number_plate_recognition
Source: S. Seitz
24-‐Sep-‐12 58
Lecture 1 - !!!
Fei-Fei Li!
Toys and Robots
Lecture 1 - !!!
Fei-Fei Li!
Mobile visual search: Google Goggles
24-‐Sep-‐12 60
Lecture 1 - !!!
Fei-Fei Li!
Mobile visual search: iPhone Apps
24-‐Sep-‐12 61
Lecture 1 - !!!
Fei-Fei Li!
Automo.ve safety
• Mobileye: Vision systems in high-‐end BMW, GM, Volvo models – “In mid 2010 Mobileye will launch a world's first applica.on of full emergency braking for collision mi.ga.on for pedestrians where vision is the key technology for detec.ng pedestrians.”
Source: A. Shashua, S. Seitz
24-‐Sep-‐12 62
Lecture 1 - !!!
Fei-Fei Li!
Vision in supermarkets
LaneHawk by EvolutionRobotics “A smart camera is flush-mounted in the checkout lane, continuously watching for items. When an item is detected and recognized, the cashier verifies the quantity of items that were found under the basket, and continues to close the transaction. The item can remain under the basket, and with LaneHawk, you are assured to get paid for it… “ Source: S. Seitz
24-‐Sep-‐12 63
Lecture 1 - !!!
Fei-Fei Li!
Vision-‐based interac.on (and games)
Microsoft’s Kinect
Source: S. Seitz Assistive technologies
Sony EyeToy
24-‐Sep-‐12 64
Lecture 1 - !!!
Fei-Fei Li!
Vision for robo.cs, space explora.on
Vision systems (JPL) used for several tasks • Panorama s.tching • 3D terrain modeling • Obstacle detec.on, posi.on tracking • For more, read “Computer Vision on Mars” by Maghies et al.
NASA'S Mars Explora.on Rover Spirit captured this westward view from atop a low plateau where Spirit spent the closing months of 2007.
Sour
ce: S
. Sei
tz
24-‐Sep-‐12 65
Lecture 1 - !!!
Fei-Fei Li!
The computer vision industry
• A list of companies here: hgp://www.cs.ubc.ca/spider/lowe/vision.html
24-‐Sep-‐12 66
Lecture 1 - !!!
Fei-Fei Li!
Today’s agenda
• Introduc.on to computer vision • Course overview
24-‐Sep-‐12 67
Lecture 1 - !!!
Fei-Fei Li!
Overall philosophy
• Breadth – Computer vision is a huge field – It can impact every aspect of life and society – It will drive the next informa.on and AI revolu.on – Pixels are everywhere in our lives and cyber space – Lectures are high-‐level, meant to be informa.ve, and covers many topics – Lots of links to references. Know where to look for references – Speak our “language”
• Depth – Computer vision is a highly technical field, i.e. know your math! – Homework meant to be challenging, both theore.cal ques.ons and
programming exercises – Master bread-‐and-‐buger techniques: face recogni.on, corners, lines, features,
op.cal flows, clustering and segmenta.on, basic object recogni.on techniques
– Course projects are your hands-‐on experience in computer vision systems and research
24-‐Sep-‐12 68
Lecture 1 - !!!
Fei-Fei Li!
Contac.ng instructor and TAs • ALL EMAIL CORRESPONDENCES TO ANYONE OF US:
– cs231a-‐aut1213-‐[email protected]
• Professor: Fei-‐Fei Li – Office hour: Tues 3:30-‐4:30pm
• Jon Krause, Ph.D, CS – Office hour: Mon 4:30-‐5:30pm
• Vignesh Ramanathan, Ph.D, EE – Office hour: Wed 3:00-‐4:00pm
• Jinchao Ye, master, CS – Office hour: TBD
• Zixuan Wang, master, CS – Office hour: Fri 3:00-‐4:00pm
24-‐Sep-‐12 69
Lecture 1 - !!!
Fei-Fei Li!
Syllabus
• Go to website…
24-‐Sep-‐12 70
Lecture 1 - !!!
Fei-Fei Li!
Course Project: overview
• 40% of your grade • Form your team:
– either 2 people or 1 person – but the quality is judged regardless of the number of people on the team
– be nice to your partner: do you plan to drop the course?
• No late days • Mandatory agendance on Dec 6 for all non-‐SCPD students
24-‐Sep-‐12 71
Lecture 1 - !!!
Fei-Fei Li!
Course Project: overview (con.nued) • Start immediately • Some important dates:
– Oct 16 • Finalize team • Project proposal due for “open project” teams
– Nov 6 • Milestone due (2-‐3 pages)
– Dec 3 • Final codes due
– Dec 4 • Final writeup due
– Dec 6 • Presenta.on
24-‐Sep-‐12 72
Lecture 1 - !!!
Fei-Fei Li!
Course Project Op.on #1: the Finding Mii Challenge
24-‐Sep-‐12 73
Lecture 1 - !!!
Fei-Fei Li!
• Original research ideas encouraged • Useful datasets:
– ImageNet (www.image-‐net.org) – PASCAL
• Need Fei-‐Fei’s approval – Email is the best way – Do it BEFORE Oct 16 (proposal submission deadline)
24-‐Sep-‐12 74
Course Project Op.on #2: Open Project
Lecture 1 - !!!
Fei-Fei Li!
Grading policy
• Problem Sets: 40% – We have 5 problem sets – Homework 0: very important! (more details…) – Late policy
• 5 free late days – use them in your ways • Ajerwards, 25% off per day late • Not accepted ajer 3 late days per PS
– Collabora.on policy • Read the student code book, understand what is ‘collabora.on’ and what is ‘academic infrac.on’
• Midterm Exam: 20% – In class: Tues, Oct 30
24-‐Sep-‐12 75
Lecture 1 - !!!
Fei-Fei Li!
Grading policy
• Course project: 40% – presenta.on: 5% – write-‐up: 10%
• clarity, structure, language, references: 3% • background literature survey, good understanding of the problem: 3% • good insights and discussions of methodology, analysis, results, etc.: 4%
– technical: 15% • correctness: 5% • depth: 5% • innova.on: 5%
– evalua.on and results: 10% • sound evalua.on metric: 3% • thoroughness in analysis and experimenta.on: 3%
• A word about ‘the curve’
24-‐Sep-‐12 76