1 Visual Perception in Humans and Machines Kostas Daniilidis Assistant Professor GRASP Lab University of Pennsylvania.

1

Visual Perception in Humans and

Machines

Kostas DaniilidisAssistant Professor

GRASP LabUniversity of Pennsylvania

2

Examples• How do we (humans) recognize faces ?

Make a machine find President Clinton’s face in the web

3

4

An interdisciplinary definition

• Computer Vision is devoted to the discovery of algorithms, representations, and architectures that embody the principles of visual capabilities.

• What are visual capabilities?– Recognizing objects and faces– Estimating shapes and distances– Moving, grasping

5

Relation to other fields

• Computer Vision is inspired from Biological Vision (Phenomenology and Models in Psychophysics and in Neurobiology) but does not try to imitate the nature's architecture or algorithms.

• Biological Vision and Psychophysics may find computational models discovered in Computer Vision useful for explaining nature.

6

Target problem in computer vision

• Compute properties of the 3D world from one or more digital images

• These properties may be– dynamic (observer and object motion)– geometric (distances, object shape)– enabling recognition

• The result may be an action (grasp an object, avoid an obstacle)

7

What is an image ?

• A gray-value image is just a set of numbers (usually from 0 to 255)

8

An image is a set of numbers

175 189 190 188 199 197 196 193 181 189 191 194 198 196 191 179 189 191 197 198 200 195 173 129 192 194 198 200 194 161 116 116 198 200 200 190 152 113 116 119 201 202 185 135 105 103 114 119 205 180 121 89 104 101 109 114

177 105 88 90 100 103 101 105

9

An image is a surface I(x,y)

10

Basic image processing operations

• Smoothing and Noise Removal

11

Blur removal

12

Edge detection•X derivativeX derivative

•Y derivativeY derivative

•Gradient magnitudeGradient magnitude

•After thresholdingAfter thresholding

13

(Sub) sampling• Shannon Theorem: Sampling frequency

must be greater than the maximal frequency in the image (therefore smooth before subsample)

14

Brightness perception

15

How do we perceive distances?

• Perspective distortion in texture, contour, shading, and a-priori knowledge

• Stereopsis (what most people believe) • Motion

16

The Eye as a Pinhole Camera:

Perspective Projection

u = X/Zu = X/Z

ZZ

XX

17

Ames’ Illusion

18

Perspective Illusions

• A-priori-knowledge bias

19

Quiz

• From which points in space is a rectangle viewed as a square (more difficult: an ellipse viewed as a circle ?

•Be careful: The center ofBe careful: The center of

the ellipse in the imagethe ellipse in the image

is not the projection of theis not the projection of the

center of the ellipse on the floor!center of the ellipse on the floor!

20

The power of vanishing points

• Perspective projection preserves cross-ratio = AC/AD : BC/BD is the same on

the street and in the image. If A is a vanishing point AC/AD = 1.

AA

BB

CC

DD

We measure A,B,C,DWe measure A,B,C,D

in pixels in the imagein pixels in the image

and form cross ratioand form cross ratio

for image and for the for image and for the street. BC is street. BC is computed from the computed from the equality of the two equality of the two ratios.ratios.

21

Stereopsis

Infer depth from the disparity between the positions of the same feature in left and right image

22

Stereo Reconstruction

23

Stereo-disparity estimation

• Search at every pixel for candidates of maximum correlation between left and right

• Estimate 3D-coordinates of point

24

The power of visual motion

• Most of the animals have monocular vision (left and right visual fields do not overlap)

• 8% of the population can not see stereo • Stereopsis is limited to a very short

depth of field (10m).

25

26

Kinetic depth effect(moving dots)

kde.mov

joh.mov

27

Self- and object-motion

ins2.gif

28

Motion artifacts

Harley.mov

29

Structure from Motion

Given a sequence of imagesfind

1. Ego-motion2. 3D-structure3. Independent motions

applying only the assumption of rigidity.

30

Motion Field and Heading Direction

31

Depth map

32

Temporal aliasing

Wagon Wheel Illusion: A wheel with a periodic radial pattern is perceived to move backwards depending on the relation between the speed, the radius of the wheel, and the period of the pattern (www.cstr.ed.ac.uk/~rjc/wagonWheel)

33

Aperture problem

Inside a small aperture displaying a small line we can estimate only the motion direction perpendicular to the line.

Aperture.mov

34

•My wife and my mother-in-law

The role of the focus of attention