3D Computer Vision and Video Computing Sensors Instructor: Zhigang Zhu City College of New York CSc I6716 Fall 2006 3D Computer Vision and Video Computing.

3D Computer Vision

and Video Computing SensorsSensors

Instructor: Zhigang ZhuCity College of New York

CSc I6716Fall 2006

3D Computer Vision and Video Computing

Topic 2 in Part 1: Sensors

3D Computer Vision

and Video Computing AcknowledgementsAcknowledgements

The slides in this lecture were adopted and modified from lectures by

Professor Allen HansonUniversity of Massachusetts at Amherst

3D Computer Vision

and Video Computing SensorsSensors

Static monocular reflectance data (monochromic or color) Films Video cameras (with tapes) Digital cameras (with memory)

Motion sequences (camcorders) Stereo (2 cameras) Range data (Range finder) Non-visual sensory data

infrared (IR) ultraviolet (UV) microwaves

Many more

3D Computer Vision

and Video Computing The Electromagnetic SpectrumThe Electromagnetic Spectrum

Visible Spectrum

700 nm 400 nm

C = f

f

3D Computer Vision

and Video Computing The Human EyeThe Human Eye

3D Computer Vision

and Video Computing The EyeThe Eye

The Retina: rods (low-level light, night vision) cones (color-vision) synapses optic nerve fibers

Sensing and low-level processing layer 125 millions rods and cones feed into 1 million nerve fibers Cell arrangement that respond to horizontal and vertical lines

Retina

RodsCones

3D Computer Vision

and Video Computing Film, Video, Digital CamerasFilm, Video, Digital Cameras

Black and White (Reflectance data only) Color (Reflectance data in three bands - red, green, blue)

3D Computer Vision

and Video Computing Color ImagesColor Images

Blue Green Red

‘Dimensions’ of an Image

Spatial (x,y)Depth (no. of components)Number of bits/channelTemporal (t)

Pixel

Spatial Resolution

Spectra Resolution

Radiometric Resolution

Temporal Resolution

3D Computer Vision

and Video Computing Across the EM SpectrumAcross the EM Spectrum

Crab Nebula

3D Computer Vision

and Video Computing Across the EM SpectrumAcross the EM Spectrum

Cargo inspection using Gamma Rays

Mobile Vehicle and Cargo Inspection System (VACIS®)

Gamma rays are typically waves of frequencies greater than 1019 Hz

Gamma rays can penetrate nearly all materials and are therefore difficult to detect

Courtesy:Science Applications International Corporation (SAIC),

3D Computer Vision

and Video Computing Across the EM SpectrumAcross the EM Spectrum

Cargo inspection using Gamma Rays

Mobile Vehicle and Cargo Inspection System (VACIS®)

Gamma rays are typically waves of frequencies greater than 1019 Hz

Gamma rays can penetrate nearly all materials and are therefore difficult to detect

Courtesy:Science Applications International Corporation (SAIC),

3D Computer Vision

and Video Computing Across the EM SpectrumAcross the EM Spectrum

Cargo inspection using Gamma Rays

Mobile Vehicle and Cargo Inspection System (VACIS®)

Gamma rays are typically waves of frequencies greater than 1019 Hz

Gamma rays can penetrate nearly all materials and are therefore difficult to detect

Courtesy:Science Applications International Corporation (SAIC),

3D Computer Vision

and Video Computing Across the EM SpectrumAcross the EM Spectrum

Medical X-Rays

3D Computer Vision

and Video Computing Across the EM SpectrumAcross the EM Spectrum

Chandra X-Ray Satellite

3D Computer Vision

and Video Computing Across the EM SpectrumAcross the EM Spectrum

From X-Ray images to 3D Models: CT Scans

3D Computer Vision

and Video Computing Across the EM SpectrumAcross the EM Spectrum

Flower Patterns in Ultraviolet

Dandelion - UV

Po

ten

till

a

3D Computer Vision

and Video Computing Across the EM SpectrumAcross the EM Spectrum

Messier 101 in Ultraviolet

3D Computer Vision

and Video Computing Across the EM SpectrumAcross the EM Spectrum

Traditional images

3D Computer Vision

and Video Computing Across the EM SpectrumAcross the EM Spectrum

Non-traditional Use of Visible Light: Range

3D Computer Vision

and Video Computing Across the EM SpectrumAcross the EM Spectrum

Scanning Laser Rangefinder

3D Computer Vision

and Video Computing Across the EM SpectrumAcross the EM Spectrum

IR: Near, Medium, Far (~heat)

3D Computer Vision

and Video Computing Across the EM SpectrumAcross the EM Spectrum

IR: Near, Medium, Far (~heat)

3D Computer Vision

and Video Computing Across the EM SpectrumAcross the EM Spectrum

IR: Finding chlorophyll -the green coloring matter of plants that functions in photosynthesis

3D Computer Vision

and Video Computing Across the EM SpectrumAcross the EM Spectrum

Microwave Imaging: Synthetic Aperture Radar (SAR)

San Fernando Valley Tibet: Lhasa River

Thailand: Phang Hoei RangeAthens, Greece

Red: L-band (24cm)

Green: C-band (6 cm)

Blue:C/L

3D Computer Vision

and Video Computing Across the EM SpectrumAcross the EM Spectrum

Radar in Depth: Interferometric Synthetic Aperture Radar - IFSAR(elevation)

3D Computer Vision

and Video Computing Across the EM SpectrumAcross the EM Spectrum

Low Altitude IFSAR

IFSAR elevation, automatic, in minutes

Elevation from aerial stereo, manually, several days

3D Computer Vision

and Video Computing Across the EM SpectrumAcross the EM Spectrum

Radio Waves (images of cosmos from radio telescopes)

3D Computer Vision

and Video Computing Stereo GeometryStereo Geometry

Single Camera (no stereo)

3D Computer Vision

and Video Computing Stereo GeometryStereo Geometry

P(X,Y,Z)

f = focal length

Optical Center

pr(x,y)

Film plane

pl(x,y)

Optical Center

f = focal length

Film plane

LEFT CAMERA RIGHT CAMERA

B = Baseline

3D Computer Vision

and Video Computing Stereo GeometryStereo Geometry

LEFT IMAGE RIGHT IMAGE

Disparity = xr - xl

P

Pr(xr,yr)Pl(xl,yl)

≈ depth

3D Computer Vision

and Video Computing Stereo ImagesStereo Images

A Short Digression

StereoscopesStereoscopes

3D Computer Vision

and Video Computing Stereo ImagesStereo Images

Darjeeling Suspension

Bridge

3D Computer Vision

and Video Computing Picture of you?Picture of you?

3D Computer Vision

and Video Computing StereoStereo

Stereograms

3D Computer Vision

and Video Computing Stereo X-RayStereo X-Ray

3D Computer Vision

and Video Computing Range SensorsRange Sensors

Light Striping

David B. Cox, Robyn Owens and Peter HartmannDepartment of BiochemistryUniversity of Western Australia http://mammary.nih.gov/reviews/lactation/Hartmann001/

3D Computer Vision

and Video Computing Why is Vision Difficult?Why is Vision Difficult?

Natural Variation in Object Classes: Color, texture, size, shape, parts, and relations

Variations in the Imaging Process Lighting (highlights, shadows, brightness, contrast) Projective distortion, point of view, occlusion Noise, sensor and optical characteristics

Massive Amounts of Data 1 minute of 1024x768 color video = 4.2 gigabytes

(Uncompressed)

3D Computer Vision

and Video Computing The Need for KnowledgeThe Need for Knowledge

Knowledge

Function

Context

Shape

SpecificObjects

GenericObjects

Structure Size

Shape

Motion

Purpose

Variation

3D Computer Vision

and Video Computing The Figure Revealed The Figure Revealed

Light Source

3D Computer Vision

and Video Computing The Effect of ContextThe Effect of Context

3D Computer Vision

and Video Computing The Effect of Context - 2The Effect of Context - 2

3D Computer Vision

and Video Computing Context, cont.Context, cont.

….a collection of objects:

3D Computer Vision

and Video Computing ContextContext

The objects as hats:

3D Computer Vision

and Video Computing

And as something else…..

‘To interpret something is to give it meaning in context.’

ContextContext

3D Computer Vision

and Video Computing Vision System ComponentsVision System Components

…..at the low (image) level, we need Ways of generating initial descriptions of the image data Method for extracting features of these descriptions Ways of representing these descriptions and features Usually, cannot initially make use of general world

knowledge

IMAGE(numbers)

DESCRIPTION(symbols)

3D Computer Vision

and Video Computing

….at the intermediate level, we need Symbolic representations of the initial descriptions Ways of generating more abstract descriptions from the initial ones (grouping) Ways of accessing relevant portions of the knowledge base Ways of controlling the processing

Intermediate level processes should be capable of being used top-down (knowledge-directed) or bottom-up (data-directed)

IMAGEIINTERMEDIATEDESCRIPTIONS

KNOWLEDGE

Vision System ComponentsVision System Components

3D Computer Vision

and Video Computing Vision System ComponentsVision System Components

….at the high (interpretation) level, we need Ways of representing world knowledge

Objects Object parts Expected scenarios (relations) Specializations

Mechanisms for Interferencing Beliefs Partial matches

Control Information Representations of

Partial interpretations Competing interpretations Relationship to the image descriptions

3D Computer Vision

and Video Computing NextNext

Anyone who isn't confused really doesn't understand the situation.

--Edward R. Murrow

Next:Image Formation

Reading: Ch 1, Ch 2- Section 2.1, 2.2, 2.3, 2.5

Questions: 2.1. 2.2, 2.3, 2.5

Exercises: 2.1, 2.3, 2.4