Multi-View Imagingchenlab.ece.cornell.edu/.../Tsuhan/20050908BrazilKeynote.pdf · 2005. 9. 8. · 1 New Frontier in Visual Communication and Networking: Multi-View Imaging Professor

1

New Frontier in Visual Communication and Networking:

Multi-View Imaging

Professor Tsuhan Chen 陳祖翰

[email protected]

Carnegie Mellon UniversityPittsburgh, USA

TsuhanChen2005

A 10-Year JourneyIEEE Multimedia Signal Processing (MMSP) Technical Committee, 1996~IEEE MMSP Workshops

Princeton 1997, Los Angeles 1998, Copenhagen 1999, Cannes 2001, St. Thomas 2002, Siena 2004, Shanghai 2005, Victoria 2006

IEEE International Conf on Multimedia (ICME)New York 2000, Tokyo 2001, Lausanne 2002, Baltimore 2003, Taipei 2004, Amsterdam 2005, Toronto 2006

IEEE Transactions on Multimedia, March 1999~Special issues: networked multimedia 2001, multimedia database 2002, multimodal interface 2003, streaming media 2004, MPEG-21 2005

2

TsuhanChen2005

First, let us talk about sampling…

Which way is it rotating?

TsuhanChen2005

Another Example

[“Beat the Devil” BMW Films 2002]

3

TsuhanChen2005

2WT numbers (in Fourier series) to represent a function of duration T and highest frequency W

Nyquist, 1928Gabor, 1946

Sampling Theorem

Shannon, 1949 (in communication theory)Whittaker, 1964 (in math)

MS ωω 2> Nyquist Rate

TsuhanChen2005

Harry Nyquist

Born in Sweden

PhD, Yale, 1917

AT&T/Bell Labs, 1917-1954

138 US patents Contributions

Quantitative study of thermal noiseVestigial sideband (VSB) transmissionNyquist diagram; stability of feedback systemsLots of signal transmission studies

1889 - 1976[IEEE History Center]

4

TsuhanChen2005

Fighting with the Nyquist Rate…

TsuhanChen2005

Super-ResolutionMultiple low-res images one high-res image

A “reconstruction” problem

5

TsuhanChen2005

How Is It Possible?

Does not really beat Nyquist…

TsuhanChen2005

It has limits in practice…

4 of these 16 of these 64 of these 256 of these

6

TsuhanChen2005

[Baker and Kanade, “Hallucinating Faces”]

We can beat Nyquist if we can…

One Single Image

images face possible all of number>>population worldhistory human36524606030 ××××××>>

Number of all possible 16×12 images 812162 ××=

Reconstruct

We can beat Nyquist with prior

TsuhanChen2005

Beating Nyquist with Stereo…

[Sawhney et. al 2001]

7

TsuhanChen2005

Beating Nyquist with Multimodality…

Input Video(Low Frame Rate)

Audio

Face/Lip-Tracking

Output Video(High Frame Rate)

TemporalSmoothing

Image Warping

Audio-to-Visual

Mapping Mouth Shapes(Viseme)

Mouth Shapesand Positions

[Chen, “Speech-assisted interpolation” 1993]

TsuhanChen2005

What else can we do to beat Nyquist?

Multiview Imaging…

8

TsuhanChen2005

3D vs. Multiview

[Digital Michelangelo Project, Stanford]

Light field of Michelangelo's statue of Night

3D (Model-Based Rendering) Multiview (Image-Based Rendering)

TsuhanChen2005

(Vx,Vy,Vz)(Vx,Vy,Vz)

(θ,ψ)(θ,ψ)

7D Plenoptic Function

[Adelson’91]

),,,,,,( tVVVf zyx λψθ

9

TsuhanChen2005

Lumigraph/Lightfield

v

us

t

(u0,v0)(s0,t0)

light ray

z

Object

p p p

Inner plane

Outer plane

4D

[Gortler et al ’96] [Levoy et al ’96]

TsuhanChen2005

Captured Images

4D Datas

t

u

v

10

TsuhanChen2005

Concentric Mosaics

3D

[Shum’99]

Cameras inward “The Matrix”

TsuhanChen2005

EyeVision

[Kanade’01]

After CorrectionBefore Correction

4D (incl. time)

Super Bowl XXXV

11

TsuhanChen2005

7D plenoptic function [Adelson’91]

5D: Stationary and monochrome [McMillan’95]

4D: Scene inside a bounded regionLumigraph [Gortler’96]Lightfield [Levoy’96]EyeVision [Kanade’01]

3D: Viewpoint along a trajectoryConcentric Mosaics [Shum’99]BulletTime [“The Matrix”]

2D: Viewpoint at a single positionPanorama [Chen’95], QuickTime VR

Summary

),,,,,,( tVVVf zyx λψθ

TsuhanChen2005

Where is research?

“Given a set of discrete samples (complete or incomplete) from the plenoptic function, the goal of IBR is to generate a continuous representation of that function” [McMillan’95]

Q: How many samples and where?

A: Need “Nyquist Sampling Theorem” for IBR

A redundancy-removal/compression problem

12

TsuhanChen2005

Recall u,v-s,t Parameterization…

t

v

0

0 0

t

v vv'f

z(v,t)

ft/z(v,t)

v

us

t

(u0,v0)(s0,t0)

light ray

z

Object

p p p

Inner plane

Outer plane

Need multidimensional spectral analysis

TsuhanChen2005

Spectral Analysis on t-v Plane

Intensity on t-v plane Spectrum

Intensity on t-v plane Spectrum

13

TsuhanChen2005

Sampling for IBR

tΩ

vΩ

tΩ

vΩ

0max =Ω−Ω vt fd

0min =Ω−Ω vt fd

0=Ω−Ω vtopt fd

(a) (b)B∆

Optimal renderingdepth determined by: π

π−

π

π−

• Lambertian surface• No occlusion• “Truncating window” analysis

[Chai et al, 2000]

Lowpass Filter

TsuhanChen2005

Optimal Sampling

tΩ

vΩ

π

π−

(a)

tΩ

vΩ

π

π−

(b)

B∆

“Fan” Filter

2x more compact50% fewer samples

Same rate as rectangular samplingEasier to design the filter

[Zhang and Chen, 2003]

14

TsuhanChen2005

We can do better than Nyquist…

Plenoptic functions are non-stationaryNon-Lambertian surfacesOcclusionPoor geometry

Non-uniform sampling is preferred

“Active IBR”Determine where to capture the imagesResulting a non-uniform sampling scheme

TsuhanChen2005

Why Non-Stationary?

Virtual View

Captured Views

C1

C4

C3C2

C

P Object

Poor GeometryVirtual View

Captured Views

C1

C4

C3C2

C

P Object

Non-LambertianVirtual View

Captured Views

C1

C4

C3C2

C

P Object

Occlusion

15

TsuhanChen2005

Concentric Mosaic as Example…

Force fromthe right

Force fromthe left

“Force” proportional to color inconsistency

TsuhanChen2005

Color Inconsistency

Virtual View

Captured Views

C1

C4

C3C2

C

P Object

4α3α2α

16

TsuhanChen2005

Example: Capturing

Non-Uniform Uniform

TsuhanChen2005

Example: Rendering

Non-Uniform Uniform

17

TsuhanChen2005

Self-Reconfigurable Camera Array

[Levoy, Stanford] [McMillan, MIT][Zhang and Chen, CMU]

TsuhanChen2005

Setup

18

TsuhanChen2005

DetailsReal-time capturing/calibration/rendering

48 webcams sensor network2 step-motors each (translation and pan)

Building the next versionMore mobile and wireless

TsuhanChen2005

NASA Successfully Demonstrates Innovative Nanosatellite System

http://www.sciencedaily.com/releases/2005/06/050619191628.htm

19

TsuhanChen2005

How about lighting? Beyond 7D…

Image-Based Relighting

TsuhanChen2005

Rendering vs. Relighting

Image-based renderingCapture the scene at multiple viewsInterpolate between views to achieve free-viewpointBut, all these under the same lighting condition

Image-based relightingCapture the scene under multiple lighting conditions“Interpolation” between lighting conditions

Actually, linear combination…

20

TsuhanChen2005

Problem Formulation

A system identification problemLighting patterns should be the basis functionsMost effective basis functions can be obtained by principal component analysis (PCA)

Image plane MM

N( )qpL ,

( )nmI ,

Q

P Lighting plane

( )qpF nm ,,

Surface Reflectance Function (SRF) Fm,n

TsuhanChen2005

Result

21

TsuhanChen2005

Result (cont.)

TsuhanChen2005

“Ultimate Photography”Experiences

“I move the camcorder around to capture the scene, but just can’t capture that immersive feeling”“I wish I had captured that angle/moment/object/ lighting…”

“Shoot at will, and render as wished”Decouple viewing from capturingImage-based rendering and relightingMosaicing/stitching (background)Both spatial and temporal interpolation

22

TsuhanChen2005

Afterthoughts…

TsuhanChen2005

Art Inspired by Sampling

12 x 16 LEDs, 8-bit Grayscale[Jim Campbell, “Portrait of a Portrait of Harry Nyquist’]

23

TsuhanChen2005

Art Inspired by Sampling

12 x 16 LEDs, 8-bit Grayscale[Jim Campbell, “Portrait of a Portrait of Claude Shannon”]

TsuhanChen2005

Art Inspired by Sampling

[Jim Campbell, “Running, Falling”]32 x 24 LEDs, 8-bit Red“It is the pixels or artifacts of the information that

are defocused by the screen…” — Jim Campbell

24

TsuhanChen2005

What does this say?

[http://www.palmyra.demon.co.uk]

Human is the best sampler…

TsuhanChen2005

Future of Visual Communication…“The most compelling shapes are those near to

our hearts: people’s faces, a gracefully moving body, a natural scene with rustling leaves and flowing water.

Evolution has tuned us to these sights.

By combining vision and graphics, capturing and creating images of these scenes may soon be within reach. …”

[Lengyel, 1998]

25

TsuhanChen2005

[Chen, SBrT’05]Forget about “visual communication” and “signal processing”!!!

as we defined them traditionally

The future is aboutMixed real and synthetic dataNew structured data

Multi-view, multi-sensor, etc.Unstructured data

It’s all about fighting with Nyquist

TsuhanChen2005

Advanced Multimedia Processing Lab

Please visit us at:

http://amp.ece.cmu.edu