CS 395: Adv. Computer Graphics Introduction to Image-Space Methods Image-Space Methods Watt & Watt: Chapter 8, "Pixel is Not a Little Square!" –AR Smith.

CS 395: Adv. Computer GraphicsCS 395: Adv. Computer Graphics

Introduction toIntroduction to

Image-Space MethodsImage-Space Methods

Watt & Watt: Chapter 8,Watt & Watt: Chapter 8,

"Pixel is Not a Little Square!" –AR Smith"Pixel is Not a Little Square!" –AR Smith

Jack TumblinJack Tumblin

[email protected]@cs.northwestern.edu

Image Space MethodsImage Space Methods

• A ‘Digital Image’ = a data structureA ‘Digital Image’ = a data structurefor displayed imagesfor displayed images

• ?What can it ?What can it dodo, besides display?, besides display?

what can we change?what can we change?– Position; Position; offset, stretch, squeeze, smear, cut,…offset, stretch, squeeze, smear, cut,…

Try It! (Java Applets)Try It! (Java Applets) http://www.mrl.nyu.edu/~hertzman/nudge/ http://www.mrl.nyu.edu/~hertzman/nudge/

– Combine:Combine: +/-*, Compositing, (depth images?) +/-*, Compositing, (depth images?)– Separate:Separate: matte or ‘segmenting’ matte or ‘segmenting’– Elaborate:Elaborate: make ‘more of the same’ picture make ‘more of the same’ picture

‘‘Real’ Images: Lens-Focused LightReal’ Images: Lens-Focused Light

• 3D3D2D Light Intensity Map 2D Light Intensity Map I(I(x,yx,y))• Angle(Angle(,,) ) Position( Position(x,yx,y) ) • ‘‘BlurringBlurring’—sharpness set by focus, lens quality’—sharpness set by focus, lens quality

I(I(x,yx,y))Image Plane Image Plane

IntensityIntensity

Ang

le(

Ang

le(

,, ))

Pos

ition

(P

ositi

on( x

,yx,y ))

Viewed SceneViewed Scene

‘‘Digital’ Images: 2D Grid of NumbersDigital’ Images: 2D Grid of Numbers

• NO intrinsic meaning, but ...NO intrinsic meaning, but ...• Widely assumed to representWidely assumed to represent

– Point Samples of a “smoothed” 2D intensity surfacePoint Samples of a “smoothed” 2D intensity surface– Uniform sampling pattern (but not always)Uniform sampling pattern (but not always)

xx

yy

(!weasel-word!)(!weasel-word!)

Real vs. DigitalReal vs. Digital

• Real Image == 2D Light Intensity map: I(x,y)Real Image == 2D Light Intensity map: I(x,y)

• Digital Image == 2D grid of numbers: I(m,n)Digital Image == 2D grid of numbers: I(m,n)

I(x,y)I(x,y)

x,yx,y

I(m,n)I(m,n)

m,nm,n

(pixels)(pixels)

Digital Images As VectorsDigital Images As Vectors

• ‘‘Stack up’ pixel values: VERY LONG vector Stack up’ pixel values: VERY LONG vector – 1 digital image == 1 point in N-dim. Space1 digital image == 1 point in N-dim. Space– Nearby points == Similar imagesNearby points == Similar images– All possible digital images: All possible digital images:

a grid of N-D pointsa grid of N-D points

– Space of all practical Space of all practical digital images: digital images:

• ~8Meg dimensions ~8Meg dimensions (2Mpix * RGBA) (2Mpix * RGBA)

• discrete, quantizeddiscrete, quantized

I(m,n)I(m,n)

m,nm,n

II0000

II0101

II0202

……II1010

II1111

II1212

……

I = I =

II00 00 I I01 01 I I02 02 II03 03 II04 04 II05 05


• Sensible element-by-element operations:Sensible element-by-element operations:– Add, subtract, scale two images: Add, subtract, scale two images:

0.5 +0.5 + (I (I11 - I - I2 2 ) = out) = out

-- ==II11 II22 outout


• Sensible element-by-element operations:Sensible element-by-element operations: ‘ ‘Alpha blending’ or ‘dissolve’:Alpha blending’ or ‘dissolve’:

--Weighted sum of two images: Weighted sum of two images: 0 < 0 < < 1 < 1

++ == II11 (1-(1-) I) I22 outout


Compositing:Compositing:

:: Use an Use an imageimage

as ‘opacity’:as ‘opacity’:

++ == .* I.* I11 (1-(1-) .* I) .* I22outout

Extended Compositing MethodsExtended Compositing Methods

• Environment MattesEnvironment Mattes

Alpha Matte Environment Matte Photograph

http://grail.cs.washington.edu/projects/envmatte/


Exploration of the space-of-all-images:Exploration of the space-of-all-images:• Image shift, scale, rotation, warp, etc.Image shift, scale, rotation, warp, etc.

as vector operation? awkward...as vector operation? awkward...• ‘‘Video Textures’ (Schodl et al., SIGG2000)Video Textures’ (Schodl et al., SIGG2000)

http://www.gvu.gatech.edu/perception/projects/videotexture/http://www.gvu.gatech.edu/perception/projects/videotexture/

• ‘‘Face Space’, ‘eigenfaces’, etc:Face Space’, ‘eigenfaces’, etc:– Gather many example images of facesGather many example images of faces

Use PCA to find desirable axes: Use PCA to find desirable axes: gender, age, facial expression...gender, age, facial expression...

• Depth images: is this still 2D? Depth images: is this still 2D?

Texture Synthesis: Learned VectorsTexture Synthesis: Learned Vectors

• Texture Synthesis/elaborationTexture Synthesis/elaborationEfros98, Wei/Levoy99, Ashikhmin2000,...Efros98, Wei/Levoy99, Ashikhmin2000,...

Input Sample

Synthesized Result


• Core Idea: Core Idea:

– for each pixel, for each pixel, make a vector of neighborhood pixel values:make a vector of neighborhood pixel values:

– Make a ‘dictionary’ of these vectors:Make a ‘dictionary’ of these vectors:given a vector , given a vector ,

– what are the most likely values of ?what are the most likely values of ?

Dictionary: crude, local pattern learningDictionary: crude, local pattern learning– Use dictionary to build a new image from Use dictionary to build a new image from

‘seed’ pixels/regions chosen at random.‘seed’ pixels/regions chosen at random.

II0000

II0101

II0202

……II1010

II1111

II1212

……

I = I =


• Refinements & Extensions:Refinements & Extensions:– Faster, simpler methods (Ashkhmin: real-time!)Faster, simpler methods (Ashkhmin: real-time!)

• http://www.cs.utah.edu/~michael/ts/http://www.cs.utah.edu/~michael/ts/ – Hertzmann(2001): Image analogiesHertzmann(2001): Image analogies

http://mrl.nyu.edu/projects/image-analogies/http://mrl.nyu.edu/projects/image-analogies/

• Given images A,B,C, ‘learn’ AGiven images A,B,C, ‘learn’ AB mapping. then find DB mapping. then find DSolves ‘A is to B’ as ‘C is to D’ problemsSolves ‘A is to B’ as ‘C is to D’ problems

• Recreates some local effects—brush strokes, blur, etc.Recreates some local effects—brush strokes, blur, etc.• Wider variety of user-guided reconstruction..Wider variety of user-guided reconstruction..

– Recent extensions: Recent extensions: video textures, depth, curves, shapes, video textures, depth, curves, shapes, textures on surfaces, colorizing B/W textures on surfaces, colorizing B/W

images...images...

Texture Synthesis: Modified VectorsTexture Synthesis: Modified Vectors

• ‘‘Image Inpainting’ –repair damaged images Image Inpainting’ –repair damaged images http://www.ece.umn.edu/users/marcelo/restoration.htmlhttp://www.ece.umn.edu/users/marcelo/restoration.html

• ‘‘Reaction/Diffusion’ textures: Reaction/Diffusion’ textures: http://www.gvu.gatech.edu/people/faculty/greg.turk/reaction_diffusion/reaction_diffusion.htmlhttp://www.gvu.gatech.edu/people/faculty/greg.turk/reaction_diffusion/reaction_diffusion.html

iteratively compute new pixel intensity from iteratively compute new pixel intensity from neighbors + rules. Image evolves over timeneighbors + rules. Image evolves over time

• Anisotropic Diffusion, Curvature Flow: Anisotropic Diffusion, Curvature Flow: iterative rules (PDEs) for progressive iterative rules (PDEs) for progressive edge-preserving image smoothingedge-preserving image smoothing

! NOT EQUIVALENT !! NOT EQUIVALENT !

? Digital image: pixel-by-pixel operations ? ? Digital image: pixel-by-pixel operations ?

? Real Image: point-by-point operations ?? Real Image: point-by-point operations ?

Source of much frustration & error!Source of much frustration & error!

Digital Image Digital Image ==== Real Image Real Image

Also see Alvy Ray Smith’s article entitled:Also see Alvy Ray Smith’s article entitled:

“A pixel is NOT a little square,“A pixel is NOT a little square,

a pixel is NOT a little square,a pixel is NOT a little square,

a pixel is NOT a little square!”a pixel is NOT a little square!” found here.found here.

Real Real DigitalDigital

– Image Image IntensityIntensity– Image Image Spectral DistributionSpectral Distribution (‘color’) (‘color’)– Image Image GeometryGeometry (Position,Angle, Distortion, …) (Position,Angle, Distortion, …)– Image Image ResolutionResolution (‘sharpness’ or ‘focus’) (‘sharpness’ or ‘focus’)

Real Images:Real Images: Smooth,Smooth,

Continuous, Continuous,

Variable, Variable,

CompleteComplete

Unlimited…Unlimited…

Digital Images:Digital Images: Sampled,Sampled, Discrete,Discrete, Quantized, Quantized, Fixed Fixed Limited …Limited …

Conversion Method?Conversion Method?

• Real Image == 2D Light Intensity map: I(x,y)Real Image == 2D Light Intensity map: I(x,y)

• Digital Image == 2D grid of numbers: I(m,n)Digital Image == 2D grid of numbers: I(m,n)

I(x,y)I(x,y)

x,yx,y

??

I(m,n)I(m,n)

m,nm,n

Digital Digital Real : ‘Reconstruction’ Real : ‘Reconstruction’

A pixel sets strength of a real A pixel sets strength of a real Basis functionBasis function(aka ‘reconstruction filter’s impulse (aka ‘reconstruction filter’s impulse

response’)response’)

I(m,n)I(m,n)

m,nm,n

00 0.50.5-0.5-0.5

1.01.0 I(x,y)I(x,y)

x,yx,y

‘‘Box’Box’




I(m,n)I(m,n)

m,nm,n

00 1.01.0-1.0-1.0

1.01.0 I(x,y)I(x,y)

x,yx,y

‘‘Linear’Linear’




I(m,n)I(m,n)

m,nm,n

00 1.01.0-1.0-1.0

1.01.0I(x,y)I(x,y)

x,yx,y

2.02.0

‘‘Cubic’Cubic’

RealRealDigital: ‘Sampling’Digital: ‘Sampling’

• If I(x,y) is ‘smooth enough’ already,If I(x,y) is ‘smooth enough’ already,

• ?Why not just grab I(x,y) values at (m,n) ??Why not just grab I(x,y) values at (m,n) ?

I(x,y)I(x,y)

x,yx,y

I(m,n)I(m,n)

m,nm,n

‘‘Smooth enough’ to Sample?Smooth enough’ to Sample?

• Because ‘smooth enough’ is Because ‘smooth enough’ is undefinedundefined;;

• Samples may hit spurious peaks, valleysSamples may hit spurious peaks, valleys

I(x,y)I(x,y)

x,yx,y

I(m,n)I(m,n)

m,nm,n

BAD!BAD!

Real Real Digital : ‘Pre-filter’ Digital : ‘Pre-filter’

‘‘Smoothing’ Defined : Linear Pre-filterSmoothing’ Defined : Linear Pre-filter

Pixel = Pixel = local weighted averagelocal weighted average of real image of real imagearound pixel sample pointaround pixel sample point

I(x,y)I(x,y)

x,yx,y

m,nm,n

00 1.01.0-1.0-1.0

1.01.0

2.02.0

‘‘Cubic’Cubic’weightsweights

WeightWeight1.01.0

Pre-Filter functionsPre-Filter functions

• Pre-filters == anti-aliasingPre-filters == anti-aliasing in Comp Graphics in Comp Graphics

• SHOULDSHOULD be done by be done by

continuous integration,continuous integration,

• USUALLYUSUALLY done by done by

super-samplingsuper-sampling

? Quality Measures ?? Quality Measures ?00 1.01.0-1.0-1.0

1.01.0

2.02.0

‘‘Bi-cubic’Bi-cubic’filterfilter

00 1.01.0-1.0-1.0

1.01.0‘‘Bi-linear’Bi-linear’

filterfilter

00 0.50.5-0.5-0.5

1.01.0

‘‘Box’Box’filterfilter

‘‘Smoothness Measures’Smoothness Measures’

• Sine wave magic:Sine wave magic:1)1) ALLALL (continuous)(continuous) functionsfunctions are a are a (possibly infinite)(possibly infinite)

weighted sum of sinusoids: weighted sum of sinusoids: f(x) = Af(x) = A00sin(wsin(w00x) + Bx) + B00cos(wcos(w00x)+x)+

A A11sin(wsin(w11x) + Bx) + B11cos(wcos(w00x)+…x)+…

2)2) ForFor ALL ALL FILTERS FILTERS (box, linear, cubic,…, anything…)(box, linear, cubic,…, anything…)

sinusoid in sinusoid in scaled, shifted sinusoid out scaled, shifted sinusoid out

THUS– compute response to ALL signals from sinusoids THUS– compute response to ALL signals from sinusoids

3)3) High-Frequency Sinusoids in a signal hold High-Frequency Sinusoids in a signal hold ALLALL of its sampling/aliasing troubles! of its sampling/aliasing troubles!

1-D Aliasing Example1-D Aliasing Example

Real image:Real image: I(x,y) =sin(2 I(x,y) =sin(2fx) f=3.0fx) f=3.0

Real

Aliasing: Aliasing: when bad conversionwhen bad conversion

scrambles the frequenciesscrambles the frequenciesof sinusoidsof sinusoids

1-D Aliasing Example1-D Aliasing Example


Sample at: Sample at: x(n) = n / 36.0 (~12 samples per cycle)x(n) = n / 36.0 (~12 samples per cycle)

Sample

Real

Aliasing ExampleAliasing Example



Reconstruct:Reconstruct: (use ‘box’ filter) (use ‘box’ filter) approximates f=3.0 approximates f=3.0

Sample

Real

Display


Real image: I(x,y) =sin(2Real image: I(x,y) =sin(2fx) f=3.0fx) f=3.0

Sample at: x(n) = n / 36.0 (~12 samples per cycle)Sample at: x(n) = n / 36.0 (~12 samples per cycle)

Reconstruct: (use ‘box’ filter) Reconstruct: (use ‘box’ filter) approximates f=3.0 approximates f=3.0

Another Real Image:Another Real Image: I(x,y) = sin(2 I(x,y) = sin(2fx) where f= 39.0fx) where f= 39.0

Sample

Real

Display

Real






Sample at: Sample at: x(n) = n / 36.0 (~0.92 samples per cycle) x(n) = n / 36.0 (~0.92 samples per cycle)

Sample

Real

Display

Sample

Real






Sample at: Sample at: x(n) = n / 36.0 x(n) = n / 36.0 (~0.92 samples per cycle)(~0.92 samples per cycle)

Reconstruct: Reconstruct: (use ‘box’ filter)(use ‘box’ filter) approximates f=3.0 !! approximates f=3.0 !!

Sample

Real

Display

Sample

Real

Display






Sample at: Sample at: x(n) = n / 144.0 x(n) = n / 144.0 ( 4.0 samples per cycle)( 4.0 samples per cycle)

Reconstruct: Reconstruct: (use ‘box’ filter)(use ‘box’ filter) approximates f=39.0 approximates f=39.0

Sample

Real

Display

SampleSample

Real

DisplayDisplay


? How much sampling is enough? ? How much sampling is enough?

• Depends on quality of Depends on quality of reconstructionreconstruction filter filter

• Rule-of-Thumb: at least 4 samples per cycle of the Rule-of-Thumb: at least 4 samples per cycle of the highest frequency sinusoidhighest frequency sinusoid

• Nyquist CriterionNyquist Criterion: 2 samples/cycle is theoretical min. : 2 samples/cycle is theoretical min.

Sample

Real

Display

SampleSample

Real

DisplayDisplay

Conclusion:Conclusion:

• Digital images are compact, indirect, Digital images are compact, indirect, ambiguous descriptions of light distributionsambiguous descriptions of light distributions

• Digital images are vectorsDigital images are vectors

• Digital image patterns can be learnedDigital image patterns can be learned

• Image-space manipulations must explicitly Image-space manipulations must explicitly address real-digital (discrete-continuous) address real-digital (discrete-continuous) conversions to avoid aliasing artifacts.conversions to avoid aliasing artifacts.




Reconstruct:Reconstruct: (use ‘box’ filter) (use ‘box’ filter) approximates f=3.0 approximates f=3.0


Sample at: Sample at: x(n) = n / 36.0 x(n) = n / 36.0

Reconstruct: Reconstruct: (use ‘box’ filter)(use ‘box’ filter) approximates f=3.0 approximates f=3.0

Sample

Real

Display

Sample

Real

Display

Image Space MethodsImage Space Methods

• Intro: What can we do with images?Intro: What can we do with images?• Re-sampling:Continuous & Discrete FiltersRe-sampling:Continuous & Discrete Filters

– Interpolation: What’s ‘in-between’ a pixel?Interpolation: What’s ‘in-between’ a pixel?– Antialiasing: What makes a good sampling?Antialiasing: What makes a good sampling?– Math Rigor: the Fourier TransformMath Rigor: the Fourier Transform– Examples: box, triangle, Mitchell-NetravaliExamples: box, triangle, Mitchell-Netravali

• Compositing: TransparencyCompositing: Transparency• Un-Compositing: Matte SeparationsUn-Compositing: Matte Separations• Further extensions: Further extensions:

– Environment MatteEnvironment Matte– Polynomial Texture MapPolynomial Texture Map– Texture Elaboration, Image Inpainting(Sigg2000)Texture Elaboration, Image Inpainting(Sigg2000)

OLDOLD

• ‘‘Image’Image’==A 2D map of light intensities from a lens==A 2D map of light intensities from a lens• ‘‘Digital Image’Digital Image’==a 2D grid of numbers (pixels)==a 2D grid of numbers (pixels)

• PROBLEMS: PROBLEMS: – What is What is betweenbetween all those points? all those points?– WhyWhy go digital? Why is it go digital? Why is it betterbetter than film? than film? – How can I combine the best of many pictures? Edit How can I combine the best of many pictures? Edit

easily?easily?

GOAL:GOAL:– More Flexibility; let me do more than just display!More Flexibility; let me do more than just display!

Image-Space TechniquesImage-Space Techniques

GOAL: More flexibility!GOAL: More flexibility!

• Compositing /Matte: cut-and-paste, Compositing /Matte: cut-and-paste, transparency , the ‘digital optical bench’ transparency , the ‘digital optical bench’ (Pixar`84)…(Pixar`84)…

• Warp: image as a ‘rubber sheet’, you can Warp: image as a ‘rubber sheet’, you can cut, stretch, and change at willcut, stretch, and change at will

• Environment Matte (Salesin99…)Environment Matte (Salesin99…)

• Video Textures(Schodl 2000)Video Textures(Schodl 2000)

• Polynomial Texture Map (Malzburg2001)Polynomial Texture Map (Malzburg2001)

CS 395: Adv. Computer Graphics Introduction to Image-Space Methods Image-Space Methods Watt & Watt: Chapter 8, "Pixel is Not a Little Square!" –AR Smith.

Documents