Shape and Appearance Models from Multiple Images Richard Szeliski Microsoft Research Workshop on Image-Based Modeling and Rendering StanfordUniversity,

Shape and Appearance Models from Multiple Images

Richard Szeliski Richard Szeliski Microsoft ResearchMicrosoft Research

Workshop on Image-Based Modeling and RenderingWorkshop on Image-Based Modeling and RenderingStanfordUniversity, March 24, 1998.StanfordUniversity, March 24, 1998.

IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 2

Image-Based ModelingImage-Based Modeling

Create 3D models from one or more imagesCreate 3D models from one or more images– calibration and camera pose recoverycalibration and camera pose recovery– correspondence (matching, tracking, stereo)correspondence (matching, tracking, stereo)– 3D model construction3D model construction– appearance extractionappearance extraction

Mix with graphics & re-render (new views)Mix with graphics & re-render (new views)


Image-Based RenderingImage-Based Rendering

Render 3D graphics from imagesRender 3D graphics from images– sprite-basedsprite-based 3D rendering (Talisman) 3D rendering (Talisman)– view interpolationview interpolation: warp and blend (: warp and blend (morphmorph) )

between several imagesbetween several images– LumigraphLumigraph: full 2D manifold of images: full 2D manifold of images– layered depth images layered depth images ((LDILDIs): voxel-based s): voxel-based

representationrepresentation


Applications

““Desktop scanning” for 3D world and Desktop scanning” for 3D world and object building (“3D home page”)object building (“3D home page”)

Collaborative design (“3D fax”)Collaborative design (“3D fax”) Virtual environment construction Virtual environment construction

(virtual tourism, home sales/redesign)(virtual tourism, home sales/redesign) Video editing and special effects:Video editing and special effects:

uncalibrated, uncontrolled videouncalibrated, uncontrolled video


Shape and Appearance Representations

Depth mapsDepth maps

Volumetric modelsVolumetric models

Surface modelsSurface models

View-based representationsView-based representations

Scene decompositions: layers/spritesScene decompositions: layers/sprites


Outline

Volumes from silhouettesVolumes from silhouettes Surface meshes from matched curvesSurface meshes from matched curves Depth maps from stereoDepth maps from stereo Range data merging and surface modelingRange data merging and surface modeling Appearance recovery (texture maps)Appearance recovery (texture maps) (Sub-) pixel-accurate, multi-view stereo(Sub-) pixel-accurate, multi-view stereo Discussion and summaryDiscussion and summary


Volumes from Silhouettes

Start with collection of “calibrated” imagesStart with collection of “calibrated” images



Convert images into binary silhouettesConvert images into binary silhouettes



Intersect generalized cones (using octree)Intersect generalized cones (using octree)



Cup on turntable exampleCup on turntable example



Advantages:Advantages:– simple to implement, fairly robustsimple to implement, fairly robust– fast executionfast execution– complete (closed) surfacecomplete (closed) surface

Limitations:Limitations:– only produces only produces line hullline hull– limited resolutionlimited resolution– sensitive to classification (thresholding)sensitive to classification (thresholding)


3D Curves from Edges

““Feature-based” stereo matchingFeature-based” stereo matching



Extract extremal and internal edgesExtract extremal and internal edges



Match curves along epipolar linesMatch curves along epipolar lines

viewing rayviewing rayepipolar planeepipolar plane

epipolar lineepipolar line



Reconstruct 3D curves…Reconstruct 3D curves…

… is there a problem?



Silhouette curves don’t match in 3DSilhouette curves don’t match in 3D

Solution: fit circular arcs in epipolar planeSolution: fit circular arcs in epipolar plane



Coffee jar exampleCoffee jar example


3D Curves from Edges Advantages:Advantages:

– correct estimates at occluding contourscorrect estimates at occluding contours– good for smoothly curved objectsgood for smoothly curved objects– provides intrinsic surface estimatesprovides intrinsic surface estimates– works on interior surface markingsworks on interior surface markings

Limitations:Limitations:– fails in highly textured regionsfails in highly textured regions– fails in textureless fails in textureless interiorinterior areas areas– incomplete surface (not closed)incomplete surface (not closed)


Dense Stereo Matching

Compute Compute depth mapdepth map using correlation using correlation



Move correlation windows along epipolar Move correlation windows along epipolar lineslines

– projected window shape depends on surface orientationprojected window shape depends on surface orientation



View extrapolation resultsView extrapolation results

inputinput depth image depth image novel view novel view [Matthies,Szeliski,Kanade’88][Matthies,Szeliski,Kanade’88]



Newer view extrapolation resultsNewer view extrapolation results

inputinput depth imagedepth image novel viewnovel view



Compute certainty map from correlationsCompute certainty map from correlations

inputinput depth map certainty map depth map certainty map


Range Data Merging

Convert sparse depths to 3D points Convert sparse depths to 3D points

Aggregate with certainty weightingAggregate with certainty weighting[Soucy [Soucy et al., et al., Curless & Levoy, Puli, …]Curless & Levoy, Puli, …]



Advantages:Advantages:– gives detailed surface estimatesgives detailed surface estimates– multi-view aggregation improves accuracymulti-view aggregation improves accuracy

Limitations:Limitations:– narrow baseline narrow baseline noisy estimates noisy estimates– fails in textureless areasfails in textureless areas– sparse, incomplete surfacesparse, incomplete surface– sensitive to non-Lambertian effectssensitive to non-Lambertian effects


3D Surface Fitting

Convert 3D points into smooth surfaceConvert 3D points into smooth surface– physically-based oriented particlesphysically-based oriented particles

[Szeliski, Tonnesen & Terzopoulos][Szeliski, Tonnesen & Terzopoulos]– triangulation and mesh simplificationtriangulation and mesh simplification

[Hoppe [Hoppe et al., ...et al., ...]]– distance functions and isosurface extractiondistance functions and isosurface extraction

[Curless & Levoy][Curless & Levoy]


Oriented Particles

Use a collection of small surface elements Use a collection of small surface elements – local coordinates: position, normal, curvaturelocal coordinates: position, normal, curvature

– interaction potentials enforce smoothness interaction potentials enforce smoothness – simulate motions using dynamics simulate motions using dynamics – local triangulation/interpolation scheme local triangulation/interpolation scheme – topology changes occur automaticallytopology changes occur automatically

nn

ee11

nnee22

ee22

ee11

interactioninteraction


Oriented Particles

Interactive particle-based surface modelingInteractive particle-based surface modeling


Texture Map Recovery

For each model patch:For each model patch:– determine visibility (item buffer)determine visibility (item buffer)

– blend together textures (weight by view)blend together textures (weight by view)


Texture Map Recovery

3D model building example3D model building example

octree octree 3D curves 3D curves texture- texture-mappedmapped


Beyond Texture-Mapped Models

Capture view-dependent appearanceCapture view-dependent appearance– recovering BRDF recovering BRDF [Sato [Sato et al.et al., Yu & Malik], Yu & Malik]– view-dependent texture maps view-dependent texture maps [Debevec [Debevec et al.et al.]]– view interpolation view interpolation [Chen & Williams, …,[Chen & Williams, …,

Seitz & Dyer]Seitz & Dyer]– lightfield and Lumigraph lightfield and Lumigraph [Levoy & Hanrahan, [Levoy & Hanrahan,

Gortler Gortler et al.et al.]]


Lumigraph Example

acquisition stageacquisition stage volumetric modelvolumetric model novel novel viewview


Multi-Image Scene Recovery

Problems with “classical” approachProblems with “classical” approach– narrow baseline narrow baseline noisy results noisy results– single depth map misses informationsingle depth map misses information– ignores (or improperly treats) occlusionsignores (or improperly treats) occlusions– ignores mixed (partially transparent) pixelsignores mixed (partially transparent) pixels


Multi-Image Scene Recovery

Goals of new stereo algorithmGoals of new stereo algorithm

– simultaneously recover simultaneously recover disparitiesdisparities, , colorscolors, and , and

opacitiesopacities (c.f. blue screen matting) (c.f. blue screen matting)

– explicitly handle occlusionsexplicitly handle occlusions

– true multi-frame setting [Collins]true multi-frame setting [Collins]

– details in [Szeliski & Golland, ICCV’98]details in [Szeliski & Golland, ICCV’98]


Plane Sweep Stereo

Sweep family of planes through volumeSweep family of planes through volume

– each plane defines an image each plane defines an image composite homography composite homography

virtual cameravirtual camera

compositecompositeinput imageinput image

projectiveprojective re-sampling of ( re-sampling of (X,Y,ZX,Y,Z))


Plane Sweep Stereo

For each depth planeFor each depth plane– compute composite (mosaic) image — compute composite (mosaic) image — meanmean

– compute error image — compute error image — variancevariance– convert to confidence and aggregate spatiallyconvert to confidence and aggregate spatially

Select winning depth at each pixelSelect winning depth at each pixel


Plane Sweep Stereo

““Stack of acetates” model (related to LDI...)Stack of acetates” model (related to LDI...)

– warp and composite (warp and composite (overover) back-to-front) back-to-front

layers (“sprites”)layers (“sprites”)

synthesizedsynthesized imageimage


Plane Sweep Stereo

Compute Compute visibilityvisibility each input/layer pair each input/layer pair

Recompute means, confidences, and opacitiesRecompute means, confidences, and opacities

input imageinput image

layer compositelayer composite


Voxel Coloring

Generalizes plane sweep camera geometryGeneralizes plane sweep camera geometry– replace plane sweep with surface sweepreplace plane sweep with surface sweep

[Seitz & Dyer][Kutulakos & Seitz][Seitz & Dyer][Kutulakos & Seitz]


Voxel Coloring

Results for dinosaur and roseResults for dinosaur and rose


Stereo with Matting

Estimate fractional opacities for pixelsEstimate fractional opacities for pixels– adjust layer “sprites” (colors and opacities) to adjust layer “sprites” (colors and opacities) to

best match input imagesbest match input images– optimization criteria:optimization criteria:

re-synthesis errorre-synthesis error color and opacity smoothnesscolor and opacity smoothness prior distribution on opacitiesprior distribution on opacities

– corresponds to MAP Bayesian estimatorcorresponds to MAP Bayesian estimator


Stereo with Matting

SRI Trees sequence exampleSRI Trees sequence example

input images input images stereo layers stereo layers


Stereo with Matting

Advantages:Advantages:– true multi-image matchingtrue multi-image matching– deals with occlusions and mixed pixelsdeals with occlusions and mixed pixels

Limitations:Limitations:– too many degrees of freedom (volume)too many degrees of freedom (volume)– breaks up surfaces into “voxels”breaks up surfaces into “voxels”– no “sub-pixel” depthsno “sub-pixel” depths


Layered Stereo

Use arbitrarily oriented sprites Use arbitrarily oriented sprites [Baker,Szeliski,Anandan’98][Baker,Szeliski,Anandan’98]

Estimate 3D plane equation for each spriteEstimate 3D plane equation for each sprite

layers (“sprites”)layers (“sprites”)


Layered Stereo Demo

SpriteViewerSpriteViewer: renders sprites with depth: renders sprites with depth


Layered Stereo

Assign pixel to different “layers” (objects, Assign pixel to different “layers” (objects, sprites)sprites)


Layered Stereo Track each layer from frame to frame, Track each layer from frame to frame,

compute plane eqn. and composite mosaiccompute plane eqn. and composite mosaic

Re-compute pixel assignment by comparing Re-compute pixel assignment by comparing original images to spritesoriginal images to sprites


Layered Stereo

Resulting sprite collectionResulting sprite collection


Layered Stereo

Estimated depth mapEstimated depth map


Layered Stereo

Re-synthesize original or novel images Re-synthesize original or novel images from collection of spritesfrom collection of sprites


Layered Stereo Per-pixel residual depth estimationPer-pixel residual depth estimation

– plane plus parallaxplane plus parallax [Anandan [Anandan et al.et al.]]– model-based stereomodel-based stereo [Debevec [Debevec et al.et al.]]

– better accuracy / fidelitybetter accuracy / fidelity– makes makes forward warpingforward warping more difficult more difficult


Layered Stereo

Advantages:Advantages:– can represent occluded regionscan represent occluded regions– can represent transparent and border (mixed) can represent transparent and border (mixed)

pixels (sprites have pixels (sprites have alphaalpha value per pixel) value per pixel)– works on texture-less interior regionsworks on texture-less interior regions

Limitations:Limitations:– fails for high depth-complexity scenesfails for high depth-complexity scenes– may need manual initialization / controlmay need manual initialization / control


Image-Based Modeling & Rendering

Grand Unified Theory of Image-Based Grand Unified Theory of Image-Based Modeling and RenderingModeling and Rendering

Design continuumDesign continuum

3D models3D models imagesimages


Modeling & Rendering

Silhouettes Silhouettes volume volume Curves Curves 3D mesh 3D mesh Stereo Stereo depth map depth map Range data mergingRange data merging 3D surface modeling3D surface modeling Texture recoveryTexture recovery Multi-view stereoMulti-view stereo

3D texture-mapped 3D texture-mapped modelmodel

View-dependent View-dependent texture mapstexture maps

Sprites with depthSprites with depth Layered Depth ImagesLayered Depth Images Colored depth mapsColored depth maps LumigraphLumigraph LightfieldLightfield

viewsviews

objectsobjects


Open Problems

Automatic scene segmentationAutomatic scene segmentation Complex scenes: forests… Complex scenes: forests… Non-static scenesNon-static scenes Non-rigid motionNon-rigid motion Moving illumination, specularities, … Moving illumination, specularities, … … … … … but potential of IBMR looks greatbut potential of IBMR looks great


Acknowledgements

ColleaguesColleagues– CMUCMU: Takeo Kanade, Geoffrey Hinton,: Takeo Kanade, Geoffrey Hinton,

Larry MatthiesLarry Matthies

– DEC CRLDEC CRL: Demetri Terzopoulos, David Tonnesen, : Demetri Terzopoulos, David Tonnesen, Sing Bing Kang, James Coughlan, Richard WeissSing Bing Kang, James Coughlan, Richard Weiss

– MicrosoftMicrosoft: Michael Cohen, Steven Gortler, Radek : Michael Cohen, Steven Gortler, Radek Grzeszczuk, Polina Golland, Grzeszczuk, Polina Golland, Heung-Yeung Shum, Simon Baker, Anandan, Mei HanHeung-Yeung Shum, Simon Baker, Anandan, Mei Han

BibliographyBibliography– see http://www.research.microsoft.com/research/vision/szeliski/IBMRsee http://www.research.microsoft.com/research/vision/szeliski/IBMR


© Microsoft Corp., 1998© Microsoft Corp., 1998

Shape and Appearance Models from Multiple Images Richard Szeliski Microsoft Research Workshop on Image-Based Modeling and Rendering StanfordUniversity,

Documents

d u solution

silhouettes u surface

d surface fitting u

edges u advantages

silhouettes u advantages

edges u feature

d points u aggregate

edges u match curves