Context-based Surface Completion Andrei Sharf, Marc Alexa, Daniel Cohen-Or.

Context-based Context-based Surface Surface

CompletionCompletion

Andrei Sharf, Marc Alexa, Daniel Cohen-OrAndrei Sharf, Marc Alexa, Daniel Cohen-Or

IntroductionIntroductionIntroductionIntroduction

Holes in models:• Imperfect range scanned

data:– Complex objects with non-visible

regions.– Misalignment of multiple-views

depth image scans.– Material reflections.

• Surface editing operations.

MotivationMotivationMotivationMotivation

Smooth filling is sufficient:• Small holes.

• Smooth surfaces.

Smooth filling is insufficient :• Surfaces containing fine

geometric detail.

• Topology of hole is more complex than a disk.

Context-based Completion Context-based Completion Context-based Completion Context-based Completion

Complete the missing region with patches that conform with its context

Smooth Context-based

MethodMethodMethodMethod

Import patches with matching context from the surface itself :

• Analyze surface characteristics.

• Find best matching patch.

• Fit imported patch to boundary.

Related WorkRelated WorkRelated WorkRelated Work

• Example-based image completion:[Drori et al. 2003; Jia and tang 2003; Sun et al. 2003;]

• Texture synthesis:[Efros and Leung 1999; Efros and Freeman 2001;Wei and Levoy 2000; Ying et al. 2001;]

• Smooth surface completion:[Curless and Levoy 1996; Davis et al. 2002; Ilic and Fua 2003; Verdera et al. 2003; Liepa 2003;]

• Model-based surface reconstruction [Savchenko et al. 2002]

• Curve analogies [Hertzmann et al. 2002]

Drori et al. 03

Liepa 03

Wei and Levoi 00

Hertzmann et al. 02

Moving to 3D SurfacesMoving to 3D SurfacesMoving to 3D SurfacesMoving to 3D Surfaces

Images have a regular spatial structure domain.

Problems in 3D:• Topology and geometry of missing region.

• Fitting a patch to the boundary of the missing region.

• Definition of similarity of shapes.

• Definition of a surface patch.

AlgorithmAlgorithmAlgorithmAlgorithm

• Given an incomplete shape• Create initial spatial subdivision• For each cell

– Compute a local shape representation.– Compute a shape signature.

• For each empty cell:– Find matching nonempty cell ω’.– Copy patch of ω’ into ω.

• Subdivide cells and repeat• Until completed region matches

its neighborhood.

AlgorithmAlgorithmAlgorithmAlgorithm

• Given an incomplete shape• Create initial spatial subdivision• For each cell

– Compute a local shape representation.– Compute a shape signature.

• For each empty cell:– Find matching nonempty cell ω’.– Copy patch of ω’ into ω.

• Subdivide cells and repeat• Until completed region matches

its neighborhood.

Shape RepresentationShape RepresentationShape RepresentationShape Representation

• Shape sampled point-set– Range scan output– Easy to merge

• Octree hierarchy on top of the point set

• Implicit surface approximation by fitting polynomials [Ohtake et al. 2003]:– General quadric:– Bivariate quadratic:– Edge or corner fitting

( ) t tQ x x Ax b x c 2 2( ) ( )Q x w Au Buv Cv Du Ev F

,i ip n

0Q x

Shape AnalysisShape AnalysisShape AnalysisShape Analysis

A local signature of a cell consists:• Implicit shape characteristics:

– Signed distance– Normal variation

• Detail amount:– Depth in octree

AlgorithmAlgorithmAlgorithmAlgorithm

• Given an incomplete shape• Create initial spatial subdivision• For each cell

– Compute a local shape representation.– Compute a shape signature.

• For each empty cell:– Find matching nonempty cell ω’.– Copy patch of ω’ into ω.

• Subdivide cells and repeat• Until completed region matches

its neighborhood.

Missing RegionMissing RegionMissing RegionMissing Region

Automatic identification:• Intersect shape approximation with cells.

• Empty cell that intersects surface is part of missing region.

Empty cell definition can alter due to refinement of missing region.

MatchingMatchingMatchingMatching

Distance metrics:• dc : Signature distance inside cells.

• da : Signature distance of adjacent cells.

• dl : Difference in amount of detail.

i c i a l ld( , ') = w d ( , ')+(1-w )d ( , ')+w d ( , ') Find most similar non-empty cell:

Matching-Candidate SetMatching-Candidate SetMatching-Candidate SetMatching-Candidate Set

• Non-empty cells of the same size

• Symmetry rotations of cells(/2, , ...)

• Rotation of all point-set with some angle (/4, /3, ...)

Transferring PointsTransferring PointsTransferring PointsTransferring Points

ICP:• Copy points into empty cell.

• Find closest point correspondence

• Align points rigidly

• Align points non-rigidly using polynomial form.

AlgorithmAlgorithmAlgorithmAlgorithm

• Given an incomplete shape• Create initial spatial subdivision• For each cell

– Compute a local shape representation.– Compute a shape signature.

• For each empty cell:– Find matching nonempty cell ω’.– Copy patch of ω’ into ω.

• Subdivide cells and repeat• Until completed region matches

its neighborhood.

David’s HairDavid’s HairDavid’s HairDavid’s Hair

Original Down-sampled

Smooth completion

Context-based completion

Completion ProcessCompletion ProcessCompletion ProcessCompletion Process

Original

Initial approximation

Final result

Manual Editing of Bunny ModelManual Editing of Bunny ModelManual Editing of Bunny ModelManual Editing of Bunny Model

Manual Editing of Knot ModelManual Editing of Knot ModelManual Editing of Knot ModelManual Editing of Knot Model

Scan of “Youth” Statue (Rear View)Scan of “Youth” Statue (Rear View)Scan of “Youth” Statue (Rear View)Scan of “Youth” Statue (Rear View)

Original Smooth Result

Scan of Human BoneScan of Human BoneScan of Human BoneScan of Human Bone

Original

Smooth

Result

Limitations: SemanticsLimitations: SemanticsLimitations: SemanticsLimitations: Semantics

Sample Rate vs. Detail Sample Rate vs. Detail FrequencyFrequencySample Rate vs. Detail Sample Rate vs. Detail FrequencyFrequency

SummarySummarySummarySummary

A fully automatic method to complete a missing region in a surface from its context.

• Completed patches geometrically conform with neighborhood.

• Incremental scale-space framework for finer approximation of the unknown region.

Future WorkFuture WorkFuture WorkFuture Work

• Explore with other spatial hierarchies.

• Enlarge the search space of examples by building a class based example set.

• Couple method with image completion methods for texture completion.

• Rotation and translation invariant signature

Acknowledgments:– David Levin and Olga Sorkine– Israel science foundation; Israeli ministry of science– Digital Michelangelo project 3D model repository; Stanford 3D

scanning repository; Darmstadt university of technology; Imager computer graphics laboratory of the university of British Columbia

Thank youThank youThank youThank you