Image-based Plant Modeling Zeng Lanling Mar 19, 2008.

Image-based Plant ModelingImage-based Plant Modeling

Zeng LanlingMar 19, 2008

1.1.Image-based Plant ModelingImage-based Plant Modeling

2.2.Image-based Image-based Tree Tree Modeling Modeling

Long Quan, Ping Tan, Gang Zeng, Lu Yuan, Jingdong Wang, Sing Bing Kang*

The Hong Kong University of Science and Technology* Microsoft Research

Image-based Plant ModelingImage-based Plant Modeling

Long Quan, Ping Tan, Gang Zeng, Lu Yuan, Jingdong Wang, Sing Bing Kang*

The Hong Kong University of Science and Technology* Microsoft Research

MotivationMotivation

• Plants are ubiquitous but difficult to model

– Complex geometry and topology

– Fine texture details

• Previous methods have limitations

– Manual intensive

– Unintuitive

– Lack of realism

FeaturesFeatures

• Only a handheld camera is used for capture

• Ability to capture complex geometry and texture

• User interaction is small

Overview of systemOverview of system…

…

3D 2D

Image Capture

Structurefrom Motion

Leaf Segmentation Leaf Reconstruction

Branch Editing

Plant Model

Render

Overview of systemOverview of system…

…

3D 2D

Image Capture

Structurefrom Motion

Leaf Segmentation Leaf Reconstruction

Branch Editing

Plant Model

Render

captured images(35-45 images)

cloud of reliable 3D points

Image Capture and Image Capture and Structure from MotionStructure from Motion

• Hand-held camera

• Use quasi-dense approach [Lhuillier & Quan 2005]

… …

Overview of systemOverview of system…

…

3D 2D

Image Capture

Structurefrom Motion

Leaf Segmentation Leaf Reconstruction

Branch Editing

Plant Model

Render

Leaf SegmentationLeaf Segmentation

• Goal: Segment 3D points and images into individual leaves

• Problem: Segmentation is subjective and ill-posed

• Our solution: Joint segmentation with user interaction

3D segmentation3D segmentation

• Automatic joint segmentation

– Graph model with joint 2D/3D distance

– Graph partition

• Interactive refinement

– User interface

– Graph update

graph model

3D segmentation3D segmentation —— —— Construct Construct 3D graph3D graph

Graph G = { V, E }:

V: 3D points recovered from SFM

E: each point connected to its K-nearest neighbors

3D segmentation3D segmentation —— —— Define joint 2D/3D distanceDefine joint 2D/3D distance

Distance between two nodes

– 3D distance : 3D Euclidean distance

– 2D distance

3 ( , )Dd p q

.p.q

( ) ( )( ) ( ) 3D 2D

3D 2D

d p,q d p,qd p,q = 1 - α + α

2σ 2σ

3 ( , )Dd p q

)(maxmax),(

],[2 ugqpd i

qpuiD

ii

p q

d2d(p,q)

= gradient of i-th image ig

3D segmentation3D segmentation—— —— GraphGraph ppartitionartition

By normalized cut [Shi & Malik 2000]

after 3D graph partition initial 3D Graph

2D segmentation2D segmentation

By two-label graph-cut algorithm

– FG: region covered by projected 3D points in a group

– BG: projections of all other points not in the group

……

……

Segmented 2D leaves Clustered 3D points

Interactive Interactive rrefinementefinement

• Click to confirm segmentation

• Draw to split and refine

• Click to merge

Sample session of user interfaceSample session of user interface

3D 3D ggraph raph uupdatepdate

By two-label graph-cut problem

– Min-cut algorithm

– Real-time visual feedback

before update split stroke after update

Overview of systemOverview of system…

…

3D 2D

Image Capture

Structurefrom Motion

Leaf Segmentation Leaf Reconstruction

Branch Editing

Plant Model

Render

Model-based leaf reconstructionModel-based leaf reconstruction

• Generic leaf extraction

• Leaf reconstruction

– Flat leaf fitting

– Boundary warping

– Texture extraction

– Shape deformation

Generic leaf extractionGeneric leaf extraction

Extract a flat leaf mesh from image

Flat leaf fittingFlat leaf fitting

Estimate position, orientation, and scale by SVD decomposition of each 3D point set

Boundary warping & texturingBoundary warping & texturing

• Match leaf boundary to 2D segmentation boundary using iterative closest point (ICP) algorithm

• Crop texture after matching

leaf boundary

segmentation boundary

Shape deformationShape deformation

Move each vertex to the closest 3D point along normal of flat leaf

Overview of systemOverview of system…

…

3D 2D

Image Capture

Structurefrom Motion

Leaf Segmentation Leaf Reconstruction

Branch Editing

Plant Model

Render

Interactive Branch EditingInteractive Branch Editing

• Automatic reconstruction is difficult due to significant occlusion

• We rely on user to:

– Add branch

– Move branch

– Edit branch thickness (through radius)

– Specify leaf

Sample session of branch editingSample session of branch editing

NephthytisNephthytis

rendering resultmesh modelone source image(1 from 35)

PoinsettiaPoinsettia

one source image (1 from 35)

recovered model novel viewpoint

ScheffleraSchefflera

one source image(1 from 40)

recovered model

Indoor treeIndoor tree

one source image(1 from 45)

recovered model

Plant editingPlant editing

recovered model after texture replacement

Texture replacement

Plant editingPlant editing

original model after cut-and-paste

Branch cut-and-paste

Reconstruction statisticsReconstruction statistics

Nephthytis Poinsettia Schefflera Indoor tree

# image 35 35 40 45

# FG pts 53,000 83,000 43,000 31,000

# leaves 30 ≈ 120 ≈ 450 ≈ 1500

# UAL 6 21 69 35

Recovered leaves 29 116 374 1036

BET (min) 5 2 15 40

UAL = user assisted leaves, BET = branch edit time

ConclusionsConclusions

• Semi-automatic image-base plant modeling

– Simple capturing

– Realistic shape and texture

• Technical contributions:

– Interactive joint segmentation

– Model-based leaf reconstruction

– Interactive branch editing

Image-based Image-based TreeTree Modeling Modeling

Ping Tan, Gang Zeng *, Lu Yuan, Jingdong Wang, Sing Bing Kang, Long Quan

The Hong Kong University of Science and Technology* Microsoft Research

DifferentDifferent

Overviwe of the systemOverviwe of the system

Branch recoveryBranch recovery

• Reconstruction of visible branches

Graph construction

Conversion of sub-graph into branches

User interface for branch refinement

• Reconstruction of occluded branches

Unconstrained growth

Constrained growth

Visible branches recoveryVisible branches recovery

Occluded branches recoveryOccluded branches recovery

Leaves reconstructionLeaves reconstruction

• Mean shift filtering

• Region split or merge

• Color-based clustering

• User interaction

Mean shift filteringMean shift filtering

Leaves reconstructionLeaves reconstruction

Adding leaves to branchesAdding leaves to branches

• Create leaves from segmentation

• Synthesizing missing leaves

ResultsResults

ResultsResults

ResultsResults

ResultsResults

Approaches to plant modelingApproaches to plant modeling

• Rule-based

– Geometric rules [Weber&Penn 1995]

– L-system [Prusinkiewicz et al. 1994] [Noser et al. 01]

– Botanical rules [De Reffye et al. 1988]

• Image-based

– Volumetric [Shlyakhter et al. 2001] [Reche et al. 2004]

– Statistical [Han et al. 2003]

• Advantages:

– Impressive-looking plants, trees, and forests

• Disadvantages:

– Difficult to use for non-expert

– Difficult to exactly match appearance of actual plants

Rule-based plant modelingRule-based plant modeling

[Weber&Penn 1995]

[Prusinkiewicz et al. 1994]

[Phillippe De Reffye et al. 1988]

• Advantages:

– Details of real plant are captured in image

• Disadvantages:

– Limited realism (visual hull)

– Not manipulable (volumetric representation)

Image-based plant modeling Image-based plant modeling

[Reche et al. 2004]

[Shlyakhter et al. 2001]

[Han et al. 2003]

Thanks!Thanks!