Srikumar Ramalingam Department of Computer Science University of California, Santa Cruz [email protected] 3D Reconstruction from a Pair of Images.

Srikumar Ramalingam Department of Computer Science University of California, Santa Cruz [email protected]

3D Reconstruction from a Pair of Images

- Problem Definition

- Previous Work

- Solution

- Experiments and Results

- Conclusion and Future work

Overview

Problem Definition

Image-1

Image-2

3D Texture-Mapped Model

Perspective View of the Model

Previous Work- Zhao, Aggarwal, Mandal and Vemuri, “3D Shape

Reconstruction from Multiple Views ”, Handbook of Image and Video Processing, pages 243-257, Al Bovik, 2000.

- Gang Xu and Zhengyou Zhang, “Epipolar Geometry in Stereo, Motion and Object Recognition”, Kluwer Academic Publishers, 1996.

- Zhang and Faugeras, “3D Dyanamic Scene Analysis-A Stereo Based Approach”, Springer-Verlag, 1992.

- Zhang, Deriche, Faugeras and Luong, “A Robust Technique for Matching Two Uncalibrated Images through the Recovery of the Unknown Epipolar Geometry”, INRIA Research Report, 1994.

- Zhang, “A New Multistage Approach to Motion and Structure Estimation: From Essential Parameters to Euclidean Motion Via Fundamental Matrix”, INRIA Research Report, 1996.

Previous Work Zhang, “Determining the Epipolar Geometry and its

Uncertainity: A Review”, INRIA Research Report, July 1996. Zhang, “A Flexible New Technique for Camera Calibration”,

Technical Report, Microsoft Research, 1998. Deriche and Giraudon, “A Computational Approach for Corner

and Vertex Detection”, INRIA Research Report, 1992.

Solution

-Feature Detection

-Getting Initial Set of Matches

-Medium Robust Correspondence

-Strong Robust Correspondence

-Camera Calibration

-3D Reconstruction

Feature Detection : Harris Corner Detection

Corner Threshold y)R(x,

)C(k.trace - )C(det y)R(x,

I II

II I C

^2

^

^2y

^

yx

^

yx

^ 2x^

Establishing Initial Set of Matches

Ambiguities in the Matches

Robust 1-1 Correspondence

-Medium Robust Matches

-Relaxation Techniques

-Strong Robust Matches

-Epipolar Geometry

Relaxation Techniques

- Winner-take-all

- Loser-take-nothing

- Some-winners-take-all - ( 1 – Max_Strength / Sec_Max_St)

Relaxation Strategies

End Result : No Ambiguities but False Matches

-Epipolar Geometry and Constraint

-Least Median of Squares

Strong Robust Estimation using Epipolar Geometry

Epipolar Geometry

0: 12Constraint mFmT

Fundamental Matrix (F) –3x3 matrix, which relates the corresponding points

Point corresponding to m lies on its epipolar line lm on the other image

Least Median of Squares – Removal of Outliers

),(),(

..3,2,1 )}({min

212

122

2

iT

iiii

ij

mFmdFmmdresidualr

mjrmed

- 8 Matches required for estimating F matrix

- Different combinations (m) of 8 matches selected

- Least median of squares algorithm is applied

If ri < Threshold, the match is discarded.

3D Reconstruction Problem is solved for Conventional Baseline Stereo System

X = b (xl+xr) / (2d)

Y = b (yl+yr) / (2d)

Z = bf / d

Intrinsic Parameters (5) Extrinsic Parameters (6)

f – focal length 3 rotational parameters, 3 translational parameters

u0, v0 – Center Intrinsic Matrix(A)

ku - unit length along x direction

kv – unit length along y direction

Angle between x and y direction

mnew(u,v) = A mold(x,y)

Need to conduct an experiment to calibrate the camera

Intrinsic and Extrinsic Parameters

0 1 0 0

0 v/sinθfk 0

0 u cotθfkfk

0v

0uu

3D Reconstruction- Triangulation

Robust Correspondence + Intrinsic Parameters Extrinsic Parameters

Robust Correspondence + Camera Parameters - 3D Points

TR

A

RTAP

IAP

tZYXPms

tZYXPms

TT

TT

,

]['

]0[

],,,[''

],,,[

2

1

Camera MatrixExtrinsic Parameters

Reconstructed 3D Model

Implementaton Pipeline

Matlab Implementations-Harris Corner Detection Algorithm (Deriche1992, Zhang1994)

-Initial Set of Matches Establishment (Zhang1994, Xu1996)

-Medium Set of Matches using Relaxation Techniques (Zhang1994, Xu1996)

-Strong Set of Matches using Epipolar Geometry (Zhang1994, Xu1996)

-Camera Calibration Experiment (Zhang1998)

-3D Points Reconstruction from Robust Matches and Camera Parameters (Zhang1994, Zhang1996, Xu1996)

-3D Polygonal Model Reconstruction (Delaunay Triangulation)

- Texture Mapping (OpenGL/C)

Standard Data Sets- Corner marked

Robust 1-1 Correspondence shown

Color Coding for Z Coordinates after 3D Reconstruction

3D Delaunay Triangulation

3D Texture Mapped Model – On Rotation

Real Data Sets and Results

Baskin Engineering Parking Scene – Two Images

Feature Points using Corner Detection process

Robust Set of Matches

Color Coding for Z Coordinates after 3D Reconstruction

Red-Max, Green – Intermediate, Blue – Min depths

3D Delaunay Triangulation

Texture Mapped 3D Model of the Scene

Perspective View of the Texture Mapped 3D Model

Camera Calibration Experiment

-Checker pattern

-3 images taken in different orientations

-Corners are marked

-Computation of camera parameters

Conclusion and Future Work

-Increasing the number of feature points

- Multiple Images

- Alternate Algorithms

- 3D Reconstruction of Urban Scenes (Faugeras 1995)

- Registration within GIS Data

Questions?