11/25/03 3D Model Acquisition by Tracking 2D Wireframes Presenter: Jing Han Shiau M. Brown, T. Drummond and R. Cipolla Department of Engineering University.

11/25/0311/25/03

3D Model Acquisition by Tra3D Model Acquisition by Tracking 2D Wireframescking 2D Wireframes

Presenter: Jing Han ShiauPresenter: Jing Han Shiau

M. Brown, T. Drummond and R. CipollaM. Brown, T. Drummond and R. Cipolla

Department of EngineeringDepartment of Engineering

University of CambridgeUniversity of Cambridge

11/25/0311/25/03

MotivationMotivation

3D models are needed in graphics, reverse 3D models are needed in graphics, reverse engineering and model-based tracking.engineering and model-based tracking.

Want to be able to do real-time tracking.Want to be able to do real-time tracking.

11/25/0311/25/03

System Input/OutputSystem Input/Output

11/25/0311/25/03

Other ApproachesOther Approaches

Optical flow/ structure from motion Optical flow/ structure from motion (Tomasi & Kanade, 1992)(Tomasi & Kanade, 1992)

- Acquire a dense set of depth measurements- Acquire a dense set of depth measurements

- Batch method: not real-time- Batch method: not real-time Point matching between images.Point matching between images.

- Feature extraction followed by geometric constrai- Feature extraction followed by geometric constraint enforcementnt enforcement

Edge extraction followed by line matching between Edge extraction followed by line matching between 3 views using trifocal tensors3 views using trifocal tensors

11/25/0311/25/03

ImprovementImprovement

Previous approaches used siPrevious approaches used single line segments, but 2D ngle line segments, but 2D wireframes allow high level wireframes allow high level user constraints to reduce thuser constraints to reduce the number of degrees of freee number of degrees of freedom (6 degree of freedom Edom (6 degree of freedom Euclidean motion constraint) uclidean motion constraint) since each new line segment since each new line segment adds 4 degrees of freedom.adds 4 degrees of freedom.

11/25/0311/25/03

3D Positions of Lines3D Positions of Lines

Internal Camera parameters are known.Internal Camera parameters are known.

Initial and final camera matrices are known Initial and final camera matrices are known through querying robot (arm) for camera pose.through querying robot (arm) for camera pose.

Edge correspondence preserved using tracking.Edge correspondence preserved using tracking.

3D position of lines computed by triangulation.3D position of lines computed by triangulation.

11/25/0311/25/03

Single Line TrackingSingle Line Tracking

Sample points are initialized along each line Sample points are initialized along each line segment.segment.

Search perpendicular to the line for local Search perpendicular to the line for local maxima of the intensity gradient.maxima of the intensity gradient.

New line position is chosen to minimize the New line position is chosen to minimize the sum squared distance to the measured edge sum squared distance to the measured edge positions. positions.

11/25/0311/25/03

Single Line TrackingSingle Line Tracking

11/25/0311/25/03

Triangulation (Single Line tracking)Triangulation (Single Line tracking)

Finding 3D line by intersecting the rays Finding 3D line by intersecting the rays corresponding to the ends of the line in the corresponding to the ends of the line in the first image with the plane defined by the line first image with the plane defined by the line in the second image.in the second image.

11/25/0311/25/03

Finding 3D LineFinding 3D Line

Find the 3D line by intersecting the world line Find the 3D line by intersecting the world line defined by the point (defined by the point (u, vu, v) in the first image, ) in the first image, with the world plane defined by the linewith the world plane defined by the line

in the second, is equivalent to solving the in the second, is equivalent to solving the linear equationslinear equations

11/25/0311/25/03

LimitationsLimitations Object edges which project to epipolar lines mObject edges which project to epipolar lines m

ay not be tracked. ay not be tracked.

In case of a pure camera translation, epipolar liIn case of a pure camera translation, epipolar lines move parallel to themselves (radially with nes move parallel to themselves (radially with respect to the epipole); but the component of a respect to the epipole); but the component of a line’s motion parallel to itself is not observable line’s motion parallel to itself is not observable locally.locally.

11/25/0311/25/03

2D Wireframe Tracking2D Wireframe Tracking

Similar to line segment tSimilar to line segment tracking, least squares mracking, least squares method is used to minimizethod is used to minimize the sum of the squared e the sum of the squared edge measurements froedge measurements from the wireframe. m the wireframe.

11/25/0311/25/03

2D Wireframe Tracking2D Wireframe Tracking

The vertex image motions are stacked into the The vertex image motions are stacked into the P-dimensional vector P-dimensional vector pp, and the measurements , and the measurements are stacked into the D-dimensional vector are stacked into the D-dimensional vector dd00..

DD is the new measurement vector due to the m is the new measurement vector due to the motion otion pp, and , and MM is the DxP measurement matrix. is the DxP measurement matrix.

Least squares is used to minimize the sum squLeast squares is used to minimize the sum squared measurement error |ared measurement error |dd||22..

11/25/0311/25/03

2D Wireframe Tracking2D Wireframe Tracking

The least squares solution is:The least squares solution is:

But in general it is not unique. It can contain arBut in general it is not unique. It can contain arbitrary components in the right nullspace of M, bitrary components in the right nullspace of M, corresponding to displacements of the vertex icorresponding to displacements of the vertex image positions that do not change the measuremage positions that do not change the measurements. Adding a small constant to the diagonaments. Adding a small constant to the diagonal of M prevents instability.l of M prevents instability.

11/25/0311/25/03

3D Model Building3D Model Building 2D wireframe tracking preser2D wireframe tracking preser

ves point correspondence.ves point correspondence.

3D position of the vertices ca3D position of the vertices can be calculated from 2 views n be calculated from 2 views using triangulation.using triangulation.

Observations from multiple vObservations from multiple views can be combined by maiiews can be combined by maintaining a 3D pdf for each ventaining a 3D pdf for each vertex rtex pp(X).(X).

3D pdf is updated on the basis of the tracked image position of the point, and the known camera.

11/25/0311/25/03

3D Model Building3D Model Building

A 3D pdf has surfaces of constant probability A 3D pdf has surfaces of constant probability defined by rays through a circle in the image pdefined by rays through a circle in the image plane. This pdf is approximated as a 3D Gaussialane. This pdf is approximated as a 3D Gaussian of infinite variance in the direction of the ray n of infinite variance in the direction of the ray through the image point, and equal, finit, variathrough the image point, and equal, finit, variance in the perpendicular plane.nce in the perpendicular plane.

11/25/0311/25/03

3D Model Building3D Model Building

The 3D pdf is the likelihood of the tracked poiThe 3D pdf is the likelihood of the tracked point position, conditioned on the current 3D posint position, conditioned on the current 3D position estimate –tion estimate –pp(w|X). (w|X).

Multiply this by the prior pdf to get the posteriMultiply this by the prior pdf to get the posterior pdf:or pdf:

11/25/0311/25/03

3D Model Building3D Model Building

X is Gaussian with mean X is Gaussian with mean mmpp and covariance m and covariance matrix Catrix Cpp, , w|Xw|X is Gaussian with mean is Gaussian with mean mmll, covari, covariance matrix ance matrix CCll, and X|w is Gaussian with mea, and X|w is Gaussian with mean n mm and covariance matrix and covariance matrix C.C.

These are the Kalman filter equations used to These are the Kalman filter equations used to maintain 3D pdfs for each point.maintain 3D pdfs for each point.

11/25/0311/25/03

Triangulation (3D Model Building)Triangulation (3D Model Building)

Instead of using multiple rays that pass Instead of using multiple rays that pass through the image point as in the case of single through the image point as in the case of single line tracking, line tracking, probability distributionprobability distribution is used. is used.

11/25/0311/25/03

Combining Tracking and Model Combining Tracking and Model BuildingBuilding

There are 6 degrees of freedom corresponding There are 6 degrees of freedom corresponding to Euclidean position in space (3 translations ato Euclidean position in space (3 translations and 3 rotations) for a rigid body. nd 3 rotations) for a rigid body.

A wireframe of P/2 points has a P-dimensional A wireframe of P/2 points has a P-dimensional vector of vertex image positions.vector of vertex image positions.

11/25/0311/25/03

Model-based 2D TrackingModel-based 2D Tracking

The velocity of an image point for a normalizeThe velocity of an image point for a normalized camera moving with translational velocity d camera moving with translational velocity UU and rotating with angular velocity and rotating with angular velocity ww about its o about its optical center isptical center is

where where ZZcc is the depth in camera coordinates anis the depth in camera coordinates and (d (u, vu, v) are the image coordinates. ) are the image coordinates.

11/25/0311/25/03

Model-based 2D TrackingModel-based 2D Tracking

Stacking the image point velocities into a P-Stacking the image point velocities into a P-dimensional vector results in dimensional vector results in

Each vector Each vector vvii forms a basis for the 6D forms a basis for the 6D

subspace of Euclidean motions in P space. subspace of Euclidean motions in P space.

11/25/0311/25/03

Model-based 2D TrackingModel-based 2D Tracking

Pros:Pros:Converting a P degree of freedom tracking problem into a 6 Converting a P degree of freedom tracking problem into a 6

degree of freedom one.degree of freedom one. Cons:Cons:

The accuracy of the model (and the accuracy of the subspace The accuracy of the model (and the accuracy of the subspace of its Euclidean motion) is poor initially.of its Euclidean motion) is poor initially.

Conclusion: Conclusion: Accumulate 3D information from Accumulate 3D information from observations and progressively apply stronger observations and progressively apply stronger constraints. constraints.

11/25/0311/25/03

Probabilistic 2D TrackingProbabilistic 2D Tracking

A second Kalman filter is used to apply weightA second Kalman filter is used to apply weighted constraints to the 2D tracking. ed constraints to the 2D tracking.

The constraints are encoded in a full PxP prior The constraints are encoded in a full PxP prior covariance matrix. covariance matrix.

A Euclidean motion constraint can be included A Euclidean motion constraint can be included by using a prior covariance matrix of the form by using a prior covariance matrix of the form

11/25/0311/25/03

Probabilistic 2D TrackingProbabilistic 2D Tracking

Writing P as Writing P as and assume and assume λλii are i are independent to get:ndependent to get:

The variance of the image motion is large in thThe variance of the image motion is large in the directions corresponding to Euclidean motioe directions corresponding to Euclidean motion, and 0 in all other directions. n, and 0 in all other directions.

The weights can be adjusted to vary the strengtThe weights can be adjusted to vary the strength of the constraints.h of the constraints.

11/25/0311/25/03

Probabilistic 2D TrackingProbabilistic 2D Tracking

To combine tracking and model building, To combine tracking and model building, errors due to incorrect estimation of depth are errors due to incorrect estimation of depth are permitted, weighted by the uncertainty in the permitted, weighted by the uncertainty in the depth of the 3D point. depth of the 3D point.

Only components of image motion due to Only components of image motion due to camera translation depend on depth. camera translation depend on depth.

11/25/0311/25/03

Probabilistic 2D TrackingProbabilistic 2D Tracking

For a 1 standard deviation error in the inverse For a 1 standard deviation error in the inverse depth, the image motions are depth, the image motions are

Stacking the image point velocities into the P-Stacking the image point velocities into the P-dimensional vector to get dimensional vector to get

11/25/0311/25/03

Probabilistic 2D TrackingProbabilistic 2D Tracking

LetLet Then Then Ignore terms due to coupling between points to Ignore terms due to coupling between points to

get get

The depth variance for each point can be compThe depth variance for each point can be computed from its 3D pdf by uted from its 3D pdf by σσZcZc = = uuttCuCu, where , where uu is is a unit vector along the optical axis and a unit vector along the optical axis and CC is the is the 3D covariance matrix. 3D covariance matrix.

11/25/0311/25/03

Probabilistic 2D TrackingProbabilistic 2D Tracking

The final form of the prior covariance matrix isThe final form of the prior covariance matrix is

Which allows image motion due to Euclidean Which allows image motion due to Euclidean motion of the vertices in 3D, and also due to motion of the vertices in 3D, and also due to errors in the depth estimation of these vertices. errors in the depth estimation of these vertices.

11/25/0311/25/03

Basic IdeasBasic Ideas

1.1. Wireframe geometry specification via user input. CWireframe geometry specification via user input. Can occur at any stage, allowing objects to be reconsan occur at any stage, allowing objects to be recons

tructed in parts.tructed in parts.

11/25/0311/25/03

Basic IdeasBasic Ideas

2. 2D tracking Kalman filter. Takes edge measureme2. 2D tracking Kalman filter. Takes edge measurements and updates a pdf for the vertex image positions. nts and updates a pdf for the vertex image positions. Maintains a full PxP covariance matrix for the image Maintains a full PxP covariance matrix for the image

positions. positions.

11/25/0311/25/03

Basic IdeasBasic Ideas

3. 3D position Kalman filter. Takes known camera, a3. 3D position Kalman filter. Takes known camera, and estimate vertex image positions, and updates a pdf nd estimate vertex image positions, and updates a pdf for the 3D vertex positions. Maintains separate 3x3 cfor the 3D vertex positions. Maintains separate 3x3 c

ovariance matrices for the 3D positions. ovariance matrices for the 3D positions.

11/25/0311/25/03

Algorithm FlowAlgorithm Flow

Combined tracking and Combined tracking and model building model building algorithm.algorithm.

3D position updates are 3D position updates are performed performed intermittently. intermittently.

11/25/0311/25/03

ResultsResults

Real time tracking and 3D reconstruction of Real time tracking and 3D reconstruction of church image.church image.

11/25/0311/25/03

ResultsResults

ME block – constructed in 2 stages exploiting ME block – constructed in 2 stages exploiting weighted model-based tracking constraints. weighted model-based tracking constraints.

11/25/0311/25/03

ResultsResults

Propagation of 3D pdfs.Propagation of 3D pdfs.

Evolution of model from initial planar hypotheEvolution of model from initial planar hypothesis.sis.

11/25/0311/25/03

ResultsResults

Objects reconstructed Objects reconstructed using the Model using the Model Acquisition System, Acquisition System, with surfaces identified with surfaces identified by hand. by hand.

Computer generated Computer generated image using image using reconstructed objects. reconstructed objects.

11/25/0311/25/03

Thanks!Thanks!

Q&AQ&A

Happy Thanksgiving!!!Happy Thanksgiving!!!