Structure from Motion
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Structure from Motion
Givenamo*onfieldes*matedfromanimagesequence
Computetheshape,orstructure,ofthevisibleobjects
Theirmo*onwithrespecttotheviewingcamera
3-D Motion and Object Structure
3/31/11 2ECEn631
3D Reconstruction CorrespondenceProblemEpipolarGeometry3DReconstruc9on
Bothintrinsicandextrinsicparametersareknown– Easytocalibrateforstereovision(fixedcameras)– Unambiguous(Eculideanstructure)
Onlytheintrinsicparametersareknowandes9mateextrinsicparameters(unknowncameramo9o)– Structurefrommo9onusingasinglecamera– Uptoanunknownscalingfactor(affinestructure)
Onlythepixelcorrespondencesaretheonlyinforma9onavailable(videofromunknownsources)– Uptoanunknownprojec9vetransforma9onoftheenvironment
3/31/11 ECEn631 3
3-D Reconstruction
EuclideanStructure AffineStructure(uptoascalefactor) Projec*veTransforma*on
3/31/11 4ECEn631
Reconstruction from a Sparse Motion Field
Largeaveragedisparitybetweenframes(slowframerates,fastmovingcamera,orveryshortobjectdistance)– Usestereovision‐likemethod,e.g.,8‐pointalgorithmappliedtoapairofframesata*me.
Smallaveragedisparitybetweenframes– Thereconstruc*ongainsstabilityandrobustnessfromthe*meintegra*onoflongsequencesofframes.
3/31/11 ECEn631 5
Large average disparity between frames
1. Calibrate camera 2. Acquire features (corners, SIFT, SURF, etc) 3. Match features (matchTemplate()) 4. Undistort feature points (not the entire image) 5. Estimate fundamental matrix F (use data
normalization and RANSAC) 6. Calculate essential matrix E from F and intrinsic
parameters 7. Calculate R and T 8. Bundle adjustment (i.e., estimation of 3D points and/
or cameras based on minimization of reprojection errors across all images and points)
3/31/11 ECEn631 6
8-point Linear Algorithm
Compute SVD of the new E=UDVT
Calculate R and T as
€
RZT (±π
2) =
0 ±1 01 0 00 0 1
Σ =σ 0 00 σ 00 0 0
=1 0 00 1 00 0 0
ˆ T =0 −Tz TyTz 0 −Tx−Ty Tx 0
€
E = ˆ T R ∈ R3×3Essential Matrix
3/31/11 7ECEn631€
R = URzT (±π
2)V T
ˆ T = URz (±π2
)ΣU T
Normalized the unknown scale factor
can be calculated if object size is
known.
Limitations
There is no guarantee that the estimated pose (R,T) is as close as possible to the true solution.
Even if we were to accept such an (R,T), a noisy image pair would not necessary give rise a consistent 3-D reconstruction.
3/31/11 8ECEn631
Small Average Disparity Factoriza*onMethod Assump*ons:
– Orthographiccameramodel– nimagepointscorrespondingtothenon‐coplanarscenepointshavebeentrackedinN>=3frames
Assump*on2almostmeansthattheprocessingstartsaVertheen*resequenceisacquired(notacceptableforsomeapplica*ons)
Cameracalibra*oncanbeignored(becauseassumingorthographicmodel),ifreconstructthe3‐Dpointsonlyuptoascalefactor
3/31/11 ECEn631 9
Pj
X
Y
Z
ii
ji ki
xij,yij
P1 P2
xi2,yi2 xi1,yi1
From a Sparse Motion Field
ii and ji are the unit vectors of the image reference frame, expressed in the world reference frame at time instant i. Ki is the cross product of the two and is the optical axis.
3/31/11 10ECEn631
Factorization Method
€
pij = [xij ,yij ]T jth image points at ith frame
j =1....n and i =1....NThe measurement matrix
W = XY where X and Y are two N×n
matrices with xijand yij as the entries.
It is simple to implement and gives numerical stable results.
3/31/11 11ECEn631
€
The registered measurement matrix
˜ W =˜ X ˜ Y
where ˜ X and ˜ Y are the X and Y
offset by the centroid ˜ x i and ˜ y i.
˜ x i =1n
xijj=1
n∑ and ˜ y i =
1n
yijj=1
n∑
3/31/11 12ECEn631
Factorization
3/31/11 ECEn631 13
€
The registered measurement matrix ˜ W can be factorized into a 2N × 3 rotation matrix R and an n× 3 shape matrix.˜ W = RS (proof skipped)
The factorization of ˜ W is not unique.˜ W = RQQ−1S is always true for any
invertible 3× 3 matrix Q.
Two constraints to help compute a factorization that is unique up to an unknown initial orientation of the world reference frame with respect to the camera frame:
The rows of R must have unit norm.
The first n rows of R (iiT) must be orthogonal to the corresponding n rows of (jiT) .
3/31/11 14ECEn631
€
The registered measurement matrix should have at most rank 3 : ˜ W = UDV T . But due to noise or imperfect matching, the rank of ˜ W is usually greater than 3 (More than three nonzero singular values along the diagonal.) This can be corrected by simply settingall but the three largest singular values in D to zero and recomputing a new ˜ W .SVD the new ˜ W to get new UDV T . Form D' by selecting the 3× 3 top left submatrix of the new D corresponding to the 3 argest singular values. Find columns corresponding to these singular value to form U' (2N× 3) and V' (n× 3), submatrics of U (2N×2N) and V (n ×n), respectively.
3/31/11 15ECEn631
€
ˆ R = U' D'1
2 =
i1T
i2T
.
.iNT
j1T
j2T
.
.jNT
ˆ S = D'1
2 V 'T = P1 P2 . . Pn[ ]
3/31/11 16ECEn631
2Nx3 rotation matrix
nx3 shape matrix
€
The rotation and shape matrices can be calaulated as R = ˆ R Q and S = Q−1 ˆ S to satisfy the constraints, where Q meets therequirements :ˆ i i
TQQTˆ i iT =1
ˆ j iTQQTˆ j i
T =1ˆ i i
TQQTˆ j iT = 0
3/31/11 17ECEn631
€
The rotation and shape matrices : R and SWhat about the translation?
€
Translation component parallel to the image plane ispropotional to the motion of the centroid of the data points on the image plane.Translation along the optical axis cannot be determined
3/31/11 18ECEn631
Reconstruction from a Dense Motion Field (Optical Flow)
StructureDensebutinaccuratees*matesofthemo*onfieldfromop*calflow
Itisaninstantaneousanalysisnotintegratedovermanyframes
3/31/11 19ECEn631
Method
Determinethedirec*onoftransla*onusingmo*onparallax
Determinealeast‐squaresapproxima*onoftherota*onalcomponentoftheop*calflowanduseittocomputedepth.
3/31/11 ECEn631 20
Motion Parallax - review
€
The relative motion field of two instantaneous conincident points does not depend on the rotational component of motion in 3 - D space.
Motion Parallax can be used to compute structure and motion from optical flow.
3/31/11 21ECEn631
Remember:
3/31/11 22ECEn631
Motion Parallax - review
€
Two points P and P in 3 - D Motion fields for P and P arevx = vx
T + vxω and vy = vy
T + vyω
v x = v xT + v x
ω and v y = v yT + v y
ω
If at an instant t the two points havethe same projection p = p = [x,y]T
, then vxω = v x
ωand vyω = v y
ω .
3/31/11 23ECEn631
Motion Parallax - review
€
By taking the difference between v and v
Δvx = vxT − v x
T = (Tz x −Tx f )( 1Z−
1Z
)
Δvy = vyT − v y
T = (Tz y−Ty f )( 1Z−
1Z
)
Δvx and Δvy (relative motion field) increase with the separation in depth between the two points P and P .
Rotation components cancel out. Relative motion field does not depend on rotation components.
3/31/11 24ECEn631
Motion Parallax - review
€
Δvx = vxT − v x
T = (Tz x −Tx f )( 1Z−
1Z
)
Δvy = vyT − v y
T = (Tz y−Ty f )( 1Z−
1Z
)
Δvy
Δvx=
Tz y−Ty fTz x −Tx f
=
y− fTy
Tz
x − f TxTz
=y− y0x − x0
For all possible rotational motions, the vector(Δvx
T ,ΔvyT ) points in the direction of p0 (the vanishing
point of th translation direction).
3/31/11 25ECEn631
Motion Parallax - review
Determine the direction of translation using motion parallax. Use two almost coincident points (approximation) to find the instantaneous epipole.
For two coincident points
For two very close points p and
€
ΔvxT =
Tz x −Tx fZ
−Tz x −Tx f
Z
ΔvyT =
Tz y−Ty fZ
−Tz y −Ty f
Z €
ΔvxT = vx
T − v xT = (Tz x −Tx f )( 1
Z−1Z )
ΔvyT = vy
T − v yT = (Tz y−Ty f )( 1
Z−1Z )
3/31/11 26ECEn631
The rotational component of the relative motion field
For two coincident points
For two very close points p and
€
vxω = v x
ω = −ω y f +ωz y+ω x xy
f−ω y x2
f
vyω = v y
ω =ω x f −ωz x −ω y xy
f+ω x y2
f
€
Δvxω = 0 and Δvy
ω = 0
€
Δvxω =ωz (y− y )+ω x
f(xy− x y )−
ω y
f(x2 − x 2 )
Δvyω = −ωz (x − x )+
ω y
f(xy− x y )−ω x
f(y2 − y 2 )
3/31/11 27ECEn631
€
ΔvxT =
Tz x −Tx fZ
−Tz x −Tx f
Z
ΔvyT =
Tz y−Ty fZ
−Tz y −Ty f
Z
€
ΔvxT = (Tz x −Tx f )( 1
Z−1Z )+ Tz
Z (x − x )
ΔvyT = (Tz y−Ty f )( 1
Z−1Z )+ Tz
Z (y− y )
3/31/11 28ECEn631
= 0 for two coincident points
Ifpandarecloseenough,alargerela*vemo*onfieldcanonlybeduetoalargedifferenceindepthbetweenthe3‐DpointsPand
3/31/11 29ECEn631
SFW from Optical Flow
Input– Intrinsicparametersfromtheviewingcamera– Adenseop*calflowfromasinglerigidmo*on
Convertimageplanevxandvytoimagepixelsusingcameraintrinsicparameters
3/31/11 ECEn631 30
€
vx =Tzx −Tx f
Z−ω y f +ωz y+
ω x xyf
−ω y x
2
f
vy =Tzy−Ty f
Z+ω x f −ωz x −
ω y xyf
+ω x y
2
f
Step 1 – Motion Field
Step 2 – Flow Differences Compute the flow differences between a point pi and all its neighbors in a small patch Qi and determine the eigenvalues and eigenvectors of the matrix.
Let λi be the greater eigenvalue and the unit eigenvector corresponding to λi .
is the direction of the relative motion field
3/31/11 31ECEn631
Step 3 – Epipoles Since relative motion field and the two points are coplanar,
€
( ˆ I i × pi )T p0 = 0
If there are N patches, then
€
ˆ I 1× p1T
ˆ I 2 × p2T
••
ˆ I N × pNT
p0 = Bp0 = 0
3/31/11 32ECEn631
Step 3 – Epipoles
Epipole p0 can be solved by using SVD on
€
λ1 0 0 0 00 λ2 0 0 00 0 • 0 00 0 0 • 00 0 0 0 λN
ˆ I 1× p1T
ˆ I 2 × p2T
••
ˆ I N × pNT
= UDV T
Estimate the epipole p0 as the olumn of V corresponding to the smallest singular value.
3/31/11 33ECEn631
Step 4 – Components of Angular Velocity
Using the intrinsic parameters and at each point pi of the image plane, we form the dot product of all N points
€
v⊥ = vxω (yi − y0 )− vy
ω (xi − x0 )Determine the angular velocity components as the least-squares solution of a system of N simultaneous equations (the above dot product equation).
3/31/11 34ECEn631
Step 5 – Translational Components
Use the intrinsic parameters and epipole coordinates to determine the translational components
€
x0 = f TxTz
y0 = fTyTz
3/31/11 35ECEn631
Step 6 – Depth
Solve for depth Z for each point using
€
vx =Tzx −Tx f
Z−ω y f +ωz y+
ω x xyf
−ω y x
2
f
vy =Tzy−Ty f
Z+ω x f −ωz x −
ω y xyf
+ω x y
2
f
3/31/11 36ECEn631
Related Documents
