Space Time Tracking Space Time Tracking ECCV 2002 ECCV 2002 Lorenzo Torresani Lorenzo Torresani Christoph Bregler Christoph Bregler
Jan 24, 2016
Space Time Tracking Space Time Tracking ECCV 2002ECCV 2002
Lorenzo TorresaniLorenzo Torresani
Christoph BreglerChristoph Bregler
OutlineOutline
ProblemProblemBackgroundBackgroundStructure from MotionStructure from MotionMatrix DecompositionMatrix DecompositionNon-Rigid Motion EstimationNon-Rigid Motion EstimationNon-Rigid Shapes EstimationNon-Rigid Shapes EstimationResultsResults
ProblemProblem
““To track feature points on non-rigid objects To track feature points on non-rigid objects without using any prior model”without using any prior model”
Rank of a MatrixRank of a Matrix
N
M
Rank (A) = Number of linearly independent vectors in
N
M
Rank (B) = Number of linearly independent vectors in
For M x N the Rank of A ≤ min (M,N)
Rank of a Matrix Rank of a Matrix cont’dcont’d
Rank C = ?
Columns of C are Linear combination of Columns of A
There are only N independent vectors in A Rank C = N
SVDSVD
SVD for a matrix A writes A as a product of three matrices:
• U • D• V
• Every m x n matrix has a singular value decomposition
Am x n
Um x n
Dn x n
VT
n x n
U,V have orthogonalcolumns
Frame 1 Frame 2 ………… Frame F
Tomasi Kanade Structure from MotionTomasi Kanade Structure from Motion
Given N 2D trajectories taken over F images, recover 3D Given N 2D trajectories taken over F images, recover 3D structure and motion (Camera pose)structure and motion (Camera pose)
• Assumption:• 3D Object is rigid• Orthographic Projection
• Tracks can be computed using any standard tracker (KLT etc)
Tomasi Kanade Structure from Motion cont’dTomasi Kanade Structure from Motion cont’d
Assume a set of P 3D points on a rigid object (structure)
S = [P1, P2 …….. PP ]
Orthographic Projection
where (u,v) are image coordinates and M is orthographic projection where (u,v) are image coordinates and M is orthographic projection matrixmatrix
Subtract mean of all u’s and v’s to center the world coordinate frame at the Subtract mean of all u’s and v’s to center the world coordinate frame at the center of the object.center of the object. This will get rid of T in the above equationThis will get rid of T in the above equation
2D coordinates of N points over F images can be 2D coordinates of N points over F images can be written in one matrixwritten in one matrix
W is called the measurement/tracking matrixW is called the measurement/tracking matrix
Rank 3
Tomasi Kanade Structure from Motion cont’dTomasi Kanade Structure from Motion cont’d
From W to R and SFrom W to R and S
Force the rank of W to be 3
SVD
StepsSteps
Matrix Decomposition of W matrix for non-Matrix Decomposition of W matrix for non-rigid objectsrigid objects
Estimate Motion Matrix using reliable set Estimate Motion Matrix using reliable set of pointsof points
Estimate shape basis (S) for all other Estimate shape basis (S) for all other feature points (unreliable)feature points (unreliable)
For Non Rigid ConstraintFor Non Rigid Constraint
3D Non Rigid Shape Model 3D Non Rigid Shape Model
Linear Combination of K Basis ShapesLinear Combination of K Basis Shapes Each basis shape is SEach basis shape is Sii 3 x P3 x P matrix describing P points matrix describing P points
S1 S2………………
SK
= l1S1 l2S2+ + … + lKSKS
Courtesy Christopher Bregler
Matrix DecompositionMatrix Decomposition Project P points of shape SProject P points of shape S Scaled Orthographic ProjectionScaled Orthographic Projection
Move world coordinate to object centeroid (This will get Move world coordinate to object centeroid (This will get rid of T)rid of T)
W Q M
-
Tracking MatrixTracking Matrix
Complete 2D Tracks or Flow
Rank of W 3K
2F x 3K 3K x P
In Tomasi Kanade it was 3
Non Rigid Motion EstimationNon Rigid Motion Estimation
Since Since W W isis rank-deficient, rank-deficient, Q Q can be estimated w/o can be estimated w/o the full availability of the full availability of WW
r <= 3Kr <= 3K point tracks will span the space of the point tracks will span the space of the trajectories of all the points (as rank of W is r)trajectories of all the points (as rank of W is r)
? = ?
W Q’ M’
known reliable tracks
Courtesy Christopher Bregler
r = 9
Trajectory ConstraintTrajectory Constraint
Q’
3D positions of point i for the K modes of deformation
t=2
t=1
t=F
. . . .= .
...frames
miwi : full trajectory
• Generate m trajectories (hypothesis) using Factored Sampling• Evaluate w by computing sum of square difference around point i.
Courtesy Christopher Bregler
• Each column mi of unreliable M is computed as expected value of posterior.
ResultsResults