This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1. Goksel Dedeoglu May 29, 2014 Embedded Lucas-Kanade Tracking:
How it Works, How to Implement It, and How to Use It
2. Copyright 2014 Texas Instruments, Inc. 2 Contributors for
Algorithms, Optimization, and Prototypes Goksel Dedeoglu
[email protected] Andrew Miller [email protected]
3. Copyright 2014 Texas Instruments, Inc. 3 Understanding
Motion: Sensors, Algorithms, Applications Application Requirements
static / moving camera frame rate motion range Algorithms model
fitting object tracking segmentation 3D reconstruction Automotive
Camera (visual) Motion Field 2D / 3D dense / sparse resolution
Accuracy (angular/magnitude) GPS Inertial Sensor Range Sensor
cross-traffic alert collision avoidance parking assist Human-Device
Interaction gesture recognition sign recognition facial expression
analysis Video Security crowd analysis action recognition traffic
analysis
4. Copyright 2014 Texas Instruments, Inc. 4 Putting the
Lucas-Kanade Tracker on the Map temporal support model complexity 2
frames 1 frame N frames single pixel kxk patch of pixels (u,v)
Lucas-Kanade
5. An Iterative Image Registration Technique with an
Application to Stereo Vision, Bruce Lucas and Takeo Kanade,
published in 1981. Tested and proven over 30+ years in practical
applications There exist generalizations to more complex object
& motion models There exist much-simplified versions that work
for very small displacements Good understanding of how & when
it works well Good Features to Track, Jianbo Shi and Carlo Tomasi,
published in 1994 Copyright 2014 Texas Instruments, Inc. 5
Lucas-Kanade Estimates Motion Between Consecutive Frames frame (t)
frame (t+1) + estimated motion (flow vectors) LK tracker
6. Assumption: brightness constancy Underdetermined system of
equations; additional constraints needed. Assuming the flow is
constant in a small neighborhood of pixels, estimate the
displacement (optical flow) vector h by minimizing Copyright 2014
Texas Instruments, Inc. 6 Understanding Lucas-Kanade kxk frame (t)
frame (t+1)displacement to be estimated h =
7. h 1. The objective function is 2. Linearize 3. Compute the
least-squares solution for the displacement h 4. Translate the
image G by hx and hy 5. Iterate until convergence! The Lucas-Kanade
Tracking Algorithm kxk kxk Copyright 2014 Texas Instruments, Inc.
7
8. h When the image regions are textured When the displacements
are small Common remedy: multi-resolution pyramids At the expense
of increased computation, more robust results expected with faster
numerical convergence The higher frame-rate the sensor, the better!
Copyright 2014 Texas Instruments, Inc. 8 When Does Lucas-Kanade
Work Best?
9. Copyright 2014 Texas Instruments, Inc. 9 How to Choose
Typical Parameters Ground Truth Motion Motion Key (in pixels) -5 +5
+5 -5 Scene patch size pyramidlevels
10. Copyright 2014 Texas Instruments, Inc. 10 Typical
Lucas-Kanade Tracking Pipeline Gaussian Pyramid Gradient Pyramid
Saliency Score Non-maximum Suppression Lucas-Kanade Feature Tracker
frame t output: flow vectors locate salient features estimate
motion of selected features multi-scale gradients multi-scale
images frame t-1
11. Copyright 2014 Texas Instruments, Inc. 11 Lucas-Kanade with
TIs Vision Library VLIB Gaussian Pyramid Gradient Pyramid Harris ,
FAST,ORB Non-maximum Suppression Lucas-Kanade Feature Tracker frame
t output: flow vectors locate salient features estimate motion of
selected features multi-scale gradients multi-scale images VLIB
VLIBVLIB VLIB VLIB frame t-1
13. Copyright 2014 Texas Instruments, Inc. 13 Studying the
Fixed-Point Approximation Floating Point (e.g., OpenCV) Fixed Point
Optimized VLIB Ground Truth Motion Motion Key (in pixels) -5 +5 +5
-5 Scene
14. Copyright 2014 Texas Instruments, Inc. 14 Live
Demonstration on TIs C6678 Keystone DSP
15. Copyright 2014 Texas Instruments, Inc. 15 Prototype:
Gesture Recognition rotate image left / right navigate to select
image
16. Computer Vision is evolving, with Open Source libraries
expanding fast, but with limited scrutiny & quality control.
There exist two Lucas- Kanade optical flow functions in OpenCV:
Embedded developers & architects beware! Copyright 2014 Texas
Instruments, Inc. 16 Quality Control for Lucas-Kanade
calcOpticalFlowPyrLK pyramid-based good results LK-brand
cvCalcOpticalFlowLK single-step algorithm can be 100x faster
handles very small motion now obsolete ground truth optical
flow
17. Lucas-Kanade is a well-understood & widely deployed
method for tracking feature points We have implemented an embedded
Lucas-Kanade tracker on the Keystone C6678 SoC; APIs are available
in TIs Vision Library VLIB Three key messages: Advantages: tested
& proven over 30+ years, works reliably in textured image
regions and small motion vectors Challenge: computationally
demanding, algorithmic extensions continue With the right
programmable processor and careful design trade- offs, Lucas-Kanade
can be implemented with cost & power consumption suitable for
embedded systems Copyright 2014 Texas Instruments, Inc. 17
Conclusions
18. An Iterative Image Registration Technique with an
Application to Stereo Vision, Bruce Lucas and Takeo Kanade,
Proceedings of the 7th International Joint Conference on Artificial
Intelligence (IJCAI '81), April, 1981, pp. 674-679. (URL)
Lucas-Kanade 20 Years On, project lead by Simon Baker & Iain
Matthews at the Robotics Institute of Carnegie Mellon University
(URL) Determining Optical Flow, Berthold Horn and Brian Schunck,
Artificial Intelligence, vol. 17, pp. 185-203, 1981. For related
algorithms, benchmarks and datasets, see Middlebury dataset:
http://vision.middlebury.edu/flow KITTI dataset:
http://www.cvlibs.net/datasets/kitti Copyright 2014 Texas
Instruments, Inc. 18 Resources for Further Investigation
19. PercepTonic is hosting a demo of the Lucas-Kanade tracker
on a TI Keystone DSP at the Technology Showcase. Please stop by!
Copyright 2014 Texas Instruments, Inc. 19 Hands-On
Demonstration