Lecture 7-Optical Flow

CAP5415-ComputerVisionLecture7-OpticalFlow

[email protected]

Lecture7:OpticalFlow

1

Readings• Szeliski,R.Ch.7• Bergenetal.ECCV92,pp.237-252.• Shi,J.andTomasi,C.CVPR94,pp.593-600.• Baker,S.andMatthews,I.IJCV2004,pp.221-255.

• SlideCredits:Szeliski andShah.• NewYorkTimes• VideofromMichaelBlack

2


Seeingthevisibleintheinvisible

3


Motion

4


U-zu-maki Glasses=>U:rabbits,zu:figure,maki:rotation

Reasons?

• thepatternsonlymovewhenyoublinkormoveyoureyes

• thearrangementofthebackgroundsofthe‘rabbits’determineswhichwaythepatternsrotate.

5


MotionPerceptionofchangeintheoutsideworldisofthehighestimportanceforalllivingbeings.

Agreatmajorityofchanges insensoryexcitationintheimageresultfrommotion.

Imagesalsochangeverymarkedlywhenalightsourceisturnedonoroff-butthischangecannotbeinterpretedasmotion.

6


WhyEstimateVisualMotion?

• VisualMotioncanbeduetoproblems– Camerainstabilities,jitter

7




• VisualMotion Indicatesdynamicsinthescene–Movingobjects,behavior,trackingobjects,analyzetrajectories

8




• VisualMotion Indicatesdynamicsinthescene–Movingobjects,behavior,trackingobjects,analyzetrajectories

• VisualMotionrevealsspatiallayout–Motionparallax

9


Today’sLecture

VisualMotionEstimation• Patch-basedmotion– OpticalFlow

• Lucas-Kanade• Horn-Schunck

• MotionModels

10


Applications

11


Motionbasedsegmentation

Structurefrommotion

Alignment(e.g.,UAVanalysis)

VideoCompression

TumorTracking

DeformationAnalysis

Nature News-Vol 525,Issue7567

12


• CVToolsthattrackhowanimalsmovearehelpingresearcherstodoeverythingfromdiagnosingneurologicalconditionstoilluminatingevolution

Higher-speedcameraseventuallyimprovedwhatcouldbecaptured.Butmovementstudiesstillneededapersontolookthroughtheresultsframebyframe,laboriouslytracingthearcofeachstep,armswingorwingflaptoextractinformationaboutanglesandforces.

13


PerceptionofMotion

14


Video

• Avideoisasequenceofframescapturedovertime– Imagedataisafunctionofspace(x,y)andtime(t)

15


I(x,y,t)

tTemporalchange

ApparentMotion

16


Presentdotsonthreemovieframes Seeonemovingdot

DescribingMotion

17


DescribingMotion• Simplestway:ImageDifferencing(intensityvalues)

18


DescribingMotion• Simplestway:ImageDifferencing(intensityvalues)

19


t

t+1

OpticalFlow

• Referstotheproblemofestimatingavectorfieldoflocaldisplacementinasequenceofimages.

20


OpticalFlow


• Whenwefixourattentiontoasinglepointandmeasurevelocitiesflowingthroughthatlocation,thentheproblemiscalledopticalflow.

21


OpticalFlow


• Whenwefixourattentiontoasinglepointandmeasurevelocitiesflowingthroughthatlocation,thentheproblemiscalledopticalflow.– Stereomatching,imagematching,tracking,…

22


EstimatingOpticalFlow

23


• Assumetheimageintensityisconstant

( )tyxI ,, ( )dttdyydxxI +++ ,,=

I

Time=t Time=t+dt


24


( )tyxI ,, ( )dttdyydxxI +++ ,, !


25


( )tyxI ,, ( )dttdyydxxI +++ ,, !I(x(t) + u.�t, y(t) + v.�t)� I(x(t), y(t), t) ⇡ 0


26



AssumingIisdifferentiablefunction,andexpandthefirsttermusingTaylor’sseries:

@I

@x

dx

dt

+@I

@y

dy

dt

+@I

@t

= 0


27



AssumingIisdifferentiablefunction,andexpandthefirsttermusingTaylor’sseries:

@I

@x

dx

dt

+@I

@y

dy

dt

+@I

@t

= 0

Ix

u+ Iy

v + It

= 0Compactrepresentation

Brightnessconstancyconstraint

Assumption:TheBrightnessConstraint

• Expressestheideaofsimilarbrightnessforthesameobjectsobservedinasequence

28




• Whenwefollowwithagivenlocationandtracetheirpositioninconsecutiveimagesofasequence,thentheproblemiscalled“featuretracking”

29




• Whenwefollowwithagivenlocationandtracetheirpositioninconsecutiveimagesofasequence,thentheproblemiscalled“featuretracking”

• Tryingtosolveasingleequationfortwovariables(uandv)àill-posed

30


Ix

u+ Iy

v + It

= 0 ApertureProblem

ApertureProblem

• Howcertain arethemotionestimates?

31


SecondAssumption:GradientConstraint

32


Velocityvectorisconstantwithinasmallneighborhood(LUCASANDKANADE)


33



E(u, v) =

Z

x,y

(Ix

u+ I

y

v + I

t

)2dxdy


34



E(u, v) =

Z

x,y

(Ix

u+ I

y

v + I

t

)2dxdy

@E(u, v)

@u=

@E(u, v)

@v= 0


35



E(u, v) =

Z

x,y

(Ix

u+ I

y

v + I

t

)2dxdy

@E(u, v)

@u=

@E(u, v)

@v= 0

2(Ix

u+ Iy

v + It

)Ix

= 0

2(Ix

u+ Iy

v + It

)Iy

= 0

Lucas-Kanade

36


PI2x

PIx

IyP

Ix

Iy

PI2y

� uv

�= �

PIx

ItP

Iy

It

�

Lucas-Kanade

37


PI2x

PIx

IyP

Ix

Iy

PI2y

� uv

�= �

PIx

ItP

Iy

It

�

Txx

Txy

Txy

Tyy

� uv

�= �

Txt

Tyt

�StructuralTensorrepresentation

Lucas-Kanade

38


PI2x

PIx

IyP

Ix

Iy

PI2y

� uv

�= �

PIx

ItP

Iy

It

�

Txx

Txy

Txy

Tyy

� uv

�= �

Txt

Tyt

�StructuralTensorrepresentation

u =Tyt

Txy

� Txt

Tyy

Txx

Tyy

� T 2xy

and v =Txt

Txy

� Tyt

Txx

Txx

Tyy

� T 2xy

39


HowdoesLucas-Kanade behave?

40


Texturedregion

Txx

Txy

Txy

Tyy

� uv

�= �

Txt

Tyt

�

Homog.region

EigenvaluesofmatrixTarehigh Eigenvaluesof

matrixTareverysmall

Atedges,matrixTbecomessingular!

Pitfalls&Alternatives

• Brightnessconstancyisnotsatisfied– Correlation basedmethodcouldbeused

41




• Apointmaynotmovelikeitsneighbors– Regularization basedmethods

42




• Apointmaynotmovelikeitsneighbors– Regularization basedmethods

• Themotionmaynotbesmall(Taylordoesnothold!)–Multi-scaleestimationcouldbeused

43


Multi-ScaleFlowEstimation

44


imageIt-1 imageI

GaussianpyramidofimageIt GaussianpyramidofimageIt+1

imageIt+1imageItu=10pixels

u=5pixels

u=2.5pixels

u=1.25pixels

Example:OpticalFlowin1D

45


v! ?

x

),( txI )1,( +txI

p

Example:OpticalFlowin1D

46


v!

xI

Spatialderivative

TemporalderivativetI

x

),( txI )1,( +txI

p

tx x

II¶¶

=px

t tII

=¶¶

=x

t

IIv -»

!Assumptions:

• Brightnessconstancy• Smallmotion

Example:OpticalFlowin1DLecture7:OpticalFlow

47

x

),( txI )1,( +txI

p

xI

tI

Temporalderivativeat2nd iteration

Iteratinghelpsrefiningthevelocityvector

Cankeepthesameestimateforspatialderivative

x

tprevious I

Ivv -¬!!

Convergesinabout5iterations

Lukas-Kanade withoutPyramid

48


Lukas-Kanade withPyramid

49


Horn&Schunck

• Globalmethodwithsmoothnessconstrainttosolveapertureproblem

50


Horn&Schunck


• Minimizeaglobalenergyfunction

51


Horn&Schunck



52


E(u, v) =

Z

x,y

[(Ix

u+ I

y

v + I

t

)2 + ↵

2(|ru|2 + |rv|2)]dxdy

Horn&Schunck



• Takepartialderivativesw.r.t.uandv:

53


E(u, v) =

Z

x,y

[(Ix

u+ I

y

v + I

t

)2 + ↵

2(|ru|2 + |rv|2)]dxdy

Horn&Schunck



• Takepartialderivativesw.r.t.uandv:

54


E(u, v) =

Z

x,y

[(Ix

u+ I

y

v + I

t

)2 + ↵

2(|ru|2 + |rv|2)]dxdy

(Ix

u+ Iy

v + It

)Ix

� ↵2ru = 0

(Ix

u+ Iy

v + It

)Iy

� ↵2rv = 0

Horn&Schunck

55


• Iterative scheme

• Yields high density flow• Fill in missing information in the homogenous regions• More sensitive to noise than local methods

2221

2221

)(

)(

yx

tk

yk

xykk

yx

tk

yk

xxkk

IIIvIuII

vv

IIIvIuII

uu

++

++-=

++

++-=

+

+

a

a

u

v

),( yx II

),( vu),( vu

OpticalFlowMatlab/C++/PythonCode

• http://people.csail.mit.edu/celiu/OpticalFlow/• http://opencv-python-tutroals.readthedocs.org/en/latest/py_tutorials/py_video/py_lucas_kanade/py_lucas_kanade.html

• https://github.com/Itseez/opencv_attic/blob/a6078cc8477ff055427b67048a95547b3efe92a5/opencv/samples/python2/lk_track.py

56


Applications: TargetTracking

57


Applications: ActionRecognition

58


yx FF , +-+-yyxx FFFF ,,, blurred +-+-

yyxx FFFF ,,,

Recognitionactionsatadistance,Efros etal.

Applications:MotionModeling

59


Flippingbetweenimage1and2. Estimatedflowfield(hueindicatesorientationandsaturationindicatesmagnitude)

Secondimageiswarped!

SlidecredittoC.Liu

Applications:MotionSegmentation

60


GlobalMotionModels(Parametric)Allpixelsareconsideredtosummarizeglobalmotion!

• 2DModels– Affine– Quadratic– Planarprojective(homography)

• 3DModels– Inst.Cameramotionmodels– Homography+epipole– Plane+parallalx

61


MotionModels

62


Translation

2 unknowns

Affine

6 unknowns

Perspective

8 unknowns

3D rotation

3 unknowns

Example:AffineMotion

63


u(x, y) = a1 + a2x+ a3y

u(x, y) = a4 + a5x+ a6y


64


u(x, y) = a1 + a2x+ a3y

u(x, y) = a4 + a5x+ a6y

Ix

u+ Iy

v + It

⇡ 0


65


u(x, y) = a1 + a2x+ a3y

u(x, y) = a4 + a5x+ a6y

I

x

(a1 + a2x+ a3y) + I

y

(a4 + a5x+ a6y) + I

t

⇡ 0


66


u(x, y) = a1 + a2x+ a3y

u(x, y) = a4 + a5x+ a6y

Eachpixelprovides1linearconstraintin6globalunknowns(a1,..,a6)

I

x

(a1 + a2x+ a3y) + I

y

(a4 + a5x+ a6y) + I

t

⇡ 0


67


u(x, y) = a1 + a2x+ a3y

u(x, y) = a4 + a5x+ a6y

Eachpixelprovides1linearconstraintin6globalunknowns(a1,..,a6)

Overallpixels,minimizetheleastsquaretofindunknowns!

Err

a1,...,a6 =X

[Ix

(a1 + a2x+ a3y) + I

y

(a4 + a5x+ a6y)]2

I

x

(a1 + a2x+ a3y) + I

y

(a4 + a5x+ a6y) + I

t

⇡ 0

Other2DMotionModels

• Quadratic

• Projective

68


u = q1 + q2x+ q3y + q7x2 + q8xy

u = q4 + q5x+ q6y + q7xy + q8y2

u =h1 + h2x+ h3y

h7 + h8x+ h9y� x

v =h4 + h5x+ h6y

h7 + h8x+ h9y� y

FlowCap:2DHumanPosefromOpticalFLow

69


Lecture 7-Optical Flow

Documents