Lecture: Motion - Stanford University

Lecture 17 -Stanford University

Lecture:Motion

JuanCarlosNiebles andRanjayKrishnaStanfordVisionandLearningLab

28-Nov-171


Whatwewilllearntoday?

• Opticalflow• Lucas-Kanade method• Horn-Schunk method• Pyramidsforlargemotion• Commonfate• Applications

28-Nov-172

Reading:[Szeliski]Chapters:8.4,8.5[Fleet&Weiss,2005]http://www.cs.toronto.edu/pub/jepson/teaching/vision/2503/opticalFlow.pdf



• Opticalflow• Horn-Schunk method• Lucas-Kanade method• Pyramidsforlargemotion• Commonfate• Applications

28-Nov-173



Fromimagestovideos• Avideoisasequenceofframescapturedovertime• Nowourimagedataisafunctionofspace(x,y)andtime(t)

28-Nov-174


Whyismotionuseful?

28-Nov-175


Whyismotionuseful?

28-Nov-176


Opticalflow• Definition:opticalflowistheapparentmotionofbrightnesspatternsintheimage

• Note:apparentmotioncanbecausedbylightingchangeswithoutanyactualmotion– Thinkofauniformrotatingsphereunderfixedlightingvs.astationarysphereundermovingillumination

GOAL: Recoverimagemotionateachpixelfromopticalflow

Source:SilvioSavarese

28-Nov-177


PicturecourtesyofSelim Temizer - LearningandIntelligentSystems(LIS)Group,MIT

Opticalflow

Vectorfieldfunctionofthespatio-temporalimagebrightnessvariations

28-Nov-178


Estimatingopticalflow

• Giventwosubsequentframes,estimatetheapparentmotionfieldu(x,y),v(x,y)betweenthem

• Keyassumptions• Brightnessconstancy:projectionofthesamepointlooksthesamein

everyframe• Smallmotion: pointsdonotmoveveryfar• Spatialcoherence: pointsmoveliketheirneighbors

I(x,y,t–1) I(x,y,t)


28-Nov-179

Lecture 17 -Stanford University 10

KeyAssumptions:smallmotions

*SlidefromMichaelBlack,CS143 2003


KeyAssumptions:spatialcoherence



KeyAssumptions:brightnessConstancy



I(x +u, y+ v, t) ≈ I(x, y, t −1)+ Ix ⋅u(x, y)+ Iy ⋅ v(x, y)+ It

• BrightnessConstancyEquation:I(x, y, t −1) = I(x +u(x, y), y+ v(x, y), t)

LinearizingtherightsideusingTaylorexpansion:

I(x,y,t–1) I(x,y,t)

0»+×+× tyx IvIuIHence,

Imagederivativealongx

→∇I ⋅ u v[ ]T + It = 0I(x +u, y+ v, t)− I(x, y, t −1) = Ix ⋅u(x, y)+ Iy ⋅ v(x, y)+ It


28-Nov-1713

Thebrightnessconstancyconstraint


Filtersusedtofindthederivatives

28-Nov-1714

𝐼" 𝐼# 𝐼$


• Howmanyequationsandunknownsperpixel?

Thecomponentoftheflowperpendiculartothegradient(i.e.,paralleltotheedge)cannotbemeasured

edge

(u,v)

(u’,v’)

gradient

(u+u’,v+v’)

If(u,v)satisfiestheequation,sodoes(u+u’,v+v’)if

•Oneequation(thisisascalarequation!),twounknowns(u,v)

∇I ⋅ u ' v '[ ]T = 0

Canweusethisequationtorecoverimagemotion(u,v)ateachpixel?


28-Nov-1715

Thebrightnessconstancyconstraint

∇I ⋅ u v[ ]T + It = 0


Theapertureproblem

Actualmotion


28-Nov-1716


Theapertureproblem

Perceivedmotion Source:SilvioSavarese

28-Nov-1717


Thebarberpoleillusion

http://en.wikipedia.org/wiki/Barberpole_illusion Source:SilvioSavarese

28-Nov-1718


Thebarberpoleillusion

http://en.wikipedia.org/wiki/Barberpole_illusion Source:SilvioSavarese

28-Nov-1719




28-Nov-1720



Solvingtheambiguity…

• Howtogetmoreequationsforapixel?• Spatialcoherenceconstraint:• Assumethepixel’sneighborshavethesame(u,v)

– Ifweusea5x5window,thatgivesus25equationsperpixel

B.LucasandT.Kanade.Aniterativeimageregistrationtechniquewithanapplicationtostereovision.InProceedingsoftheInternationalJointConferenceonArtificialIntelligence,pp.674–679,1981.


28-Nov-1721


• Overconstrained linearsystem:

Lucas-Kanade flow


28-Nov-1722


• Overconstrained linearsystem

ThesummationsareoverallpixelsintheKxKwindow

Leastsquaressolutionford givenby


28-Nov-1723

Lucas-Kanade flow


Conditionsforsolvability– Optimal(u,v)satisfiesLucas-Kanadeequation

Doesthisremindanythingtoyou?

WhenisThisSolvable?• ATA shouldbeinvertible• ATA shouldnotbetoosmallduetonoise

– eigenvaluesl1 andl 2 ofATA shouldnotbetoosmall• ATA shouldbewell-conditioned

– l 1/l 2 shouldnotbetoolarge(l 1 =largereigenvalue)


28-Nov-1724


• EigenvectorsandeigenvaluesofATArelatetoedgedirectionandmagnitude• Theeigenvectorassociatedwiththelargereigenvaluepointsin

thedirectionoffastestintensitychange• Theothereigenvectorisorthogonaltoit

M=ATAisthesecondmomentmatrix!(Harriscornerdetector…)


28-Nov-1725


Interpretingtheeigenvalues

l1

l2

“Corner”l1 and l2 are large,l1 ~ l2

l1 and l2 are small “Edge” l1 >> l2

“Edge” l2 >> l1

“Flat” region

Classificationofimagepointsusingeigenvaluesofthesecondmomentmatrix:


28-Nov-1726


Edge

– gradientsverylargeorverysmall– large l1,smalll2


28-Nov-1727


Low-textureregion

– gradientshavesmallmagnitude– small l1,smalll2


28-Nov-1728


High-textureregion

– gradientsaredifferent,largemagnitudes– large l1,largel2


28-Nov-1729


ErrorsinLukas-Kanade

Whatarethepotentialcausesoferrorsinthisprocedure?– SupposeATAiseasilyinvertible– Supposethereisnotmuchnoiseintheimage

• When our assumptions are violated– Brightness constancy is not satisfied– The motion is not small– A point does not move like its neighbors

• window size is too large• what is the ideal window size?

*FromKhurram Hassan-Shafique CAP5415ComputerVision2003


– Can solve using Newton’s method (out of scope for this class)

– Lukas-Kanade method does one iteration of Newton’s method• Better results are obtained via more iterations

Improvingaccuracy• Recalloursmallmotionassumption

• This is not exact– To do better, we need to add higher order terms back in:

• This is a polynomial root finding problem

It-1(x,y)

It-1(x,y)

It-1(x,y)

*FromKhurramHassan-ShafiqueCAP5415ComputerVision2003


IterativeRefinement• Iterative Lukas-Kanade Algorithm

1. Estimate velocity at each pixel by solving Lucas-Kanade equations

2. Warp I(t-1) towards I(t) using the estimated flow field- use image warping techniques

3. Repeat until convergence

*FromKhurramHassan-ShafiqueCAP5415ComputerVision2003


Whendotheopticalflowassumptionsfail?

28-Nov-1733

Inotherwords,inwhatsituationsdoesthedisplacementofpixelpatchesnotrepresentphysicalmovementofpointsinspace?

1. Well, TV is based on illusory motion – the set is stationary yet things seem to move

2. A uniform rotating sphere – nothing seems to move, yet it is rotating

3. Changing directions or intensities of lighting can make things seem to move – for example, if the specular highlight on a rotating sphere moves.

4. Muscle movement can make some spots on a cheetah move opposite direction of motion. – And infinitely more break downs of optical flow.




28-Nov-1734



Horn-Schunk methodforopticalflow

• Theflowisformulatedasaglobalenergy function whichisshouldbeminimized:

28-Nov-1735




• Thefirstpartofthefunctionisthebrightnessconsistency.

28-Nov-1736




• Thesecondpartisthesmoothnessconstraint.It’stryingtomakesurethatthechangesbetweenframesaresmall.

28-Nov-1737




• 𝛼 isaregularizationconstant.Largervaluesof𝛼 leadtosmootherflow.

28-Nov-1738




• Bytakingthederivativewithrespecttouandv,wegetthefollowing2equations:

28-Nov-1739



• Bytakingthederivativewithrespecttouandv,wegetthefollowing2equations:

• WhereiscalledtheLagrangeoperator.Inpractice,itismeasuredusing:

• whereistheweightedaverageofumeasuredat(x,y).

28-Nov-1740



• Nowwesubstitutein:

• Toget:

• Whichislinearinuandvandcanbesolvedforeachpixelindividually.

28-Nov-1741


IterativeHorn-Schunk

• Butsincethesolutiondependsontheneighboringvaluesoftheflowfield,itmustberepeatedoncetheneighborshavebeenupdated.

• Soinstead,wecaniterativelysolveforuandvusing:

28-Nov-1742


Whatdoesthesmoothnessregularizationdoanyway?

• It’sasumofsquaredterms(aEuclidiandistancemeasure).• We’reputtingitintheexpressiontobeminimized.• =>Intexturefreeregions,thereisnoopticalflow

Regularizedflow

Opticalflow

• =>Onedges,pointswillflowtonearestpoints,solvingtheapertureproblem.

Slidecredit:SebastianThurn


DenseOpticalFlowwithMichaelBlack’smethod

• MichaelBlacktookHorn-Schunk’s methodonestepfurther,startingfromtheregularizationconstant:

• Whichlookslikeaquadratic:

• Andreplaceditwiththis:

• Whydoesthisregularizationworkbetter?

28-Nov-1744




28-Nov-1745


• Keyassumptions(ErrorsinLucas-Kanade)

• Smallmotion: pointsdonotmoveveryfar

• Brightnessconstancy:projectionofthesamepointlooksthesameineveryframe

• Spatialcoherence: pointsmoveliketheirneighbors

Recap


28-Nov-1746


Revisitingthesmallmotionassumption

• Isthismotionsmallenough?– Probablynot—it’smuchlargerthanonepixel(2nd ordertermsdominate)– Howmightwesolvethisproblem?

* Fro

m K

hurra

mH

assa

n-Sh

afiq

ueC

AP54

15 C

ompu

ter V

isio

n 20

03

28-Nov-1747


Reducetheresolution!

* Fro

m K

hurra

mH

assa

n-Sh

afiq

ueC

AP54

15 C

ompu

ter V

isio

n 20

03

28-Nov-1748



image Iimage H

Gaussianpyramidofimage1 Gaussianpyramidofimage2

image2image1 u=10 pixels

u=5 pixels

u=2.5 pixels

u=1.25 pixels

Coarse-to-fineopticalflowestimation

28-Nov-1749


image Iimage J

Gaussianpyramidofimage1(t) Gaussianpyramidofimage2(t+1)

image2image1

Coarse-to-fineopticalflowestimation

runiterativeL-K

runiterativeL-K

warp&upsample

.

.

.


28-Nov-1750


OpticalFlowResults

* Fro

m K

hurra

mH

assa

n-Sh

afiq

ueC

AP54

15 C

ompu

ter V

isio

n 20

03

28-Nov-1751


OpticalFlowResults

* Fro

m K

hurra

mH

assa

n-Sh

afiq

ueC

AP54

15 C

ompu

ter V

isio

n 20

03

• http://www.ces.clemson.edu/~stb/klt/• OpenCV

28-Nov-1752




28-Nov-1753


• Keyassumptions(ErrorsinLucas-Kanade)

• Smallmotion: pointsdonotmoveveryfar

• Brightnessconstancy:projectionofthesamepointlooksthesameineveryframe

• Spatialcoherence: pointsmoveliketheirneighbors

Recap


28-Nov-1754


Reminder:Gestalt– commonfate

28-Nov-1755


Motionsegmentation• Howdowerepresentthemotioninthisscene?


28-Nov-1756


• Breakimagesequenceinto“layers”eachofwhichhasacoherent(affine)motion

MotionsegmentationJ.WangandE.Adelson.LayeredRepresentationforMotionAnalysis.CVPR1993.


28-Nov-1757


• Substitutingintothebrightnessconstancyequation:

yaxaayxvyaxaayxu

654

321

),(),(

++=++=

0»+×+× tyx IvIuI

Affinemotion


28-Nov-1758


0)()( 654321 »++++++ tyx IyaxaaIyaxaaI

• Substitutingintothebrightnessconstancyequation:

yaxaayxvyaxaayxu

654

321

),(),(

++=++=

• Eachpixelprovides1linearconstraintin6unknowns

[ ] 2å ++++++= tyx IyaxaaIyaxaaIaErr )()()( 654321!

• Leastsquaresminimization:

Affinemotion


28-Nov-1759


Howdoweestimatethelayers?• 1.Obtainasetofinitialaffinemotionhypotheses

– Dividetheimageintoblocksandestimateaffinemotionparametersineachblockbyleastsquares• Eliminatehypotheseswithhighresidualerror

• Mapintomotionparameterspace• Performk-meansclusteringonaffinemotionparameters

–Mergeclustersthatarecloseandretainthelargestclusterstoobtainasmallersetofhypothesestodescribeallthemotionsinthescene


28-Nov-1760







28-Nov-1761






2.Iterateuntilconvergence:•Assigneachpixeltobesthypothesis

–Pixelswithhighresidualerrorremainunassigned•Performregionfilteringtoenforcespatialconstraints•Re-estimateaffinemotionsineachregion


28-Nov-1762


Exampleresult

J. Wang and E. Adelson. Layered Representation for Motion Analysis. CVPR 1993. Source:SilvioSavarese

28-Nov-1763




28-Nov-1764


Usesofmotion

• Trackingfeatures• Segmentingobjectsbasedonmotioncues• Learningdynamicalmodels• Improvingvideoquality

– Motionstabilization– Superresolution

• Trackingobjects• Recognizingeventsandactivities

28-Nov-1765


Estimating3Dstructure


28-Nov-1766


Segmentingobjectsbasedonmotioncues

• Backgroundsubtraction– Astaticcameraisobservingascene– Goal:separatethestaticbackground fromthemovingforeground


28-Nov-1767


• Motionsegmentation– Segmentthevideointomultiplecoherentlymovingobjects

Segmentingobjectsbasedonmotioncues

S.J.Pundlik andS.T.Birchfield,MotionSegmentationatAnySpeed,ProceedingsoftheBritishMachineVisionConference(BMVC)2006


28-Nov-1768


Z.Yin andR.Collins,"On-the-flyObjectModelingwhileTracking,"IEEEComputerVisionandPatternRecognition(CVPR'07),Minneapolis,MN,June2007.

Trackingobjects


28-Nov-1769


Synthesizingdynamictextures

28-Nov-1770


Super-resolution

Example:Asetoflowqualityimages


28-Nov-17


Super-resolution

Eachoftheseimageslookslikethis:


28-Nov-17


Super-resolution

Therecoveryresult:


28-Nov-17


D. Ramanan, D. Forsyth, and A. Zisserman. Tracking People by Learning their Appearance. PAMI 2007.

Tracker

Recognizingeventsandactivities


28-Nov-1774


JuanCarlosNiebles,HongchengWangandLiFei-Fei,UnsupervisedLearningofHumanActionCategoriesUsingSpatial-TemporalWords,(BMVC),Edinburgh,2006.


28-Nov-1775


Crossing– Talking– Queuing– Dancing– jogging

W.Choi &K.Shahid&S.SavareseWMC2010



28-Nov-1776


W.Choi,K.Shahid,S.Savarese,"Whataretheydoing?:CollectiveActivityClassificationUsingSpatio-TemporalRelationshipAmongPeople",9thInternationalWorkshoponVisualSurveillance(VSWS09)inconjuction withICCV09

28-Nov-1777


Opticalflowwithoutmotion!

28-Nov-1778


Whatwehavelearnedtoday?


28-Nov-1779

[Fleet&Weiss,2005]http://www.cs.toronto.edu/pub/jepson/teaching/vision/2503/opticalFlow.pdf

Reading:[Szeliski]Chapters:8.4,8.5

Lecture: Motion - Stanford University

Documents