MULTI-RESOLUTION BLOCK MATCHING MOTION ESTIMATION …

MULTI-RESOLUTION BLOCK MATCHING MOTION ESTIMATIONWITH DEFORMATION HANDLING USING GENETIC

ALGORITHMS FOR OBJECT TRACKING APPLICATIONS

Harish Bhaskar and Helmut BezLoughborough University, U.K.

[email protected], [email protected]

Keywords: Block motion estimation, affine, deformation handling, genetic algorithms, multi-resolution, object tracking.

Abstract: Motion Estimation is a popular technique for computing the displacement vectors between objects or attributesbetween images captured at subsequent time stamps. Block matching is a well known technique of motionestimation that has been successfully applied to several applications suchas video coding, compression andobject tracking. One of the major limitations of the algorithm is its ability to cope with deformation of objectsor image attributes within the image. In this paper we present a novel scheme for block matching that combinesgenetic algorithms with affine transformations to accurate match blocks. The model is adapted into a multi-resolution framework and is applied to object tracking. A detailed analysis of the model alongside criticalresults illustrating its performance on several synthetic and real-time datasets is presented.

1 INTRODUCTION

Motion estimation techniques aim at deducingdisplacement vectors for objects or image attributesbetween two consecutive frames (A. Gyaourovaand Cheung, 2003). The main idea behind blockmatching motion estimation strategies is to divide theimage frame into blocks and match blocks betweensuccessive frames within a search window usingspecific search techniques (A.Barjatya, 2005). It isclear that the two distinct phases that make up anyblock matching method is block partitioning andblock searching. The block partitioning scheme isconcerned with dividing the original image frameinto non-overlapping regions. Block partitioning canbe performed using the fixed size or variable sizemethods (C-C.Chang, 2006) (F.J.Ferri and J.Soret,1998). The block search mechanism is the processof locating the block in the destination frame thatbest matches the block in the anchor frame using aspecific matching criterion (Turaga and M.Alkanhal,1998).

Different models have been developed in litera-ture to accomplish robust motion estimation usingblock based techniques. (C-W.Ting and L-M.Po,

2004) propose the use of different search schemeswith fixed and variable block partitioning methodsto accomplish robust estimation. In a similar studyby (M.Wagner and D.Saupe, 2000), a quad-treeblock motion estimation scheme is proposed. Othermethods of variable block matching have also beenproposed, particularly in the form of polygon ap-proximation, mesh based (Y.Wang and O.Lee, 1996)and binary trees. Another class of block matchingmethods that have recently used for deformationhandling particularly in applications of object track-ing is the deformable block matching (O.Lee andY.Wang, 1995). In a study by (J.H.Velduis andG.W.Brodland, 1999), deformable block matchinghas been adapted for use in tracking cell particles. Abilinear transformation is used with block matchingto handle deformation. In the context of deformablemodels,triangular or mesh based block decomposi-tion is much popular (Y.Wang and A.Vetro, 1996).The idea behind these schemes is to partition imageframes using techniques of finite element analysis,triangulation (M.Yazdi and A.Zaccarin, 1997), meshgrid etc. and employ deformable block matchingof vertex points to handle complex motion changesduring motion estimation. In the context of meshbased methods, a nodal based scheme for block

169

matching is also popular. According to the nodalscheme, mesh is generated such that nodes liesacross object boundaries and a simple search oflinear motion of these nodal position from the anchorframe to the destination frame will be able to sufficedeformation (O.Lee and Y.Wang, 1995). In thispaper, we shall highlight a framework that integratesa vector quantization based block partitioning methodto an genetic algorithm based search scheme withaffine parametrization to accomplish robust, accuratemotion estimation with deformation handling. Themodel is built on a multi-resolution platform withperformance feedback.

2 PROPOSED MODEL

The proposed model constitutes of different phases.The first phase is the multi-resolution platform thatthe framework is based on. The platform combinesa scale space representation of data with a multi-resolution level analysis. A multi-resolution modelaims at capturing a wide range of levels of detail of animage and can in-turn be used to reconstruct any oneof those levels on demand. The distinction betweendifferent layers of an image is determined by the res-olution. A simple mechanism of tuning the resolutioncan add finer details to coarser descriptions providinga better approximation of the original image. Mathe-matically, we can represent the above analysis in thefollowing way. If the resolution is represented usingλ, then the initial level is associated withλ = 0 is 1and that with any arbitrary resolutionλ is 1

2λ . If fλ isthe image at resolutionλ, then at resolutionλ+1,

fλ+1 = fλ + Γλ (1)

where Γλ is the details at resolutionλ. In con-trast, the scale space representation of data deals withrepresenting images in such a way that the spatial-frequency localizations are simultaneously preserved.This is achieved by decomposing images into a setof spatial-frequency component images. Scale spacetheory, therefore, deals with handling image struc-tures at different scale such that the original image canbe embedded into a one-parameter family of derivedcomponent images thereby allowing fine-scale struc-tures to be successively suppressed.Mathematically,to accomplish the above, a simple operation of con-volution can be used. However, it is important tonote that the overhead of using the convolution op-erator is kept low. For any given imageI(x, y), itslinear scale space representation is composed of com-ponentsLϑ(x, y) defined as a convolution operator of

the imageI(x, y) and a Gaussian kernel of the form:

Gϑ(x, y) =1

2πϑe−

x2+y2

2ϑ (2)

, such that

Lϑ(x, y) = Gϑ(x, y) ∗ I(x, y) (3)

whereϑ = σ2 is the variance of the Gaussian. Per-formance based feedback automates the selection ofrelevant resolution and scale for any particular framepair. A brief algorithm describing the process is asfollows.

• Initialize the resolutionsλ[1:q] to [0, 1, 2, ..., q] andscalesϑ[1:q] to [1, 2, 3, ..., q + 1] for any value ofq (4 chosen for this experiment).

• Select the median of resolutions as the initial start-ing resolution and scale. The median is 2 in ourexperiments and the chosen values of(λ, ϑ) are(2, 3)

• Input at any time instantt, two successive framepairs of a video sequence,(ft, ft+1).

• Re-sample the imagesft and ft+1 into the se-lected resolution using bi-cubic interpolation

• Convolve the image at selected scale (in matchingpositions with the resolution) with a Gaussian ker-nel to obtain a filtered output(Gϑ ∗ft, Gϑ ∗ft+1)

• Perform Motion Estimation of these input imagesat this scale-resolution using the motion estima-tion algorithm specified in the subsection belowand reconstruct the target frame using the esti-mated motion parameters.

• Evaluate the performance of the model usingthe metrics: PSNR, Entropy and Time as in(H.Bhaskar and S.Singh, 2006)

• If the frame pair processed is(ft, ft+1) at t = 1then automatically slide up to a higher resolutionand repeat process by incrementingt. Otherwise,if t > 1 then ifPSNRt > PSNRt−1 then slidedown to lower resolution - scale otherwise slideup to higher resolution - scale combination.

• Repeat the process for all frame pairs

The second phase of the algorithm deals withmotion estimation. For the purpose of motionestimation we extend the technique of deformableblock matching that combines the process of blockpartitioning, block search and motion modeling. Avector quantization based block partitioning schemeis combined with a genetic algorithm based searchmethod for robust motion estimation (H.Bhaskarand S.Singh, 2006). We extend the basic model insuch a way that block deformation is handled using a

ICINCO 2007 - International Conference on Informatics in Control, Automation and Robotics

170

combined genetic algorithm affine motion model.

The block partitioning phase remains unchangedwhile the genetic algorithm based block searchscheme is altered to include the affine transforma-tions. In the subsection below, a detailed algorithm ofthe modified block search scheme based on geneticalgorithm and affine transforms is presented.

2.1 Vector Quantization Based BlockPartitioning

The vector quantization scheme for block partitioningillustrated in (H.Bhaskar and S.Singh, 2006) has beenused in the proposed deformable block matching. Itis important to realize that the image framesft andft+1 that is input to this stage of the algorithm refersto the filtered output of the previous stage. Accordingto the vector quantization scheme, image frames arepartitioned based on the information content presentwithin them. The model separates regions of inter-est based on clustering and places a boundary sep-arating these regions. For this, the vector quantiza-tion mechanism uses the gray level feature attributesfor separating different image regions and the centerof mass of different intersection configurations is em-ployed to deduce the best partition suitable for the im-age frames.

2.2 Affine-Genetic Algorithm MotionModel

The idea behind the genetic algorithm affine motionmodel combination is to use the affine transformationequation on every block during fitness function eval-uation. The algorithm for the block search scheme isas follows.

The genetic algorithm based block matching al-gorithm described below is used to match thecentroid of any block from the partitioned structureof frameft to its successive frameft+1 at differentangles theta and parameters shear and scale. Theinputs to the genetic algorithm are the blockbt andthe centroid(xc, yc) of the block.

• Parameter Initialization: The variable parametersof the genetic algorithm will be the genes in thechromosomes. In our experiments they will bethe the pixel displacement value inx andy direc-tions, the angle theta of the input block, the shearfactors and scale(rx, ry) are encoded as the chro-mosome(Tx, Ty, θ, s, rx, ry). The translation, ro-tation and scale parameters of the model are ini-tialized using the phase correlation and log-polar

Figure 1: Phase Correlation.

transforms. This speeds up the genetic algorithmsearch scheme and also increases the accuracy ofestimation.

– Translation parameters using phase correlation:The phase correlation technique is a frequencydomain approach to determine the translativemovement between two consecutive images. Asimple algorithm illustrating the process of de-termining an approximate translative motioncharacteristics between two images is as fol-lows.∗ Consider the input blockbt and its corre-

sponding block at the successive framebt+1

∗ Apply a window function to remove edge ef-fects from the block images

∗ Apply a 2D Fourier transform to the im-ages and produceFt = Ψ(bt) andFb+1 =Ψ(bt+1); whereψ is the Fourier operator.

∗ Compute the complex conjugate ofFt+1,multiply the Fourier transforms element-wiseand normalize to produce a normalized crosspower spectrumNPS using

NPS =FtF

∗

t+1

|FtF∗

t+1|(4)

∗ Apply inverse Fourier transform on the nor-malized power spectrum to obtainPS =ψ−1(NPS); whereψ−1 is the inverse Fourieroperator.

∗ Determine the peak as the the translative co-ordinates using

(∆x,∆y) = argmax(PS) (5)

∗ An illustration describing the process of phasecorrelation using a sample image is as shownin Figure 1.

– Rotation and Scale using Log-Polar Trans-forms: The log-polar transform is a conformalmapping of points on cartesian plane to pointson the log-polar plane. The transformation canaccommodate an arbitrary rotations and a rangeof scale changes. If an block image in the carte-sian plan is represented usingb(x, y), then thelog polar transform of the block image with ori-ginO at location(xo, yo) is

MULTI-RESOLUTION BLOCK MATCHING MOTION ESTIMATION WITH DEFORMATION HANDLING USINGGENETIC ALGORITHMS FOR OBJECT TRACKING APPLICATIONS

171

Figure 2: Log Polar Transform.

b∗(ψ, φ) = bξ(x, y) (6)

where,ψ = Mlog(r + α), α is any constantr =

√

(x− xo)2 + (y − yo)2 andφ = tan−1 y−yo

x−xo

In order to determine the approximate valuesof rotation and scale using the log-polar trans-forms, we convert the image frames into the logpolar domain and then use phase correlation be-tween the log-polar images to identify the rota-tion and scale parameters as in Figure 2.

• Population Initialization: A populationP of thesen chromosomes representing(Tx, Ty, θ, s, rx, ry)is generated from uniformly distributed randomnumbers where,

– 1 ≤ n ≤ limit and limit (100) is the maxi-mum size of the population that is user defined.

– The values of pre-initialized parameters such astranslational, rotational and scale are generatedwithin a small range of their initial value.

• To evaluate the fitnessE(n) for every chromo-somen:

– Extract the pixels locations corresponding tothe block from frameft using the centroid(xc, yc) and block size information

– Affine Transforming these pixels using thetranslation parameters(Tx, Ty), rotation angleθ, shear factors and scalerx, ry using,24 x′

y′

1

35 =

24 1 s 0

0 1 0

0 0 1

3524 rx 0 0

0 ry 0

0 0 1

3524 cosθ −sinθ 0

sinθ cosθ 0

0 0 1

3524 1 0 Tx

0 1 Ty

0 0 1

3524 x

y

1

35– If bt represents the original block under consid-

eration,b∗t+1 represents the block identified atthe destination frame after transformation and(h,w) the dimensions of the block, then the fit-nessE can be measured as the mean absolutedifference (MAD).

MAD =1

hw

h∑

i=1

w∑

j=1

∣

∣bt(i, j) − b∗t+1(i, j)∣

∣

(7)

Figure 3: 2D Deformable Block Matching.

• Optimization: Determine the chromosome withminimum errornemin = n whereE is mini-mum. As this represents a pixel in the block,determine all the neighbors (NHk) of the pixel,where1 ≤ k ≤ 8.

– For all k, determine the error of matching as inFitness evaluation.

– If E(NHk) < E(nemin), thennemin = NHk

• Selection: Define selection probabilities to selectchromosomes for mutation or cloning.

• Cross-Over: All chromosomesncr that are cho-sen for cross-over are taken into the next gener-ation after swapping one or more random genesbetween every successive chromosome.

• Mutation: All Chromosomesnmu chosen for mu-tation are replaced with uniformly distributed ran-dom values for centroid, angle, shear, scale andsqueeze.

• Termination: Three termination criterion arespecified in the proposed model. Check if anycondition is satisfied, otherwise iterate until ter-mination.

– Zero Error: If a chromosome returned an errorvalue zero through fitness evaluation, Or

– Maximum Generations: If the number of gener-ations (i.e. process loops) exceeds a predefinedthreshold, Or

– Stall Generations: If the number of stall gen-erations (i.e. process loops where there is nochange in the fitness values) exceeds a prede-fined threshold.

3 RESULTS AND ANALYSIS

Detailed results and analysis of the proposed model ispresented in this section of the paper. On the secondpart of this section we demonstrate how the motionestimation scheme is adapted to object tracking appli-cations.

ICINCO 2007 - International Conference on Informatics in Control, Automation and Robotics

172

1 2 3 4 5 613.5

14

14.5

15

15.5

16

16.5

17

17.5Time Compexity Plot

Video Number

Tim

e T

aken

Per

Fra

me

Pai

r

VBMA with Affine Parametrization

1 2 3 4 5 67

7.5

8

8.5

9

9.5

10

10.5

11

11.5

12

Video Number

Tim

e T

aken

Per

Fra

me

Pai

r

Time Complexity Plot

DVQBMA

1 2 3 4 5 629

29.5

30

30.5

31

31.5

32

32.5

33

33.5

34

Video Number

PS

NR

Val

ues

(dB

)

Variable Block Matching with Affine Parametrization

1 2 3 4 5 630

30.5

31

31.5

32

32.5

33

33.5

34

34.5

35

Video Number

PS

NR

Val

ues

(dB

)

PSNR Plot

DVQBMA

1 2 3 4 5 60

0.02

0.04

0.06

0.08

0.1

0.12

0.14

Video Number

Rel

ativ

e E

ntro

py

Variable Block Matching with Affine Parametrization

1 2 3 4 5 60

0.02

0.04

0.06

0.08

0.1

0.12

Video Number

Rel

ativ

e E

ntro

py

Entropy Plot

DVQBMA

Figure 4: Performance Comparison of Proposed Model toBaselines.

3.1 Performance Evaluation of MotionEstimation

In Figure 3 we illustrate the stages of the proposedblock matching scheme. The first and the second im-ages illustrate the original frame and the transformedframe of a sample synthetic image. Through theother images we illustrate how the genetic algorithmis used to identify the optimal motion parameters.Different configurations are evolved through increas-ing generations getting the solution closer to optimal.The red block represents the objects original positionin the anchor frame and the green boundary specifiesthe location of the block during block searching usinggenetic algorithm at different generations. To furtheraffirm the performance of the model on differentreal time datasets, we perform experiments of themodel on 6 different video data each containingaround 40 frames. The averaged performances oneach videos are measured using time, relative entropyand PSNR metrics and compared to the baselinemodel in Figure 4. The baseline model uses affineparametrization with other search schemes on avariable block partitioned data. We also compare theproposed model against the original block matchingmodel that does not handle deformation and a rotationinvariant model.

It is very evident that the averaged time complex-ity of the proposed motion estimation mechanism thathandles deformation still does not match the require-ments of real-time. However, with a multi-resolutionoptimization approach it might well be possible to im-prove the time efficiency. The results compare wellwith the quad-tree block matching mechanism withaffine parametrization. There is a clear advantage inusing the proposed strategy for deformation handlingthan an extension to any other variable block parti-tioning scheme with sub-optimal search. The qual-ity of motion estimation is recorded and compared inthe graphs. It can be observed that there is clear im-

provement in the quality of motion estimation whendeformation of objects is handled during motion es-timation. In comparison to the baseline model, thereis clear increase of about 2dB in the PSNR values. Aclear increase in the PSNR values can be noted dur-ing the progressive improvements in the model fromthe basic framework to the rotation invariant modeland finally to the deformation handling model. Thisclearly indicates how useful deformation handling isduring motion estimation. A very similar trend canalso be visualized between different models whencompared against the performance metric of relativeentropy. The reconstructions made from the defor-mation handling model match closer to the expectedoutcome of the image frame. This highlights the ac-curacy and robustness of the strategy in accomplish-ing motion estimation. In comparison to the baselinemodel there is a clear improvement in the values ofrelative entropy.

3.2 Object Tracking Applications

In this section we describe how the motion estima-tion mechanism above can be adapted to object track-ing applications and also analyze how the efficiencyof motion estimation influences the quality of objecttracking. We have extended the model for applicationin object tracking through simple clustering of fea-tures characteristics including motion information. Touse the proposed model into object tracking motionvectors are clustered such that the moving group ofblocks possessing similar motion and feature charac-teristics will form the object of interest. Trajectoriesare plotted using the center of mass location of theblocks that constitute the objects. We have tested theapproach on a number of different datasets. We havedisplayed the results of the model on some of them.Figure 5 illustrates the motion trajectory (representedusing red dots) of a single/multiple object trackedover different time stamps. As the model does notperform object segmentation, produces a number ofsmall unwanted trajectories that have been removedthrough manually entered semantic information. Thesemantic information can be of the form of velocityinformation of the object in motion, color of the mov-ing objects etc. We have in our experiments displayedthe motion trajectory of the group of blocks that havebeen tracked longest on the image sequences. Gener-ally in any scene this information corresponds to theobject of interest. The first two images are the tra-jectories of the proposed model and a polygon shapefeature based nearest neighbor tracking scheme pro-posed in (H.Bhaskar and S.Singh, 2005). As it can beclearly observed the trajectory of the baseline model

MULTI-RESOLUTION BLOCK MATCHING MOTION ESTIMATION WITH DEFORMATION HANDLING USINGGENETIC ALGORITHMS FOR OBJECT TRACKING APPLICATIONS

173

Figure 5: Object Trajectory of Sample Video Sequences.

disintegrates once the object has complex deforma-tion whereas the proposed scheme continues to handledeformation reliably for the entire video. The mainreason to this is that the model relies on image seg-mentation and polygon based shape approximation.The second group of 3 images illustrate the trackingof multiple objects in an image sequence. This se-quence is also an example of a very noisy sequencewith most of its background moving as well. Finally,examples of human tracking are also illustrated. Inthe first, we have extracted the trajectory of the bodyof the person moving and displayed it. The model ac-tually produces different trajectories for moving partsof hands, legs, face etc. as in the next image. Thesecond image is the output of the shape feature basedbaseline model. Again the technique fails to track ob-jects immediately after a complex deformation is no-ticed.

4 CONCLUSION

In this paper we presented a novel deformation han-dling mechanism for block matching based on ge-netic algorithm that can be extended for use in ob-ject tracking applications. The model combines thevector quantization based variable block partitioningand applies an affine based genetic algorithm match-ing scheme for block matching. We have also pre-sented results on several real time datasets to illus-trate the proof of concept. Analysis of the results onthe model has proved that the model is robust and re-liable for tracking deformational changes in objects invideo sequences.

REFERENCES

A. Gyaourova, C. K. and Cheung, S.-C. (2003). Blockmatching for object tracking. Technical report,Lawrence Livermore Nation Laboratory.

A.Barjatya (2005). Block matching algorithms for motionestimation. Technical report.

C-C.Chang, L-L.Chen, T.-S. (2006). Multi-resolution basedmotion estimation for object tracking using genetic al-gorithm. InVIE Conference.

C-W.Ting, W.-H. and L-M.Po (2004). Fast block matchingmotion estimation by recent-biased search for multi-ple reference frames. InInternational Conference onImage Processing (ICIP).

F.J.Ferri, J. J. and J.Soret (1998). Variable-size blockmatching algorithm for motion estimation using aperceptual-based splitting criterion. InInternationalConference on Pattern Recognition (ICPR), page 286.

H.Bhaskar and S.Singh (2005). Multiple particle tracking inlive cell imaging using green fluorescent protein (gfp)tagged videos. InInternational Conference on Ad-vances in Patter Recognition, pages 792–803.

H.Bhaskar, R. and S.Singh (2006). A novel vector quan-tization based block matching strategy using geneticalgorithm search. Submitted to Pattern RecognitionLetters.

J.H.Velduis and G.W.Brodland (1999). A deformableblock-matching algorithm for tracking epithelial cells.In Image and Vision Computing, pages 905–911.

M.Wagner and D.Saupe (2000). Video coding withquadtrees and adaptive vector quantization. InPro-ceedings EUSIPCO.

M.Yazdi and A.Zaccarin (1997). Interframe coding usingdeformable triangles of variable size. InInternationalConference on Image Processing, page 456.

O.Lee and Y.Wang (1995). Motion compensated predic-tion using nodal based deformable block matching.In Visual Communications and Image Representation,pages 26–34.

Turaga, D. and M.Alkanhal (1998). Search algorithms forblock-matching in motion estimation.

Y.Wang and O.Lee (1996). Use of 2d deformable meshstructures for video compression, part i - the synthe-sis problem: Mesh based function approximation andmapping. InIEEE Trans. Circuits and Systems onVideo Technology, pages 636–646.

Y.Wang, O. and A.Vetro (1996). Use of 2d deformable meshstructures for video compression, part ii - the analysisproblem and a region-based coder employing an activemesh representation. InIBID, pages 647–659.

ICINCO 2007 - International Conference on Informatics in Control, Automation and Robotics

174