K.-S. Choi and S.-J. Ko Sch. of Electr. Eng., Korea Univ., Seoul, South Korea IEEE, Electronics Letters Issue Date : June 10 2010 Hierarchical Motion Estimation.

K.-S. Choi and S.-J. Ko

Sch. of Electr. Eng., Korea Univ., Seoul, South Korea

IEEE, Electronics Letters Issue Date : June 10 2010

Hierarchical Motion Estimation AlgorithmUsing Reliable Motion Adoption

OUTLINE• Hierarchical structure

• Conventional Hierarchical BMAs (HBMAs)

• Hierarchical motion correction

• Experimental results

• Conclusions

Hierarchical structure• A pyramid of multi-level

• Low resolution images

obtained by down-sampling

or low-pass filter.

• Rough estimation

• Relatively large block

• Relatively large searching

range

• Four child blocks

Conventional Hierarchical BMAs (HBMAs)[3]• The estimate of motion vectors at a lower resolution level.

• Propagate motion vectors from a lower to a higher resolution level.

• Duplication or bilinear interpolation

• Produce consistence motion vector fields

If block B is only included in one moving object The true motion vectors of the four child blocks are usually very close to V.

Otherwise, if block B covers two or more different objects The true motion vectors of the child blocks may be significantly different from V.

Child block belongs to the parent block or one of its neighboring blocks.

Smallest motion compensation error is chosen as the motion vector of the child block

[3] Wang, D., and Lauzon, D. : ‘Hybrid algorithm for estimating true motion fields’, Opt. Eng., 2000, 39, (11), pp. 2876–2881

The actual motion of small objects is hardly predicted either spatiotemporally or hierarchically! => hierarchical motion correction

V FS (l = 0, 1,…, L − 1): L-level MVFs by FS-VBSMEl

V (l=0,1,…,L) : resulting MVFs l

Predict from upper level MVF and refinement• Initially, VFS is utilized as the prediction at the first loop ( l = L – 1 )

• Small refinement search range(RSR)

• To maintain the smoothness of the resulting MVF

• Similarly to conventional HBMAs.

L-1

Propagation of MV between adjacent blocks

• if a block at a higher level has more than one moving object with a different motion,

• an inaccurate MV is predicted for the current block from the parent block.

• The current block and one of its adjacent blocks having a different parent block may belong to the same object.

• Replaced with the MV of the adjacent block S i, j.

Adoption of actual motion of objects(1/2)• the motion of small objects has not been predicted accurately at the upper levels.

• VFS(i, j) is utilized as another MV candidate.

• Since VFS does not only contain meaningful information for any complex motion of variable sized objects but also random MVs especially at the low levels.

• The reliability of VFS(i, j) should be first examined

l

l

l

Adoption of actual motion of objects(2/2)• Reliability measure

• MVD represents the cost of discontinuity in the MVF induced when VFS (i, j) is adopted for the current block.

• V (i, j) is replaced by VFS(i, j) when following condition is satisfied :

C1: minimum discontinuity cost imposed on VFS(i, j) λ: a weighting factor for level l

l

l

l l

MVD prevents VFS(i, j) from being adopted with small amount of reduction in the match-error criterion.

l

EXPERIMENTAL RESULTS• To compare the results of the proposed algorithm objectively and subjectively

• 3DRS

• Ha’s algorithm

• HBMA [3]

Stefan sequenceFrame 13 Frame 14

Resulting M

VFs

Demonstrate the accuracy of the MVFs objectively• three test video sequences

• Even frames have been skipped

• Applied to a motion-compensated frame interpolation (MCFI) method [5] with the resulting MVF of each algorithm.

[5] Wang, T.-S., Choi, K.-S., Jang, H.-S., and Ko, S.-J.: ‘Enhanced frame rate up-conversion method for UHD Video’. Proc. IEEE Int. Conf. on Consumer Electronics, January 2010, p. 4.4–3

CONCLUSION• Propose a hierarchical true motion estimation algorithm

• Spatially propagates reliable MVs

• Replaces inaccurate MVs with more reliable MVs in the multi-level MVFs

• Experimental results have shown that

• Produces consistent MVs within each object

• Estimates the different motion of various small objects accurately