Error Resilience & Error Concealment Based on Motion Vector Correction & Missing Frame Prediction Chen-Yu Tseng ( 曾禎宇 ) Department of Electronic Engineering,

Error Resilience & Error Concealment Based on

Motion Vector Correction &

Missing Frame Prediction

Chen-Yu Tseng (曾禎宇 )Department of Electronic Engineering,

National Chiao Tung University,Taiwan, R.O.C.

Introduction

Transmitted Video

WirelessNetwork

Packet loss

Received Video

Error Resilience

Error Concealment

Outline

• Compressed Video Over Wireless Network• Prior Arts of Error Concealment• Proposed Error Concealment

– Motion Vector Correction– Motion Recovery

• Proposed Error Resilience Based on Motion Vector Correction

• Experiments• Conclusion

Compressed Video Over Wireless Network

time

IP

PP

P

P

time

IP

PP

P

P

Encoded Sequence

Decoded Sequence

Packet Loss

Error PropagationError Propagation

Packet-BasedVideo Transmission

Prior Arts of Error Concealment

• Block-Level Concealment– Spatial Based– Spatial-Temporal Based– Motion Vector Field Based

• Frame-Level Concealment

MissingBlock

CorrectlyReceived Parts

Block-Level Concealment

• Spatial Interpolation Based [9-11]

Original Sequence Edge Preserving Bilinear Interpolation


• Spatial-Temporal Based [12-15]

Missing

MB

Substitute

MB

Reference

Frame

Left MB Left MB Right MB Right MB

Top MB Top MB

Bottom

MB

Bottom

MB

Current

Frame

Missing MBMissing MBSubstitute

MBSubstituteMB

ReferenceFrame

CurrentFrame

Boundary Matching Algorithm (BMA)W. M. Lam et al. [12]


• Motion Vector Field Based [16-17]

Missing MBMissing MBSubstitute

MBSubstituteMB

ReferenceFrame

CurrentFrame

RecoveredMVRecoveredMV

Lost

MV

Current MV

Field

Recovered

MV Field

Interpolation in MV Field

Whole-Frame Loss

Example of 3G Applications:• QCIF Video Transmission at 64 kbps.

Frame Rate: 10 fps.Average 800 bytes per frame

• General Packet Size: 1kBPacket Loss May Cause Whole-Frame

Loss!!

Frame-Level Concealment

• Motion Vector Recovery Based on Optical Flow

SubstituteMBSubstituteMB

ReferenceFrame

CurrentMissing Frame

RecoveredMVRecoveredMVRef MV

Frame-Level Concealment

S. Belfiore, M. Grangetto, E. Magli, G. Olmo, “An error concealment algorithm for streaming video,” Proc. ICIP 2003, 2003 [22]

S. Belfiore, M. Grangetto, E. Magli, G. Olmo, “Concealment of whole-frame loss for wireless low bit-rate video based on multiframe optical flow estimation,” IEEE Trans. Multimedia, vol. 7, no. 2, pp. 316-329, Apr. 2005. [23]

P. Baccicht, D. Bagni, A. Chimienti, L.Pezzoni, and F. Rovati, “Frame concealment for H.264/AVC decoders,” IEEE Trans. Consumer Electronics, vlo. 51, no. 1, pp. 227-233, Feb. 2005. [24]

Zhenyu Wu and J. M. Boyce, “An error concealment scheme for entire frame losses based on H.264/AVC,” Proc. ISCAS 2006, May 2006.[25]

Optical Flow (OF)

• Motion Consistency

t-2

t-1

ttime

t-2

t-1

t

Optical Flow

Motion Recovery Based on Optical Flow

time

t-2

t-1

t

Optical Flow

Original Optical Flow

Corrupted Optical Flow

Packet lost

time time

Optical Flow Estimation

• General Model of Optical Flow Equation[27]

• Motion Vector substitutes for OF

0),,(

dt

tssdx VH (1)

0),,(

),,(),,(

),,(),,(

t

tssxtssv

s

tssxtssv

s

tssx VHVHV

V

VHVHH

H

VH (2)

Fn-1 Fn

(i, j)(iF, jF)

1,njiMV

1,njiFMV 1

,njiFMV

1,

1,,

1,,

1,,

1,,

where

,

nji

nVji

nVji

nHji

nji

njFiF

MVFMV

FMVjjF

FMViiF

MVMV

Motion Vector Projection

MV Versus OF

• MV Block Matching

• OF Motion Trajectory

Reference FrameForeman #4

Current FrameForeman #5

Block Matching

Temporal Aliasing Problem

Projected MVProjected MV

Projected MVProjected MV

Wrong MV

t

t-1

t-2

t

t-1

t-2

MissingFrame

Correct MV Projection Wrong MV Projection

CorrectMV

Pixel-based Concealment

• Schematic of Belfiore’s Algorithm [22]

ReferenceFrameBuffer

Estimation ofoptical flow

with temporal regularization

Spatial regularizationof FMV field

Projection of prev. frame

ontomissing frame

Interpolationof missing pixels

Filtering and downsampling








ontomissing frame




• Estimation of optical flow with temporal regularization

TERM MVH MVH

TERM MVH MVH

MVMV

MVMV

nn-1n-2

L

k

kji

nji MVH

LFMV

1,

1,

1

There are other temporal interpolators for the MVH, including median filter and weighted averages decaying overtime. Belfiore et al. [22] find that mean value consistently yields the best result, thus validating the linear velocity assumption.








ontomissing frame



tjFiFFMV ,

1,tjiFMV

t-1

t








ontomissing frame



tjFiFFMV ,

1,tjiFMV

t-1

t

Pixel-based vs. Block-Based

tjFiFFMV ,

1,tjiFMV

t-1

t

1,tjiFMV

t-1

t

tjFiFFMV ,

Pixel-based Block-based

Block-based Concealment

• Schematic of P. Baccicht’s Algorithm [24]

Reference frame buffer

Searching of suitable reference

Motion vector

projection

Statistics collection

MB-Level MV field estimation

Block-Level MV field estimation

Picture reconstruction

1,tjiFMV

t-1

t

tjFiFFMV ,

Problems of MV Projection

• Conflict State

• Non-covered State

1,tjiFMV

t-1

t

tjFiFFMV ,

Non-covered State

Conflict State

Problems of MV Projection

• Conflict State

• Non-covered State

Possible Reasons– Wrong Motion Vector– Object Warping, Occlusion, or Non-

translation Motion.– Frame Boundary

Outline





Motion Vector Correction

• Motion Vector True Motion Trajectory

• Property of True Motion Trajectory– Temporal Consistency– Spatial Consistency

t-2 t-1 t

TemporalConsistency

SpatialConsistency

Motion Vector CorrectionBased on Fuzzy Logic

Motion Vector Field

Nji

tji

tji

tji

tjit

ji

tji

Nji

tji

tji

TRSR

TRSRR

MVRMVC

,,,

','','','

',',

',',

tjiMV ,Original MV:

Corrected MV:

Reliability of :

Spatial Reliability:

Temporal Reliability:

tjiMVC ,

tjiMV ','

tjiR ','

tjiSR ','

tjiTR ','

tjiTD ,

tjiSD ,

tjiR ,

tjiR ','

SpatialDifference

TemporalDifference

Temporal Reliability

• Temporal Difference

t-2

t-1

t

time

t-2

t-1

t

time

Low temporal difference

High temporal reliability

High temporal difference

Low temporal reliability

,,

,,where

2

,

,

1,

1,,,

tji

tji

tjFiF

tjBiB

tji

tji

MVjijFiF

MVjijBiB

MVMVMVTD

)(exp1

1

,,

Tht

ji

tji TDTD

TR

tjiTR ,

1

0t

jiTD ,

ThTD

Spatial Reliability

• Spatial Difference

Motion Vector Field Motion Vector Field

tji

tji

tji

tji

tji

tji

tji

tji

tji

tji

tji

tji

tji

tji

tji

tji

MVMVMVMVMVMV

MVMVMVMVMVMV

MV

MVMVSD

1,1,,1,11,1,

1,1,,1,1,1,1

,

,,,

if ,2

if ,2

,

)(exp1

1

,,

Tht

ji

tji SDSD

SR

tjiSR ,

1

0t

jiSD ,

ThSD

Motion Vector CorrectionBased on Fuzzy Logic

Motion Vector Field

Nji

tji

tji

tji

tjit

ji

tji

Nji

tji

tji

TRSR

TRSRR

MVRMVC

,,,

','','','

',',

',',

tjiTD ,

tjiSD ,

tjiR ,

x

y

Original Frame Original MV Field Corrected MV Field

MV Projection with MV Correction

Original MV Field Corrected MV Field

x

y

tjFiFFMV ,

1,tjiFMV

t-1

t

tjFiFFMV ,

1,tjiFMV

t-1

t

Proposed MV Projection

• Why Concealment?

Release Error Propagation

time

IP

PP

P

P

Error PropagationError Propagation

time

IP

PP

P

P

Error FrameError Frame

Error FrameError Frame

Can We Skip from Error Frame?Can We Skip from Error Frame?

MV Projection Forward vs. Backward

Original Optical Flow


time time

Packet lost


time

Packet lost

Forward MV ProjectionForward MV Projection

Backward MV ProjectionBackward MV Projection


OFReference Frame

Missing Frame

Forward projected OF

(a) Forward motion projection

OFReference Frame

Missing Frame

OF

Backward projected OF

(b)Backward motion projection


1,2,, njiMVjijBiB

Fn-1

Fn

(iB, jB)

Fn+1

Backward Projected MV

Backward Projected MV Missing

frame

1,njiMV (i, j)

Fn-1 Fn

1,njiMV

(i, j)

1,njiFMV 1

,njiFMV

(iF, jF)

Missing frame

1,

1,,

1,,

1,,

1,,

where

,

nji

nVji

nVji

nHji

nji

njFiF

MVFMV

FMVjjF

FMViiF

MVMV

Advantage ofBackward MV Projection

• Easy to implement.

• Avoid from conflict or non-cover states.

• Invariance of I-frame position.

Forward Motion Projection

time

IP

P

P

PP

time

Missing frame

PI

P

P

PP

Missing frame

time

Missing frame

IP

P

P

PP

time

Missing frame

PI

P

P

PP

No MV

Backward Motion Projection

Outline





Schematic of Proposed System

Encoder with MV Correction

WirelessNetwork Decoder with

Missing FramePrediction

Error Resilience Error Concealment

Motion Vector Correction

• Spatial-Temporal MV Correction (3-D)

x

y

time

n-1

n

n-2

……

Iterative Correction

MV Field

32.8000

32.9000

33.0000

33.1000

33.2000

33.3000

33.4000

33.5000

33.6000

0 1 2 3 4 5 6

IterationPS

NR

1數列

The number of MV correction iteration effects the result of concealment.

The PSNR is the concealment result (foreman FLR10) with different number of MV correction iteration

Proposed 2-StepMotion Vector Correction

MVn-1MVn-1 MVn MVn+1MVn+1 MVn+2MVn+2MVn-2MVn-2 ….….

MVn-1MVn-1 MVnMVn MVn+1MVn+1 MVn+2MVn+2MVn-2MVn-2 ….….





Forward MV correctionForward MV correction

time

Iteration

Backward MV correctionBackward MV correction

Concealment PSNRwith Different MV Correction

2-Step: 33.4666 dBBidirectional: 33.1826 dB

The PSNR is the concealment result (foreman FLR10)

Motion Compensation PSNR

With MV Correction : 32.0382 dBWithout MV Correction : 34.5538dB

PSNR Drop PSNR Drop

(foreman)

PSNR Drop

Effect: Increase the bit rateReason: Inconsistent Motion

MV correction is based on motion consistency.

In this situation,concealment is also hard to work.

We preserve the original MV to avoid MC PSNR drop.

PSNR Drop Control Factor

ME MC

MVCorrection

MC

If Residual_MADcorrect > f * Residual_MADuncorrect

MVcorrect = MVoriginal

Residualcorrect = Residualoriginal

f is the control factor

MC_PSNR vs. EC_PSNRfactor MC_PSNR EC_PSNR

inf 32.0382 33.8803

6 32.4231 33.8397

5 32.5454 33.7881

4 32.7421 33.6529

3.5 32.9059 33.5187

3.25 32.9981 33.4395

3 33.106 33.3602

2.75 33.2406 33.2511

2.5 33.4009 33.132

2.25 33.5861 32.8189

2 33.789 32.5876

1.75 34.0185 32.2453

1.5 34.2771 31.5792

1.25 34.5392 31.0878

1 34.6723 30.544

0 34.5538 30.2423

Foreman QCIFMC_PSNR: Motion Compensation PSNREC_PSNR: Error Concealment PSNR(Backward MV concealment FLR10)

foreman_EC/MC

30

30.5

31

31.5

32

32.5

33

33.5

34

34.5

35

0 1 2 3 4 5 6 7MAD_Factor

PSN

R

EC_PSNR

MC_PSNR

MC_PSNR vs. EC_PSNR

30

30.5

31

31.5

32

32.5

33

33.5

34

34.5

31.5 32 32.5 33 33.5 34 34.5 35

MC_PSNR

EC_P

SNR

Without Refinement

factor = inf

factor = 0

Foreman QCIFMC_PSNR: Motion Compensation PSNREC_PSNR: Error Concealment PSNR(Backward MV concealment FLR10)

Error Concealment Results

• Foreman QCIF Backward Concealment FLR10 (#63)

MAD Factor = inf MAD Factor = 2.75

Refinement of Concealment

• MV Temporal Consistency Check

• Frame Spatial Consistency Checkt-2 t

MV Temporal Consistency Check

Difft(i, j) = || MVt(i, j) – MVt-2(iB, jB) ||,Where (iB, jB) = (i, j) + 2*MVt(i, j)

If Difft(i, j) > Diff_temporal_TH MVt(i, j) = MV’t(i, j) , if Diff_v(i, j) < Diff_h(i, j), MV’t(i, j) = ( MVt(i-1, j) + MVt(i+1, j) )/2 else MV’t(i, j) = ( MVt(i, j-1) + MVt(i, j+1) )/2 where Diff_v(i, j) = || MVt(i-1, j) - MVt(i+1, j) || and Diff_h(i, j) = || MVt(i, j-1) - MVt(i, j+1) ||

t-2 t

MV RefinementMV Refinement

Frame Spatial Consistency Check

difference difference

difference

difference

If difference of the four directionsare all greater then Diff_spatial_TH MVt(i, j) = MV’t(i, j)

4x4 block

MC_PSNR V.S. EC_PSNR

Foreman QCIFMC_PSNR: Motion Compensation PSNREC_PSNR: Error Concealment PSNR(Backward MV concealment FLR10)EC_r_PSNR: Error Concealment PSNR

with MV refinement

factor MC_PSNR EC_PSNR EC_r_PSNR

inf 32.0382 33.8803 34.2188

6 32.4231 33.8397 34.1206

5 32.5454 33.7881 34.0954

4 32.7421 33.6529 34.0545

3.5 32.9059 33.5187 34.0389

3.25 32.9981 33.4395 33.9509

3 33.1060 33.3602 33.9143

2.75 33.2406 33.2511 33.8724

2.5 33.4009 33.1320 33.8054

2.25 33.5861 32.8189 33.6826

2 33.7890 32.5876 33.5845

1.75 34.0185 32.2453 33.4763

1.5 34.2771 31.5792 33.2406

1.25 34.5392 31.0878 32.8728

1 34.6723 30.5440 32.5500

0 34.5538 30.2423 32.4962

foreman_EC/MC

30

30.5

31

31.5

32

32.5

33

33.5

34

34.5

35

0 1 2 3 4 5 6 7MAD_Factor

PSN

R

EC_PSNR

MC_PSNR

EC_r_PSNR

MC_PSNR V.S. EC_PSNR

30

30.5

31

31.5

32

32.5

33

33.5

34

34.5

31.5 32 32.5 33 33.5 34 34.5 35

MC_PSNR

EC_P

SNR

Without Refinement

factor = inf

factor = 0

With Refinement

MC PSNR (foreman qcif)

MAD Factor = 0 : 34.5538dBMAD Factor = 2.75: 33.2406 dBMAD Factor = inf : 32.0382 dB

MAD Factor = 0 : 34.5538dBMAD Factor = 2.75: 33.2406 dBMAD Factor = inf : 32.0382 dB


MAD Factor = inf; PSNR =33.8803MAD Factor = 2.75; PSNR =33.2511MAD Factor = 2.75; with MV refinementPSNR =33.8724

MAD Factor = inf; PSNR =33.8803MAD Factor = 2.75; PSNR =33.2511MAD Factor = 2.75; with MV refinementPSNR =33.8724

Foreman QCIF Backward Concealment FLR10


• Foreman QCIF Backward Concealment FLR10 (#63)

a. b. c.a. MAD Factor = infb. MAD Factor = 2.75c. MAD Factor = 2.75 with MV refinement

Outline





Experiment of Concealment

Encoded Video

Sequence

VideoTransmission

SimulatorDecoder

ErrorConcealment

OutputVideo

Sequence

[7] Chih-Heng Ke, Cheng-Han Lin, Ce-Kuen Shieh, Wen-Shyang Hwang, “A Novel Realistic Simulation Tool for Video Transmission over Wireless Network”, The IEEE International Conference on Sensor Networks, Ubiquitous, and Trustworthy Computing (SUTC2006), June 5-7, 2006, Taichung, Taiwan.

Results

Foreman QCIF, 15 fpsPLR 1%without EC: 32.4318 dB

FMP: 35.2562 dBBMP: 35.5058 dB

FMP: Forward MV ProjectionBMP: Backward MV Projection with MV correction

Concealment Results

Foreman QCIF #54, 15 fps PLR 1%

Forward MV Projection Backward MV Projection

Results

flower QCIF, 15 fpsPLR 2%without EC: 30.4504 dB

FMP: 33.4728 dBBMP: 34.5167 dB

FMP: Forward MV ProjectionBMP: Backward MV Projection with MV correction

Concealment PSNR

Sequence　

PLR　

　 PSNR 　 Gain 　

No EC FMP BMP FMP BMP

foreman 1%

32.4318

35.2562

35.5058

2.8244 3.074

　 2%29.74

8134.51

0034.95

324.761

95.205

1

　 5%24.65

8533.51

5834.30

828.857

39.649

7

flower 1%32.66

1134.13

6334.51

671.475

21.855

6

　 2%30.45

0433.47

2834.17

683.022

43.726

4

　 5%20.65

3330.29

2432.26

649.639

111.61

31

stefan 1%31.26

9333.43

2333.47

902.163

02.209

7

　 2%26.32

8831.90

6732.09

475.577

95.765

9

　 5%18.42

8929.19

9529.51

3510.77

0611.08

46

Conclusion

• We propose a new motion vector correction method to make MV approximate true motion trajectory.

• We propose an error resilient scheme based on MV correction.

• We propose a new whole-frame concealment based on backward MV projection which is simple to implement.

Reference

1. Thomas Wiegand, Gary J. Sullivan, Gisle Bjontegaard, and Ajay Luthra, “Overview of the H.264/AVC Video Coding Standard,” IEEE Trans. on Circuits Syst. Video Technol., vol. 13, No. 7, pp.560 – 576, July 2003

2. Iain E G Richardson, “H.264 and MPEG-4 Video Compression, ” John Wiley, 2003

3. Thomas Stockhammer, Miska M. Hannuksela, and Thomas Wiegand, “H.264/AVC in Wireless Environment,” IEEE Trans. On Circuits Syst. Video Technol., vol. 13, No. 7, July 2003

4. Stephan Wenger, “H.264/AVC Over IP,” IEEE Trans. On Circuits Syst. Video Technol., vol. 13, No. 7, July 2003

5. Thomas Stockhammer, Thomas Wiegand, Tobias Oelbaum, and Florian Obermeier, “Video coding and transport layer techniques for H.264/AVC-based transmission over packet-lossy networks,” Proc. ICIP 2003, vol. 3, September 2003.

6. J. Klaue, B. Rathke, and A. Wolisz, "EvalVid – A Framework for Video Transmission and Quality Evaluation", In Proc. of the 13th International Conference on Modelling Techniques and Tools for Computer Performance Evaluation, Urbana, Illinois, USA, September 2003.

7. Chih-Heng Ke, Cheng-Han Lin, Ce-Kuen Shieh, Wen-Shyang Hwang, “A Novel Realistic Simulation Tool for Video Transmission over Wireless Network”, The IEEE International Conference on Sensor Networks, Ubiquitous, and Trustworthy Computing (SUTC2006), June 5-7, 2006

8. Network Simulator, http://www.isi.edu/nsnam/ns/9. Zhang Rongfu, Zhou Yuanhua, and Huang

Xiaodong, “Content-adaptive spatial error concealment for video communication,” IEEE Trans. On Consumer Electronics, vol. 50, Feb. 2004.

10. Olivia Nemethova, Ameen Al-Moghrabi and Markus Rupp, “Flexible Error Concealment for H.264 Based on Directional Interpolation,” International Conference on Wireless Networks, Communications and Mobile Computin, vol. 2, June 2005.

11. Steven Beesley, Andrew Armstrong, Christos Grecos, “An Edge Preserving Spatial Error Concealment Technique for the H.264 Video Coding Standard,” Research in Microelectronics and Electronics 2006, Ph. D., June 2006.

http://www.isi.edu/nsnam/ns/



Reference

12. W. M. Lam, A. R. Reibman, and B. Liu, “Recovery of lost orerroneously received motion vectors,” in Proc. ICASSP’93, pp. V417-V420, Apr. 1993.

13. E. T. Kim, S.-J. Choi, and H.-M. Kim, “Weighted boundary matching algorithm for error concealment in the MPEG-2 video bit stream,” Signal Process., vol. 73, pp. 291–295, Mar. 1999.

14. Yan Chen, Au, O., Chiwang Ho, and Jiantao Zhou, “Spatio-temporal boundary matching algorithm for temporal error concealment,” Proc. ISCAS 2006, May 2006.

15. Agrafiotis, D., Bull, D.R., Canagarajah, C.N., “Enhanced Error Concealment With Mode Selection,” IEEE Trans. on Circuits and Systems for Video Technology, vol. 16, August 2006.

16. M. E. Al-Mualla, C. N. Canagarajah, and D. R. Bull, “Error concealment using motion field interpolation,” Proc. ICIP 1998, Vol. 2, pp. 512—516, October 1998.

17. Jinghong Zheng, Lap-Pui Chau, “A temporal error concealment algorithm for H.264 using Lagrange interpolation,” Proc. ISCAS 2004, vol. 2, May 2004.

18. Jae-Young Pyun, Jun-Suk Lee, Jin-Woo Jeong, Jae-Hwan Jeong, and Sung-Jea Ko, “Robust error concealment for visual communications in burst-packet-loss networks,” IEEE Trans. On Consumer Electronics, vol. 49, November 2003

19. S. Gnavi, M. Grangetto, E. Magli, and G. Olmo, “Comparison of rate allocation strategies for H.264 video transmission over wireless lossy correlated networks,” in Proc. ICME\—IEEE Int. Conf. Multimedia and Expo, 2003.

20. E. N. Gilbert, “Capacity of a burst-noise channel,” Bell Syst. Tech. J., vol. 39, pp. 1253-1265, Sept. 1960.

21. E. 0. Elliott, “Estimates of error rates for codes on burst-noise channels,” Bell Syst. Tech. J., vol. 42, pp. 1977-1997, Sept. 1963.

22. S. Belfiore, M. Grangetto, E. Magli, G. Olmo, “Concealment of whole-frame loss for wireless low bit-rate video based on multiframe optical flow estimation,” IEEE Trans. Multimedia, vol. 7, no. 2, pp. 316-329, Apr. 2005.

23. S. Belfiore, M. Grangetto, E. Magli, G. Olmo, “An error concealment algorithm for streaming video,” Proc. ICIP 2003, 2003.

Reference

24. Baccichet, P.;and Chimienti, A., “A low complexity concealment algorithm for the whole-frame loss in H.264/AVC,” IEEE 6th Workshop on Multimedia Signal Processing, October 2004.

25. P. Baccicht, D. Bagni, A. Chimienti, L.Pezzoni, and F. Rovati, “Frame concealment for H.264/AVC decoders,” IEEE Trans. Consumer Electronics, vol. 51, no. 1, pp. 227-233, Feb. 2005.

26. Zhenyu Wu and J. M. Boyce, “An error concealment scheme for entire frame losses based on H.264/AVC,” Proc. ISCAS 2006, May 2006.

27. A. M. Tekalp, Digital Video Processing. Englewood Cliffs, NJ: Prentice- Hall, 1995.

Error Resilience & Error Concealment Based on Motion Vector Correction & Missing Frame Prediction Chen-Yu Tseng ( 曾禎宇 ) Department of Electronic Engineering,

Documents