Temporal Video Denoising Based on Multihypothesis Motion Compensation Liwei Guo; Au, O.C.; Mengyao Ma; Zhiqin Liang; Hong Kong Univ. of Sci. & Technol.,

Temporal Video Denoising Based on Multihypothesis Motion Compensation

Liwei Guo; Au, O.C.; Mengyao Ma; Zhiqin Liang; Hong Kong Univ. of Sci. & Technol., Clear Water Bay Circuits and Systems for Video Technology(CSVT), IEEE 2007

Outline

• Introduction• Video Signal Model With Multihypothesis MC• Multihypothesis Motion Compensated Filter

(MHMCF)– The Proposed Linear Temporal Filter – MHMCF– Implementation Issues– Performance Analysis

• Experimental Results• Conclusions

Introduction

• Spatial correlation denoising: 2-D Kalman filter [1], 2-D Wiener filter [2], wavelet shrinkage [3], non-local means [4] etc.

• Until now there are few temporal denoising methods presented in the literature.

• These temporal predictions, defined as its motion-compensated hypotheses for the current pixel.

Video Signal Model With Multihypothesis MC

• We present a novel model of residue(zm) for multihypothesis MC:

• Let the mean and the variance of be and respectively.

• We propose a linear model for this relationship:

f : the current pixel of Fk

Pm : the motion compensated prediction of f from Fk-m

Video Signal Model With Multihypothesis MC

Video Signal Model With Multihypothesis MC

• For video with large motion, the correlation tends to decrease faster than small motion.

• Large b implies video with large motion.• Large a implies texture regions.

Multihypothesis Motion Compensated Filter (MHMCF)

• The Proposed Linear Temporal Filter – MHMCF• Implementation Issues– Motion Estimation– Parameters Estimation

• Performance Analysis

-The Proposed Linear Temporal Filter – MHMCF

• Assumptions:– Video sequence is contaminated by additive zero-

mean random noise.– The noise source is stationary over spatial and

temporal domain, and independent of residue(zm).

• The noise-corrupted video signal f’ and p’m:

• We propose MHMCF to estimate the current pixel f :

• For simplicity, we rewrite (3) as:

• We define the objective function of MHMCF:

• Minimizing is equal to Minimizing [16, p. 273].

-The Proposed Linear Temporal Filter – MHMCF

Random varianbles

Let

• By = 0 and = 0 , the optimal w and d that minimize are:

• As zm and nm are independent, and gm is

independent with each other:

-The Proposed Linear Temporal Filter – MHMCF

• The optimal w and d that minimize are:

• Large implies low temporal correlation.• When , then d = 0, w0 = 1, wm = 0, and

no filtering will be applied.

-The Proposed Linear Temporal Filter – MHMCF

-Implementation Issues• Motion Estimation:– MHMCF needs to perform ME with respect to

every reference frame.– Fast ME algorithm, PMVFAST [17], is employed.– Experiments show that PMVFAST compared to full

search, about 1% denoising error is increased.• Parameters Estimation:

?

-Implementation Issues

• Parameters Estimation:– : We select the minimum 3% out of the total

block variances (their average is ) :

– and : Let be the noisy residue.• Then ( ), since n0 and nm are all

zero-mean:• As gm and n0 are independent:

-Performance Analysis

• The estimation error :• MHMCF is an unbiased estimator leading to

and error variance .• Combining (3), (6), (9),(11), and (12):

– We have the model of residue variance .– The remaining noise in the reference frame is the

estimation error:

?

-Performance Analysis

Small motion

Smooth regions

-Performance Analysis

Experimental Results

• Denoising Performance: JNT[9] STVF[10]

WRSTF[11]

Experimental Results

Experimental Results

Experimental Results

• Computational Complexity :– In terms of the number of ADD and MUL performed

to process a frame in CIF resolution (352 288).

Conclusions

• A temporal denoising filter MHMCF is developed for the removal of noise in video.

• MHMCF has very good noise suppression capability while using fewer inputs than other proposed filters.

• MHMCF is a purely temporal filter, spatial blurring is avoided and most spatial details could be preserved.