Robust Kernel Estimation with Outliers Handling for Image Deblurring Jinshan Pan 1,2 , Zhouchen Lin 3,4, * , Zhixun Su 1,5 , and Ming-Hsuan Yang 2 1 School of Mathematical Sciences, Dalian University of Technology 2 Electrical Engineering and Computer Science, University of California at Merced 3 Key Laboratory of Machine Perception (MOE), School of EECS, Peking University 4 Cooperative Medianet Innovation Center, Shanghai Jiaotong University 5 National Engineering Research Center of Digital Life http://vllab.ucmerced.edu/ ˜ jinshan/projects/outlier-deblur/ Abstract Estimating blur kernels from real world images is a chal- lenging problem as the linear image formation assumption does not hold when significant outliers, such as saturated pixels and non-Gaussian noise, are present. While some existing non-blind deblurring algorithms can deal with out- liers to a certain extent, few blind deblurring methods are developed to well estimate the blur kernels from the blurred images with outliers. In this paper, we present an algo- rithm to address this problem by exploiting reliable edges and removing outliers in the intermediate latent images, thereby estimating blur kernels robustly. We analyze the effects of outliers on kernel estimation and show that most state-of-the-art blind deblurring methods may recover delta kernels when blurred images contain significant outliers. We propose a robust energy function which describes the properties of outliers for the final latent image restoration. Furthermore, we show that the proposed algorithm can be applied to improve existing methods to deblur images with outliers. Extensive experiments on different kinds of challenging blurry images with significant amount of out- liers demonstrate the proposed algorithm performs favor- ably against the state-of-the-art methods. 1. Introduction Recent years have witnessed significant advances in single-image deblurring [13, 15, 18]. The success of state- of-the-art methods mainly stems from accurate restoration of sharp edges [3, 5, 14, 30, 36] or strong priors of natural images and blur kernels [2, 7, 17, 23, 29, 32, 38, 39]. While these methods are able to deblur natural images well, they are less effective for blurred inputs with significant amount of outliers (e.g., saturated areas and non-Gaussian noise) as illustrated by examples in Figure 1. * Corresponding author. (a) (b) (c) (g) (d) (e) (f) (j) (k) (l) (m) (p) (n) (o) (h) (i) (q) Figure 1. Effects of outliers on blind deblurring. (a) A real cap- tured blurred image with large saturated regions, e.g., light streaks and blobs in red boxes. (b)-(e) Intermediate results generated by Cho and Lee [3], Xu and Jia [36], Xu et al. [38], and Pan et al. [27]. (f) Salient edges extracted by the proposed algorithm (shown by Poisson reconstruction). (g)-(h) Deblurred results of Xu and Jia [36] and Hu et al. [12]. (i) Deblurred result by the proposed algorithm. (j)-(o) Estimated kernels by Xu and Jia [36], Krishnan et al. [17], Levin et al. [21], Zhong et al. [40], Xu et al. [38], Pan et al. [27], and Pan et al. [26]. (p) Estimated kernel by the proposed algorithm. The red boxes in (a)-(e) and (g)-(h) con- tain saturated regions (best viewed on high-resolution displays). The main reason that most algorithms do not perform well is that the gray levels of the pixels in the bright or specular regions are clipped to the maximum value (e.g., 255) during the image formation process due to the limited quantization range of camera sensors. Thus, the linear blur model, 2800
9
Embed
Robust Kernel Estimation With Outliers Handling for Image ...openaccess.thecvf.com/content_cvpr_2016/papers/Pan... · Robust Kernel Estimation with Outliers Handling for Image Deblurring
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Robust Kernel Estimation with Outliers Handling for Image Deblurring
Jinshan Pan1,2, Zhouchen Lin3,4,∗, Zhixun Su1,5, and Ming-Hsuan Yang2
1School of Mathematical Sciences, Dalian University of Technology2Electrical Engineering and Computer Science, University of California at Merced
3Key Laboratory of Machine Perception (MOE), School of EECS, Peking University4Cooperative Medianet Innovation Center, Shanghai Jiaotong University
5National Engineering Research Center of Digital Life
Figure 11. A synthetic example with saturated regions and im-
pulse noise (best viewed on high-resolution displays with zoom-
in).
detection of light streaks from a blurred image for kernel
estimation. When light streaks are not detected well, this
method is less effective as shown in Figure 11(d). Although
the method [40] is designed to deal with Gaussian noise, it
is less effective for saturated areas. We also note that the
method [26] is able to deal with saturated images, but less
effective for this example due to noise (See Figure 11(e)).
In contrast, the proposed algorithm is able to detect the sat-
urated regions, which facilitates kernel estimation and the
deblurred results contain fine textures. We further compare
with outlier handling methods [4, 34]. As [4, 34] mainly
focus on non-blind image deblurring with outliers, we use
our estimated kernels to generate the final results. With our
estimated kernel, high-quality deblurred results can be ob-
tained by [4]. As the method by [34] is mainly designed for
handling saturated areas, it is less robust to impulse noise.
Real Images: We use a real example to evaluate the pro-
posed algorithm. Figure 12(a) shows a real captured exam-
ple with several saturated areas and unknown noise. Again,
state-of-the-art methods [17, 21, 38] do not perform well on
this example due to effects of saturated areas. Method [12]
also fails to generate clear results due to unavailable light-
streaks (See Figure 12(e)). Although the method by [4] gen-
erates much clearer results by our estimated kernel, the de-
blurred image still contains significant artifacts. In contrast,
our method successfully estimates the blur kernel and gen-
erates a better deblurred result. Moreover, the comparison
results shown in Figure 12(g) and (h) demonstrate that the
proposed algorithm with M is able to remove outliers in the
kernel estimation.
Non-Uniform Deblurring: We evaluate the proposed al-
gorithm against the state-of-the-art methods [35, 38] for
non-uniform image deblurring. Figure 13 shows one real-
captured image from [35] in which the proposed algorithm
performs favorably with sharper results.
Images without Outliers and Noise: As discussed in Sec-
tion 4.4, the proposed algorithm can be applied to deblur
images without containing outliers and noise. Figure 14(a)
shows an example without outliers from [38]. The proposed
method is able to detect the positions of ambiguous edges
2806
(a) Input (b) [17] (c) [38] (d) [21]
(e) [12] (f) Our kernel+[4] (g) Without M (h) Ours
Figure 12. Real example with numerous saturated regions (e.g.,
the light blobs). The kernel size is estimated at 45× 45 pixels.
(a) Input (b) [35] (c) [38] (d) Ours
Figure 13. Real captured non-uniform example from [35] (best
viewed on high-resolution displays with zoom-in).
(a) Input (b) Xu et al. [38] (c) Ours (d) Detected M
Figure 14. A blurred image without outliers. The parts enclosed in
blue boxes in (b) contain ringing artifacts. The size of estimated
kernel is 45 × 45 pixels (best viewed on high-resolution displays
with zoom-in).
where the linear convolution model does not hold from ex-
tracted salient edges (See Figure 14(d)). The estimated la-
tent image contains fewer ringing artifacts compared to the
result by [38].
Benchmark Datasets with and without Outliers: We use
the benchmark dataset by Levin et al. [20] for quantitative
evaluation in which we add the salt and pepper noise (as it
is one of the most common outliers [1]) to each image. The
noise density is set to be 0.01. We evaluate the performance
of the proposed algorithm against the state-of-the-art meth-
ods [3, 7, 17, 21, 28, 36] and one deblurring algorithm that
also deals with noise [40]. The error ratio metric [20] is
used for quantitative evaluations. Figure 15(a) shows that
the proposed algorithm achieves favorable results against
state-of-the-art methods. We further evaluate the proposed
algorithm using the images with different noise densities.
Figure 15(b) shows that the proposed algorithm performs
well even when the noise density is high.
We create a dataset containing 10 ground truth images
with saturated regions and 8 kernels from [20]. The size of
the saturated regions in this dataset is from 5×5 to 400×400pixels. Similar to [4], we stretch the intensity histogram
range of each image into [0, 2] and apply 8 different blur
1 2 3 4 5 6 7 8 9 10Error ratios
0
10
20
30
40
50
60
70
80
90
100
Succ
ess
rate
(%)
Fergus et al.Cho and LeeXu and JiaKrishnan et al.Levin et al.Zhong et al.Perrone and FavaroOurs
1 2 3 4 5 6 7 8 9 1016
18
20
22
24
26
28
30
32
Noise density (%)
Ave
rage
PSN
R v
alue
s
Cho and LeeXu and JiaKrishnan et al.Zhong et al.Perrone and FavaroOurs
im1 im2 im3 im4 im5 im6 im7 im8 im9 im100
5
10
15
20
25
30
Ave
rage
PSN
R V
alue
s
Blurred imagesCho and LeeXu and JiaKrishnan et al.Zhong et al.Xu et al.Hu et al.Pan et al.Ours
(a) (b) (c)
Figure 15. Quantitative evaluation on the dataset with outilers. (a)
Results on the dataset with salt and pepper noise. (b) PSNR values
of blind deblurring methods on the 10 input images with noise
density from 1% to 10%. (c) Results on the dataset with saturated
regions (best viewed on high-resolution displays with zoom-in).
im1 im2 im3 im4 Average16
18
20
22
24
26
28
30
32
34
Ave
rage
PSN
R V
alue
s
Blurred imagesFergus et al.Shan et al.Cho and LeeXu and JiaKrishnan et al.Hirsch et al.Whyte et al.Ours
1.5 2 2.5 310
20
30
40
50
60
70
80
90
100
Error ratios
Succ
ess
rate
(%)
Fergus et al.Shan et al.Cho and LeeXu and JiaKrishnan et al.Levin et al.Zhong et al.Xu et al.Pan et al.Ours
1 1.5 2 2.5 3 3.5 4 4.5 5Error ratios
0
10
20
30
40
50
60
70
80
90
100
Succ
ess
rate
(%)
Cho and LeeXu and JiaKrishnan et al.Levin et al.Sun et al.Michaeli and IraniOus
(a) Results on [15] (b) Results on [20] (c) Results on [30]
Figure 16. Quantitative evaluations on two benchmark datasets.
kernels to generate blurred images where the pixel intensi-
ties are clipped into the range of [0, 1]. Finally, we add 1%
random noise on each blurred image. Figure 15(c) shows
that the proposed algorithm achieves favorable results com-
pared with state-of-the-arts. We note that the proposed al-
gorithm can also be applied to deblur images with Gaussian
noise. More experimental results are included in the sup-
plemental document.
In addition, we use the natural image deblurring
datasets [15], [20], and [30] for evaluation with correspond-
ing metrics [15, 20]. Figure 16 shows that the proposed
algorithm performs well on both datasets against the state-
of-the-art blind deblurring methods.
6. Conclusions
In this work, we propose a robust kernel estimation al-gorithm in which effective edges are selected for deblur-ring images containing significant amount of outliers. Wepresent detailed analysis on the effects of outliers on ker-nel estimation. Furthermore, we show that the proposedmethod can be applied to improve the accuracy of exist-ing blind deblurring methods. In the final deconvolutionstep, we develop a robust method to restore the latent im-age under the guidance of the proposed outlier-aware func-tion where the effects of outliers are minimized. Exten-sive experimental evaluations on real images and bench-mark datasets demonstrate the proposed algorithm performsfavorably against the state-of-the-art methods for uniformas well as non-uniform deblurring.
Acknowledgements: J. Pan is supported by a scholarship from
China Scholarship Council. Z. Lin is supported by National Basic
Research Program of China (973 Program) (No. 2015CB352502),
NSFC (No. 61272341 and 61231002), and MSRA. Z. Su is sup-
ported by the NSFC (No. 61572099 and 61320106008). M.-
H. Yang is supported in part by the NSF CAREER Grant (No.
1149783), NSF IIS Grant (No. 1152576), and a gift from Adobe.
2807
References
[1] L. Bar, N. Kiryati, and N. A. Sochen. Image deblurring in
the presence of impulsive noise. IJCV, 70(3):279–298, 2006.
[2] J.-F. Cai, H. Ji, C. Liu, and Z. Shen. Framelet based
blind motion deblurring from a single image. IEEE TIP,
21(2):562–572, 2012.
[3] S. Cho and S. Lee. Fast motion deblurring. In SIGGRAPH
Asia, pages 145:1–145:8, 2009.
[4] S. Cho, J. Wang, and S. Lee. Handling outliers in non-blind
image deconvolution. In ICCV, pages 495–502, 2011.
[5] T. S. Cho, S. Paris, B. K. P. Horn, and W. T. Freeman. Blur
kernel estimation using the radon transform. In CVPR, pages
241–248, 2011.
[6] Z. Farbman, R. Fattal, D. Lischinski, and R. Szeliski. Edge
preserving decompositions for multi-scale tone and detail
manipulation. In SIGGRAPH, 2008.
[7] R. Fergus, B. Singh, A. Hertzmann, S. T. Roweis, and W. T.
Freeman. Removing camera shake from a single photograph.
In SIGGRAPH, pages 787–794, 2006.
[8] M. D. Grossberg and S. K. Nayar. Modeling the space
of camera response functions. IEEE TPAMI, 26(10):1272–
1282, 2004.
[9] S. Harmeling, S. Sra, M. Hirsch, and B. Scholkopf. Multi-
frame blind deconvolution, super-resolution, and saturation
correction via incremental EM. In ICIP, pages 3313–3316,
2010.
[10] J. A. Hertz, A. S. Krogh, and R. G. Palmer. Introduction to
the Theory of Neural Computation. Westview Press, 1991.
[11] M. Hirsch, C. J. Schuler, S. Harmeling, and B. Scholkopf.
Fast removal of non-uniform camera shake. In ICCV, pages
463–470, 2011.
[12] Z. Hu, S. Cho, J. Wang, and M.-H. Yang. Deblurring low-
light images with light streaks. In CVPR, pages 3382–3389,
2014.
[13] J. Jia. Mathematical models and practical solvers for uni-
form motion deblurring. Cambridge University Press, 2014.
[14] N. Joshi, R. Szeliski, and D. J. Kriegman. PSF estimation
using sharp edge prediction. In CVPR, pages 1–8, 2008.
[15] R. Kohler, M. Hirsch, B. J. Mohler, B. Scholkopf,
and S. Harmeling. Recording and playback of camera
shake: Benchmarking blind deconvolution with a real-world
database. In ECCV, pages 27–40, 2012.
[16] D. Krishnan and R. Fergus. Fast image deconvolution using
hyper-Laplacian priors. In NIPS, pages 1033–1041, 2009.
[17] D. Krishnan, T. Tay, and R. Fergus. Blind deconvolution
using a normalized sparsity measure. In CVPR, pages 2657–
2664, 2011.
[18] S. Lee and S. Cho. Recent advances in image deblurring. In
SIGGRAPH Asia 2013 Course, 2013.
[19] A. Levin, R. Fergus, F. Durand, and W. T. Freeman. Image
and depth from a conventional camera with a coded aperture.
In SIGGRAPH, pages 70–78, 2007.
[20] A. Levin, Y. Weiss, F. Durand, and W. T. Freeman. Under-
standing and evaluating blind deconvolution algorithms. In
CVPR, pages 1964–1971, 2009.
[21] A. Levin, Y. Weiss, F. Durand, and W. T. Freeman. Efficient
marginal likelihood optimization in blind deconvolution. In
CVPR, pages 2657–2664, 2011.
[22] L. B. Lucy. An iterative technique for the rectification of