Image Demosaicing: a Systematic Survey

Image Demosaicing: a Systematic Survey

Xin LiXin Li11, Bahadir Gunturk, Bahadir Gunturk22 and Lei Zhang and Lei Zhang33

11Lane Department of CSEE, West Virginia UniversityLane Department of CSEE, West Virginia University22Dept. of ECE, Louisiana State UniversityDept. of ECE, Louisiana State University33Dept. of Computing, The Hong Kong Polytechnic UniversityDept. of Computing, The Hong Kong Polytechnic University

Growth of Image Demosaicing CommunityGrowth of Image Demosaicing Community

Published papers since Year 1999

Distribution of researchers across the world

AcknowledgementAcknowledgement

Yap-peng Tan at Nanyang Technological University Yap-peng Tan at Nanyang Technological University (NTU), Singapore(NTU), Singapore

David Alleysson at University Pierre Mendes France David Alleysson at University Pierre Mendes France (UPMF)(UPMF)

Daniele Menon at University of Padova, ItalyDaniele Menon at University of Padova, Italy King-Hong Chung at the Hong Kong Polytechnic King-Hong Chung at the Hong Kong Polytechnic

University (HKPU)University (HKPU) Dmitriy Paliy at Tampere University of Technology, Dmitriy Paliy at Tampere University of Technology,

FinlandFinland Chung-Yen Su at National Taiwan Normal UniversityChung-Yen Su at National Taiwan Normal University Darian Muresan previously with Cornell University Darian Muresan previously with Cornell University Keigo Hirakawa at Harvard UniversityKeigo Hirakawa at Harvard University

Outline of This TalkOutline of This Talk

Color science backgroundColor science background– Scientific basis of color-filter-array (CFA)Scientific basis of color-filter-array (CFA)

Image demosaicing problem formulationsImage demosaicing problem formulations– Deterministic in the frequency domain Deterministic in the frequency domain – Statistical in the spatial domainStatistical in the spatial domain

Categorization of existing methodsCategorization of existing methods– Sequential vs. parallel reconstructionSequential vs. parallel reconstruction– Performance evaluationPerformance evaluation

Comparison resultsComparison results– Two image data sets: Kodak CD and IMAX HDTwo image data sets: Kodak CD and IMAX HD

Concluding remarks and open questionsConcluding remarks and open questions– What have we learned? What lies ahead? What have we learned? What lies ahead?

Trichromatic Color Vision Trichromatic Color Vision

(L)(M)(S)

Biological Demosaicing ProblemBiological Demosaicing Problem

Human vision system (HVS) solves this biological demosaicingproblem so well that trichromacy does not affect spatial acuity1

1Alleysson, D.; Susstrunk, S.; Herault, J., "Linear demosaicing inspired by the human visual system," IEEE Transactions on Image Processing,, vol.14, no.4, pp. 439-449, April 2005

Computational Demosaicing ProblemComputational Demosaicing Problem

3-CCD camera

single-CCD camera

S=(R,G,B)

zS=(zR,zG,zB)

Computational Demosaicing Problem (Con’d)Computational Demosaicing Problem (Con’d)

)ˆ,ˆ,ˆ(ˆ BGRS

Objective: minimize the distortion between S and S for the class of images of interests.

^

Statistical vs. Deterministic FormulationStatistical vs. Deterministic Formulation

Original S zG zR/zB

)|(

)|(),|(),|(

HzP

HSPHSzPHzSP

S

SS

Bayesian perspective

Spectral perspective1

1David Alleysson et al., “Frequency selection demosaicking: a review and a look ahead”

Categorization of Existing Demosaicing Categorization of Existing Demosaicing MethodsMethods

H

Hinter Hintra

sequential parallel

Luminance

Chrominance

(Post-processing)

Iterative methods

Vector median filter

Neural network (NN)or VQ-based

Selected Example: Sequential Demosaicing Selected Example: Sequential Demosaicing

(blue channel is processed in a similar fashion to red channel)

)()()(

)|()|()(),,(

GBPGRPGP

GBPGRPGPBGRP

luminance chrominance

Edge-sensitive/directional interpolation, local polynomial approximation … …

Experimental Set-upExperimental Set-up

Eleven latest algorithms have been used in our Eleven latest algorithms have been used in our comparisoncomparison– Lu & Tan (LT): TIP Oct. 2003Lu & Tan (LT): TIP Oct. 2003– Alternating projection (AP): TIP Sep. 2002Alternating projection (AP): TIP Sep. 2002– Adaptive homogeneity-directed (AHD): TIP Mar. 2005Adaptive homogeneity-directed (AHD): TIP Mar. 2005– Successive approximation (SA) with edge-weighted improvement: Successive approximation (SA) with edge-weighted improvement:

TIP Feb. 2005+TCE May 2006TIP Feb. 2005+TCE May 2006– Lukac’s CCA method with post-processing: TCE 2004+ICIP2004Lukac’s CCA method with post-processing: TCE 2004+ICIP2004– Frequency-domain demosaicing (FD): TIP April 2005Frequency-domain demosaicing (FD): TIP April 2005– Directional filtering and a posteriori decision (DFPD): TIP Jan. Directional filtering and a posteriori decision (DFPD): TIP Jan.

20072007– Variance of color difference (VCD): TIP Oct. 2006Variance of color difference (VCD): TIP Oct. 2006– Directional Linear MMSE estimation (DLMMSE): TIP Dec. 2005Directional Linear MMSE estimation (DLMMSE): TIP Dec. 2005– Local polynomial approximation (LPA): IMA 2007Local polynomial approximation (LPA): IMA 2007– Adaptive filtering (AF): TIP Oct. 2007Adaptive filtering (AF): TIP Oct. 2007

Performance Evaluation ProtocolsPerformance Evaluation Protocols

Objective measures: PSNR values + SCIELab metricsObjective measures: PSNR values + SCIELab metrics Subjective evaluation: very limited (mainly to visually Subjective evaluation: very limited (mainly to visually

inspect the severity of various artifacts)inspect the severity of various artifacts)

IMAX test images KODAK test images

PSNR Performance Comparison on KODAK setPSNR Performance Comparison on KODAK set

S-CIELab Measure Comparison on KODAK SetS-CIELab Measure Comparison on KODAK Set

Subjective Performance Comparison ExamplesSubjective Performance Comparison Examples

originaloriginal LTLT APAP AHDAHD SASA CCACCA

FDFD DFPDDFPD VCDVCD DLDL LPALPA AFAF

PSNR Performance Comparison on IMAX setPSNR Performance Comparison on IMAX set

Subjective Performance Comparison ExamplesSubjective Performance Comparison Examples

originaloriginal LTLT APAP AHDAHD SASA CCACCA

AFAF DFPDDFPD VCDVCD DLDL LPALPA NEDINEDI

Discussions and PerspectivesDiscussions and Perspectives What have we learned?What have we learned?

– Color-related problems are hard and our understanding Color-related problems are hard and our understanding of color demosaicing problem remains ad-hocof color demosaicing problem remains ad-hoc

– Many demosaicing techniques might appear different Many demosaicing techniques might appear different but essentially follow a similar motivation but essentially follow a similar motivation

– Kodak image set is a poor benchmark despite its Kodak image set is a poor benchmark despite its popularitypopularity11

What are important questions ahead?What are important questions ahead?– Establishment of an alternative benchmark data set for Establishment of an alternative benchmark data set for

demosaicing researchdemosaicing research– Design of new-generation CFADesign of new-generation CFA22 and video demosaicing and video demosaicing

techniquestechniques– Exploration of its relationship to other tasks such as Exploration of its relationship to other tasks such as

compression, denoising and forensicscompression, denoising and forensics– … …… …

1Xiaolin Wu et al., “Improved color demosaicking in weak spectral correlation” 2Keigo Hirakawa and Patrick J. Wolfe “Second-generation CFA and demosaicking designs”

Conclusion Demosaicing is never an isolated problem; Instead of paying attention to PSNR values, it is often more fruitful to rethink this problem under the context of electronic imaging and ask the right question first.

Ad-hoc Fusion of Different Demoisaicing MethodsAd-hoc Fusion of Different Demoisaicing Methods