Lecture 9 Perceptual Image Quality Assessment 09-Perceptual IQ… · Lin ZHANG, SSE, 2016 Lecture 9 Perceptual Image Quality Assessment Lin ZHANG, PhD School of Software Engineering

Lin ZHANG, SSE, 2016

Lecture 9Perceptual Image Quality Assessment

Lin ZHANG, PhDSchool of Software Engineering

Tongji UniversityFall 2016


Contents

• Problem definition• Full reference image quality assessment• No reference image quality assessment• Summary


Problem DefinitionPlease rank these images according to their visual quality

(a) (b)

(c) (d)

A subjective process

Can we have some algorithms to measure the image quality? And the results is highly consistent with the human judgments

Our goal in this lecture!


Problem Definition

• The goal of the IQA research is to develop objective metrics for measuring image quality and the results should be consistent with the subjective judgments

• Classification of the IQA problem• Full reference IQA (FR‐IQA)

• The distortion free image is given. Such an image is considered to have a perfect quality and is called reference image. A set of its distorted versions are also provided. Your task is to devise an algorithm to evaluate the perceptual quality of distorted images


Problem Definition


• Classification of the IQA problem• Reduced reference IQA (RR‐IQA)

• The distorted image is given; The reference image is not available; however, partial information of the reference image is known


Problem Definition


• Classification of the IQA problem• Reduced reference IQA (RR‐IQA)


Problem Definition


• Classification of the IQA problem• No reference IQA (NR‐IQA)

• Only the distorted image is given. Or more accurately in such a case, we cannot call it as "distorted" image since we do not know the corresponding distortion‐free reference image. You need to design an algorithm to evaluate the quality of the given image


Contents

• Problem definition• Full reference image quality assessment

• Application scenarios• Problem of the classical FR‐IQA metric—MSE• Error visibility method• Structural Similarity (SSIM)• Feature Similarity (FSIM)• Performance metrics

• No reference image quality assessment• Summary


Application Scenarios

• Quantify the performance of de‐noising algorithms

I

simulation

'IAI

algoA

BIalgoB

denoising results

Which algorithm is better?

has better quality than

We need to design a metric function f having the following property:

( , ) ( , )A Bif f I I f I Ihas better quality than ; AI BI

otherwise, BI AISuch an f is our desired FR‐IQA metric



• Quantify the performance of compression algorithms

I

Which compression algorithm is better?

AI

algoA

BI

algoB

compression results

We also need a FR‐IQA metric



• FR‐IQA metrics usually can be used in the following applications• Measure the performance of some image enhancement or restoration algorithms, such as algorithms for denoising, deblurring, dehazing, etc

• Measure the performance of image compression algorithms• Used to adjust parameters of some image processing algorithms


Contents





Problem of the Classical FR‐IQA Metric—MSE

• MSE (mean squared error) is a classical metric to measure the similarity between two image signals

• MSE is a point‐to‐point based measure

• Advantages• Easy to compute• Easy to optimize• Clear physical meaning: energy

• What’s the problem?

image ximage y

1/221

i iiN

x y

MSE


• MSE is point‐to‐point and doesn’t care about ordering

MSE = 1600, MSSIM = 0.6373 MSE = 1600, MSSIM = 0.0420

MSE thinks that the similarity between I1and I2 and the similarity between I3and I4 are the same; this contradicts with the human intuition

1I

2I

reorder

3I

reorder

4I



1/ 221

i ii

x yN

+ 30

+ (rand sign)* 30

MSE = 900

SSIM = 0.9329

MSE = 900

SSIM = 0.2470

Don’t care about the sign




• Mean Squared Error

1/ 221

i ii

E x yN

signal samplesare independent

signal sampleshighly correlated

• Natural Images

highly structured

Conflict


Contents





Error Visibility Method: Idea

• Representative work• Frequency weighting[Mannos & Sakrison ’74]• Sarnoff model [Lubin ’93]• Visible difference predictor [Daly ’93]• Perceptual image distortion [Teo & Heeger ’94]• DCT‐based method [Watson ’93]• Wavelet‐based method [Safranek ’89, Watson et al. ’97]

distorted signal = reference signal + error signal

Quantify error signal perceptually


Error Visibility Method: Framework

• Goal: simulate relevant early HVS components• Structures motivated by physiology• Parameters determined by psychophysics


• Contrast sensitivity function

CSF

10-2

10-1

100

spatial frequency (cycles/degree)

no

rma

lize

d s

en

sit

ivit

y

10-1

100

101

102

In this image, the contrast amplitude depends only on the vertical coordinate, while the spatial frequency depends on the horizontal coordinate. Observe that for medium frequency you need less contrast than for high or low frequency to detect the sinusoidal fluctuation

Error Visibility Method—HVS Properties Modeling


Error Visibility Method—HVS Properties Modeling

• Masking

highly visible

weak masking

hardly visible

strong masking


Error Visibility Method—Difficulties

• Natural image complexity problem• Based on simple‐pattern psychophysics

• Quality definition problem• Error visibility = quality ?


Contents





Structural Similarity (SSIM)

Purpose of vision: extract structural information

Quantify structural distortion

• Questions:• How to define structural/nonstructural distortions?• How to separate structural/nonstructural distortions?



• What are structural/non‐structural distortions?

non‐structural distortions

luminance change contrast change

Gamma distortion spatial shift JPEG blocking wavelet ringing

blurring noise contamination

structural distortions




distortedimage

originalimage




structuraldistortion

distortedimage

originalimage

nonstructuraldistortion





+

distortedimage

originalimage






+

distortedimage

originalimage

+



Structural Similarity (SSIM)—Computation

For two corresponding local patches x and y in two images

Luminance Comparison

Contrast Comparison

Structure Comparison

CombinationSimilarity Measure

( )x y

( , )l x y

( , )c x y

( , )s x y

is the mean intensity of x (y),

( )x y is the standard deviation of x (y),

xy is the covariance of x and y,

Assume that x and y are vectorized as 1 2, ,..., Nx x xx 1 2, ,..., Ny y yyand

1

1 Nx i

ix

N

1/ 2

2

1

1 Nx i x

ix

N

1

1 Nxy i x i y

ix y

N



12 2

1

2( , ) x y

x y

Cl

C

x y 22 22

2( , ) x y

x y

Cc

C

x y 33

( , ) x yx y

Cs

C

x y, ,

1 2

2 2 2 21 2

2 2( , ) ( , ) ( , ) ( , ) x y xy

x y x y

C CSSIM l c s

C C

x y x y x y x y

Then, the structure similarity between x and y are defined as 1 2 3, ,C C C are fixed constants, and usually set 3 2 / 2C C

If the image contains M local patches (defined by a sliding window), the overall image quality is

1

1SSIM ( , )M

i ii

SSIMM

x y



[Wang & Bovik, IEEE Signal Proc. Letters, ’02][Wang et al., IEEE Trans. Image Proc., ’04]

distortion/similaritymeasure withinsliding window

original image

distorted image

quality map

pooling

quality score

1 22 2 2 2

1 2

(2 )(2 )( , )

( )( )x y xy

x y x y

C CSSIM

C C

x y



original image

Gaussian noise corrupted image

absolute error map

SSIM index map



JPEG2000 compressed image

original image

SSIM index map

absolute error map



JPEG compressed image

original image

SSIM index map

absolute error map


Comparison between MSE and SSIM

MSE=0, SSIM=1 MSE=309, SSIM=0.928 MSE=309, SSIM=0.987

MSE=309, SSIM=0.580 MSE=309, SSIM=0.641 MSE=309, SSIM=0.730

original Image


Comparison between MSE and SSIM

referenceimage

initialimage

converged image(best SSIM)

equal-MSEcontour

converged image(worst SSIM)


Summary about SSIM

• Structural similarity (SSIM) metric measures the structure distortions of images

• In implementation, SSIM measures the similarity of two local patches from three aspects, luminance, contrast, and structure

• The quality scores predicted by SSIM is much more consistent with human judgments than MSE

• SSIM is now widely used to gauge image processing algorithms

In the next section, you will encounter an even more powerful IQA metric, FSIM


Contents


• Application scenarios• Problem of the classical FR‐IQA metric—MSE• Error visibility method• Structural Similarity (SSIM)• Feature Similarity (FSIM)

• Phase congruency• Feature similarity index (FSIM)

• Performance metrics



Phase Congruency

• Why is phase important?

2 ( )( ) ( ) ( ) ( )i ux i uf x F u f x e dx A u e Fourier transform( )u is called the Fourier phase or the global phase

• Phase is defined for a specified frequency• The Fourier phase indicates the relative position of the frequency components

• Phase is a real number between


Phase Congruency

• Why is phase important?

Fourier

Hilbert

Reconstruction results

From Fourier’s amplitude

From Fourier’s phase

From Fourier’s phase + Hilbert’s amplitude


Phase Congruency

• Local phase analysis

Question: What are the frequency components (and the associated phases) at a certain position in a real signal f(x) ?

Fourier transforms cannot answer such questions


Phase Congruency

• Local phase analysis

( ) ( ) ( )A Hf x f x if x Analytic signal needs to be constructed

where1( ) ( )* ( ), ( )Hf x h x f x h x x

is called the Hilbert transform of f(x)( )Hf x

Instantaneous phase: ( ) arctan 2 ( ), ( )Hx f x f x

Instantaneous amplitude: 2 2( ) ( ) ( )HA x f x f x

seems local, but not so since HT is a global transform( )x


Phase Congruency

• Local phase analysisThus, local complex filters whose responses are analytic signals themselves are used instead

That isIf is a complex filter and ( ) ( ) ( )e og x g x ig x

( )* ( ) ( )* ( ) ( )* ( )e og x f x g x f x ig x f x is an analytic signal, then, the local phase (instead of the instantaneous phase) of f(x)is defined as

( ) arctan 2 ( ) * ( ), ( ) * ( )o ex g x f x g x f x The local amplitude is

2 2( ) ( ) * ( ) ( ) * ( )e oA x g x f x g x f x


Phase Congruency

• Local phase analysisThus, local complex filters whose responses are analytic signals themselves are used instead

That isIf is a complex filter and ( ) ( ) ( )e og x g x ig x

( )* ( ) ( )* ( ) ( )* ( )e og x f x g x f x ig x f x is an analytic signal,

and are called a quadrature pairogeg

What are the commonly used quadrature pair filters?See the next sections!


Phase Congruency

• Gabor filter

' 2 ' 2

'2 2

1( , ) exp exp 22 x y

x yG x y i fx

where ' 'cos sin , sin cosx x y y x y

(1)


Phase Congruency

• Gabor filter

' 2 ' 2

'2 2

1( , ) exp exp 22 x y

x yG x y i fx


(1)

John Daugman, University of Cambridge, UK

Denis Gabor, 1900~1979, Nobel Prize Winner


Phase Congruency

• Gabor filter

' 2 ' 2

'2 2

1( , ) exp exp 22 x y

x yG x y i fx


(1)

Primary Cortex


Phase Congruency

• Gabor filter

' 2 ' 2

'2 2

1( , ) exp exp 22 x y

x yG x y i fx


(1)

J. G. Daugman, Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by two‐dimensional visual cortical filters, Journal of the Optical Society of America A, 2(7):1160–1169, 1985.


Phase Congruency

• Log‐Gabor filter• It is also a quadrature pair filter; defined in the frequency domain

2202 2 2

log /( , ) exp exp

2 2j

jr

G

where is the orientation angle, is the center frequency, controls the filter’s radial bandwidth, and determines the angular bandwidth

/j j J 0r

radial part angular part Log‐Gabor


Phase Congruency—Motivation

• Gradient‐based feature detectors• Roberts, Prewitt, Sobel, Canny et al…..• Find maximum in the gradient map • Sensitive to illumination and contrast variations• Poor localization, especially with scale analysis• Difficult to use—threshold problem. One does not know in advance what level of edge strength corresponds to a significant feature



Harris corners, Harris corners, 1 7



• Phase congruency is proposed to overcome those drawbacks• Totally based on the local phase information• A more general framework for feature definition• Invariant to contrast and illumination variation• Offers the promise of allowing one to specify universal feature thresholds


Phase Congruency—Definition

• First appears in [1] • It is more like the human visual system• It postulates that features are perceived at points of maximum phase congruency

[1] M.C. Morrone, J. Ross, D.C. Burr, and R. Owens, Mach bands are phase dependent, Nature, vol. 324, pp. 250‐253, 1986

[all the following discussions will be based on this observation]


Phase Congruency—Definition

• Features from the PC view. Fourier components are all in phase in the two cases


Phase Congruency—Computation

• Now the widely used to method to compute phase congruency is [1]

• In [1], Kovesi proposed a framework to compute PC by using quadrature pair filters

[1] P. Kovesi, Image features from phase congruency, Videre: Journal of Computer Vision Research, vol. 1, pp. 1‐26, 1999


Phase Congruency—Computation

denote the even‐symmetric and odd‐symmetric wavelets at a scale

,e on nM Mn

( ), ( ) ( )* , ( )*e on n n ne x o x I x M I x M

( ) ( ), ( ) ( )n nn n

F x e x H x o x

The amplitude and phase of the transform at a given wavelet scale is given by

2 2( ) ( ) ( )n n nA x e x o x

and can be estimated as:( )F x ( )H x

( )( )( )n

n

E xPC xA x

2 2( ) ( ) ( )E x F x H x

( )( )( )

nn

n

o xx arctge x


Phase Congruency—Example


Contents


• Application scenarios• Problem of the classical FR‐IQA metric—MSE• Error visibility method• Structural Similarity (SSIM)• Feature Similarity (FSIM)

• Phase congruency• Feature similarity index (FSIM)

• Performance metrics



Feature Similarity Index (FSIM)

• A state‐of‐the‐art method proposed in [1]

[1] Lin Zhang, Lei Zhang, Xuanqin Mou, and David Zhang, FSIM: A feature similarity index for image quality assessment, IEEE Trans. Image Processing, vol. 20, pp. 2378‐2386, 2011



• A state‐of‐the‐art method proposed in [1]• Motivations

• Low‐level feature inspired• Visual information is often redundant• low‐level features convey most crucial information• Image degradations will lead to changes in image low‐level features

Thus, an IQA index could be devised by comparing the low‐level features between the reference image and the distorted image

What kinds of features?



• Phase congruency • Physiological and psychophysical evidences• Measure the significance of a local structure

• Gradient magnitude• PC is contrast invariant. However, local contrast indeed will affect the perceptive image quality

• Thus, we have to compensate for the contrast• Gradient magnitude can be used to measure the contrast similarity



• Phase congruency—An example



• Gradient magnitude

1 10 * ( ), 0 0 0 * ( )16 16

3 3 10 3x yG f G f

x x

Scharr operator to extract the gradient

Gradient magnitude (GM):

2 2x yG G G



• FSIM computationGiven two images, f1 and f2Their PC maps, PC1 and PC2Their GM maps, G1 and G2

PC similarity 1 2 12 21 2 1

2 ( ) ( )( )( ) ( )PC

PC PC TSPC PC T

x xx

x x

GM similarity 1 2 22 21 2 2

2 ( ) ( )( )( ) ( )G

G G TSG G T

x xx

x x

( ) ( ) ( )FSIM

( )PC G m

m

S S PCPC

x

x

x x xx

where 1 2( ) max ( ), ( )mPC PC PCx x x

T1 is a constant

T2 is a constant



• Extended to a color IQASeparate the chrominance from the luminance

0.299 0.587 0.1140.596 0.274 0.3220.211 0.523 0.312

Y RI GQ B

I1(I2) and Q1(Q2) be the I and Q channels of f1 and f2

1 2 32 2

1 2 3

2 ( ) ( )( )( ) ( )I

I I TSI I T

x xx

x x1 2 4

2 21 2 4

2 ( ) ( )( )( ) ( )Q

Q Q TSQ Q T

x xx

x x

( ) ( ) ( ) ( ) ( )FSIM

( )PC G I Q m

Cm

S S S S PCPC

x

x

x x x x xx


Feature Similarity Index (FSIM)—Schematic diagram


Summary

• FSIM is a HVS‐driven IQA index• HVS perceives an image mainly based on its low‐level features

• PC and gradient magnitude are used• PC is also used to weight the contribution of each point to the overall similarity of two images

• FSIM is extended to FSIMC, a color IQA index

• FSIM (FSIMC) outperforms all the other state‐of‐the‐art IQA indices evaluated


Contents





Performance Metrics

• How to evaluate the performance of IQA indices?• Some benchmark datasets were created

• Reference images (quality distortion free) are provided• For each reference image, a set of distorted images are created; they suffer from kinds of quality distortions, such as Gaussian noise, JPEG compression, blur, etc; let’s suppose that there are altogether N distorted images

• For each distorted image, there is an associated quality score, given by subjects; thus, altogether we have N scores

• For distorted images, we can compute their objective quality scores by using an IQA index f; we can get N quality scores

• f’s performance can be reflected by the rank order correlation coefficients between and

1{ }N

i is

1{ }N

i io

1{ }N

i is 1{ }N

i io


Performance Metrics

• How to evaluate the performance of IQA indices?

Spearman rank order correlation coefficient (SRCC)

2

12

61

( 1)

N

ii

dSRCC

N N

where di is the difference between the ith image's ranks in the subjective and objective evaluations.

Note: in Matlab, you can compute the SROCC by usingsrcc = corr(vect1, vect2, 'type', 'spearman')


Performance Metrics

• How to evaluate the performance of IQA indices?

Kendall rank order correlation coefficient (KRCC)

0.5 ( 1)c dn nKRCC

N N

where nc is the number of concordant pairs and nd is the number of discordant pairs

Note: in Matlab, you can compute the SROCC by usingkrcc = corr(vect1, vect2, 'type', ‘kendall')


Performance Metrics

• Popular used benchmark datasets for evaluating IQA indices

Database name

Reference Images

Distorted images

Observer numbers

Distortion types

TID2013 [1] 25 2000 971 24

TID2008 [2] 25 1700 838 17

CSIQ [3] 30 866 35 6LIVE [4] 29 779 161 5

[1] http://www.ponomarenko.info/tid2013.htm[2] http://www.ponomarenko.info/tid2008.htm[3] http://vision.okstate.edu/?loc=csiq[4] http://live.ece.utexas.edu/research/Quality/


Performance Metrics—Comparison of IQA Indices

FSIM FSIMC MS‐SSIM VIF SSIM IFC VSNR NQM

TID2013

SRCC 0.8015 0.8510 0.7859 0.6769 0.7417 0.5389 0.6812 0.6392

KRCC 0.6289 0.6665 0.6047 0.5147 0.5588 0.3939 0.5084 0.4740

TID2008

SRCC 0.8805 0.8840 0.8528 0.7496 0.7749 0.5692 0.7046 0.6243

KRCC 0.6946 0.6991 0.6543 0.5863 0.5768 0.4261 0.5340 0.4608

CSIQSRCC 0.9242 0.9310 0.9138 0.9193 0.8756 0.7482 0.8106 0.7402

KRCC 0.7567 0.7690 0.7397 0.7534 0.6907 0.5740 0.6247 0.5638

LIVESRCC 0.9634 0.9645 0.9445 0.9631 0.9479 0.9234 0.9274 0.9086

KRCC 0.8337 0.8363 0.7922 0.8270 0.7963 0.7540 0.7616 0.7413

Note: For more details about full reference IQA, you can refer to http://sse.tongji.edu.cn/linzhang/IQA/IQA.htm


Contents

• Problem definition• Full reference image quality assessment• No reference image quality assessment

• Background introduction• Our proposed method: IOUML

• Summary


Background introduction—Problem definition• No reference image quality assessment (NR‐IQA)

•Devise computational models to estimate the quality of a given image as perceived by human beings

• The only information an NR‐IQA algorithm receives is the image whose quality is being assessed itself



How do you think the quality of these two images?

Though you are not provided the ground‐truth reference images, you may judge the quality of these two images as poor



How do you think about the qualities of these images?Rank them

Remember that you DONOT know the ground‐truth “high quality” reference image


Background introduction—Typical methods• Opinion‐aware approaches

• These approaches require a dataset comprising distorted images and associated subjective scores

• At the training stage, feature vectors are extracted from images and then the regression model, mapping the feature vectors to the subjective scores, is learned

• At the testing stage, a feature vector is extracted from the test image, and its quality score can be predicted by inputting the feature vector to the learned regression model



feature vectors



• BIQI [1]• BRISQUE [2]• BLIINDS [3]• BLIINDS‐II [4]• DIIVINE [5]• CORNIA [6]• LBIQ [7]

Proposed by Bovik’s group, Univ. Texas

http://live.ece.utexas.edu/



– [1] A. Moorthy and A. Bovik, A two‐step framework for constructing blind image quality indices, IEEE Sig. Process. Letters, 17: 513‐516, 2010

– [2] A. Mittal, A.K. Moorthy, and A.C. Bovik, No‐reference image quality assessment in the spatial domain, IEEE Trans. Image Process., 21: 4695‐4708, 2012

– [3] M.A. Sadd, A.C. Bovik, and C. Charrier, A DCT statistics‐based blind image quality index, IEEE Sig. Process. Letters, 17: 583‐586, 2010

– [4] M.A. Sadd, A.C. Bovik, and C. Charrier, Blind image quality assessment: A natural scene statistics approach in the DCT domain, IEEE Trans. Image Process., 21: 3339‐3352, 2012

– [5] A.K. Moorthy and A.C. Bovik, Blind image quality assessment: from natural scene statistics to perceptual quality, IEEE Trans. Image Process., 20: 3350‐3364, 2011

– [6] P. Ye, J. Kumar, L. Kang, and D. Doermann, Unsupervised feature learning framework for no‐reference image quality assessment, CVPR, 2012

– [7] H. Tang, N. Joshi, and A. Kapoor. Learning a blind measure of perceptual image quality, CVPR, 2011


Background introduction—Typical methods• Opinion‐unaware approaches

• These approaches DONOT require a dataset comprising distorted images and associated subjective scores

• A typical method is NIQE [1]• Offline learning stage: constructing a collection of quality‐aware features from pristine images and fitting them to a multivariate Gaussian (MVG) model

• Testing stage: the quality of a test image is expressed as the distance between a MVG fit of its features and

[1] A. Mittal et al. Making a “completely blind” image quality analyzer. IEEE Signal Process. Letters, 20(3): 209-212, 2013.


Contents


• Background introduction• Our proposed method: IL‐NIQE

• Motivations and our contributions• NIS‐induced quality‐aware features• Pristine model learning• IL‐NIQE index• Experimental results

• Summary


Motivations[1]

• Opinion‐unaware approaches seems appealing, so we want to propose an opinion‐unaware approach

• Design rationale• Natural images without quality distortions possess regular statistical properties that can be measurably modified by the presence of distortions

• Deviations from the regularity of natural statistics, when quantified appropriately, can be used to assess the perceptual quality of an image

• NIS‐based features have been proved powerful. Any other NIS‐based features?

[1] Lin Zhang et al., A feature‐enriched completely blind image quality evaluator, IEEE Trans. Image Processing 24 (8) 2579‐2591, 2015


Contributions• A novel “opinion‐unaware” NR‐IQA index, IL‐NIQE (Integrated Local‐NIQE)• A set of prudently designed NIS‐induced quality‐aware features

• Bhattacharyya distance based metric to measure the quality of a local image patch

• A visual saliency based quality score pooling scheme• A thorough evaluation of the performance of modern NR‐IQA indices


Contents




• Summary


• Statistics of normalized luminance• The mean subtracted contrast normalized (MSCN) coefficients have been observed to follow a unit normal distribution when computed from natural images without quality distortions [1]

• This model, however, is violated when images are subjected to quality distortions; the degree of violation can be indicative of distortion severity

IL‐NIQE—NIS‐induced quality‐aware features

( , )nI x y

[1] D.L. Ruderman. The statistics of natural images. Netw. Comput. Neural Syst., 5(4):517-548, 1994.


• Statistics of normalized luminance


( , ) ( , )( , )( , ) 1n

I x y x yI x yx y

,( , ) ( , )K L

k lk K l L

x y I x k y l

2,( , ) ( , ) ( , )K L

k lk K l L

x y I x k y l x y

where

Conforms to Gaussian


• Statistics of normalized luminance•We use a generalized Gaussian distribution (GGD) to model the distribution of


( , )nI x y

( ; , ) exp

2 1 /x

g x

Density function of GGD,

Parameters are used as quality‐aware features which can be estimated from {In(x, y)} by MLE

,


• Statistics of MSCN products• The distribution of products of pairs of adjacent MSCN coefficients, In(x, y)In(x, y+1), In(x, y)In(x+1, y), In(x, y)In(x+1, y+1), and In(x, y)In(x+1, y-1), can also capture the quality distortion



• Statistics of MSCN products• They can be modeled by asymmetric generalized Gaussian distribution (AGGD),


exp / , 01 /

( ; , , )exp / , 0

1 /

ll r

l r

rl r

x xg x

x x

The mean of AGGD is 2 / / 1 /r l

, , ,r l are used as “quality‐aware” features


• Statistics of partial derivatives and gradient magnitudes• We found that when introducing quality distortions to an image, the distribution of its partial derivatives, and gradient magnitudes, will be changed



• Statistics of partial derivatives and gradient magnitudes


1(a)

1(b)

1(d)

1(c)

1(e)




-0.015 -0.01 -0.005 0 0.005 0.01 0.0150

1

2

3

4

5

6

partial derivative (normalized)

Perc

enta

ge (%

)

Fig. 1(a)Fig. 1(b)Fig. 1(c)Fig. 1(d)Fig. 1(e)

0 0.005 0.01 0.0150.5

1

1.5

2

2.5

3

3.5

gradient magnitude (normalized)

Perc

enta

ge (%

)

Fig. 1(a)Fig. 1(b)Fig. 1(c)Fig. 1(d)Fig. 1(e)




Partial derivatives* ( , ), * ( , )x x y yI I G x y I I G x y

where, 2 24 2

2 2

4 2

( , ) exp2 2

( , ) exp2 2

x

y

x x yG x y

y x yG x y

Gradient magnitudes2 2( , ) x yGM x y I I


• Statistics of partial derivatives and gradient magnitudes• We use a GGD to model the distributions of Ix (or Iy) and take its parameters as features

• We use a Weibull distribution [1] to model the distribution of the gradient magnitudes and use the parameters as features,


1 exp , 0

; ,

0, 0

aa

a

a xx xh x a b b b

x

a and b are used as features

[1] J.M. Geusebroek and A.W.M. Smeulders. A six-stimulus theory for stochastic texture. Int. J. Comp. Vis., 62(1): 7-16, 2005.


• Statistics of image’s responses to log‐Gabor filters• Motivation: neurons in the visual cortex respond selectively to stimulus’ orientation and frequency, statistics on the images’ multi‐scale multi‐orientation decompositions should be useful for designing a NR‐IQA model



• Statistics of image’s responses to log‐Gabor filters• For multi‐scale multi‐orientation filtering, we adopt the log‐Gabor filter,


2

20

22

log

222 ,

j

rG e e

where is the orientation angle, is the center frequency, controls the filter’s radial bandwidth, and determines the angular bandwidth

/j j J 0r

radial part angular part Log‐Gabor


• Statistics of image’s responses to log‐Gabor filters


With log‐Gabor filters having J orientations and N center frequencies, we could get response maps

, ,{( ( ), ( )) :| 0,..., 1, 0,..., 1}n j n je o n N j J x xwhere and represents the image’s response to the real and imaginary part of the log‐Gabor filter

, ( )n je x , ( )n jo x

We extract the quality‐aware features asa) Use a GGD model to fit the distribution of {en,j(x)} (or {on,j(x)}) and take

the model parameters α and β as features.b) use a GGD to model the distribution of partial derivatives of {en,j(x)} (or

{on,j(x)}) and also take the two model parameters as features.c) Use a Weibull model to fit the distribution of gradient magnitudes of

{en,j(x)} (or {on,j(x)}) and take the corresponding parameters a and b as features


• Statistics of colors• Ruderman et al. showed that in a logrithmic‐scale opponent color space, the distributions of the image data conform well to Gaussian [1]


[1] D.L. Ruderman et al. Statistics of cone response to natural images: implications for visual coding. J. Opt. Soc. Am. A, 15(8): 2036-2045, 1998.


• Statistics of colors


RGB to logarithmic signal with mean subtracted,( , ) log ( , ) log ( , )

( , ) log ( , ) log ( , )( , ) log ( , ) log ( , )

x y R x y R x yx y G x y G x yx y B x y B x y

where means the mean of logX(x,y)>

to opponent color space

1

2

3

( , ) ( ) / 3

( , ) ( 2 ) / 6

( , ) ( ) / 2

l x y

l x y

l x y

For natural images, l1, l2, and l3 conform well to Gaussian




-0.2 -0.1 0 0.1 0.20

1

2

3

4

5

6

7

8

Perc

enta

ge (%

)

Fig. 3(a)Fig. 3(b)Fig. 3(c)

l1 coefficients-0.2 -0.1 0 0.1 0.20

2

4

6

8

10

12

Perc

enta

ge (%

)


l2 coefficients

-0.2 -0.1 0 0.1 0.20

2

4

6

8

10

12

14

16

18

20

Perc

enta

ge (%

)


l3 coefficients

3(a)

3(b)

3(c)




We use Gaussian to fit the distribution of l1, l2, and l3,2

22

1 ( )( ; , ) exp22xf x

For each l1, l2, and l3 channel, we estimate the two parameters ζ and ρ2 and take them as quality‐aware features


Contents




• Summary


• The pristine model acts as a “standard” for representing characteristics of high quality images

• It is learned from a pristine image set collected by us, which contains 92 high quality images

Pristine model learning

Sample high quality images


• Step 1: for each pristine image, it is partitioned into patches

• Step 2: high contrast patches are selected based on local variance field

• Step 3: for each selected patch, the quality‐aware features are extracted. Thus, we can get a feature vector set,

Pristine model learningP P

1{ :| 1,..., }, di ii M x x

where M is the number of patches and d is the feature dimension

d is very large, so we need a further dimension reduction operation


• Step 4: dimension reduction by PCA

Pristine model learning

Suppose is the dimension reduction matrix,d m m d

After the dimension reduction,1d

ix ' 1T mi i

x x

• Step 5: feed into a MVG model and regard it as the pristine model

'1{ }

Mi ix

1/2 1/2

1 1( ) exp22

Tmf

x x v x v

where v is the mean vector and is the covariance matrix The mean vector and the covariance matrix of the pristine model are denoted as v1 and 1


Contents




• Summary


• Step 1: partition the test image into patches• Step 2: for each patch, we extract from it a feature vector; thus, we can get a feature vector set,

IL‐NIQE indexP P

1{ :| 1,..., }, di t ii M y y

where Mt denotes the number of patches extracted from test image

• Step 3: reduce the dimension of yi as' ' 1,T mi i i

y y y

• Step 4: fit a MVG from and denote its covariance matrix as

'1{ } t

Mi iy

2


• Step 5: the quality qi of patch i is measured as

IL‐NIQE index

1

' '1 21 12

T

i i iq

v y v y

Such a metric is inspired from the Bhattacharyya distance

• Step 6: visual saliency guided quality pooling• High salient patches are given high weights• Patch saliency si is computed as the sum of saliency values covered by patch i

• For saliency computation, we use the Spectral Residual approach [1]

1 1

/t tM M

i i ii i

q q s s

[1] X. Hou and L. Zhang. Saliency detection: a spectral residual approach. CVPR’07, 1-8, 2007.


Offline pristine model learning

…

pristine images

n high-contrast patches

…

patch extraction

n feature vectors

feature extraction

MVG parameters and

MVG fitting

Online quality evaluation of a test image

test image

k image patches

…

patch extraction

k feature vectors

feature extraction

quality score computation for each patch

1 2, ,..., kq q qfinal quality score pooling

1/k iiq q k

μ


Contents




• Summary


Protocol• Protocol for experiments

• Experiments are conducted on TID2013, CSIQ, LIVE, LIVE Multiply‐Distortion

• Spearman rank order correlation coefficient (SRCC) and Pearson linear correlation coefficient (PLCC)

YES1225LIVE MD2YES1225LIVE MD1NO5799LIVENO6866CSIQYES243000TID2013

Contains multiply‐distortions?

Distortion Types No.

Distorted Images No.Dataset

Benchmark image datasets used


Protocol• IL‐NIQE was compared with

• “opinion‐aware” approaches• BIQI, BRISQUE, BLIINDS2, DIIVINE, and CORNIA

• “opinion‐unaware” approaches• NIQE and QAC


Cross‐datasets evaluation• Drawback of single‐database evaluation strategy

• It cannot faithfully measure the prediction performance of NR‐IQA indices since it cannot reflect the “blindness”

• At the training stage the “opinion aware” approaches had already met all the possible distortion types that would appear in the testing stage

• Consequently, we will train the “opinion aware” approaches on one dataset and test their performances on other rest datasets


Cross‐datasets evaluation—Training on LIVE

TID2013 CSIQ MD1 MD2SRCC PLCC SRCC PLCC SRCC PLCC SRCC PLCC

BIQI 0.394 0.468 0.619 0.695 0.654 0.774 0.490 0.766BRISQUE 0.367 0.475 0.557 0.742 0.791 0.866 0.299 0.459BLIINDS2 0.393 0.470 0.577 0.724 0.665 0.710 0.015 0.302DIIVINE 0.355 0.545 0.596 0.697 0.708 0.767 0.602 0.702CORNIA 0.429 0.575 0.663 0.764 0.839 0.871 0.841 0.864NIQE 0.311 0.398 0.627 0.716 0.871 0.909 0.795 0.848QAC 0.372 0.437 0.490 0.708 0.396 0.538 0.471 0.672

IL‐NIQE 0.493 0.586 0.813 0.852 0.891 0.902 0.882 0.895

Evaluation results when being trained on LIVE


Cross‐datasets evaluation—Training on LIVE

BIQI BRISQUE BLIINDS2 DIIVINE CORNIA NIQE QAC IL‐NIQE

SRCC 0.458 0.424 0.424 0.435 0.519 0.429 0.402 0.598PLCC 0.545 0.548 0.525 0.595 0.643 0.512 0.509 0.672

Weighted‐average performance derived from last table


Cross‐datasets evaluation—Training on TID2013

Evaluation results when being trained on TID2013

LIVE CSIQ MD1 MD2SRCC PLCC SRCC PLCC SRCC PLCC SRCC PLCC

BIQI 0.047 0.311 0.010 0.181 0.156 0.175 0.332 0.380BRISQUE 0.088 0.108 0.639 0.728 0.625 0.807 0.184 0.591BLIINDS2 0.076 0.089 0.456 0.527 0.507 0.690 0.032 0.222DIIVINE 0.042 0.093 0.146 0.255 0.639 0.669 0.252 0.367CORNIA 0.097 0.132 0.656 0.750 0.772 0.847 0.655 0.719NIQE 0.906 0.904 0.627 0.716 0.871 0.909 0.795 0.848QAC 0.868 0.863 0.490 0.708 0.396 0.538 0.471 0.672

IL‐NIQE 0.898 0.903 0.813 0.852 0.891 0.902 0.882 0.895


Cross‐datasets evaluation—Training on TID2013

Weighted‐average performance derived from last table

BIQI BRISQUE BLIINDS2 DIIVINE CORNIA NIQE QAC IL‐NIQE

SRCC 0.074 0.384 0.275 0.172 0.461 0.775 0.618 0.860PLCC 0.250 0.491 0.349 0.251 0.527 0.821 0.744 0.881


• We have the following findings• “Opinion aware” indices depend much on the training dataset; it can be seen that these approaches perform better when being trained on LIVE than when being trained on TID2013

• The proposed method IL‐NIQE can achieve the best results nearly in all cases

• The prominent performance of IL‐NIQE indicates that if being designed properly, an “opinion unaware” approach could obtain much better prediction performance than their “opinion aware” counterparts

Cross‐datasets evaluation


Contents

• Problem definition• Full reference image quality assessment• No reference image quality assessment• Summary


• The research in IQA aims to propose computational models to compute the image quality in a subjective‐consistent manner

• IQA problems can be classified as FR‐IQA, RR‐IQA, and NR‐IQA problems according to the availability of the reference information

• Quality scores predicted by the modern FR‐IQA methods can be highly consistent with the subjective ratings

• There is still a large room for development of NR‐IQA methods

Summary


Thanks for your attention

Lecture 9 Perceptual Image Quality Assessment 09-Perceptual IQ… · Lin ZHANG, SSE, 2016 Lecture 9 Perceptual Image Quality Assessment Lin ZHANG, PhD School of Software Engineering

Documents