ENEE631 Digital Image Processing (Spring'09) Image Forensics & Feature Extraction Spring ’09 Instructor: Min Wu Electrical and Computer Engineering Department,

ENEE631 Digital Image Processing (Spring'09)

Image Forensics & Feature ExtractionImage Forensics & Feature Extraction

Spring ’09 Instructor: Min Wu

Electrical and Computer Engineering Department,

University of Maryland, College Park

bb.eng.umd.edu (select ENEE631 S’09) [email protected]

ENEE631 Spring’09ENEE631 Spring’09Lecture 23 (4/27/2009)Lecture 23 (4/27/2009)

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

04

)

ENEE631 Digital Image Processing (Spring'09)Lec 23 – Image Forensics & Feature

Extraction [2]

Overview and LogisticsOverview and Logistics

Last Time:

– Robust watermark via spread spectrum embedding– Two major types and unification of embedding strategies– Digital forensic fingerprinting for traitor tracing

Today:

– Non-intrusive image forensics– Useful image features and feature extraction techniques

Gradient based; Projection based

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

04

)


Extraction [7]

Recall: Issues to Address in Data HidingRecall: Issues to Address in Data Hiding

Tradeoff among conflicting requirements– Imperceptibility– Robustness & security– Capacity

Key elements of data hiding– Perceptual model– Embedding one bit– Multiple bits– Uneven embedding capacity– Robustness and security– What data to embed

Up

per

L

ayer

s

Uneven capacity equalization

Error correction

Security

……

Low

er

Lay

ers

Imperceptible embeddingof one bit

Multiple-bit embedding

Coding of embedded data

Robustness

Capacity

Imperceptibility

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

ba

sed

on

Re

sea

rch

Ta

lks

’98

-’04

)


Extraction [8]

Embedded Fingerprint for Tracing TraitorsEmbedded Fingerprint for Tracing Traitors

Insert special signals to identify recipients

– Deter leak of proprietary documents– Complementary protection to encryption– Consider imperceptibility, robustness, traceability

– Attacks mounted by single and multiple users

Collusion Attack Collusion Attack (to remove fingerprints)(to remove fingerprints)

AliceAlice

BobBob

Colluded CopyColluded Copy

Unauthorized Unauthorized rere--distributiondistribution

Fingerprinted docfor different users

Collusion Attack Collusion Attack (to remove fingerprints)(to remove fingerprints)

AliceAlice

BobBob

Colluded CopyColluded Copy

Unauthorized Unauthorized rere--distributiondistribution

Fingerprinted docfor different users

Extract Extract FingerprintsFingerprints

Suspicious Suspicious CopyCopy

101110 …101110 …

Codebook

Alice, Bob, …

Identify Identify TraitorsTraitors

Extract Extract FingerprintsFingerprints

Suspicious Suspicious CopyCopy

101110 …101110 …

Codebook

Alice, Bob, …

Identify Identify TraitorsTraitors

Multi-user Attacks

Traitor Tracing

President

Satellite Image

Alice

Bob

Carl

w1

w2

w3

LeakLeak


Extraction [11]

Fingerprinting Topographic MapFingerprinting Topographic Map

– Traditional protection: intentionally alter geospatial content

– Embed much less intrusive digital fingerprint for a modern protection

• 9 long curves are marked; 1331 control points used to carry the fingerprint

1100x1100 Original Map Fingerprinted Map


Extraction [12]

Collusion-Resistant Fingerprinting of MapsCollusion-Resistant Fingerprinting of Maps

2-User Interleaving Attack5-User Averaging Attack

. . .

Can survive combined attacks of collusion + print + scan

Can extend to 3-D Digital Elevation Map


Extraction [14]

Many Forensic Questions …Many Forensic Questions …

arise from military, intelligence, law enforcement, and commercial applications

What type of sensor was used?

Which camera brand took this picture? What model?

What processing has been done?

– Has it been tampered? manipulated?

What technologies were employed?

– Given two images, are they acquired by devices with similar imaging technologies?


Extraction [15]

Break down the info. processing chain into individual components

Identify algorithms and parameters employed in major components of a digital device or processing system

Exploit Intrinsic Fingerprints via Component ForensicsExploit Intrinsic Fingerprints via Component Forensics

Ref: Swaminathan/Wu/Liu in ICASSP’06 and IEEE Trans. Info Forensics & Security (’07)

Color Filter Array (CFA)

ColorInterpolation

White Balancing

Real world scene Digital imageCamera Components

… Sensors …


Extraction [19]

Forensic Estimation and IdentificationForensic Estimation and Identification

Establish a processing model and estimate parameters– Small # possibilities => exhaustive search or by classifier design – More continuous valued parameters => analyze w/ estimation theory

Example: color interpolation in digital camera– Approximate by texture classification and linear filter

(one set of interpolation coeff. for smooth, horizontal & vertical)

– Find best linear estimate of filter coeff. in each class(least-square type of method for robustness)

– Find CFA pattern in a search space that minimizes fitting errors

CFA Interpolation

R ?

? ?

R ?

? ?

R ?

? ?

R ?

? ?

? ?? ?

CandidateCFA pattern

A x = b Interp. equation set

b ~ interpolated pixels; A ~ each row is for neigh-borhood of a interp. pixel (based on directly sensed, known pixels)

Interp. coeff. x and fitting error for each region type and color


Extraction [20]

Experiments with Images from Digital CamerasExperiments with Images from Digital Cameras

Camera Model Camera Model

123456789

10

Canon Powershot A75Canon Powershot S400Canon Powershot S410Canon Powershot S1 IS Canon Powershot G6Canon EOS Digital REBELNikon E4300 Nikon E5400Sony Cybershot DSC P7Sony Cybershot DSC P72

111213141516171819

Olympus C3100Z/C3020ZOlympus C765UZMinolta DiMage S304Minolta DiMage F100Casio QV-UX2000FujiFilm Finepix S3000FujiFilm Finepix A500Kodak CX6330Epson PhotoPC 650

19 cameras and 200 image blocks per camera model

– 512 x 512 regions with maximum gradients chosen for analysis (s.t. have substantial revealing evidence on color interpolation)


Extraction [21]

Detecting Which Camera Detecting Which Camera BrandBrand Took the Image Took the Image

Canon Nikon Sony Olympus Minolta Casio Fuji Kodak Epson

Canon 96% * * * * * * * *

Nikon * 83% 5% * * * * * *

Sony * * 90% * * * * * *

Olympus * * * 93% * * * * *

Minolta 8% * * * 81% * * * *

Casio * * * 6% * 89% * * *

Fuji * * * * 7% * 87% * *

Kodak * * * * * * * 89% *

Epson * * * * * * * * 100%

Interpolation coefficients as features for classifier Average accuracy: 90% for 9 camera brands on uncontrolled scenes Best related work under controlled, uncompressed setting on input scenes

84% for 3 brands, uncompressed [Kharrazi et al’ 05]; 96% for 3 brands [Bayram et al’ 06]

(* denotes values below 4%)


Extraction [23]

Tampering DetectionTampering Detection

Explore intrinsic fingerprints left by various processing modules

– To infer the algorithms and parameters employed in various components of the digital device and processing systems

– New traces or vanished old traces suggests potential post-camera operations

Estimated coeff.

From direct camera output

After post-camera filtering

ColorInterpolation

ColorSensors

Scene Optical Lens System CAMERA

Other SoftwareProcessing

ATampering

/ Stego

B

Black: Sony P72; White: Canon Powershot S410

Grey: Classified as other cameras with low confidence


Extraction [24]

Overview of Scanner ModelOverview of Scanner Model

Tri-linear CCD sensors

Tri-linear CCD sensorsHardcopy

graphic data

Lamp & MirrorsLamp & Mirrors

Lens

Tri-linearcolor filter array

Scanner head

Motion systemMotion system

Shift RegisterAmplifier

A/D Converter

Shift RegisterAmplifier

A/D Converter

Post-processing

Interpolation, Color transformation White balancing, Exposure controlNoise reduction,…

Digitalimage

Software operation

Scanning noise

StatisticalStatisticalnoise featuresnoise features

Scanner modelIdentification


Extraction [26]

E.g.: Noise Features from Wavelet AnalysisE.g.: Noise Features from Wavelet Analysis

ScannedImage I

One stagewavelet

decomposition

HHHLLH

Subband STD &goodness of

Gaussian fitting

Statisticalfeatures

f (3)(I), f (4)(I)

Digital photograph Scanner model 1 Scanner model 2

Histogram Mean and STD Gaussian distribution Goodness of Gaussian fitting

Fitting errorFitting error

HH,HL,LH sub-bandsRGB components2x3x3 = 18 features


Extraction [28]

Acquisition Forensics w/ Noise + Interp. FeaturesAcquisition Forensics w/ Noise + Interp. Features

Q1) What type of device was used to capture the image?

94% accuracy in identifying device type

Q2) What brand/model of the device captured the image?

Cellphone cameras: 98% accuracy over 5 brands

Standalone cameras: 90% over 19 camera models from 9 camera brands

Scanners: 93% accuracy over 9 scanner brands

Cell Phone Camera

Standalone Camera

Scanner

Computer Generated

Input Image

Brand/Model

IdentificationAcquisition Device

Type Identification

Further Forensic Analysis

Sony

Samsung

Nokia

AudiovoxMotorola

CanonFujiFilm

CasioMinolta

Epson

Microtek

AcerScan

Canon

Step 1Step 2


Extraction [30]

Useful Image Features Useful Image Features

and Feature Extraction Techniquesand Feature Extraction Techniques


Extraction [31]

Types of Image Processing TasksTypes of Image Processing Tasks

Image in, Image out– Codec (compression-decompression)– Image enhancement and restoration– Digital watermarking

=> May require both analysis and synthesis operations Intermediate output may be non-image like (coded stream),

but end output should reconstruct into an image close/relate to the input

Image in, Features out

– Features may be used for classification, recognition, and other studies


Extraction [32]

Line FeaturesLine Features

Profile – Project image intensity or other

properties along a given direction– Useful for document and object

analysis

Example-1 locate text linesExample-2 reading banker font

1-D signal via narrow reading head Equiv. to vertical projection,

then 1st order derivative.

Identify colinear points and linesvia Hough transform

Figure 12.7 from Gonzalez’s book resource: “American Bankers Association E-13B font character set and corresponding waveforms


Extraction [33]

Hough TransformHough Transform

(x, y) space parameter space (a, b) y = a x + b b = x a + y

(, ) space: x cos + y sin =

– Representing in (, ) space has similarity to Radon transform

Ref: P. V. C. Hough, "Method and means for recognizing complex patterns." U.S. Patent 3,069,654, 1962.

Figures from Gonzalez’s book resource for Chapter 10


Extraction [34]

– For each point (x0, y0), lines of all angles passing it form a sinusoid curve in (, ) space

– (, ) curves corresponding to colinear points intersect at a point (0, 0)

=> useful for line detection

Illustration of Illustration of Hough TransformHough Transform

Figures from http://en.wikipedia.org/wiki/Hough_transform

(, ) space: x cos + y sin =


Extraction [35]

Example: Detecting Lines in Hough Transf. DomainExample: Detecting Lines in Hough Transf. Domain

Intensity peaks of the transformed results correspond to the (, ) of the respective lines

Can extend to other patterns such as circles

Figures from http://en.wikipedia.org/wiki/Hough_transform


Extraction [36]

Corner DetectorCorner Detector

Corner: intersection of 2 edges– Large variation in local intensity in multiple directions– Alleviate “aperture problem” in motion analysis & image matching

Harris-Stephens detector (1988)– Examine (weighted) Sum of Squared Difference

between an image patch& neighborhood

– Build a 2x2 Harris matrix A Describes 2nd order derivatives of the above SSD

averaged over neighborhood

– Examine the two eigen values of A If both close to 0, there are no features of interest at pixel (x,y). If one close to 0 and the other has large positive value, then an edge is

found. If both have large, distinct positive values, then a corner is found. May avoid computing eigen: examine M = det(A) – k tr(A)2 with k =

0.04-0.15

Ref. and examples at http://en.wikipedia.org/wiki/Corner_detection


Extraction [38]

More on Gradient-based Salient Feature PointsMore on Gradient-based Salient Feature Points Recall operations for robust edge detection ~ e.g. Canny

– Laplacian of Gaussian (LoG) [Gonzalez 3/e 10.2.6] effectively bandpass filtering to suppress noise when taking

derivatives

– Ridge of gradient magnitude and edge linking Local extrema of gradient; seek stable edge info

Recall multiresolution analysis

Achieve invariance to scaling and rotation

– Take account of multiple resolution scale levels– Represent orientation w.r.t. dominant or canonical direction

=> Scale Invariant Feature Transform (SIFT) by D. Lowe Give about 2000 stable “keypoints” for a typical 500 x 500 image Each keypoint is described by a vector of 4 x 4 x 8 = 128 elements

(over 4x4 array of 8-bin gradient histograms in keypoint neighborhood)


Extraction [39]

Recall: Robust Edge DetectorRecall: Robust Edge Detector Apply LPF to suppress noise, then apply edge detector or

derivative operations

E.g. Laplacian of Gaussian (LoG): in shape of Mexican hat

Figures from Gonzalez-Woods 2/e online slides.


Extraction [41]

SIFT: Employ “Scale Space Extrema”SIFT: Employ “Scale Space Extrema” Examine Differences of Gaussian filtered

images at nearby scale and k Avoid low contrast & poorly defined DoG peaks

Figures from Lowe’s IJCV 2004


Extraction [42]

Examples of Difference of GaussianExamples of Difference of Gaussian

Examples from SIFT tutorial notes by Estrada/Jepson/Fleet (2004)


Extraction [43]

Examples from SIFT tutorial notes by Estrada/Jepson/Fleet (2004)


Extraction [44]

(weighted) Gradient orientation histogram near a stable DoG extrema– Peaks in histogram correspond to dominant orientation– Also include other directions close to the peak as keypoints for higher stability– Measure properties of a keypoint relative to its assigned orientation to gain

rotational invariance

Keypoint Descriptor records gradient pattern of each keypoint

– As a set of 8-bin orientation histograms over 4x4 nearby blocks

SIFT: Rotational Invariant Keypoint Descriptor SIFT: Rotational Invariant Keypoint Descriptor

Figure from Lowe’s IJCV 2004


Extraction [45]

Summary of Today’s LectureSummary of Today’s Lecture

Readings

– Image Forensics: IEEE Signal Proc. Magazine March 2009

– Line and corner detection: Gonzalez’s 3/e book 10.2.7

– Stable Gradient Features ~ SIFTDavid G. Lowe (2004). "Distinctive Image Features from Scale-Invariant Keypoints". International Journal of Computer Vision 60 (2): 91–110.

For more explorations

– Boundary representation: Gonzalez’s 3/e book 11.1 – 11.2

– Morphological operation: Gonzalez’s book Chapter 9

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

04

)


Extraction [46]


Extraction [47]

Pattern MatchingPattern Matching Minimum distance classifier: compare with mean feature vector of

each class (if known or can be learned)

Matching by correlation– Reduce sensitivity via normalized correlation (correlation coeff.)– Exhaustive search for size & orientations ~ computationally expensive

– Learn more on pattern classification Overview: Chapter 12 of Gonzalez’s book ENEE731 Statistical Pattern Recognition; CS Machine Learning course

Figure from Gonzalez’s 2/e book resource


Extraction [48]

Boundary Boundary RepresentationRepresentation

Chain code

4-way or 8-way connectivity

Features/measurement obtained from object’s chain code

– Perimeter; Area; Centroid

Figure from Gonzalez’s book resource


Extraction [49]

Boundary DescriptorsBoundary Descriptors

Simple descriptors

– boundary length, diameter– curvature (rate of change of slope/tangent); B-splines

Shape number

– The 1st difference of smallest magnitude of chain code to normalize chain code’s start point and orientation

Fourier descriptor

– Record point coordinates w/ a sequence of complex #– DFT coeff. of the complex sequence as Fourier descriptors

Statistical moments

– mean, variance, and higher-order moments of boundary segments and/or 1-D representation of boundary e.g. r()

ENEE631 Digital Image Processing (Spring'09) Image Forensics & Feature Extraction Spring ’09 Instructor: Min Wu Electrical and Computer Engineering Department,

Documents

enee631 spring09 lecture

enee631 s09

enforced feature

original feature value

additive embedding

limited robustness

higher robustness

wu electrical