Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation

1

Multiple target detection in video using quadratic multi-frame

correlation filtering

Ryan KerekesOak Ridge National Laboratory

B. V. K. Vijaya KumarCarnegie Mellon University

March 17, 2008

2

Outline

PART I• Correlation filtering overview• Database

PART II• Multi-frame correlation theory• Multi-frame simulation results

3

CF

Why use correlation filters?

• Shift invariance: useful when we don’t know:– Target locations in the scene– Number of targets in the scene

• Distortion tolerance: CF’s can be trained to tolerate some distortion and reject false targets

• Graceful degradation: noise, occlusion

False patterns

Scaled versionsTraining pattern

4

Correlation peak metric

• Often we will measure peak sharpness– Peak-to-sidelobe ratio (PSR)

5

Correlation peak metric

• Often we will measure peak sharpness– Peak-to-sidelobe ratio (PSR) metric– invariance to overall image brightness

PSR = 7.46

6

FFT

Correlation filter usage

• Typically computed in the frequency domain– 2 FFTs– complexity reduction

• PSR computed at every point in frequency domain– 4 more FFTs

IFFT

input

FT of filter

correlation array

FT of PSR window

4FFTs

PSR array

*

7

Linear filter vs. quadratic filter

• Consider thresholding the output– results in a discriminant at each shift

pixel 2

pixe

l 3

pixel 2

pixe

l 3

threshold

class 1class 2

example: elliptical boundary

threshold

linear filter quadratic filter

8

Turntable database

• Captured rotational imagery of three targets• Depression angles used: 17 , 19 , 21 , 23• Green background used to aid in segmentation

17°

211cmtarget

camera

turntable

green background

9

Filter training

• Each filter trained on scaled and out-of-plane-rotated images of true- and false-class targets

0° 3° 30°

…

…

…

…

Azimuth

… …Scale (PHTW)

True

False (see

n)

Missile

Leopard

False (unsee

n)Abrams

16

20

10

Filter comparisons

• Showed that RQQCF filters outperform others– each filter bank trained to divide up 360 azimuth range– different number of filters/bank, equal computation

filter design

LEGEND

EOTSDF Eigen-filter

UOTSDF Unconstrained OTSDF

EMACH Extended MACH

CPCF/UPCF

Polynomial CFs

RQQCF/SSQSDF

Quadratic filters

11

PART IIMulti-frame correlation filtering

MFCF

12

CF in video sequences

• Three types of approaches to the problem

CF

CF

CF

Frame 1

Frame 2

Frame 3

…

THRESHOLD

THRESHOLD

THRESHOLD

Correlation planes

Test images

DetectionsTYPE 1: SINGLE-FRAME

13

CF in video sequences

• Three types of approaches to the problem

CF

CF

CF

Frame 1

Frame 2

Frame 3

…

THRESHOLD

THRESHOLD

THRESHOLD

Correlation planes

Test images

DetectionsTYPE 2: DETECTION-FIRST (CLASSICAL TRACKING)

Relate information

14

CF in video sequences

• Three types of approaches to the problem

CF

CF

CF

Frame 1

Frame 2

Frame 3

…

THRESHOLD

THRESHOLD

THRESHOLD

Correlation planes

Test images

DetectionsTYPE 2: DETECTION-FIRST (CLASSICAL TRACKING)

Relate information

Information loss due to thresholding

step

Data association

problem due to false alarms

15

CF’s in video sequences

• Three types of approaches to the problem

CF

CF

CF

Frame 1

Frame 2

Frame 3

…

THRESHOLD

THRESHOLD

THRESHOLD

Correlation planes

Test images

DetectionsTYPE 3: RELATION-THEN-DETECTION (our work)

Relate information

16

Efficient output combination

• We want to combine outputs while preserving information– Avoid preliminary thresholding step – Avoid assumptions on number of targets– Avoid a large computational increase

• We have developed a probabilistic approach– We can derive defensible theory– We can confidently utilize all available information

( )Pr | 1, 2,target frame frame …

17

Typical detection schematic

18

Probability mapping

• We map each correlation value to probability of target presence using all available information

PSR array probability array

mapping

high likelihood of target (PSR ≈ 10)

reasonable likelihood of target (PSR ≈ 4)

19

PSR score distributions

• Must assume distributions for target scores and non-target scores for each filter– (we used Gaussians)

Example from a FLIR sequencetarget non-target (clutter)

20

Target motion model

• Must assume transition probability function between adjacent frames– can take velocity into account

• Function of displacement (position-independent)– (we used centered, rotationally symmetric functions)

Gaussian exponential uniform

21

Posterior probability array

• We can derive a pixel-wise mapping function for the posterior probability array

“enhanced output”

“knowledge” from Frames 1

and 2

map

Frame 3 prior probs.

Frame 3 output

Frame 3 posterior probs.

information from Frame 3

22

i-th frame posterior probability array

i-th frame prior probability array

score pdf of filter l on target k

Mapping function

• Various properties of the mapping function

Uses prior arrays from all filters

Uses PSR scores from all filters

Already normalized “filter communication”

sub-class (filter) index

frame index

23

Mapping function

• Example of a mapping in a two-class problemPSR value target probability

24

Prior probability array

• We can derive the prior probability array for the next frame: based on convolution

Frame 3 posterior probs.

“predictor” for Frame 4

motion model priorformula

“knowledge” from Frames 1,

2, and 3 Frame 4 prior probs.

25

i-th frame posterior array

i-th frame prior array

Motion model function

Emergence/disappearance probability functions

Subclass transition probs.

,

Prior array formula

• Various properties of the prior array formula

probability of target emergence

ε(x)

non-zero probability

region

zero probability region

26

Prior array formula

• Various properties of the prior array formula

probability of target disappearance

δ(x)

non-zero probability

region

zero probability region

i-th frame posterior array

i-th frame prior array

Motion model function

Emergence/disappearance probability functions

Subclass transition probs.

,

27

Prior array formula

• Various properties of the prior array formula

only one convolution per filter

* i-th frame posterior array

i-th frame prior array

Motion model function

Emergence/disappearance probability functions

Subclass transition probs.

,

28

Prior array formula

• Various properties of the prior array formulauses posterior arrays from all

filters

subclass transition probability

“filter communication”

i-th frame posterior array

i-th frame prior array

Motion model function

Emergence/disappearance probability functions

Subclass transition probs.

,

29

Multi-frame schematic

mapFrame i

Frame ioutput

(to next iteration)

THRESHOLD

Frame i priors

Frame iposterior

probabilitiesMotion model

Frame i+1 priorspriorformula

detections

30

Multi-frame schematic

mapFrame i

Frame ioutput

(to next iteration)

THRESHOLD

Frame i priors

Frame iposterior

probabilitiesMotion model

Frame i+1 priorsbig

formula

detections

Table lookup

2 FFTs

31

Synthetic sequence demo

Sequence

Prior probability

arrayOriginal output

Posterior probability

array

log-likelihood values

PSR values

32

Synthetic sequence demo

• MFCF can handle multiple targets– Example: sequence with 3 true-class targets, 3 false-class

MFCF detections

single-frame

MFCF

33

Synthetic sequence demo

• False alarms reduced in target and non-target areas@ 90% detection rate

target paths

single-frame MFCF

34

FLIR sequence results• Two FLIR sequences tested • Two types of noise each• MFCF offers greater performance improvement in

white noise than in compression noise

H.264 compression noise (65:1)

AWGN, SNR=20dB

Sequence L2117

MFCF

35

Noise vs. clutter in MFCF

• MFCF favors filters with better clutter rejection– Example: sequence Synthetic3, noisy (SNR = 20dB)– Varied number of eigenvalues (Ne) retained in filter design

single-frame multi-frameworse than

single-frame

36

Summary of main findings

MFCF algorithm:• MFCF is best-suited for handling temporally

uncorrelated noise• Clutter rejection should be handled by the filter(s)• Filter communication can help reject false targets• MFCF degrades gracefully under parameter

changes

37

Future work

• Multi-view correlation filtering (multiple cameras)

scene

View 1 View 2