Data fusion and multitarget tracking: some interests for military and automobile … · 2013. 8. 22. · Automobile applications 1. Multi-lane detection and tracking 2. Ego-localisation

Data fusion and multitarget tracking:

some interests for military and

automotive applications.

Evangeline POLLARD

Just a few words about me…

One year internship

DLR Munich, Germany

Master and PhD degree

Paris/Grenoble,France

Post-doc

Versailles and Sherbrooke (Canada)

Visitor researcher

Berkeley, CA 2

Outlook

1. Military application: convoy detection and tracking

1. Multi-target tracking, a brief overview

2. Hybridization of CPHD filter and MHT

3. Bayesian network for convoy detection

2. Multi-target detection and tracking with uncalibrated

aerial videos

1. Detection

2. Tracking

3. Automotive applications

1. Multi-lane detection and tracking

3

Outlook

1. Military application: convoy detection and tracking

1. Multi-target tracking, a brief overview

2. Hybridization of CPHD filter and MHT

3. Bayesian network for convoy detection

2. Multi-target detection and tracking with uncalibrated

aerial videos

1. Detection

2. Tracking

3. Automotive applications

1. Multi-lane detection and tracking

4

Battlefield surveillance

zone of interest

GMTI data

Videos SAR images

Geographical information System (GIS)

5

General problem

Goal

6

Situation assessment

How many targets on the scene ?

What is their behavior ?

Are they objects of interest ?

Methods:

• 1st step: Using GMTI sensor to detect agregates

• Algorithm weaknesses for closely spaced target tracking

• Use a promising algorithm: the PHD filter (Probability Hypothesis Density)

• 2nd step: Integrate other data types to determine if the detected

aggregates are convoys or not.

Convoy detection

and if so, how many targets are in

Convoy detection

Methods

GMTI data

• GMTI (Ground Moving Target Indicator) data:

• High traffic density

• High maneuverability of ground targets

• Environment complexity (roads, mountains, ...)

• Sensor limitations: measurement noise, spatial and temporal bias,…

• False alarms , PD<1 and spawned targets

7

Single object tracking

solved by Kalman filter equation

with linear/Gaussian assumption

8

Optimal Bayesian Filter: Kalman filter

1

1

11111

1

1)()()(

k

k

kkkkkkk

k

kkkdxZxfxxfZxf

Propagation of the probability density function (pdf) of xk

[Mahler01] : Detecting, tracking and classifying group targets: a unified approach, Proc.

of SPIE Vol. 4380

)(

)()(

)(1

1

1

k

kkk

k

kkkkkkkk

kkk

Zzf

Zxfxzf

Zxf

Prediction

Update

Estimator )(ˆ supargk

kk

x

kkZxfx

a priori pdf

a posteriori pdf

motion model previous pdf

normalization

likelihood a priori pdf

9

Multi-target tracking

State space

• Varying number of targets:

• Birth targets

• Stationary targets

• Output of the observation

zone

• False alarm

• Non-detection

Observation space

solved by MHT,

JPDAF, particle

filter…

and CPHD filter

10

Random Finite Set (RFS)

Target set Xk modeled as a RFS

k

X

kk

X

kkk

kk

BSX

11

)()(11

• Measurement set Zk modeled as a RFS

k

Xx

kk

k

xZ

)(θ

• : Survival targets between iteration k and iteration k-1

• : Spawned targets

• : Birth targets

1kk

S

1kk

B

k

• : Target originated measurement

• : false alarms k

θ

k

11

Multi-sensor/Multi-target Bayes filter

1

1

11111

1

1)()()(

k

k

kkkkkkk

k

kkkdXZXfXXfZXf

Propagation of the joint probability density function (jpdf) of RFS Xk

[Mahler03] : Multitarget Bayes Filtering via First-Order Multitarget Moments, IEEE AES, Vol. 39, No 4

)(

)()(

)(1

1

1

k

kkk

k

kkkkkkkk

kkk

ZZf

ZXfXZf

ZXf

Prediction

Update

Estimator )(ˆ supargk

kk

X

kkZXfX

a priori jpdf

a posteriori jpdf

motion model previous jpdf

normalization

measurement likelihood a priori jpdf

12

13

PHD definition

S

kdxxvSX )(E

• : first-order statistical moment of the multitarget posterior, also

called intensity function or Probability Hypothesis Density k

v

PHD filter principle

(x))()().((x)111 kkkkskk

dvxfPv

kZz

kkdk

kkd

kkdk

dvzPz

vxzP

vPxv

)(.)g()(

(x)).g(.

(x)1)(

1

1

1

• : survival probability between iteration k and iteration k-1

• : transition function knowing the previous state

• : birth intensity

.1kk

f

k

• : detection probability

• : measurement likelihood

• : clutter intensity

sP

dP

)g( xz

k

Prediction

Estimation

14

Several implementation

[Vo06] : Analytical implementation of the Gaussian Mixture Probability Hypothesis Density

Filter, IEEE SP, 2006

15

Le Cardinalized PHD: principle the number of targets is considered as a random variable p

the corresponding pdf is conjointly propagated over time

[Mahler07] : PHD filters of higher order in target number, IEEE AES, 2007

Prediction

Update

Estimator )(ˆ suparg npN

kk

n

kk

a posteriori pdf normalization

Measurement likelihood a priori pdf

Sum of hypotheses for the n targets to be

(n-j) birth l survival (l-j)non survival

16

Labeling

labeling

17

kkN̂

11

ˆ kk

N

Labeled GM-CPHD (1/2)

• : Gaussian set of size

GG

kNiikikikk

Pmw,...,1,,,

,,

Principle

G

kN

kG

• : track set of size describing the target trajectory kk

N̂k

T

kk

NijkikikikksPx

ˆ,...,1,1,,,,,,ˆ

TT

Evaluate the track-to-Gaussian association matrix of size k

A

0

1),( nm

kA

If the Gaussian component n is associated to the track m

otherwise

G

kkkNN ˆ

weight state covariance

state covariance score

Principle

Goal

18

19

• Maximization of the weight matrix 3 tracks

4 Gaussian components

kW

• Minimization of the cost matrix

Contributions to the labeled GMCPHD

kC

If the Gaussian n can be associated to the predicted track m

Otherwise

0

, n k

k

w W

0 1 . 0 7 . 0 0

6 . 0 0 0 0

0 0 0 9 . 0

k W

0 6 . 0 7 . 0 0

1 . 0 6 . 0 7 . 0 0

0 0 0 9 . 0

k W If the Gaussian n can be associated to the predicted track m

Otherwise

0

) , ( n m c k C

0 18 . 5 67 . 3 0

52 . 0 85 . 2 83 . 4 0

0 0 0 55 . 4

k C

Labeled GMCPHD (2/2)

Means

Comparison between the IMM-MHT and the

GMCPHD filter

IMM-MHT GMCPHD

Target position

estimation ++ +

Target velocity

estimation ++ -

Number of targets

estimation - ++

Computational

complexity + ++

Hybridization

++

++

++

+

Creation of a hybrid algorithm which would combine the

advantages of both of them

• IMM-MHT: Interacting Multiple Model – Multiple Hypothesis Tracker

• GMCPHD: Gaussian Mixture Cardinalized Probability Hypothesis Density

20

Hybridization

[Pollard11] : E. Pollard, B. Pannetier, M. Rombaut, “Hybrid algorithms for Multitarget tracking using the

MHT and the GMCPHD”, IEEE Aerospace and Electronic Systems

21

22

Scenario

Convoy velocity: 10m/s

Isolated target velocity: 15m/s

Root Mean Square Error in position

Convoy of 6 targets Independent

target

quality

23

Root Mean Square Error in velocity

Convoy of 6 targets Independent

target

quality

24

Track length ratio

Convoy of 6 targets Independent

target

quality

25

A convoy detection process

Elaboration of a new algorithm combining the advantages of the

GMCPHD and the IMM-MHT with road constraints

+ No performance drop when targets are close together

Aggregate detection

26

27

Convoy: definition and analysis

• Convoy definition

• Number of targets > 2

• Low and constant velocity

• Military type

• Stay on sight

• On the road

• System analyze

• Asynchronous data

• Heterogeneous data

• Random variables

• Missing data

• Temporal evolution

Dynamic Bayesian Network

X1 X2

X3

3

1

321))((),,(

i

iiXPaXPXXXP

Convoy modeling by using Dynamic

Bayesian Network

k-1 k

[Pollard09a] : E. Pollard, B.

Pannetier, M. Rombaut, “Convoy

detection processing by using the

hybrid algorithm (GMCPHD/VS-

IMMC-MHT) and Dynamic

Bayesian Networks ”, Fusion 2009,

Seattle

28

29

Convoy probablity

Time (in s)

30

Number of target estimation

: set of unique value of

Outlook

1. Military application: convoy detection and tracking

1. Multi-target tracking, a brief overview

2. Hybridization of CPHD filter and MHT

3. Bayesian network for convoy detection

2. Multi-target detection and tracking with uncalibrated

aerial videos

1. Detection

2. Tracking

3. Automobile applications

1. Multi-lane detection and tracking

2. Ego-localisation by fusing GPS and proprioceptive data

31

General problem

Detection

Camera motion

Parallax effects with

urban objects

Low image parameters

Unknown camera

parameters

Tracking Extended targets

Probability of detection <1

Hidden zone in urban areas

Spawned targets

High false alarm rate

[Pollard09b]: E. Pollard, A. Plyer, B.

Pannetier, F. Champagnat, G.

Lebesnerais, “GM-PHD Filters for

Multi-Object Tracking in

Uncalibrated Aerial Videos”, Fusion

2009, Seattle

32

Image motion decomposition

Image motion = ground plane motion

+ parallaxe motion

+ moving object motion

residuals

filtered using

size requirements

Optical flow

algorithm

parametric registration

(homography)

33

Ground plane motion (1/3)

Ransac

Region segmentation

Region selection

Dense estimation

mask

Grond plane motion

regions

inliers

Images

Point tracking

Initial transformation

34

Ground plane motion (2/3)

Ransac

Region segmentation

Region selection

Dense estimation

mask

parametric registration

regions

inliers

Images

Point tracking

Initial transformation

[FOLKI]: G. Le Besnerais, F. Champagnat, “Dense

optical flow estimation by iterative local window

registration”, in ICIP’05, IEEE, vol. 1, p. I-137-140

35

Ground plane motion (3/3)

Ransac

Region segmentation

Region selection

Dense estimation

mask

regions

inliers

Images

Point tracking

Initial transformation

parametric registration

36

Ground plane motion (2/3)

Ransac

Region segmentation

Region selection

Dense estimation

mask

regions

inliers

Images

Point tracking

Initial transformation

parametric registration

37

Postprocessing

Image segmentation based

on the norm of the residual motion

After object selection

Selection of moving objects • Edge detection: select region with high density of edges

• Morphological processing: regularize region shape

• Final selection on area (use prior information on object’s size)

38

Detection results

39

GM-CPHD tracking

40

Outlook

1. Military application: convoy detection and tracking

1. Multi-target tracking, a brief overview

2. Hybridization of CPHD filter and MHT

3. Bayesian network for convoy detection

2. Multi-target detection and tracking with uncalibrated

aerial videos

1. Detection

2. Tracking

3. Automotive applications

1. Multi-lane detection and tracking

41

Intelligent vehicle

processing

unit

laser

actuators HMI

camera radar GPS

gyrometer accelerometer

exteroceptif

proprioceptif odometer

42

X

Y

local map

model

• ego-vehicle

• obstacles

• environment - marking lanes

camera

laser

com

scene

Situation assessment

43

Goals

Multi lane

detection

Multi-camera system

X

Y

2

210xaxaay

Number of marking lines

Shape of marking lines

44

Issues

Missing marking line

Identification ambiguity

Curves

Texture changes

Line width change

Shadow

Light condition change

False alarm

45

General scheme 1. Marking feature extraction

Correction

Prediction Association

Propagation

Kalman Filter

Transferable Belief Model

Estimated

marking

Estimated

marking

Classification

Estimated

marking

Road shape

2. Multi-lane detection 3. Estimation

46

Extraction of road markings primitives

Computed only in a region of interest to limit false points

Horizon

Car cover

(height, variance)

Maximum height

[Pollard11a] : Lane Marking Extraction with Combination Strategy and Comparative

Evaluation on Synthetic and Camera Images, ITSC, 2011 47

Local threshold extractor – LT: Local Threshold

– SLT: Symmetrical Local Threshold

Tg

width

LT

SLT

Drawback

• Depends on the width of

the neighborood

• Only horizontal markings

• Sensitive to marks on the

road

48

Background detection extractor – MLT: Median Local Threshold (50th percentile)

– PLT: Percentil Local threshold (43th percentile)

Median filter

Median value 43th percentile value

49

Multi-lane detection

50

Projection according to the road shape

51/52

Multi-lane tracking

Marking lane 2

210xaxaay

state vector

state eq.

Observations

observation eq

solved by Kalman filter

perspective: use of an IMM for dealing with high curve situation [PollardXX]Road Lane Marker Detection and Estimation: a new algorithm and its complete

Evaluation Process, submitted to IEEE ITS

Performance evaluation

53/52

Conclusion

Multitarget Tracking, Situation Assessment,

Object of Interest Detection

Data Fusion, Uncertainty Management,

Evidence Theory, Bayesian Networks

Particle filtering

GMTI, SAR, GPS, GIS, video images

Thank you for your attention!

54

The Joint Directors of Laboratories

[Steinberg98] : Revisions to the JDL data fusion model, Proceedings of SPIE

Multi-target tracking Data fusion

55