Aleksandar Ivanovic, Thomas S. Huang Backmen Institute for Advanced Science and Technology, UIUC

A Probabilistic Framework For Segmentation and Tracking of Multiple Non Rigid Objects for Video Surveillance

Aleksandar Ivanovic, Thomas S. Huang

Backmen Institute for Advanced Science and Technology, UIUC

Outline

Introduction Pixel probability model

Background model Foreground probability model

Connected components matching Object detection Experimental results Conclusion

Introduction

In video surveillance, reliable segmentation of moving objects is essential for successful event recognition

Park and Aggarwal A Gaussian mixture model is used at the pixel level

to train and classify individual pixel colors Markov Random Field (MRF) framework is used at

the blob level to merge the pixels into coherent blobs and to register inter-blob relations

A coarse model of the human body (head, upper body, lower body) is applied at the object level as empirical domain knowledge to resolve ambiguity due to occlusion and to recover from intermittent tracking failures

Introduction

Elgammal and Davis Use maximum likelihood estimation to estimate the

best arrangement for people Modeling these regions involves modeling their

appearance (color distributions) as well as their spatial distribution with respect to the body

Assumption: targets are visually isolated before occlusion so that we can initialize their models

Gomila and Meyer Each image of a sequence is segmented and

represented as a region adjacency graph Object tracking becomes a particular graph-

matching problem, in which the nodes representing the same object are to be matched

Background model

Each Lu*v* dimension is modeled with a single Gaussian

Initialize the background pixel model No foreground objects, computing the

statistics over the training sequence Otherwise, use a bootstrapping algorithm

[9] Make use of the Mahalanobis distance

to achieve segmentation, which corresponds to the probability that pixel belongs to the background

[9] Gutchess et al., “A background model initialization algorithm for video surveillance”

),( yxM b

),( yxp

Mahalanobis distance

Mahalanobis distance between each pixel and the corresponding background pixel:

),(*

),(*),(*

),(*

),(*),(*

),(

),(),(),(

var

var

var

yxv

yxvyxv

yxu

yxuyxu

yxL

yxLyxLyxM

mean

mean

meanb

),( yxM b

Foreground probability model (1) For each pixel , we have:

Feature vector Label (0 for background, 1 for foreground)

: absolute distance of the current pixel to the background pixel in the RGB color space

where , , , and are the means over the video frame in RGB space

),(),(

),(),(

),(),(),(

yxByxB

yxGyxG

yxRyxRyxD

mean

mean

mean

),( yxp)],(),,(),,([),( yxPyxDyxMyxA hb

),( yxF),( yxD

),( yxRmean ),( yxGmean ),( yxBmean

Foreground probability model (2) A color similarity measure

Computed from the cumulative histogram of all tracked objects

Computed as the number of pixels in the bin that contains divided by the number of the pixels in the color histogram

Histogram A histogram for one component (R, G, B)

describes the distribution of the number of pixels for that component color in the 16 bins

),( yxPh

),( yxp

0

10

20

30

40

50

60

70

80

90

100

0-15 16-31 32-47 … 224-239 240-255

Foreground probability model (3) Bayesian Network (BN)

Model the relationship of pixel label with feature vector

Model and using Gaussian mixture model (GMM) using Gaussians

The probability the a pixel belongs to the foreground is :

)1()1|()0()0|(

)1()1|(),(

FPFAPFPFAP

FPFAPyxPf

),( yxF

),( yxA

)0|( FAP )1|( FAP

5K

),( yxPf

Blob Level

This paper doesn’t mention! Foreground pixels with the same color

distribution are labeled in the same class Relabeling the disjoint blobs in

connected component analysis Adjacency Color Similarity Small blob

Components matching (1)

Find the objects in the new frame is binarized using an adaptive threshold A union find algorithm is used to find its 8-

connected components that correspond to foreground objects 4-connected components 8-connected components

Foreground object matching cases: One-to-one matching Many-to-one matching One-to-many matching

),( yxPf

Components matching (2)

Use probabilistic matching to solve the problems mentioned above

(foreground object, connected component): , can be described with feature vector :

: horizontal and vertical size of the bounding box

: the size in pixels : color histogram of object/component : centroid of all the pixels of an

object/connected component : index (i.e. or )

)](),(),(),(),(),([)( tytxtHtStytxtk ccss))(),(( tytx ss

)(tS

)(tH t

))(),(( tytx cc

t it jt

)(if

))(),(( jcif

)( jc

Components matching (3)

Derive information for matching a foreground object to a connected component :

: size change as : Euclidean distance between , : similarity between , : horizontal size change as : vertical size change as

If and are matched, label with , otherwise

)(if )( jc

)],(),,(),,(),,(),,([),( jiYCjiXCjiHSjiEDjiSCjim

),( jiSC )(/)( iSjS

),( jiED ))(),(( iyix cc ))(),(( jyjx cc

),( jiHS )(iH )( jH

),( jiXC

),( jiYC

)(/)( ixjx ss

)(/)( iyjy ss

)(if )( jc ))(),(( jcif

1M 0M

Components matching (4)

Probability of matching One-to-one matching:

Occlusion :

Object color similar to background:

Probability of foreground object disappeared is , where is dummy node

)),(|1( jimMP

)),(|1()),(|0(

)),(|1()(

jimMPjimMP

jimMPjiP

)),(|1()),(|0(

)),(|1()( 21 jimMPjimMP

jimMPjiiP

cc

c

)),(|1()),(|0(

)),(|1()( 21

cc

c

jimMPjimMP

jimMPjjiP

)( 0 djP )( 0jc

d

Object Detection

How to decide the connected components should become new object: Define a set of feature

: size of the connected component : distance to the nearest location of a

foreground object : a shape feature, defined as : color similarity of object candidate to the

average foreground object Label if candidate is not new object,

otherwise label

],,,[ CSSHLCST SLCSH 2perimeter

areaSH

CS

0N

1N

Experimental Results

Test a 55 minute long indoor sequence

(a)(f)Foreground object

(b)Probability based only on background model

(c)Probability of foreground

(d)(g)Segmented objects using only background model

(e)(h)Segmented objects using probability of foreground

Conclusion

Contributions of this paper: A new probabilistic framework for pixel

segmentation and for matching of objects to blobs

A framework can account for grouping of objects

A method robust to initialization The matching formulation is better able

to model multi-object tracking and gives more reliable segmentation results