iowa - home.eng.iastate.eduhome.eng.iastate.edu/~namrata/gatech_web/iowa.pdf · Title: iowa.dvi Created Date: 5/2/2005 6:25:50 AM

Particle Filters for Change Detection and Shape Tracking

Namrata Vaswani

School of Electrical and Computer Engineering

Georgia Institute of Technology

http://users.ece.gatech.edu/∼ namrata

Change Detection and Shape Tracking 1

Outline

• Optimal Filtering, Particle Filtering

• Slow and Sudden Change Detection in Nonlinear Systems

– Application: Abnormal “Shape Activity” Detection

• Particle Filtering for Continuous Closed Curves (Contours)

– Tracking moving and deforming objects


Optimal Filtering, Particle Filtering


State Space Model

Yt−1 Yt

qt

gt−1gt

Xt−1 Xt

• State transition model:

Xt = ft(Xt−1) + nt, qt(Xt|Xt−1) = pn(Xt − ft(Xt−1))

• Observation model:

Yt = ht(Xt) + wt, gt(Yt|Xt) = pw(Yt − ht(Xt))

• Hidden Markov Model (HMM) assumption satisfied


Filtering and Tracking

• Filtering : Estimating expected value of stateXt (and of any function of

the state), given all observations untilt, Y1:t.

• Tracking: Evaluating above using observations untilt − 1

• Complete Solution: evaluate prediction & filtering (posterior)distribution

πt|t−1(dx) = Pr(Xt ∈ dx|Y1:t−1) : Prediction

πt△= πt|t(dx) = Pr(Xt ∈ dx|Y1:t) : Posterior


Exact Solution

• t = 0: Posterior ofX0 given no observations is its prior,π0|0 = p0

• Bayes’ rule applied to system and observation model att:

Prediction dist. πt|t−1(dxt) =

∫

xt−1

qt(xt|xt−1)πt−1(dxt−1)dxt

Filtering dist. πt(dxt) =gt(Yt|xt)πt|t−1(dxt)∫

xgt(Yt|x)πt|t−1(dx)

• System & observation model linear, Gaussian: Kalman filter

• Any general system: approx. solution using a Particle Filter


Particle Filter [Gordon et al’93]: Basic Idea

• Sequential Monte Carlo method, approx. true filter as numberofMonte Carlo samples (“particles”), N → ∞

• GivenπNt−1, perform importance sampling/ weighting, followed by

resampling to approx. the Bayes’ recursion:πNt

πNt|t−1

πt πNt

Resample

wit ∝ gt(Yt|x

it)

Weight

xit ∼ qt

πNt−1

Importance Sample

Yt

• Usingγt(xt|x(i)1:t−1, Y1:t) = qt(xt|x

(i)t−1) as importance density


Slow and Sudden Change Detection


Application: Abnormal Activity Detection

• Activity performed by a group of moving and interacting objects, treatedas point objects (“landmarks”)

• Objects (landmarks): People, Vehicles, Robots, Human bodyparts

• Dynamics of group of landmarks: moving and deforming shape

• “Normal Activity”: Modeled as a landmark shape dynamical model

• “Abnormal Activity”: change in learned shape dynamical model,could be slow or sudden and whose parameters were unknown


Example: Group of People Deplaning

A ‘normal activity’ frame Abnormality

Figure 1:Airport example: Passengers deplaning


Dynamical Model for Landmark Shapes

• Observation: Vector of observed object locations (Configuration)

• State: [Shape, Translation, Scale, Rotation, Velocities]

• Observation model:ht : S ×R2 ×R

+ ×S0(2) → R2k, Gaussian noise

• System model:

– Gauss-Markov model on shape velocity, parallel transported totangent space of the current shape

– Gauss-Markov model on group (scale, rotation, translation)velocity

• Detect changes in shape using posterior distribution of shape givenobserved object locations


The Change Detection Problem

• Partially Observed and Nonlinear System satisfying HMM property

• Given the observationsY1, Y2, ...Yt, detect, as quickly as possible, ifa change occurred in the dynamics of the stateXt

– Parameters of changed system unknown

– Change can be slow or sudden


Notation

Yt−1 Yt

qt

gt−1gt

Xt−1 Xt

• Prior: Given no observations,Xt ∼ pt(.)

• Posterior:Xt|Y1:t ∼ πt(.)

• Superscripts: 0 (unchanged system),c (changed system)

• X0t ∼ p0

t (.), Xct ∼ pc

t(.)


Slow and Sudden Changes

• Slow change: small change magnitude per unit time, “tracked” bythe filter, i.e. ||πc,0,N

t − πct || is small

• Sudden change: “filtered out” (“loses track”)

– Duration much smaller than “response time” of filter.

– Response time (time taken to track) depends on

∗ System noise variance

∗ Observation noise variance

∗ Number of particles,N


Existing Work: Change Detection in Nonlinear Systems

• Fully observed state(no observation noise,ht invertible)

– CUmulative SUM, generalized CUSUM, negative log likelihood

• Partially observed state

– Known change parameters

∗ Linearization techniques followed by CUSUM∗ CUSUM (usest + 1 PFs att), modified CUSUM[Sadjadi et al’02]

– Unknown change parameters: few existing solutions

∗ generalized CUSUM not tractable[Andrieu et al’2004]

∗ Tracking Error [Bar-Shalom]

∗ negative Log Likelihood of Observations (OL)∗ Fail to detect slow changes


Slow change detection, Unknown parameters

• Fully observed state:

– negative Log Likelihood of state of unchanged system,

− log p0t (Xt) = − log p0

t (h−1t (Yt))

• Partially observed state (significant observation noise):

– Why not use Min. Mean Square Error estimate of this ?

• Our statistic is exactly this MMSE estimate:

ELL(Y1:t) , E[− log p0t (X)|Y1:t]


Computing the Statistics[Vaswani, ACC’2004]

• Expected (negative) Log Likelihood of state (ELL)

ELL(Y1:t) = E[− log p0

t (Xt)|Y1:t] = Eπt[− log p0

t (X)]

• For sudden changes, can use

– (negative) log of Observation Likelihood (OL)

OL(Y1:t) = − log pY(Yt|Y1:t−1) = − log Eπt|t−1[gt(Yt|X)]

– Tracking Error (TE) [Bar-Shalom]

TE = ||Yt − Yt||2, Yt = E[Yt|Y1:t−1] = Eπt|t−1

[ht(X)]

– OL ≈ TE (to first order) for white Gaussian observation noise


Detection Thresholds

• ELL Threshold: ThELL = EY0

1:t[ELL0] + k

√

Var(ELL0)

EY0

1:t[ELL0] = EY0

1:t[E[− log p0

t (Xt)|Y0

1:t] = h(p0

t ) = h(X0

t )

h(.): Differential entropy

• OL Threshold: ThOL = EY0

1:t[OL0] + k

√

Var(OL0)

EY0

1:t[OL0] = h(Y0

t |Y0

1:t−1)

• Choosek based on allowed false alarm probability

• Declare a change if either ELL or OL exceeds its threshold


Change Detection Algorithm

Particle Filter

(Observation)

πNt−1

πNt

YesYes

πNt|t−1

xit ∼ qt

wit ∝ gt(Yt|x

it)

πtN

Change (Slow)Change (Sudden)

ELL > ThELL?OL > ThOL?

Yt


ELL v/s OL (or TE)

• Slow Change:

– PF: stable under mild assumptions, tracks slow change well

– OL & TE rely on error introduced by change to detect

– Error due to change small: OL, TE fail or take longer to detect

– Estimate of posterior close to true posterior of changed system

– ELL detects as soon as change magnitude becomes detectable

• Sudden Change:

– PF loses track: OL & TE detect immediately

– ELL detects based on “tracked part of the change”

– ELL fails or takes longer


ELL Approximation Errors

• Exact filtering error : Wrong state transition kernel for changed

observations

• Particle Filtering Error : Finite number of particles:N

• Bounding error : Log Likelihood is an unbounded function, stability

and PF convergence results exist for bounded functions


Complementariness of ELL and OL [Vaswani, ACC’2004]

Theorem. ELL approx. error,errc,0,Nt , is upper bounded by an increasing

function ofOLc,0,Nτ , tc ≤ τ ≤ t, i.e.

errc,0,Nt ≤

t∑

τ=tc

eOLc,0,Nτ ω1(σ

2obs)ω2(ǫ

c,0τ ) + const

Implication for a“detectable” change (true value of ELL large):

• OL fails to detect a change=⇒ ELL detects

• ELL fails to detect=⇒ OL detects


Stability of ELL Error [Vaswani, ACC’2004]

Theorem. Average ELL approximation error iseventually monotonicallydecreasing (and hence stable), for large enoughN if

- Change lasts for a finite time

- ft(Xt) continuous for all t

- π0 has compact support

- gt(Yt|x) (as a function of x) has compact support, for allYt

- The convergence of the bounded approx. of ELL is uniform in time

• Based on optimal filter stability results of [LeGland & Oudjane’02]

• Valid for anyunbounded function of state(not just ELL)

• Errorasymptotically stableunder stronger assumptions


Contributions

• ELL detects a change before loss of track (very useful). OL orTracking Error detect after partial loss of track.

• Complementary behavior of ELL & OL for slow & sudden changes

• Stability of the total ELL approximation error for large N

• Relation to Kerridge Inaccuracy and a sufficient condition for theclass of detectable changes using ELL[Vaswani, ACC’04]

• ELL error upper bounded by increasing function of “rate ofchange”, increasing derivatives of all orders[Vaswani, ICASSP’04]


Simulated Example: ELL and OL Plots for Increasing Rates of Change

Xt = Xt−1 + nt + bt, bt = rσsys for t=5 to t=15

Yt = X3

t + wt

No Change:r=0 (blue),

Slow: r=0.5 (red), r=2 (magenta),Medium: r=2 (green),Sudden: r=5 (cyan)

0 5 10 15 20 25 30 35 40 45 500

2

4

6

8

10

12

14

Time

ELL

ELL plot for increasing rates of change

p=0 p=0.5σ

noisep=1σ

noise

p=2σnoise

p=5σ

noise

0 5 10 15 20 25 30 35 40 45 500

10

20

30

40

50

60OL plot for varying change rates

Time

OL

← GOES TO ∞

r=0 r=0.5r=1 r=2 r=5

ELL OL


Videos

• Group of People Deplaning: Normal activity sequence

• Abnormality (One person walking away in an abnormal direction)

• Human Action Tracking & Abnormality Detection


Group of People: Abnormality Detection

Abnormality (one person walking away) begins att = 5

0 5 10 15 20 25 30 35 40 45 500

5

10

15

20

25

30

t

ELL

NormalAbnormal, vel=1Abnormal, vel=4Abnormal, vel=32

0 5 10 15 20 25 30 35 40 45 500

50

100

150

200

250

300

350

t

Obs.

like

lihood

NormalAbnormal, vel=1Abnormal, vel=4Abnormal, vel=32

ELL OL


Group of People: “Temporal Abnormality” Detection

Abnormality (one person stopped in path) begins att = 5

ELL Plot


ROC Curves: “Slow” Abnormality Detection

0 10 20 30 40 50 601

2

3

4

5

6

7

Mean time between false alarms

Det

ectio

n de

lay

ELL, vel = 1

σ2obs

=3σ2

obs=9

σ2obs

=27σ2

obs=81

0 10 20 30 40 50 6018

20

22

24

26

28

30

Mean time between false alarms

Det

ectio

n de

lay

Tracking error, vel = 1

σ2obs

=3σ2

obs=9

σ2obs

=27σ2

obs=81

ELL Detects TE: Takes much longer


Human Actions: Abnormality Detection

• Abnormality begins at t = 20

• NSSA detects using ELL without loss of track

ELL Tracking Error


Future Research

• Changed Parameter Estimation

• Practical implications of the “rate of change” bound result and thestability result for particle filter design

• Applications and Performance Analysis

– Abnormal activity detection and activity segmentation

– Neural signal processing (changes in STRFs of auditory neurons)

– Acoustic tracking (changes in target motion model)

– Communications applications: tracking slowly varying channels,

congestion detection in networks


Particle Filtering for Continuous Closed Curves

(PF for Infinite Dimensional State Spaces)


(System Model)(Observation Model)

(object contour, velocities)State

Filter

Observation(Image)

(camera noise)(system noise)

ft(.)

t = t + 1

ht(.)+ +Xt Yt

t = t + 1

System (Object motion + deformation)

Sensor (Camera)Observation

πt(Xt|Y1:t)

ntwt


Continuous Closed Curve (“Contour”)

• A smooth locus of points traced out by a mapping of the unit interval[0, 1] into R

2, with C(0) = C(1)

• Representations (Infinite dimensional)

– Parametric: C(p) = [Cx(p), Cy(p)], Cx(p) & Cy(p) are smoothfunctions ofp ∈ [0, 1], C(0) = C(1)

∗ All re-parameterizations also represent the same curve

– Implicit: Zero level set of a higher dimensional function,φ(x, y),i.e. it is the collection of all points{x, y ∈ R

2 : φ(x, y) = 0}

• Finite dim: B-splines, Fourier descriptors, Marker particle, Landmarks


Motion and Deformation [Yezzi,Soatto’02]

• “Motion” : global motion, a finite dimensional group e.g. Affine

• “Deformation” : local shape deformations, infinite dimensional

”deform”(local)(global)

“move”

• Examples:

– Fishcan move in space and also deform its shape

– Human heart only deforms,Human hand moves and deforms


The Problem

• Track a moving and deforming object from an image sequence

• State: Object Contour, Affine velocity, Deformation velocity

• Contour: Infinite dimensional level set representation

• Observation: Image (noisy nonlinear function of contour)

• Goal: Estimate the contour & velocities, given all past images(filter out the state from noisy observations)


Existing Work: Tracking Continuous Curves

• Finite dim. parametric repr. (B-splines) + Particle filter:Condensatione.g. [Isard, Blake’98]

Tracks only affine deformations. Cannot handle large changes in curve length

• Fixed finite dim marker particle repr. + Kalman filter:e.g. [Terzopoulos, Szelisky’92], [Peterfreund’99]

Prediction step correlated with current observation. Needexplicit observations

of contour. Cannot handle large changes in curve length

• Level set representation (infinite dim) + Linear observers:[Jackson et al’04], [Niethammer et al’04]

Observers for contour, velocity uncoupled. Need explicit observations


Issues we address...

1. Implicit observations: Yt = Image(t)

2. Coupled observersfor contour and velocity: use particle filtering

3. Prediction independentof current observation (image)

4. Infinite dim repr. of contour & velocity

5. Particle filtering expensive in this case

• Generating samples from a very large dim noise distrib.

• No. of particles for accurate filtering increases with noisedim


Observation Model

object

background

N(u2, σ2

obs)

N(u1, σ2

obs)

• Chan and Vese model for image formation. Image:Yt, Contour:Ct

gt(Yt|Ct) = e−

Ecv(Ct,Yt)

σ2obs

Ecv =

∫

Cint

(Yt(x, y) − u1)2dxdy +

∫

Coutt

(Yt(x, y) − u2)2dxdy


Solution 1: Curve Evolution + PF

[Rathi,Vaswani,Tannenbaum,Yezzi] (Accepted for CVPR 2005, Oral)

• Constant velocity Gauss-Markov model on affine deformationand azero velocity model on local deformation

• Contour: Ct, Affine velocity: ρt, Deformation velocity: vt

• System Model:

ρt = ρt−1 + nt, nt ∼ N (0, Σρ)

Ct = Ct−1 + ∆tgaffine(Ct−1, ρt) + vtN, vt ∼ N (0, ΣC)

• Observation model: Chan and Vese model of image formation

g(Yt|Ct) ∝ e−

Ecv(Ct,Yt)

σ2obs


PF Algorithm: Importance Sampling Step

• Sampleρ(i)t from its transition kernelN (ρ

(i)t−1, Σρ)

• TakeC(i)t = C

mode,(i)t where

Cmode,(i)t = arg min

Ct

[||Ct − Caff,(i)t−1 ||ΣC

+ Ecv(Ct, Yt)]

Caff,(i)t−1 = C

(i)t−1 + ∆tgaffine(C

(i)t−1, ρ

(i)t )

Approx Solution: Start with Ct = Caff,(i)t−1 , run few steps of gradient

descent to minimizeEcv

– Can be understood as sampling from a Gaussian approx. to theoptimal importance density,p(xt|x

(i)t−1, Yt) using [Doucet’98]

– No randomness in samplingC(i)t : still works well in practice


Solution 2: Time Varying Finite Dim. Deformation

[Vaswani,Yezzi, Rathi,Tannenbaum] (Submitted CDC’05)

• Approx. curve deformation using a time varying finite dim. basis

• Assume: For sometime, “most curve deformation” occurs in finiteno. of dimensions,K: “effective basis”, for e.g.

K = 2 dim. B-spline basis will suffice

• Assume:Changes in effective basis detected & estimated accurately


Contour Deformation Model

• Local contour deformation: K basis functions,bj(p) & K-dim speed,v along basis directions

Deformation velocity, β(p) ≈K

∑

j=1

bj(p)vj N(p)

• Global Motion (Affine): gaffine(C, ρ), 6-dim affine velocity,ρ

• Contour at t, Ct deforms as

Ct+1 = Ct + ∆t [gaffine(Ct, ρt+1) +

K∑

j=1

bjvj,t+1 N]


• Gauss Markov model onρt

and vt

vt+1 = vt + νv,t+1, νv,t ∼ N (0, Σv,t∆t)

ρt+1

= ρt+ νρ,t+1, νρ,t ∼ N (0, Σρ,t∆t)

• Used B-splines as the “‘effective basis” for deformation velocity (knots

move at eacht according to contour deformation model)

• “Effective basis” is piecewise constant

– Detect change in basis at every time instant

– If changed, re-estimate newK and new basis functionsbj


One problem...

• Part of contour not changing: like an “unknown static parameter”

– Resampling can result in loss of a good particle (if badobservation), new particles never generated: divergence

• Solution: Monte Carlo PF for “static parameter” [Papavasiliou’2004]

– No resampling for “static parameter” particles

– For each particle of “parameter”,run PF for rest of state space

– Proven to beasymptotically stableunder certain assumptions

– Our problem: treat unchanging contour as “static parameter”


PF Algorithm

1. At t = 0, generateM “static parameter” (contour) particles

2. At eacht, for m = 1 to M do,

(a) Run anN -particle PF

(b) Move B-spline knots, re-evaluate basis functions using new knots

(c) Weight themth contour particle usingq(M) past observations

3. Effective Basis Change Detect:If change go to step 3, else go to step 1

4. Effective Basis Re-estimation

(a) Estimate new basis dimension, learn basis vectors

(b) Re-sample particles of the static parameter (contour)


Summarizing...

Curve Evolution + PF

• Generates only a 6 dim system noise distribution

• “Curve evolution” to get mode for non-affine deformation (expensive)

• Bad observations: “loss of track” (1 particle left)

• Gets back in track slowly, can get stuck in local minima

Time Varying Finite Dim. Deformation

• Generates a K+6 dim system noise distribution

• Works w/o mode evaluation, but moving B-spline knots expensive

• Bad observations: “loss of track” but gets back faster

MCPF-PF: Back in track immediately (static particles not resampled)


Future Research

• Proof of asymptotic stability of the algorithm

• “Effective basis” change detection and re-estimation

– Other possible “effective basis” representations, geometric basis?

– Deviation from uniform B-spline knot separation

– Local tracking error, More B-spline knots where large deformations

– ELL w.r.t. the pdf of deformation velocity before change

– Posterior expected square distance to a reference contour

• Extension to surface tracking


Contributions

• First implementable solution to particle filtering for infin itedimensional state space. Many possible applications:

– Volume image segmentation as a 2D tracking problem

– Tracking human heart, detecting abnormality

– Tracking spatio-temporal receptive fields of auditory neurons

– Tracking principal subspaces (array signal processing)

• Finite dim. parametrization of deformation of a continuous curve

• Posterior mode detection using curve evolution (imp. sampling)

– Faster algorithm: Only MCPF + posterior mode detection


Summary of Recent Research

• Detection & Estimation Problems, Applications in image andvideo

– Pattern classification algorithms[IEEE Trans. IP, Accepted]

∗ Face recognition, Image/Video retrieval, Feature matching

– “Shape Activity” [IEEE Trans. IP, Accepted] [CVPR’03]

∗ Abnormal activity detection : stationary shape activity∗ Human action tracking & abnormality detection: nonstationary SA∗ Activity sequence segmentation: piecewise stationary SA

– Change detection in nonlinear systems[ACC’04, ICASSP’04,’05]

– Particle filtering for continuous closed curves[CVPR’05]


• Future research interests

– Biomedical Image & Signal Processing

– Shape analysis in Computer Vision

– Optimal Filtering theory, algorithms & applications

– Information theory for estimation/detection, video compression


Research Plan

• Biomedical Image and Signal Processing

– Use of dynamical models and tracking

∗ Dynamical models for disease progression?∗ Track the human heart, detecting abnormality∗ Volume image segmentation as 2D tracking?∗ Neural signal processing

– Shape Matching/Classification

• Particle filtering for infinite dimensional state spaces

– Asymptotic stability

– Effective basis representation, change detection, estimation

– Applications


• Particle filtering under system model error

– Changed system model parameter estimation

– Implications of my results for improved Particle Filter design

– Applications in neural and acoustic signal processing

• “Landmark Shape Activities”

– Dealing with time varying number of landmarks

– Activity Segmentation

– Using different sensors, sensor fusion


Acknowledgements

• Landmark Shape for Modeling Activity: Joint work with

Dr. Amit RoyChowdhury and Dr. Rama Chellappa

• Filtering Continuous Closed Curves: Joint work with

Dr. Anthony Yezzi, Yogesh Rathi, Dr. Allen Tannenbaum


iowa - home.eng.iastate.eduhome.eng.iastate.edu/~namrata/gatech_web/iowa.pdf · Title: iowa.dvi Created Date: 5/2/2005 6:25:50 AM

Documents