iowa - home.eng.iastate.eduhome.eng.iastate.edu/~namrata/gatech_web/iowa.pdf · Title: iowa.dvi Created Date: 5/2/2005 6:25:50 AM

Particle Filters for Change Detection and Shape Tracking

Namrata Vaswani

School of Electrical and Computer Engineering

Georgia Institute of Technology

http://users.ece.gatech.edu/∼ namrata

Change Detection and Shape Tracking 1

Outline

• Optimal Filtering, Particle Filtering

• Slow and Sudden Change Detection in Nonlinear Systems

– Application: Abnormal “Shape Activity” Detection

• Particle Filtering for Continuous Closed Curves (Contours)

– Tracking moving and deforming objects

Optimal Filtering, Particle Filtering

State Space Model

Yt−1 Yt

gt−1gt

Xt−1 Xt

• State transition model:

Xt = ft(Xt−1) + nt, qt(Xt|Xt−1) = pn(Xt − ft(Xt−1))

• Observation model:

Yt = ht(Xt) + wt, gt(Yt|Xt) = pw(Yt − ht(Xt))

• Hidden Markov Model (HMM) assumption satisfied

Filtering and Tracking

• Filtering : Estimating expected value of stateXt (and of any function of

the state), given all observations untilt, Y1:t.

• Tracking: Evaluating above using observations untilt − 1

• Complete Solution: evaluate prediction & filtering (posterior)distribution

πt|t−1(dx) = Pr(Xt ∈ dx|Y1:t−1) : Prediction

πt△= πt|t(dx) = Pr(Xt ∈ dx|Y1:t) : Posterior

Exact Solution

• t = 0: Posterior ofX0 given no observations is its prior,π0|0 = p0

• Bayes’ rule applied to system and observation model att:

Prediction dist. πt|t−1(dxt) =

xt−1

qt(xt|xt−1)πt−1(dxt−1)dxt

Filtering dist. πt(dxt) =gt(Yt|xt)πt|t−1(dxt)∫

xgt(Yt|x)πt|t−1(dx)

• System & observation model linear, Gaussian: Kalman filter

• Any general system: approx. solution using a Particle Filter

Particle Filter [Gordon et al’93]: Basic Idea

• Sequential Monte Carlo method, approx. true filter as numberofMonte Carlo samples (“particles”), N → ∞

• GivenπNt−1, perform importance sampling/ weighting, followed by

resampling to approx. the Bayes’ recursion:πNt

πNt|t−1

πt πNt

Resample

wit ∝ gt(Yt|x

Weight

xit ∼ qt

πNt−1

Importance Sample

• Usingγt(xt|x(i)1:t−1, Y1:t) = qt(xt|x

(i)t−1) as importance density

Slow and Sudden Change Detection

Application: Abnormal Activity Detection

• Activity performed by a group of moving and interacting objects, treatedas point objects (“landmarks”)

• Objects (landmarks): People, Vehicles, Robots, Human bodyparts

• Dynamics of group of landmarks: moving and deforming shape

• “Normal Activity”: Modeled as a landmark shape dynamical model

• “Abnormal Activity”: change in learned shape dynamical model,could be slow or sudden and whose parameters were unknown

Example: Group of People Deplaning

A ‘normal activity’ frame Abnormality

Figure 1:Airport example: Passengers deplaning

Dynamical Model for Landmark Shapes

• Observation: Vector of observed object locations (Configuration)

• State: [Shape, Translation, Scale, Rotation, Velocities]

• Observation model:ht : S ×R2 ×R

+ ×S0(2) → R2k, Gaussian noise

• System model:

– Gauss-Markov model on shape velocity, parallel transported totangent space of the current shape

– Gauss-Markov model on group (scale, rotation, translation)velocity

• Detect changes in shape using posterior distribution of shape givenobserved object locations

The Change Detection Problem

• Partially Observed and Nonlinear System satisfying HMM property

• Given the observationsY1, Y2, ...Yt, detect, as quickly as possible, ifa change occurred in the dynamics of the stateXt

– Parameters of changed system unknown

– Change can be slow or sudden

Notation

Yt−1 Yt

gt−1gt

Xt−1 Xt

• Prior: Given no observations,Xt ∼ pt(.)

• Posterior:Xt|Y1:t ∼ πt(.)

• Superscripts: 0 (unchanged system),c (changed system)

• X0t ∼ p0

t (.), Xct ∼ pc

Slow and Sudden Changes

• Slow change: small change magnitude per unit time, “tracked” bythe filter, i.e. ||πc,0,N

t − πct || is small

• Sudden change: “filtered out” (“loses track”)

– Duration much smaller than “response time” of filter.

– Response time (time taken to track) depends on

∗ System noise variance

∗ Observation noise variance

∗ Number of particles,N

Existing Work: Change Detection in Nonlinear Systems

• Fully observed state(no observation noise,ht invertible)

– CUmulative SUM, generalized CUSUM, negative log likelihood

• Partially observed state

– Known change parameters

∗ Linearization techniques followed by CUSUM∗ CUSUM (usest + 1 PFs att), modified CUSUM[Sadjadi et al’02]

– Unknown change parameters: few existing solutions

∗ generalized CUSUM not tractable[Andrieu et al’2004]

∗ Tracking Error [Bar-Shalom]

∗ negative Log Likelihood of Observations (OL)∗ Fail to detect slow changes

Slow change detection, Unknown parameters

• Fully observed state:

– negative Log Likelihood of state of unchanged system,

− log p0t (Xt) = − log p0

t (h−1t (Yt))

• Partially observed state (significant observation noise):

– Why not use Min. Mean Square Error estimate of this ?

• Our statistic is exactly this MMSE estimate:

ELL(Y1:t) , E[− log p0t (X)|Y1:t]

Computing the Statistics[Vaswani, ACC’2004]

• Expected (negative) Log Likelihood of state (ELL)

ELL(Y1:t) = E[− log p0

t (Xt)|Y1:t] = Eπt[− log p0

t (X)]

• For sudden changes, can use

– (negative) log of Observation Likelihood (OL)

OL(Y1:t) = − log pY(Yt|Y1:t−1) = − log Eπt|t−1[gt(Yt|X)]

– Tracking Error (TE) [Bar-Shalom]

TE = ||Yt − Yt||2, Yt = E[Yt|Y1:t−1] = Eπt|t−1

[ht(X)]

– OL ≈ TE (to first order) for white Gaussian observation noise

Detection Thresholds

• ELL Threshold: ThELL = EY0

1:t[ELL0] + k

Var(ELL0)

1:t[ELL0] = EY0

1:t[E[− log p0

t (Xt)|Y0

1:t] = h(p0

t ) = h(X0

h(.): Differential entropy

• OL Threshold: ThOL = EY0

1:t[OL0] + k

Var(OL0)

1:t[OL0] = h(Y0

1:t−1)

• Choosek based on allowed false alarm probability

• Declare a change if either ELL or OL exceeds its threshold

Change Detection Algorithm

Particle Filter

(Observation)

πNt−1

YesYes

πNt|t−1

xit ∼ qt

wit ∝ gt(Yt|x

Change (Slow)Change (Sudden)

ELL > ThELL?OL > ThOL?

ELL v/s OL (or TE)

• Slow Change:

– PF: stable under mild assumptions, tracks slow change well

– OL & TE rely on error introduced by change to detect

– Error due to change small: OL, TE fail or take longer to detect

– Estimate of posterior close to true posterior of changed system

– ELL detects as soon as change magnitude becomes detectable

• Sudden Change:

– PF loses track: OL & TE detect immediately

– ELL detects based on “tracked part of the change”

– ELL fails or takes longer

ELL Approximation Errors

• Exact filtering error : Wrong state transition kernel for changed

observations

• Particle Filtering Error : Finite number of particles:N

• Bounding error : Log Likelihood is an unbounded function, stability

and PF convergence results exist for bounded functions

Complementariness of ELL and OL [Vaswani, ACC’2004]

Theorem. ELL approx. error,errc,0,Nt , is upper bounded by an increasing

function ofOLc,0,Nτ , tc ≤ τ ≤ t, i.e.

errc,0,Nt ≤

eOLc,0,Nτ ω1(σ

2obs)ω2(ǫ

c,0τ ) + const

Implication for a“detectable” change (true value of ELL large):

• OL fails to detect a change=⇒ ELL detects

• ELL fails to detect=⇒ OL detects

Stability of ELL Error [Vaswani, ACC’2004]

Theorem. Average ELL approximation error iseventually monotonicallydecreasing (and hence stable), for large enoughN if

- Change lasts for a finite time

- ft(Xt) continuous for all t

- π0 has compact support

- gt(Yt|x) (as a function of x) has compact support, for allYt

- The convergence of the bounded approx. of ELL is uniform in time

• Based on optimal filter stability results of [LeGland & Oudjane’02]

• Valid for anyunbounded function of state(not just ELL)

• Errorasymptotically stableunder stronger assumptions

Contributions

• ELL detects a change before loss of track (very useful). OL orTracking Error detect after partial loss of track.

• Complementary behavior of ELL & OL for slow & sudden changes

• Stability of the total ELL approximation error for large N

• Relation to Kerridge Inaccuracy and a sufficient condition for theclass of detectable changes using ELL[Vaswani, ACC’04]

• ELL error upper bounded by increasing function of “rate ofchange”, increasing derivatives of all orders[Vaswani, ICASSP’04]

Simulated Example: ELL and OL Plots for Increasing Rates of Change

Xt = Xt−1 + nt + bt, bt = rσsys for t=5 to t=15

Yt = X3

t + wt

No Change:r=0 (blue),

Slow: r=0.5 (red), r=2 (magenta),Medium: r=2 (green),Sudden: r=5 (cyan)

0 5 10 15 20 25 30 35 40 45 500

ELL plot for increasing rates of change

p=0 p=0.5σ

noisep=1σ

p=2σnoise

0 5 10 15 20 25 30 35 40 45 500

60OL plot for varying change rates

← GOES TO ∞

r=0 r=0.5r=1 r=2 r=5

ELL OL

Videos

• Group of People Deplaning: Normal activity sequence

• Abnormality (One person walking away in an abnormal direction)

• Human Action Tracking & Abnormality Detection

Group of People: Abnormality Detection

Abnormality (one person walking away) begins att = 5

0 5 10 15 20 25 30 35 40 45 500

NormalAbnormal, vel=1Abnormal, vel=4Abnormal, vel=32

0 5 10 15 20 25 30 35 40 45 500

lihood

NormalAbnormal, vel=1Abnormal, vel=4Abnormal, vel=32

ELL OL

Group of People: “Temporal Abnormality” Detection

Abnormality (one person stopped in path) begins att = 5

ELL Plot

ROC Curves: “Slow” Abnormality Detection

0 10 20 30 40 50 601

Mean time between false alarms

ELL, vel = 1

σ2obs

=27σ2

obs=81

0 10 20 30 40 50 6018

Mean time between false alarms

Tracking error, vel = 1

σ2obs

=27σ2

obs=81

ELL Detects TE: Takes much longer

Human Actions: Abnormality Detection

• Abnormality begins at t = 20

• NSSA detects using ELL without loss of track

ELL Tracking Error

Future Research

• Changed Parameter Estimation

• Practical implications of the “rate of change” bound result and thestability result for particle filter design

• Applications and Performance Analysis

– Abnormal activity detection and activity segmentation

– Neural signal processing (changes in STRFs of auditory neurons)

– Acoustic tracking (changes in target motion model)

– Communications applications: tracking slowly varying channels,

congestion detection in networks

Particle Filtering for Continuous Closed Curves

(PF for Infinite Dimensional State Spaces)

(System Model)(Observation Model)

(object contour, velocities)State

Filter

Observation(Image)

(camera noise)(system noise)

t = t + 1

ht(.)+ +Xt Yt

t = t + 1

System (Object motion + deformation)

Sensor (Camera)Observation

πt(Xt|Y1:t)

Continuous Closed Curve (“Contour”)

• A smooth locus of points traced out by a mapping of the unit interval[0, 1] into R

2, with C(0) = C(1)

• Representations (Infinite dimensional)

– Parametric: C(p) = [Cx(p), Cy(p)], Cx(p) & Cy(p) are smoothfunctions ofp ∈ [0, 1], C(0) = C(1)

∗ All re-parameterizations also represent the same curve

– Implicit: Zero level set of a higher dimensional function,φ(x, y),i.e. it is the collection of all points{x, y ∈ R

2 : φ(x, y) = 0}

• Finite dim: B-splines, Fourier descriptors, Marker particle, Landmarks

Motion and Deformation [Yezzi,Soatto’02]

• “Motion” : global motion, a finite dimensional group e.g. Affine

• “Deformation” : local shape deformations, infinite dimensional

”deform”(local)(global)

“move”

• Examples:

– Fishcan move in space and also deform its shape

– Human heart only deforms,Human hand moves and deforms

The Problem

• Track a moving and deforming object from an image sequence

• State: Object Contour, Affine velocity, Deformation velocity

• Contour: Infinite dimensional level set representation

• Observation: Image (noisy nonlinear function of contour)

• Goal: Estimate the contour & velocities, given all past images(filter out the state from noisy observations)

Existing Work: Tracking Continuous Curves

• Finite dim. parametric repr. (B-splines) + Particle filter:Condensatione.g. [Isard, Blake’98]

Tracks only affine deformations. Cannot handle large changes in curve length

• Fixed finite dim marker particle repr. + Kalman filter:e.g. [Terzopoulos, Szelisky’92], [Peterfreund’99]

Prediction step correlated with current observation. Needexplicit observations

of contour. Cannot handle large changes in curve length

• Level set representation (infinite dim) + Linear observers:[Jackson et al’04], [Niethammer et al’04]

Observers for contour, velocity uncoupled. Need explicit observations

Issues we address...

1. Implicit observations: Yt = Image(t)

2. Coupled observersfor contour and velocity: use particle filtering

3. Prediction independentof current observation (image)

4. Infinite dim repr. of contour & velocity

5. Particle filtering expensive in this case

• Generating samples from a very large dim noise distrib.

• No. of particles for accurate filtering increases with noisedim

Observation Model

object

background

N(u2, σ2

N(u1, σ2

• Chan and Vese model for image formation. Image:Yt, Contour:Ct

gt(Yt|Ct) = e−

Ecv(Ct,Yt)

σ2obs

(Yt(x, y) − u1)2dxdy +

(Yt(x, y) − u2)2dxdy

Solution 1: Curve Evolution + PF

[Rathi,Vaswani,Tannenbaum,Yezzi] (Accepted for CVPR 2005, Oral)

• Constant velocity Gauss-Markov model on affine deformationand azero velocity model on local deformation

• Contour: Ct, Affine velocity: ρt, Deformation velocity: vt

• System Model:

ρt = ρt−1 + nt, nt ∼ N (0, Σρ)

Ct = Ct−1 + ∆tgaffine(Ct−1, ρt) + vtN, vt ∼ N (0, ΣC)

• Observation model: Chan and Vese model of image formation

g(Yt|Ct) ∝ e−

Ecv(Ct,Yt)

σ2obs

PF Algorithm: Importance Sampling Step

• Sampleρ(i)t from its transition kernelN (ρ

(i)t−1, Σρ)

• TakeC(i)t = C

mode,(i)t where

Cmode,(i)t = arg min

[||Ct − Caff,(i)t−1 ||ΣC

+ Ecv(Ct, Yt)]

Caff,(i)t−1 = C

(i)t−1 + ∆tgaffine(C

(i)t−1, ρ

(i)t )

Approx Solution: Start with Ct = Caff,(i)t−1 , run few steps of gradient

descent to minimizeEcv

– Can be understood as sampling from a Gaussian approx. to theoptimal importance density,p(xt|x

(i)t−1, Yt) using [Doucet’98]

– No randomness in samplingC(i)t : still works well in practice

Solution 2: Time Varying Finite Dim. Deformation

[Vaswani,Yezzi, Rathi,Tannenbaum] (Submitted CDC’05)

• Approx. curve deformation using a time varying finite dim. basis

• Assume: For sometime, “most curve deformation” occurs in finiteno. of dimensions,K: “effective basis”, for e.g.

K = 2 dim. B-spline basis will suffice

• Assume:Changes in effective basis detected & estimated accurately

Contour Deformation Model

• Local contour deformation: K basis functions,bj(p) & K-dim speed,v along basis directions

Deformation velocity, β(p) ≈K

bj(p)vj N(p)

• Global Motion (Affine): gaffine(C, ρ), 6-dim affine velocity,ρ

• Contour at t, Ct deforms as

Ct+1 = Ct + ∆t [gaffine(Ct, ρt+1) +

bjvj,t+1 N]

• Gauss Markov model onρt

and vt

vt+1 = vt + νv,t+1, νv,t ∼ N (0, Σv,t∆t)

= ρt+ νρ,t+1, νρ,t ∼ N (0, Σρ,t∆t)

• Used B-splines as the “‘effective basis” for deformation velocity (knots

move at eacht according to contour deformation model)

• “Effective basis” is piecewise constant

– Detect change in basis at every time instant

– If changed, re-estimate newK and new basis functionsbj

One problem...

• Part of contour not changing: like an “unknown static parameter”

– Resampling can result in loss of a good particle (if badobservation), new particles never generated: divergence

• Solution: Monte Carlo PF for “static parameter” [Papavasiliou’2004]

– No resampling for “static parameter” particles

– For each particle of “parameter”,run PF for rest of state space

– Proven to beasymptotically stableunder certain assumptions

– Our problem: treat unchanging contour as “static parameter”

PF Algorithm

1. At t = 0, generateM “static parameter” (contour) particles

2. At eacht, for m = 1 to M do,

(a) Run anN -particle PF

(b) Move B-spline knots, re-evaluate basis functions using new knots

(c) Weight themth contour particle usingq(M) past observations

3. Effective Basis Change Detect:If change go to step 3, else go to step 1

4. Effective Basis Re-estimation

(a) Estimate new basis dimension, learn basis vectors

(b) Re-sample particles of the static parameter (contour)

Summarizing...

Curve Evolution + PF

• Generates only a 6 dim system noise distribution

• “Curve evolution” to get mode for non-affine deformation (expensive)

• Bad observations: “loss of track” (1 particle left)

• Gets back in track slowly, can get stuck in local minima

Time Varying Finite Dim. Deformation

• Generates a K+6 dim system noise distribution

• Works w/o mode evaluation, but moving B-spline knots expensive

• Bad observations: “loss of track” but gets back faster

MCPF-PF: Back in track immediately (static particles not resampled)

Future Research

• Proof of asymptotic stability of the algorithm

• “Effective basis” change detection and re-estimation

– Other possible “effective basis” representations, geometric basis?

– Deviation from uniform B-spline knot separation

– Local tracking error, More B-spline knots where large deformations

– ELL w.r.t. the pdf of deformation velocity before change

– Posterior expected square distance to a reference contour

• Extension to surface tracking

Contributions

• First implementable solution to particle filtering for infin itedimensional state space. Many possible applications:

– Volume image segmentation as a 2D tracking problem

– Tracking human heart, detecting abnormality

– Tracking spatio-temporal receptive fields of auditory neurons

– Tracking principal subspaces (array signal processing)

• Finite dim. parametrization of deformation of a continuous curve

• Posterior mode detection using curve evolution (imp. sampling)

– Faster algorithm: Only MCPF + posterior mode detection

Summary of Recent Research

• Detection & Estimation Problems, Applications in image andvideo

– Pattern classification algorithms[IEEE Trans. IP, Accepted]

∗ Face recognition, Image/Video retrieval, Feature matching

– “Shape Activity” [IEEE Trans. IP, Accepted] [CVPR’03]

∗ Abnormal activity detection : stationary shape activity∗ Human action tracking & abnormality detection: nonstationary SA∗ Activity sequence segmentation: piecewise stationary SA

– Change detection in nonlinear systems[ACC’04, ICASSP’04,’05]

– Particle filtering for continuous closed curves[CVPR’05]

• Future research interests

– Biomedical Image & Signal Processing

– Shape analysis in Computer Vision

– Optimal Filtering theory, algorithms & applications

– Information theory for estimation/detection, video compression

Research Plan

• Biomedical Image and Signal Processing

– Use of dynamical models and tracking

∗ Dynamical models for disease progression?∗ Track the human heart, detecting abnormality∗ Volume image segmentation as 2D tracking?∗ Neural signal processing

– Shape Matching/Classification

• Particle filtering for infinite dimensional state spaces

– Asymptotic stability

– Effective basis representation, change detection, estimation

– Applications

• Particle filtering under system model error

– Changed system model parameter estimation

– Implications of my results for improved Particle Filter design

– Applications in neural and acoustic signal processing

• “Landmark Shape Activities”

– Dealing with time varying number of landmarks

– Activity Segmentation

– Using different sensors, sensor fusion

Acknowledgements

• Landmark Shape for Modeling Activity: Joint work with

Dr. Amit RoyChowdhury and Dr. Rama Chellappa

• Filtering Continuous Closed Curves: Joint work with

Dr. Anthony Yezzi, Yogesh Rathi, Dr. Allen Tannenbaum

iowa - home.eng.iastate.eduhome.eng.iastate.edu/~namrata/gatech_web/iowa.pdf · Title: iowa.dvi Created Date: 5/2/2005 6:25:50 AM

Documents

IOWA ACCELERATED BRIDGE CONSTRUCTION...

home.eng.iastate.eduhome.eng.iastate.edu/~jdm/ee590-Old/Plex...

home.eng.iastate.eduhome.eng.iastate.edu/~jdm/ee303/quiz 4.....

Compound Interest Tables -...

Chapter 5 Moving Object Detection and Compression in IR...

Hy-Vee back page August 31st Iowa.pdf

home.eng.iastate.eduhome.eng.iastate.edu/~jdm/ee552/LineTabl...

home.eng.iastate.eduhome.eng.iastate.edu/~shermanp/AERE355/l...

599 V3l - Iowapublications.iowa.gov/23324/1/Mammals of...

home.eng.iastate.eduhome.eng.iastate.edu/~shermanp/AERE355/l...

Chapter 5 Moving Object Detection and Compression in...

home.eng.iastate.eduhome.eng.iastate.edu/~shermanp/AERE355/l...

home.eng.iastate.eduhome.eng.iastate.edu/~tge/ce326/hyd.pdfA...

Summary of Papers -...

The daily Gate City and constitution-Democrat (Keokuk ......

Waterfowl in Iowa › ... › waterfowl-in-iowa.pdf ·...