WALLFLOWER PRINCIPLES AND PRACTICE OF BACKGROUND MAINTENANCE Costache Theodor “Hermann Oberth” Faculty of Engineering – Advanced Computing Systems Master.

WALLFLOWER PRINCIPLES AND PRACTICE OF BACKGROUND

MAINTENANCE

Costache Theodor “Hermann Oberth” Faculty of Engineering – Advanced Computing Systems Master

OVERVIEW

Introduction

The Wallflower Algorithm

Experiments

Analysis and Principles

Conclusions

Video surveillance systems seek to automatically identify people, objects, or events of interest in diff erent kinds of environments.

A common element of such surveillance systems is a module that performs background subtraction for diff erentiating background pixels from foreground pixels.

The diffi cult part of background subtraction is not the diff erencing itself, but the maintenance of a background model.

INTRODUCTION

An ideal background maintenance system would be able to avoid the following problems:

INTRODUCTION

Moved objects

Time of day

Light switch

Waving trees

Camouflage

Bootstrapping

Foreground aperture

Sleeping person

Waking person Shadows

Wallfl ower was intended to solve as many of the canonical problems as possible.

To handle problems that occur at various spatial scales, it processes images at the pixel, region, and frame levels.

The pixel- level processing makes the prel iminary classifi cations of foreground versus background and also handles adaptation to changing backgrounds. The pixel- level avoids many of the common problems immediately: moved objects, t ime of day, waving trees, camoufl age, and bootstrapping.

The region-level considers inter-pixel relationships that might help to refi ne the raw classifi cation of the pixel level; doing so avoids the foreground aperture problem.

The frame level addresses the l ight switch problem: it watches for sudden changes in large parts of the image and swaps in alternate background models that explain as much of the new background as possible

THE WALLFLOWER ALGORITHM

THE PIXEL LEVEL

The idea is to segment whole objects, rather than isolated pixels, as foreground.

When an object is homogeneously colored, the problem is an instance of the classic vision problem where a moving homogeneous region exhibits no perceivable motion.

Moving homogeneous regions are necessarily bracketed by pixels which occur on the frontier edges of object movement, and which exhibit identical properties as those inside the region

So, we cast new foreground regions that have been discovered by the pixel level as seed regions that are grown by backprojecting the pixel values that occur in the seeds.

THE REGION LEVEL

A s e a c h n e w p a i r o f r a w a n d f o re g ro u n d - m a r ke d i m a g e s , a n d , a r r i v e s ,

C o m p u t e i m a g e d i ff e re n c e s ( a a n d b ) :

C o m p u t e t h e s u b s e t o f p i xe l s w h i c h o c c u r a t t h e i n t e r s e c t i o n o f a d j a c e n t p a i r s o f d i ff e re n c e d i m a g e s a n d t h e p re v i o u s f o re g ro u n d i m a g e ( c ) :

Fi n d 4 - c o n n e c t e d re g i o n s , , i n , d i s c a rd i n g re g i o n s c o n s i s t i n g o f l e s s t h a n p i xe l s

C o m p u t e , t h e n o rm a l i z e d h i s t o g r a m o f e a c h , a s p ro j e c t e d o n t o t h e i m a g e ( s i s a p i xe l v a l u e ) :

B a c k p ro j e c t h i s t o g r a m s i n : Fo r e a c h , c o m p u t e , a n d f ro m e a c h p o i n t i n t h e i n t e r s e c t i o n , g ro w , t h e 4 - c o n n e c t e d re g i o n s i n t h e i m a g e ,

w h e re w e u s e = 1 6 , = 8 , ε = 0 . 1 .

THE REGION LEVEL

THE FRAME LEVEL

EXPERIMENTS

EXPERIMENTS

Background maintenance seems simple at fi rst. But it turns out to be a problem rich with hard cases and subtle tradeoff s. Therefore, a set of principles to which background maintenance modules should adhere, need to be established.

ANALYSIS AND PRINCIPLES

•No Semantics

Principle 1

•Proper Initial Segmentation

Principle 2

•Stationarity Criteria

Principle 3

•Adaptation

Principle 4

• Multiple Spatial Levels

Principle 5

The Wallfl ower algorithm outputs false posit ives in the waving trees sequence, where part of the sky, considered background by the pixel- level, becomes foreground after region-level processing. The region-level a lgorithm is therefore an unsound heurist ic, the use of which is not just ifi ed in general because it is an attempt to extract object semantics from low-level v is ion.

Semantic diff erentiation of objects should not be handled by the background maintenance module.

A background maintenance module handles the default model for everything in a scene that is not modeled expl ic it ly by other processing modules.

The module performing background maintenance should not attempt to extract the semantics of foreground objects on its own.

While background maintenance might be useful in determining gross traffi c stat ist ics of objects such as people and cars, which can alternately be moving or motionless, attempts to use it a lone as a preprocessing step for cont inuous, accurate tracking are bound to fai l .

PRINCIPLE 1 – NO SEMANTICS

As long as there is to be a background maintenance module, we must diff erentiate the task of fi nding foreground objects from the task of understanding

Background subtraction should segment objects of interest when they fi rst appear (or reappear) in a scene .

In environments where adaptation is necessary, maintaining foreground objects as foreground is not a reasonable task for background modelers, since such accurate maintenance requires semantic understanding of foreground.

Object recognition and tracking can be computationally expensive tasks – good background subtraction eliminates the need to perform these tasks for each frame, on every subregion.

PRINCIPLE 2 – PROPER INITIAL SEGMENTATION

The principle of proper init ia l segmentation depends on the notion of object sal ience. Objects are sal ient when they deviate from some invariant property of the background. The key quest ion, then, is how this invariance is modeled and what i t means to deviate from it .

Backgrounds, for instance, are not necessari ly defi ned by absence of motion. Consider a fl utter ing leaf on a tree. As the leaf moves on and off a pixel , that pixel ’s value wi l l change radical ly. No unimodal distr ibut ion of pixel values can adequately capture such a background, because these models impl ic it ly assume that the background, apart from some minimal amount of noise, is stat ic . Indeed, al l unimodal models fa i led to capture the complexity in the background required to handle the waving trees and camoufl age experiments.

An appropriate pixel-level stationarity criterion should be defi ned. Pixels that satisfy this criterion are declared background and ignored.

We defi ne stat ionarity as that qual i ty of the background that a part icular model assumes to be approximately constant. Carelessness in defi ning the stat ionarity cr i ter ion can lead to the waving trees and camoufl age problems.

PRINCIPLE 3 – STATIONARITY CRITERIA

Backgrounds often change, even with liberal definitions of stationarity.

The background model must adapt to both sudden and gradual changes in the background.

The line between gradual and non-gradual changes should be chosen to maximize the distinction between events that cause them.

PRINCIPLE 4 – ADAPTATION

Sudden l ight changes were best handled by normalized block correlation, eigenbackground, and Wallfl ower. On the other hand, neither eigenbackgrounds nor block correlation deals with the moved object problem or the bootstrapping problem, because they lack adaptive pixel level models. So, our fi nal principle is the fol lowing:

Background models should take into account changes at diff ering spatial scales.

Most background maintenance algorithms maintain either pixel-wise models or whole-frame models, but not both. Pixel-level models are necessary to solve many of the most common background maintenance problems, while the l ight switch and time-of-day problems suggests that frame-wide models are useful, as well.

A good background maintenance system is l ikely to explicitly model changes that happen at diff erent spatial scales. Much of Wallfl ower’s success is attributable to its separate models for pixels and frames.

PRINCIPLE 5 – MULTIPLE SPATIAL LEVELS

Background maintenance, though frequently used for video surveillance applications, is often implemented ad hoc with little thought given to the formulation of realistic, yet useful goals. We presented Wallflower, a system that attempts to solve many of the common problems with background maintenance. Comparison of Wallflower with other algorithms establishes a case for five principles that we proposed based on analysis of the experiments

CONCLUSION