Monitoring Camera Video Analysis Vera Kettnaker. Motivation Cameras are everywhere...…but how can we make sense of all this information ? NYT 3/19/98:

Monitoring Camera Video Analysis

Vera Kettnaker

Motivation

Cameras are everywhere...…but how can we make sense of all this information ?

NYT 3/19/98: > 80 cams on short route through Manhattan

Current use of monitoring cameras (1)

Video from most cameras is analyzed by users

On the web: Traffic cams, weather cams, Seismo-cam….

Current use of monitoring cameras (2)

Computer Vision event detection

Recording and preventing law violations: running red lights, accidents, theft & robbery, planting bombs in airports…

Current use of monitoring cameras (3)

Single camera content analysis systems

Commercial example: compute traffic statistics for traffic planning

Volume/counts (# of vehicles) Lane Occupancy (% time lane is occupied) Speed (average speed in mph or kph) Density (average density = volume/speed) Headway (average headway in seconds) Length (average vehicle length in ft/meters) Vehicle Classification by user-selectable lengths (5 bins) Delay (average delay in seconds) Queue Length

Current use of monitoring cameras (4)

Systems of multiple, overlapping cameras

Construct multi-view mosaics or project into common coordinate system

large area can be monitored even if there are obstaclesCool application: One company sells a small, fast, submersible remote vehicle as a “3 camera, 240 degree view, 6,000' depth underwater survey system”

Quote: “You can even seem to swim with sharks, hundredsof them ! All in comfort, safety, and with friends or family.”

System in this talk

Several non-overlapping cameras monitor environmentin which objects move along pathways, e.g. corridors, streets, highways

Open environment : objects enter and leave, it is not known how many different people were visible.

All single camera observations are collected into globaldatabase.

Task

Task: reconstruct the paths of all objects through the environment = link successive occurrences of the same object

Application: trajectory statistics are useful for traffic planning

Preprocessing

Single-camera tracker extracts moving object info that is stored in database.

database

Single-camera information extraction

1. Use combination of background subtraction and Boykov-Veksler-Zabih motion algorithm to segment moving objects.

Main problem: strong reflections and shadows

2. Represent people by the color of their clothing: How much area of the person is covered in which color ? = coarse histograms; choice of color space important ! Varies over time: use probabilistic model

Main problems: • shadows • different lighting conditions• different viewing angles • clothing colors difficult: beige, grey, jeans • inclusion of background

Visual matching alone won’t be sufficient to solve the task !

Single-camera information extraction (2)

Exploiting additional information

Observation: the pathway structure imposes constraints on when and where moving objects can reappear.

Input

C1

C3

C4

C2

1/32/3

1/2

1/2

2.Traffic statistics:• transition probabilities

• transition time probabilities

• frequency of new objects.

1. Database of observation intervals

Output

Requirements:• most likely according to - usual traffic behavior - similarity of visual appearance • chains must be mutually exclusive “Quality measure”

Solution = set of chains

Difficult to efficiently compute a solution that satisfies both requirements ! If we only have a black-box quality measure, we can’t do much better than brute force search. (Bad !! Huge search space….)

Efficient computation

1. Step of decomposition: quality of set of chains =combination of quality measure of each chain separately

Key for efficient computation: decompose quality measure so that we can make decisionsmore independent of each other.

Decomposing the quality measure - Step 2

Transformation such thatquality of a chain =combination of quality measure of each link

What does the decomposition mean?

First decomposition step (chains) Main obstacle: # chains is a global property of a solution Relation between #chains & quality is not simple:

Second decomposition step (links) Requires additional assumptions -- Markov transition probabilities -- transitivity of visual similarity

Adding a chain can increase or decrease the qualityof the solution !

Transformation into a Linear Program

Weighted Assignment Problem

& exclusivity constraints

Decomposition,

Transformation

max quality

Special case of a Linear Program

1. Mutual exclusiveness of chains:• in-degree of successor = 1 (predecessor = real o.i. or ‘NEW’)

• out-degree of predecessor <=1

2. Minimize total link costs

Weighted Assignment Problem

Two roles for each observation interval:predecessor and successor

Can be solved in cubic time via Munkres algorithm. see OR courses...

NEW

... ...

1

N

N

2

3

1

2

3

N

Matrix elements = link terms

Matrix representation

?

Very unlikely matches:occurred long time ago

# comparison pairsgrows very big with time

Pruning # comparisons: Focus Sets

Observation: links that are less likely than will notbe part of the optimal solution Conservative estimate: which links are plausible candidates ? Use upper bound on visual match quality relative time window for each pair of cameras

NEW

Focus set =set of plausiblematch candidates

So far: batch algorithm

Not appropriate for monitoring systems !• problem size becomes very big• continuous output of answers desirable

Problem: we can prove that there is no online algorithm that returns the correct result for all inputs.

Proposal: Approximate online algorithm that uses limited, semantically meaningful lookahead.

Online Algorithm

3. Solve this subproblem.

Online Algorithm

1. Determine possible match candidates for A.V

AFocus set A

W

X

B

C

Y

ZFocus set B

Focus set C

2. Which other focus setsoverlap with A’s focus set ?

W

X

4. Fix A’s assignment

• Inclusion of A in previous lookaheads consistency with past• variant: include also the contestants of the contestants of ... trade off waiting time and correctness

Evaluation

Evaluation with 8 minute 4-camera experiment

Results

• 2 out of 28 observation intervals are linked to wrong predecessor• Additionally: 2 links are missing for reoccurences after unnaturally long disappearance time• small focus sets

Summary

• Introduced Multi-camera Path Reconstruction Task

• visual appearance matching aided by knowledge of traffic statistics

• Transformation into a Weighted Assignment Problem: efficient computation

• Focus sets, online version

• Preliminary experimental results are promising

Performance of online algorithm

Fraction of runtime as comparedto the batch version

Error fraction: ~ 2%

Problem size in # observations

For size 100: 10x faster