TEST

Fudan University

HaishanWu

8/27/2009

Acquiring 3D Motion Trajectories of Large

Numbers of Swarming Animals

Social behavior: a spectacular sight of

nature

Scientists are interested in this

phenomenon

Nature Method 2009 PNAS 2008 Nature 2005

PRL 1995 Siggraph 1987

They need 4D motion trajectories!

4D(3D+t) trajectory of each individual is imperative:

How they

interact with

each other?

How do they make collision

avoidance decision?

What is the flock

property?

Our results! We successfully obtained 3D flying trajectories of

Drosophila melanogaster (fruit fly) group comprising

hundreds of individuals

Multi-view Geometry Technique

It is , however, a challenging problem

Animal groups with collective action contain large

variable number of interacting individuals

Moving targets themselves in a swarm usually

resemble each other in appearance

Frequent occlusions

make event state-of-the

art tracking method failed

Appearance feature

will be also ineffectual for tracking or

matching across various views

Tracking or matching

failures will lead to the

broken or

incomplete

trajectories in 3D

space

TraMaL: Our framework to solve these

challenges

TraMaL: Tracking-Matching-Linking

Novelties and contributions of our

approach

Each of the three components of this framework can be

modeled as linear assignment problem (LAP), and our

method works in near real time.

Our computationally efficient data association method, is

able to deal with large variable number of targets.

MECL(maximum epipolar co-motion length) is used to

match the trajectories . It is able to handle tracking errors

and calibrations errors

Linking model incorporate motion and temporal information

to make the final trajectories as complete as possible.

Related work

Many complex models for tracking: JPDA, MHT… Exponential complexity is a serious disadvantage .

Most related papers:

CVPR 2007: cluster based method for data association: the accuracy of the tracking trajectories is not reported

Nature Method 2008: robust model for SPT(single particle tracking): their data association method did not consider particle true and temporal disappearance, their track fragments tend to incomplete.

Related work

Numerous methods for stereo matching. A survey is available in CVPR 2007. But most the them combine the appearance feature and epipolar constrains

Most related paper:

ICCV 2007: REM will discard many potential matching candidates (as shown in experiments)

IJCV 2006: aim to align video sequences captured in various cameras, which is achieved by matching partial trajectories of several moving objects

Related work: other fields

Computational fluid dynamics: A Spatial-Temporal Matching Algorithm for 3D Particle Tracking Velocimetry. PhD thesis : Reconstruct from 4 different views, then track particles in 3D object space. Trajectories will be broken frequently because of matching and tracking ambiguities.

Biology community: A Simple Vision-Based Algorithm for Decision Making in

Flying Drosophila ,Current Biology 2008: 5 cameras to track 20 fruit flies. But as reported, their approach is only able to track up to 2 targets simultaneously

The STARFLAG handbook on collective animal behavior , Animal Behaviour,2008: obtained the 3D coordinates of a large bird flock, but 4D trajectories is unavailable.

Our method: TraMaL

Tracking

2 steps: single particle state updating and data

association

State updating: Markov assumption and alpha-beta

filter (simple, yet efficient)

Tracking, the first LAP (linear

assignment problem )

The first LAP is:

To handle these

difficulties:

Matching, the second LAP

Now our problem is how to assign (match) a 2D trajectory in one view to that of another view, which can be also formulated as a LAP .

The appearance feature of targets, however, is useless in our experiments, so how to define the matching cost?

Observations: Two true matching trajectories will submit to epipolar constrains The longer the length of matched trajectories, the larger the possibility of

correct matching is.

Matching cost function

MECL(maximum epipolar co-motion length)

For a rectified image pair in two different views:

Matching cost function

Solve LAP recursively :

Matching overlap

One observation:

Linking, the third LAP

3D trajectories can be recovered by

using multi-view geometry technique.

However, tracking errors make

these trajectories broken into

segments

We solve this problem by formulating it

as pairwise tracklet matching problem,

namely a LAP

Linking, cost function

We incorporate temporal and kinematic information into

linking cost.

temporal information

kinematic information

Results in simulating data sets

Tracking Performance:

Parameters:

1. : depends on the

intensity of noise

2. : determined in an

adaptive way

3. : 1-5 frame

Simulated particle swarms:

Results in simulating data sets

Matching Performance:

Parameters:

: depends on accuracy of camera calibration

Results in simulating data sets

Linking Performance

Parameters:

1. For non-overlap

candidates, maximum

frame difference

equals to

2. For overlap candidates,

maximum overlap

length also equals to

Results in simulating data sets

Compared to:

Evaluation metrics:

Trajectory completeness factor (TCF), which measures on average the ratio of a ground truth trajectory length covered by the reconstructed trajectories. TCF is 100% when all reconstructed trajectories are correct.

Trajectory fragmentation factor (TFF), which measures on average the number of acquired trajectories used to match one ground truth trajectory. The ideal score of TFF is 1, the larger this value is, the worse the performance on maintaining particle identity.

Results in simulating data sets

Results in real world datasets

We use our method to obtain 3D flying trajectories of Drosophila melanogaster (fruit fly) group comprising hundreds of individuals (more than 150 detected targets in each frame).

Experiment setup:

The fruit flies flied freely in an acrylic box of size 35cm by 35cm by 25cm, where the background was illuminated by white plane lights. Two synchronized and calibrated Sony HVR-V1C video cameras working in high speed mode at 200fps were used to capture the scene from different views. The image resolution was 960×540 and 200 frames were captured for the experiments.

Results in real world datasets

Near 400 4D trajectories are obtained All the obtained trajectories Some longer trajectories

We hope our results could lead to deeper understanding of

fly social behavior of fruit fly group.

Future work

Build an advanced probabilistic model for MECL

Test our framework on bird flock or fish school data sets

Maybe we have a chance to analyze the trajectories and

discover something interesting