REAL-TIME 3D MODELING OF VEHICLES IN LOW-COST MONOCAMERA SYSTEMS

1

Real-time 3D modeling of vehicles in

low-cost monocamera systems

1

M. Nieto, L. Unzueta, A. Cortés, A. Barandiaran, O. Otaegui, and P. Sánchez

2

Outline

Introduction Context

Objectives

Motivation

System overview Examples

Proposed solution

Calibration

Segmentation

2D blob extraction

3D tracking

Bayesian framework Likelihood

Prior

Real-time operation MH

From 3D to 1D search

Finite posterior evaluations

Results

2Algarve (Portugal) - VISAPP 2011/03/05-07

3

Introduction

Context

Algarve (Portugal) - VISAPP 2011/03/05-07 3

“Intrusive” technology (ILD) Non-intrusive technology (video)

4

Introduction

Conditions

Real-time solution

High speed roads

Objectives

Count and classify vehicles (light and heavy)


“Any” perspective

Low-cost camera

5

Introduction

Motivation

Projective distortion


6

Introduction

Motivation

Length error (2D vs 3D)


7

Introduction


Motivation

Width error (2D vs 3D)

8

System overview


Examples

9

System overview

Proposed solution


10

System overview

Calibration and rectification


Radial distortion

Perspective distortion

Metric information

We can obtain the approximate intrinsic and extrinsic parameters

of the calibration from the homography

11

System overview

Segmentation

Pixel-level classification

Colour (IHLS) and gradient


12

System overview

2D blob extraction


Remove small blobs

Merge (it. #1) Merge (it. #2)

13

System overview

3D tracking

Infer 3D boxes that projects onto detected 2D blobs

Propagate through time


14

Bayesian framework


Likelihood

(Motion) Prior

Models

State (t-1) State (t)

15

Bayesian framework

MCMC-based particle filter Recursive Bayes’ rule

Approximate with a set of samples (particles)

Obtain samples with Metropolis-Hastings


Posterior LikelihoodPrediction

Sample-basedapproach

16

2D blob

3D best box

3D hypotheses

Bayesian framework

Likelihood model


Plane perpendicular to the raypassing through

Projection of into the ray

17

Bayesian framework

Prior vehicle models


18

Real-time operation

Real-time is a hard requirement

High speed roads + short stretchs = few frames

MCMC particle filter

Propagate a chain of samples using Metropolis Hastings

It might be slow (depend on the processor)

Simplifications are welcome

One single sample


Check discrete posterior values and compute argmax


19

Real-time operation

Metropolis-Hastings

Full computation

Single-sample posterior representation


Previous set of particles

Single previoushypothesis

20

Real-time operation



s.t.

Measures the distance of the hypothesis

to the ray

Let us consider onlypoints on the ray

Need to sample 3D space WHL

Need to sample 1D ray

21

Real-time operation

Finite (small) number of posterior evaluations

Approximate MAP as product of most likely model and prior

Project each vehicle model to the ray given by the 2D observation


We reduce the search to the

evaluation of M points of the ray

These points are the projection of the set of prior models on the ray

22

Results

Remarkable capabilities

Flexibility to adapt to large vehicles


24

Results

Capabilities

Multiple / simultaneous vehicles

Static / moving vehicles


25

Results

Capabilities

Adapts to visible part of vehicles

Better fits when several frames have led the filter to converge


26

Results

Known problems / difficulties


Visual artifacts

Merging vehicles

Shadows

Occlusions / perspectiveNight time

Dark vehicles

27

Results

Classification


28

Results

Recall & precision in challenging scenarios

6+ hours

13000+ vehicles


Sequence N_L N_P R_L (%) P_L (%) R_P (%) P_P (%)

Dusk 1662 118 98.01 98.61 94.92 95.73

Rain & Shadow 4516 627 95.70 94.84 92.98 87.80

Traffic jam 4796 563 98.33 98.91 91.47 91.80

Dusk and rain 968 115 92.25 88.42 87.83 81.45

Perspective 614 101 91.86 92.16 86.14 84.47

Color noise 561 23 99.82 99.12 86.96 86.96

Average 13117 1547 95.99 95.34 90.05 88.03

29

Results


0,8

0,85

0,9

0,95

1

0,8 0,85 0,9 0,95 1

Pre

cisi

on

Recall

Dusk

Rain and shadows

Traffic jam

Dusk and rain

Perspective

Color noise

0,8

0,85

0,9

0,95

1

0,8 0,85 0,9 0,95 1

Pre

cisi

on

Recall

Dusk

Rain and shadows

Traffic jam

Dusk and rain

Perspective

Color noise

Light vehicles Heavy vehicles

Dusk Rain and shadows Traffic jam Dusk and rain Perspective Colour noise

30

Conclusions

Vehicle detection and tracking 3D volume estimation

Real-time operation Using some approximations within the particle filter framework

Flexible Automatically adapts to illumination changes

Requires calibration to work under different perspectives

Average counting and classification rates above 90% in challengingscenarios

Future work: interaction between objects, autocalibration, automatic selection of parameters…


3131

Dr.-Ing. Marcos Nieto DoncelInvestigador/Researcher

[email protected]

http://marcosnieto.net/

Algarve (Portugal) - VISAPP 2011/03/05-07

32

Results

Night time performance


REAL-TIME 3D MODELING OF VEHICLES IN LOW-COST MONOCAMERA SYSTEMS

Technology