Research and Future Perspectives on Intelligent Video Surveillance Systems

Research and Future Perspectives on Intelligent Video Surveillance Systems

Monique THONNAT

Senior Scientist Head of Orion research team

INRIA Sophia AntipolisFRANCE

2

Introduction 1/4

Which Security Problems? Safety and security of goods and human beings

Safety: protection in case of accident or incident (e.g.: fall)

Security: protection against a malicious act (ex.: bomb) Evolution

1980: guarding, human surveillance 1990: start of video cameras setting up, remote

surveillance Highways, parking lots, subways, malls …

2000: explosion of video-surveillance Set of new laws Better understanding of the general public Threats of tragic events, terrorism acts

3

Introduction: control room 2/4

4

Introduction: issue 3/4

Definition: intelligent video surveillance automatic analysis of the video streams

Why ? Cost: 1 human operator /6-10 video streams Current paradox : the more there are video

cameras, the less these video camera are observed E.g. 600 000 video cameras in London need for the policemen to look at stored videos tapes

Effectiveness: duration of vigilance for an operator: > 1 or 2hours: % of the attention is lost

5

Introduction 4/4

Huge information flow Few pertinent information

Video streams

Video cameras Surveillance control rooms

Selection of the information Increase of the detection rate

Intelligent video

surveillance software

Intelligent video

surveillance software

Alarms

Video cameras

Video streams

Surveillance control rooms

6

Intelligent Video Surveillance 1/3demonstration

From object detection to complex event recognition(e.g.: violence)

7

Intelligent Video Surveillance Definition: Data captured by video surveillance cameras Real time and automated analysis of video sequences Video understanding= from people detection and

tracking to behavior recognition

Recognition of complex behaviors: of individuals (e.g. fraud, graffiti, vandalism, bank

attack) of small groups (e.g. fighting) of crowds (e.g. overcrowding) interactions of people and vehicles (e.g. aircraft

refueling)

Intelligent Video Surveillance 2/3

8

Intelligent Video Surveillance 3/3Typical problems

Metro station surveillance Surveillance inside trains

Building access control Airport monitoring

9

Video Understanding for Intelligent Video Surveillance 1/13

Definition:Cognitive Vision is a new research field mixing: Computer vision techniquesfor object detection, description, categorization and tracking Artificial intelligence techniquesfor knowledge acquisition, reasoning (e.g. spatial and temporal

reasoning,…), learning (e.g. categories, structures, parameters…)

Software engineering techniquesfor vision software design, integration, reusability, evaluation

Reference: http:www.eucognition.org/ecvision site and roadmap

10

Intelligent Videosurveillance:

How? A Cognitive vision approach for video understanding mixing:

computer vision: 4D analysis (3D + temporal analysis)

artificial intelligence: a priori knowledge (scenario,

environment)

software engineering: reusable software platform (VSIP)

Video Understanding for Intelligent Video Surveillance

2/13

11


4 D analysis:multi-cameras

tracking

Video understanding

People detection

and tracking

Scenario recognition

A PRIORI KNOWLEDGE:• 3d models of the environment • Camera calibration• Scenario Models

Alarms

People detection

and tracking

Interpretation of the videos from pixels to alarms

12


4/13

SegmentationSegmentation ClassificationClassification TrackingTracking Scenario RecognitionScenario Recognition

Alarms

access to forbidden

area

3D scene modelScenario models A priori Knowledge

Objective: Interpretation of videos from pixels to alarms

13

Behavior recognition: approach based on a priori knowledge

model of the empty scene (3D geometry and semantics)

models of predefined scenarios

a language for representing scenarios based on

combination of states and events more than 20 states and 20 events can be used

a reasoning mechanism for real time detection of states,

events and scenarios (e.g. temporal reasoning,

constraints solving techniques)


5/13

14

3D Scene Model: Barcelona Metro Station Sagrada Famiglia mezzanine (European project ADVISOR)


6/13

15

States, Events and Scenarios : State: a spatio-temporal property involving one or several actors on a time interval

Ex : « close», « walking», « seated»

Event: a significant change of states

Ex : « enters», « stands up», « leaves »

Scenario: a long term symbolic application dependent activity

Ex : « fighting», « vandalism»


7/13

16

Vandalism scenario description :

Scenario(vandalism_against_ticket_machine, Physical_objects((p : Person), (eq : Equipment, Name = “Ticket_Machine”) )

Components ((event s1: p moves_close_to eq) (state s2: p stays_at eq)

(event s3: p moves_away_from eq) (event s4: p moves_close_to eq)

(state s5: p stays_at eq) ) Constraints ((s1 != s4) (s2 != s5)

(s1 before s2) (s2 before s3) (s3 before s4) (s4 before s5) ) ) )


8/13

Scenario Recognition : Temporal constraints

17

Video Understanding for Intelligent Video Surveillance 9/13Vandalism in metro (Nuremberg, Germany)

18


3d Scene Model of 2 bank agencies

objet du contexte

mur et portezone d’accès

salle du coffrerue

rue

salle automates

zone d’entrée de l’agence

zone des distributeurs

zone de jour/nuit

zone devant le guichet

zone derrière le guichet

zone d’accès au bureau du

directeur

zone de jour

ported’entrée

porte salleautomates

armoire

guichet

commode

19


11/13Bank Monitoring: Bank attack scenario description :

scenario Bank_attack_one_robber_one_employeephysical_objects: ((employee : Person), (robber : Person), z1: Back_Counter, z2: Entrance_Zone, z3: Front_Counter, z4: Safe, d: Safe_door)

components: (State c1 : Inside_zone(employee, z1)) (Event c2 : Changes_zone(robber, z2,z3))

(State c3 : Inside_zone(employee, z4)) (State c4 : Inside_zone(robber, z4))) constraints : ((c2 during c1) (c2 before c3) (c1 before c3) (c2 before c4) (c4 during c3) (d is open))

20


12/13

bank monitoring Recognition of a bank attack scenario: an employee is e behind a counter, an aggressor enters, goes behind the counter, then he goes with the emplyee towards theSTR (secured technical room), they enter in the STR then they leave ethe STR and they go toward the exit of the agency.

Examples : Brussels and Barcelona Metro Surveillance

Exit zone

Jumping over barrier

Blocking

Overcrowding

Fighting

Group

behavior

Crowd

behavior

Individual

behavior

Groupbehavior


13/13

21

22

Impact: Visual surveillance of metro stations, bank agencies, trains,

buildings and airports

5 European projects (PASSWORDS, AVS-PV, AVS-RTPW,

ADVISOR, AVITRACK)

4 contracts with End-users companies (metro, bank, trains)

2 transfer activities with Bull (Paris) and Vigitec (Brussels)

Cooperation over more than 11 years with partners

Creation in 2005 of a start-up Keeneo www.keeneo.com

Conclusion 1/5

23

Conclusion 2/5

Hypotheses: fixed cameras 3D model of the empty scene predefined behavior models

Results:

+ Behavior understanding for Individuals, Groups of people, Crowd or Vehicles

+ an operational language for video understanding (more than 20 states and events)

+ a real-time platform (10 to 25 frames/s)

24

Conclusion 3/5 Current issues

Systems have poor performances over time, can be hardly modified and do not use enough a priori knowledge

shadowsstrong perspectivetiny objects

close view

clutterlightingconditions

25

Knowledge Acquisition Design of learning techniques to complement a

priori knowledge: Frequent events, scenario model learning European project CARETAKER

Object description Fine human shape description:3D posture models Crowd description: European project SERKET

Reusability is still an issue for vision programs Video analysis robustness Dynamic configuration of programs and parameters

Conclusion: Where we go 4/5

26

Conclusion 5/5 Posture Recognition

Current image and binary image

Instantaneous posture

Postures recognised along the time

27

Conclusion 5/5 Crowd Behavior

Motion direction detection

Abnormal direction of people in a crowd

28

Video Understanding demo?

Airport Apron Monitoring “Unloading Operation” European AVITRACK project

Research and Future Perspectives on Intelligent Video Surveillance Systems

Documents