Body detection, tracking and analysis E-TEAM Participants (9): FORTH, ACV, BILKENT, SZTAKI, ICG, University of Amsterdam, University of Surrey, Technion,

Body detection, tracking and analysisE-TEAM

Participants (9):

• FORTH, ACV, BILKENT, SZTAKI, ICG, University of Amsterdam, University of Surrey, Technion, UPC

E-Team Leader: Montse Pardàs (Cristian Cantón) (UPC)

Participants

FORTH: Antonis Argyros, Panos Trahanias ACV: Herbert Ramoser Bilkent: Ugur Gudukbay, Enis Cetin, Yigithan Dedeoglu, B. Ugur

Toreyinç SZTAKI: Tamas Sziranyi ICG: Horst Bischof University of Amsterdam: Thang Pham, Michiel van Liempt,

Arnold Smeulders University of Surrey: Bill Christmas Technion/MM: E.Rivlin, M. Rudzsky UPC: Montse Pardas, Jose Luis Landabaso, Cristian Canton

Description

Relevant to WP5 (Single modality processing) and WP11 (Integration and Grand Challenges: Detecting and interpreting humans and human behaviour in videos)

Objective: To increase collaboration in:

– Body detection. Using for instance background learning techniques in both single and multi-camera environments. Persons will be identified by means of classification techniques.

– Body tracking. By means of models (e.g., templates, 3D models, classifiers) and appropriate motion prediction.

– Body analysis. Body models are being used for analysis and tracking. They can range from simple to complex models, depending on the applications.

UPC application: smart rooms

Object localization and tracking task in indoor environments surveyed by multiple fixed cameras

UPC: Detection and tracking

The method uses a foreground separation process at each camera, based on Stauffer and Grimson background learning

A 3D-foreground scene is modeled and discretized into voxels making use of all the segmented views

Voxels are grouped into blobs

Color information together with other characteristic features of 3D object appearances are temporally tracked using a template-based technique

Cam 1

ForegroundSegmentation

3D Reconstruction & Connected Components Analysis

Voxel Coloring

Size

Position &Velocity

Object / Candidate Feature Matching

Kalman Predictor

Feature Extraction

Cam 2

ForegroundSegmentation

Cam N

ForegroundSegmentation

3D Labels

BLO

B EX

TRAC

TIO

NO

BJEC

T TR

ACK

ING

Histogram

UPC: 3D Blob Extraction

UPC: Body and gesture analysis

Aim: obtain the body posture of several people present in a room.

Many pattern analysis challenges can be addressed in this framework:

– Gesture analysis: Scence understanding and classification (who is doing what?

i.e. someone raises his hand to ask a question) Friendly and non-intrusive Human Computer Interfaces (HCI)

– Gait analysis: Biometrics Motion disorders detection and diagnosis

UPC: Model based analysis

Aim: Extract the posture of a human body based on a hierarchical representation of its skeleton.

Example (I) – Simple Model

Simple body model

Position analysis, simple body action (standing up, walking,…).

Example (II) – Not so simple model

Related publications:

- C.Canton-Ferrer, J.R.Casas, M.Pardàs, Towards a Bayesian Approach to Robust Finding Correspondences in Multiple View Geometry Environments, CGGM, Atlanta (USA). LNCS 3515:281-289, Springer-Verlag, 2005.

- C.Canton-Ferrer, J.R.Casas, M.Pardàs, Projective Kalman Filter: Multiocular Tracking of 3D Locations Towards Scene Understanding, MLMI, Edinburgh (UK). To appear in LNCS, 2005.

Example (III) – Skeleton model

From the voxels data-set, extract information about the structure and the position of the joints of our skeleton model.

Stick body model

Gesture analysis, Gait analysis, Biometrics,…

UPC

Possible collaboration:

– Introduce classification of the detected objects in the smart-room context

– Introduce new techniques of gesture or activity recognition in the smart room context

– Support other groups in the extension from single camera to multi camera

– Introduce body models in other groups applications– New applications/analysis methods over our data

(availability to generate multi-camera data)

University of Amsterdam

Reconstruction of trajectories of people in street surveillance videos:

– People detection: state-of-the-art from literature

– Tracking algorithm: our own work with solid software implementation

– People matching: our own work in general object matching with color invariant descriptors (software is still under development)

Sztaki (Szriyanzi)

Aim: Extraction of simple biometric motion of walking and human actions from videos

Method: – The method works with spatio-temporal input information to

detect and classify typical patterns of human movement. – Real-time operations– New information-extraction and temporal-tracking method

based on a simplified version of the symmetry pattern extraction, which pattern is characteristic for the moving legs of a walking person. This pattern also helps in recognizing human events of more people and unusual actions.

Symmetry patterns of walking humans

Feature extraction – Identification of the leading leg

Leading leg: the “staning” leg from 2 steps, Ratio of integrated leg-areas

d

Sztaki (Chetverikov)

Robust Structure-from-Motion, 3D motion segmentation and grouping

– Given a set of feature points tracked over the frames, we can do robust SfM in presence of more than 50% outliers.

– Based on that, we can do robust 3D motion segmentation of multiple objects in presence of occlusion, outliers, and for moving camera.

– Recently, we have also developed a novel method for grouping the segmented parts, in order to decide which of them are related. For example, one can determine if an object rotates around an axis defined by another object.

Bilkent

Human body extraction, tracking and activity recognition from video sequences.

– Body detection and extraction based on motion detection and object shape based classification techniques, background learning and silhouette shape-based object classification.

– Multi-person and single person tracking: Correspondence-based whole body tracking and model-based body part tracking methods.

– Human action recognition: Tracking results will be combined with activity models (action templates), Hidden Markov models and dynamic programming techniques.

ACV

Fast Spatio-Temporal tracking based on Principal Curves

ACV

Back-projected reconstructed trajectories

ACV

Possible cooperation:– Applying our tracking methods to your data– Benchmarking, evaluation of motion detection,

tracking performance– Algorithms for fast computation of informative

descriptors (for recognition and tracking tasks)

ICG

People detection based on an On-line Adaboost method, which is embedded in a learning framework that can train a Person detector without hand labelling.

Appearance based tracking of people based on an on-line classifier.

Both methods are based on integral orientation histogram features and are able to run in real-time on a standard PC

ICG

Possible contributions:– Various sequences we use for testing our

methods (some of them with ground truth). – Combining our methods with other techniques to

improve the robustness and applicability.

Technion

Detection of moving objects Tracking of detected targets Classification to one of

predefined classes:– human, – human group,– animal,– Vehicle

E.Rivlin, M.Rudzsky, R.Goldenberg, U.Bogolmolov and S.Lapchev.,ICPR'02Y.Bogomolov, G.Dror, S.Lapchev, E.Rivlin, M.Rudzsky. BMVC’03

The classes handled by the system

Technion

Classification of moving objects:– Single human (walking, running,

crawling)

walk run run45

Tracking

Technion

Possible collaboration in research of human body detection, tracking and motion analysis in multi-camera environments

University of Surrey

Automated Audio-Visual Analysis Work on recognition of activities in the context of sports

videos and visual surveillance. We are concentrating on 2-D analysis, using shape and motion cues.

Also work on 3-D representations for human activity recognition.

We have available a public domain (LGPL) C++ library that includes a good framework for integrating different types of video sources & outputs

Example

FORTH

FORTH has developed a hand detector and tracker which:

– Handles multiple, potentially occluding blobs– Supports detection of the fingers of hands– Provides 3D information for the contours of the tracked blobs– Operates with potentially moving cameras– Robust performance under considerable illumination changes– Real time performance (>30fps)– Has already been employed in many applications (cognitive

interpretation of human activities, a prototype human-computer interaction system, landmark detection in robot navigation experiments, etc)

FORTH - Example

FORTH

• On-going and future research activities:• Investigation of the use of additional cues (motion,

shape, etc) for model-based human motion detection and tracking

• Development of inference mechanisms to handle missing parts and uncertain detection estimates

• Research in gesture recognition and human activity interpretation

E-team possible cooperation

Main outcome of e-teams: joint research papers! Ideas:

– Extend methods developped for single camera to multiple-camera applications

– Exchange databases / applications– Extend systems using tools from other groups– Create sub-groups for:

Body detection Body tracking Object classification Gesture analysis

E-team possible cooperation

How?– Software exchanges (executables, code, …)– Students visits

Two weeks, financed by MUSCLE A few months, with student grants Create a list: who wants to host or send someone in a very

specific subject

– For every sub-group publish on the Muscle web the on-going collaborations (title, partners, results…)

Face detection and recognitionE-TEAM

Participants (3):

• ICG, AUTH, UPC

E-Team Leader: Not decided yet (M.Pardàs, C. Cantón) (UPC)

Note: If this E-TEAM is too small it could be embedded into the Body E-TEAM

ICG

Researchers: Horst Bischoff Face detection, tracking and recognition based

on local orientation histograms On-line Adaboost algorithm as an algorithm to

cope with all this tasks Real time operation

AUTH

Researchers: Ionnis Pitas, Nikos Nikolaidis Expertise in face detection, tracking and

verification based on several detection techniques developed for greyscale and color images.

Techniques based on morphological elastic graph matching.

UPC

Researchers: Ferran Marqués, Verónica Vilaplana

Measures

Measures Classifier

Perceptual Model

Basic Descriptors

Basic Descriptor #1Basic Descriptor #2

…

Basic Descriptor #M

Basic Descriptors

Basic Descriptor #1Basic Descriptor #2

…

Basic Descriptor #M

Perceptual Model

Shape Descriptor

Shape Descriptor

Specific Descriptors

Specific Descriptor #1

Specific Descriptor #2

…

Specific Descriptor #N

Specific Descriptors

Specific Descriptor #1

Specific Descriptor #2

…

Specific Descriptor #N

BPTRegion

BPTRegion

Final Decision

Node Extension

Node Extension

Measures&

Soft Classifiers

Measures&

Soft Classifiers

Hard Classifiers

Hard Classifiers

Frontal Face: Perceptual Model • Candidate selection (in maroon):

– Non-complete representation of the object.

• Shape descriptor (in orange):– Union of regions that may not be linked in the BPT.

• Candidate selection (in maroon):– Non-complete representation of the object.

• Shape descriptor (in orange):– Union of regions that may not be linked in the BPT.