Image Processing and Analysis: Applications and Trendstavares/downloads/publications/comunicacoes/... · Image Processing and Analysis: Applications and Trends ... Face Images , CompIMAGE

Image Processing and Analysis: Applications and Trends

Joo Manuel R. S. Tavares

[email protected] www.fe.up.pt/~tavares

AES ATEMA 2010 (Le Quebec, CANADA) Int. ConferenceMontreal & Quebec City, CANADA

June 27 to July 03 , 2010on

Advances and Trends in Engineering Materialsand their Applications

Outline

1. Introduction

2. Segmentation

3. Motion Tracking

4. Analysis of Objects: Matching, Registration and Morphing

5. 3D Reconstruction

6. Conclusions

7. Research Team

2Image Processing and Analysis: Applications and TrendsJoo Manuel R. S. Tavares

Introduction

The researchers of Computational Vision aim the development of algorithms to perform in fully or semi-automatically manner operations and tasks carry out by the (quite complex) humans vision system

4Image Processing and Analysis: Applications and Trends

Original images

Azevedo et al. (2010), Three-dimensional reconstruction and characterization of human external shapes from two-dimensional images using volumetric methods, Computer Methods in Biomechanics and Biomedical Engineering 13(3): 359-369

Computational 3D model built voxelized and poligonized

Introduction


Image processing and analysis are topics of the most importance for our Society

Algorithms of image processing and analysis are frequently used, for example, in:

Medicine Biology Industry Natural Sciences Engineering

Examples of common tasks involving algorithms of image processing and analysis are:

noise removal geometric correction segmentation, recognition (2D-4D) motion tracking and analysis, including matching, registration and morphing (2D-4D) 3D reconstruction


Introduction


Introduction: Usual Computational Pipeline for Image Processing and Analysis

Image Processing and Analysis: Applications and Trends 6

Image(s) enhancement

Image(s) segmentation / features extraction

tracking

matching

morphing

Image(s)

motionanalysis registration

image processing

image analysis /computational

visionJoo Manuel R. S. Tavares

3D vision

computer vision

Segmentation

Segmentation It is intended to identify in a full or semi- automatic manner

objects (2D/3D) presented in static images or in image sequences

The most usual methodologies are based on template matching, statistical, geometric or physical modeling, or neuronal networks

It is one of the most usual operations involved in the computational analysis of objects from images, and very often it is the first important step of image processing and analysis

Frequent problems: noise, low resolution, reduce contrast, shapes not previously known, occlusion, multiple objects, etc.

Joo Manuel R. S. Tavares Image Processing and Analysis: Applications and Trends 8

Segmentation Example: segmentation of contours in dynamic

pedobarography (Otsu method, morphologic operators)


Original images After segmentation

Bastos & Tavares (2004), Improvement of Modal Matching Image Objects in Dynamic Pedobarography using Optimization Techniques, Lecture Notes in Computer Science 3179:39-50

camera mirror

contact layer + glass

reflected light glass

pressure opaque layer

lamp

lamp transparent layer

Segmentation Example: analysis of damage in composite materials

(binarization and region analysis)


Original image After segmentation

Damage areaMeasures obtained

Marques et al. (2009), Delamination Analysis of Carbon Fibre Reinforced Laminates: Evaluation of a Special Step Drill , Composites Science and Technology, 69(14): 2376-2382

Segmentation Example: hardness evaluation from indentation images

(Johannsen and Bille threshold, region growing)


Vickers hardness Brinell hadnessFilho et al. (2010), Brinell and Vickers Hardness Measurement using Image Processing and Analysis Techniques, Journal of Testing and Evaluation 38(1):88-94

Segmentation Example: analysis of material microstructures (neuronal

network)


Original images After segmentationAlbuquerque et al. (2008), A New Solution for Automatic Microstructures Analysis from Images Based on a Backpropagation Artificial Neural Network, Nondestructive Testing and Evaluation 23(4):273-283

Segmentation Example: evaluation of nickel alloy secondary phases from

Scanning Electron Microscopic images (neuronal network)


Original image Image segmented

Albuquerque et al. (2010), Automatic Evaluation of Nickel Alloy Secondary Phases from SEM Images, Microscopy Research and Technique, doi: 10.1002/jemt.20870 (in press)

Segmentation Example: evaluation on synthetic material porosity from

optical microscopic images (neuronal network)


Original images withtraining pixels

Image segmented

Albuquerque et al. (2010), New computational solution to quantify synthetic material porosity from optical microscopic images, Journal of Microscopy, doi: 10.1111/j.1365-2818.2010.03384.x (in press)

Segmentation Example: control of a servomechanism by

gesture recognition (orientation histograms)


Tavares et al. (2005), Control a 2-Axis Servomechanism by Gesture Recognition using a GenericWebCam, International Journal of Advanced Robotic Systems 2(1):39-44

Running mode

Conversion for 256 gray levels

Orders orientation histogram vectors

initialization (stored in memory)

1 2 3 4

Orders images acquisition (one by one)

Command image

acquisition Conversion for 256

gray levels

Comparison of the images orientation histogram vector

with the stored vectors of the preset orders images

Associated command

order

1 3 4

1 2 4

1 2 3

2 3 4

4

2

1

Learning mode

Segmentation Example: detection of breast tumours from mammography

images (Hough transform)


Chagas et al. (2007), An Application of Hough Transform to Identify Breast Cancer in Images, VIPimage 2007, 363-368

Original image After segmentation

Segmentation Example: object recognition (image template matching)


Carvalho & Tavares (2005), Metodologias para identificao de faces em imagens: Introduo e exemplos de resultados, CMNI 2005

fft fft

ift

( )3ift D CC( )2ift D CC

max CC

Original imageTemplate image

Segmentation Example: segmentation of facial features

(deformable geometric template)


Carvalho & Tavares (2006), Two Methodologies for Iris Detection and Location in Face Images, CompIMAGE 2006, 129-134Carvalho & Tavares (2007), Eye detection using a deformable template in static images, VipIMAGE 2007, 209-215

Original image and associated energy fields

Segmentation of the iris using adeformable template (a circle)

Segmentation of an eye using an

deformable template

Segmentation Example: segmentation of skin regions in images (statistical

model)


Skin samples used to build the statistical model Original image and

segmentation obtained

Carvalho & Tavares (2005), Metodologias para identificao de faces em imagens: Introduo e exemplos de resultados, CMNI 2005

Probably functionused


Segmentation Example: segmentation of scene background/foreground in

image sequences (statistical models)

Backgroundsubtraction

Foreground objectdetection

Vasconcelos & Tavares (2008), Image Segmentation for Human Motion Analysis: Methods and Applications, WCCM8 / ECCOMAS 2008

Original images

Segmentation Example: segmentation of faces and hands in images

(Active Shape Model)


Segmentations achieved (initial, intermediate and final steps)Vasconcelos & Tavares (2008), Methods to Automatically Built Point Distribution Models for Objects like Hand Palms and Faces Represented in Images, Computer Modeling in Engineering & Sciences 36(3):213-241

Segmentation Example: segmentation of faces in images (Active Appearance

Model)


Vasconcelos & Tavares (2008), Methods to Automatically Built Point Distribution Models for Objects like Hand Palms and Faces Represented in Images, Computer Modeling in Engineering & Sciences 36(3):213-241

Segmentations achieved (initial, intermediate and final steps)

Segmentation Example: analysis of the vocal tract shape during speech

production from MRI (Active Shape Models)


Vasconcelos et al. (2010), Towards the Automatic Study of the Vocal Tract from Magnetic Resonance Images, Journal of Voice, doi:10.1016/j.jvoice.2010.05.002 (in press)

Intermediate segmentation II

Originalimage

Finalsegmentation

Intermediate segmentation I

Segmentation Example: analysis of the vocal tract shape during speech

production from MRI (Active Appearance Models)


Vasconcelos et al. (2010), Towards the Automatic Study of the Vocal Tract from Magnetic Resonance Images, Journal of Voice, doi:10.1016/j.jvoice.2010.05.002 (in press)

Intermediate segmentations

Initialsegmentation

Finalsegmentation

Intermediate segmentations

Segmentation Example: segmentation of medical images (active contours -

snakes)


Original image andinitial contour

Final contour

Tavares et al. (2009), Computer Analysis of Objects Movement in Image Sequences: Methods and Applications, International Journal for Computational Vision and Biomechanics 2(2): 209-220

Segmentation Example: segmentation of objects in images (deformable

contour, FEM, Lagrange equation)


Original images and initial contours Final contoursGonalves et al. (2008), Segmentation and Simulation of Objects Represented in Images using Physical Principles, Computer Modeling in Engineering & Sciences 32(1):45-55

rubberk = 200N/m14s

Segmentation Example: segmentation of medical images (level-set method)


Original image Initial segmentation Final segmentation

Perdigo et al. (2005), Gerao de modelos de malhas de elementos finitos a partir de imagens mdicas 2D, Encontro_1_Biomecnica, 81-85

Segmentation Example: segmentation of pelvic floor from MRI (level-set

method, prior knowledge, shape influence field)


Ma et al. (2010), A shape guided C-V model to segment the levator ani muscle in axial magnetic resonance images, Medical Engineering & Physics, doi:10.1016/j.medengphy.2010.05.002 (in press)

Pelvic floor segmented

Segmentation Example: new computational framework for medical image

segmentation (VC++, OpenCV, ITK)


Interface of our platform

Ma et al. (2008), Segmentation of Structures in Medical Images: Review and a New Computational Framework, CMBBE2008

Segmentation Example: segmentation of organs from female pelvic cavity

MRI (using our platform)


Ma et al. (2010), A Review of Algorithms for Medical Image Segmentation and their Applications to the Female Pelvic Cavity, Computer Methods in Biomechanics and Biomedical Engineering 13(2):235-246

Region Growing Watershed

Malladis methodGeodesic Active Contour

Motion Tracking

Motion Tracking It is intended to track the motion (and/or the deformation) of

objects along image sequences In this area, the methodologies based on optical flow, block

matching and stochastic methods are widespread Usually, it concerns the estimation of the motion involved,

the management of the features being tracked, and the analysis of the motion tracked as well as its quantification

Usual problems: non-rigid motions, geometric distortions, non-constant illumination, occlusions, noise, multiple objects, etc.


Motion Tracking Computational framework to

track features in image sequences(Kalman Filter or Unscented KalmanFilter, optimization, Mahalanobisdistance, management model)


Pinho & Tavares (2009), Tracking Features in Image Sequences with Kalman Filtering, Global Optimization, Mahalanobis Distance and a Management Model, Computer Modeling in Engineering & Sciences 46(1):51-75

Pinho & Tavares (2009), Comparison between Kalman and Unscented Kalman Filters in Tracking Applications of Computational Vision, VipIMAGE 2009


Motion Tracking Kalman Filter

Optimal recursive Bayesian stochastic method One of its drawbacks is the restrictive assumption of Gaussian

posterior density functions at every time step Many tracking problems involve non-linear motions (i.e. human gait)

Image Processing and Analysis: Applications and Trends 34Joo Manuel R. S. Tavares

Motion Tracking Example: tracking marks in gait analysis (Kalman filter,

Mahalanobis distance, optimization, management model)


Prediction Uncertainty Area Measurement Correspondence Result

Pinho et al. (2005), Human Movement Tracking and Analysis with Kalman Filtering and Global Optimization Techniques, ICCB 2005, 915-926Pinho & Tavares (2009), Tracking Features in Image Sequences with Kalman Filtering, Global Optimization, Mahalanobis Distance and a Management Model, Computer Modeling in Engineering & Sciences 46(1):51-75

(5 frames)



Pinho et al. (2007), Efficient Approximation of the Mahalanobis Distance for Tracking with the Kalman Filter, International Journal of Simulation Modelling 6(2):84-92

(547 frames)

Pinho et al. (2005), A Movement Tracking Management Model with Kalman Filtering, Global Optimization Techniques and Mahalanobis Distance, LSCCS, Vol. 4A:463-466

Motion Tracking Example: tracking mice in long image sequences (Kalman

filter, Mahalanobis distance, optimization, management model)


Motion Tracking Unscented Kalman Filter

A set of sigma-points from the distribution of the state vector is propagated through the true non-linearity, and the parameters of the Gaussian approximation are then re-estimated

Addresses the main shortcomings of Kalman Filter, as well as of Extended Kalman Filter, and is more suitable for non-linear motions


Motion Tracking Example: tracking the centre of a square that is moving

according to a non-linear model (Kalman Filter (KF) and Unscented Kalman Filter (UKF))


Motion equations:

0 0

22( 1) 12.51 ,12.51

with 6, 10

x x ii iy xi i

x y

= + += +

= =

#6

+ predictionsx measurementsx corrections

Kalman Filter results:


(8 frames)

Motion Tracking Example: tracking the centre of a square that is moving

according to a non-linear model (Kalman Filter (KF) and Unscented Kalman Filter (UKF)) cont.


Motion equations:

0 0

22( 1) 12.51 ,12.51

with 6, 10

x x ii iy xi i

x y

= + += +

= =

(8 frames)

Tracking error (predicted/real state)


Motion Tracking Example: tracking the motion of three mice in a real image

sequence (Kalman Filter (KF) and Unscented Kalman Filter (UKF))

+ predictionsx measurementsx corrections

(22 frames)




sequence (Kalman Filter (KF) and Unscented Kalman Filter (UKF)) cont.

Kalman Filter results Unscented Kalman Filter results


(22 frames)



sequence (Kalman Filter (KF) and Unscented Kalman Filter (UKF)) cont.


(22 frames)Tracking error (predicted/real state)

Motion Tracking Influence of the adopted filter: Kalman Filter (KF) and

Unscented Kalman Filter (UKF) If the motion is highly nonlinear, then UKF justifies its superior

computational load Otherwise, KF with the undertaken matching (association)

methodology can accomplish efficiently the tracking Hence, the decision between KF or UKF is application-dependent

Frequently, UKF gets superior results However, when the computational load is somewhat constrained, KF

with a suitable matching strategy can be a good tracking solution


Analysis of Objects:Matching, Registration and

Morphing

Analysis of Objects Matching

It is regularly used in the computational analysis of objects from images, for example, to register (i.e. align) objects, recognize objects, attain 3D information, analyze the motion tracked, and so forth

Generally, it is achieved by considering invariant objects characteristics, as curvature, or displacements in a global space (like in modal space)

Common problems: occlusion, non-rigid deformations, high shape variations, etc.


Analysis of Objects Registration

It is commonly required in order to compare objects in images acquired at different time instants or according to distinct conditions

It is essential, for example, in Medicine to follow up the evaluation of patients diseases from images

Usually, it is achieved by considering objects characteristic features, as points of maximum curvature, and their matching followed by the estimation of the involved transformation

Common problems: key and invariant features not easily identified, occlusion, non-rigid deformations, severe shape variations, etc.


Analysis of Objects Morphing (i.e. simulation)

It is an especially used in Computer Graphics, but also very useful in the analysis of objects from images, for example, to estimate the deformation involved between two objects or between two configurations of an object, to simulate the transitions between two shapes acquired under a high temporal gap, etc.

Normally, it is attained by considering simple geometric transformations

However, when it must be considered the real behavior of the objects, physical methodologies and modeling as, for example, FEM, should be considered

Common difficulties are related with the estimation of the involved forces and with the properties of the adopted (virtual) material

The adequate matching of the objects is crucial


Matching Using physical or geometrical modeling and modal matching


Modeling(physical or geometrical)

Eigenvalues / eigenvectors computation

Matching matrix

assembly

Contour 1

Contour 2

Matches achievement(optimization)

Modeling(physical or geometrical)

Eigenvalues / eigenvectors computation

Bastos & Tavares (2006), Matching of Objects Nodal Points Improvement using Optimization, Inverse Problems in Science and Engineering 14(5):529-541


Example: matching contours in dynamic pedobarography (FEM, modal matching, optimization)

Matching


Original images Matched contours

Bastos & Tavares (2004), Improvement of Modal Matching Image Objects in Dynamic Pedobarography using Optimization Techniques, Lecture Notes in Computer Science 3179:39-50

camera mirror

contact layer + glass

reflected light glass

pressure opaque layer

lamp

lamp transparent layer

Tavares & Bastos (2010), Improvement of Modal Matching Image Objects in Dynamic Pedobarography using Optimization Techniques, Progress in Computer Vision and Image Analysis, Chapter 19, 339-368


Matching Example: matching contours and surfaces in dynamic

pedobarography (FEM, modal analysis, optimization)


Image of dynamic pedobarography

Tavares & Bastos (2005), Improvement of Modal Matching Image Objects in Dynamic Pedobarography using Optimization Techniques, Electronic Letters on Computer Vision and Image Analysis 5(3):1-20

Matching foundbetween two contours

Matching found between two intensity (pressure) surfaces (2 views)

Matching found between iso-contours (2 views)


Registration Registration of contours in images (geometrical modeling,

optimization, dynamic programming)


Oliveira & Tavares (2008), Algorithm of dynamic programming for optimization of the global matching between two contours defined by ordered points, Computer Modeling in Engineering & Sciences 31(11):1-11


Registration Example: registration of contours in images (geometrical

modeling, optimization, dynamic programming)


Original images and contours

Matched contours before registration

Matched contours after registration

Oliveira & Tavares (2009), Matching Contours in Images through the use of Curvature, Distance to Centroidand Global Optimization with Order-Preserving Constraint, Computer Modeling in Engineering & Sciences 43(1):91-110


Registration Example: registration of images in pedobarography

(geometrical modeling, optimization, dynamic programming)


Original images and contours Contours and images before and after registration

Oliveira et al. (2009), Rapid pedobarographic image registration based on contour curvature and optimization, Journal of Biomechanics 42(15):2620-2623Joo Manuel R. S. Tavares

Registration Example: registration of images in pedobarography (Fourier

transform)


Original images Images before andafter registration

Oliveira et al. 2010, Registration of pedobarographic image data in the frequency domain, Computer Methods in Biomechanics and Biomedical Engineering (in press)


Registration Example: registration of images in pedobarography (Hybrid

method: Contours registration or Fourier transform based registration + Optimization of a Similarity Measure MSE, MI or XOR)


Original images Images before andafter registration

Oliveira & Tavares 2010, Novel Framework for Registration of Pedobarographic Image Data, Medical & Biological Engineering & Computing (submitted)


Registration Registration of image sequences in dynamic

pedobarography (spatial and temporal registration)


Oliveira & Tavares 2010, Spatio-temporal Registration of Pedobarographic Image Sequences, Journal of Biomechanics (submitted)

Registration Example: registration of image sequences in dynamic

pedobarography (spatial and temporal registration)

Image Processing and Analysis: Applications and Trends

57


Original sequencesbefore registration

Preprocessed sequences

Original image sequences

Sequencesafter registration

57

Morphing


Physical morphing/simulation of contours in images (FEM, modal analysis, optimization, Lagrange equation)


Example: morphing contours in images (FEM, modal analysis, optimization, Lagrange equation)

Matching found

Deformations estimated

Morphing


Gonalves et al. (2008), Segmentation and Simulation of Objects Represented in Images using Physical Principles, Computer Modeling in Engineering & Sciences 32(1):45-55

Original images


3D Reconstruction

3D Reconstruction It is intended to achieve the 3D reconstruction of objects or

scenes from images In this area, the following methodologies are common:

external shapes active techniques (with energy projection or relative motion), passive techniques (without energy projection nor relative motion) and of space carving; inner shapes 2D segmentation (contours, for example) and data interpolation

Usually, it involves tasks of camera calibration, data segmentation, matching, triangulation and interpolation

Common problems: geometric distortions, bad or unstable illumination, occlusion, noise, multiple objects, complex shapes and topologies, etc.


3D Reconstruction Example: 3D reconstruction of organs from medical images

(segmentation 2D, marching cubes, Delaunay)


Segmentation done in a 2D slice and the 3D reconstruction

obtained

3D reconstruction of some structures of an arm

Perdigo et al. (2005), Sobre a Gerao de Malhas Tridimensionais para fins Computacionais a partir de Imagens Mdicas, CMNI 2005

3D Reconstruction Example: 3D reconstruction of organs from medical images

(segmentation 2D, loft, smooth, Delaunay)


Segmentation donein a 2D slice

Pelvic floorreconstructed

Reconstructed organs from a pelvic

cavityPimenta et al. (2006), Reconstruction of 3D Models from Medical Images: Application to Female Pelvic Organs, CompIMAGE 2006, 343-348Alexandre et al. (2007), 3D reconstruction of pelvic floor for numerical simulation purpose, VipIMAGE2007, 359-362

slices

3D Reconstruction Example: 3D reconstruction of a scene using a technique

of active vision (dense stereo vision)


Disparity mapobtained

Original image pair

Azevedo et al. (2006), Development of a Computer Platform for Object 3D Reconstruction using Active Vision Techniques, VISAPP 2006, 383-388

3D Reconstruction Example: 3D reconstruction of objects by space carving


Azevedo et al. (2008), 3D Object Reconstruction from Uncalibrated Images using an Off-the-Shelf Camera, Advances in Computational Vision and Medical Image Processing: Methods and Applications, 117-136

Original images Computational 3D model built voxelizedand poligonized

3D Reconstruction Example: 3D reconstruction of objects by space carving


Original images Computational 3D model built voxelizedand poligonized

Azevedo et al. (2010), Three-dimensional reconstruction and characterization of human external shapes from two-dimensional images using volumetric methods, Computer Methods in Biomechanics and Biomedical Engineering 13(3): 359-369

Conclusions

Conclusions

The area of image processing and analysis is very complex and demand, but of raised importance in many domains

Numerous hard challenges exist, as for example, adverse conditions in the image acquisition process, occlusion, objects with complicate shapes, with topological variations or undergoing complex motions

Considerable work has already been developed, but important and complex goals still to be reached

Methods and methodologies of other research areas, as of Mathematics, Computational Mechanics, Medicine and Biology, can contribute significantly for their reaching

For that, collaborations are welcome


Research Team(Computational Vision)

Research Team (Computational Vision)

PhD students (15): In course: Raquel Pinho, Patrcia Gonalves, Maria

Vasconcelos, Ilda Reis, Teresa Azevedo, Daniel Moura, Zhen Ma, Elza Chagas, Victor Albuquerque, Francisco Oliveira, Eduardo Ribeiro, Antnio Gomes, Joo Nunes, Alex Araujo, Sandra Rua

MSc students (13): In course: Carlos Faria, Elisa Barroso, Ana Jesus, Veronica

Marques, Diogo Faria Finished: Daniela Sousa, Francisco Oliveira, Teresa Azevedo,

Maria Vasconcelos, Raquel Pinho, Lusa Bastos, CndidaCoelho, Jorge Gonalves

BSc students (2) Finished: Ricardo Ferreira, Soraia Pimenta


Image Processing and Analysis:Applications and Trends


[email protected] www.fe.up.pt/~tavares

Slide Number 1OutlineIntroductionIntroductionIntroductionIntroduction: Usual Computational Pipeline for Image Processing and AnalysisSegmentationSegmentationSegmentationSegmentationSegmentationSegmentationSegmentationSegmentationSegmentationSegmentationSegmentationSegmentationSegmentationSegmentationSegmentationSegmentationSegmentationSegmentationSegmentationSegmentationSegmentationSegmentationSegmentationSegmentationMotion TrackingMotion TrackingMotion TrackingMotion Tracking Motion TrackingMotion TrackingMotion TrackingMotion TrackingMotion TrackingMotion TrackingMotion TrackingMotion TrackingMotion TrackingAnalysis of Objects:Matching, Registration and MorphingAnalysis of ObjectsAnalysis of ObjectsAnalysis of ObjectsMatchingMatchingMatchingRegistrationRegistrationRegistrationRegistrationRegistrationRegistrationRegistrationMorphingMorphing3D Reconstruction3D Reconstruction3D Reconstruction3D Reconstruction3D Reconstruction3D Reconstruction3D ReconstructionConclusionsConclusionsResearch Team(Computational Vision)Research Team (Computational Vision)Slide Number 71

Image Processing and Analysis: Applications and Trendstavares/downloads/publications/comunicacoes/... · Image Processing and Analysis: Applications and Trends ... Face Images , CompIMAGE

Documents