01.AdvancesinPatternRecognition

PROCEEDINGS OF THE SIXTH INTERNATIONAL CONFERENCE ON

Illllllllll Illllllllll

P A T T E R N RECOGNIT ION

E D I T O R P I N A K P A N I P A L

!

I Hill II P A T T E I! N R E C O G N I T I O N

PROCEEDINGS OF THE SIXTH INTERNATIONAL CONFERENCE ON

Mil II Mill

I Mill II P A T T E R N R E C O G N I T I O N Ind ian S ta t i s t i ca l I ns t i t u t e , Ko lka ta . India

2 - A Janua ry 2007

I T 0 I P I N A K P A N I P A L

Indian Statistical Institute, India

^jjp World Scientific NEW JERSEY • LONDON • SINGAPORE • BEIJING • SHANGHAI • H O N G K O N G • TAIPEI • CHENNAI

Published by

World Scientific Publishing Co. Pte. Ltd.

5 Toh Tuck Link, Singapore 596224

USA office: 27 Warren Street, Suite 401-402, Hackensack, NJ 07601

UK office: 57 Shelton Street, Covent Garden, London WC2H 9HE

British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library.

ADVANCES IN PATTERN RECOGNITION Proceedings of the Sixth International Conference on Advances in Pattern Recognition (ICAPR 2007)

Copyright © 2007 by World Scientific Publishing Co. Pte. Ltd.

All rights reserved. This book, or parts thereof, may not be reproduced in any form or by any means, electronic or mechanical, including photocopying, recording or any information storage and retrieval system now known or to be invented, without written permission from the Publisher.

For photocopying of material in this volume, please pay a copying fee through the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA. In this case permission to photocopy is not required from the publisher.

ISBN-13 978-981-270-553-2 ISBN-10 981-270-553-8

Printed in Singapore by B & JO Enterprise

PREFACE

The Electronics and Communication Sciences Unit (ECSU) of the Indian Statistical Institute is organizing the sixth International Conference on Advances in Pattern Recognition (ICAPR 2007) at the Indian Statistical Institute, Kolkata, from 2 n d to 4 t h January, 2007. Since the advent of knowledge based computing paradigm, pattern recognition has become an active area of research involving scientists and engineers from different disciplines of the physical and earth sciences. A number of conferences are being organized every year, which act as platforms to present and exchange ideas on different facets of pattern recognition. It is needless to mention that ICAPR has carved out a unique niche within this list of conferences on pattern recognition, particularly for its continued success in focusing upon application-driven research. We are confident that the programme of this ICAPR will be as exciting as the previous ones.

You may be aware of the overwhelming response that we received since the publication of the call for papers for ICAPR 2007 in February 2006. We received 123 papers from 32 different countries. Given the constraint on time like any other three-day conference, it was indeed difficult for us to select a very few out of these high-quality technical contributions. I am thankful to the learned members of the programme committee whose untiring effort helped me to ultimately select a total of 68 papers for oral presentation. The selected papers represent a number of important frontiers in pattern recognition ranging from Biometrics, Document Analysis, Image Registration & Transmission to traditional areas like Image Segmentation, Multimedia Object Retrieval, Shape Recognition and Speech & Signal Analysis. I am happy that we shall see an excellent balance of theory and application focused research in the programmes of ICAPR 2007. Another important aspect of the programme will be a series of invited talks by renowned exponents in the field of pattern recognition and related areas. We look forward to listening to plenary speakers Prof. K. V. Mardia, Prof. T. Tan and Prof. E. J. Delp. I am confident that it would also be a rewarding experience for all of us to interact with our invited speakers Prof. I. Bloch and Prof. V. Lakshmanan.

Publication of a proceedings of this standard requires tremendous infrastructural support. I am fortunate to have a very active advisory committee who extended their support whenever we required it. The organizing committee is working hard to make the event a grand success. The editorial workload was huge bu t Bibhas Chandra Dhara and Partha Pratim Mohanta, made it easy for me through their hard work and dedication.

I must acknowledge the staffs of the ECSU for their untiring support to conference secretariat. Particularly I am thankful to N. C. Deb, D. K. Gayen and S. K. Shaw for their support to technical work. The administrative responsibilites are being organized by S. K. Seal, S. Sarkar, D. Mitra, R. Chatterjee, D. Shaw, S. S. Das and supported by S. Deb.

I am also thankful to WebReview Team, Katholieke Universiteit Leuven, ESAT/COSIC, to let me use their WebSubmision and WebReview Software. This made our job easier. Of course, the World Scientific editorial team lent a graceful touch to the printed format of this publication. I also acknowledge the help of Subhasis Kumar Pal for maintaining our webserver problem free.

We also thank our sponsors for their kind help and support. I conclude with my heartfelt thanks to the contributors, for submitting their papers, and who now are

ready to present it before the august audience of ICAPR. I am sure that this collection of papers and their presentation will motivate us to explore the further research and the advances made in pattern recognition. Thank you.

Pinakpani Pal Electronics and Communication Sciences Unit Indian Statistical Institute

INTERNATIONAL ADVISORY COMMITTEE

Chairman

Sankar Kumar Pal, India

M e m b e r s

Shun-ichi Amari, Japan Gabriella Sanniti di Baja, Italy

Horst Bunke, Switzerland Bulusu Lakshmana Deekshatulu, India

S.C. Dutta Roy, India Vito Di Gesu, Italy J.K. Ghosh, India

Anil K. Jain, USA Nikola Kasabov, New Zealand

Rangachar Kasturi, USA M. Kunt, Switzerland

C.T. Lin, Taiwan M.G.K. Menon, India

A.P. Mitra, India Heinrich Niemann, Germany

Witold Pedrycz, Canada V.S. Ramamurthy, India

C.R. Rao, USA Erkki Oja, Finland

Lipo Wang, Singapore Jacek M. Zurada, USA

viii

General Chair

D. Dutta Majumder, ISI, Kolkata

Plenary Chair

Nikhil Ranjan Pal, ISI, Kolkata

Tutorial Chair

Bhabatosh Chanda, ISI, Kolkata

Organizing Committee

Arun Kumar De (Chairman), ISI, Kolkata Partha Pratim Mohanta (Convener), ISI, Kolkata

B. D. Acharya, DST, New Delhi Aditya Bagchi, ISI, Kolkata

Bhabatosh Chanda, ISI, Kolkata Bidyut Baran Chaudhuri, ISI, Kolkata Narayan Chandra Deb, ISI, Kolkata Malay Kumar Kundu, ISI, Kolkata

Jharna Majumdar, ADE, Bangalore Dipti Prasad Mukherjee, ISI, Kolkata

C. A. Murthy, ISI, Kolkata Nikhil Ranjan Pal, ISI, Kolkata

Srimanta Pal, ISI, Kolkata S. Rakshit, CARE, Bangalore

Kumar Sankar Ray, ISI, Kolkata Bimal Roy, ISI, Kolkata

S. K. Sarkar, NPL, New Delhi Swapan Kumar Seal, ISI, Kolkata

Bhabani Prasad Sinha, ISI, Kolkata

ix

INTERNATIONAL PROGRAMME COMMITTEE

Abhik Mukherjee, BESU, Shibpur Abraham Kandel, University of South Florida Tampa

Amit Das, BESU, Shibpur Amita Pal, ISI, Kolkata

Amitabha Mukerjee, IIT, Kanpur Anup Basu, University of Alberta

Basabi Chakraborty, Iwate Prefectural University, Japan Bhabatosh Chanda, ISI, Kolkata

Bhargab Bhattacharya, ISI, Kolkata Bidyut Baran Chaudhuri, ISI, Kolkata

Bimal Roy, ISI, Kolkata Brian C. Lovell, The University of Queensland, Australia

C. A. Murthy, ISI, Kolkata C. V. Jawahar, HIT, Hyderabad

Dipti Prasad Mukherjee, ISI, Kolkata Hisao Ishibuchi, Osaka Prefecture University, Japan

Irina Perfiljeva, University of Ostrava, Czech Republic Isabelle Bloch, ENST, France

Jayanta Mukhopadhyay, IIT, Kharagpur Koczy T. Laszlo, Hungary

Kumar Shankar Ray, ISI, Kolkata Lipo WANG, Nanyang Technological University, Singapore

Malay Kundu, ISI, Kolkata Mrinal Mondal, University of Alberta, Canada

Nikhil Ranjan Pal, ISI, Kolkata Niladri Chatterjee, IIT, Delhi

Okyay Kaynak, Bogazici University, Turkey Olli Simula, Helsinki University of Technology, Finland Oscar Castillo, Tijuana Institute of Technology, Mexico

Punam Saha, University of Pennsylvania, USA Ryszard S. Choras, Institute of Telecommunications, Poland

Sanjoy Saha, JU, Kolkata Sansanee Auephanwiriyakul, Chiang Mai University, Thailand

Scott Acton, University of Virginia, USA Sid Ray, Monash University, Australia

Somnath Sengupta, IIT, Kharagpur Soo-Young Lee, Korea Advanced Institute of Sc. & Technology, Korea

Subhashis Banerjee, IIT, Delhi Subhasis Choudhury, IIT, Bombay

Sukhendu Das, IIT, Madras Sung-Bae Cho, Yonsei University, Korea

Takeshi Furuhashi, Nagoya University, Japan Visvanathan Ramesh, Siemens Corporate Research Inc., USA

Yutaka Hata, University of Hyogo, Japan Pinakpani Pal (Chairman), ISI, Kolkata

ADDITIONAL REVIEWERS

Aditya Bagchi, ISI, Kolkata Arijit Bishnu, IIT, Kharagpur

Arun K. De, ISI, Kolkata Ashish Ghosh, ISI, Kolkata

Bibhas Chandra Dhara, Jadavpur University Debrup Chakrabarty, CINVESTAV, IPN, Mexico

Durga Prasad Muni, ISI, Kolkata Mandar Mitra, ISI, Kolkata

Nilanjan Ray, University of Alberta, USA Oscar Montiel, Tijuana Institute of Technology, Mexico Patricia Melin, Tijuana Institute of Technology, Mexico

Roberto Sepulveda, Tijuana Institute of Technology, Mexico Somitra Kumar Sanadhya, ISI, Kolkata

Srimanta Pal, ISI, Kolkata Subhamay Maitra, ISI, Kolkata

Swapan Kumar Parui, ISI, Kolkata Umapada Pal, ISI, Kolkata Utpal Garain, ISI, Kolkata

SPONSORS

Adobe

RD INDIA SCIENCE LAB

Reserve Bank of India

CONTENTS

Xlll

Preface v

International Advisory Committee vii

Organizing Committee viii

International Programme Committee ix

Additional Reviewers x

Sponsors xi

Part A Plenary Lecture 1

Why Statistical Shape Analysis is Pivotal to the Modern Pattern Recognition? 3 Kanti V. Mardia

Part B Invited Lectures 13

On the Interest of Spatial Relations and Fuzzy Representations for Ontology-Based Image Interpretation 15 Isabelle Block, Celine Hudelot and Jamal Atif

A Technique for Creating Probabilistic Spatio-Temporal Forecasts 26

V. Lakshmanan and Kiel Ortega

Part C Biometrics 33

An Efficient Measure for Individuality Detection in Dynamic Biometric Applications 35 B. Chakraborty and Y. Manabe

Divide-and-Conquer Strategy Incorporated Fisher Linear Discriminant Analysis: An Efficient Approach for Face Recognition 40

S. Noushath, G. Hemantha Kumar, V. N. Manjunath Aradhya and P. Shivakumara

Ear Biometrics: A New Approach 46 Anupam Sana, Phalguni Gupta and Ruma Purkait

Face Detection using Skin Segmentation as Pre-Filter 51 Shobana L., Anil Kr. Yekkala and Sameen Eajaz

Face Recognition Using Symbolic KDA in the Framework of Symbolic Data Analysis 56 P. S. Hiremath and C. J. Prabhakar

Minutiae-Orientation Vector Based Fingerprint Matching 62 Li-min Yang, Jie Yang and Yong-liang Zhang

Recognition of Pose Varied Three-Dimensional Human Faces Using Structured Lighting Induced Phase Coding 66

Debesh Choudhury

Writer Recognition by Analyzing Word Level Features of Handwritten Documents 73 Prakash Tripathi, Bhabatosh Chanda and Bidyut Baran Chaudhuri

Part D Clustering Algorithms 79

A New Symmetry Based Genetic Clustering Technique for Automatic Evolution of Clusters 81 Sriparna Saha and Sanghamitra Bandyopadhyay

A Non-Hierarchical Clustering Scheme for Visualization of High Dimensional Data 88 G. Chakraborty, B. Chakraborty and N. Ogata

An Attribute Partitioning Approach to Correlation Connected Clusters 93

Vijaya Kumar Kadappa and Atul Negi

Part E Document Analysis 99

A Hybrid Scheme for Recognition of Handwritten Bangla Basic Characters Based on HMM and MLP Classifiers 101

U. Bhattacharya, S. K. Parui and B. Shaw

An Efficient Method for Graphics Segmentation from Document Images 107 S. Mandal, S. P. Chowdhury, A. K. Das and B. Chanda

Identification of Indian Languages in Romanized Form 112 Pratibha Yadav, Girish Mishra and P. K. Saxena

Online Bangla Handwriting Recognition System 117

K. Roy, N. Sharma, T. Pal and U. Pal

Oriya Off-Line Handwritten Character Recognition 123 U. Pal, N. Sharma and F. Kimura

Recognition of Handwritten Bangla Vowel Modifiers 129 S. K. Parui, U. Bhattacharya and S. K. Ghosh

Template-Free Word Spotting in Low-Quality Manuscripts 135 Huaigu Cao and Venu Govindaraju

Unconstrained Handwritten Digit Recognition: Experimentation on MNIST Database 140 V. N. Manjunath Aradhya, G. Hemantha Kumar and S. Noushath

Part F Image Registration and Transmission 145

An Adaptive Background Model for Camshift Tracking with a Moving Camera 147 R. Stolkin, I. Florescu, G. Kamberov

Colour and Feature Based Multiple Object Tracking Under Heavy Occlusions 152 Pabboju Sateesh Kumar, Prithwijit Guha and Amitabha Mukerjee

DCT Properties as Handle for Image Compression and Cryptanalysis 157 Anil Kr. Yekkala, C. E. Veni Madhavan and Narendranath Udupa

Genetic Algorithm for Improvement in Detection of Hidden Data in Digital Images 164 Santi P. Maity, Prasanta K. Nandi and Malay K. Kundu

High Resolution Image Reconstruction from Multiple UAV Imagery 170 Jharna Majumdar, B. Vanathy and Lekshmi S.

Image Registration and Object Tracking via Affine Combination 175 Nilanjan Ray and Dipti Prasad Mukherjee

Progressive Transmission Scheme for Color Images Using BTC-PF Method 180 Bibhas Chandra Dhara and Bhabatosh Chanda

Registration Algorithm for Motion Blurred Images 186 K. V. Arya and P. Gupta

Part G Image Segmentation 191

Aggregation Pheromone Density Based Change Detection in Remotely Sensed Images 193 Megha Kothari, Susmita Ghosh and Ashish Ghosh

Automatic Brain Tumor Segmentation Using Symmetry Analysis and Deformable Models 198 Hassan Khotanlou, Olivier Colliot and Isabelle Bloch

Edge Recognition in MMWave Images by Biorthogonal Wavelet Decomposition and Genetic Algorithm 203

C. Bhattacharya and V. P. Dutta

Extended Markov Random Fields for Predictive Image Segmentation 208

R. Stolkin, M. Hodgetts, A. Greig and J. Gilby

External Force Modeling of Snakes Using DWT for Texture Object Segmentation 215 Surya Prakash and Sukhendu Das

I-FISH: Increasing Detection Efficiency for Fluorescent Dot Counting in Cell Nuclei 220 Shishir Shah and Fatima Merchant

Intuitionistic Fuzzy C Means Clustering in Medical Image Segmentation 226 T. Chaira, A. K. Ray and 0. Salvetti

Remote Sensing Image Classification: A Wavelet-Neuro-Fuzzy Approach 231 Saroj K. Meher, B. Uma Shankar and Ashish Ghosh

Part H Multimedia Object Retrieval 237

An Efficient Cluster Based Image Retrieval Scheme Using Localized Texture Pattern 239

Saumen Mandal, Sanjoy Kumar Saha, Amit Kumar Das and Bhabatosh Chanda

Feature Selection Based on Human Perception of Image Similarity for Content Based Image Retrieval 244 P. Narayana Rao, Chakravarthy Bhagvati, R. S. Bapi, Arun K. Pujari and B. L. Deekshatulu

Identification of Team in Possession of Ball in a Soccer Video Using Static and Dynamic Segmentation 249 V. Pallavi, Jayanta Mukherjee, A. K. Majumdar, Shamik Sural

Image Retrieval Using Color, Texture and Wavelet Transform Moments 256

R. S. Choras

Integrating Linear Subspace Analysis and Iterative Graphcuts For Content-Based Video Retrieval 263 P. Deepti, R. Abhilash and Sukhendu Das

Organizing a Video Database Around Concept Maps 268 K. Shubham, L. Dey, R. Goyal, S. Gupta and S. Chaudhury

Statistical Bigrams: How Effective Are They in Text Retrieval? 274 Prasenjit Majumder, Mandar Mitra and Kalyankumar Datta

Part I Pattern Recognition 279

Adaptive Nearest Neighbor Classifier 281 Anil K. Ghosh

Class-Specific Kernel Selection for Verification Problems 285 Ranjeeth Kumar and C. V. Jawahar

Confidence Estimation in Classification Decision: A Method for Detecting Unseen Patterns 290

Pandu R Devarakota and Bruno Mirbach

ECG Pattern Classification Using Support Vector Machine 295 S. S. Mehta and N. S. Lingayat

Model Selection for Financial Distress Classification 299 Srinivas Mukkamala, Andrew H. Sung, Ram B. Basnet, Bemadette Ribeiro and Aarmando S. Vieira

XV11

Optimal Linear Combination for Two-Class Classifiers 304 0. Ramos Terrades, S. Tabbone and E. Valveny

Support Vector Machine Based Hierarchical Classifiers for Large Class Problems 309 Tejo Krishna Chalasani, Anoop M. Namboodiri and C. V. Jawahar

Unsupervised Approach for Structure Preserving Dimensionality Reduction 315 Amit Saxena and Megha Kothari

Part J Shape Recognition 319

A Beta Mixture Model Based Approach to Text Extraction from Color Images 321 Anandarup Roy, Swapan Kumar Parui and Utpal Roy

A Canonical Shape-Representation for a Polygon 327 Sukhamay Kundu

A Framework for Fusion of 3D Appearance and 2D Shape Cues for Generic Object Recognition 332

Manisha Kalra and Sukhendu Das

Constructing Analyzable Models by Region Based Technique for Object Category Recognition 338 Yasunori Kamiya, Yoshikazu Yano and Shigeru Okuma

DRILL: Detection and Representation of Isothetic Loosely Connected Components without Labeling 343

P. Bhowmick, A. Biswas and B. B. Bhattacharya

Pattern Based Bootstrapping Method for Named Entity Recognition 349

A sif Ekbal and Sivaji Bandyopadhyay

SCOPE: Shape Complexity of Objects using Isothetic Polygonal Envelope 356

Arindam Biswas, Partha Bhowmick and Bhargab B. Bhattacharya

Segmental K-Means Algorithm Based Hidden Markov Model for Shape Recognition and its Applications 361

Tapan Kumar Bhowmik, Swapan Kumar Parui, Manika Kar and Utpal Roy

Part K Speech and 1-D Signal Analysis 367

Automatic Continuous Speech Segmentation Using Level Crossing Rate 369

Nagesha and G. Hemantha Kumar

Automatic Gender Identification Through Speech Analysis 375 Anu Khosla and Devendra Kumar Yadav

Error-Driven Robust Particle Swarm Optimization for Fuzzy Rule Extraction and Structure Estimation 379

Sumitra Mukhopadhyay and Ajit K. Mandal

xvm

HMM Based POS Tagger and Rule-Based Chunker for Bengali 384 Sivaji Bandyopadhyay and Asif Ekbal

Non-Contemporary Robustness in Text-Dependent Speaker-Recognition Using Multi-Session Templates in an One-Pass Dynamic-Programming Framework 391

V. Ramasubramanian, V. Praveen Kumar and S. Thiyagarajan

Some Experiments on Music Classification 396 Debrup Chakraborty

Text Independent Identification of Regional Indian Accents in Spoken Hindi 401 Kamini Malhotra and Anu Khosla

Part L Texture Analysis 405

An Efficient Approach for Texture Classification with Multi-Resolution Features by Combining Region and Edge Information Using a Modified CSNN 407

Lalit Gupta and Sukhendu Das

Upper Bound in Model Order Selection of MRF with Application in Texture Synthesis 413 Arnab Sinha and Sumana Gupta

Wavelet Features for Texture Classification and Their Use in Script Identification 419

P. S. Hiremath and Shivashankar S.

Author Index

PART A

Plenary Lecture

Why Statistical Shape Analysis is Pivotal to the Modern Pattern Recognition?

Kanti V. Mardia

Department of Statistics University of Leeds,

Leeds, West Yorkshire LS2 9JT, UK E-mail: [email protected]

www. maths, leeds. ac. uk

There have been great strides in shape analysis in this decade. Pattern recognition, image analysis, and morphometries have been the major contributors to this area but now bioinformatics is driving the subject as well, and new challenges are emerging; also the methods of pattern recognition are evolving for bioinformatics. Shape analysis for labelled landmarks is now moving to the new challenges of unlabelled landmarks motivated by these new applications. ICP, EM algorithms, etc. are well used in image analysis but now Bayesian methods are coming into the arena. Dynamic Bayesian networks are other developments. We will discuss the problem of averaging, image deformation, projective shape and Bayesian alignment. The aim of this talk will be to convince the scientists that statistical shape analysis is pivotal to the modern pattern recognition.

Keywords: Bayesian analysis; Bioinformatics; Protein gel; Deformation; Average image; Discrimination; Penalized likelihood.

1. Introduction

We have reviewed the topic over the years starting from two volumes1'2 in 1993, 1994. The subsequent reviews until 2001 include papers3"6 . Since then the subject has grown especially for shapes on manifold, eg. two recent workshops in the USA of the American Institute of Mathematics in 2005 and the Institute of Mathematical Applications in 2006. Also our Leeds Annual Statistical Research (LASR) Workshops ht tp : / /www .maths . leeds .ac .uk/ S ta t i s t i cs /workshop have been keeping abreast of the field especially in relation to shapes and images. An excellent treatment of recent developments in shape analysis including shape manifold can be found in the edited volume of Krim and Yezzi.7 Further stride has been due to its new connections with Bioinformatics - a field bursting with challenges. The field of shape analysis as

'covered until 1998 by Diyden and Mardia8 has domi-nated mainly by labelled shape analysis. New innovations are now emerging in unlabelled shape analysis. Perhaps Cross and Hancock9 is one of the early statistical papers in the image area via EM-algorithm. Glasbey and Mardia10 gave some different perspectives through penalized likelihood methods for images. A cross-over to Bioinformatics can also be seen for example in Richmond et al.11

A Bayesian hierarchical model for unlabelled shape is proposed in Green and Mardia12 which has

not tried images yet (only bioinformatics). Mardia et al.13 have given a hybrid approach for image deformation and discrimination where some landmarks are labelled. One of the tools for deformation has been through a part of thin plate spline (TPS) but many other radial functions can be used. Mardia et al.14 have shown how TPS gives advantages over various radial functions using Brodatz type texture images. Thus, it is important to distinguish between labelled and unlabelled configurations, finite and 'infinite' number of points, outline or solid shape, linear or nonlinear transformations, parametric or non-parametric methods, and so on. We now describe some special topics.

2. Labelled Shape Analysis

Consider a configuration of points in M.m. For pattern recognition, applications generally m = 2 or 3. "Shape" deals with the residual-structure of this configuration when certain transformations are filtered out. More specifically, the shape of a configuration consists of its equivalence class under a group of transformations. Important groups for machine vision are the similarity group, the affine group and the projective group. Here the group action describes the way in which an image is captured. For instance if two different images of the same scene are obtained using a pinhole camera, the corresponding transformation between the two images is the composition of two central projections, which is a projective trans-

4 Why Statistical Shape Analysis is Pivotal to the Modern Pattern Recognition?

formation. If the two central projections can be approximated by parallel projections, which is the case of remote views of the same planar scene, the projective transformation can be approximated by an affine transformation. Further, if these parallel projections are orthogonal projections on the plane of the camera, this affine transformation can be approximated by a similarity transformation. Therefore, the relationships between these shapes as follows: if two configurations have the same similarity shape then they automatically have the same affine shape; if they have the same affine shape they will have the same projective shape. For example, two squares of different sizes have the same similarity, affine and projective shape whereas a square and a rectangle have the same affine and projective shape but not the same similarity shape. On the other hand, a square and a kite have the same projective shape but not the same affine shape.

The word "shape" often refers in statistics to similarity shape where only the effects of translation, scale and rotation have been filtered out (see for example, Dryden and Mardia,8). In recent years, substantial progress has been made in similarity shape analysis since appropriate shape space (e.g. Kendall's space) and shape coordinates (e.g., Bookstein coordinates) have been available. A simple example of Bookstein coordinates is for the shape of a triangle where the shape coordinates are obtained after taking one of the vertices as the origin and rotating the triangle so that the base of the triangle lies on the x-axis, and then rescaling the base to the unit size. The motivation behind such coordinate systems is similar to those in directional statistics where to analyze spherical data one requires a coordinate system such as longitude and latitude (see for example, Mardia and Jupp,15). Similar type of coordinates are avail-able for affine shape (Goodall and Mardia,16). For affine shape in 2-D, we can obtain shape coordinates by using three points that determine the direction and the origin of the axes, and the unit length between the points on each of these two axes.

A convenient projective shape space as well as an appropriate coordinate system for this shape space has been put forward by Mardia and Patrangenaru17

where in 2-D, now the coordinate frame consists of four points (0, 0), (0, 1), (1, 0) and (1, 1). This allows reconstruction of three-dimensional image given two-dimensional multiple views of a scene. A "sym

metrical" approach for projective shape space has been given in Kent and Mardia.18

3. Unlabelled Shape Analysis and Bioinformatics

Various new challenging problems in shape matching have been appearing from different scientific areas including Bioinformatics and Image Analysis. In a class of problems in Shape Analysis, one assumes t h a t the points in two or more configurations are labelled and these configurations are to be matched after filtering out some transformation. Usually the transformation is a rigid transformation or similarity transformation. Several new problems are appearing where the points of configuration are either not labelled o r the labelling is ambiguous, and in which some points do not appear in each of the configurations. An example of ambiguous labelling arises in understanding the secondary structure of proteins, where we are given not only the 3-dimensional molecular configuration but also the type of molecules (amino acids) a t each point. A generic problem is to match such two configurations, where the matching has to be invariant under some transformation group.

There are other related examples from Image Analysis such as matching buildings when one has multiple 2-dimensional views of 3-dimensional objects (see, for example, Cross and Hancock9). The problem here requires filtering out the projective transformations before matching. Other examples involve matching outlines or surfaces (see, for example, Chui and Rangarajan19). Here there is no labelling of points involved, and we are dealing with a continuous contour or surface rather than a finite number of points. Duta et al.20 give a specific example of unlabelled matching solid shapes.

Green and Mardia12 build a hierarchical Bayesian model for the point configurations a n d de-rive inferential procedure for its parameters. I n particular, modelling hidden point locations as a Poisson process leads to a considerable simplification. They discuss in particular the problem when only a linear or affine transformation has to be filtered out. They also provide an implementation of the resulting methodology by means of Markov chain Monte Carlo (MCMC) samplers. Under a broad parametric family of loss functions, an optimal Bayesian point estimate of the matching matrix have been constructed, which turns out to depend on a single parameter of

Kanti V. Mardia 5

the family. Also discussed there is a modification to the likelihood in their model to make use of partial label ('colour') information at the points. The principal innovations in this approach are (a) the fully model-based approach to alignment, (b) the model formulation allowing integrating out of the hidden point locations, (c) the prior specification for the rotation matrix, and (d) the MCMC algorithm.

We now give some details together with an example.

3 .1 . Notation

Consider again two configurations of unlabelled landmarks in d dimensions, Xj, j = 1 , . . . , J and yk, k = 1 , . . . , K, respectively, represented as matrices x ( J x d) and y (K x d), where J is not necessarily the same as K. The objective is to find suitable subsets of each configuration and a suitable transformation, such the the two subconfigurations become closely matched. One of the key parameters is the matching matrix of the configurations which is represented by M (J x K), where Mjk indicates whether the points in the two configurations Xj and yk are matched or not. That is, Mjk = 1, if Xj matches yk, and 0 otherwise. Note that M is the adjacency matrix for the bipartite graph representing the matching, and that Y^,j k Mjk — L number of matches.

The transformation g, say, lies in a specified group Q of transformations. Depending on the application, suitable choices for Q include (a), translations, (b) rigid body transformations, (c) similarity transformations, (d) affine transformations and (e) projective transformations.

It is sometimes notationally helpful to add an extra column (k = 0) to M to yield Mo, where rrijo = 1 — JZfe=i mjk = 1 if Xj is not matched to any yk, and 0 otherwise. The matrix M (or equivalently A/0) us called a "hard" labelling because each element is either 0 or 1. It is also helpful to consider "soft" labellings given by a J x (K + 1) matrix M*, say, where 0 < m*k < 1 and ^Zk=0 m*k = 1. There is now no constraint on ]T\ m*k. Note that M is symmetric in the roles of j and k, but Mo and M* are not.

Thus the overall matching objective can be restated as finding a matrix M and transformation g such that Xj « g(yk) for j , k with mjk = 1, as measured, e.g., by a sum of squares criterion. In computational geometry, the problem is termed the largest common point set (LCP) problem. We consider two

related statistical models to tackle the problem.

3.2. Some statistical approaches

Model 1: Regression models In this approach we condition on the landmark positions for one configuration y and on the matching matrix M (or equivalently Mo), and then model the distribution of the landmarks of the other configuration x. In the hard version of the model, the landmarks Xj, j = 1 , . . . , J are taken to be conditionally independent with

Xj ~ Nd(g{yk),o-2Id), when rrijk = 1 for some k,

(1)

and Xj ~ Nd{g{y0),alld), when mj0 = 1.

Here OQ » <T2 is large, and yo is located in the center of the y configuration, and the latter distribution is meant to represent a broad distribution for those x landmarks not matched to any y landmarks. The matching parameters M and transformation g e Q are to be estimated.

There is also a soft model under which the landmarks {XJ}, j = 1 , . . . , J are treated as conditionally independent identically distributed observations from a mixture distribution,

K

Xj ~ n0Nd(g(y0),alld) + ^^kNd{g{yk), cr2/d), fe=i

where Ylo ^k = 1- ^ n e hard membership function Mo does not appear in this formulation, but the posterior probabilities that Xj comes from the class for yk form a soft membership matrix M*.

This soft model has been used by several authors including Cross and Hancock,9 Luo and Hancock,21 Walker,22 Chui and Rangarajan,19 and Kent et al.23 In this case the EM algorithm can be used to compute the MLEs (at least locally), and it takes the form of a simple explicit iterative updating algorithm. In general it converges quite quickly in this context, though the solution depends heavily on the starting value for g (Kent et al.,23), as well as the parameters a and oo- Dryden et al.24 have treated multiple configurations.

Model 2: Bayesian Hierarchical Model The matching problem can also be given a formulation which is symmetric in the roles of j and k by introducing a set of unknown latent sites {fii, i =

6 Why Statistical Shape Analysis is Pivotal to the Modern Pattern Recognition?

1 , . . . , J} to represent a collection of "true" locations. In particular, given rrijk = 1 for some j and k, the key assumption in this approach is that there is a value of i such that

XJ ~ N{in,a2I), g(yj) ~ N{m,a2I).

This approach has been implemented in a Bayesian framework by Green and Mardia (2006) with inference carried out by MCMC. In particular, the hidden points {ni} are generated by a Poisson process with some rate A and the proportion of matches is governed by a parameter p. When g comes from the rigid body group g(y) = Ay+a, where A is a rotation matrix and a is a translation vector, the likelihood can be shown to take the form

likelihood oc I T j,k:mjk = l

p<j>{{xj - Ayk - a}/<T\/2)

(2) where 4>{.) is the density of iV(0,1), after integrating out the {p-i}- This construction defines a hard version of Model 2; it is also possible to define a soft version. We add suitable priors on A, y and a (see, Green and Mardia12).

Estimation for the hard versions of Models 1 and 2 is difficult to carry out analytically. Model 2 feels more natural for this situation and an elegant and powerful MCMC algorithm, which avoids the need for reversible jump steps, has been developed by Green and Mardia.12 The soft versions can be tackled by EM, though the choice of starting point is critical since the likelihood (or posterior) will generally be highly multimodal.

3.3. Example: Matching protein gels

The objective in this example is to match two elec-trophoretic gels automatically given two gel images (see Fig. 1). However we will assume that the images have been preprocessed and we are given the locations of the centres of 35 proteins on each of the two gels. The correspondence between pairs of proteins, one protein from each gel, is unknown, so our aim is to match the two gels based on these sets of unlabelled points. We suppose that it is known that the transformation between the gels is affine. In this case, experts have already identified 10 points; see Horgan et al.25 Based on these 10 matches, the linear part of the transformation is estimated a priori (Dryden & Mardia,8 pp. 20-1, 292-6).

Here, we have only to make inference o n the translation parameter and the unknown matching between certain of the proteins. Figure 2 gives the matches from this method. Note that all o f the expert-identified matches, points 1 to 10 in each set, are declared to be matches with high probability in the Bayesian analysis.

4. Image Deformations

Deformation is a basic tool of image analysis (see, for example, Toga,26 Glasbey and Mardia,10 Hajnal et al.,27 and Singh et al.,28) which maps a region S to a region T in Rd. Consider a landmark-based example. Let U, i = 1 , . . . , k be the configuration of landmarks in the "starting" or "source" image S, and xt, i = 1 , . . . , k, be their homologues in a "target" image T. A useful deformation that takes the i ' s onto the z's and maps every point of S onto some poin t of T is the thin-plate spline (Bookstein29 ; Dryden and Mardia8). In the applied literature, e.g. radiology or evolutionary biology, this is often called a "model", though it is not actually serving that role: i t is a prediction function. In Toga26 for instance, none of the deformations introduced to relate pairs of neu-roanatomical images are assigned standard errors, as they would necessarily be assigned when these a r e estimates of some underlying model.

One common method of fitting a deformation is to use a thin-plate spline for each coordinate of the deformation. It is well-known that the thin-plate spline can also be given a stochastic interpretation as a predictor for a certain (intrinsic) stochastic process conditioned to match the observed values a t the source landmarks. An advantage of the stochastic approach is that confidence limits on the predicted values can be provided. Mardia et al.30 reviewed the stochastic framework and associated prediction error in detail.

There are two common strategies for fitting deformations given information at a set of landmarks. One involves a minimizing a roughness penalty, e.g. for a thin-plate spline, and another involves prediction for a stochastic process, e.g. for a self-similar intrinsic random field. The stochastic approach allows parameter estimation and confidence limits for the predicted deformation. An application is presented there in Mardia et al.30 from a study of breast data and how the images deform as a function of the imaging procedure.

Kanti V. Mardia 7

C&

(a)

(b)

Fig. 1. Gel Images

Mardia et al.14 address the problem of the distortion effect produced by different types of non-linear deformation strategies on textured images. The images are modelled by a Gaussian random field. They give various examples to illustrate that the model generates realistic images. They consider two types of deformations: a deterministic deformation and a landmark based deformation. The latter includes

Fig. 2. The 17 most probable matches in the gel data, + symbols signify x points, o symbols the y points, linearly transformed by premultiplication by the ixed affine transformation. The solid line for each of the 17 matches joins the matched points, and represents the inferred translation plus noise.

various radial basis type deformations including the thin-plate splines based deformation. The effects of deformations are assessed through Kuilback Leibler divergence measure. The measure is estimated by statistical sampling techniques. It is found empirically that this divergence measure is approximately distributed as a lognormal distribution under various different deformations. Thus a coefficient of variation based on logdivergence provides a natural criterion to compare different types of deformations. They find that the thin-plate splines deformation is almost optimal over the wider class of the radial type deformations.

A new method of a compositional approach to multiscale image deformation is given in de Souza et al.31 Here a general framework is presented for the application of image deformation techniques, in which a smooth continuous mapping is modelled by a piecewise linear transformation, with separate components at a number of discrete scales. These components are combined by composition, which has several advantages over addition. The ideal transformation between two images is the best compromise between matching criteria and smoothness constraints, and have been estimated using both deterministic and stochastic analyses.

8 Why Statistical Shape Analysis is Pivotal to the Modern Pattern Recognition?

5. Penalized Image Averaging and Discrimination

The importance of statistically based objective discrimination methods is difficult to overstate. The need for such is in many areas. We describe here semi-landmark the procedure of Mardia et al.13

Glasbey & Mardia10 provide a landmark free method based on a penalized likelihood to discriminate. However, where landmark information is readily available it is judicious to make full use of it in any discrimination procedure. Mardia et al.13 expand upon Glasbey & Mardia's10 method by incorporating landmarks. The approach combines Glasbey and Mardia's method on the full image with Procrustes methods for landmark data; it is a fusion of landmark and standardized image information. The use of the extra landmark information improves the discrimination, giving a wider separation in the stu-dentized difference in means of within and between group measures. In addition, the use of landmarks significantly improves the computational speed compared with the landmark free approach.

The penalized likelihood is comprised of similarity and distortion parts. The likelihood measures the similarity between images after warping and the penalty is a measure of distortion of a warping. The images they discriminate, the measures of similarity consist of normalized image information in the two-and three-dimensional settings respectively.

We now give some details together with an example.

5.1. Image averaging

The basic strategy to obtain an average of a sample

of images is as follows. We assume the images possess

easily identifiable landmarks. The strategy is

(1) Obtain the mean shape of landmarks using the Procrustes mean.

(2) Register the sample of images to the mean shape. (3) Warp all the images to the mean shape using say

thin plate spline. (4) Interpolate the images (if necessary) to give a ho

mologous coordinate system over all the sample images.

(5) Average the images over the homologous coordinate system.

5.2. Image distance and discrimination

We now describe a general strategy for image discrimination based on penalized likelihood.

5.2.1. Image distance

Define a measure of distance between two images /$ and fj, both in m dimensions, to be

P(fi, fj) = T(fh ft) + \L(Xi, Xj), (3)

where T(fi,fj) is a standardised difference i n texture/surface information, L(Xj,Xj) is a landmark based distance between images and A is a weighting parameter. Next, for a sample of n images / i ; • • •, fn, in TTI dimensions with corresponding landmarks X i , . . . , X n , define an average image by minimising

n n

F = ^ T ( / i , / ) + A ^ L ( X i , M ) , (4) i = l t = l

with respect to A and fi. Here / is the mean image and T and L are as defined in quantity (3). An algorithm to minimise this quantity is described in Section 5.3. The optimum average image is referred to as the perturbed Procrustes average when it is evaluated by an algorithmic method.

Discriminant analysis. Consider the case where we wish to allocate an individual image to one of two populations, say HA and l i s - Suppose we are given samples of training data from the distinct populations, these are / i , . • •, / m from n ^ and gi, • •. ,gn

from HB- Firstly, we obtain the optimum image averages, / and g via Section 5.3. Then we carry out discriminant analysis on the variables P(h, f) and P(h,g), where P is given in (3) and h is an individual image.

In order to use (3) we need to select an appropriate: weightingj)arameter A. We select A which attains the maximum separation between the training samples according to P(ft, / ) , P{gj,f), P{fi,g) and P(gj,g), i = 1 , . . . ,m; j = 1 , . . . ,n, that is the distance between each image and each average image. We obtain the optimal choice of A by selecting that which optimises the studentised difference between P within group and P between group means. Here the within group consists of P(fi, f) and P{gj,g) and the between group consists of P(fi,g) and P(gj,f), i = 1 , . . . , m; j = 1 , . . . , n. Mardia et al.13 have given an explicit expression for optimum A.

Kanti V. Mardia 9

Assuming that training data has been used to obtain the population mean images and the optimum A has been derived, we can define a discriminant rule, based on image distances, to allocate a new image h in the classical way, eg. Fisher's rule.

5.2.2. Specific image distances in the two dimensional setting

Full Procrustes with texture information. We define a measure of distance between two images fc (t) and fj(t), via warping one to the other, to be

A( /* , / i ) = Et(/i(*X i-X1(*)) /.x - / J ( t ) ) 2 + AdF(X,,X i)

2 . {0)

The measure of distance Pi given in equation (5) is comprised of a similarity part and a distortion of shape part through the Procrustes distance. The parameter A determines the relative weighting between similarity and distortion. Also, it is worth commenting that we could have alternatively used d/p in (5) for balance in the measure, but it is still valid to use dp- Note, this measure does not detect size differences. However, a size measure could be added to the distance measure Pi. A suitable size term is the ratio of centroid sizes of the landmark configurations.

Bending energy with texture information methods. Here, an alternative approach to discrimination is given, where the bending energy matrix is used instead of the Procrustes distance. Define the second measure of distance between two images /,(£) and fj(t), via warping one to the other, to be

PuifiJj) = E t C f i ^ - x ^ ) ) - fj(t))2+ A vec(X;)TBlock(B(X.,), B(XJ)))vec(Xi),

(6) where B(X.j) is the bending energy matrix for the deformation with Xj as the source. The measure of distance Pi given in equation -(6}-^ comprised of a similarity part and a distortion part. Again the parameter A determines the relative weighting between similarity and distortion. Note Pi does not account for any affine differences between images. This is due to the bending energy being affine invariant. Therefore, this is only appropriate where very little of the warp from one configuration to another can be accounted for by an affine transformation. As with Pi given in quantity (5), Pi in (6) does not detect size differences.

5.3. Perturbed Procrustes averages

The optimum average image can be arrived at by perturbation of the Procrustes mean shape via the following optimisation procedure: Algorithm 1 1. Set A = Aj. 2. Obtain /z, the Procrustes mean shape of X i , . . . , X n . 3. Obtain / = ± ]T"=1 * / 1 _ > X i ( / i ) . evaluate F. 4. Perturb fj, to give fj,pert. 5. Obtain / = £ £"=i tfj^-oc^/i), evaluate Fnew f o r Uperf I f Fnew < F a ccep t /X p e r t ,

i.e set H=npert. 6. Repeat steps 3, 4 and 5 until F cannot be reduced further. 7. Set A = A J + i , repeat steps 2 to 6 until F cannot be reduced further.

Here ^/i—x^/t) denotes the tth image warped to the mean shape, fj,. The final / is the average image. Note in Algorithm 1, /z is only changed if the objective function improves. This algorithm possesses an MCMC-like property, however unlike MCMC methods in this case only improvements due to perturbation are accepted.

In this algorithm various choices of distance could be used. However, the objective function F must penalise affine transformations in the perturbation otherwise severe shearing occurs. This will not occur where the Procrustes distance is used. We propose Gaussian updates from a bivariate normal distribution on each landmark as the perturbation method. Mardia et al.13 have used a systematic grid search for the choice of A.

5.4. Example: Fish

Here we consider photographic images obtained under controlled conditions of two species of fish , haddock and whiting. These are part of a data set of ten haddock and ten whiting images. Eight of each species are randomly selected as training data leaving two from each species as test data. In total sixteen corresponding landmarks are defined on each of the fish in this example (see Figures 3, 4).

Effect on average of perturbation . The extent of the improvement in the perturbed average have been assessed under a criteria based on the the distance Pi from equation (5) and found Pi to be smaller.

10 Why Statistical Shape Analysis is Pivotal to the Modem Pattern Recognition?

Effect on discrimination of perturbation. Here we wish to assess if the perturbed averages improve the discrimination procedure. In order to do this, Mardia et al.13 consider the studentised differences between Pi within species and Pi between species for A = 48.1 million. This value attains the optimal separation for the Procrustes'average images. Taking A = 48.1 million is not the optimal value to discriminate using the perturbed average, however parity enables comparison between the studentised differences. We find that the perturbed average gives a greater separation between species.

Another example of reconstruction through a stochastic deformation template related to degratded fish is given in de Souza et al.32 which may be applicable for other fish images - rather than photographs as here.

Fig. 3. Haddock 1.

Fig. 4. Whiting 1.

The details of the allocation for the training and test data are given in Mardia et al.13 All the fish were correctly allocated.

6. Discussion

We have given here a personal view of why shape analysis is pivotal to pattern recognition in a very broad sense. Still the field is young and we hope to see many new developments coming from interdisciplinary research. Mardia and Gills33 have identified three themes for statistics in the 21st century. First, statistics should be viewed in the broadest way for scientific explanation or prediction of any phenomenon. Second, the future of statistics lies in a holistic approach to interdisciplinary research. Third, a change of attitude is required by statisticians - a paradigm shift - for the subject to go forward.

References

1. Mardia, K.V. and Kanji, G. (eds.) (1993). Statistics and Images: Vol L Carfax Publishing Co. Ltd., Abingdon, Oxfordshire.

2. Mardia, K.V. (ed.) (1994). Statistics and Images: Vol. II Carfax Publishing Co. Ltd., Abingdon, Oxfordshire.

3. Dryden, I.L., Mardia, K.V. and Walder, A.N. (1997). Review of the use of context in statistical image analysis. Journal of Applied Statistics, 24, pp513-538.

4. Mardia, K.V. (1997). Bayesian image analysis. J. Theoretical Medicine, 1, pp63-77.

5. Glasbey, C.A. and Mardia, K.V. (1998). A review of image warping methods. J. Appl. Statist., 25, ppl§5-171.

6. Mardia, K.V. (2001). Shapes in Images. In Pattern Recognition Prom Classical to Modern Approaches, edited by Pal, S.K. and Pal, A., World Scientific, ppl47-167.

7. Krim, H. and Yezzi, Jr. A. (eds). (2006). Statistics and Analysis of Shapes. BIrkhauser, Boston.

8. Dryden, I.L. and Mardia, K.V. (1998). Statistical Shape Analysis. J. Wiley, Chichester.

9. Cross, A. D. J. k Hancock, E. R. (1998). Graph matching with dual-step EM algorithm. IEEE Trans. Patt Anal Mach. Intell. 20, 1236-53.

10. Glasbey, C.A. and Mardia, K.V. (2001). A Penalized Likelihood Approach to Image Warping (with discussion). Journal of Royal Statistical Society, Series B, 63, pp465-514.

11. Richmond, N.J., Willett, P. and Clark, R.D. (2004). Alignment of three-dimensional molecules using an image recognition algorithm. Journal of Molecular Graphics and Modelling, 23, pp 199-209.

12. Green, P.J. and Mardia, K.V. (2006). Bayesian alignment using hierarchical models, with applications in protein bioinformatics. Biometrika, 93 , pp235-254.

13. Mardia, K.V., McDonnell, P. and Linney, A.D. (2006). Penalised image averaging and discriminations with facial and fishery applications. Journal of

Kanti V. Mardia 11

Applied Statistics, 33, pp339-369. 14. Mardia, K.V., Angulo, J.M. and Goitia, A. (2006).

Synthesis of image deformation strategies. Image and Vision Computing, 24, ppl-12.

15. Mardia, K.V. and Jupp, RE. (2000). Directional Statistics, 2nd edition. J. Wiley, Chichester.

16. Goodall, C.R. and Mardia, K.V. (1993), Multivariate aspects of shape theory. Ann. Statist., 21 , 848-866.

17. Mardia, K.V. and Patrangenaru, V. (2005). Directions and projective shapes. Annals of Statistics, 33, ppl666-1699.

18. Kent, J.T. and Mardia, K.V. (2006). A new representation for projective shape. LASR2006 Proceedings. Edited by: Barber, S., Baxter, P.D., Mardia, K.V. and Walls, R.E., Leeds University Press, pp75-78.

19. Chui, H. and Rangarajan, A. (2003). A new point matching algorithm for non-rigid registration. Computer Vision and Understanding, 89, 114-141.

20. Duta, N., Jain, A.K. and Mardia, K.V. (2002). Matching of palm prints. Pattern Recognition Letters, 23 , pp477-485.

21. Luo, B. and Hancock, E.R. (2001). Structural Matching using the EM algorithm and singular value decomposition. IEEE Trans. PAMI, 23, 1120-1136.

22. Walker, G. (2000). Robust, non-parametric and automatic methods for matching spatial point patterns. PhD thesis, University of Leeds.

23. Kent, J. T., Mardia, K. V. & Taylor, C. C. (2004). Matching problems for unlabelled configurations. In Bioinformatics, Images and Wavelets, Proceedings of LASR 2004, Ed. R. G. Aykroyd, S. Barber & K. V. Mardia, pp. 33-6. Leeds: Leeds University Press.

24. Dryden, I.L., Hirst, J.D. and Melville, J.L. (2006).

Statistical analysis of unlabelled point sets: comparing molecules in chemoinformatics. Biometrics to appear.

25. Horgan, G.W., Creasey, A. and Fenton, B. (1992). Superimposing two-dimensional gels to study genetic variation in malaria parasites. Electrophoresis, 13, 871-875.

26. Toga, A.W. (1999). Brain Warping. Academic Press, San Diego.

27. Hajnal, J.V., Hill, D.L.G. and Hawkes, D.J. (2001). (Eds) Medical Image Registration. CRC Press, London.

28. Singh, A.J., Goldgof, D. and Terzopoulos, D. (1998). (Eds) Deformable Models in Medical Image Analysis. IEEE Computer Soc, Los Alamitos, California.

29. Bookstein, F.L. (1991). Morphometric Tools for Landmark Data: Geometry and Biology. Cambridge University Press, Cambridge.

30. Mardia, K.V., Bookstein, F.L., Kent, J.T. and Meyer, C.R. (2006b). Intrinsic random fields and image deformations. Journal of Mathematical Imaging and Vision, in press.

31. de Souza, K.M.A., Kent, J.T., Mardia, K.V. and Glasbey, C. (2007). A compositional approach to multiscale deformation. In preparation.

32. de Souza, K.M.A., Kent, J.T. and Mardia, K.V. (1999). Stochastic templates for Aquaculture Images and a Parallel Pattern Detector. J. Roy. Statist. Soc. C. 48, pp211-227.

33. Mardia, K. V. and Gilks, W. (2005), "Meeting the statistical needs of 21st-century science," Significance, 2, 162-165.

PART B

Invited Lectures

15

On the Interest of Spatial Relations and Fuzzy Representations for Ontology-Based Image Interpretation

Isabelle Bloch, Celine Hudelot1 and Jamal Atif2

Ecole Nationale Superieure des Telecommunications - GET-Telecom Paris CNRS UMR 5141 LTCI - Signal and Image Processing Department

46 rue Barrault, 75013 Paris, France E-mail: Isabelle. [email protected]

1 Current address: Ecole Centrale Paris, Grande Voie des Vignes, 92 295 Chdtenay-Malabry Cedex, France E-main: [email protected]

2Current address: Universite des Antilles et de la Guyane, Guyane, FVance

E-main: [email protected]

In this paper we highlight a few features of the semantic gap problem in image interpretation. We show that semantic image interpretation can be seen as a symbol grounding problem. In this context, ontologies provide a powerful framework to represent domain knowledge, concepts and their relations, and to reason about them. They are likely t o be more and more developed for image interpretation. A lot of image interpretation systems rely strongly on descriptions of objects through their characteristics such as shape, location, image intensities. However, spatial relations are very important too and provide a structural description of the imaged phenomenon, which is often more stable and less prone to variability than pure object descriptions. We show that spatial relations can be integrated in domain ontologies. Because of the intrinsic vagueness we have to cope with, at different levels (image objects, spatial relations, variability, questions to be answered, etc.), fuzzy representations are well adapted and provide a consistent formal framework to address this key issue, as well as the associated reasoning and decision making aspects. Our view is that ontology-based methods can be very useful for image interpretation if they are associated to operational models relating the ontology concepts to image information. In particular, we propose operational models of spatial relations, based on fuzzy representations.

Keywords: Image interpretation, semantic gap, ontology, spatial relations, fuzzy sets, fuzzy relations, brain imaging.

1. Introduction

The literature acknowledges several attempts towards formalization of some domains. For instance in medicine, noticeable efforts have led to the development of the Neuronames Brain Hierarchy(http: / / bralninfo.rprc.washington.edu/) and the Foundational model of anatomy (FMA) ( h t t p : / / s i g . biostr.Washington .edu/projects/ fm/AboutFM.html)at the University of Washington, or Neuranat (ht tp: / /www.chups. jussieu.fr /ext / neuranat)in Paris at CHU La Pitie-Salpetriere. Generic formalizations of spatial concepts were also developed and specified in different fields, for spatial reasoning in artificial intelligence, for Geographic Information Systems, etc.

In a parallel domain, well formalized theories for image processing and recognition appeared in the image and computer vision community.

Noticeably, both types of developments still remain quite disjoint and very few approaches try to use the abstract formalizations to guide image in

terpretation. The main reason is to be found in the so called "semantic gap", expressing the difficulty to link abstract concepts with image features. This problem is also related to the symbol grounding problem.

In this paper we highlight a few features of the semantic gap problem in image interpretation. We show that semantic image interpretation can be seen as a symbol grounding problem in Section 2. In this context, ontologies provide a powerful framework to represent domain knowledge, concepts and their relations, and to reason about them. Therefore, they are likely to be more and more developed for image interpretation. We briefly explain the potentials of ontologies towards this aim in Section 3. A lot of image interpretation systems rely strongly on descriptions of objects through their characteristics such as shape, location, image intensities. However, spatial relations are very important too, as explained in Section 4, and provide a structural description of the imaged phenomenon, which is often more stable and less prone to variability than pure object descriptions. We show

16 On the Interest of Spatial Relations and Fuzzy Representations for Ontology-Based Image Interpretation

that spatial relations can be integrated in domain ontologies. Because of the intrinsic vagueness we have to cope with, at different levels (image objects, spatial relations, variability, questions to be answered, etc.), fuzzy representations are well adapted and provide a consistent formal framework to address this key issue, as well as the associated reasoning and decision making aspects. This question is addressed in Section 5. Our view is that ontology-based methods can be very useful for image interpretation if they are associated to operational models of spatial relations (and other concepts), in particular based on fuzzy representations. These operational models contribute to reduce the semantic gap. We provide some hints on this integration in Section 6.

As a typical application where all these issues are raised, we illustrate our purpose with examples in brain image interpretation.

2. Semantic gap in image interpretation and symbol grounding

The symbol grounding problem has been first introduced in artificial intelligence by Harnad in,1 as an answer to the famous Searle's criticisms of artificial systems.2 It is defined in1 through the fundamental question: How is symbol meaning to be grounded in something other than just more meaningless symbols. As underlined in the literature, symbol grounding is still an unsolved problem (see e.g.3).

In the robotics community, this problem was addressed as the anchoring problem:4 a special form of symbol grounding needed in robotic systems that incorporate a symbolic component and a reasoning process. The anchoring process is defined as the problem of creating and maintaining the correspondence between symbols and sensor data that refer to the same physical object.

In our case, artificial systems are not robotic systems but image interpretation systems. As the former, they incorporate a symbolic component. Some similarities between Anchoring and Pattern Recognition have been underlined in,5 in order to assess the potentiality of using ideas and techniques from anchoring to solve the pattern recognition problem and vice versa. Similarly, we argue that image interpretation could greatly benefit from such a correspondence. Indeed, the image interpretation problem can be defined as the automatic extraction of the meaning of an image. The image semantics cannot be

considered as being included explicitly in the image itself. It rather depends on prior knowledge on the domain and the context of the image. It is therefore necessary to ground the digital representation of an image (perceptual level) with the semantic interpretation that a user associates to it (linguistic level). In the image indexing and retrieval community, this problem is called the semantic gap problem, i.e. the lack of coincidence between the information that one can extract from the visual data and the interpretation of these data by a user in a given situation.6

Our view is that image interpretation can be seen as a symbol grounding problem, i.e. the dynamical process of associating image data to human interpretations by taking into account the influence of external factors such as the social environment (application domain, interpretation goal, ...) or the physical environment of the interpretation. Indeed, image interpretation is the process of finding semantics and symbolic interpretations of image content. This problem has the same nature as the physical grounding of linguistic symbols in visual information in the case of natural language processing systems.7'8 In our case, linguistic symbols are application domain concepts defined by their linguistic names and their definition.

Example: In cerebral image interpretation, concepts can be: brain: part of the central nervous system located in the head, caudate nucleus: a deep gray nucleus of the telencephalon involved with control of voluntary movement, glioma: tumor of the central nervous system that arises from glial cells,...

Rather than being constrained by a grammar and a syntax as in a formal or natural language, the concepts are organized in a semantic knowledge base which describes their semantics and their hierarchical and structural dependencies.

Example: The human brain is a structured scene and spatial relations are highly used in the anatomical brain description (e.g. the left thalamus is to the left of the third ventricle and below the lateral ventricle).

This structural component, in the form of spatial relations, plays a major role in image interpretation. This aspect is detailed in Section 4.

Ontologies are useful to represent the semantic knowledge base. They entail some sort of the world view, i.e. a set of concepts, their definitions and their relational structure which can be used to describe and reason about a domain. This aspect is detailed

Isabelle Block, Celine Hudelot and Jamal Atif 17

in Section 3. As underlined by Cangelosi in,9 a symbol

grounding mechanism, as language itself, has both an individual and a social component. The individual component called Physical Symbol Grounding refers to the ability for a system to create an intrinsic link between perceptions and symbols. The Social Symbol Grounding refers to the ability to communicate with other systems by the creation of a shared lexicon of perceptually-grounded symbols. It is strongly related to the research on human language origins and evolution where external factors such as cultural and biological evolution are primordial.

N.tii u\im\ l.*t .-„.•*. l . \ .

i i . , f >•-,

»: Mumliwi

. » • < • * t •

• • ' • * ; •

livMiK^V.S."' * * ^ 9X >¥*>•)?

denotes

Semantic Kmnv-Sedge 1 brain 1

1 ' 1 1 . ] t 1

1

real world /•V"> ' (fvujlfiiiig;

perceptions by visual sensors Visual Percepts

Image pixels regions descriptors..

Fig. 1. Physical and external symbol grounding for image interpretation.

In the case of image interpretation systems, these two components of the symbol grounding are also essential and take the following form: on the one hand, the physical symbol grounding consists of the internal creation of the link between visual percepts (image level) and a known semantic model of the part of the real world which concerns the application domain (domain semantic level). On the other hand, in order to enable communication and interoperability with humans or other systems, this grounded interpretation must capture a consensual information accepted by a group. As a consequence a social external symbol grounding component raises for image interpretation. Moreover, image interpretation systems operate in a dynamic environment which is prone to changes and variations. The interpretation process is highly influenced by external factors such as the environmental context, the perception system

or the interpretation goal and it has to adapt itself to these external factors. As a consequence, image interpretation is a distributed and adaptive process between physical symbol grounding and external symbol grounding as shown in Figure 1.

3. Ontologies for image interpretat ion

In knowledge engineering, an ontology is defined as a formal, explicit specification of a shared conceptualization.10 An ontology encodes a partial view of the world, with respect to a given domain. It is composed of a set of concepts, their definitions and their relations which can be used to describe and reason about a domain. Ontological modeling of knowledge and information is crucial in many real world applications such as medicine for instance.11

Let us mention a few existing approaches involving jointly ontologies and images. By using ontologies, the physical symbol grounding consists in ontology grounding,12 i.e. the process of associating abstract concepts to concrete data in images. This approach is considerably used in the image retrieval community to narrow the semantic gap. In,13

the author proposes to ground, in the image domain, a query vocabulary language used for content-based image retrieval using supervised machine learning techniques. A supervised photograph annotation system is described in,14 using an annotation ontology describing the structure of an annotation, irrespectively of the application domain, and a second ontology, specific to the domain, which describes image contents. Another example concerns medical image annotation, in particular for breast cancer,15 and deals mainly with reasoning issues. But image information is not direcly involved in these two systems. Other approaches propose to ground intermediate visual ontologies with low level image descriptors,16-18

and are therefore closer to the image interpretation problem. In,19 the enrichment of the Wordnet lexicon by mapping its concepts with visual-motor information is proposed.

As the main ontology language OWL is based on description logics, a usual way to implement the grounding between domain ontologies (or visual ontologies) and image features is the use of concrete domains as shown in Figure 2.

Description logics20 are a family of knowledge-based representation systems mainly characterized

18 On the Interest of Spatial Relations and Pazzy Representations for Ontology-Based Image Interpretation

Table 1. Description logics syntax and interpretation.

Constructor atomic concept

individual Top

Bottom atomic role conjunction disjunction

negation existential restriction universal restriction

value restriction number restriction

Subsumption Concept definition

Concept assertion Role assertion

Syntax A a T JL

r CUD CUD

-rC 3r.C Vr.C

9 r.{A] (>nR) (<nR) C C D C = D

a:C (a,b) : R

Example Human

Lea Thing

Nothing has-age

Human n Male Male U Female

-i Human 3has-child.Girl

Vhas-child. Human 9has-child.{Lea} (> 3 has-child)

(< 1 has-mother) Man C Human

Father = Man n 3 has-child.Human

John:Man (John,Helen):has-child

Semantics A1 CAX

a? € A 1

T I = A I

±.x = $x

Rx C A 1 x Ax

c x nD x

CXUDX

AX\CX

{x € A1 | 3y € A1 : (x,y) € Rx A y € C 1 } {x 6 A z | Vj/ € A z : (x, y) € RT =s- y £ C1} {x € Ax | 3y € A2" : (x, y) € flx => y = a x }

{ x € A x | K l r l f o y ) € « * } ! > * } {xeA 1 ! \{y\(x,y)eRx}\<n}

C1 CD* CX = DX

ax €CX

(ax,bx)eRx

Abstract domain

Rose R3: hasCoior

Concrete domain (image)

Pink (RGB values)

(shape descriptors)

Fig. 2. tion.

Importance of concrete domains in image interpreta-

by a a set of constructors that enable to build complex concepts and roles from atomic ones. A semantics is associated with concepts, roles and individuals using an interpretation J = (A1, -1), where A 1 is a non empty set and -J is an interpretation function that maps a concept C to a subset CJ of A 1 or a role r to a subset R1 of A J x A z . Concepts correspond to classes. A concept C represents a set of individuals (a subset of the interpretation domain). Roles are binary relations between objects. Table 1 describes the main constructors and a syntax for description logics.

Concrete domains are expressive means of description logics to describe concrete properties of real world objects such as their size, their spatial extension or their color. They are of particular interest for image interpretation, as illustrated in Figure 2. Indeed, they allow performing anchoring for a particular application, hence reducing the semantic gap. This grounding approach using description log

ics and concrete domains has been used by several authors21,22 for the automation of semantic multimedia annotation.

4. Importance of spatial relations

Spatial relations between objects of a scene or an image is of prime importance, as highlighted in different domains, such as perception, cognition, spatial reasoning, Geographic Information Systems, computer vision. In particular, the spatial arrangement of objects provides important information for recognition and interpretation tasks, in particular when the objects are embedded in a complex environment like in medical or remote sensing images.23'24 Human beings use extensively spatial relations in order to describe, detect and recognize objects: they allow to solve ambiguity between objects having a similar appearance, and they are often more stable than characteristics of the objects themselves (this is typically the case of anatomical structures).

Many authors have stressed the importance of topological relations, but distances and directional relative position are also important, as well as more complex relations such as "between", "surround", "among", etc. Freeman25 distinguishes the following primitive relations: left of, right of, above, below, behind, in front of, near, far, inside, outside, surround. Kuipers24,26 considers topological relations (set relations, but also adjacency which was not considered by Freeman) and metrical relations (distances and directional relative position).

Isabelle Block, Celine Hudelot and Jamal Atif 19

Spatial reasoning can be defined as the domain of spatial knowledge representation, in particular spatial relations between spatial entities, and of reasoning on these entities and relations (hence the importance of relations). This field has been largely developed in artificial intelligence, in particular using qualitative representations based on logical formalisms. In image interpretation and computer vision, it is much less developed and is mainly based on quantitative representations. In most domains, one has to be able to cope with qualitative knowledge, with imprecise and vague statements, with polysemy, etc. This calls for a common framework which is both general enough to cover large classes of problems and potential applications, and able to give raise to instantiations adapted to each particular application. Ontologies appear as an appropriate tool towards this aim. This shows the interest of associating ontologies and spatial relations for symbol grounding and image interpretation. Figure 3 illustrates a part of an ontology of spatial relations.27

Spatial Relation

X Topological Relation Metric Relation

Adjacent; Directional Relation Distance Relation

Binary Directional Relation

Ternary Directional Relation

Right to 11 Left to | |ln Front otj Close to Far from

hierarchy relation

Fig. 3. Excerpt of the hierarchical organization of spatial relations in the ontology of.27

As mentioned in,28 several ontological frameworks for describing space and spatial relations have been developed recently. In spatial cognition and linguistics, the project OntoSpace(http://www. ontospace.uni-bremen.de/ twiki/bin/view/Main/WebHome) aims at developing a cognitively-based commonsense ontology for space. Some interesting works on spatial ontologies can also be found in Geographic Information Science29 or in medicine concerning the formalization of anatomi

cal knowledge.30"32 All these ontologies concentrate on the representation of spatial concepts according to the application domains. They do not provide an explicit and operational mathematical formalism for all the types of spatial concepts and spatial relations. For instance, in medicine, these ontologies are often restricted to concepts from the mereology theory.31

They are therefore useful for qualitative and symbolic reasoning on topological relations but there is still a gap to fill before using them for image interpretation.

Example: internal brain structures are often described trough their spatial relations, such as: the left caudate nucleus is inside the left hemisphere; it is close to the lateral ventricle; it is outside (left of) the left lateral ventricle; it is above the thalamus, etc. In case of pathologies, these relations are quite stable, but more flexibility should be allowed in their semantics.33

This example raises the problem of assigning semantics to these spatial relations, according to the application domain: what do concepts such as "close to" or "left" mean when dealing with brain images? Should this meaning be adapted depending on the context (possible pathology, etc.)? These questions can be addressed by using fuzzy models.

5. Importance of fuzzy representat ions

Usually vision and image processing make use of quantitative representations of spatial relations. In a purely quantitative framework, spatial relations are well defined for some classes of relations, unfortunately not for intrinsically vague relations (such as directional ones for instance). Moreover they need a precise knowledge of the objects and of the types of questions we want to answer. These two constraints can be relaxed in a semi-qualitative framework, using fuzzy sets. This allows to deal with imprecisely defined objects, with imprecise questions such as are these two objects near to each other?, and to provide evaluations that may be imprecise too, which is useful for several applications, where spatial reasoning under imprecision has to be considered. Note that this type of question also raises the question of polysemy, hence the need for semantics adapted to the domain. This is an important question to be solved in the symbol grounding and semantic gap problems.

Fuzzy set theory finds in spatial information pro-

20 On the Interest of Spatial Relations and Fuzzy Representations for Ontology-Based Image Interpretation

cessing a growing application domain. This may be explained not only by its ability to model the inherent imprecision of such information (such as in image processing, vision, mobile robotics...) together with expert knowledge, but also by the large and powerful toolbox it offers for dealing with spatial information under imprecision. This is in particular highlighted when spatial structures or objects are directly represented by fuzzy sets. If even less information is available, we may have to reason about space in a purely qualitative way, and the symbolic setting is then more appropriate. In artificial intelligence, mainly symbolic representations are developed and several works addressed the question of qualitative spatial reasoning (see34 for a survey). For instance in the context of mereotopology, powerful representation and reasoning tools have been developed, but are merely concerned by topological and part-whole relations, not by metric ones.

Limitations of purely qualitative spatial reasoning have already been stressed in,35 as well as the interest of adding semiquantitative extension to qualitative value (as done in the fuzzy set theory for linguistic variables36'37) for deriving useful and practical conclusions (as for recognition). Purely quantitative representations are limited in the case of imprecise statements, and of knowledge expressed in linguistic terms. As another advantage of fuzzy representations, both quantitative and qualitative knowledge can be integrated, using semi-quantitative (or semi-qualitative) interpretation of fuzzy sets. These representations can also cope with different levels of granularity of the information, from a purely symbolic level, to a very precise quantitative one. As already mentioned in,25 this allows us to provide a computational representation and interpretation of imprecise spatial constraints, expressed in a linguistic way, possibly including quantitative knowledge. Therefore the fuzzy set framework appears as a central one in this context. Several spatial relations have led to fuzzy modeling, as reviewed in.23

Spatial reasoning aspects often imply the combination of various types of information, in particular different spatial relations. Again, the fuzzy set framework is appropriate since it offers a large variety of fusion operators38'39 allowing for the combination of heterogeneous information (such as spatial relations with different semantics) according to different fusion rules, and without any assumption on an underlying

metric on the information space. They also apply on various types of spatial knowledge representations (degree of satisfaction of a spatial relation, fuzzy representation of a spatial relation as a fuzzy interval, as a spatial fuzzy set, etc.). These operators can be classified according to their behavior, the possible control of this behavior according to the information to combine, their properties, and their specificities in terms of decision.40 For instance, if an object has to satisfy, at the same time, several spatial constraints expressed as relations to other objects, the degrees of satisfaction of these constraints will be combined in a conjunctive manner, using a t-norm. If the constraints provide a disjunctive information, operators such as t-conorms are then appropriate. It is the case for example for symmetrical anatomical structures that can be found in the left or right parts of the human body. Operators with variable behavior, as some symmetrical sums, are interesting if the aim is a reinforcement of the dynamics between low degrees and high degrees of satisfaction of the constraints. In particular, this facilitates the decision since different situations will be better discriminated.

Let us come back to ontologies from the point of view of uncertain knowledge and imprecise information. A major weakness of usual ontological technologies is their inability to represent and to reason with uncertainty and imprecision. As a consequence, extending ontologies in order to cope with these aspects is a major challenge. This problem has been recently stressed out in the literature. Several approaches have been proposed to deal with uncertainty and imprecision in ontology engineering tasks.41'42 The first approach is based on probabilistic extensions of the standard OWL ontology language(http: / / www.w3.org/TR/owl-features/) by using Bayesian networks.43'44 The probabilistic approach proposes to first enhance the OWL language to allow additional probabilistic markups and then to convert the probabilistic OWL ontology into the directed acyclic graph of a Bayesian network with translation rules. As the main ontology language OWL is based on description logics,20 another approach to deal with uncertainty and imprecision is to use fuzzy description logics.45-48 Fuzzy description logics can be classified according to the way fuzziness is introduced into the description logics formalism. A good review can be found in.49 In particular, a common way for description logics with concrete domains is to intro-

Isabetle Block, Celine Hudelot and Jamal Atif 21

duce fuzziness by using fuzzy predicates in concrete domains as described in.50

Another approach is to introduce fuzziness directly in the concrete domains, which then become fuzzy concrete domains. This is particularly interesting for image interpretation.

Example: Using fuzzy representations of spatial relations in the image domain leads to restricted search area for the caudate nucleus, based on the knowledge that it is to the right and close to the lateral ventricles. This is illustrated in Figure 4-

(a) (b

(c) (d)

Fig. 4. (a) The right ventricle is superimposed on one slide of the original image (MM here). The search space of the object "caudate nucleus" corresponds to the conjunctive fusion of the spatial relations "to the right of the right ventricle" (b) and "close t o the right ventricle" (c). The fusion result is shown in (d).

Example: Typically brain image interpretation may have to cope with abnormalities such as tumors. Our system allows instantiating generic knowledge expressed in the ontology to adapt to the specific patient 's case. The fuzzy representations provide an efficient way to represent inter-individual variability, which are a key point in such situations. They can be further revised or specified according to the visual features extracted from the image and matched with the symbolic representation.

Using fuzzy representations, it is possible to deal with such cases, for instance by enlarging the areas where an object can be found, which amounts to relax the definition of the fuzzy relation.

In summary, fuzzy representations have several

advantages:

• they allow representing the imprecision which is inherent to the definition of a concept; for instance, the concept "close to" is intrinsically vague and imprecise, and its semantics depends on the context in which objects are embedded, on the scale of the object and on their environment;

• they allow managing imprecision related to the expert knowledge in the concerned domain;

• they constitute an adequate framework for knowledge representation and reasoning, reducing the semantic gap between symbolic concepts and numerical information.

6. Towards the integration of ontologies, spatial relations and fuzzy models

To conclude this presentation, we summarize ongoing developments carried out in our team, towards the construction of a spatial relation ontology enhanced with fuzzy representations and its use for image interpretation. This work aims at integrating all important features underlined in this paper. A global scheme of our approach is provided in Figure 5.

Our recent work addresses the important problems highlighted in this paper in several ways.27,52

We propose to reduce the semantic gap between numerical information contained in the image and higher level concepts by enriching ontologies with a fuzzy formalism layer. More specifically, we introduce an ontology of spatial relations and propose to enrich it by fuzzy representations of these relations in the spatial (image) domain. The choice of spatial relations is motivated on the one hand by the importance of structural information in image interpretation, and on the other hand by the intrinsically ambiguous nature of most spatial relations. This ontology has been linked to the part of FMA related to brain structures, as illustrated in Figure 5.

As another contribution, this enriched ontology can support the reasoning process in order to recognize structures in images, in particular in medical imaging. Different types of reasoning become then possible: (i) a quite general reasoning may consist in classifying or filtering ontological concepts to answer some queries; (ii) at a more operational way, the ontology and the fuzzy representations can be used to deduce spatial reasoning operations in the images and to guide image interpretation tasks such

22 On the Interest of Spatial Relations and Fuzzy Representations for Ontology-Based Image Interpretation

generic knowledge

H'*!••.'! m.iti.r.-':

JV.'"J iil'U-.tUTil

it'c.cipti >n

Knowledge of »pocifi': CHSOS

F.rmn inm'.'

J- M Mt.yl

l l fMltr, • T ^ ^ l Pathoiuqr.il]

;.rt:.o^

In'-I'iavnii lunnis

Graph based representation of the generic model G

Fuzzy modeling of spatial relations

Dealing wi th s specific

•PI H i

r i IP:-!.'.

Leammg jjroesdofn

Or i-i-w -ibifti f ? * f l

L-.J

Stepl: j i*] learning spatial I j relations (adjacency, i j distance, orientation) | j of the generic model

} using healthy cases

Step 2: • learning spatial relation for specif cases * deducing stable relations for each class of patholog *".

Hi&ry *wmtHfafte

Generic model ^ adaptation using ^ ^ knowledge of specific V ^ case and results of the W teaming procedure #

•^•Sil] - . i 'S l / l '•,

kifo JfkA* /?*&

fe5

Graph based propagation process to update the graph and to

represent the tumor impact on the

surrounding structures

Enrichment of the

* :••• pfcirt i is ' l l iSHf » i

"" v : 3 ,Spa t i a l relation ^ j ontology

concepts

Spatial relations between anatomical concepts

faiit!?I' njrl*-js

3pita. patialPclatipnRight Of PJJH Uwal tnt Kit

3pllu<SFMdPe!£ nCIa t To figs I La'rabtntn k

B r a i n a n a t o m v srniasspaiaipda HADO * of n ht ih jam

c o n c e p t s

Fig. 5. Overview of our framework. Ontological engineering is used to represent the symbolic knowledge useful to interpret cerebral images. In particular, a spatial relation ontology is used to enrich the brain ontology by the description of the spatial structure of the brain. A graph based representation of the brain including learned fuzzy representations of spatial relations is derived from the generic model and from an image database. This graph is used to guide the segmentation and the recognition of cerebral structures. This framework is also useful to deal with pathological cases by an adaptation of the knowledge and the reasoning process. The second scheme displays a part of an ontology of brain anatomy (excerpt of the FMA5 1) enhanced with our fuzzy spatial relations ontology. The concepts of the spatial relation ontology are prefixed by p i .

as localization of objects, segmentation, and recognition. An illustration is provided in Figure 6 for the

recognition of internal brain structures. Another enrichment of the model consists of

the representation of domain knowledge by graphs,

Isabelle Bloch, C&line Hudelot and Jamal Atif 23

DomaiR Ont&ftfgj-

' > . - 7 - ^ - T - -

* " * " ' w " : " " l "" " " ™ " '

Fact Base C"!; 5«<jt^,si«frii, ytoiwrews (\

lifcsJKfc JU^^WSrtftto^lattHftjtK *; K?: *3rsy, »Kttfw jx*HisH»wfi< P

v&, ij> ^ua^^ojthias^iAMa&ffi^ C^. 'on^^flaatf! ^tj&HHjsns",

fla<* 1WJ«X^„^aR'*t*J,.-^3ftKa»i<fU'; Cw; Caw imawRf .Jwwi^wttt^

iia», fa^'»4^„v, j»M8«i»^wt^H***i

Image domain

if5^ <

* a . *Siai - *fr\i i9lh?w^!Sd / /"• I

R4 R2

/ Rl

\ 1

) i m \ \ \ \ \ R3

Fig. 6. The right lateral ventricle corresponds to the spatial region Rl in the image. The domain ontology describes spatial relations between several grey nuclei and the lateral ventricles. These relations are exploited to identify each individual structure.

which include fuzzy models of spatial relations, used to guide the recognition of individual structures in images.53 The inclusion of such structural models, as intermediate representation domains between symbols and images, deals with the physical symbol grounding problem, and also contributes to reduce the semantic gap. However pathological cases may deviate substantially from generic knowledge. We propose to adapt the knowledge representation to take into account the possible influence of pathologies on the spatial organization, based on learning procedures. We also adapt the reasoning process, based on graph based propagation and updating. These features of our approach are detailed in.52 A result is illustrated in Figure 7.

Fig. 7. An axial slice of a 3D MRI, with segmented tumor and some anatomical structures.

The enriched ontology contributes to reduce the

semantic gap and to answer some symbol grounding questions, which are difficult and still open problems in image interpretation. It provides tools both for knowledge acquisition and representation and for its operational use. It has an important potential in model-based recognition that deserves to be further explored, in particular for medical image interpretation. The framework described in this section focuses on spatial relations, but similar principles can be applied to other types of information that could be involved in image interpretation.

A c k n o w l e d g m e n t s : This work has been partly supported by grants from Region Ile-de-France, GET and ANR.

Re fe r ences

1. S. Harnad, Physica 42, 335 (1990). 2. J. Searle, Behavioral and Brain Sciences 3, 417

(1980). 3. M. Taddeo and L. Floridi, Journal of Experimental

and Theorical Artificial Intelligence (2006). 4. S. Coradeschi

and A. Saffiotti, Robotics and Autonomous Systems 43, 85 (2003), Special issue on perceptual anchoring. Online at http://www.aass.oru.se/Agora/RAS02/.

5. I. Bloch and A. Saffiotti, Some similarities between anchoring and pattern recognition concepts, in AAAI Fall Symposium on Anchoring Symbols to Sensor Data in Single and Multiple Robots Systems, 2001.

6. A. Smeulders, M. Worring, S. Santini, A. Gupta and R. Jain, IEEE Transactions on Pattern Analysis and Machine Intelligence 22, 1349 (2000).

7. D. Roy, Computer Speech and Language 16 (2002). 8. P. Vogt, Artificial Intelligence 167, 206 (2005). 9. A. Cangelosi, Pragmatics and Cognition, Special Is

sue on Distributed Cognition (2006), in press. 10. T. R. Gruber, Towards Principles for the Design

of Ontologies Used for Knowledge Sharing, in Formal Ontology in Conceptual Analysis and Knowledge Representation, eds. N. Guarino and R. Poli (Kluwer Academic Publishers, Deventer, The Netherlands, 1993).

11. P. Zweigenbaum, B. Bachimont, J. Bouaud, J. Charlet and J. Boisvieux, Meth Inform Med 34, p. 2 (1995).

12. A. Jakulin and D. Mladenic, Ontology grounding, in Conference on Data Mining and Data Warehouses, (Ljubljana, Slovenia, 2005).

13. C. Town, Machine Vision and Applications (2006). 14. A. Schreiber, B. D. ans J. Wielemaker and

B. Wielinga, IEEE Intelligent Systems 16, 66 (2001). 15. B. Hu, S. Dasmahapatra, P. Lewis and N. Shadbolt,

24 On the Interest of Spatial Relations and Fuzzy Representations for Ontology-Based Image Interpretation

Ontology-based medical image annotation with description logics, in 15th IEEE International Conference on Tools with Artificial Intelligence, 2003.

16. C. Hudelot, N. Maillot and M. Thonnat, Symbol grounding for semantic image interpretation: from image data to semantics, in Proceedings of the Workshop on Semantic Knowledge in Computer Vision, ICCV, (Beijing, China, 2005).

17. W. Z. Mao and D. A. Bell, Integrating visual ontologies and wavelets for image content retrieval, in DEXA Workshop, 1998.

18. V. Mezaris, I. Kompatsiaris and M. G. Strintzis, Eurasip Journal on Applied Signal Processing 2004, 886 (2004).

19. G. Guerra-Filho and Y. Aloimonos, Towards a sensorimotor wordnet: Closing the semantic gap, in Proceedings of the Third International Wordnet Conference, January 2006.

20. F. Baader, D. Calvanese, D. McGuinness, D. Nardi and P. Patel-Schneider, The Description Logic Handbook: Theory, Implementation and Applications (Cambridge University Press, 2003).

21. M. Aufaure and H. Hajji, Multimedia Information Systems , 38 (2002).

22. K. Petridis, D. Anastasopoulos, C. Saathoff, N. Tim-mermann, I. Kompatsiaris and S. Staab, Engineered Applications of Semantic Web Session (SWEA) at the 10th International Conference on Knowledge-Based & Intelligent Information & Engineering Systems (KES 2006) (2006).

23. I. Bloch, Image and Vision Computing23, 89 (2005). 24. B. J. Kuipers and T. S. Levitt, AI Magazine 9, 25

(1988). 25. J. Freeman, Computer Graphics and Image Process

ing 4, 156 (1975). 26. B. Kuipers, Cognitive Science 2, 129 (1978). 27. C. Hudelot, J. Atif and I. Bloch, Ontologie de rela

tions spatiales floues pour l'interpretation d'images, in Rencontres francophones sur la Logique Floue et ses Applications, LFA 2006, (Toulouse, France, 2006).

28. J. Bateman and S. Farrar, Towards a generic foundation for spatial ontology, in International Conference on Formal Ontology in Information Systems (FOIS-2004), (Trento, Italy, 2004).

29. R. Casati, B. Smith and A. Varzi, Ontological Tools for Geographic Representation, in Formal Ontology in Information Systems, ed. N. Guarino (IOS Press, Amsterdam, 1998) pp. 77-85.

30. O. Dameron, Symbolic model of spatial relations in the human brain, in Mapping the Human Body: Spatial Reasoning at the Interface between Human Anatomy and Geographic Information Science, (University of Buffalo, USA, 2005).

31. M. Donnelly, T. Bittner and C. Rosse, Artificial Intelligence in Medicine 36, l(January 2006).

32. S. Schulz, U. Hahn and M. Romacker, Modeling anatomical spatial relations with description log

ics, in Annual Symposium of the American Medical Informatics Association. Converging Information, Technology, and Health Care (A MI A 2000), (Los Angeles, CA, 2000).

33. J. Atif, H. Khotanlou, E. Angelini, H. Duffau and I. Bloch, Segmentation of Internal Brain Structures in the Presence of a Tumor, in MICCAI, (Copenhagen, 2006).

34. L. Vieu, Spatial Representation and Reasoning in Artificial Intelligence, in Spatial and Temporal Reasoning, ed. O. Stock (Kluwer, 1997) pp. 5-41.

35. S. Dutta, International Journal of Approximate Reasoning 5, 307 (1991).

36. L. A. Zadeh, Information Sciences 8, 199 (1975). 37. D. Dubois and H. Prade, Fuzzy Sets and Systems:

Theory and Applications (Academic Press, New-York, 1980).

38. D. Dubois and H. Prade, Information Sciences 36, 85 (1985).

39. D. Dubois, H. Prade and R. Yager, Merging Fuzzy Information, in Handbook of Fuzzy Sets Series, Approximate Reasoning and Information Systems, eds. J. Bezdek, D. Dubois and H. Prade (Kluwer, 1999)

40. I. Bloch, IEEE Transactions on Systems, Man, and Cybernetics 26, 52 (1996).

41. P. da Costa et al. (Eds), Proceedings of the ISWC Workshop on Uncertainty Reasoning for the Semantic Web, 2005).

42. E. Sanchez (ed.), Fuzzy Logic and the Semantic Web (Elsevier, 2006).

43. Z. Ding, Y. Peng and R. Pan, A Bayesian Approach to Uncertainty Modelling in OWL Ontology, in International Conference on Advances in Intelligent Systems-Theory and Applications (AISTA20O4), (Luxembourg-Kirchberg, Luxembourg, 2004).

44. Y. Yang and J. Calmet, Ontobayes: An ontology-driven uncertainty model, in International Conference on Intelligent Agents, Web Technology and Internet Commerce (IAWTIC05), 2005.

45. S. Holldobler, T. Khang and H. Storr, Proceedings InTech/VJFuzzy 2002, 25 (2002).

46. Y. Li, B. Xu, J. Lu, D. Kang and P. Wang, A family of extended fuzzy description logics, in 29th Annual International Computer Software and Applications Conference (COMPSAC'05), (IEEE Computer Society, Los Alamitos, CA, USA, 2005).

47. G. Stoilos, G. Stamou and J. Pan, Handling imprecise knowledge with fuzzy description logic, in International Workshop on Description Logics (DL 06), (Lake District, UK, 2006).

48. U. Straccia, Description logics with fuzzy concrete domains, in 21st Conference on Uncertainty in Artificial Intelligence (UAI-05), eds. F. Bachus and T. Jaakkola (AUAI Press, Edinburgh, Scotland, 2005).

49. M. d'Aquin, J. Lieber and A. Napoli, Etude de quelques logiques de description floues et de formal-ismes apparentes, in Rencontres Francophones sur la

Isabelle Bloch, Celine Hudelot and Jamal Atif 25

Logique Flout et ses Applications, (Nantes, France, 2004).

50. U. Straccia, A fuzzy description logic for the semantic web, in Fuzzy Logic and the Semantic Web, ed. E. SanchezCapturing Intelligence (Elsevier, 2006) pp. 73-90.

51. C. Rosse and J. L. V. Mejino, Journal of Biomedical Informatics 36, 478 (2003).

52. J. Atif, C. Hudelot, I. Bloch and E. Angelini, From Generic Knowledge to Specific Reasoning for Medical Image Interpretation using Graph-based Representations, in International Joint Conference on Artificial Intelligence IJCAI'07, (Hyderabad, India, 2007).

53. O. Colliot, O. Camara and I. Bloch, Pattern Recognition 39, 1401 (2006).

26

A Technique for Creating Probabilistic Spatio-Temporal Forecasts

V. Lakshmanan

University of Oklahoma and National Severe Storms Laboratory

E-mail: lakshmanOou.edu

Kiel Ortega

School of Meteorology University of Oklahoma

Keywords: Probability; Atmospheric Science

Probabilistic forecasts can capture uncertainty better and provide significant economic benefits because the users of the information can calibrate risk. For forecasts in earth-centered domains to be useful, the forecasts have to be clearly demarcated in space and time. We discuss the characteristics of a good probability forecast - reliability and sharpness - and describe well-understood techniques of generating good probability forecasts in the literature. We then describe our approach to addressing the problem of creating good probabilistic forecasts when the entity to be forecast can move and morph. In this paper, we apply the technique to severe weather prediction by formulating the weather prediction problem to be one of estimating the probability of an event at a particular spatial location within a given time window. T h e technique involves clustering Doppler radar-derived fields such as low-level shear and reflectivity to form candidate regions. Assuming stationarity, the spatial probability distribution of the regions is estimated, conditioned based on the level of organization within the regions and combined with the probability that a candidate region becomes severe. For example, the probability that a candidate region becomes tornadic is estimated using a neural network with a sigmoid output node and trained on historical cases.

1. Motivation

A principled estimate of the probability that a threat will materialize can be more useful than a binary yes/no prediction because a binary prediction hides the uncertainty inherent in the data and predictive model from users who will make decisions based on that prediction. A principled probabilistic prediction can enable users of the information to calibrate their risk and can aid decision making beyond what simple binary approaches yield.1 Probabilistic forecasts can capture uncertainty better and provide significant economic benefits because the users of the information can calibrate risk.

Techniques to create good probabilistic forecasts are well understood, but only in situations where the predictive model is a direct input-output relationship. If the threats in consideration move and change shape, as with short-term weather forecasts, the well-understood techniques can not be used directly. For forecasts in earth-centered domains to be useful, the forecasts have to be clearly demarcated in space and time. This paper presents a data mining approach to address the problem of creating principled probabilistic forecasts when the entity to be forecast can

move and change shape. The rest of the paper is organized as follows.

The characteristics of good probabilistic forecasts, and standard data mining approaches to create such probabilistic forecasts are described in Section 2. The limitations of the standard data mining approaches in creating clearly demarcated forecasts in space and time are explained, and the first part of our predictive model (to create principled spatial forecasts) is explained in Section 3. The second part of model, to create principled temporal forecasts that can be tied with the spatial forecast, is explained in Section 4. The way of tying together the two probabilities is explained in Section 5. Results of applying this approach to predicting liquid content are described in Section 6.

2. Probabilistic Forecasts

A good probability forecast has two characteristics (See Figure 1): (a) it is reliable. For example, 30% of all the times that the threat is forecast as 30% likely to occur, the threat should occur and (b) it is sharp, i.e. the probability distribution function of the forecast probabilities should not be clustered around

V. Lakshmanan and Kiel Ortega 27

P(f) ,/

Fig. 1. A good probability forecast needs to be both reliable (left) and sharp (right).

the a priori probability. Instead, there should be lots of low and high-probability events and relatively few mid-probability events.

In many cases, there are three probabilities of interest: (a) the probability that an event will occur, (b) the probability that the event will occur at a particular location (c) the probability that the event will occur at a particular location within a specified time window. When stated like this, most decision makers will aver that it is the third probability that is of interest to them,2 but that is not what they are commonly provided. Even though research studies implicitly place spatial and temporal bounds on their training set, it has long been unclear how to explicitly form the spatial and temporal variability of the probability field. Thus, if probabilities are presented to decision maker, those probabilities are usually of the first type. The probability that an event will occur is commonly estimated through the formation of a mapping between input features and the desired result on a set of historical cases .3 This mapping function may take the form of a neural network, a support vector machine or a decision tree. In the case of a neural network, the choice of a sigmoid function as the output node is sufficient to ensure that the output number is a probability, provided that the training set captures the a priori probabilities of the true input feature space.4 In the case of maximum marginal methods such as support vector machines or bagged, boosted decision trees, scaling the output using a sigmoid function yields the desirable property that the output is a probability .5

3. Spatial probability forecast

Thus, techniques to estimate the probability of an event are well understood. The estimation of spa

tial probabilities when the threat under consideration is stationary can be performed using the standard formalisms, for example,6 on soil variability. If the threat is stationary, then the value of each of the input fields at a pixel or statistical or morphological values computed in the neighborhood of a pixel may be used as the input features to a classifier. If the classifier is a neural network with a sigmoid output node or a Platt-scaled maximum marginal technique, then the output of the classifier yields a spatial probability.

One approach to probabilistic forecasts when the threat areas are not stationary is to use kriging,7 a geospatial interpolation approach. Naturally, such an approach is limited to studies where the kriging resolution can be finer than the rate of transformation. This is a condition that exists in slow-moving systems such as diseases (studied by8), but not in fast moving systems such as thunder storms.

For fast moving threats, no principled approach to estimating probability fields exists. This is because the input features at a point in space and time affect the threat potential at a different point N minutes later. Thus, a simple input/output mapping is insufficient because which location will be affected, and when it will be affected needs to be known. Yet, in practical situations, the time and location that will be affected is not known with certainty. Consequently, it is necessary to assume a spatial probability distribution associated with the locations that will be affected at a given time instant. Once this second probability is introduced, prior work on principled probability estimates is no longer applicable. A new formulation is needed in the development of a spatiotemporal framework to estimate the probability of occurrence of moving or spreading threats to account for the dynamics of spatial probability field

28 A Technique for Creating Probabilistic Spatio- Temporal Forecasts

of area at risk over time. One can develop the spatiotemporal formulation

first by building an ontology of threat precursors with identified features. Each of the features signals a probability distribution of threats in space and time. Threat probabilities can be combined from multiple features and dynamics among these features to estimate the probability of a threat occurring at a particular location within a specific time window. One factor that needs to be considered is that even if the spatial and temporal distribution of the locations that will be affected by a particular feature is estimated, whether the feature will lead to the threat still needs to be estimated. This, of course, is a problem that has been thoroughly addressed in the literature on data mining algorithms.

The formulation can be easily described by comparing to an analogue of a set of billiards balls (See Figure 2) and asking: what is the future position of any ball on the table? Each grid point in the set of continuous fields will be considered akin to a ball, and the driving force of a set of points, such as a storm cell boundary will be considered akin to the ring.

9l\

Fig. 2. The probabilistic formulation assumes rigid bodies operated upon by a driving force that permits individual degrees of motion.

We will make the simplified assumption of space-time stationarity: the probability distribution of a particle in space is identical to the probability distribution of the particle in time (See Figure 3). Loosely, this assumption is similar to assuming that the probability distribution obtained by tossing 50 coins is the same as the probability distribution obtained by

tossing the same coin 50 times. This assumption has to be made on faith in situations where one does not have 50 coins; we will never have 50 identical thunderstorms or hazardous events.

By making the assumption of space-time stationarity and from the motion estimated by the hybrid technique described in the next section, one can formulate the spatial probability distribution of a ball on the billiards table.

If motion estimates of a moving potential threat are available, the variability of the motion estimates themselves can be used to gauge the probability distribution of the motion estimates (Figure 3). Under stationarity assumptions, the historical probability distribution can be used to estimate the future spatial probability distribution of this current billiard ball.

There is one further complication, however. The future expected position of a billiard ball depends on how tightly the balls are packed or on how much control the forcing function exerts. If the balls are loosely packed, the movements of different balls are independent. If the balls are tightly packed, all the balls move together. The actual movement of the balls will be somewhere in between the two extremes. The actual balance depends on the problem at hand and will likely be estimated from the data.

If X is the event of interest (such as the probability that there will be a tornado at a particular location within the next 30 minutes), then simple Bayesian analysis yields:

P(X) = P(X\packed) x P(packed)+ P(X\notpacked) x (1 - P(packed)) ^ '

The first quantity, P(X\packed): the probability of lightning given that all the grid points associated with a thunderstorm move together - can be estimated quite readily using numerical integration of the probability distribution corresponding to motion vectors that will yield a lightning strike at this location based on where the threats are currently present.

The third quantity, P(X\notpacked): the probability of lightning given that the grid points associated with a thunderstorm move independently (in other words, the state transition from a cell to its neighbor is independent of other cells in a thunderstorm) can be computed through probability analy-

V. Lakshmanan and Kiel Ortega 29

p=P(ui)P(vi;

*x

/ Fig. 3. By assuming that the probability distribution in space is identical to the probability distribution in time (stationarity), the estimated probability distribution in time from historical data can be used to create a spatial probability field. The figure shows how the data values of one component of the velocity vector in time (top) is used to estimate a probability distribution of that component (bottom left). This probability distribution, though estimated from the time field, is considered the spatial distribution in order to formulate the probabilistic location of a grid cell (bottom right) at time T into the future.

Sis:

P{X\notpacked) = 1 - ( (1 - P{xl))x (1-P(x2))x

(2)

(l-P(xN)))

where P(xl), P(x2), etc. are the spatial probabilities that the individual grid points will end up impacting this location. The weighting factor P(packed) needs to be estimated from historical data. We anticipate that the P (packed) will depend upon the type of event under consideration and needs to be estimated from the data - a single number may not work for all problems.

4. Estimating Movement

There are two broad methods of estimating movement: (a) optical flow methods including cross-correlation and spectral methods that minimize a displacement error with sub-grids of two frames of a sequence of images9'10 (b) object tracking methods that identify objects in the frames of a sequence and correlate these objects across time.11 The object tracking methods provide the ability to track small-scale features and extract trends associated with those features but miss large-scale transformations. In comparison, the optical flow methods yield more accurate estimates over large scales, but miss fine-scale movements and do not provide the ability to extract trends.12

Nevertheless, trends are very important in a

data-mining predictive approach. A measured value may bear no correlation with whether a threat will materialize, but sharp increases or decreases of some measurements are often reliable indicators of coming threats. Thus, even though optical flow methods can yield more accurate estimates of movement than object-tracking methods, optical flow methods cannot be djrectly applied to problem domains in which object-specific motion and trend information is critical.

One way to achieve the high accuracy of optical flow methods while retaining the trending capability of object-based tracking is to create a hybrid technique. It is possible to find objects by clustering the input features using a multi-scale hierarchical approach.12 This hybrid technique will not correlate objects across frames as in a typical object-tracking scenario. Instead, the objects should be correlated in the current frame with the images in the previous frame using the current shape of the object itself as the domain in which displacement error will be minimized. To extract trends, the object domain can then be simply displaced by the temporally adjusted amount in previous frames and the change in statistical properties of the object computed and used for prediction. Such a hybrid technique, as Figure 4 illustrates, neatly sidesteps the problems associated with splitting and merging that are commonly associated with object-tracking methods. Motion estimates from pairs of frames are smoothed across time using a Kalman filter.13

30 A Technique for Creating Probabilistic Spatio- Temporal Forecasts

• & ,

o O

Fig. 4. A hybrid tracking technique of correlating objects in the current frame to pixels in the previous frame can yield trend information as well as the high accuracy associated with optical flow methods.

Fig. 5. Top row: Values of Vertical Integrated Liquid (VIL) in kg/m2 and reflectivity (Z) in dBZ at T= t 0 - Middle row: Probabilistic forecast of VIL I 20, Z I 0, Z i 30 at T=4 0 4- 30 Bottom row: Actual values of VIL and Z at T = t 0 + 30

5. Crea t ing a spa t io- tempora l probabi l i ty forecast

In the short term (under 60 minutes), extrapolation forecasts of storm location often provide reasonable results. Therefore, a data mining approach to probabilistic forecasts of severe weather hazards holds promise of being able to predict where already-occurring severe weather is likely to be. At the same time, research studies have shown that it Is possible to predict, with a high degree of skill, which storms are likely- to initiate lightning (one measure of storm severity) and where new storms are likely to form. What has been missing Is a formal mechanism of combining these two disparate sets of studies. Our

spatlotemporal formalism provides the framework to combine the two probabilities to yield, for example, a lightning probability forecast.

The suggested approach is to apply the various components described above to train the system as follows:

® Cluster the input remotely sensed data into candidate regions

m Using the hybrid motion estimation technique, associate current severe weather to threat signals that occurred N minutes ago.

® Train the associated threat signals to the severe weather activity using a neural network capable of providing predictive probabilities.

V. Lakshmanan and Kiel Ortega 31

Probabilistic Forwcasf Reiiabiiity

t o

0.3 0 4 0.5 0.6 0.7 Forecast probability

Reliability-

sharpness of Probabilistic Frxecas!

01 02 03 04 05 08 07 OS 0.9 1 Forecast Probabtlrty

Sharpness

Fig. 6. Reliability and sharpness of Vertical Integrated Liquid (VIL in kg/m2) and radar reflectivity (Z in dBZ) forecasts for a Mar 8, 2002 case in the central United States.

• Use the spatiotemporal framework and the motion estimates derived from the hybrid technique to estimate spatial probabilities.

• Estimate the weighting factor P (packed) to create optimal (in terms of reliability and sharpness) probability factors.

The trained model and pattern recognition techniques can be applied in real-time on routinely arriving satellite, radar and model data to predict up to N minutes into the future. The resulting probabilistic forecasts can be evaluated using three metrics: (a) reliability (b) sharpness (c) Radar operating characteristics (ROC) curves, created by setting risk thresholds that vary from 0 to 1 and determining the probability of detection and false alarm rate on a grid-point by grid-point basis.

6. Results and Conclusion

The technique of the paper was applied to predicting the spatial distribution of vertical integrated liq-uid(VIL;14) and radar reflectivity 30 minutes into the

future. Examples of the original, forecast and actual images are shown in Figure 5 The reliability and sharpness diagrams when evaluated over a 6 hour period is shown in Figure 6.

Reflectivity forecasts were sharp, but not reliable. The reflectivity forecasts were usually underestimates of the true probability. The VIL probability forecasts were both reliable and sharp, so a probabilistic forecast of VIL would be of high utility to users of weather information. The lead time, of 30 minutes, is higher than possible using deterministic forecast techniques. Thus, the technique described in this paper works for severe weather situations (such as VIL), but not for any weather (as highlighted by the poor performance on reflectivity). Future work will involve testing against more useful diagnostics of severe weather - the initiation of lightning and the potential for a tornado.

Acknowledgments

Funding for this research was provided under NOAA-OU Cooperative Agreement NA17RJ1227. The statements, findings, conclusions, and recommendations are those of the authors and do not necessarily reflect the views of the National Severe Storms Laboratory (NSSL) or the U.S. Department of Commerce.

References

1. A. H. Murphy, The value of climatological categories and probabilistic forecast in the cost-loss ratio situations, Monthly Weather Review , 803 (1977).

2. I. Adrianto, T. M. Smith, K.A.Scharfenberg and T. Trafalis, Evaluation of various algorithms and display concepts for weather forecasting, in 21st Int'l Conf. on Inter. Inf. Proc. Sys. (IIPS) for Meteor., Ocean., and Hydr., (Amer. Meteor. Soc, San Diego, CA, Jan. 2005).

3. M. Richard and R. P. Lippman, Neural network classifiers estimate bayesian a posteriori probabilities, Neural Computation 3, 461 (1991).

4. C. Bishop, Neural Networks for Pattern Recognition (Oxford, 1995).

5. J. Piatt, Advances in Large Margin Classifiers (MIT Press, 1999), ch. Probabilistic outputs for suppot vector machines and comparisons to regularized likelihood methods.

6. J. Prevost and R. Popescu, Constitutive relations for soil materials, Electronic Journal of Geotechnical Engineering (1996).

32 A Technique for Creating Probabilistic Spatio- Temporal Forecasts

7. M. Oliver and R. Webster, Kriging: a method of interpolation for geographical information systems, Int. J. Geographical Information Systems 4, 313 (1990).

8. P. Goovaerts, Geostatistical analysis of disease data: visualization and propagation of spatial uncertainty in cancer mortality risk using poisson kriging and p-field simulation, Int. J. Health Geography 5 (2006).

9. J. Barron, D. Fleet and S. Beauchemin, Performance of optical flow techniques, Int'l J. Comp.Vis. 12, 43 (1994).

10. J. Tuttle and R. Gall, A single-radar technique for estimating the winds in tropical cyclones, Bull. Amer. Met. Soc. 80(Apr. 1999).

11. M. Dixon, Automated storm identification, tracking and forecasting - a radar-based method, P h D thesis, University of Colorado and National Center for Atmospheric Researchl994.

12. V. Lakshmanan, R. Rabin and V. DeBrunner, Multi-scale storm identification and forecast, J. Atm. Res. , 367(July 2003).

13. R. Kalman, A new approach to linear filtering and prediction problems, Trans. ASME - J. Basic Engr. , 35(March 1960).

14. D. R. Greene and R. A. Clark, Vertically integrated liquid water - A new analysis tool, Mon. Wea. Rev. 100, 548 (1972).

PART C

Biometrics

An Efficient Measure for Individuality Detection in Dynamic Biometric Applications

B. Chakraborty

Faculty of Software and Information Science Iwate Prefectural University

Japan 020-0193 E-mail: [email protected]

Y. Manabe

Graduate School of Software and Information Science Iwate Prefectural University

Japan 020-0193

Individuality detection from dynamic information is needed to counter the threat of forgery in biometric authentication system. In this work a new similarity measure has been proposed for discrimination of genuine online handwriting from forgeries from dynamic time series signal. The simulation experiment shows that the measure is more effective compared to DP matching for measuring intraclass and interclass variation among several writers for writer verification by online handwriting.

Keywords: Dynamic Biometrics, Individuality Detectio

1. Introduction

Biometric technologies are rapidly gaining importance with the growing need of information security. Current biometric technologies for personal identity verification are mostly based on static information such as face, fingerprint, iris, blood vessel etc. Due to increasing threat of forgery it is becoming necessary to extend biometric technologies to include dynamic information like hand gesture, body movement or online handwriting.

The main point of an efficient authentication system is to minimize false acceptance rate as well as false rejection rate. One of the key problems in biometrics is the intraclass variation. Nearly no two structures are completely identical although they are of same origin. To achieve a successful identity detection, intra class variation should be as minimum as possible while inter class variation being maximum. In the case of dynamic biometric technologies such as writer verification by on-line handwriting, one generally has to deal with multivariable time series to include dynamical information. In the process of verification, we need to find out similarity or dissimilarity between two time series (signal) in order to discriminate between genuine handwriting with forgeries. The similarity measures used for static feature values (expressed as multidimensional vectors) in classification/identification problems can not be used directly for calculating similarities between two

Translation Error, Similarity Measure

multivariate time series signal. In this work a similarity measure for compar

ing two trajectories has been proposed which can be applied for individuality detection by distinguishing online handwriting of different persons based on dynamic information. The proposed measure is based on translation error defined by Wayland in 1 for detecting determinism in a time series. Simulation experiments has been done for the problem of writer identification by online handwriting and the usefulness of the measure has been shown in comparison to popular DP matching method in measuring intraclass and interclass variation among several writers. The next section represents a brief introduction to writer verification by online handwriting followed by proposed algorithm of measuring similarity between trajectories generated by pen-tip movement on writing tablet. Section 3 contains simulation experiments and results. Section 4 represents conclusion and future direction.

2. Dynamic Biometrics: Online Handwriting

Online signature verification as a means of person identification is under intense investigation for a long time and a lot of research paper has been published in this area 23 .4 Automatic writer authentication methods using online handwriting are broadly classified

36 An Efficient Measure for Individuality Detection in Dynamic Biometric Applications

into two categories : function based and parameter based .5 In parametric approaches, only the parameters abstraced from the signal are used. Though they are simpler with high computation speed but task of selecting right parameters is difficult and during the processing the dynamical information is lost. In functional approach the two complete signals are compared which generally yields better results.6 For the past two decades the use of DTW (Dynamic Time Warping) based on DP matching algorithm to find the best matching path, in terms of the least global cost, between an input signal and a template, has become a major technique in signature verification,7

8 Though DTW is quite sucessful it has two main drawbacks: heavy computational load and warping of forgeries. Other popular recent methods are based on HMM.9

In this paper we consider online handwriting as a multivariate time series signal generated by the pen movement on writing pad, the variables are x,y coordinates of pen-tip movement, writing pressure, pen inclination with respect to x and y co-ordnates etc. The path of the hand movement during pen-up position is also interpolated in the time series along with the movement of the pen-down position. A similarity measure for measuring similarity between generated time series signals corresponding to two samples of same writing has been proposed and is explained in the following section.

2.1. Proposed Algorithm of Measuring Similarity

The time series signal generated by online handwriting is considered to be originated from the dynamics of the hand movement. The trajectories of a writing piece of sample should be similar for the same writer but different for different writers as handwriting is considered to be dependent on individual. Here we propose a novel measure for evaluating similarity of time series signals based on Translation Error, originally proposed by Wayland. Wayland test1 is a widely used method for detecting determinism from a time series. We have modified Wayland's algorithm in order to evaluate similarity between handwriting time series. The proposed measure is based on the following concepts.

by delay coordinate embedding method from a online handwriting signal.

(2) Constructed trajectory reflects the individuality of the time series.

(3) Several multi-dimensional trajectories constructed from the several handwriting time series of same characters generated by the specific writer have almost same dynamics within a certain error.

(4) Thus, similarity between time series signals can be evaluated by translation error of constructed trajectories.

A deterministic time series signal [^(i)]^!, can be embedded as a sequence of time delay co-ordinate vector vs(t), known as experimental attractor, with an appropriate choice of embedding dimension m and delay time r for reconstruction of the original dynamical system as follows:

vs(t + l) = f(vs(t)) vs(t) = \s(t), s(t + r),...,s(t + (m- 1 ) T ) ) ]

where / denotes reconstructed system having one-to-one correspondence with original system. Though the present work is not concerned with detecting determinism in time series, delay vector embedding is used here for extracting the local wave pattern of the time series as shown in Fig. (1) which act as a local feature.

• t v. (1)1

vf(2) 4 4 4

v.O) •

(1) Multi-dimensional trajectory can be constructed Fig. 1. Delay Coordinate Embedding and Local Wave Form

B. Chakraborty and Y. Manabe 37

•

Fig. 2. Translation in Multi-Dimensional Phase space

We propose a measure for calculating distance between trajectories for a th sample and b th sample of the same piece of writing from the translation of delay vectors in the reconstructed space based on translation error defined in Wayland test as follows:

(1) Si(t) is defined as the time series signal generated from online handwriting and vSi (t) is the corresponding embedded vector for i th sample, vSa (t) and vSb(t) denote the delay vectors respectively for a th and b th samples.

(2) A random vector vSa(k) is chosen from vSa(t). Let the nearest vector of vSa(k) from vSb(t) be vSb(k'), where \k — k'\ < Tth, where Tth denotes a threshold value ensuring a small region for k nearest neighbours (shown in Fig. (2) )

(3) For the vectors (vSa(k) and vSb(k')), the transition in the each orbit after one step are calculated as follows;

V3a(k) = vSa(k+l)-vSa(k), (1)

VSb(k') = vSb(k' + l)-vSb(k'). (2)

(4) Translation error etrans, is calculated from VSa(k) and VSb(k') as

i(\v.A*)-v\ , \v.>W)-v\ etrans ~ n\ |y-| "•" ly-i /> V"/

where V denotes average vector between VSa (k)

andVS(1(fc')-

(5) etrans is calculated for L times for different selection of random vector vSa (k) and the median of e\rans (i = 1,2,..., L) is calculated as

M(etrans) = Median(e\rans,..., e tL

rans). (4)

The final translation error Etrans is calculated by taking the average, repeating the procedure M times to supress the statistical error generated by random sampling in the previous step.

1 M

Et will be different if we interchange a th and 6 th sample in step 2 of the above algorithm i,e. random vector is chosen from delay vectors of b th sample. Wayland demonstrated that the translation error Et tends to 0 if the time series is deterministic and tends to 1 if it is random.

2.2. Measure for Individuality Detection

The translation error Et defined above is used here to define a measure of variation between same and different writer's handwriting samples as follows:

(1) Intra sample variation of similarity measure from a single writer.

(2) The distance between intra sample distribution and inter sample distribution of several writers.

Now for calculation of (1) online handwriting samples for a particular piece of writing for a particular writer has to be taken iV times. The translation error between (N x N — N) samples has to be computed to get the distribution of individual variation for a particular writer. The co-efficient of variation for N samples is defined as

CV = °Sim

fJ-Sim — N(N-l) 2 ^ i = l a i m ,

asim = y N(N-I) ]LiIi (Sinn - ^Sim)2

where er| im and [isim denote the variance and mean of the distribution of translation error for N samples.

For calculation of (2), online handwriting sample for a particular piece of writing for several (K) writers have to be taken N times. The difference between the distribution of translation error for same writer

38 An Efficient Measure for Individuality Detection in Dynamic Biometric Applications

(number of samples N(N — 1), group P) and the distribution of translation error for this writer and the other (K-l) writers ( number of samples NK, group Q) are calculated by Fratio as follows:

MS- . — bbinUr lvl Winter — 2 —1

(Lf?- , — , ssi"i'" M&tntra — N(N-l) + NK-2

SSinter = N(N - l)(fiP - fiPQ)2 + NK^Q -

V-PQ?

SSintra = N(N - l)a2P + NKa2

Q

UP, HQ, fipQ represent average of group P, group Q and group P and Q pulled together as one group while op and OQ represent variance of group P and Q respectively. For correct identification of a writer, MSinter should be as high as possible while MSintra

should be as low as possible, making Fr high for good authetication results.

3. Simulation Experiment and Results

3.1. Data Preparation

In order to ebvaluate the efficiency of the proposed similarity measure a small scale simulation experiment has been done with 5 (A, B, C, D, E) writers, all of them wrote the word 'Software' in katakana, a Japanese alphabet system known to be difficult for capturing individuality of handwriting. A piece of sample in shown in Fig. (3) The writers used WACOM tablet " intuose 3" for writing and each writer wrote 10 times in two sittings producing 100 samples per writer. In this study we used only x-y coordinates of the pen tip movement. WACOM tablet "intuosjj 3" produces 200 points /sec which is too many for computation.

Fig. 3. Handwriting Character samples

3.2. Pre-processing

A preprocessing has been done to sample non equidistant points such that points in the curve region is taken into account more than the points in the relatively straight portion. If the angle between the line joining ( xt-\,yt-\) and (xt,yt) and the line joining (xt-i,yt-i) and (xt+i,yt+i) is less than a certain threshold (Qth) the point (xt,yt) is dropped. Secondly the time series values of the co-ordinates x(t) and y(t) are normalized to lie between 0 and 1.

3.3. Simulation Experiment

Translation error (TE) and DP matching distance values are calculated for 90 trajectory pairs corresponding to 10 samples for each particular writer and 400 trajectory pairs for a particular writer and other writers. That is here, group P has 90 samples and group Q had 400 samples. For calculation of translation error (TE) the delay vectors of the trajectories are constructed with simple assumption of embedding dimemsion m = 3 and delay T = 1. The other parameters are set as follows:

Qth = 1.0, L = 50 and M = 10 (in step 5 of proposed algorithm).

3.4. Simulation Results

Table. 1 represents the intra writer variability values using translation error and DP matching as the measures. The ratio of standard deviation and average value of the distribution of both the measurements is taken for calculating the index. It is shown that for both the time series, translation error is a better measure for identity detection. Table. 2 represents the avarage Fr of each writer in comparison with rest of the writers. Here also translation error is found to be better than DP matching. Specially for writers C and D the low values of Frati0 using DP measure for x co-ordinate time series indicate that intra writer variation compared to inter writer variation is high enough to discover any individuality and thus can not be used for personal identification. Fr

for y(t) values seems better but FT based on T E measure for both the time series are far better than those values based on DP measure. Thus translation error can be used to detect the individuality of writer C and D as Fr corresponding to TE of writer C and

B. Chakraborty and Y. Manabe 39

D are quite high. In fact in our experiment writer C

and D are novice in using writing tablet and their

sample writings show greater variation from sample

to sample compared to writer A, B and E. It seems

tha t in spite of variation in sample writing, T E is

bet ter measure than D P to identify the particular

writer.

4 . C o n c l u s i o n

In this work translation error (TE) , a measure for

detecting determinism in a t ime series, is used to

define a distance measure and an algorithm for cal

culating the distance between the trajectories of two

online handwrit ing samples of same piece of writing

has been proposed. The effectiveness of the measure

for possible use as a dynamic biometric technolgy for

identity detection has been examined in comparison

to the one of the popular technique, D P matching

algorithm, in writer verification problem with online

handwrit ing.

Table 1. Intra writer variability for different writers

Writer A Writer B Writer C Writer D Writer E Average

Time Series X TE

0.096 0.152 0.160 0.118 0.126 0.130

DP 0.213 0.232 0.334 0.223 0.237 0.248

Time Series Y TE

0.104 0.139 0.126 0.102 0.127 0.120

DP 0.252 0.301 0.212 0.193 0.253 0.242

In the simple experiment conducted here it is

found that translation error based similarity or dis

tance calculation of trajectories are bet ter than D P

matching based measure. This measure can also be

used for feature evaluation for multivariate time se

ries. In our experiment the values in the Tables in

dicate tha t x trajectories are bet ter than y trajec

tories for identity detection. This measure also can

be extended to consider multivariate t ime series for

authentication problems using all the t ime series in

formation combined together. At present we are con

ducting experiments with a larger number of writers

and would like to apply this measure for wri ter ver

ification problem from online handwrit ing.

Table 2. FTatio of one writer with rest of the writers

Writer A Writer B Writer C Writer D Writer E Average

Time Series X TE

346.88 210.11 104.76 167.06 244.39 212.84

DP 178.24 185.85 4.68 0.14 85.04 90.79

Time Series Y TE

235.61 144.34 91.88 142.75 92.91 141.30

D P 104.11 141.84 107.30 39.49 210.85 120.72

R e f e r e n c e s

1. R. Wayland et al, Physical Review Letters 70(5), 580 (1993).

2. V. S. Nalwa, Proc. of IEEE85,215 (1997). 3. A. Khalmatov and B. Yanikoglu, Pattern Recogni

tion Letter 26(15),2400 (2005). 4. h t t p : / / i r i s . u s e . e d u / V i s i o n - N o t e s /

Bibliography/char1012.html 5. F. Lecrec and R. Plamondon,International Journal

of Pattern Recognition and Artificial Intelligence, 8(3), 643 (1994).

6. R. Plamondon and G. Lorette,Pattern Recognition 22, 107 (1989).

7. P. Zhao et al., IEICE Trans. Inf. & Syst E79-D(5),535 (1996).

8. H. Feng and C. C. Wah, Pattern Recognition Letter 24(16) ,2943 (2003).

9. M. M. Shafie and H. R. Robiee.Proc. of ICDAR 2003 (2003).

40

Divide-and-Conquer Strategy Incorporated Fisher Linear Discriminant Analysis: An Efficient Approach for Face Recognition

S. Noushath*, G. Hemantha Kumar and V. N. Manjunath Aradhya

Department of Studies in Computer Science University of Mysore

Mysore-570006, INDIA E-mail: nawali-naushad@yahoo. co. in*

P. Shivakumara

Department of Computer Science School of Computing

National University of Singapore Singapore

E-mail: [email protected]

Fisher linear discriminant analysis (FLD) is one of the most popular feature extraction methods in patter recognition and can obtain a set of so-called projection directions such that the ratio of the between-class and the within-class scatter matrices reaches its maximum. However in reality, the dimension of the patterns will be so high t h a t the conventional way of obtaining Fisher projections makes the computational task a tedious one. To alleviate th is problem, in this paper, divide-and-conquer strategy incorporated FLD (dcFLD) is presented with two objectives: one is to sufficiently utilize the contribution made by local parts of the whole image and the other is to still follow the same simple mathematical formulation as FLD. In contrast to the traditional FLD method, which operates directly on t he whole pattern represented as a vector, dcFLD first divides the whole pattern into a set of subpatterns and acquires a set of projection vectors for each partition to extract corresponding local sub-features. These local sub-features a re then conquered to obtain global features. Experimental results on several image databases comprising of face and object reveal the feasibility and effectiveness of the proposed method.

Keywords: Divide-and-conquer Strategy; Fisher Linear Discriminant Analysis; Principal Component Analysis; Face Recognition; Object Recognition

1. Introduction

Principal Component Analysis (PCA)1 and Fisher Linear Discriminant analysis (FLD),2 respectively known as eigenface and fisherface method, are the two state-of-the-art subspace methods in face recognition. Using these techniques, a face image is efficiently represented as a feature vector of low dimensionality. The features in such subspace provide more salient and richer information for recognition than the raw image. It is this success, which has made face recognition (FR) based on PCA and FLD very active, although they have been investigated for decades.

To further exploit the potentiality of PCA and FLD methods, new techniques (Refs. 3,4 are two similar methods found in the literature referred respectively as 2DFLD and 2DLDA. In all our discussions, whenever we refer to,4 it also implies3 and vice-versa.) called 2DPCA5 and 2DLDA3,4 were respectively proposed. Although these methods proved to be efficient in terms of both computational time and

accuracy, a vital unresolved problem is that these methods require huge feature matrix for the representation. To alleviate this problem, the (2£>)2LDA6

method was proposed which gave the same or even higher accuracy than the 2DLDA method. Further, it has been shown in6 that, 2DLDA essentially works in the row-direction of images. In this way, alternative-2DLDA (A2DLDA)6 was also proposed which works in the column direction of images. By simultaneously combining both row and column directions of images, 2-Directional 2DLDA, i.e. (2D)2LDA was proposed. Unlike 2DLDA and A2DLDA methods, the DiaFLD7

method seeks optimal projection vector along the diagonal direction of face images by enlacing both row and column information at the same instant. Furthermore, 2DFLD and DiaFLD were combined together in7 to achieve efficiency in terms of both accuracy and storage requirements.

In spite of the success achieved due to the above mentioned variations3'4'6'7 of the original FLD method, there are still some serious flaws that needs

S. Noushath, G. Hemantha Kumar, V. N. Manjunath Aradhya and P. Shivakumara 41

to be addressed. The main disadvantage of the FLD method is that, when the dimensionality of given pattern is very large, extracting features directly from these large dimensional patterns cause some processing difficulty such as the computational complexity for large scale scatter matrices constructed by the training set. Furthermore, due to utilizing only the global information of images, this is not effective under extreme facial expression, illumination condition and pose, etc. Hence, in this paper, we have made a successful attempt to overcome the aforesaid problems by first partitioning a face image into several smaller sub-patterns (sub-images), and then a single FLD is applied to each of them. It was reported that the changes due to lighting condition and facial expressions emphasize more on specific parts of the face than others.8 Consequently, variations in illumination or expression in the image will only affect some sub-images rather than the whole image, and therefore the local information of a face image may be better represented. Furthermore, the sub-pattern dividing process can also help in increasing the diversity, making the common and class-specific local features be identified more easily.9 Meaning that, the different contributions made by different parts of images are more emphasized which in turn helps to enhance the robustness to both illumination and expression variation. These are the reasons which motivated us to go for subpattern dividing process to overcome the aforementioned drawbacks of the original FLD method.

In the first step of this method, an original whole pattern denoted by a vector is partitioned into a set of equally sized sub-patterns in a non-overlapping way and all those sub-patterns sharing the same original feature components are respectively collected from the training set to compose a corresponding sub-patterns training set. In the second step, FLD is performed on each of such sub-patterns training set to extract its features. At last, a single global feature is synthesized by concatenating each sub-patterns FLD projected features together. Finally, a nearest neighbor classifier is used for subsequent recognition. The experiment on different image databases will provide the comparison of the classification performance of dcFLD with other linear discrimination methods.

The rest of the paper is organized as follows: The algorithm is detailed in section 2. Experiments

are carried out in section 3 to evaluate dcFLD and other subspace analysis methods using wide range of image databases. Finally, conclusions are drawn in section 4.

2. Proposed dcFLD

There are three main steps in the proposed dcFLD algorithm: (1) Partition face images denoted by a vector into sub-patterns, (2) FLD is performed subpattern-by-subpattern to extract features from a set of large dimensional patterns, and (3) classify an unknown image.

2.1. Image Partition

Suppose that there are iV training samples Ak(k = 1,2,..., iV), denoted by m-by-n matrices, which contain C classes, and the ith class d has n* samples. Now, an original whole pattern denoted by a vector is partitioned into K d-dimensional subpatterns in a non-overlapping way and reshaped into d-by-K matrix Aj = (Aji, Aj2, • • •, AJK) with Aij being the j t h

subpattern of Ai and i — 1,... ,N,j = I,... ,K. Now to form the j t h training subpattern set TSj, we respectively collect j t h subpattern of Ai and i=l,..,N, j=l,..,K. In this way, K separate subpatterns are formed.

2.2. Apply FLD on K subpatterns

Now according to the second step, conventional FLD is applied to the j t h subpattern set [TSj]to seek corresponding projection sub vectors Uj = (uji,Uj2,--,Ujt) by selecting t eigenvectors corresponding to t largest eigenvalues based on maximizing the ratio of the determinants of the between-class and the within-class scatter matrices of the projected samples. Analogous to the fisherface method,2 define the j t h between-class and within-class sub scatter matrices,Gbj and GWJ , respectively as follows:

c

Gbj = Y,n*(Av - Ai)(AH ~ Ai)T (!) t = i

c GwJ = H E (Akj - AijKAv - Aijf (2)

i = l Akj&Ci

Here, A, = i YLi=\ Aij 3 = 1 ,2,3, . . . , K are sub-pattern means, Aij = ^- Y^Ji=i Aij is ith class j t h

42 Divide-and-Conquer Strategy Incorporated Fisher Linear Discriminant Analysis: An Efficient Approach for Face Recognition

subpattern mean and Akj is the j subpattern of kth sample belonging to the ith class. After obtaining all individual projection sub vectors from the partitioned sub patterns, extract corresponding sub features Yj from any subpattern of given whole pattern Z = (Zi, Z2 , . . . , ZK) using the following equation:

Yj = UjZj (3)

Now synthesize them into a global feature as follows:

Y = (¥?,...,¥?)? = (Z?UU...,ZTUK)T (4)

It is interesting to note that when K = 1 and d = m x n, dcFLD reduces to the standard Fisherface method. Thus we can say that the Fisherface method is a special case of the proposed dcFLD method.

2.3. Classification

In this process, in order to classify an unknown face image I, the image is first vectorized and then partitioned into K sub-patterns (Ii,h, • • • ,IK) in the same way as explained in section 2.1. Using the projection sub vectors, sub features of the test sample I is extracted as follows:

Fj=Ujlj Vj = l,2,...,K (5)

Since one classification result for the unknown sample is generated independently in each subpattern, there will be total K results from K subpatterns. To combine K classification result from all subpatterns of this face image I, a distance matrix is constructed and denoted by D(I) = (dij)NxK, where di} denotes the distance between the corresponding j t h patterns of the I and the ith person, and d^ is set to 1 if the computed identity of the unknown sample and the ith persons identity are identical, 0 otherwise. Consequently, a total confidence value that the test image I finally belong to the ith person is defined as:

K

Td{I) = Y,di} (6)

And the final identity of the test image I is determined as follows:

Identity(I) = argtmax(TCi(I)) l^i^N (7)

2.4. Image Reconstruction

In the whole-pattern based approach, the feature vector and the eigenvectors can be used to reconstruct

the image of a face. Similarly, in this sub-pattern based approach, a face image can be reconstructed in the following way:

A ^ (Uj . FVJ) + Aj Vj = l,...,K (8)

Where Uj indicates the projection vectors of j t h sub-pattern obtained through PCA, FVJ is the feature vector of the image which we are seeking to reconstruct, and Aj is the j t h sub-pattern mean.

3. Experiments

In this section, a series of experiments are presented to evaluate the performance of the proposed dcFLD by comparing with existing methods.1 '2 '4^710 ,11 All our experiments are carried out on a PC machine with P4 3GHz CPU and 512 MB RAM memory under Matlab7 platform. For all the experiments, the nearest neighbor classifier is employed for classification. If extra explanation is not given, it is understood that the experiment is repeated for 25 times by varying the number of projection vectors t (where i=l,2,..,20,25,30,35,40,45). Since t has a considerable impact on classification performance as well on the dimension of subpattern (for the proposed dcFLD method), we choose that t which corresponds to best classification result on the image set.

3.1. Image databases

The aforementioned algorithms are tested on several image databases comprising of face and object. We carried out the experiments on two face databases: ORL12 and the Yale2 and also on an object database namely COIL-20.13 In ORL database, there are 400 images of 40 adults, 10 images per person while Yale database contains 165 Images of 15 adults, 11 image per person. Images in Yale database features frontal view faces with different facial expression and illumination condition. Besides these variations, images in ORL database also vary in facial details (with or without glasses) and head pose.

The COIL-20 is a database of 1440 gray-scale images of 20 objects. The objects were placed on a motorized turntable against a black background. The turntable was rotated through 360 degrees to vary object pose with respect to fixed camera. Images of the objects were taken at pose interval of 5 degrees, which corresponds to 72 images per object.

S. Noushath, G. Hemantha Kumar, V. N. Manjunath Aradhya and P. Shivakumara 43

Table 1. Best recognition accuracy for varying training sample number and dimension of subpattern

Dimension of subpatter(d-.fC)

92-112

112-92

161-64

322-32

644-16

2576-4

2

88.25(11)

87.00(12)

87.00(06)

87.00(10)

85.50(07)

82.00(20)

4

94.50(11)

94.75(18)

94.75(13)

94.50(14)

94.25(18)

92.75(19)

5

97.00(08)

96.00(08)

97.25(15)

96.50(09)

96.50(12)

94.00(18)

6

99.00(06)

98.50(12)

99.00(09)

98.25(09)

98.00(09)

95.75(08)

8

99.75(04)

100.0(06)

99.75(05)

100.0(06)

99.75(08)

99.25(09)

3.2. Results on the face databases

We first conduct an experiment with ORL database. As noted above, 40 people, 10 images of each person with the size of 112 x 92 are used. Our preliminary experiments show that the classification performance of the proposed method is impacted by the dimension of subpattern (d). In order to determine the effect of dimension of subpattern on the available data, we check the classification performance by varying both number of training samples and dimension of subpattern. For this, we randomly chose p images from each class to construct the training database, the remaining images being used as the test images. To ensure sufficient training a value of at least 2 is used for p. It can be ascertained from Table 1 that the recognition accuracy of the proposed method is greatly influenced by the size of the subpattern dimension (d). It is apparent from the table that, smaller the d (or larger the number of subpatterns K), better is the recognition accuracy. Value in the parenthesis denotes the number of projection vectors used to attain the best recognition accuracy. It is also observed that the recognition accuracy is comparatively better when d and K take on the values 161 and 64 respectively. Hence in all our later experimentations on ORL, we use these values for d and K.

The next set of experiments on ORL database is conducted by each time randomly selecting 5 images per person for training, the rest 5 per person for testing. This experiment is independently carried out 40 times, and the averages of these experiments results are tabulated in Table 2.

Experiments on the Yale database are carried out by adopting leave-one-out strategy, that is, leaving out one image per person each time for testing and all of the remaining images are used for training. Each image is manually cropped and resized to

235 x 195 pixels in our experiment. This experiment is repeated 11 times by leaving out a different image per person every time. Results depicted in Table 2 are the average of 11 times results. Here the values of d and K are empirically fixed to 235 and 195 respectively, to yield optimal recognition accuracy. In table 2, /x is defined as follows:

number of selected eigenvectors fi = : — - f- : x 100 number of all the eigenvectors

Here all the eigenvectors are sorted in the descending order of their corresponding eigenvalues, and selected eigenvectors are associated with the largest eigenvalues.

It can be determined from Table 2 that on ORL, dcFLD achieves better performance improvement over FLD and other methods. Moreover, the experiments on Yale database significantly exhibits the efficiency of proposed method under varied facial expressions and lighting configurations. It can be seen from the table that, on Yale database, dcFLD achieves up to 4-10% performance improvement over PCA, up to 4-9% improvement over the Fisherface method, up to 1-4% improvement over A2DLDA method. In contrast to 2DLDA and (2D)2LDA methods, significant improvement in accuracy is achieved (i.e. up to 13%). Thus we can say that not only does dcFLD prevails stable under aes-thetical conditions (for ORL), but also exhibits efficiency and high robustness under the condition when there is wide variations in both lighting configurations and facial expressions (for Yale).

Finally, to further exhibit the essence of this sub-pattern based approach over the conventional whole-pattern based approach, we conduct a simple reconstruction test. Taking one image from the ORL as

44 Divide-and-Conquer Strategy Incorporated Fisher Linear Discriminant Analysis: An Efficient Approach for Face Recognition

Table 2. Accuracy comparison of various approaches.

Database

ORL

Yale

H

10.0

12.5

20.0

6.66

10.00

13.33

16.66

20.00

PCA

90.43

92.46

93.61

83.47

86.77

85.95

87.60

87.60

FLD

93.73

94.00

93.93

88.42

88.42

86.77

88.42

86.77

2DLDA

93.80

94.73

92.86

88.42

85.12

85.12

85.12

84.29

A2DLDA

93.67

94.88

93.88

93.38

92.56

92.56

92.56

92.56

2D2LDA

93.33

92.71

93.88

90.08

91.73

88.42

85.12

80.16

2DPCA

93.95

93.78

93.00

86.77

87.60

87.60

87.60

87.60

DiaFLD

93.80

94.66

94.00

89.25

89.25

88.42

87.60

87.60

DiaFLD+2DFLD

93.73

92.48

93.00

92.56

92.56

92.56

90.90

89.25

dcFLD

94.63

94.70

94.15

93.38

92.56

95.04

93.38

93.38

II IMB 9H m i H i I

Fig. 1. five reconstructed images by whole-pattern based approacfa(Top-row) and sub-pattern based approach (Bottom-row)

an example, we can determine its five reconstructed images for varying projection vectors t (where 1=10, 20, 30, 40,50). These images are shown in Fig. 1. It is quite evident that the sub-pattern dividing approach yields higher quality image than the whole-pattern based approach, when using similar amount of principal components. Note that our objective is to demonstrate the effectiveness of the subpattern based approach over the whole-pattern based approach. Hence in both the approaches, we considered the PCA features for reconstruction. It is also well known that, PCA has better image reconstruction accuracy than the FLD method.

3.3. Results on the object database

Inspired by the conviction that the successful method developed for FR1 '5 should be extendable for object recognition as in10,11 respectively, in this section, we verify the applicability of the dcFLD method

for objects by considering COIL-20 database. This database contains gray level images of size 128 x 128 pixels. In this experiment, we empirically fixed the values of both d and K to 128.

For comparative analysis of various approaches, we consider first 36 views of each object for training and the remaining views for testing. So, size of both the training and the testing database is 720. Table 3 gives the comparison of nine methods on top recognition accuracy, corresponding dimension of feature vector/matrices and running time costs. It reveals that the top recognition accuracy of the proposed dcFLD method is higher than other methods. These results certify the pertinence of the dcFLD method for object database apart from the face images. But, the only demerit of the proposed- method is that it consumes more time compared to other contemporary methods such as 2DPCA, 2DLDA, etc. This is due to the fact that the proposed method involves the drudgery of subpattern set formation and then obtaining corresponding projection vectors and features. But, as the present-day computers have ample processor speed, it has no practical influence on the classification performance of the proposed method.

4. Conclusions and future work

In this paper, a novel subspace analysis method called dcFLD is proposed for efficient and robust face/object recognition. The proposed method utilizes the separately extracted local information from each sub-pattern set and thereby possesses robustness and better recognition performance. Indeed, the practicality of the dcFLD is well evidenced in the experimental results for the Yale face database. In that,

S. Noushath, G. Hemantha Kumar, V. N. Manjunath Aradhya and P. Shivakumara 45

Table 3. Comparison of Different Methods on COIL-20 database.

Methods Top Recognition Accuracy Running Time(s) Dimension

PCA 1 0

FLD

2DPCA 1 1

2DLDA

A2DLDA

2D2LDA

DiaFLD

DiaFLD+2DFLD

dcFLD

93.89

91.25

94.30

93.05

88.88

94.72

93.75

94.10

95.77

157.91

177.29

30.45

33.69

29.94

59.69

29.91

61.64

129.52

35

40

128 x 5

128 x 7

9 x 128

11x11

128x6

9x9

19x128

dcFLD improves recognition accuracy by a minimum

of 3-4% than the other discrimination methods. Fur

thermore, the results on COIL-20 object database

also show tha t the proposed method is feasible and

effective. Also we believe tha t this method is equally

effective in scenarios where a person's face is oc

cluded by sunglass, scarf etc. which is an interesting

issue for future work.

Nevertheless, there are still some aspects of

dcFLD method tha t deserve further study. Is the pro

posed approach the best way of subdividing the full

pa t tern . In addition, dcFLD needs more coefficients

for image representation than FLD. Is there any way

to reduce the number of coefficients and in the mean

t ime to keep up the same accuracy? Finally, it is still

unclear as how to choose the optimal number of sub-

pat te rns to obtain the bet ter recognition accuracy.

These are some crucial issues which give scope for

future work.

R e f e r e n c e s

1. M. Turk and A. Pentland, Journal of Cognitive Neu-roscience 3, 71 (1991).

2. P. Belhumeur, J. Hespanha and D. Kriegman, IEEE Transactions on Pattern Analalysis and Machine Intelligence 19, 711 (1997).

3. H. Xiong, M. Swamy and M. Ahmed, Pattern Recognition 38, 1121 (2005).

4. M. Li and B. Yuan, Pattern Recognition Letters 26, 527 (2005).

5. J. Yang, D. Zhang, A. Frangi and J. Yang, IEEE Transactions on Pattern Analalysis and Machine Intelligence 26, 131 (2004). S. Noushath, G. H. Kumar and P. Shivakumara, Pattern Recognition 39, 1396 (2006). S. Noushath, G. Kumar and P. Shivakumara, Neu-rocomputing 69, 1711 (2006). A. M. Martnez, IEEE Transactions on Pattern Analalysis and Machine Intelligence 24, 748 (2002). X. Tan, S. Chen, Z. Zhou and F. Zhang, IEEE Transactions on Neural Networks 16, 875 (2005).

10. H. Murase and S. Nayar, International Journal of Computer Vision 14, 5 (1995). P. Nagabhushan, D. Guru and B. Shekar, Pattern Recognition 39, 721 (2006). www.uk.research.att .com/facedatabase.html.

13. wwwl.cs.Columbia.edu/CAVE/research/softlib/ coi l -20.html.

6

7.

9

11

12

Ear Biometrics: A New Approach

Anupam Sana and Phalguni Gupta

Indian Institute of Technology Kanpur Kanpur(U.P.), India-208016

E-mail: {sanupam,pg}@iitk.ac.in

Ruma Purkait

Department of Anthropology Saugor University

Saugor-470003, India E-mail:r. [email protected]

Abstract. The paper presents an efficient ear biometrics system for human recognition based on discrete Haar wavelet transform. In the proposed approach the ear is detected from a raw image using template matching technique. Haar wavelet transform is used to decompose the detected image and compute coefficient matrices of the wavelet which are clustered in its feature template. Decision is made by matching one test image with 'n' trained images using Hamming distance approach. It has been implemented and tested on two image databases pertaining to 600 individuals from IITK and 350 individuals from Saugor University, India. Accurcy of the system is more than 96%

Keywords: Haar wavelet; Wavelet Coefficient Matrices; Ear detection; Template matching; Hamming distance.

1. Introduction

Biometrics is the automated method of identifying or verifying the identity of an individual on the basis of physiological and behavioral characters. Ear which is easily detectable in profile images can be advocated as a recognition tool. Ear has few advantages over facial recognition technology. It is more consistent compared to face as far as variability due to expression, orientation of the face and effect of aging, especially in the cartilaginous part is concerned. Its location on the side of the head makes detection easier. Data collection is convenient in comparison to the more invasive technologies like iris, retina, fingerprint etc.

The possibility of ear as a tool for human recognition has been first recognized by the French Criminologist, Alphonse Bertillon 1 . Nearly a century later, Alfred Iannarelli2 has devised a non-automatic ear recognition system where more than ten thousand ears have been studied and found no two ears are exactly alike. Burger and Burger3'4 have proposed an ear biometrics approach which is based on building neighborhood graph from Voronoi diagrams of the detected edges of ear. But its main disadvantage is the detection of erroneous curve segments i.e. this system may not be able to differentiate the real ear edges and non edges curves. Choras5 also has used an approach for feature extraction based on contour

detection, but the method too suffers from the same disadvantage of erroneous curve detection. Hurley et a/.6 have proposed an approach on force field transformation to determine the energy lines, wells and channels of ear. Although it has been experimented on a small data, the results are found to be quite promising. Victor et al.7 have presented an approach based on the Principal Component Analysis (PCA) on face and ear recognition. Face based system has given a better performance than ear. In a similar experiment by Chang et al.8 no significant difference between the performance of ear and face has been found. Moreno et al.,9 have analysed ear by neural classifiers and macro-features extracted by compression networks. It has achieved better recognition result (without considering rejection thresholds) using only compression network. Chen and Bhanu10 have worked on 3D ear recognition using local surface descriptor for representing ear in 3D. The system performance is evaluated on a real range image database of 52 subjects.

This paper has proposed a novel approach for feature extraction from the ear image. Wavelet is an emerging technique for image analysis. Haar wavelet is simple and reliable in this field. So in this paper discrete Haar Wavelet transformation is applied on ear images and wavelet coefficients are extracted. Section 2 discusses the proposed approach. The experimen-

tal results are presented in Section 3. The proposed conclusions are given in the last section.

2. T h e Proposed Approach

This section discusses a new approach for feature extraction in ear biometrics. The ear images generally have large background, so there is a need to detect ear which is done by template matching and detected image is scaled into constant size. R*om these ear images features are extracted as Haar wavelet coefficient from wavelet decomposed image and feature template is stored for each training image. So testing feature template which is extracted using above mention way is matched with large set of training template and gives best set of matches based on individual matching score. The present approach is implemented in four major steps. Step 1 is image acquisition where ear image is captured in camera in the laboratory environment. Step 2 is image preprocessing where EGB image is converted in gray-scale image, ear is detected and then scale is normalized. Step 3 is feature extraction where unique features of ear are extracted and are stored as trained template. Then in Step 4 matching is performed to get the best matches.

2 .1 . Image Acquisition

Ear images are collected at two centers, Indian Institute of Technology Kanpur (IITK) and Saugor University. The images are captured in the laboratory environment where moisture and humidity are normal and illumination is constraint. One bright light source is used to illuminate the whole ear to reduce the shading of ear edges that generally appear in the image. A stand is used to place chin of a person over there that reduces the rotational effect on image. At IITK, a CCD camera (JAI CV-M90 3-CCD RGB color camera) from a distance of 28 to 28 cm has been used to take the images of the ear. Using ITI camera configurator three images of each ear have been acquired for six hundred subjects. Another set of images have been taken of three hundred and fifty subjects at the Anthropology department laboratory in Saugor University. These have been captured using a digital camera (Kodak Easy Share CX7330, 3.2Megapixel) from a distance of 24 to 28cm from the subject. For each of these subjects, three images of the ear are taken. Out of the three images so cap-

Anupam Sana, Phalguni Gupta and Ruma Purkait 47

tured, two has been used for training and one for testing purpose.

Uj (h)

Fig. 1. Image database (a) IITK database, (b) Saugor University database.

2.2. Image Preprocessing

The raw image may not be suitable for feature extraction due to its large background so some preprocessing are required to make it suitable. The important steps involved are: Grayscale conversion. Ear Detection, Scale Normalization.

2.2.1. Grayscale Conversion

RGB images are converted into grayscale by using the following formula:

Jp = 0.2989*/(B)+0.5870*I(G)+0.1140*7(B) (1)

where I(R) = red channel value of RGB image, 1(G) = green channel value of the RGB image, 1(B) = blue channel value of RGB image, Ig = an intensity image with integer values ranging from a minimum of zero to a maximum of 255.

2.2.2. Ear Detection

Ear detection is implemented using simple template matching technique. Here first a set of images are manually cropped to get the different sizes of ear images. These images are decomposed into level 2 using Haar wavelet. These decomposed images are trained as templates. And then the input raw image is also decomposed into level 2 using same technique. Thereafter, each template is retrieved from the

48 Ear Biometrics: A New Approach

y *

1

$(x) :

0 i s

Fig. 3. Haar scaling function $(x).

trained set and matched with the same sized overlap-, ping block of the decomposed input image. Thus for each trained template the best matched block in input image is traced. Among those blocks the best matched block is chosen and the corresponding region from the original image is extracted (Fig. 2),

scaling function <j>(x) has values one for closed interval of time(x) from 0 to 1 and zero for other interval of time. </>(x) is formulated as follows:

<f>(x) {i if 0 < x < 1

otherwise (4)

Haar wavelet function &(x) is a step function has value 1 for time (x) greater than equals to 1 and less than | . 9(x) has also value -1 for time (x) greater than equal to | and less than 1. \P(x) has 0 values for other interval. W(x) is discontinuous at time(x) equal to 0, | , 1. 9(x) is defined by

1 Vx € [0,1/2) tf(x) = { - l V x € [1/2,1)

0 otherwise (5)

Fig. 2. (a)Raw ear image, (b) Ear traced, (c) Detected ear.

2.2.3. Scale Normalization

The cropped ear image may be of varying size so the feature set of images may also vary. Hence the images are normalized to a constant size by resizing technique. If two images are in different sizes, e.g. one is of size (x'xy*) and the other is of size (x" xy"), then two images are mapped to constant size.

2.3.

I(xfJy

,) = I(x1y)

I(x\yn = I(x,y)

Wavelet Tmnsform and Featum Extraction

(2)

(3)

This section discusses wavelet transform and introduces the method for the feature extraction from wavelet coefficients.11 Every wavelet is defined by one scaling function and one wavelet function. The Haar

TOO

1/2

1 x

Fig. 4. The Haar wavelet #(a?).

A standard decomposition of an image (two dimensional signals) is easily done by first performing a one dimensional transformation on each row followed by one dimensional transformation of each column. Thus the input ear image is decomposed (Figure 3) into approximation (CA), vertical (CV), horizontal (CH) and diagonal (CD) coefficients using the wavelet transformation and the approximation coefficient (CA) is further decomposed into four coefficients. The sequences of steps are repeated to get the coefficients from the four level wavelet transform. After analyzing the coefficients at all the four levels it is found that the coefficients of level 1 to level 4 are almost the same. In order to reduce redundancy from among the coefficients of different levels only one of them is chosen. Thus all the diagonal, vertical and horizontal coefficients for the fourth level are chosen to reduce space complexity and discard the redundant information. The feature vector comprises of following coefficients [CD4 CV4 CH4], This coefficient matrix which represents the unique ear

Anupam Sana, Phalguni Gupta and Ruma Purkait 49

l"LevelHaar WTavefet transform

Oil CD1 CHI OT

2"1 Level Haar wsvelet transform

CA2 CD2 CH2

Fig. 5. Wavelet decomposition level.

pattern is binarised comparing the sign (negative or positive) of its element. The binary feature template of approximataion coefficient matrix from level two wavelet transform is shown in Fig. 6.

^ ^ ^ Haar Wavelet l ^ y ^ l

Fig. 6. Wavelet transform of Ear into level two.

2.4. Matching

Testing binary template (S) matched with the trained template (T) of the database using Hamming distance. Hamming Distance(HD) between two templates (nxm) calculated using the equation

HD = 5 ^ T i j ® $ j n x m

(6)

Here templates T and S are XOR-ed element wise and HD is computed, which is the matching score between training and testing templates. Therefore, for each trained template there is a matching score and best matches can be chosen. The matching score of all the templates are statistically analyzed in the next section.

3. Experimental Result

The experiment is conducted on two different databases, IITK and Saugor University. The IITK image database has images belonging to 600 individuals, with three images acquired per person (600x3). While the database collected at Saugor University

consists of 350 (350x3) individuals. In Fig. 7 false acceptance rate (FAR) and false rejection rate (FRR) is plotted at different threshold values (i.e. in between 0 and 1). From the two curves in this figure it is found that the system is giving equal error rate (EER) 3.4% and 2.4% at threshold 0.303 and 0.323 for IITK and Saugor University image database respectively. In Fig. 8 the accuracy is plotted at different threshold values and it is found that at EER threshold accuracy value is 96.6% and 97.6% respectively and system is giving maximum accuracy of 97.8% and 98.2% at 0.29 and 0.28 threshold value for IITK and Saugor University image database respectively. The Receiver Operating characteristic (ROC) is shown in Fig. 9. From two curves in this figure it is observed that the system is giving more than 95% genuine acceptance at FAR lower than 1%.

Plot of FAR /FRR curve

Fig. 7. FAR and FRR curve for IITK database and Saugor University database.

4. Conclusion

The proposed ear biometrics has certain advantages over those reported earlier. This paper has proposed a new approach of ear biometrics. As two training images are sufficient for database preparation the time taken to test with the testing image is significantly reduced. The ear pattern template size being small make it easy to handle. For this reasons it can be applicable on any broad scale or small scale society. Our approach achieved more than 96% accuracy for the two databases indicating a fairly reliable system.

50 Ear Biometrics: A New Approach

Plot of accuracy cuive

Fig. 8. Accuracy Curve for IITK database and Saugor University database.

Plot of ROC Curve

99.5

99

£98.5

™ 98 tx 0)

g 97.5

S 97

.1 96.5

S * 95.5

95

! \

:

':

;-'.-./_

;

1 i

— ROC-IITK - -ROC- Sagour

• ^ - : - - - - : ~ - - -

'..-S" s

f\ * • /

;

.jr

Jill

/

10 10 10 10 10

False Acceptance Rate (%) -- >

Fig. 9. Receiver Operating Characteristic Curve for IITK database and Saugor University database.

5. A c k n o w l e d g e m e n t

The study is supported by the Ministry of Commu

nication and Information Technology and University

Grants Commission, Govt, of India. The authors ac

knowledge Ms. Hunny Mehrotra for preparing the

manuscript and for assistance in conducting the ex

periments and Mr. Pradeep Nayak for assistance in

conducting the experiments.

References

1. A. Bertillon, La photographie judiciaire,avec un ap-pendice sur la classification et identification anthro-pometriques, in Gauthier-Villars, (Paris, 1890).

2. A. Iannarelli, Ear identification, in Proceedings of International Workshop Frontiers in Hand/writing Recognition, (Paramont Publishing Company, Fremont, California, 1989).

3. M. Burge and W. Burger, Personal identification based on pea, in Proceedings of the 21st Workshop of the Austrian Association for Pattern Recognition, 1997.

4. M. Burge and W. Burger, Ear biometrics in computer vision, in Proceedings of the 15th International Conference on Pattern Recognition, 2000.

5. M. Choras, Ear biometrics based on geometrical feature extraction, in Electronic Letters on Computer Vision and Image Analysis, 2005.

6. D. J. Hurley, M. S. Nixon and J. N. Carter, A new force field transform for ear and face recognition, in Proceedings of the IEEE International Conference of Image Processing, 2000.

7. B. Victor, K. Bowyer and S. Sarkar, An evaluation of face and ear biometrics, in Proc. Intl Conf. Pattern Recognition, 2002.

8. K. Chang, S. S. K. W. Bowyer and B. Victor, Comparison and combination of ear and face images in appearance-based biometrics, in IEEE Transactions on Pattern Analysis and Machine Intelligence, 2003.

9. B. Moreno, A. Sanchez and J. Velez, On the use of outer ear images for personal identification in security applications security technology, in Proceedings. IEEE 33rd Annual 1999 International Carna-han Conference, 1999.

10. H. Chen and B. Bhanu, Contour matching for 3d ear recognition, in Seventh IEEE Workshops on Application of Computer Vision(WACV/MOTION-05), 2005.

11. S. Pittner and S. Kamarthi, Feature extraction from wavelet coefficients for pattern recognition tasks, in IEEE Transactions on Pattern Analysis and Machine Intelligence, 1999.

51

Face Detection using Skin Segmentation as Pre-Filter

Shobana L., Anil Kr. Yekkala and Sameen Eajaz

Philips Electronics India Ltd. E-mail: {shobana.lakshminarasimhan, anil.yekkala, sameen.eajaz

Face Detection has been a topic of research for several years due to its vast range of applications varying from Security Surveillance to Photo Organization. Face detection algorithms are also used by Face Recognition algorithms for locating faces in an image, before performing the recognition. In the available literature on Face Detection, there exists a Face Detection algorithm provided by Viola and Jones, very robust in terms of detection rate, but its speed is not very satisfactory. In this paper, we propose a Pre-flltering step using skin segmentation and some minor modification to the algorithm provided by Viola Jones. The proposed Pre-filtering step and modifications improve the speed of the algorithm approximately by 2.8 times. It also improves the precision of the algorithm by reducing the number of false detections.

Keywords: Face Detection; Viola Jones; Skin Segmentation; Pre-Filter

1. Introduction

Several methods have been proposed in the literature for performing Face Detection. The available methods for Face Detection can be broadly categorized under two approaches. The first approach1'4'9'12 is based on conducting an exhaustive search of the entire image using windows of varying sizes and finding the face regions using features like Haar Cascades, Eigen Features and others. The second approach2,3 is based on the skin segmentation using color properties and simple texture measures, then finding connected regions and finally, filtering the non-face regions using face profiles and facial features. Even though the first approach based on the exhaustive search provides a very robust algorithm in terms of detection, its speed is not very satisfactory. For example, using the Viola Jones approach takes almost 2-3 seconds to detect a face in an image of size 1024x768 on a Pentium processor. Though the second approach based on skin segmentation provides very satisfactory results in terms of speed, it fails to provide a robust algorithm, since the algorithm fails for images with Complex backgrounds. Also, the false detection rate is quite high, since most of the time skin-colored objects are also detected as faces. In this paper, we present an improvement to the Viola Jones based Face Detection algorithm to improve its speed and precision as well by reducing the number of false detections, without affecting the recall rate. These improvements are achieved by using skin segmentation as a Pre-Filter and with some modification to the approach followed by Viola Jones.

The following sub-sections provide a brief

overview of the skin segmentation techniques in the available literature on Face Detection and also an overview of the Viola Jones Face Detection algorithm.

1.1. Skin Segmentation based Face Detection

Based on several studies6'13'14 conducted in the past, it has been found that in color images the skin regions can be detected based on their color properties. It has been observed that in case of RGB color models the proportion of red, green and blue are within some prescribed limits for skin regions. Similarly, in case of YC(,Cr model the chrominance components C;, and C r are within the prescribed limits for skin regions. Simple texture measures are used to improve the skin segmentation performance. Skin segmentation based Face Detection is done in three steps. 1. Skin segmentation based on the RGB color properties of the skin. 2. Finding the connected regions 3. Filtering out the non-face regions. The advantages of skin segmentation based Face Detection is its simplicity and its being not affected by the geometric variations of the face. Its main disadvantage is its sensitivity to the illumination conditions.

1.2. Viola Jones Method

The Viola Jones1 approach is based on conducting an exhaustive search of the image using varying size windows, starting from a base window of size 24x24, gradually increasing the window size by a fixed factor of 1.25 (or any value ranging from 1.2 to 1.3).

52 Face Detection using Skin Segmentation as Pre-Filter

A

C

B

1

D

1

2

4

Fig. 2. Integral Sum.

Each window region is classified as a face or non-face region based on simple features. The features used are reminiscent of Haar basic functions proposed in.11 Three kinds of features are used, namely, two-rectangular features, three-rectangular features and four-rectangular features. In two-rectangular features the difference between the sums of pixel intensities of the two rectangular regions is considered. In case of three-rectangular features the sums of the pixel intensities within the outside rectangles is subtracted from that of the center rectangle. Finally, in case of four-rectangular feature the difference between the sum of the pixel intensities of the diagonal pairs of rectangles is considered. Figure 1 shows the set of rectangular features used.

Ill SB

Fig. 1. Rectangular Features.

In order to make the computations of rectangular features faster, an intermediate representation of the image called integral image is used. The integral image at any point (x,y) is the sum of the pixels above and to the left of (x,y).

y

ii(x,y) = 2_J2-^I{J,k) j=0 fc=0

Using integral image, the sum of the pixel intensities within any rectangular region in the image can

be computed with just three sum operations. For example, in Figure 2 the sum of the pixel intensities of the rectangle D can be computed with four array references. The value of the integral image at location 1 is the sum of the pixel intensities in rectangle A. The integral image value at location 2 is A + B, at location 3 is A + C, and at location 4 is A + B + C + D. The intensity sum within D can be computed as 4 + 1 — (2 + 3).

Viola Jones Face Detector uses a boosting cascade of rectangular features. The cascade contains several stages. The stages are trained in such a way that they progressively eliminate the non-face candidates. The initial stages in the cascade are simple so that they can be evaluated fast and can reject maximum number of non-face candidates. The later stages are comparatively complex.

Each stage consists of a set of features. Each input window is evaluated on the first stage of the cascade, by computing the set of features belonging to the first stage. If the total feature sum of the stage is below a pre-defined threshold, then the window is discarded, otherwise it passes to the next stage. The same procedure is repeated at each stage. Since most of the windows in an image do not contain facial features, most of the windows get discarded in the initial stages, hence reducing the number of computations.

As the Viola Jones face detector is insensitive to small variations in scale and position of the windows, a large number of detections occur around a face region. To combine these regions into a single region, a grouping algorithm is applied. The grouping algorithm groups all the overlapping rectangles that lie within an offset of 25% of their size into a single rectangle. A region is detected as a face only when a minimum of three face rectangles are obtained around that face region. This confidence check is used to reduce the number of false detections. For a non-face region, the probability of getting three or more overlapped rectangles is relatively low.

2. Proposed Approach for Face Detection

We present an improvement to the Viola Jones algorithm by using skin segmentation as a pre-filter and modifying the search strategies and merge and group methods. With the proposed modification, the algorithm is performed in three steps. Figure 3 gives the overview of the proposed algorithm.

Shobana L., Anil Kr. Yekkala and Sameen Eajaz 53

11, s(x,V) = ^2Y2Iss(j,k)

I •miMg.

j=0 fe=0

Fin ting Skin Regions

Feature based Filtering

Fig. 3. Overview of the proposed algorithm.

2.1. Skin Segmentation

In the proposed method, skin segmentation is used as the pre-filtering step to reduce the number of windows to be searched. Skin segmentation is performed by detecting the skin pixels using color properties in the RBG color space. For each pixel, the algorithm checks whether the proportions of red and blue are within the prescribed limits. The method can be described by the following equation.

if ((r < 0.34; and (r < 0.58; and (g > 0.22; and (s < o.3s;;

hs = 1 else

hs =0 end

where, r - R ' ~ fi+G+B' 9 =

G R+G+B

A pixel with the Skin Segmentation output Iss=l indicates that it is a skin pixel and that with 0 indicates a non-skin pixel. The skin segmentation is designed in such a way that rejecting a skin region is considered costlier then accepting a non-skin region. Once the skin segmentation is performed, the output binary image Iss is represented as an integral image, similar to the intensity sum. This representation results in faster computation of the number of skin pixels by using just 3 sum operations. Since skin segmentation is based on color properties, this pre-filtering step can be used only for color images.

2.2. Searching For Face Regions

Unlike the Viola Jones algorithm, the proposed approach does not perform an exhaustive search to locate the face regions. Instead, it finds the probable face regions based on the number of skin pixels present in the region. If the number of skin pixels present in a region is above a pre-defined threshold, the region is considered to be a candidate face region. Based on experimental results, this threshold was found to be 60%. The Haar features are computed for each candidate face region using the same approach as Viola Jones detection to check whether the region is a face region or not. Unlike the Viola Jones approach, the Face Detection starts with the largest possible window size depending on the image size and gradually scales down to a window size of 24x24. The advantage of starting from the largest window size is that if a bigger window is detected as a face region, the smaller windows within that face region that are lying at an offset of more than 25% of the bigger window size can be discarded. Note that the regions that are lying within an offset of 25% are not discarded because these windows are required for the Merge and Group algorithm.

2.3. Merge and Group

The merge and group technique is similar to that followed in the Viola Jones Face Detection. Once the merging is performed according to Viola Jones method, the algorithm checks for overlapping face regions with percentage of overlap greater then 50%. If such overlapping regions exist, then the region with lower average value of feature sum is eliminated. This results in improved precision.

3. Resul ts

A comparative study on the performance in terms of speed of the proposed method with respect to standard Viola Jones method is shown in Table 1.

The study was conducted on a set of 452 images obtained from Caltech Image database. The comparison is done based on the percentage of windows processed in each stage using the proposed method and

54 Face Detection using Skin Segmentation as Pre-Filter

Table 1. Percentage of regions processed in each stage.

Stage

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

No of

features

9

16

27

32

52

53

62

72

83

91

99

115

127

135

136

137

155

159

169

181

196

197

199

200

211

% of Windows

searched

in Viola Jones

method

100

68.15636

37.44727

23.80884

12.71177

5.70148

2.563128

0.885991

0.49522

0.300651

0.200682

0.117771

0.083492

0.05391

0.036424

0.026771

0.021732

0.015506

0.012858

0.010599

0.008351

0.007104

0.006146

0.005452

0.004961

% of Windows

searched

Proposed

method

36.738

25.92015

14.18722

8.981527

4.884061

2.211218

1.001229

0.341391

0.19646

0.122056

0.083506

0.049768

0.037018

0.024601

0.017609

0.013751

0.011666

0.00891

0.007694

0.00662

0.005504

0.004849

0.004308

0.003919

0.003656

that using the Viola Jones method. The comparison clearly shows that in each stage the number of features computed using the proposed method is approximately around 2.8 times lesser then the number of features computed using the standard Viola Jones method. Hence there is an improvement by a factor of 2.8 in speed. It can be further verified that the average number of features computed using the standard Viola Jones method is 2.8 times the average number of features computed using the proposed method. Hence it can be concluded that the proposed method results in an improvement of speed by a factor of 2.8. This has been further verified by

the performance analysis. Table 2 shows that proposed method has not af

fected the recall rate much, but has improved the precision rate.

Table 2. Detection Rate.

Recall

Precision

Viola Jones method

0.97

0.86

Proposed Method

0.96

0.95

The Precision and Recall are computed as follows. Rprnll Noof facesdetected*100

Noof facesdetected+Noof facesmissed

Precision = Noofcorrectdetections*100 Noofcorrectdetections+Noofwrongdetections

The results of detection can be seen in Figure 4.

4. Conclusions

In this paper we have proposed modifications to the Viola Jones approach of Face Detection. The modifications include addition of a Pre-Filtering step using Skin Segmentation, modification in the search algorithm by starting the search from the largest window and excluding the smaller windows from the search if they are within a detected face region. In addition, a modification to the Merging algorithm has been suggested. With the proposed modification the speed of the algorithm has improved by a factor of 2.8 and the precision rate has also improved by 10% without affecting the recall rate much. Hence it can be concluded that using Skin segmentation as a pre-filter for detecting faces in color images and starting the search from a larger window size are useful extensions to the Viola Jones method. The proposed approach of using skin segmentation as a pre-filter can be easily extended to any face detection algorithms that use sliding window techniques, such as face detection technique based on eigen features.

References

1. Micheal J Jones, "Robust Real Time Face Detection", International Journal of Computer Vision, 57(2), 137-154 (2004).

Shobana L.t Anil Kr. Yekkala and Sameen Eajaz 55

Fig. 4. Face Detection Results

2. Pedro Fonseca, Jan Nesvadba, "Face Detection in the Compressed domain", International Conference on Image Processing , (2004).

3. R. L. Hsu, M. Abdel Mottaleb and A. K. Jain, "Face Detection in Color Images", IEEE Transactions on Pattern Analysis and Machine Intelligence, v, o (1). 24, no. 5, pp. 696-706, May 2002.

4. Diedrik Marius, Sumita Pennathur and Klint Rose, "Face Detection using color thresholding and Eigen-image template matching".

5. Rapheal Feraud, Olivier J. Beraier, Jean Emmanuel Viallet and Michel Collobert, "A Fast and Accurate Face Detector Based on Neural Networks", IEEE Transactions on Pattern Analysis and Machine Intelligence. Vol. 23, No. 1 (Jan 2001).

6. Jean-Christophe Terrillon, Mahdad N. Shirazi, Hideo Fukamachi and Shigeru Akamatsu, "Comparative Performance of Different Skin Chrominance Models and Chrominance Spaces for the Automatic Detection of Human Faces in Color Images".

7. Sung K and Poggio T, "Example-based learning for viewbased face detection", IEEE Pattern Analysis Machine Intelligence. , (1998).

8. Rowley H, Baluja S and Kanade T., "Neural network-based face detection", IEEE Pattern Anal

ysis Machine Intelligence. , (1998). 9. Shneiderman H and Kanade T, "A statistical

method to 3D object detection applied to faces and cars", International Conference on Computer Vision. , (2000).

10. Roth D, Yang M and Ahuja N, "A snowbased face detector", Advances in Neural Information Processing Systems. , (2000).

11. Papageorgiou C, Oren M and Poggio T, "A general framework for object detection", International Conference on Computer Vision. , (1998).

12. M. Turk, A. Pentland. "Eigenfaces for Recognition", Journal of Cognitive Neuroscience. Vol. 3 N o . 1, 71-86 (1991).

13. Son Lam Phung, Abdesselam Boezerdoum and Douglas Chai, "Skin Segmentation using Color Pixel Classification: Analysis and Comparision", IEEE Transactions on Pattern Analysis and Machine Intelligence. Vol. 27 No . 1, (Jan 2005).

14. Filipe Tomaz and Tiago Candeias and Hamid Shah-bazkia, "Improved Automatic Skin Detection in Color Images", Proc. Vllth Digital Image Computing: Techniques and Applications. , 10-12 (Dec 2003).

56

Face Recognition using Symbolic KDA in the Framework of Symbolic Data Analysis

P. S. Hiremath

Department of Computer Science Gulbarga University,

Gulbarga-585106, Karnataka, India E-mail: [email protected]

C. J. Prabhakar

Department of Computer Science Kuvempu University

Shankarghatta-577451,Karnataka, India E-mail: [email protected]

In this paper we present one of the symbolic factor analysis method called as symbolic kernel discriminant analysis (symbolic KDA) for face recognition in the framework of symbolic data analysis. Classical factor analysis methods (specifically classical KDA) extract features, which are single valued in nature to represent face images. These single valued variables may not be able to capture variation of each feature in all the images of same subject; this leads to loss of information. The symbolic KDA Algorithm extracts most discriminating non-linear interval type features; they optimally discriminate among the classes represented in the training set. The proposed method has been successfully tested for face recognition using two databases, ORL and Yale Face database. The effectiveness of the proposed method is shown in terms of comparative performance against popular classical factor analysis methods such as eigenface method and fisherface method. Experimental results show that symbolic KDA outperforms the classical factor analysis methods

Keywords: Symbolic Data Analysis; Face Recognition; Interval Type Features; Symbolic Factor Analysis Methods.

1. Introduction

Of the appearance based face recognition methods,4 '5 '1 2 , 1 6 , 2 6 , 2 9 those utilizing LDA techniques 10 ,11 ,28 '35have shown promising results. However, statistical learning methods including the LDA based ones often suffer from the so called small sample size (SSS) problem encountered in high dimensional pattern recognition tasks where the number of training samples available for each subject is smaller than the dimensionality of the sample space. Therefore numerous modified versions of the LDA were proposed. These modified versions have shown promising results as it is demonstrated in 3,34,6,30,20,23,33 There are two ways to address the problem. One option is to apply linear algebra techniques to solve the numerical problem of inverting the singular within class scatter matrix. For example, Tian et al, utilize the pseudo inverse to complete this task. Also, some researchers15'34 recommended the addition of a small perturbation to the within class scatter matrix so that it becomes non-singular. However, the above methods typically computationally expensive since the scatter matrices are very large. The second option is a subspace approach, such as the one fol

lowed in the development of the Fisherfaces method3

where PCA is firstly used as a preprocessing step to remove the null space of within class scatter matrix and then LDA is performed in the lower dimensional PCA subspace. However, it has been shown that the discarded null spaces may contain significant discriminatory information19 to prevent this from happening solutions without a separate PCA step, called direct LDA methods have been proposed recently in.6 '30 '23

Although successful in many cases, linear methods fail to deliver good performance when face patterns are subject to large variations in view points, which result in a highly non convex and complex distribution. The limited success of these methods should be attributed to their linear nature. As a result, it is reasonable to assume that better solution to this non linear problem could be achieved using non linear methods, such as the so called kernel machine techniques.17'25 Among them, kernel principal component analysis (KPCA)27 and kernel Fisher discriminant analysis (KFD)24 have aroused considerable interest in the fields of pattern recognition and machine learning. KPCA was originally developed by Scholkopf et al., in 1998, while KFD was first

P. S. Hiremath and C. J. Prabhakar 57

proposed by Mika et a/., in 1999.24 Subsequent research saw the development of series of KFD algo-rithms.2 '24-31 '32 '35

The denning characteristic of KFD based algorithms is that they directly use the pixel intensity values in a face image as the features on which to base the recognition decision. The pixel intensities that are used as features are represented using single valued variables. However, in many situations same face is captured in different orientation, lighting, expression and background, which lead to image variations. The pixel intensities do change because of image variations. The use of single valued variables may not be able to capture the variation of feature values of the images of the same subject. In such a case, we need to consider the symbolic data analysis (SDA),1 '7 '8 , 9 '1 8 in which the interval-valued data are analyzed.

In this paper, new appearance based method is proposed in the framework of Symbolic Data Analysis (SDA),1 '9 namely, symbolic KDA for face recognition, which are generalization of the classical KDA to symbolic objects. In the first step, we represent the face images as symbolic objects (symbolic faces) of interval type variables. The representation of face images as symbolic faces accounts for image variations of human faces under different lighting conditions, orientation and facial expression. It also drastically reduces the dimension of the image space without losing a significant amount of information. Each symbolic face summarizing the variation of feature values through the different images of the same subject. In the second step, we applied symbolic KDA algorithm to extract interval type non-linear discriminating features. According to this algorithm, In the first phase, we applied kernel function to symbolic faces, as a result a pattern in the original input space is mapped into a potentially much higher dimensional feature vector in the feature space, and then perform in the feature space to choose subspace dimension carefully. In the second phase, Symbolic KDA is applied to obtain interval type non-linear discriminating features, which are robust to variations due to illumination, orientation and facial expression. Finally, minimum distance classifier with symbolic dissimilarity measure1 is employed for classification. Proposed method has been successfully tested using two standard databases ORL and Yale face database.

The remainder of this paper is organized as fol

lows: In section 2, the idea of constructing the symbolic faces is given. Symbolic KDA is developed in section 3. In section 4, the experiments are performed on the ORL and Yale face database whereby the proposed algorithm is evaluated and compared to other methods. Finally, a conclusion and discussion are offered in section 5.

2. Construction of Symbolic Faces

Consider the face images T i , r 2 , . . . ,T„, each of size N x M from a face image database. Let ft = {Ti, T2, • •., r „ } be the collection of n face images of the database, which are first order objects. Each object T; e ft,/ = 1 , . . . , n, is described by a feature vector (Yi,...,Yp), of length P = NM, where each component Yj,j = l , . . . , p , is a single valued variable representing the intensity values of the face image Ti.

An image set is a collection of face images of m different subjects; each subject has same number of images but with different orientations, expressions and illuminations. There are m number of second order objects(face classes) denoted by E = { c i , . . . , c m } , each consisting of different individual images T; e ft, of a subject. We have assumed that images belonging to a face class are arranged from right side view to left side view. The view range of each face class is partitioned into q sub face classes and each sub face class contains r number of images. The feature vector of kth sub face class ck of ith face class Cj, where k = 1,2,.. . , q, is described by a vector of p interval variables Yi,...,Yp, and is of length p = NM. The interval variable Yj of kth sub face class ck of ith face class is described as

Yj(ck) = [xk

j,xk

j}. (1)

where x% and xk< are minimum and maximum in-tensity values, respectively, among j t h pixels of all the images of sub face class ck. This interval incorporates information of the variability of j t h feature inside the kth sub face class c\. We denote

Xk = (Y1(ck),...,Yp(c

k)),i = l,...m,k=l,...,q.

(2) The vector Xk of interval variables is recorded for each kth sub face class ck of ith face class. This vector is called as symbolic face and is represented as:

X(ck) = (ak1,...,a

kp). (3)

58 Face Recognition using Symbolic KDA in the Framework of Symbolic Data Analysis

where a^ = Y}(c?) = [x^x^] j = 1 , . . . ,p, k = 1 , . . . ,q and i = 1 , . . . , m. We represent the qm symbolic faces by a matrix X of size p x qm , consisting of column vectors Xf, i = 1, . . . ,m and k = 1 , . . . ,q.

3. Acquiring Non Linear Subspace using symbolic KDA Method

Let us consider the matrix X containing qm symbolic faces pertaining to the given set fi of images belonging to m face classes. The centers x^ € 3? of the intervals ahj = Yj(c^) = [x^,x*j], are given by

kc _ fiij + Sij] V ~ 2

where j = 1 , . . . ,p, k = 1 , . . . , q and i = 1 , . . . , m. The p x qm data matrix X c containing the centers

(4)

r. . a;?, e 3? of the intervals for qm symbolic faces. The

p- dimensional vectors X]° = (x^0,..., xkip ), X% =

(s.ii ,...,Xip) and Xt = (xkt,..., xk

p) represents the centers, lower bounds and upper bounds of the qm symbolic faces, respectively. Let $ : W —> F be a nonlinear mapping between the input space and the feature space, the nonlinear mapping, $, usually defines a kernel function. Let K e ^gmxim define a kernel matrix by means of dot product in the feature space:

Kij = (HXi) • *{Xj)) (5)

In general, the Fisher criterion function in the feature space F can be defined as

VTS*V J(V) = (6)

VTS*V

where V is a discriminant vector, Sf and 5 * are the between class scatter matrix and the within class scatter matrix respectively. The between class scatter matrix and the within class scatter matrix in the feature space F are defined below:

face class i, mf is the mean of the mapped symbolic faces in face class i, m* is the mean across all mapped qm symbolic faces. From the above definitions, we have S* = Sf + S*. The discriminant vectors with respect to the Fisher criterion are actually the eigenvectors of the generalized equation SfV = \S®V. According to the theory of the reproducing kernel, V will be an expansion of all symbolic faces in the feature space. That is, there exist coefficients bi(L = 1 , . . . , m) such that

qm

V=J2bL$(xt) = HA (9) L=l

H = ^C

Mx^)...^(x!c),...,^(xm^...^(xic)} where A = (bi,...,bgm). Substituting equation (9) into equation (6). We can obtain the following equation:

ATKWKA J(A) = (10)

ATKKA

where K is a kernel matrix, W = diag(W\,..., Wm), Wi is a ft x ft matrix whose elements are ^-. From the definition of W, it is easy to verify that W is a qm x qm block matrix. In fact, it is often necessary to find s discriminant vectors, denoted by a\,..., as, to extract features. Let V = [a\,..., as}. The matrix V should satisfy the following condition:

V = argmax \vTsbv\ \vTstv\ (11)

Sf = - E ^m? - <)("»* - mof (7) HX?)), (*(XmC)

where Sb = KWK and St = KK. Since, each symbolic face Xk is located between the lower bound symbolic face X_t and upper bound symbolic face Xt, so it is possible to find most discriminating non-linear interval type features [Bj, Bi ]. The lower bound features of each symbolic face X* is given by

B! = V?*(X2),l = l,...,s. (12)

where *(X*) = [ ^ ( X ^ ) • * (*? )> . . . ,*(xf) •

$(Xk),..-, « « C ) i = l

1 m Q i

st =—E E(*(x"c) - m?)(*(xin - mt)

*(2C?))]. Similarly the upper bound features of each sym

bolic face Xjfc is given by

»=i fc=i (8)

where X-f denotes the kth symbolic face of ith face class, qi is the number of training symbolic faces in

B,fc = Vf * ( * * ) , / l , . . . , s . (13)

Let Ctest = [ r i , . . . , r ; ] be the test face class contains face images of same subject with different expression, lighting condition and orientation. The test

symbolic face Xtest is constructed for test face class Ctest as explained in the section 2. The lower bound test symbolic face of test symbolic face Xtest is described as X_test ~ (£ies t! • • • ,£pest)-Similarly, the upper bound test symbolic face is described as X~test = (x\est,.. •,xt

pest).

The interval type features [ 5 t e s t; B ] of test sym

bolic face Xtest are computed as:

Rtest = V^{Xtest). (14)

Btest = V^(Xtest). (15)

where I = 1 , . . . , s.

4. Experimental Results

Face recognition system using symbolic KDA method identify the face by computing nearest face image for a given unknown face images using a minimum distance classifier with Minkowsky's symbolic dissimilarity measure proposed by De Carvalho and Diday [1] is employed for classification. The proposed symbolic KDA method with polynomial kernels is experimented with the face images of the ORL and Yale databases. The effectiveness of proposed methods is shown in terms of comparative performance against five popular face recognition methods. In particular, we compared our algorithms with eigenfaces29 fisherfaces3 symbolic PCA13 and symbolic KPCA.14 The experimentation is done on system with CPU:Pentium 2.5 GHz.

4.1. Experiments using ORL database

We assess the feasibility and performance of the proposed symbolic LDA on the face recognition task using ORL database. The ORL face database is composed of 400 images with ten different images for each of the 40 distinct subjects. All the 400 images from the ORL database are used to evaluate the face recognition performance of proposed methods. We have manually arranged the face images of same subject from right side view to left side view. Six images are randomly chosen from the ten images available for each subject for training, while the remaining images are used to construct the test symbolic face for each trial. Table 1 presents the experimental results for each method corresponding to ORL database. The experimental results show that the proposed method with polynomial kernel of degree

P. S. Hiremath and C. J. Prabhakar 59

three outperforms the classical factor analysis methods.

Table 1. Comparison of classification performance using ORL Database.

Methods

Fisherfaces

Eigenfaces

Symbolic PCA

Symbolic KPCA

Symbolic LDA

Symbolic KDA

Training time (sec)

98

102

38

87

85

19

F earure Dimension

86

189

71

109

34

28

Recognition Rate (%)

92 8

87.65

94 85

89 15

96 00

98 50

4.2. Experiments on the Yale Face database

The experiments are conducted using Yale database to evaluate the excellence of the symbolic KDA for the face recognition problem. The Yale Face database consists of a total 165 images obtained from 15 different people, with 11 images from each person. In our experiments, 9 images are randomly chosen from each class for training, while the remaining two images are used to construct test symbolic face for each trial. The recognition rates, training time and optimal subspace dimension are listed in Table 2. From Table 2, we note that the symbolic KDA method with polynomial kernel of degree using smaller number of features outperforms the classical factor analysis methods with a larger number of features.

Table 2. Comparison of classification performance using Yale Face database.

Methods

Fisherfaces

Eigenfaces

Symbolic PCA

Symbolic KPCA

Symbolic LDA

Symbolic KDA

Trarung time (sec)

59

85

35

43

98

12

Feature Dimension

23

110

41

32

56

15

Recognition Rate (%)

8995

82.04

91 15

92 00

94 55

98 15

5. Conclusion

In this paper, we introduce a novel symbolic KDA method for face recognition. Symbolic data representation of face images as symbolic faces, using interval variables, yield desirable facial features to cope up with the variations due to illumination, orientation and facial expression changes. The feasibility of

60 Face Recognition using Symbolic KDA in the Framework of Symbolic Data Analysis

the symbolic KDA has been tested successfully on

frontal face images of ORL and Yale databases. Ex

perimental results show tha t symbolic KDA method

with polynomial kernel of degree three leads to su

perior recognition rate as compared to classical fac

tor analysis methods. The proposed symbolic KDA

outperforms symbolic PCA, symbolic LDA and sym

bolic K P C A under variable lighting conditions, ori

entations and expressions.

The proposed symbolic KDA has many advan

tages compared to classical factor analysis methods.

The drawback of classical factor analysis methods is

tha t in order to recognize a face seen from a par

ticular pose and under a particular illumination, the

face must have been previously seen under the same

conditions. The symbolic KDA overcomes this limi

tat ion by representing the faces by interval type fea

tures so tha t even the faces seen previously in differ

ent poses, orientations and illuminations are recog

nized. Another important merit is tha t we can use

more than one probe images with inherent variabil

ity of a face for face recognition. Therefore, symbolic

KDA improve the recognition accuracy as compared

to classical factor analysis methods at reduced com

putat ional cost. This is clearly evident from the ex

perimental results. Further, the symbolic KDA yields

significantly bet ter results than other symbolic factor

analysis methods.

R e f e r e n c e s

1. Bock, H. H., Diday, E. (Eds.) 2000. Analysis of Symbolic Data. Springer Verlag.

2. Baudat, Anouar, "Genralized Discriminant Analysis using Kernel Approach", Neural Computation, 2385-2404, 2000.

3. P. Belhumeur, J. Hespanha, D. Kriegman,1997. Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection. IEEE Transaction on PAMI.19 (7): 711-720.

4. Bruneli and Poggio: Face Recognition: Features versus Templates, vol -15. IEEE Trans. Pattern Analysis and Machine Intelligence, (1993) 1042-1052.

5. Chellappa, Wilson, Sirohey, 1995. Human and machine recognition of faces: A survey, vol - 83(5). Proc. IEEE) 705-740.

6. Chen, Liao, Ko, Lin, Yu,2000. Anew LDA based face recognition system which can solve the small sample size problem. Pattern Recognition 33, 1713-1726.

7. Choukria, Diday, Cazes, 1995. Extension of the principal component analysis to interval data. Presented at NTTS'95: New Techniques and Technologies for statistics, Bonn.

8. Choukria, Diday, Cazes, 1998. Vertices Principal Component Analysis with an Improved Factorial Representation. In: A. Rizzi, M. Vichi, H. Bock (eds.): Advances in Data Science and Classification. Pp.397-402, Springer Verlag.

9. Diday, 1993. An Introduction to symbolic data analysis. Tutorial at IV Conf. IFCS.

10. K. Etemad and R. Chellappa, 1997. "Discriminant Analysis for Recognition of Human Face Images", J. Optical Soc. Am. vol-14, pp 1724-1733.

11. Fisher, 1938. The statistical utilization of multiple measurements, Ann. Eugenics, 8, 376-386.

12. M. A. Grudin, 2000. "On internal representations in face recognition systems", Pattern recognition, vol.33, no.7, pp.1161-1177.

13. Hiremath. P. S, Prabhakar. C. J, 2005. "Face Recognition Technique using Symbolic PCA Method", Proc. Int. Conf. on Pattern Recognition and Machine Intelligence (PreMI'05), Kolkata, 266-271, Springer Verlag.

14. Hiremath. P. S, Prabhakar. C. J, "Face Recognition Technique using Symbolic kernel PCA Method", Proc. Int. Conf. on Cognition and Recognition (COGREC'05), Mysore, 801-805, Allied Publishers (2005).

15. Hong, Yang,1991. Optimal discriminant plane for a small number of samples and design method of classifier on the plane. Pattern Recognition 24(4),317-324.

16. Kirby, Sirovich, 1990. Applications of the Karhunen-Loeve procedure for the characterization of human faces, v-12(l). IEEE Trans. Pattern Anal. Machine Intell. 103-108.

17. Kernel Machines http:/ /www.kernel-machines, org, 2000.

18. Lauro, Verde, Palumbo, 1997. Analysis of symbolic data, Bock and Diday (Eds), Springer Verlag.

19. Liu, Cheng, Yang, 1992a. A generalized optimal set of discriminant vectors. Pattern Recognition 25,731-739.

20. Liu, Wechsler, 2002. Gabor feature based classification using the enhanced Fisher linear discriminant model for face recognition. IEEE Trans. Image Process. 11(4), 467-476.

21. Liu, Wechsler, 2000: Robust coding schemes for indexing and retrieval from large face databases, IEEE Trans. On image processing 9, 132-137.

22. Liu, Cheng, Yang, 1993. Algebraic feature extraction for image Recognition based on an optimal discriminant criterion, Pattern Recognition, 26, 903-911.

23. Lu, Plataniotis, Venetsanopoulos, 2003b. Face Recognition using LDA based algorithms. IEEE Trans Neural Networks, 1(1), 195-200.

24. Mika, Ratsch, Scholkopf, Muller, "Fisher Discriminant Analysis with kernels" Proc. IEEE Int Workshop Neural Networks for signal Processing, 41-48,1999.

25. Muller.Mika.Ratsch, Tsuda, Scholkopf,2001, " An

P. S. Hiremath and C. J. Prabhakar 61

Introduction to kernel based Learning Algorithms", IEEE Trans. Neural Networks, voll2, 1810201.

26. A. Pentland, B. Moghaddam and T. Starner, 1994. "View based and modular Eigenfaces for Face Recognition" , Proc. Computer Vision and Pattern Recognition, pp.84-91.

27. Scholkopf and A. Smola, and K. Muller, "Nonlinear Component Analysis as a kernel Eigenvalue Problem", Neural Computation, vol.10, pp.1299-1319, 1998.

28. D. Swets, J. Weng, 1996. Using discriminant eigen-features for image retrieval. IEEE. Transactions on PAMI.18, 831-836.

29. Turk, Pentland, 1991. Eigenfaces for Recognation, v-3. J Cognitive Neuro Science, 71-86.

30. Yu, Yang, 2001. A Direct LDA algorithm for high dimensional data with application to face recognition,

Pattern Recognition, 34(7), 2067-2070. 31. M. H. Yang, "Kernel Eigenfaces vs. Kernel Fish-

erfaces: Face Recognition using Kernel Methods", Proc. Fifth IEEE Int'l Conf. Automatic Face and Gesture Recognition, pp.215-220,2002.

32. M. H. Yang, N. Ahuja and D. Kriegman, " Face Recognition Using Kernel Eigenfaces", Proc. IEEE Int'l Conf. Image Processing, 2000.

33. Ye, Li, 2004. LDA/QR. An efficient and effective dimension reduction algorithm and its theoretical foundation. Pattern Recognition, 27(9), 1209-1230.

34. Zhao, Chellappa, Phillips, 1999. Subspace linear discriminant analysis for face recognition. Technical Report, CS-TR4009, University of Maryland.

35. Zhao, Chellappa, Phillips, Rosenfeld, 2003. Face Recognition: A literature survey, ACM Comput. Surveys, 35(4), 399-458.

62

Minutiae-Orientation Vector Based Fingerprint Matching

Li-min Yang*, Jie Yang and Yong-liang Zhang

Institute of Image Processing and Pattern Recognition Shanghai JiaoTong University (SJTU),

Shanghai, 200240, P.R. China E-mail: [email protected]. cri*

Fingerprint matching is an important problem in fingerprint identification. In this paper, we propose a novel minutiae-orientation vector(MOV) for fingerprint matching. MOV combines both orientation information and neighborhood minutiae information. A set of reference point pairs is identified based on MOV. Alignment is done coordinately and directionally based on these reference point pairs. Matching score is computed. The experimental results show that the proposed matching scheme is very competitive as compared with those algorithms, which have been participated in FVC2004 DB2.A.

Keywords: Fingerprint alignment; Fingerprint matching; Minutiae-orientation vector.

1. Introduction

Fingerprints have been used for identifying individual from the ancient time. Fingerprint identification is the most widely used biometric authentication method at present. Many fingerprint matching methods have been reported in literatures. Among them the minutiae-based methods are most popular. Fingerprint can be represented by a set of minutiae, including ridge endings and bifurcations. Each minu-tia can be described by its spatial location associated with the direction and the minutia type.

A novel fingerprint feature named the adjacent feature vector(AFV) was defined in [1]. An AFV consists of four adjacent relative orientations and six ridge counts of a minutia. An AFV-based fuzzy scoring strategy is introduced to evaluate similarity levels between matched minutiae pairs. [2] proposed a triangular matching method to cope with the deformation and validate the matching by dynamic time warping. [3] used orientation-based minutia descriptor for minutiae matching. [4] integrated point pattern and texture pattern by firstly aligning two fingerprints using point pattern and then matching their texture features. [5] defined a novel feature vector for each fingerprint minutia based on the global orientation field. These features are used to identify corresponding minutiae between two fingerprint impressions by computing the Euclidean distance between vectors. [6] proposed a minutiae matching technique, which used both local and global struc

tural information. [7] developed a method to evenly align two sets of minutiae based on multiple reference minutiae (MRM). MRM is distributed in different regions of fingerprint and two sets of minutiae are globally and evenly aligned. Thus a pair of unpaired corresponding minutiae far away from the reference minutiae in SRM-based method maybe paired in MRM based method.

In this paper, we propose a novel minutiae-orientation vector(MOV) for each minutia, which combines both neighborhood minutiae and surrounding orientation information. Based on MOV, one main-reference point pair and a set of associate-reference point pairs are identified. Alignment is performed according to these reference point pairs. Matching score is computed thereafter. Experiments are conducted on the public domain collection of fingerprint images, DB2.A in FVC2004. Comparing with earlier works like [1], the strength of the proposed scheme lies in that the MOV structure combines both neighborhood minutiae and surrounding orientation information. MOV has limited dimensions, which means shorter computational time. Further more, the multi-reference point pairs mechanism is more robust to deformations than one-reference point pair method.

The rest of this paper is organized as follows. Section 2 describes the proposed minutiae-orientation vector. Our fingerprint matching method is presented in Sec. 3. The experimental results are reported in Sec. 4. Section 5 concludes this paper.

Li-min Yang, Jie Yang and Yong-liang Zhang 63

2. Definition of the Novel Minutiae-Orientation Vector(MOV)

In general, a minutia point M from a fingerprint minutiae set can be described by a feature vector given by:

F={x,y,w). (1)

where (x, y) is its coordinate, w is its orientation. Generally, the orientation of minutia is in the

range [—n/2, it/2\. Considering that the difference of 30° may result in the difference of 150° due to the effect of rotation of fingerprint image on the directions of ridges, the orientation difference of two minutia wi and o)2 can be calculated as follows:5

d(o)i,0)2) =

if -it/2 < (wi - o>2) < Tt/2,

if -it < (wi - (j2) < -Tt/2, 0)i — 0)2 — 7T

if it/2 < (o)i - o)2) < it.

Let M(x, y, o>) denotes an arbitrary minutia from a fingerprint image, we define our novel minutiae-orientation vector(MOV) as follows:

Firstly, draw a circle C around M with center M(x, y) and radius 6 * T (T is the local average ridge distance, empirically determined). Let 6\ = o), #2 = o)+7r/4, 03 = W+7I-/2 and 04 = o)+37r/4. We plot four lines l\,h,h, and Z4 along the angles 61,82,03, and 64 with respect to X axis through the minutia point M. Label the eight intersection points of circle C and lines h,h,h, and Z4 as C?,C?,--- ,C8

M. C f has coordinate (XCM,yc\i) and orientation O)CM, where i = 1,2, • • • , 8 (illustrated in Fig. 1).

.'-, ? ,

Secondly, search around M to find 8 selective minutiae, M : , M 2 , - - - ,M 8 . Af" has coordinate (xl,yl) and orientation o)1, where i = 1,2, • • • ,8. It is worth to notice that each selective minutia M%

should satisfy the following two conditions (illustrated in Fig. 1):

DMMi = min DMMk

where k = 1,2, ••• ,N,f(i) = [o)+(i-l)x45°,o)-Mx 45°), o)M M i denotes the direction from M to minutia M%, and DMM1 denotes the distance between M and minutia Ml, i = 1,2, • • • ,8, TV is the number of minutiae on this fingerprint.

The proposed novel minutiae-orientation vec-tor(MOV) at minutia point M can be defined as:

MOV(M) = { d ( o ) , o ^ ) , d ( o ) , o / ) ,

d(o>, UMMi), DMMi }i=l,2,- • ,8 (3)

where ^MCM € (—7r/2,7r/2) is the direction of M to

This vector MOV(M) of the minutia M is invariant to rotation and translation of the fingerprint images. As we know, if two minutiae from two images of the same finger are corresponding minutiae, they will have similar MOVs.

3. Fingerprint Matching Based on M O V

A minutia can be described by its position, direction and type (ridge ending or ridge bifurcation). Only minutia position and minutia direction are used in this paper. In order to align two point sets before calculating the matching score, we need to identify a set of reference point pairs.

3.1. Reference Point Pairs Identification

Suppose Tfe and Qi (where k = 1,2, •••if, I = 1.2, • • • L, K,L are the numbers of minutiae detected on T and Q, respectively.) are two minutiae from

Fig. 1. Illustration of minutiae-orientation vector(MOV)

64 Minutiae-Orientation Vector Based Fingerprint Matching

template fingerprint and query fingerprint, respectively. The similarity level between Tk and Qi is calculated as:

S{Tk,Qi) Thr~TZk'Ql) if <£(Tk,Qi)<Thr,

0 otherwise. (4)

where <£(Tk,Qt) = \MOV{Tk) - MOV{QL)\ is the Euclidean distance between Tk and Qi's MOVs, and Thr is the threshold. The similarity level S(Tk,Qi), 0 < S(Tk,Qi) < 1, describes a matching certainty level of two MOVs instead of simply matched or not. S(Tk,Qi) = 1 implies a perfect match while S(Tk,Qi) = 0 implies a total mismatch.

Note that not all dimensions of the MOV exist. When calculating the similarity level between Tk and Qi, only the dimensions existed in both minutiae are employed to decide the similarity level.

The best matched minutiae pair (A, B) is obtained by maximizing the similarity level as:

S(A,B) = max(S(Tk,Ql)). (5)

Minutiae pair (A, B) is identified as the main-reference point pair, while A and B's surrounding intersection point pairs (Cf,Cf), i = 1,2, •••8 are identified as associate-reference point pairs.

3.2. Fingerprint Alignment

Aligning two minutiae sets T and Q is to rotate and shift one of them in order to make minutiae in T overlap approximately the corresponding counterparts in

Q. After obtaining main-reference point pair

(A(xa,ya,wa), B(xb,yb,Ub)), and associate-reference point pairs (C/*(a;?,j/?,u;?),Cf(zj.yj.wj)), i = 1,2, • • • 8, the rotation parameter Aw between template minutiae set T and query minutiae set Q is computed by:

AW = a ( W 6 - W a ) + / 3 , ^ K b - < ) . (6) i

where a and $ are weights, and a + ]>3/?i = l , i =

l ,2 , - - -8 . Rotate Cf to Cf by angle Aw with respect to

reference point B, according to the following formula:

cos Aw sin Aw \ ( x\ — Xb sin AUJ - cos Aw J \yf - Vb

(7)

where (Xf, Yf) is the coordinate of Cf . Then the translation vector (Ax, Ay)T between

T and Q can be computed by :

(8)

where 7 and 77* are weights, and -y + ^2rji — 1, i =

1,2, ••-8. After obtaining the translation and rotation vec

tor (Ax, Ay) and Aw, we translate and rotate all the minutiae on query set Q, according to the following formula:

(cos Aw sin Aw 0 sin Au> — cos Aw 0

0 0 1 (9)

where (xi,yi,Wi)T, represents minutiae on the query minutiae set and (xf ,yf,wf)T represents the corresponding aligned minutiae.

Let Q' denote the new minutiae set of the query fingerprint after transformation with the estimated translation and rotation parameters.

3.3. Matching Score Computation

For the transformed minutiae set Q', we re-compute the MOVs of each minutia. By calculating and comparing the similarity level of minutiae's MOVs, we find the corresponding minutiae pairs between the transformed minutiae set {Q'[}i=i,2,••• ,L and the originally extracted minutiae set {Tfc}fe=ii2)... tK- Let {rc„,<5cn}n=i,2,-,AP, N' < mm(K,L), denote the corresponding minutiae pairs, then the final matching score Macore between the query and template fingerprints can be calculated according to the following equation:

JV'

Ms, "£s(TCn,Q'cJ. (10) n = l

where the similarity level S(TCn,Q'Cn) is computed according to Eq. (4).

Li-min Yang, Jie Yang and Yong-liang Zhang 65

4. Experimental Results

The experiments reported in this paper have been conducted on the public domain collection of fingerprint images, DB2.A of FVC2004.8 The fingerprints of DB2_A are acquired through optical sensor "U.are.U 4000" by Digital Persona, and each fingerprint has a size of 328x364 pixels by 500 dpi. This database contains 800 fingerprints captured from 100 different fingers, eight impressions for each finger. According to FVC rules,8 each fingerprint is matched against the remaining fingerprints of the same finger in genuine match. Hence, the total number of genuine tests is (8x7xl00)/2 = 2800. In imposter match, the first fingerprint of each finger is matched against the first fingerprint of the remaining fingers. Hence, the total number of false acceptance tests is (100x99)/2 = 4950.

hta. DtakhK*!'..

Fig. 2. ROC curve on FVC2004 DB2.A obtained by the proposed algorithm

Table 1. Comparison of the proposed algorithm with P039 and P071 on DB2-A

Algorithm

P039 Our Algorithm P071

EER (%)

1.58 2.04 2.59

FMR 100(%)

2.18 2.23 3.14

FMR 1000(%)

5.79 3.30 4.96

Average Match Time(s)

0.83 0.45 0.57

We tested the relationship between FMR(False Match Rate) and FNMR(False Non-Match Rate) on DB2.A. The Receiver Operating Characteristic

(ROC) curve obtained by the proposed algorithm is illustrated in Fig. 2. In Table 1, we compare the performance of the proposed scheme with the results of the two best algorithms called "P039" and "P071" on DB2.A of FVC2004. According to the ranking rule in terms of EER in FVC2004, the proposed scheme is in the second place on DB2_A.

5. Conclusions

In this paper, we propose a MOV-based matching scheme. For each minutia, a minutiae-orientation vector(MOV) is constructed. It is rotation and translation invariant. Based on MOV, a set of reference point pairs between template and query fingerprint minutiae sets is identified. Alignment is done according to these reference point pairs. Fingerprint matching based on the proposed MOVs is processed. Experimental results on the public domain collection of fingerprint images, DB2-A of FVC2004 show that fingerprint matching based on the proposed MOV can achieve good performance.

References

1. Xifeng Tong, Jianhua Huang, Xianglong Tang, Darning Shi, Fingerprint minutiae matching using the adjacent feature vector, Pattern Recognition Letters 26 (2005) 1337-1345

2. Z. M. Kovcs-Vajna, A fingerprint verification system based on triangular matching and dynamic time warping, IEEE Trans. Pattern Anal. Mach. Intell. 22 (11) (2000) 1266-1276

3. M. Tico, P. Kuosmanen, Fingerprint matching using an orientation-based minutia descriptor, IEEE Trans. Pattern Anal. Mach. Intell. 25 (8) (2003) 1009-1014

4. A. Ross, A. K. Jain, J. Reisman, A hybrid fingerprint matcher, Pattern Recognition 36 (7) (2003) 1661-1673

5. Jin Qi, Suzhen Yang, Yangsheng Wang, Fingerprint matching combining the global orientation field with minutia, Pattern Recognition Letters 26 (2005) 2424-2430

6. Jiang, X., Yau, W. Y., Fingerprint minutiae matching based on the local and global structures In: Proc. Internat. Conf. Pattern Recognition, ICPR2000, Barcelona, Spain, vol. 2, pp. 1042-1045

7. En Zhu, Jianping Yin, Guomin Zhang, Fingerprint matching based on global alignment reference minutiae, Pattern Recognition 38 (2005) 1685-1694

8. Biometric Systems Lab., Pattern Recognition and Image Processing Lab., Biometric Test Center., [Online] Available:http://bias.csr.unibo.it/fvc2004/

66

Recognition of Pose Varied Three-Dimensional Human Faces Using Structured Lighting Induced Phase Coding

Debesh Choudhury

Instruments Research and Development Establishment Photonics Division,

Raipur Road, Dehradun 248008, Uttaranchal, India E-mail: [email protected]

Face recognition received a lot of attention in the recent past. Most of the reported techniques are based on use and analysis of the two-dimensional (2D) information available in the 2D facial images. Since human faces are originally three-dimensional (3D) objects, association of 3D sensed information can make face recognition more robust and accurate. This paper reports a method for recognition of faces by utilizing phase codes obtained from projected structured light patterns onto the faces. The phase differences associated with the distorted projected patterns are detected and the computed phase maps are utilized to synthesize complex signature functions, spatial frequency distributions of which are directly proportional to the computed phase maps and hence to the original 3D face shape. The synthesized signature functions of the test faces are compared with that of the target face by digital cross-correlation. Analyses of the cross-correlation intensities (squared modulus) complete the recognition process. Preliminary experimental results are presented for faces with wide variations of pose (out-of-plane head rotations).

Keywords: Biometrics; face recognition; 3D object recognition.

1. Motivation

Out of all the biometric authentication areas, the topic of human face recognition research received perhaps the most attention.1'2 A lot of techniques exist in the literature for face recognition utilizing feature analysis, eigenface approach, neural processing, global two-dimensional (2D) matching of 2D images etc.3'4 All these techniques involve utilization or analysis of information obtained from 2D facial images. Since a human face is originally a 3D object, direct association of 3D shape information would make face recognition more accurate and robust.

2. Introduction

Three-dimensional face recognition is a relatively recent trend, although face recognition based on 3D modeling of the 2D facial image have been well-addressed. The face recognition research, in general, is experiencing a paradigm shift towards 3D face recognition that utilize directly (or indirectly) sensed 3D shape information.2 A recent literature review5 points to a variety of methods those utilized range images (and or stereo images) and applied several techniques, such as segmentation, feature analysis, principal component analysis, Hausdorff distance method, iterative closest point approach etc. for the purpose of 3D face recognition.

The face recognition in the 3D regime has sev

eral advantages. 3D face recognition overcomes the limitations due to pose (viewpoint or orientation) and lighting variations. It may also solve the problem of discriminating live face from fake faces6 because faking a 3D face would be more difficult than to fake a 2D image. However, different 3D recognition schemes may require different forms of 3D signatures. Moreover, standard 3D face databases are yet to be available freely. Nevertheless, superior performance of 3D face recognition techniques over the image-based ones calls for more studies in this area. In what follows we consider a technique for recognition of 3D faces using directly sensed 3D shape induced phase coding.7

A structured light pattern (SLP) is projected onto the faces. The 3D depth variations in the faces induce proportional distortions in the projected patterns. The SLP projected face images are captured using a CCD camera. The phase differences associated with the distortions of the SLP projections are extracted by using a Fourier-fringe analysis method.8 The extracted phase values characterize the 3D shape of the faces. The detected phase maps are utilized as shape codes to synthesize 2D spatial harmonic functions. The encoded spatial functions can be used as unique signatures for the face objects. The synthesized signature functions of the test faces are compared with that of the target face by digital cross-correlation (CrossCorr). The CrossCorr inten-

Debeah Choudhury 67

Camera SLP projected face image

Signature data base

Aexp{i2np^i(x,y)fi.x,y)}

Aexp{i2Tt$$2(x,y)j{x,y)}_

Aexp {i2n$Mx,y)flx,y)}

Asxp{i2nPfn(x,y)fi.x,y)}

Phase map ty0(x,y)

v

o Spatial coding

Aexp{ i2n$$0(x,y)fix,y)}

H Cross-correlation of the signature functions

Aexp {i2nP$t(x,y)fix,y)} ® Aexp {i2itffy0(x,y)f{x,y)}

1} I Noise Correlation peak

or Noise and no peak

Peak

Pig. 1. Schematic diagram of the proposed face recognition system along with the algorithm of the phase coded cross-correiatkm.

sities (squared modulus) are analyzed. The signal-to-noise-ratio (SNR) of CrossCorr results gives a quantitative measure for recognition or rejection of a face class.

3. Theory

Figure 1 represents the schematics of the proposed face recognition system. The configuration consists in a white-light projector connected with a computer. A SLP from the computer, say, a sinusoidal grating, is projected on the human face object positioned in front of a uniform background screen. A CCD camera is connected with the same computer that captures the projected patterns with and without the face object. The captured images are processed to compute the associated phase difference using Fourier-fringe analysis.8 In short, the first order Fourier spectra of the distorted and the undistorted SLP images are

selected by spatial filtering and their inverse Fourier transforms are computed. Now, the associated phase difference is extracted by computing the arctangent of the product of the inverse Fourier transform of the first order spectrum of the distorted SLP and the conjugate of that of the undistorted SLP.8 This computed phase map characterizes the 3D shape of the face and the phase values are, in general, wrapped, i.e., the phase values may have 2n ambiguity due to large depth variations in the face objects. The wrapped phase maps are used for synthesizing the signature functions by spatial coding. The phase coding algorithm and the associated computational process are described in the lower dashed block of Fig.l. If 4>t(x, y) and <l>o(x, y) represent the computed phase map corresponding to the SLP projected target face image and a general test object face image respectively, we define the signature functions st(x, y) and sa(x, y) of the target and the test object faces respectively as

st(x,y) = Aexp{i2n@<l>t(x,y)f(x,y)} , (1)

sa(x,y) = Aexp{i2ir0(t>o(x,y)f(x,y)} , (2)

where (x,y) represent the rectangular spatial coordinates, f(x, y) is a function of the coordinates, i = \ /^- l , f$ is a positive factor (fraction or whole number) and A is a constant. The function f(x,y) is used for spatial coding, which can be any suitable function of the coordinates, linear or powered. The signature functions st(x, y) and s0(x, y) represent the 3D shape information of the face objects spatially encoded in terms of spatial modulations. The spatial frequency distributions of the signatures are proportional to the computed phase map corresponding to the face objects. The signature functions of t he faces contain high spatial frequency information, hence we prefer to compare them using the frequency domain correlation pattern recognition method9 instead of the weE-known image domain techniques such as principle component analysis. The cross-correlation operation between the signature functions of the target face and the test face may be expressed as

c(x,y)= / l St(u,v)S*(u,v)N(u,v)

x exp[—i2w(ux + vy)]dudv (3)

where (u, v) is the spatial frequency coordinates corresponding to the spatial coordinates (x,y), St(u,v)

68 Recognition of Pose Varied Three-Dimensional Human Faces Using Structured Lighting Induced Phase Coding

and S0(u, v) represent the Fourier transforms of the signature functions St(x,y) and s0{x,y) respectively, * signifies complex conjugate, and N(u, v) is a notch type spatial filter that blocks the zero order components of St(u,v) and passes the higher orders satisfying the relation

N(u,v) = 1 — circ v ^

M t \min (4)

The minimum value of \4>t\ is used, because the spatial filter distribution pertaining to the computed phase map corresponding to the target face falls within that range. The j3 factor may be used to control the sensitivity of the spatial coding, and may be suitably selected after some test experiments. The Fourier transform St (u, v) may explicitly be written

as 10

+oo +oo

St(u,v) •= \ \ st{x,y)exp{-i2ir(ux+ vy)}dxdy

—oo —oo

+ o o + o c

= x//-p| i iM».(,.,)/fc»)-«x-n.}K*. — oo —oo

(5) Referring to the well-known method of stationary phase,11 it can be noticed that the main contribution to the integral of equation (5) comes from the points where the phase

ip(x,y)= (3(j)t (x,y)f(x,y)-ux- vy (6)

is stationary, i.e.,

s - ' S - 0 - (7)

Now, partially derivating equation (6) with respect to x and y and applying the method of stationary phase,11 we can have

u = (3

v = (3

,df(x,y) dMx,y) 4>t{x,y)—zz— +f(x^y)

{x,y)

dx

df(x,y) dy

+ f{x,y)

dx

d<l>t{x,y) dy

, (8)

• (9)

The face signature points (x, y) that give a dominant contribution to the spectrum distribution at (u, v) are governed by equations (8) and (9). Similarly, the spectrum distribution [S0(u,v)\ of the signature function [s0(x,y)] due to the general test face can also be expressed. If 4>0 ^ <j>t, the spatial frequency spectrum distributions of the general test face signature and the target face signature are different, and therefore they are uncorrelated. On the otherhand, if 4>o ~ 4>t, the spectrum distributions of the test face signature and the target face signature overlap in the spatial frequency domain and they are correlated.

3.1. Personalized spatial coding

It is interesting to point out the possibility of assigning different spatial coding functions for different persons. As for example, the coding functions fi(x,y), h{x,y), h{x,y) may be used to synthesize the signature functions of the persons 1, 2, 3 respectively. The signature function of the same face (say person 1) will not be the same for a different coding function [say with f2(x, y)}, because according to equations (8) and (9), the spatial frequency distribution of a face signature function is dependent of the coding function f(x,y). Person-specific coding function will render the spatial coding more stringent and will help to reinforce secure face recognition in an authentication scenario.

4. Results and discussions

The feasibility is experimented with faces of 20 persons. We put f(x,y) = x - y in equations (1) and (2). Movies of SLP projected face sequences are captured with variations in poses (i.e., out-of-plane head rotations) in both left-right and up-down directions (about 90 degrees) using a commercial digital camera (Sony Cybershot).

Debesh Choudhury 69

Fig. 2. Experimental results: (a) SLP projected 1st face; (b) same with rotated head; (c) SLP projected 2nd face; (d) phase map of (a); (e) phase map of (b); (f) phase map of (c); (g) signature of (a); (h) signature of (b); (i) signature of (c); (j) CrossCorr of (g) and (h); (k) CrossCorr of (h) and (i); (1) CrossCorr of (i) and (g);

The extracted frames are processed using the proposed spatial phase coding algorithm (Fig.l). The SLP projected face images are of 128x128 pixel size. The value of /3 is kept 0.05 throughout. Experimental results with two face samples are shown in Fig.2. The SLP projected face images are shown in Fig.2(a), Fig.2(b) (faces of the same person with different poses, i.e., faces of the same class) and Fig.2(c) shows that of a different person. The associated phase map corresponding to the depth-induced distortions in Fig.2(a), Fig.2(b) and Fig.2(c) are computed and are shown in Fig.2(d), Fig.2(e) and Fig.2(f) respectively. These phase maps are utilized to synthesize the signature functions. The real parts of the signature functions of the face images of Fig.2(a), Fig.2(b) and Fig.2(c) are shown in Fig.2(g), Fig.2(h) and

Fig.2(i) respectively. The normalized intensity of the CrossCorr function of Fig.2(g) and Fig.2(h), that of Fig.2(h) and Fig.2(i), and that of Fig.2(i) and 2(g) are shown respectively in Fig.2(j), Fig.2(k) and Fig.2(l). The high peak in Fig.2(j) exemplifies correct recognition between face signatures of pose varied faces of the same class, whereas Fig.2(k) and Fig.2(l) contain no well-defined peak but only noise, which clearly demonstrate discrimination of faces of the false class.

We present some more results to show the effects of pose variation on the cross-correlations. Figure 3 shows some sample results of two face objects. The SLP projected pose varied faces of a person are shown in Fig.3(a) - Fig.3(f). The SLP projected frontal face image of the same person is shown in Fig.3(g), whereas Fig.3(h) shows the same of another person. The CrossCorr between the signature functions of the faces of Fig.3(a) - Fig.3(f) with that of the face of Fig.3(g) are respectively shown in Fig.3(i) - Fig.3(n) which contain sharp correlation peaks. These sharp peaks are the evidences of correct recognition with pose varied faces of the same class. Figures 3(o) - 3(t) show the CrossCorr between the signature functions of the faces of Fig.3(a) - Fig.3(f) with that of the face of Fig.3(h). The CrossCorr in Fig.3(o) - Fig.3(t) contain no well-defined peak but only noise, which signify mismatch for faces of different class. Therefore, the proposed spatially coded signature functions of the 3D faces can be utilized to recognize the true class faces and to reject the false class faces.

We also show an example of personalized spatial coding by using two different spatial coding functions [f(x, y)\, results of which are shown in Fig.4. The SLP projected face images are shown in Fig.4(a), Fig.4(b) and Fig.4(c), all of which belong to the same class, i.e. they are faces of the same person. The signature functions corresponding to the faces of Fig.4(a) and Fig.4(b) are shown in Fig.4(d) and Fig.4(e) respectively with a coding function f(x, y) = x — y. Figure 4(f) shows the signature function corresponding to the face image of Fig.4(c) but with a coding function f{x,y) = 0.5a; - y. The normalized intensity of the CrossCorr function of Fig.4(d) and Fig.4(e), that of Fig.4(e) and Fig.4(f), and that of Fig.4(f) and Fig.4(d) are shown respectively in Fig.4(g), Fig.4(h) and Fig.4(i). In these CrossCorr results, only Fig.4(g) shows a sharp peak, but the other

70 Recognition of Pose Varied Three-Dimensional Human Faces Using Structured Lighting Induced Phase Coding

Y Z>' y ' /

(j)

/f" if

(k) (1) (m) (n)

' " > > • - .

' *>,-; (o) (P)

r * *W (q) (r) (s) (t)

Fig. 3. More experimental results with pose variations: (a) - (f) SLP projected pose varied face of a person; (g) SLP projected frontal face of the same person; (h) SLP projected frontal face of another person; (i) - (n) CrossCorr of signatures of (a) - (f) with that of (g); (o) - (t) CrossCorr of signatures of (a) - (f) with that of (h);

CrossCorr show only noise, i.e., the face of the same person with different coding functions don't correlate and match. Therefore, the signature functions can be made different by using different spatial coding functions. This gives a choice for personalized coding that can render more secure recognition.

Since, a correct recognition in our technique is evidenced by a high correlation peak around the centre, we can define the SNR as the ratio of the maximum correlation peak intensity around the centre to the mean noise in a rectangular area (128x128 pixel) around the centre (excluding a 21x21 pixel area at the centre where the correct peak is situated). The SNR of CrossCorr is computed for the pose variant face signatures. The frontal face signature is cross-correlated with the pose varied face signatures of the same person (true class) for head rotations in both left-right and up-down directions. The frontal face signature of a second person (false class) is also cross-

correlated with the pose varied face signatures of the first person. Plots of the computed SNR values versus the pose varied (left-right) face signature numbers for two human objects are shown in Fig.5. The SNR of CrossCorr between the faces of the same person show high values (more than 100), whereas the SNR of CrossCorr for different persons' face is low (less than 20). Therefore, it is possible to recognize pose varied faces of the same class (person) and reject the pose varied faces of different class (person) by analyzing the SNR of CrossCorr. Similar plots can be obtained with pose variation in the up-down direction. The rate of false recognition is nil in our feasibility experiments using face objects of 20 persons with wide variations of out-of-plane head rotations (about 90 degrees) in both left-right direction (25 poses approx) and up-down direction (25 poses approx).

We have utilized the correlation pattern recog-

Debesh Choudhwry 71

\Afe - * -,JSk* ..•#&•;*' (d) (e) (f)

V •**- *x '>3-^.yi**

(g) (h) (i)

Fig. 4. Experimental results with different coding functions: (a) SLP projected 1st face; (b) same with rotated head; (c) almost same as (a); (d) signature of (a); (e) signature of (b); (f) signature of (c) with different coding function; (g) CrossCorr of (d) and (e); (h) CrossCorr of (e) and (f); (i) CrossCorr of (f) and (d);

o "5 200

True Class -3K-False Class - -O

*-*"*"•

>-<)-oTo-o--{>-o--'^o-cv-p-A'>-<>,

<— Left side head rotation Frontal Right side head rotation —5*

Pose varied signature number

Fig. 5. SNR versus pose varied face signature plot.

nition9 method for matching, although other popular techniques, such as principal component analysis (PCA) or FisherFaces method, might have been tried. While PCA and FisherFaces work in the image domain, correlation pattern recognition works in the frequency domain and offer advantages such as shift-invariance, the ability to accommodate in-class image variability.12 Also, the spatial coding in our proposed

method creates rather higher spatial frequencies in the coded signatures, which calls for a spatial frequency sensitive matching technique and frequency domain correlation pattern recognition method satisfies that. In-plane rotation and scale are not considered in the present study, which can be surmounted using matured techniques based on wavelet and circular transforms.13,14

Although, the results presented here are based on the test experiments carried out on a very limited database of our own, because standard 3D face databases are yet to be available freely, the feasibility of the proposed 3D face recognition algorithm is successfully proved. Preliminary tests with wide pose variations show the promise of the proposed technique. It is worth mentioning that explicit reconstruction of the 3D shape is not required which is a tedious job. However, the system has to be calibrated as any other structured lighting based system.15 More rigorous testing are required with changed conditions, such as for expression variations, with and without glasses, beards, intentional disguise etc. Since, the 3D shape of a face may change with age (years), because the flesh and the tissues may change drastically, so the 3D signature database must be updated regularly for authentication purposes.

5. Conclusion

A technique for recognition of pose varied three-dimensional faces is presented. It utilizes the three-dimensional shape cues of the faces obtained by structured light pattern projection induced spatial phase coding. Experimental feasibility is tested using pattern projected face images with wide variations in pose captured by a commercial digital camera. The test experiments bring out the novelty with excellent results of true class face recognition and false class face rejection.

Acknowledgments

All the people who agreed to expose their faces for the experiments deserve sincere thanks and gratitudes. Some helps from Mr. S. K. Chaukiyal are acknowledged. The author is indebted to Dr. A. K. Gupta and Director, IRDE for permission to present this work.

72 Recognition of Pose Varied Three-Dimensional Human Faces Using Structured Lighting Induced Phase Coding

References

1. S. Z. Li and A. K. Jain, eds., Handbook of Face Recognition (Springer, 2005).

2. Reports on Face Recognition Grand Challenge initiatives, http://www.frvt.org/FRGC.

3. W. Zhao, R. Chellappa, A. Rosenfield and J. Phillips, ACM Comput. Surv. 12, 399 (2003).

4. S. K. Zhou and R. Chellappa, J. Opt. Soc. Am. A 22, 217 (2005).

5. K. W. Bowyer, K. Chang and P. Flynn, Comput. Vis. Imag. Underst. 101, 1 (2006).

6. J. Li, Y. Wang, T. Tan and A. K. Jain, in Biometric Technology for Human Identification, A. K. Jain, N. K. Ratha, Eds., Proc.SPIE 5404, 296 (2004).

7. D. Choudhury and M. Takeda, Opt. Lett. 27, 1466 (2002).

8. M. Takeda and K. Mutoh, Appl. Opt. 22, 3977 (1983).

9. B. V. K. Vijaya Kumar, A. Mahalanobis and R. D. Juday, Correlation Pattern Recognition (Cambridge, New York, 2006).

10. D. Choudhury and M. Takeda, in Optical Information Systems, B. Javidi and D. Psaltis., eds., Proc.SPIE 5202, 168 (2003).

11. M. Born and E. Wolf, Principles of Optics (Perga-mon, 1989).

12. K. Venkataramani, S. Qidwai and B. V. K. Vijayaku-mar, IEEE Trans Systems Man Cybernetics: Part C 35, 411 (2005).

13. Y. Sheng and D. Roberge, Appl. Opt. 38, 5541 (1999).

14. S. Roy, H. H. Arsenault and D. Lefebvre, Opt. Eng. 42, 813 (2003).

15. R. Legarda-Saenz, T. Bothe and W. P. Juptner, Opt. Eng. 43, 464 (2004).

Writer Recognition by Analyzing Word Level Features of Handwritten Documents

Prakash Tripathi and Bhabatosh Chanda

Ecetronics and Communication Sciences Unit Indian Statistical Institute

Kolkata 700108, India E-mail: [email protected]

Bidyut Baran Chaudhuri

Computer vision and Pattern Recognition Unit Indian Statistical Institute

Kolkata 700108, India

Writer recognition based on handwriting is important from security as well as judiciary point of view. In this paper, based on static off-line data, we have proposed a novel writer recognition methodology using word level micro-features and simple pattern recognition technique. The result is reasonably good and encouraging, and shows usefulness of computational feaures proposed here.

Keywords: writer recognition, handwritten document, word bounding box, word level feature, K-nearest neighbour.

1. Introduction

Analysis of handwritten documents is taken up by the research community to deal with two kinds of problems: (i) What is written and (ii) who has written the concerned document. In this work we deal with the second problem. This has immense application potential in security system and judicial framework. For example, sample of handwriting may be considered as biometric of the writer.2 So this may be used as a means of authentication. Analysis of handwriting from the viewpoint of determining the writer has great bearing on the criminal justice system. Writer individuality rests on the hypothesis that each individual has components of handwriting that is distinct from those of another individual. Handwriting has long been considered individual property, as evidenced by the importance of signatures in documents. However, this hypothesis has not been subjected to rigorous scrutiny with the accompanying experiment, testing, and peer review.

Each writer can be characterized by his own handwriting, by the reproduction of details and unconscious practices. This is why in certain cases the handwriting samples may be treated as a biometric, like fingerprints. The problem of writer identification arises frequently in the court of justice where the judge has to come to a conclusion about the authenticity of a document (e.g. a will, bill or receipt). Need also arises in banks for signature verification, or in

some institutes that analyze manuscripts of ancient authors and are interested in the genesis of the texts. In order to come to a conclusion about the identity of an unknown writer, two tasks may be considered:

(1) The writer identification task concerns the retrieval of handwritten samples from a database using the sample under study as a graphical query. It provides a subset of relevant candidate documents, which the expert (s) will concentrate on.

(2) The writer verification task must come to a conclusion about two samples of handwriting and determines whether they are written by the same writer or not.

When dealing with large databases, the writer identification task can also be viewed as a filtering step prior to the verification task.

Analysis of handwriting by computer has a long history3 and one of its first applications was in signature verification.4 However, that was a very constrained search and templates of handwritten word(s), which is usually distinctive from person to person, are used for verification. Identifying and/or verifying writer based on handwriting style (not limited to a constrained set of words) is relatively new area with pioneering work done by Srihari.5 The basic tasks are (i) extraction of style or forensic dependent features6 from documents, and (ii) then the features are analyzed with the help of some pattern clas-

74 Writer Recognition by Analyzing Word Level Features of Handwritten Documents

sification / recognition algorithms.7'8 For example, clustering and HMM based methodologies are proposed by Bensefia et a/.9'10 These are mainly off-line methods based on static data. On-line methods are primarily based on dynamic features like pen pressure, direction of strokes, etc. l,lx Some approaches adopt both static and dynamic features.12

The objective of this work is to develop a novel writer recognition system for handwritten documents using the static features relevant to writing style. Writing style consists of character shape (e.g., loop formation, curvature, corners, etc. ), word formation (e.g, inter-character gap, aspect ratio, interword gap, etc.) and line formats (e.g., skew, straight-ness, inter-line spacing, etc. ). Particularly in this paper we have explored the capabilities of word level features and employed a simple pattern recognition methodology in writer recognition. The experiment is carried on the documents written in an Indian language named Bangla.

The remaining of this paper is organized as follows. Section 2 describes the data acquisition strategy. Proposed methodology is described in section 3. Experimental results and discussion are given in section 4 followed by concluding remarks in section 5.

2. Data Acquisition

For such experiment the data acquisition is very critical to draw any meaningful conclusion. Thus, our objective was to obtain a set of samples that would capture variations in handwriting among and within writers. It means that we need handwriting samples from multiple writers, as well as multiple samples from each writer. The handwriting samples of the sample population should have the following properties.

(1) They should be sufficient in number to exhibit normal writing habits and to portray the consistency with which particular habits are executed.

(2) For comparison purposes, they should have similarity in texts, in writing circumstances, and in

writing purposes. (3) The text content should be meaningful and it

should have continuity.

Several factors may influence handwriting style, e.g., gender, age, ethnicity, handedness, the learing system of handwriting, the subject matter (content), writing protocol (written from memory, dictated, or copied out), writing instrument (pen and paper), changes in the handwriting of an individual over time, etc.5 We tried to ensure that document content captures as many features as possible. Only some of these factors were considered in the experimental design. The other factors will have to be part of a different study. However, the same experimental methodology can be used to determine the influence factors not considered.

2.1. Source Documents

One of the two almost similar source documents in Bangla script, which were to be copied by each writer is shown in Fig.l. They are concise, containing 158 words and 120 words respectively and are composed of most frequently occurring words in Bangla language. A table of some such words and their frequencies obtained from a moderately large Bangla Corpus is shown in Fig.2. In addition, the source documents also contain punctuation, distinctive letter and a general document structure that allowed extracting macro-document attributes such as word and line spacing, line skew, etc. Each participant (writer) was required to copy one of the source documents four times in his/her most natural handwriting, using plain, unlined sheets, and a ballpoint pen. The repetition was to determine, for each writer, the variation of handwriting from one occasion to the next. We collected documents from 30 writers resulting in a total of 120 documents. Each handwritten document was scanned and converted into a two-tone image using a resolution of 300 dpi. An example of such image is shown in Fig.3.

Prakash Tripathi, Bhabatosh Chanda and Bidyut Baran Chaudhuri 75

?RJ> Ft tfPft ^ ^Tl 33 *5RJ £J>R *R$ f SRSI f%f F SR t%|. SOT ^ 3 Cot *RR 3

• r*ff w iroi «fprai?r «rq*u ^ ^ is$8 *

T "Q ?wsr i$*ro?f «npr *>nft # R ^ ^ p jt^fo ^ i faft ^rftft $M *KrRi w H R J ^rtw $n ¥R ww, ^M ^ TO ifirai «* ICTI CT *nw «iftiw<i «$ *t0i ^ *

$, ^S*R RST ^rtpr 43 t ^ i ^ft^ ^RJ ^ !? WHI ^ «*R*. *W vO>R 1*R 13 *fo «r sclera WRJ w*r ftg ifi f ^5^5 ^

Fig. 1. A portion of the original document used to collect handwriting.

- - \ sT7

2 S > .

1\>- !"'

^W\

*.?7'^ ^1 ^ / x

; svju -

s - > ^

1 tSOl

*r

',• -,; ,

*

j

'V

>

"w

M^2 i > V

• z - s ^

1'. s

' i d ' . •<

• ]f„)H

1 0 ( ^ 1 " ,

l ' , ^ ' / -

,-,-

»- ;

5

-

-i r f

^)' (^

. ~ ~. 1

• r .

< - ? 2 >

Oi ~*

S1 , 1

^ 7 1

82v

Fig. 2. Some most frequently used Bangla words and their frequencies obtained from a corpus.

3. Proposed methodology

A digital binary image J is denned over a discrete domain Z2 obtained by sampling the continuous domain along two orthogonal directions such that the value of I{i,j) is either 0 or 1. In this section we describe preprocessing, feature extraction and pattern recognition techniques adopted for the development of the intended system for writer recognition.

3.1. Preprocessing

Preprocessing comprises of following basic steps.

(1) Noise removal: Stray black pixels (positive noise) of / are removed by binary morphological opening with 3 x 3 structuring element (SE). Let the result be I.

(2) Estimation of line count: Connected components of I are labeled. Statistics of run-length of 0-pixels within and between the words are obtained. A rectangular SE is designed based on these statistics and the image is closed with this SE to form word-blobs (see Fig.4). Let the result be Ic- Then Ic is vertically scanned to estimate the line number and line spacingi.13

Fig. 4. Illustartes word blobs obtained by closing.

(3) Extraction of word bounding box: Based on the line count and line spacing as well as the vertical scan of each word-blob, we determine the word bounding box. An example of word bounding boxes obtaind by the system is shown

-*^&> SYTJM'S' "3^?srr, «?v&<crj £=sr?i ^ i P '

2*-(X3r -xz-f^s,, -35f?sf-^cJ rasf5?-ajTR

Fig. 3. An example of handwritten document.

76 Writer Recognition by Analyzing Word Level Features of Handwritten Documents

in Fig.5. The result of this step is not 100% accurate; however, it is not very critical for the final results.13

J^<P QfS^T:^\ Fig. 5. Illustartes word bounding boxes obtained by the system. Note the errors occured though noise is cleaned in step 1.

(4) Thinning: The / is thinned for the extraction of structural features. Let the thinned image be

It-14

(5) Medial axis transform and skeletonization: The / also undergoes medial axis transform from which local maxima are extracted. Let the results be Im and Is, respectively.

3.2. Feature extraction

We distinguish between two types of features: conventional features and computational features. Conventional features are the handwriting attributes that are commonly used by the forensic document examination community. These features are obtained from the handwriting by naked eye under microscope. Computational features are those having known software/ hardware techniques of extraction. Computational features can be divided into macro-and micro-features, depending on whether they pertain globally to the entire handwritten sample, e.g., uniformity in line spacing and straightness of lines, or are extracted locally, e.g., inter-character gaps and loop formation. Macro-features can be extracted at the document level (entire handwritten manuscript) or at least a paragraph; while micro-features may be extracted at line, word, and character levels. As mentioned earlier, in this work we confine ourselves to word level features, which we consider of two types:

(1) Box level features (e.g., spread of stroke, depth of stroke, aspect ratio and density) that are computed over the entire word bounding box, and

(2) Cell level features (e.g., isolated points, corner, crossing, branch, end point, pixel density) that

are computed over non-overlapping division of word bounding box.

3.3. Box level features

Let the fc-th bounding box of a word be represented by Wk with two diagonal points (ri ,ci) and (r2,C2) in the image. Then the box level features are computed from the preprocessed images are as follows.

(1) Thickness of stroke Ts\ This roughly refers to the variable thickness of stroke caused due to speed of writing and is computed as

'E(r,c)EWkI(r'C)

(2) Width of stroke Ws: This this may be another way of computing the thickness of stroke due to speed of writing and is computed as

z2(r,c)ewk -f(r>c) Ds = l -

s(r,c)eWk Im(r,c)

(3) Spread of stroke Ss: This roughly refers again to the thickness of stroke due to speed of writing and is computed as

2^(r,c)€Wk Is\T,c) S. = l -

i(r,c)eWk Im(r,C)

(4) Aspect ratio A: This gives an idea of elongation of word due to speed and style of writing and is computed as

A = (r2 -n)/{c2 - c i )

mt Fig. 6. Division of bounding box into cells.

3.4. Cell level features

Each word bounding box is divided into m x n cells. An example of such division for 3 x 4 cells is shown in Fig.6. On each of the cells six features are computed.

(1) Number of isolated points: Isolated points are those pixels which have no neighbours. This is computed from the thinned image It-

Prakash Tripathi, Bhabatosh Chanda and Bidyut Baran Chaudhuri 77

(2) N u m b e r of e n d points : End points are those

pixels which have only one neighbour. This is

computed from It-

(3) N u m b e r of cross po ints : Cross points are

those pixels where two strokes cross or have V

junction. This is computed based on the connec

tivity number 1 4 using It.

(4) N u m b e r of branch points : Branch points are

those pixels which have ' T ' or 'Y' junction. This

is computed based on the connectivity number

using It.

(5) P i x e l dens i ty: This is the rat io of number of

black pixels and the cell size. This is computed

using the thinned image I.

(6) N u m b e r of corners: This is related to the

smoothness of the strokes. This is computed

using the thinned image / based on the facet

model.1 5

Thus, a total of N = 6mn + 4 features per word is

computed, which will be used for writer recognition.

It may be noted tha t since the features reflect dif

ferent kinds of geometrical and structural at tr ibutes,

we do not employ any transformation (e.g., PCA) for

feature reduction.

3 .5 . Recognition

Training set is a set of document images for which

writers ' identity are known. It contains all the fea

tures computed for all the words of all the writers.

Writer information is at tached with the feature vec

tor of every word in training set. Each word of train

ing documents is represented by a point in the N-

dimensional feature space. For testing a table con

sisting of all the writers along with individual counter

is created. The counters are initialized to zero. For

a test document, every word is used to recognize

the writer using K-nearest neighbour algorithm. The

counter for corresponding K writers is incremented.

After examining all the words of the test document,

the writer having maximum count is taken as the

writer of the document. The K-nearest neigbour ap

proach is simple and gives consistently good results.

4. E x p e r i m e n t a l R e s u l t s a n d D i s c u s s i o n

The da ta set consists of 4 copies of 30 different writ

ers. For each writer, out of four documents, three

were considered as training da ta set and one as test

data. Words were extracted automatically from ev

ery training document image. For each word 76 word

level features (4 box level and 72 cell level features

considering m = 3 and n = 4) were extracted. So,

N = 76. Euclidean distance is employed to measure

the similarity and K is taken to be 3. Using leave one

out strategy, all of the 120 documents are used as

test set, and out of these 120 documents writers are

correctly recognized in 114 cases making the recog

nition rate of 95%. It may be noted tha t if the value

of K is chosen 1 or 2, the recognition ra te goes down

to 67.2% and 70%, respectively. This implies tha t to

make the system invariant to the intra-writer varia

tion high value of the K should be taken. Here K = 3

is a reasonable choice, as we have three handwri t ten

documents of each writer in the training set. Recog

nition score for the words for a randomly selected

writers is shown in Table 1.

5. Conc lus ion

Here we have presented a novel algorithm for writer

recognition based on handwri t ten documents. In this

work only micro-features at word level are used. Sim

ple K-nearest neighbour algorithm with Euclidean

distance measure has been used and the result ob

tained is very encouraging.

Table 1. Recognition score for the words for some writers (ts: Test, tr: Training).

ts

tr

1 2 3 4 5 6 7 8

1

539 100 65 48 62 29 39 71

2

24 502 12 113 115 86 65 127

3

97 46 505 53 40 42 51 92

4

14 77 20 687 104 85 128 104

5

27 107 19 126 958 54 143 54

6

12 60 3 85 44 345 62 66

7

18 70 26 134 181 76 591 83

8

30 105 43 73 56 94 82 575

References

1. K. Yu, Y. Wang and T. Tan, Writer identification using dynamic features, in Proc. Intl. Con}, on Bioin-formatics and its Applications (ICBA '04), (Florida, USA, 2004).

2. A. K. Jain, B. Ruud and P. S. (eds), Biometrics: Personal identification in networked society (Kluwer Academic, Boston, 1999).

3. E. C. Greanias, P. F. Meagher, R. J. Norman and P. Essinger, IBM J. Res. Develop. 7, 14 (1963).

78 Writer Recognition by Analyzing Word Level Features of Handwritten Documents

4. K. K. Ho, H. Schroder and G. Leedham, Codebooks for signature verification and handwriting recognition, in Proc. of the Australian and New Zealand Intelligent Information Systems Conference, (Australia, 2006).

5. S. N. Srihari, S.-H. Cha, H. Arora and S. Lee, J. Forensic Science 47 (2002).

6. M. Tapiador and J. A. Sigenza, Writer identification method based on forensic knowledge, in Proc. Intl. Conf. on Bioinformatics and its Applications (ICBA'04), (Florida, USA, 2004).

7. K. Steinke, Pattern Recognition 14, 357 (1981). 8. E. N. Zois and V. Anastassapoulous, Pattern Recog

nition 33, 101 (2000). 9. A. Bensefia, T. Paquet and L. Heutte, Pattern Recog

nition Letters 26, 2080 (2005).

10. A. Bensefia, T. Paquet and L. Heutte, Electronic Letters on Computer Vision and Image Analysis 5, 72 (2005).

11. K. P. Zimmerman and M. J. Varady, Pattern Recognition 18, 63 (1985).

12. W. Jin, Y. Wang and T. Tan, Text-independent writer identification based on fusion of dynamic and static features, in Proc. Intl. Workshop on Biometric Recognition Systems, (Beijing, China, 2005).

13. A. K. Das, A. Gupta and B. Chanda, Image Processing and Communication 3, 85 (1997).

14. B. Chanda and D. D. Majumdar, Digital Image Processing and Analysis (Prentice-Hall of India, New Delhi, 2000).

15. R. M. Haralick and L. G. Shapiro, Computer and robot Vision (Addison-Wesley, Reading, 1992).

PARTD

Clustering Algorithms

81

A New Symmetry Based Genetic Clustering Technique for Automatic Evolution of Clusters

Sriparna Saha and Sanghamitra Bandyopadhyay

Machine Intelligence Unit Indian Statistical Institute

Kolkata, India Email: {sripama-r, sanghami}@isical.ac.in

In this article, a new symmetry based genetic clustering algorithm is proposed which automatically evolves the number of clusters as well as the proper clustering of any data set. Strings comprise both real numbers and the don't care symbol in order to encode a variable number of clusters. A newly proposed index, Sj/m-index, is used as a measure of the validity of the clusters that is optimized by the genetic algorithm. Sym-'mdex uses a new point symmetry based distance. The algorithm is therefore able to detect both convex and non-convex clusters that possess the symmetry property. Kd-tree based nearest neighbor search is used to reduce the complexity of finding the closest symmetric point. Effectiveness of the proposed method is demonstrated for several artificial data sets and one real-life data set.

Keywords: Clustering, Genetic Algorithms, Point Symmetry Based Distance, Real encoding

1. Introduction

Clustering1 is a fundamental problem in data mining with innumerable applications spanning many fields. In order to mathematically identify clusters in a data set, it is usually necessary to first define a measure of similarity or proximity which will establish a rule for assigning patterns to the domain of a particular cluster centroid. One of the basic feature of shapes and objects is symmetry. Su and Chou have proposed a point symmetry (PS) distance based similarity measure.2 This work is extended in3 in order to overcome some of the limitations existing in.2 It has further been shown in4 that the PS distance proposed in3 also has some serious drawbacks and a new PS distance (dps) is defined in4 in order to remove these drawbacks. For reducing complexity of point symmetry distance computation, Kd-tree based data structure is used.

if-means is a widely used clustering algorithm that was used in2 in conjunction with the earlier PS based distance. However, if-means has three major limitations: it requires the a priori specification of the number of clusters (K), it often gets stuck at sub-optimal solutions based on the initial configuration and it can detect only hyper-spherical shaped clusters. The real challenge in this situation is to be able to automatically evolve a proper value of K as well as providing the appropriate clustering of a data set. There exists a Genetic Algorithm (GA) based clustering technique, GCUK-clustering,5 which is able to automatically evolve the appropriate clustering for hyperspherical data sets. However for clusters with

other than hyperspherical shapes, this algorithm is likely to fail, as it uses, like the K-means, the Euclidean distances of the points from the respective cluster centroids for computing the fitness value. In this article a variable string length GA based clustering technique is proposed. Here assigment of points to different clusters is done based on point symmetry based distance rather than Euclidean distance. This enables the proposed algorithm to automatically evolve the appropriate clustering of all types of clusters, both convex and non convex which have some symmetrical structures. The chromosome encodes the centres of a number of clusters, whose value may vary. A new cluster validity index named Sym-'mdex is utilized for computing the fitness of the chromosomes. The effectiveness of the proposed genetic clustering technique for evolving the appropriate partitioning of a dataset is demonstrated on four artificial and one real-life data sets having different characteristics.

2. Proposed Algorithm (VGAPS)

In this section the new clustering algorithm is described in detail. It includes determination of number of clusters as well as appropriate clustering of the data set. This genetic clustering technique is subsequently referred to as variable length genetic clustering technique with point symmetry based distance (VGAPS).

In GAs, the parameters of the search space are encoded in the form of strings (called chromosomes). A collection of such strings is called population. Ini-

82 A New Symmetry Based Genetic Clustering Technique for Automatic Evolution of Clusters

tially a random population is created, which represents different points in the search space. An objective/fitness function is associated with each string that represents the degree of goodness of the solution encoded in the string. Based on the principle of survival of the fittest, a few of the strings are selected and each is assigned a number of copies that go into the mating pool. Biologically inspired operators like crossover and mutation are applied on these strings to yield a new population. The process of selection, crossover, and mutation continues for a fixed number of generations or till a termination condition is satisfied.

2.1. String representation

The chromosomes in VGAPS are made up of real numbers (representing the coordinates of the centres) as well as don't care symbol ' # ' . Note that real encoding of the chromosome is adopted since it is more natural to the problem under consideration. The value of K is assumed to lie in the range [Kmim Kmax), where Kmin is chosen equal to 2 unless specified otherwise. The length of a string is taken to be Kmax where each individual gene position represents either an actual center or a don't care symbol.

2.2. Population initialization

For each string i in the population (i = 1 , . . . P where P is the size of the population), a random number Ki in the range [Kmin,Kmax\ is generated. This string is assumed to encode the centres of Ki clusters. For initializing these centres, Ki points are chosen randomly from the data set. These points are distributed randomly in the chromosome. Let us consider the following example.

Example: Let Kmin = 2 and Kmax = 10. Let the random number Ki be equal to 4 for chromosome i. Then this chromosome will encode the centers of 4 clusters. Let the 4 cluster centers (4 randomly chosen points from the data set) be (10.0, 5.0) (20.4, 13.2) (15.8, 2.9) (22.7, 17.7). On random distribution of these centers in the chromosome, it may look like # (20.4, 13.2) # # (15.8, 2.9) # (10.0, 5.0) (22.7, 17.7) # # .

2.3. Fitness computation

This is composed of two steps. Firstly points are assigned to different clusters by the newly developed point symmetry based distance dps. Next, the cluster validity index, Sym-mdex, is computed and used as a measure of the fitness of the chromosome.

2.3.1. Newly developed point symmetry based distance, dps

The proposed point symmetry based distance dps(x,c) associated with point x with respect to a center c is defined as follows: Let a point be x. The symmetrical (reflected) point of x with respect to a particular centre c is 2 x c — x. Let us denote this by x*. Let the first and second unique nearest neighbors of x* be at Euclidean distances of d\ and a\ respectively. Then

dps(x,c) = -^~Y^- x de(x,c) (1)

where de (x, c) is the Euclidean distance between the point x and c.

Some important observations about the proposed point symmetry distance dps(x,c) are as follows:

(1) Instead of computing Euclidean distance between the original reflected point x* = 2 xc — x and its first nearest neighbor as in2 and,3 here the average distance between x* and its first and the second unique nearest neighbors have been taken. Consequently the term, (di + d2)/2 will never be equal to 0, and the effect of de (x, c), the Euclidean distance, will always be considered.

(2) Considering both d\ and di in the computation of dpS makes the PS-distance more robust and noise resistant. From an intuitive point of view, if both di and d^ of x with respect to c is less, then the likelihood that x is symmetrical with respect to c increases. This is not the case when only the first nearest neighbor is considered which could mislead the method in noisy situations.

It is evident that the symmetrical distance computation is very time consuming. Computation of dps(x~i,c) is of complexity 0(n). In order to compute the nearest neighbor distance of the reflected point of a particular data point with respect to a cluster center efficiently, we have used Kd-tree based nearest neighbor search.

Sriparna Saha and Sanghamitra Bandyopadhyay 83

Kd-tree Based Nearest Neighbor Computation: A K-dimensional tree, or Kd-tree is a space-partitioning data structure for organizing points in a K-dimensional space. A Kd-tree uses only those splitting planes that are perpendicular to one of coordinate axes. ANN(Approximate Nearest Neighbor) is a library written in C++, 6 which supports data structures and algorithms for both exact and approximate nearest neighbor searching in arbitrarily high dimensions. In this article, ANN is used to find d± and di in Equation 1 efficiently. The ANN library implements a number of different data structures, based on Kd-trees and box-decomposition trees, and employs a couple of different search strategies. The Kd-tree data structure has been used in this article. Here ANN is used to find d\ and d<i in Equation 1 efficiently. The Kd-tree structure can be constructed in 0(nlogn) time and takes 0(n) space, while the search complexity is O(logn).

2.3.2. Assignment of points

Here for each point Xi, 1 < i < n is assigned to cluster k iff dps(x~i,Ck) < dps(xi,c~j), j = 1,...,K, j ^ k and (dpa(xi,ck)/de(xi,ck)) < 8. For (dpa(xi,Ck)/de(xi,Ck)) > 8, point x"i is assigned to some cluster m iff de(xi,cm) < de(x~i,Cj), j = 1,2. ..K, j 7 m. In other words, point x is assigned to that cluster with respect to whose centers its PS-distance is the minimum, provided this value is less than some threshold 8. Otherwise assignment is done based on the minimum Euclidean distance criterion as normally used in5 or the iif-means algorithm. The value of 8 is kept equal to the maximum nearest neighbor distance among all the points in the data set. It is to be noted that if a point is indeed symmetric with respect to some cluster centre then the symmetrical distance computed in the above way will be small, and can be bounded as follows. Let d™^ be the maximum nearest neighbor distance in the data set. That is

djvjv5 = maxi=\,...NdNN(xi), (2)

where d;vAr(Si) is the nearest neighbor distance of Xj. Assuming that x* lies within the data space, it may be noted that

dx < d~f- (3) Ideally, a point x is exactly symmetrical with respect to some c if di = 0. However considering the un

certainty of the location of a point as the sphere of radius d™^ around x, we have kept the threshold 8 equals to d'fi'ff. Thus the computation of 8 is automatic and does not require user intervention. After the assignments are done, the cluster centres encoded in the chromosome are replaced by the mean points of the respective clusters.

2.3.3. Fitness calculation

The fitness of a chromosome is computed using the newly developed Sym-'mdex. Let K cluster centres be denoted by Q where 1 < i < K and number of points in each cluster is equal to n, where i = 1,...K. Then Sym index is defined as follows:

Sym(K) = ( i x i - x DK), (4)

where K is the number of clusters. Here,

K

EK = Y,E* (5)

such that

Ei^^d^ix^Ci) (6)

and

DK = maxfi=l \\ci - Cj\\ (7)

DK is the maximum Euclidean distance between two cluster centers among all centres. d*g(xj,c7) is computed by Equation 1 with some constraint. Now first two nearest neighbors of xj — 2 x ct — Xj will be searched among the points which are already in cluster i i.e., now the first and second nearest neighbors of the reflected point Xj of the point x~j with respect to Ci and xj should belong to the ith cluster. The objective is to maximize this index in order to obtain the actual number of clusters and to achieve proper clustering. The fitness function for chromosome j is defined as 1/Symj, where Symj is the Sym index computed for the chromosome. Note that minimization of the fitness value ensures maximization of the Sym index.

Explanation: As formulated in Equation 4, Sym is a composition of three factors, these are 1/K, 1/EK and DK- The first factor increases as K decreases; as Sym needs to be maximized for optimal clustering, so it will prefer to decrease the value of

84 A New Symmetry Based Genetic Clustering Technique for Automatic Evolution of Clusters

K. The second factor is the within cluster total symmetrical distance. For clusters which have good symmetrical structure, EK value is less. This, in turn, indicates that formation of more number of clusters, which are symmetrical in shape, would be encouraged. Finally the third factor, DK, measuring the maximum separation between a pair of clusters, increases with the value of K. Note that value of DK is bounded up by the maximum separation between a pair of points in the data set. As these three factors are complementary in nature, so they are expected to compete and balance each other critically for determining the proper partitioning.

2.4. Genetic Operations

The following genetic operations are performed on the population of strings for a number of generations. Selection: Conventional proportional selection is applied on the population of strings. Here, a string receives a number of copies that is proportional to its fitness in the population. Crossover. During crossover each cluster centre is considered to be an indivisible gene. Single point crossover, applied stochastically with probability yuc, is explained below with an example. Example: Suppose crossover occurs between the following two strings: # (20.4, 13.2) # # (15.8, 2.9)| # (10.0, 5.0) (22.7, 17.7) # # and (13.2, 15.6) # # # (5.3, 13.7)| # (10.5, 16.2) (7.9, 15.3) # (18.3, 14.5) Let the crossover position be 5 as shown above. Then the offspring are # (20.4, 13.2) # # (15.8, 2.9)| # (10.5, 16.2) (7.9, 15.3) # (18.3, 14.5) and (13.2, 15.6) # # # (5.3, 13.7)| # (10.0, 5.0) (22.7, 17.7) # # Mutation: Mutation is applied on each chromosome with probability fim. Mutation is of three types. (1) Each valid position (i.e., which is not ' # ' ) in a chromosome is mutated with probability fxm in the following way. A number 6 in the range [0,1] is generated with uniform distribution. If the value at that position is v, then after mutation it becomes v x (1 ± 26), if v ^ 0, otherwise for v = 0 it will be equal to ±26. The '+ ' or '-' sign occurs with equal probability. (2) One randomly generated valid position is removed and replaced by ' # ' . (3) One

randomly chosen invalid position is replaced by randomly chosen point from the data set. Any one of the above mentioned types of mutation is applied randomly on a particular chromosome if it is selected for mutation.

2.5. Termination Criterion

In this article the processes of fitness computation, selection, crossover, and mutation are executed for a maximum number of generations. The best string having the lowest fitness (i.e., largest Sym index value) seen up to the last generation provides the solution to the clustering problem. We have implemented elitism at each generation by preserving the best string seen up to that generation in a location outside the population. Thus on termination, this location contains the centres of the final cluster.

3. Implementation Results

The experimental results showing the efectiveness of VGAPS algorithm are provided for five artificial and one real life data sets. The description of the data sets are given in Table 1. Data_6_2 and Data_4_3 are used in7 while the other data sets can be obtained on request to the authors. The cancer data set is obtained from [www.ics.uci.edu/~mlearn/MLRepository.html]. Each pattern of Cancer dataset has nine features. There are two categories in the data: malignant and benign. The two classes are known to be linearly separable. There are a total of 683 data points in the data set.

VGAPS is implemented with the following parameters (determined after some experimentations): \ic = 0.8, /xm = 0.02. The population size P is taken to be equal to 100. Kmin and Kmax are set equal to 2 and y/n respectively where n is the total number of data points in the particular data set. VGAPS is executed for a total of 30 generations. Note that it is shown in Ref.8 that if exhaustive enumeration is used to solve a clustering problem with n points and K clusters, then one requires to evaluate l/K ^2i-i(—l)K~:'jn partitions. For a data set of size 50 with 2 clusters this value is 249 — 1 (i.e., of the order of 1015). If the number of clusters is not specified a priori, then the search space will become even larger and utility of GAs is all the more evident. For

Sripama Saha and Sanghamitra Bandyopadhyay 85

all the data sets, as is evident from Table 1, VGAPS is able to find out appropriate number of clusters and the proper partitioning. Figures 1, 2, 3, 4 and 5 show the final clustered results obtained after application of VGAPS on Sym_5_2, Sym.3_2, Ring_3_2, Data.6.2 and Data_4_3. For cancer data set it is not possible to show the clustered result visually. The obtained cluster centres are (3.013453, 1.266816, 1.378924, 1.304933, 2.056054, 1.293722, 2.080717, 1.215247, 1.105381), (7.130802, 6.696203, 6.670886, 5.700422, 5.451477, 7.780591, 6.012658, 5.983122, 2.540084) respectively. And the actual cluster centres are (2.9640, 1.3063,1.4144,1.3468,2.1081,1.3468, 2.0833,1.2613, 1.0653) and (7.1883, 6.5774, 6.5607, 5.5858, 5.3264, 7.6276, 5.9749, 5.8577, 2.6025) respectively.

Table 1 also shows the performance of GCUK-clustering5 optimizing Davies-Bouldin index (DB-index) for all the data sets. As is evident, GCUK-clustering is able to detect the proper number of clusters as well as proper clustering for DataJL2 and Data_4_3 but it fails for Sym_5_2, Sym_3.2 and Ring_3_2. Figure 6, 7 and 8 show the clustering result obtained by GCUK-clustering on Sym_5_2, Sym.3_2 and Ring.3.2 respectively. GCUK-clustering obtained, incorrectly, 6, 2 and 5 clusters for these three data sets respectively. Clustering results on DataJL2 and Data_4_3 obtained by GCUK-clustering are same as that of VGAPS and are therefore omitted.

To compare the performance of VGAPS with that of GCUK-clustering,5 for the real-life data set, Minkowski Score (MS)9 is calculated after application of both the algorithms. MS is a measure of the quality of a solution given the true clustering. For MS, the optimum score is 0, with lower scores being "better". For Cancer dataset, MS score is 0.3233 for VGAPS and 0.37 for GCUK-clustering. From the above results it is evident that VGAPS is not only able to find the proper cluster number, but it also provides significantly better clustering (both visually as in Figure 1-5, and also with respect to the MS scores). Moreover, we have also conducted statistical test ANOVA, and found that the difference in the mean MS values over ten runs obtained by VGAPS and GCUK-clustering are statistically significant. For Cancer, mean difference in mean MS obtained by two algorithms over ten runs is -4.67 E-02 which is statistically significant (significance value

is 0.00).

4. Conclusion

In this paper a genetic algorithm based clustering technique, VGAPS clustering, is proposed which assigns the data points to different clusters based on the point symmetry based distance and can automatically evolve the appropriate clustering of a data set. A newly developed symmetry based cluster validity index named Sym index is utilized for computing the fitness of the chromosomes. The proposed algorithm has the capability of detecting both convex and non-convex clusters that possess the symmetry property.

The effectiveness of the clustering technique is demonstrated for several artificial and real life data sets of varying complexities. The experimental results show that VGAPS is able to detect proper number of clusters as well as proper clustering from a data set having any type of clusters, irrespective of their geometrical shape and overlapping nature, as long as they possess the characteristic of symmetry.

Table 1. Results obtained with the different data sets using VGAPS and GCUK. Here, # pts, # dim, # AC, # OC denotes respectively number of points in the data set, number of dimension, actual number of clusters, obtained number of clusters

Name # p t s # d i m # A C #oc VGAPS GCUK

Sym.5.2 850 2 5 5 6 Sym-3J2 600 2 3 3 2 Ring.3.2 350 2 3 3 5 Data.6-2 300 2 6 6 6 Data.4.3 400 3 4 4 4 Cancer 683 9 2 2 2

Fig. 1. Clustered Sym_5_2 using VGAPS where 5 clusters are detected

86 A New Symmetry Based Genetic Clustering Technique for Automatic Evolution of Clusters

Fig. 2. Clustered Sym_3_2 using VGAPS where 3 clusters are detected

Fig. 6. Clustered Sym_5_2 using GCUK-clustering where 6 clusters are detected

. - • • • • ; : " 7

2 °$*$ °°

m& irtii?

Fig. 3. Clustered Ring-3.2 using VGAPS where 3 clusters Fig. 7. Clustered Sym.3-2 using GCUK-clustering where 2 are detected clusters are detected

41

#

^4 ^ S

&

* &

Fig. 4. Clustered Data-6-2 using VGAPS where 6 clusters are detected

Fig. 8. Clustered Ring-3_2 using GCUK-clustering where 5 clusters are detected

Fig. 5. Clustered Data-4_3 using VGAPS where 4 clusters are detected

R e f e r e n c e s

1. B. S. Everi t t , S. L a n d a u and M. Leese, Cluster Analysis (London: Arnold, 2001).

2. M.-C. Su and C.-H. Chou, IEEE Transactions Pattern Analysis and Machine Intelligence 2 3 , 674 (2001).

3. C. H. Chou, M. C. Su and E. Lai, S y m m e t r y as a new measure for cluster validity, in 2nd WSEAS Int. Conf. on Scientific Computation and Soft Computing' 2002, pp . 209-213.

4. S. Bandyopadhyay and S. Saha, Pattern Recog. (Revised.).

5. S. Bandyopadhyay and U. Maulik, Pattern Recognition 3 5 , 1197 (2002).

Sriparna Saha and Sanghamitra Bandyopadhyay 87

6. D. M. Mount and S. Arya, Ann: A library for approximate nearest neighbor searching (2005), http://www.cs.umd.edu/~mount/ANN.

7. U. Maulik and S. Bandyopadhyay, Pattern Recognition 33, 1455 (2000).

8. M. de Berg, M. V. Kreveld, M. Overmars and

O. Schwarzkopf, Cluster Analysis for Application (Academic Press, 1973).

9. A. Ben-Hur and I. Guyon, Detecting stable clusters using principal component analysis in methods in molecular biology (Humana press, 2003).

A Non-Hierarchical Clustering Scheme for Visualization of High Dimensional Data

G. Chakraborty and B. Chakraborty

Faculty of Software and Information Science Iwate Prefectural University

Japan 020-0193 E-mail: {goutam, basabi}&soft.iwate-pu.ac.jp

N. Ogata

Graduate School of Software and Information Science Iwate Prefectural University

Japan 020-0193

Clustering algorithms with data visualization capability is needed for discovering structure in multidimensional data. Self Organizing Maps(SOM) are widely used for visualization of multidimensional data. Though SOMs are simple to implement, they need heavy computation as the dimensionality increases. In this work a simple non hierarchical clustering scheme has been proposed for clustering and data visualization of high dimensional data in two dimension. Simple simulation experiments show that the algorithm is effective in clustering and visualization compared to SOM while takes much lesser time than SOM.

Keywords: Non hierarchical clustering, Data visualization, Self Organizing Map, Multi Dimemsional Scaling

1. Introduction

Clustering algorthims123 are needed for data mining tasks in the the process of understanding and discovering the natural structure and grouping in a data set. It has also a wide range of applications in other areas like data compression, information retrieval or pattern recognition/classification. Data visualization techniques provide further assistance in this process by visual representation of the data. In case of high dimensional data, understanding of structure is difficult as human cannot visualizes in dimensions more than three. Most clustering algorithms do not work efficiently for high dimensional data due to existence of noisy and irrelevant attributes. High dimensional data sets are often sparse in some dimensions and show clustering tendency in some subspaces.

Subspace clustering45 is an extension of traditional clustering to find clusters in proper subspaces for high dimensional data. Multidimensional Scaling (MDS) is another approach6 to cluster high dimensional data into a lower dimensional space in data visualization for exploring similarities in data. Much of the work in cluster visualization has been done for hierarchical clustering. For non-hierarchical clustering, Self organizing Maps (SOM) are popular. SOMs invented by Kohonen7 reduce the data dimensions by producing a map in 1 or 2 dimension and display similarities in the data through the use of self orga

nizing neural networks. SOMs are simple to construct and easy to understand but the major problem with SOMs is that they are computationally expensive. As the data dimension increases, dimension reduction visualization techniques become more important, but unfortunately the computation time also increases for SOM.

In this work a novel non hierarchical clustering scheme is proposed to lower compuational load with data visualization effect comparable to SOM. Our technique is simple to implement and can be implemented for any dimension. The proposed algorithm is represented in the next section followed by simulation experiments and results in the following section. Final section contains conclusion and direction for improvement.

2. Proposed Clustering Scheme

The main idea of the proposed scheme is as follows. Let us consider a point in two dimension corresponding to each high dimensional point in the data set. Then any two points in the two dimension should be moved closer or far apart ( Fig. 1) depending on the similarity or dissimilarity in the original high dimensional space. This process will be initiated for any two random points and iterated for a large number of times. Eventually the clusters formed in two dimension will produce a image of the clusters present

G. Chakraborty, B. Chakraborty and N. Ogata 89

in the data in the original high dimensional space (Fig. 2). The actual algorithm for computation is as follows.

i i : : : : : : : : : : : :

IE

/ " - \t~~ __

simirality

Fig. 1. Moving two dimensional points according to similarity in original space.

3i_: • _ J I _ T | " " It'

\?~ | ~ ~ ~

I E " I | i 3 1 " _ _ : i r ~ ~<l 1*" :

i ^ * ^ r ~7 , 3 1 V

j. X ^ J J j , , I -H ! x i r i ; ! ; J I _ ; ^ ~ » t ^ | r ^ ;7 _\ lzc _ :

j i . 3^ 0>1E L . . r E ' a J £-,»-• -J S I L ' 4 ^ * 'Z

\ , 3 C | i . ' : i * " " -JT - •"

c = >

Fig. 2. Start and Final configuration of the two dimensional space.

(1) Let us consider D to be the data set containing n m-dimensional data points (X\, X2, • ••, Xn) to be clustered, where Xt = xn, X&,..., xim.

(2) Normalization of the data: At the first step, the data points are to be normalized to (Yi, Y2, • • •, Yn) as following:

Vij = -mini'

maxij ~mi7iij

where y,j is the normalized value of j th component (dimension) of the ith data point. miriij = min(xij,X2j, • • • ,xnj) and maxij — max(xij,X2j, • • • ,xnj) represent the minimum and maximum values of the j th attribute (dimension) of the data respectively.

(3) Two dimensional data generation: Random two dimensional n data points are generated corre

sponding to the data points in the original m dimension(left portion in the Fig 2).

(4) Similarity Calculation: Any two data points Y)t and Yi in m dimension is selected and their similarity is calculated by Manhatton distance as:

Mkl = \Yk-Yl\ As the value of Manhatton distance depends

on number of dimensions, scaling has been done to limit the values to lie between 0 - 0.5 by the following equation

m m + Mki

Rki represents the degree of relatedness of the data points Yk and Yj

(5) Moving the data points in two dimension: Now according to the value of Rki, the data points in two dimension corresponding to Yfe and Y; are moved. The actual movement value is calculated as follows:

Movement Value = i _ ax fit

+ b where the parameters a and 6 can be calcu

lated from the following equations.

1 <*i

0.01

1

+ b

a 2 = a— + b U.o

(2)

(3)

a\ and a2 represent the movement of points in two dimension for high similarity of the original points and low similarity of original points respectively. These two parameters are to be set by users from the value 0.01 to 0.5. The movement of two dimensional points according to the degree of relatedness of the original points are shown in Fig. 1 and Fig. 3.

(6) The above procedure (step 4 and step 5) has to be repeated a large number of times until the data are clustered in two dimension as shown in (Fig 2)

3. Simulation Experiments and Results

Simple simulation experiments have been done for checking the efficiency of the the proposed clustering scheme compared to SOM. Two sets of data are used in the experiment.

90 A Non-Hierarchical Clustering Scheme for Visualization of High Dimensional Data

Fig. 3. Movement value according to degree of Relatedness.

3.1. Vowel data set

This data set8 contains three dimensional array of (three formant frquencies) vowel data of 10 vowels for different speakers uttered several times, a total of 300 data. The three dimensional view of the data set is shown in Fig 4. Now the proposed algorithm and SOM is simulated with this data. For SOM the learning epoch is 10,000 times. For the proposed algorithm the number of iteration has been done for 100,000 times. Though the number of iteration is larger in the proposed algorithm than in SOM, the total time taken is less for the proposed algorithm as time for one iteration is lesser than for one iteration for SOM. The total time taken for SOM is around 2700 sec while for the proposed algorithm it was nearly 100 sec.

Fig. 4. Vowel data set.

Fig. 5 represents the result of SOM by Vowel

data. Fig 6 represents the result of the proposed clustering scheme. It is seen that in the two dimension there are ten discrete clusters corresponding to ten classes of vowels. Though the results are comparable to the output of SOM which also clearly shows ten clusters, the time taken for clustering by the proposed scheme is far less than that of SOM.

Fig. 5. Result of Vowel data by SOM.

0.524

0.5235

0.523

0.5225

0.522

0.5215

*

-

-

-

''*&& ' **&§&. "

1 2 3 * 4 :--5 s 6 7 • 8 9

10 v

-

-

^ **#%..

0.487 0.4875 0.488 0.4885 0.489 0.4895 0.49 0.4905 0.491 0.4915

Fig. 6. Result of Vowel data for the Proposed Algorithm

3.2. Fisher Iris Data

The proposed algorithm has also been simulated with Fisher Iris data.9 This is a four dimensional data set containing three classes of Iris plant (Iris Setosa, Iris Versicolour, Iris Verginica) each with 50 sample points, a total of 150 data points. As the data set is four dimensional it is impossible to visualize in four

G. Chakraborty, B. Chakraborty and N. Ogata 91

dimension. Simulation experiments for clustering by SOM and the proposed scheme have been done for 10,000 times and 100,000 times respectively as before. Fig. 7 and Fig 8 represents the result of SOM and the result of the proposed scheme. In this case also the results show that the clustering result of the proposed algorithm is similar to the results obtained by SOM but the time taken for the proposed clustering scheme is lesser than time taken by SOM.

11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26

Fig. 7. Result of the Iris data by SOM.

-

.

'»%,

vs VG »

-

0.49952 0-499525 0.49953 0.499535 0.49954 0.499545 0.49955 0.499555

Fig. 8. Result of Iris data by Proposed Algorithm.

4. Conclusion

In this work a non hierarchical clustering scheme has been proposed for data visualization of multi dimensional data in two dimension. The clusters generated in two dimension retain the properties of the data

i,e. similarity/dissimilarity between the data points in the original dimension. The algorithm is simple and less time consuming than more widely used Self Organizing Map for non hierarchical clustering. The clustering results obtained by SOM and the proposed algorithm is almost similar.

In the present work we have done simulation with 3 and 4 dimensional data sets but the algorithm can be applicable to data sets with larger dimension. The computational load with increase of dimension in the proposed algorithm is not much compared to the increase of computational load with dimension in case of SOM as in this algorithm we used manhatton distance for similarity measurement. With the use of euclidean distance, the computational load increases but clusteting result does not differ much. So we preferred manhatton distance as the similarity measure. Here in this simple experiment, in both the cases, it is found that the resulting clusters clearly show the data structure like SOM but take much lesser time than SOM for execution.

Further experiment with data of higher dimension has to be done to justify the efficiency of the algorithm. Moreover we have only experimented with the data sets which have clear clusters, we need also to experiment with datasets with overlapping cluster to see the effect of the proposed algorithm. In our simulation we have decided at the start to iterate the process for 100,000 times. We need to set a proper stopping criterion depending on some cluster validity measures. We are currently working on this problem and hope to report in near future.

References

1. A. K. jain and R. C. Dubes, Algorithms for Clustering Data (Prentice-Hall, Englewood Cliffs, NJ, 1988).

2. L. P. wang and X. J. Fu, Data Mining with Computational Intelligence, (Springer, Berlin, 2005).

3. V. S. Tseng and C. P. Kao ' Efficiently Mining Gene Expression Data via a Novel Parameterless Clustering Method', IEEE/ACM Transactions on Computational Biology and Bioinformatics 2, 355 - 365, (2005).

4. L. Parsons, E. Haque and H. Liu, ACM SIGKDD Explorations Newletter 6(1),90, (2004).

5. X. J. Fu and L. P. Wang, ' Data dimensionality reduction with application to simplifying RBF network structure and improving classification performance', IEEE Trans. System, Man, Cybern, Part B, 33, 399-409, (2003).

92 A Non-Hierarchical Clustering Scheme for Visualization of High Dimensional Data

6. J. Kruskal and M. Wish, Multidimensional Scaling (Sage Publications, London, 1978).

7. T. Kohonen, Self- Organizing Maps (New York: Springer-Verlag, 1997).

8. UCI Machine Learning Repository (http://www.ics.uci.edu/ mlearn/databases/).

9. J. C. Bezdek, 'Pattern Recognition with Fuzzy Objective Functions (Plenum Press, NY, 1981).

10. J. Lee and D. Lee ' An improved cluster labeling method for support vector clustering', IEEE Transactions on Pattern Analysis and Machine Intelligence 27, 461 - 464, (2005).

93

An Attribute Partitioning Approach to Correlation Connected Clusters

Vijaya Kumar Kadappat'* and Atul Negi*

t Department of Computer Applications Vasavi College of Engineering, Hyderabad 500031, India.

Email: [email protected]. com, kadappakumar@gmail. com

* Department of Computer and Information Sciences University of Hyderabad, Hyderabad 500046, India. Email: [email protected], [email protected]

Correlation Connected Clusters are objects that are grouped based upon correlations in local subsets of data. The method 'Computing Clusters of Correlation Connected objects' (4C)1 uses DBSCAN and Principal Component Analysis (PCA) to find such objects. In this paper, we present a novel approach that attempts to find correlation connected clusters using an attribute partitioning approach to PCA. We prove that our novel approach is computationally superior to the 4C method.

Keywords: Clustering, DBSCAN, PCA, SubXPCA, correlation connected clusters, attribute partitioning

1. Introduction

Cluster analysis is widely used for data exploration in data mining. A recently proposed clustering technique, 'Computing Clusters of Correlation Connected objects' (4C technique) is based on DBSCAN2 and Principal Component Analysis (PCA). The 4C method was proved to be useful in various data sets such as molecular biology, time sequences, etc. 4C finds correlations which are local to subset of the data and are not global, since dependency between features can be different for different subgroups of data set. The 4C method was proved to be superior to DBSCAN, CLIQUE, etc.1 However, 4C is not suitable for high dimensional data due to inability of DBSCAN handling such data. PCA is a crucial aspect of 4C method. In 4C method, PCA is used to find correlations in the neighbourhood of a core object and PCA plays vital role in 4C in finding correlated clusters. A clustering approach based on subsets of attributes was previously presented by Friedman and Meulman.3 Following this line of thinking, SubXPCA4 was proposed to improve PCA. SubXPCA finds local subsets of attributes and combines them globally like PCA does. More importantly, SubXPCA4 was found to be computationally superior to PCA for high dimensional data. In this paper we attempt to improve 4C method based upon our insight into advantages of attribute partitioning approach to PCA (i.e. SubXPCA) for high dimensional data and we prove computational superiority of our approach over 4C.

The organization of the paper is as follows. In section 2, we review the salient aspects of 4C1 and a review of SubXPCA is presented in section 3. We propose 'an Attribute Partitioning approach to Correlation Connected Clusters' (AP-4C) and time efficiency of AP-4C over 4C is proved in section 4. We conclude with some comments in section 5.

2. Computing Clusters of Correlation Connected objects (4C)

We review briefly the 4C method here and a detailed discussion may be found in Bohm et oZ.'s work.1 Consider X = { X i . . . X N } , the set of N patterns, each of d dimensionality. 4C algorithm: We use the definitions of core object, correlation dimension, direct correlation-reachability, Ne * (Xi) etc., as defined by Bohm et al.x for our discussion and we give them here - correlation dimension: Let S C X, A < d, V = {Aj . . . A^}, the eigenvalues of S in descending order and 6 £ 5i((5 ss 0). S forms a A— dimensional linear correlation set w.r.t. 5 if atleast d — X eigenvalues of S are close to zero. Let S C X be a linear correlation set w.r.t. S € 5ft. The number of eigenvalues with Aj > 6 is called correlation dimension of S. Direct correlation reachability (DirReach^(P2,Pi))-Let e,S £ 5ft and /x, A € N. A point Pi e D is direct correlation-reachable from a point P2 € D w.r.t. e, n, 6 and A if P2 is correlation core object, the correlation dimension of e-neighbourhood

94 An Attribute Partitioning Approach to Correlation Connected Clusters

of Pi, Ne(Pi) is atmost A and Pj 6 7VeCp2(P2).

Cp2 is correlation similarity matrix, Nf''(P) = {Xi e D\max{distP(P, Xj), distXi(Xit P)} < e}, where distP(P,Xi) = y/(P - Xi)CP(P - Xi)T.

Input: X, pattern (object) set; e, neighbourhood; fi, number of points in e—neighbourhood; A, upper bound for correlation dimension; 6, threshold to select correlation dimension. Output: Every object is assigned a cluster-id or marked as noise. Assumption: Each pattern (object) in X is marked as unclassified initially. Method: For each unclassified pattern (object), Xi € X repeat the steps 1 — 3

l.Test whether Xi is a correlation core object as follows: l.l.Compute e—neighbourhood of Xi, iV^Xi). 1.2.If Number of elements in Ne(Xi) > fi, then

1.2.1. Compute covariance matrix of patterns in AT£(Xi), i.e. M X | .

1.2.2.If Correlation Dimension of A^Xi) < A 1.2.2.1.Compute correlation similarity

matrix, Mx, and Ne X ' (X;)

1.2.2.2.Test if number of patterns

inATeMx'(Xi) >n

2.If (Xi is a correlated core object) then Expand a new cluster as follows: 2.1.Generate new cluster-id.

2.2.Insert all Xj e N6 ' (Xi) into queue, Q. 2.3.While (Q is not empty) repeat the steps

2.3.1 - 2.3.4. 2.3.l.E = first object in Q. 2.3.2.Compute

R = {Xj e X\DirReaehi>*(E,Xj)}, where DirReachf,^(E,'X.j) indicates Xj is direct correlation reachable from correlation core object E.

2.3.3.For each Xj € R repeat the steps 2.3.3.1-2.3.3.2 2.3.3.l.If Xj is unclassified or noise then

Assign current cluster-id to Xj. 2.3.3.2.1f Xj is unclassified then insert

Xj into queue Q. 2.3.4.Remove E from queue Q.

3.1f (Xi is not a correlated core object) then Mark Xi as noise.

4C method was found to be very useful to find correlations in local subsets of data such as microbiology, e-commerce. 4C makes use of PCA to find correlations in the data set, which is not suitable for high dimensional data, hence 4C consumes a large chunk of time for such data sets. To counter this problem, we make use of SubXPCA, a computationally more efficient variation of PCA which is described in the following section.

3. Cross-sub-pattern Correlation based Principal Component Analysis (SubXPCA)

We review briefly SubXPCA here and a detailed discussion may be found in the work of Negi and Kadappa.4

SubXPCA Algorithm: In the following steps, we use the indices as follows. 1 < i < N; 1 < j < k; 1 < p < dt; 1 < s < kr

1.Apply classical PCA for each sub-pattern set as follows:5

1.1. Partition datum, Xi suitably into fc(> 2) equally-sized sub-patterns, each of size dt (= d/k) and SPj is the set of j t h

sub-patterns of Xi, Vi. 1.2. For every SPj, repeat the following steps

1.2.1 to 1.2.4. 1.2.1.Compute covariance matrix, (Cj)dtxdi-1.2.2.Compute eigenvalues (A£) and corr

esponding eigenvectors (e^). 1.2.3.Select r (< d{) eigenvectors correspo

nding to first r largest eigenvalues obtained in step 1.2.2. Let Ej be the set of r eigenvectors (column vectors) selected in this step.

1.2.4. Extract r local features (PCs) from SPj by projecting SPj onto Ej as follows. Let Yj be the reduced data in this step, and is given by

(Yj)Nxr = (SPj)Nxdi(Ej)dtxr 1.3. Collate Yj, Vj, as shown below. Let Z

denotes such combined data. Z; = (j/i(i, l),yi(i,2),...,yi(i,r)... and so on . . . yk(i, 1 ) , . . . , yk{i, r)) where Zi is the ith

row of Z, which corresponds to Xi. And (Vjih l),Vj(i,2),...,Vj(i,r)) is ith r owofYj .

2. Apply classical PCA on Z obtained in step 1.3. 2.1.Compute Final covariance matrix, (CF)(kr)x(kr

Vijaya Kumar Kadappa and Atul Negi 95

for the data Z. 2.2. Compute eigenvalues (Xf) and correspon

ding eigenvectors (ef). 2.3.Select rF (< kr) eigenvectors corresponding

to first rF largest eigenvalues obtained in step 2.2. Let E F be the set of rF eigenvectors selected in this step.

2.4.Extract rF global features (PCs) by projecting Z (obtained in step 1.3) onto E F . Let Z F

be the data obtained after projection and is given by ( Z F ) N x r F = ( Z ) N x k r ( E F ) k r x r F

2.5.ZF is the final reduced pattern data (corresponds to original data, X), which can be used for subsequent classification, recognition, etc.

SubXPCA was proved to be computationally more efficient than PCA and the successful classification results on 5 UCI repository of Machine Learning databases was also presented by Negi and Kadappa.4

4. An Attribute Partitioning approach to Correlation Connected Clusters (AP-4C)

In this section we present our clustering method, AP-4C, which is based on DBSCAN and SubXPCA.

4.1. AP-4C Algorithm:

To find correlation dimension of e-neighbourhood of core object (See step 1 of sec. 2), 4C method uses classical PCA which is computationally expensive. To ease this problem, AP-4C method uses SubXPCA to find eigenvalues and eigenvectors, to compute correlation dimension. The algorithm is as follows.

Input: X, pattern (object) set; e, neighbourhood; fi, number of points in e—neighbourhood; A, upper bound for correlation dimension; 5, threshold to select correlation dimension

Output: Every object is assigned a cluster-id or marked as noise

Assumption: Each pattern (object) in X is marked as unclassified initially

Method: For each unclassified pattern (object), X; 6 X repeat the steps 1 - 3 .

l.Test whether Xi is correlation core object as follows: 1.1.Compute e-neighbourhood of Xi, 7Vf(Xi). 1.2.If Number of elements in 7Ve(Xi) > u, then

1.2.1.Find Correlation Dimension of iv£(Xi)

using SubXPCA4 as follows: 1.2.1.1.Partition each pattern in 7V£(Xi)

of pattern Xi, into k sub-patterns, find sub-covariance matrices, project sub-patterns on them and find final covariance matrix (Cx,) of the same neighbourhood as given in step 1.2.1 and 2.1 of sec. 3.

1.2.1.2.Compute eigenvalues and eigenvectors of Cx, obtained in step 1.2.1.1.

1.2.1.3.Count the number of eigenvalues greater than S. If the count < A then 1.2.1.3.1.Compute correlation sim

ilarity matrix Cx., and JVfx'(Xi)

1.2.1.3.2.Test if number of patterns in -/Ve

Cx'(Xi) > / i Steps 2 and 3: are same as 4C method (See sec. 2), except that M is replaced with C. hence for brevity we do not reproduce it here.

To give a better conceptual conprehension of our method we summarize it in figure 1.

4.2. Time complexities of 4C and AP-4C

In PCA techniques, where covariance matrix is computed explicitly, a large amount of time is spent in computing covariance matrix, and an insignificant amount of time for other tasks such as finding eigenvalues. Hence, we focus on time complexity of covariance matrices of PCA and SubXPCA used in 4C and AP-4C methods respectively. Consider X = { X i , . . . , X N } , the set of TV patterns of size d. Time complexity to calculate covariance matrix (C) in PCA, Tc' It involves computation of d(d + l ) /2 parameters, and a total of Nd(d + l ) /2 operations and Tc is given as Tc = 0(Nd2). Time complexity to calculate k sub-covariance matrices (Cj) in SubXPCA, Tfx, is given as follows.

TFl = O(kNdf) (1)

where k is number of sub-patterns per pattern and O(Ndf) is the time complexity of calculating a sub-pattern covariance matrix, Cj. Time complexity to calculate final covariance matrix (CF) in SubXPCA, I>2 , is given by

TF2 = Q(Nk2r2) (2)

96 An Attribute Partitioning Approach to Correlation Connected Clusters

('lusters awl/or Noise objects <C:

A No

V

Select next unclassified \ ; •

Mnd ('-neighbourhood of X. N (X );

Find Correlation Dimension of N(.(Xf) using SubXPCA:

u) Partition each object into sub-objects b) Extract local features from each i'!l sub-objects set using PCA c) Apply PCA on locally extracted data to get eigen values d) If (number of eigen values > 8) < A. then

(i) Compute correlation similarity matrix, C" ;

(ii) If Count (objects in (^-neighbourhood of X. vv.r.t. C ) > u then

X is- correlation core object

No

Expand cluster w.r.l. X. using concept of direct

correlation-reachabililv

Expansion of cluster is completed

Fig. 1. The flow chart of proposed AP-4C algorithm

where r < di, and di, is sub-pattern size. From eqs.

(1) and (2), t ime complexity to calculate all covari-

ance matrices of SubXPCA, T>, is given by

7> = O(kNdf) + 0{Nk2r2) (3)

Time complexity of 4C method, T±c, is discussed in1

and is reproduced here.

TiC = 0(NTc + Nd3) = 0(NNd2 + Nd3) (4)

Vijaya Kumar Kadappa and Atul Negi 97

On the same lines, time complexity of AP-4C method, TAP, is given by

TAP = 0(NTF+Nkdf) = 0(N(kNdf+Nk2r2)+Nkdf

(5) where di is sub-pattern size and k is number of sub-patterns per pattern (See sec. 3).

Theorem 4.1. TF < Tc, Vr < diy/(k-l)/k, where 2 < k < d/2, is number of sub-patterns per pattern, r is number of chosen projection (eigenvectors per Sj and di is sub-pattern size (See step 1 in sec 3)

Proof. To prove that TF < Tc

From eq. (3), TF = kNdf + N(kr)2

TF = (l/k)Nd2 + {d2/d2)N{kr)2 (since di = d/k) TF = (l/k)Nd2 + (k2/d2) [Nd2] r2

TF = (l/k)Nd2 + (r2/d2) [Nd2] (since di = d/k)

TF = a+r^)[Nd2} (6)

Obviously, TF < Nd2 if (± + £ ) < 1

TF < Nd2 if r2/df < (1 - 1/fc)' Hence TF <Tc,Vr < disJ(k-\)/k The theorem follows. •

Lemma 4.1. Umr^i,k-~2TF « (\/k)Tc-

Proof of Lemma. Since r < d\, it is true that r > 1 (From step 1.2.3 in sec 3). To prove TF w (l/k)Tc, we have to minimize (r2/d2) (from eq. (6)). It is true that (r2/d2) reaches minimum possible value, as r tends to minimum possible value and d; tends to maximum possible value. Also it is true that (r2/d2) reaches minimum possible value, as r —> 1 and di —• (d/2) (since r > 1 and Theorem 4.1). Since di = (d/k), (di —> d/2) implies k —• 2. From previous discussion, (r2/d2) reaches minimum possible value, as r —-> 1 and fc->2. Hence the Lemma follows. •

By Lemma 4.1, TF as (l/k)Tc is true for smaller values of k and r. However, in practice, r may not be chosen as 1 (i.e. smallest possible value), especially when k is small, since the classification rate may get reduced due to less number of PVs (r). Hence some tradeoff between r and k is required to achieve good classification rate and time efficiency.

Lemma 4.2. TAP < T^c-

Proof of Lemma. From Theorem 4.1, TF < Tc

(7) Consider Second term from eq (5), i.e. Nkdf. Nkdf = (l/k2)Nd3 (since dt = d/k) Nkdf < Nd3 (since k > 2) (8) Therefore, from eqs. (4), (5), (7) and (8) the Lemma follows. •

4.3. Why is AP-4C more efficient than 4C ?

AP-4C uses SubXPCA to compute eigenvalues which are used to find correlation dimension of e-neighbourhood of a core object (See step 1.2.1 of sec. 4.1). Similarly 4C uses classical PCA for the same. In PCA methods where covariance matrix is comuted explicitly, most of the time is consumed for the computation of covariance matrix alone. In contrast to PCA (where a single large covariance matrix, C, is computed), SubXPCA computes k, (k > 2), smaller sub-pattern covariance matrices (Cj), one for each sub-pattern set, Sj, and one final covariance matrix (C F ) . By the Theorem 4.1, it is obvious that TF < Tc, Vr < di\J(k — l)/k, where r is number of chosen projection vectors per Sj, di is sub-pattern size and k is number of sub-patterns per pattern. The upperbound for r (i.e. di^/(k — l)/k) is reasonably large and in practice, we choose first few salient features (i.e. r is small in general), therefore, the computation of final covariance matrix, C F , becomes trivial. Finally, SubXPCA is faster by nearly k times to PCA (See Lemma 4.1). The concept of partitioning is the basic reason for the lesser time complexity of SubXPCA. Since we use SubXPCA in AP-4C for finding correlation dimension of e—neighbourhood, AP-4C is thus faster than 4C, which uses PCA and the same is proved in Lemma 4.2. It was found that classification results of SubXPCA and PCA were significantly same.4

5. Conclusions and Future work

In this paper, we have proposed a new and efficient 4C method, AP-4C, suitable for high dimensional data. Theoretical proofs reveal that AP-4C is more efficient than 4C. 4C becomes a special case of AP-4C if (i) the number of sub-patterns per pattern in SubXPCA is taken as 1 and (ii) step 2 of SubXPCA

98 An Attribute Partitioning Approach to Correlation Connected Clusters

is omit ted. AP-4C may be extensively used in high

dimensionality da ta mining applications.

In the near future, we a t t empt to improve our

method by using ideas from 'clustering objects on

subset of a t t r ibutes ' , 3 where the relevant a t t r ibute

subsets for each individual cluster can be different

and partially (or completely) overlap with those of

other clusters.

R e f e r e n c e s

1. C. Bohm, K. Railing, P. Kroger and A. Zimek, Computing clusters of correlation connected objects, in

Proc. of SIGMOID ACM, (Paris, France, 2004). 2. A. K. Pujari, Data mining Techniques (Universities

Press, India, 2002). 3. J. Friedman and J. Meulman, J. of Royal Statistical

Society. 66, 815 (2004). 4. A. Negi and V. K. Kadappa, An experimental study

of sub-pattern based principal component analysis and cross-subpattern correlation based principal component analysis, in Pnoc. Image and Vision Computing international confernce, (Univ. of Otago, New Zealand, 2005).

5. S. Chen and Y. Zhu, Pattern Recognition 37, 1081 (2004).

PART E

Document Analysis

101

A Hybrid Scheme for Recognition of Handwritten Bangla Basic Characters Based on H M M and MLP Classifiers

U. Bhattacharya*, S. K. Parui and B. Shaw

Computer Vision and Pattern Recognition Unit Indian Statistical Institute

203, B. T. Road, Kolkata-700108, INDIA E-mail: ujjwal&isical.ac.in*

This paper presents a hybrid approach to recognition of handwritten basic characters of Bangla, the official script of Bangladesh and second most popular script of India. This is a 50 class recognition problem and the proposed recognition approach consists of two stages. In the first stage, a shape feature vector computed from two-directional-view-based strokes of an input character image is used by a hidden Markov model (HMM). This HMM defines its states in a data-driven or adaptive approach. The statistical distribution of the shapes of strokes present in the available training database is modelled as a mixture distribution and each component is a state of the present HMM. The confusion matrix of the training set provided by the HMM classifier of the first stage is analyzed and eight smaller groups of Bangla basic characters are identified within which misclassifications are significant. The second stage of the present recognition approach implements a combination of three multilayer perceptron (MLP) classifiers within each of the above groups of characters. Representations of a character image at multiple resolutions based on a wavelet transform are used as inputs to these MLPs. This two stage recognition scheme has been trained and tested on a recently developed large database of representative samples of handwritten Bangla basic characters and obtained 93.19% and 90.42% average recognition accuracies on its training and test sets respectively.

Keywords: Handwritten character recognition; Bangla character recognition; stroke feature; wavelet based feature; HMM; MLP.

1. Introduction

Although significant development has already been achieved on recognition of handwriting in scripts of developed nations, not much work has been reported on Indie scripts. However, in the recent past, significant research progresses could be achieved towards recognition of printed characters of Indian scripts including Bangla.1 Unfortunately, the methodology for printed character recognition cannot be extended towards recognition of handwritten characters. The development of a handwritten character recognition engine for any script is always a challenging problem mainly because of the enormous variability of handwriting styles.

Many diverse algorithms/schemes for handwritten character recognition2'3 exist and each of these has its own merits and demerits. Possibly the most important aspect of a handwriting recognition scheme is the selection of an appropriate feature set which is reasonably invariant with respect to shape variations caused by various writing styles. A large number of feature extraction methods are available in the literature.4

India is a multilingual country with 18 constitutional languages and 10 different scripts. In the

literature, there exist a few studies on recognition of handwritten characters of different Indian scripts. These include Ref. 5 for Devanagari, Ref. 6 for Tel-ugu, Ref. 7 for Tamil, Ref. 8 for Oriya among others. Bangla is the second most popular Indian script. It is also the script of two other Indian languages, viz., Assamese and Manipuri. On the other hand, Bangla is the official language and script of Bangladesh, a neighbour of India. Several off-line recognition strategies for handwritten Bangla characters can be found in Refs. 9-11. A few other works dealing with off-line recognition of handwritten Bangla numerals include Refs. 12,13.

Many of the existing works on handwritten Bangla character recognition are based on small databases collected in laboratory environments. On the other hand, it is now well established that a scheme for recognition of handwriting must be trained and tested on a reasonably large number of samples. In the present work, a recently developed large and representative database14 of handwritten Bangla basic characters has been considered.

In the present article, a novel hybrid scheme for recognition of Bangla basic characters has been proposed. In the first stage, a shape vector representing all of certain directional view-based strokes of an in-

102 A Hybrid Scheme for Recognition of Handwritten Bangla Basic Characters Based on HMM and MLP Classifiers

put character is fed to our HMM classifier. The set of posterior probabilities provided by the HMM determines the smaller group of the input character among N such groups of confusing character shapes. In the second stage, three MLP classifiers for the particular group independently recognize the character using its representations at three fine-to-coarse resolutions based on a wavelet transform. Final recognition is obtained by applying the sum rule15 for combination of the output vectors of three MLPs of the particular group.

An HMM is capable of making use of both the statistical and structural information present in handwritten shapes.16 A useful property of the present HMM is that its states are determined in a data-driven or adaptive approach. The shapes of the strokes present in the training set of our image database of handwritten Bangla characters are studied and their statistical distribution is modelled as a multivariate mixture distribution. Each component in the mixture is a state of the HMM. This model is robust in the sense that it is independent of several aspects of input character sample such as its thickness, size etc. In the proposed approach, above HMM is used to simplify the original fifty class recognition problem into several smaller class problems.

Wavelets have been studied thoroughly during the last decade17 and its applicability in various image processing problems are getting explored. In the present work, Daubechies18 wavelet transform has been used to obtain multiresolution representations of an input character image. Distinct MLPs are employed to recognize the character at each of these resolution levels. Final recognition results are obtained by combining the above ensemble of MLPs. Such a multiresolution recognition approach was studied before for numeral recognition problem12 and was observed that it is robust with respect to moderate noise, discontinuity or small changes in rotation.

2. Stage I of the Recognition Scheme

The first stage of the proposed recognition scheme consists of a preprocessing module, extraction of directional view based strokes from character image, computation of stroke features and designing a classifier based on HMM.

2.1. Preprocessing

For cleaning of possible noise in an input character image, it is first binarized by Otsu's thresolding technique19 followed by smoothing using median filter of window size 5. No size normalization is done at the image level since it is taken care of during feature extraction. A sample image from the present database and the effect of smoothing on extracted strokes by the subsequent module is shown in Figs. 1(a) - 1(1).

"3* <T* "eft ({:\ z^ T-(«) (6) <B <A V) """GO

ZJI *3* "c^ ^ ^ s - ^ « ) (H) CO 0') <# CO

Fig. 1. (a) Image of a sample character; (6) image in (a) after binarization; (c) vertical strokes obtained from (6); (d) vertical strokes obtained after pruning (c); (e) horizontal strokes obtained from (6); ( /) horizontal strokes obtained after pruning (e); (g) image of the sample character after smoothing; (h) image of the smooth character sample after binarization; (i) vertical strokes obtained from (h); (j) vertical strokes obtained after pruning (i); (k) horizontal strokes obtained from (h); (I) horizontal strokes obtained after pruning (k).

2.2. Stroke Extraction

Horizontal and vertical strokes present in the input character shape are obtained in the form of digital curves each of which is one-pixel thick and in which all the pixels except the terminals have exactly two 8-neighbours. Certain two directional view based approach is considered for extraction of these strokes. In this approach, a binarized image E consisting of only the vertical strokes present in the input character shape is obtained by removing all the object pixels of the binarized character image other than those for which right or east 4-neighbour is a background pixel, that is, where the pen movement is upward or downward. In other words, the object pixels of the input sample image that are visible from the east form the vertical strokes as shown in Fig. 2(a). Similarly, another binarized image S consisting of only the horizontal strokes is generated. The object pixels of the binarized character image whose bottom or south 4-neighbour is a background pixel, that is, where the pen movement is side-wise form the horizontal strokes as shown in Fig. 2(b). Thus, here,

U. Bhattacharya, S. K. Parui and B. Shaw 103

connected components of images E and S are respectively vertical and horizontal strokes. Vertical strokes consisting of pixels less than 20% of the height and horizontal strokes consisting of pixels less than 20% of the width of the input image are ignored during further processing. Such vertical and horizontal strokes for a character shape in Fig. 1(a) are respectively shown in Figs. l ( j) and 1(1).

(a) (b)

Fig. 2. Strokes in a character sample are shown, (a) Dark pixels indicate E image; (b) dark pixels indicate S image.

2.3. Computation of Feature Components

On each stroke, six points Pi,i = 0, . . . , 5 are obtained such that the curve lengths Pj_iPj, i = 1,...,5 are equal with P$ and P5 are respectively the lowest and highest pixels for a vertical curve or the leftmost and rightmost pixels for a horizontal curve. The feature components corresponding to each stroke from an input character are obtained as {oti,ct2,- • • ,cts,X,Y,L}, where on is the angle between the line Pi- iPj and the positive X-axis, (X, Y) is its centroid and L is the length of the stroke. It represents the shape, size and position of the stroke with respect to the character image. The quantities a* are invariant under scaling and the remaining three quantities X, Y and L are normalized with respect to the character height.

For an input character, all the strokes are arranged from left to right and obtained the observation sequence O — (0\,..., OT), where T is the number of strokes and Oi = {an, aa,..., ai5,Xi, Yt, Lt} corresponds to the i-th stroke in the above arrangement.

2.4. HMM Based Classifier

The HMM designed for the present classification task is a non-homogeneous one and the classifier used in the first stage of the proposed recognition scheme consists of 50 such HMMs denoted by 7,-, j = 1,2,.. . , 50 one for each of the underlying 50 classes.

For an input pattern X, the probability values P(0 |7 j ) are computed for all j = 1,2,.. . , 50 using the well known forward and backward algorithms.20

Finally X is assigned to class c e {1 ,2 , . . . , 50} such that c = arg max {P(0 |7 , )} . Computational de-

l< j<50

tails are avoided due to space constraints. It has been assumed above that the feature val

ues {oti,a-2,... ,a5,X,Y,L} corresponding to the strokes obtained from training samples of each class follow a mixture of K 8-dimensional Gaussian distributions. The unknown parameters of this mixture distribution for different choices of K are obtained using the well known EM algorithm.21 The optimum value of K for each class is determined by using the Bayesian information criterion (BIC).22 The mean vectors of these K distributions are called shape primitives for the corresponding character class and these form the state space of the associated HMM. Thus, here, the states of an HMM are not determined a priori but are constructed adaptively on the basis of the training set of character samples of the respective class. The parameters of the underlying HMMs (such as the initial and transition probability matrices) are computed on the basis of the above states.

3. Final Classification

An analysis of the recognition performance of the above HMM based classifier on both the training and test sets shows that a significant percentage of mis-classifications occurred within several smaller groups of character classes. This pattern of misclassifications is shown in Table 1.

In the second stage, fresh classification is performed for each of the above sub-groups. It significantly improves the overall recognition accuracy of the first stage. However, in this context, it is obvious that if an input sample in the first stage of its classification is misclassified in a group other than its own (as shown in Table 1), then the second stage of the present classification scheme fails to correctly classify the sample. An input sample misclassified in the first stage as another character of its own group, gets a second chance of being properly classified by the second stage of classification. Brief details of the final classification scheme is given below. A similar scheme23 was used before for recognition of hand printed Bangla numerals.

104 A Hybrid Scheme for Recognition of Handwritten Bangla Basic Characters Based on HMM and MLP Classifiers

Table 1. Groups of confusing character shapes provided by the HMM classifier

Group No.

1

2

3

4

5

6

7

8

Groups of confusing characters

3T 3It * * , * $ Sfcfr* ^ *r TF * 3 \3 !ft" T5 v» 3 * *r*r*r*TTrwws' irsr * ?&* T^s \? % t <* j>*rFJto!T«t*r?r*rT>

Total

No. of 1 in the

Training

600

1200

1200

1500

1800

3000

2700

3000

15000

samples group

Tea

400

400

707

1000

1200

1974

1800

2000

9481

Within '

Tiaining

22.50

12.00

20.50

10.27

30.17

33.00

15.85

27.23

23.05

reclassification percentage group Between groups Total

Test

24.00

12.25

21.22

11.90

32.83

37.13

17.06

29.30

25.67

Training

3.83

3.92

5.17

5.60

3.67

6.77

4.96

6.23

5.37

Test

4.50

4.25

5.94

7.50

3.92

952

6.78

7.65

6.98

Training

26.33

15.92

25.67

15.87

33.84

39.77

20.81

33.46

28.42

Test

28.50

16.50

27.16

19.40

36.75

46.65

23.84

36.95

32.65

3.1. Multi-resolution Representation of a Character Image

The bounding box (minimum rectangle enclosing the shape) of a character image is first normalized to the size 64 x 64. Wavelet decomposition algorithm24is applied to this normalized image recursively for two times to obtain an 16 x 16 smooth.. .smooth approximation of the original image as the final decomposition. Here, we also obtain a 32 x 32 smooth.. .smooth approximation of the original image at an intermediate stage. Images of a sample character at multiple resolutions are shown in Figs.3. The images at these three fine-to-coarse resolutions are gray-valued ones and as before Otsu's thresholding technique is used for their binarization.

(a) (b) (c) id)

Fig. 3. Image of a character sample at different resolutions: (o) The original image; (6) Size normalized (64 x 64) image; (6) Wavelet transformed image at resolution (32 x 32); (d) Wavelet transformed image at resolution (16 x 16).

3.2. Classification at Multiple Resolutions

For an input character, images at each of the above three fine-to-coarse resolutions are fed to the input

layers of three different MLP networks. Optimal size of the hidden layer in each case has been determined through near exhaustive simulation runs. The size of the output layer is equal to the number of classes involved in the recognition task and this number varies for different groups of Table 1. These MLPs are trained using the well known backpropagation training algorithm. For the same sample of any group, the values at the output layers of the corresponding three MLP classifiers are usually different and this is the reason that we have combined the output values of these MLPs obtaining better recognition accuracy.

3.3. Combination of MLP Classifiers at Multiple Resolutions

It is now very popular in the OCR community to consider combination of multiple classifiers for better recognition performance than that of a single classifier. There exist various approaches for such combination of outputs of multiple classifiers. These combination approaches have recently been studied in Ref. 25. Such combination approaches include product rule, sum rule, median rule, majority voting etc.

It has been observed that the product rule provides the worst results which are also worse than any of the individual classifiers while the performance of the sum and median rule are good. Also, performance of the majority voting rule is reasonably good. The recognition results shown in the next Section have been obtained by combining the three MLP classifiers using sum rule.

U. Bhattacharya, S. K. Parai and B. Shaw 105

4. Experimental Results and there are 9481 samples in the test set.

Most of the existing off-line recognition studies on handwritten characters of Indian scripts are based on different databases collected either in laboratory environment or from smaller groups of the concerned population. However, recently a large and representative database of handwritten Bangla basic characters was developed and the simulation results of the proposed recognition approach have been obtained based on the training and test samples of this database.

Vc^i" **«-* -&\S^

i'S* ^tSv& s f 6 ^ -

^5PJ " * *0 -zl-sn*

$ £> $ ^^^ %%-s-

o o o

O o f

r \s--w\y

s ^

Fig. 4. Samples from the present database of handwritten Bangla characters; three samples in each category are shown.

4.1. Database of Handwritten Bangla Basic Characters

Samples of the present database were collected by distributing several standard application forms among different groups of population of the state of West Bengal in India and the purpose was not disclosed. Subjects were asked to fill-up these forms in Bangla by putting one character per box. Such handwritten samples were collected over a span of more than two years.

Sample characters of the present database are stored as grayscale images at 300 d.p.i. resolution. These are TIF files with 1 byte per pixel. A few samples images from the present database are shown in Fig. 4.

The present database consists of 24481 images of isolated Bangla basic characters explicitly divided into training and test sets. The training set consists of 15000 samples with 300 samples for each category

4.2. Recognition Results of the First Stage

In the first stage of the present hybrid recognition scheme based on an HMM classifier, horizontal and vertical strokes have been extracted as described in Section 2.2. Parameters of the HMM corresponding to a character class are estimated using shape vectors computed from its set of strokes as described in Section 2.3. For example, 7 is the minimum value of K and was obtained for the 42-th character of Bangla alphabet while 18 is the maximum value of K and was obtained for the 24-th character of the alphabet.

The first stage of the present scheme provided only 71.58% and 67.35% average recognition accuracies respectively on the training and test sets of the present database. However, from Table 1, it is seen that 23.05% and 25.67% of the misclassifications occurred within 8 smaller groups consisting of 2 to 10 different character classes.

4.3. Recognition Results of the Second Stage

Combination of three MLP classifiers within each of the above smaller groups of Bangla characters provided 98.48% and 97.21% average recognition accuracies respectively for the training and test samples. Individual recognition accuracies at the second stage for each of the above groups are shown in Table 2.

Thus, at the end of the second stage, the overall final recognition accuracies are respectively 93.19% and 90.42% on the training and test sets of the present database of handwritten Bangla basic characters.

5. Conclusions

In the present work, we considered a hybrid approach to recognition of handwritten Bangla basic characters. In this approach, both HMM and MLP classifiers have been used. Shape features are used by the HMM classifier of the first stage while pixel images at multiple resolutions are used by MLP classifiers of the second stage. The approach of the second stage was studied before providing acceptable recognition accuracies in smaller problems such as recognition of

106 A Hybrid Scheme for Recognition of Handwritten Bangla Basic Characters Based on HMM and MLP Classifiers

Table 2. Recognition performance of the 2nd stage of the proposed scheme

Group No.

Characters belonging to different groups

Samples correctly classified Correct recognition (%) in respective groups within each group during the 1 st stage by the 2nd stage

Training Test Training Test

3T 31T

^ It T? ? 1 \3 \B 'ST ^ 'S "»

»J JJ IT T ¥ * ^ 7T V *f

j*ro _ j toJT e r*r?r a tT5

577

1153

1138

1416

1734

2797

2566

2813

382

383

665

925

1153

1786

1678

1847

100

99.57

99.29

99.08

99.13

97.57

98.17

97.87

98.95

98.17

98.35

97.95

98.01

96.75

97.26

95.78

Total 14194 8819 98.48 97.21

handwri t ten Bangla numerals. However, when it is a

larger class problem, the performance of the scheme

is not equally satisfactory. This is the reason for our

choice of shape features with an HMM in the first

stage and MLP-based multi-resolution recognition

approach in the latter stage.

References

1. B. B. Chaudhuri, U. Pal, Pattern Recognition, 31, 531-549 (1998).

2. R. Plamondon, S. N. Srihari, IEEE Trans. Patt. Anal, and Mach. Intell., 22, 63-84 (2000).

3. N. Arica, F. Yarman-Vural, IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews, 31, 216 - 233 (2001) .

4. O. D. Trier, A. K Jain, and T. Taxt, Pattern Recognition, 29, 641 - 662 (1996) .

5. K. R. Ramakrishnan, S. H. Srinivasan, S. Bhaga-vathy, Proc. of the 5th ICDAR, 414-417 (1999).

6. M. B. Sukhaswami, P. Seetharamulu, A. K. Pujari, Int. J. Neural Syst., 6, 317-357 (1995).

7. R. M. Suresh, L. Ganesan, Proc. of Sixth IC-CIMA'05, 286-291 (2005).

8. S. Mohanti, IJPRAI.12, 1007-1015 (1998). 9. U. Bhattacharya, S. K. Parui, M. Sridhar, F.

Kimura, CD Proc. IICAI,1357-1376 (2005). 10. T. K. Bhowmik, U. Bhattacharya, S. K. Parui, Proc.

ICONIP, 814-819 (2004) . 11. F. R. Rahman, R. Rahman, M. C. Fairhurst, Pattern

Recognition, 35, 997-1006 (2002). 12. U. Bhattacharya, B. B. Chaudhuri, Proc. of ICDAR,

Seoul, 322-326 (2005).

13. U. Bhattacharya, T. K. Das, A. Datta, S. K. Parui, B. B. Chaudhuri, International Journal for Pattern Recognition and Artificial Intelligence, 16, 845-864 (2002).

14. www.isical.ac.in/~ujjwal/download/database.html, "OFF-LINE HANDWRITTEN CHARACTER DATABASE".

15. J. Kittler, M. Hatef, R. P. W. Duin and J. Matas, IEEE Trans, on Patt. Anal, and Mach. Intell. 20 226-239 (1998).

16. H. Park and S. Lee Pattern Recognition, 29, 231-244 (1996).

17. A. Graps, IEEE Computational Science and Engineering, 2(2), (1995).

18. I. Daubechies, IEEE Trans, on Information Theory. 36, 961-1005 (1990).

19. N. Otsu, IEEE Trans. Systems, Man, and Cybernetics, 9, 377-393 (1979)

20. L. R. Rabiner, Proc. Of the IEEE, 77(2), 257-285 (1989)

21. K. Fukunaga, Introduction to Statistical Pattern Recognition, Academic Press, San Diego, 2nd Ed., 1990.

22. J. Bernardo and A. Smith, Bayesian Theory. John Wiley & Sons, 1994.

23. U. Bhattacharya and B. B. Chaudhuri, Proc. of the 7th Int. Conf. on Document Analysis and Recognition, vol. I, 16-20 (2003).

24. S. G. Mallat, IEEE Trans, on Pattern Anal, and Machine Int., 11(7), 674 -693 (1989).

25. I. K. Ludmila, IEEE Trans. On Patt. Anal, and Mach. Intell., 24, 281-286 (2002).

107

An Efficient Method for Graphics Segmentation from Document Images

S. Mandal, S. P. Chowdhury and A. K. Das

CST Department Bengal Engineering and Science University

Howrah - 711 103, India E-mail: {sekhar,shyama,amit} @cs.bees.ac.in

B. Chanda

Electronics and Communication Sciences Unit Indian Statistical Institute Kolkata - 700 108, India

E-mail: [email protected]

Major constituents of any document are text, graphics and half-tones. While half-tone c a n be characterised by its inherent intensity variation, text and graphics share common characteristics except difference in spatial distribution. The success of document image analysis systems depends on the proper segmentation of t e x t and graphics as text is further subdivided into other classes like, heading, table, math-zones. Segmentation of graphics is essential for better OCR performance and vectorization in computer vision applications. Graphics segmentation from text is particularly difficult in the context of graphics made of small components (dashed or dotted lines etc.) which have many features similar to texts. Here we propose a very efficient technique for segmenting all sorts of graphics from document pages.

Keywords: Document Image Analysis (DIA), Graphics Segmentation

1. Introduction

Due to its immense potential for commercial applications research in Document Image Analysis (DIA) supports a rapidly growing industry including OCR, vectorization of engineering drawings and Vision systems. Commercial document analysis systems are now available for storing business forms, performing OCR on typewritten/handwritten text, and compressing engineering drawings. Graphics detection is one of the first application areas in the document processing system. However, until today we do not have any efficient method for detecting for all type of graphics appeared in frequently used real life documents. Here, we are mainly focusing on the segmentation of the graphics from a document page, which is already half-tone segmented, and may not be even fully skew corrected.

This paper is organised as follows. Section 2 describes past research. Proposed method is detailed in Section 3. Concluding section (Section 4) contains experimental results and remarks.

2. Past work

Graphics segmentation is attempted and reported by many researchers.1-11 Quite a few are in the domain of text graphics separation and in many cases text

strings are separated out thus indirectly segmenting graphics as left-overs. Texture based identification is proposed by4 exploiting the nature of the texture of regular text to be supposedly different from that of graphics by using Gabor filters. We have come across many references where graphics are identified from engineering drawings for the purpose of vectorization.10-13 However, engineering drawings are special cases of document images containing predominantly graphics portion.

In a nutshell we are aware of three different approaches for separating out graphics from text; they are;

(1) Directional morphological filtering.

The technique is applied to locate all linear shapes and are considered to be text; effectively leaving other objects as graphics. This works well for simple maps14 and may have inherent problems in dealing with more complex scenes.

(2) Extraction of lines and arcs etc.

Relying on transform15 or vectorization16

many had tried t o isolate the graphical objects from text. This approach works well for engineering drawings.

108 An Efficient Method for Graphics Segmentation from Document Images

(3) Connected component Analysis.

A set of rules (based usually on the spatial characteristics of the components (text and graphics) is used to analyse connected components to filter out the components. The algorithms can handle complex scenario and can be tuned to deal increasingly complex documents. One of the best examples of this class is the work done by Fletcher and Kasturi.1

Here we elaborate, the well known and frequently referred endeavour by Fletcher and Kasturi.1 This is a deliberate choice as our approach, in many ways, resembles their approach based on connected component analysis They have done text-string separation from mixed graphics thereby indirectly segmenting the graphics part. On the other hand, we tried to separate graphics from the document containing both text and graphics using simple spatial properties.

The major steps of Flecher and Kasturi's algorithm are as follows:

1) Connected component generation: This is done by component labeling process which also computes the maximum and minimum co-ordinates of the bounding rectangles.

2) Filtering using area/ratio: This is used to separate large graphical components. The filtering threshold is determined according to the most populated area and the average area of the connected components. The working set to be used by the next step is now reduced to text strings and small graphical components.

3) Co-linear component grouping: Hough transform is used on the centroids of the rectilinear bounding boxes of each component. This operation consolidates grouping of co-linear components.

4) Logical-grouping: Co-linear components are grouped to words by using information like position of each component, inter character and inter word gap threshold.

5) Text string separation: The words along a text line is segmented from the rest of the image.

The algorithm is robust to changes in font size a

and style. It is by and large skew independent but sensitive to touching and fragmented characters. The

aassumed that the maximum font size in the page should be less than 5 times of the minimum font size used in the same

page

inherent connectivity assumption weakens with images of degraded/worn out documents due to a large number of touching and fragmented characters. As the aspect ratio in graphics and text vary widely the dependence on area/ratio filter is unwieldy. The algorithm is also computationally expensive like any other Hough transform based approach.

Next we present the technique used by us to separate graphics which is based on similar techniques but computationally cheaper and more effective in segmenting line art made of dotted or dashed lines or very short line segments irrespective of their orientation

3. Proposed Method

We have started with gray image of the document and using a texture based technique17 half-tones are removed. Next, it is converted to binary image by the well known technique proposed by Otsu;18 so we do the graphics segmentation with binary images.

Graphics segmentation using connected component analysis yields good results if the lines and arcs forming the graphical component are all connected. Here are a pair of examples shown in Fig. 1. Note that the graphics (single big connected component) is totally removed in both the cases.

However graphics made of dotted (or dashed) lines or short line segments are difficult to detect as the size of the individual components are similar to text characters. Therefore, the individual connected component does not signify anything, but a sequence (or group) of such components together represent graphics. The presence of such graphics which are difficult to detect is shown in Fig. 2.

Special care is taken in our approach to segment out graphics made of dotted and dashed lines in any orientation.

Our approach to detect graphics made of small components is based on grouping of small disconnected components supposedly form a big (spatially) component. In order to segment these cases we have made use of grouping of small components starting from any particular point by observing

(1) adjacency, (2) overlapping, (3) pseudo continuity and (4) stroke width

S. Mandal, S. P. Chowdhury, A. K. Das and B. Chanda 109

Fig. 1. Segmentation using connected component analysis; First row shows two pages with graphics made of connected components and the second row shows the result after graphics removal.

(a) (b)

Fig. 2. Graphics made of small components (a) Dotted lines along with big connected components; (b) Graphics with dotted and dashed lines.

of nearby components. It may be observed that the text characters do

possess similar properties but there are subtle differences. For example, two characters are adjacent to each other mostly in horizontal direction and they share overlapping of their extended bounding boxes, again, in horizontal direction. As the orientation of

the document is known and the text line detection of skew free document is also known there should not be any confusion in identifying two nearby characters belonging to two adjacent text lines. Thus, grouping of lines made of dots or dashes and small arcs is possible as elaborated next.

We start with the characteristics of graphics made with dashed or dotted lines;

(1) Number of foreground to background transitions is 1 in vertical or horizontal directions.

(2) Ratio of height to width is within a range of 0.5 to 2.

(3) Ratio of foreground to background pixel within the bounding box encompassing a component is more than 0.5. This rule is exclusive for dots only.

(4) Two components would treated as adjacent if their extended (5 times) bounding boxes have spatial overlap.

Applying the above rules we could group small components forming part of a graphics object. However, we have introduced more checking to rule out the possibilities of grouping small components available in normal text. We form a group of valid components (small) if the count goes more than 5. This is a check against the possibilities of grouping 5 consecutive dots (dots of i and j) together.13.

For grouping of solid short arc segments we use the following rules.

(1) Number of foreground to background transitions is 1 in vertical or horizontal directions.

(2) The ratio of pen width of the two components will have a range of 0.5 to 2.0.

(3) Extended skeleton of the components will have spatial overlaps.

At this juncture; the terms pen width and extended skeleton need be explained.

Pen width is a measure of the stroke width of the arcs forming a graphics. It is computed by drawing four lines (with 0, 45, 90 and 135 degree slopes) from all the points in the skeleton to the object boundary and computing average of the minimum of those radial lines. It is defined as below:

Pw = Average of all skeleton points (Minimum o f ( / ° , / 9 0 , ( / 4 5 + / 1 3 5 ) / 2 ) )

It may be noted that minimum of 3 radial lines are

bConsider the word "divisibility" which has got 5 is'

110 An Efficient Method for Graphics Segmentation from Document Images

J A ^ V ^ - ' V .

Fig. 4. Result of graphics Segmentation. Original Images in the left column and segmented images are in the right column.

actually taken as in actual calculation average of the lines in slopes 45 and 135 degrees are considered. The

pen width is used to verify that adjacent components belonging to a group if the pen width has a permitted variation. Note that for text portion pen width variation is limited and the same is true in case of graphics made up of short lines and arcs where abrupt changes are unlikely.

Extension of the skeleton is done by dividing the skeleton of the component into four parts along the principal axis. So we get three control points (points of divisions) in the skeleton. We make a copy of the lower half and upper half of the skeleton. Take their mirror images and join them from the lower and upper end points (trying to maintain the slope). This is shown in Fig. 3.

/ / / / i f I

|(a) fbj (cl (d) (e) |

Fig. 3. Expansion of the skeleton taking the mirrored and flipped lower and upper half of the skeleton; a) Original component; (b) Its skeleton; (c) Lower part of the skeleton; (d) upper part of the skeleton and (e) Extended skeleton.

This effectively extends the skeleton as shown in the figure roughly maintaining the original slope in both the ends. So, in one sense it is a pseudo window creation strategy in which the components, in their extended form, come closer to each other. Thereafter, the adjacency conditions are checked to form a group. Adjacency conditions need be fine tuned to accommodate adjacent components in bends, crossings and corners. Without such a measure, a single curved graphics component may be segmented as multiple one as the component at the corners or bends would be the missing links. To accommodate them we will accept the number of vertical or horizontal transitions be more than 1 but the pen width remains within the range of 0.5 to 2.0.

The results of the graphics separation is shown in Fig. 4 for a number of cases.

S. Mandal, S. P. Chowdhury, A. K. Das and B. Chanda 111

4. E x p e r i m e n t a l R e s u l t s

We have carried out the experiments using around

200 samples taken from the U W - I , U W - I I databases

as well as our own collection of scanned images from

variety of sources e.g., books, reports, magazines and

articles. The summary of the results is presented in

Table 1 for graphics zones whose ground t ru thed in

formation is available in the U W - I and II database.

Note tha t the ground- t ru th information for our col

lection is done manually. All experiments were car

ried out in a p4 based high end P C and all the pro

grams are writ ten in C and the average segmentation

time excluding the half-tone removal is around 3.4

second.

Table 1. Segmentation Performance (in %)

Cla

ssi

fied

(BG)

(SG)

(OC)

Big Graphics

(BG)

98

0

2

Actual Small

Graphics (SG)

0

92

8

Other Components

(OC)

2

10

88

The table shows near perfect result for graph

ics with big connected components. Result for small

graphics is also very impressive; however we fail to

cluster some of them as they are dispersed in the

graphical portion. It may be noted tha t this segmen

tat ion algorithm is fully automatic and the param

eters used work satisfactorily for a wide variety of

fonts and styles.

R e f e r e n c e s

1. L. Fletcher and R. Kasturi, IEEE Transaction on Pattern Analysis and Machine Intelligence 10 (6), 910 (1988).

2. O. T. Akindele and A. Belaid, Page segmentation

by segment tracing, in ICDAR93, (Tsukuba, Japan, 1993).

3. K. C. Fan, C. H. Liu and Y. K. Wang, Pattern Recognition Letters Vol. 15, 1201 (1994).

4. A. K. Jain and S. Bhattacharjee, Machine Vision and Application 5, 169 (1992).

5. A. K. Jain and B. Yu, Page segmentation using document model, in Proc. ICDAR'97, Ulm, Germany, August 1997.

6. A. K. Das and B. Chanda, Segmentation of text and graphics from document image: A morphological approach, in International Conf. on Computational linguistics, Speech and Document Processing (ICCLSDP'98); Feb. 18-20,Calcutta, India:, 1998.

7. T. Pavlidis and J. Zhou, Computer Vision Graphics and Image Processing vol. 54, 484 (1992).

8. F. M. Wahl, K. Y. Wong and R. G. Casey, CGIP 20, 375 (1982).

9. W. Liu and D. Dori, Computer Vision and Image Understanding 70(3), 420 (1998).

10. C.-C. Han and K.-C. Fan, Pattern Recognition 27(2), 261 (1994).

11. T. Pavlidis, CVGIP 35, 111 (1986). 12. J. Song, F. Su, C. Tai, J. Chen and S. Cai, Line

net global vectorization: An algorithm and its performance evaluation, in Proc. CVPR, Nov. 2000.

13. J. Chiang, S. Tue and Y. Leu, Pattern Recognition 12, 1541 (1998).

14. H. Luo and R. Kasturi, Improved Directional Morphological Operations for separation of Characters from Maps/Graphics, in K. Tombre and A. K. Chhabra, editors, Graphics Recognition - Algorithms and Systems, LNCS, Volume 1389, (Springer-Verlag, 1998), pp. 35-47.

15. A. Kacem, A. Belaid and M. B. Ahmed, IJDAR 4, No . 2, 97 (2001).

16. D. Dori and L. Wenyin, Vector based Segmentation of Text Connected to Graphicss in Engineering Drawings, in Advances in Structural and Syntactical Pattern Recognition, P. Perner, P. Wang and A. Rosenfeld, editors, volume 1121 of LNCS, (Springer-Verlag, August 1996), pp. 322-331.

17. A. K. Das and B. Chanda, Extraction of half-tones from document images: A morphological approach, in Proc. Int. Conf. on Advances in Computing, Calicut, India, Apr 6-8, 1998.

18. N. Otsu, IEEE Trans. SMC, 9 N o . 1, 62 (1979).

112

Identification of Indian Languages in Romanized Form

Pratibha Yadav*, Girish Mishra and P. K. Saxena

Scientific Analysis Group Defence Research and Development Organization

Ministry of Defence Metcalfe House, Delhi-110054> India

E-mail: [email protected]*

This paper deals with identification of romanized plaintexts of five Indian Languages - Hindi, Bengali, Ma-nipuri, Urdu and Kashmiri. Fuzzy Pattern Recognition technique has been adopted for identification. Suitable features/characteristics are extracted from training samples of each of these five languages and represented suitably through fuzzy sets. Prototypes in the form of fuzzy sets are constructed for each of these five languages. The identification is based on computation of dissimilarity with prototypes of each of these languages. It is computed using a dissimilarity measure extracted through fuzzy relational matrices. The identifier proposed is independent of dictionary of any of these languages and can even identify plaintext without word break-ups. The identification can be used for automatic segregation of plain texts of these languages while analysing intercepted multiplexed interlieved Speech/Data/Fax communication on RF channel, in a computer network or Internet.

Keywords: measures

Fuzzy sets; Fuzzy relations; Linguistic charateristics; Fuzzy Pattern Recognition (FPR); Fuzzy distance

1. Introduction

In digital communications, be it terrestrial or satellite based, many channels are used to carry across multiplexed Speech/Data/Fax, of course with suitable modulation and following required protocols. In case of Networks such as Internet also, where TCP/IP protocol is followed, communications also take place in the form of packets of Speech/Data/Fax. Since English had been a language of common use for many, most of text communication has been in English language. With the emergence of need for Information flow and free exchange of ideas among various communities, application softwares are being developed for languages other than English. Regional languages are emerging as a viable media for both written as well as spoken communication. When it comes to secure communication, languages also provide a natural barrier apart from security through encryption. Thus, while monitoring/intercepting such plain traffic, though the protocol followed helps in segregating texts from voice and fax, yet the problem still remains to segregate text messages into different regional languages without expert domain knowledge. In case, the communication is protected through encryption the problem becomes more complex when one needs decryption first and then go for text/language identification. It is this problem that needs to be addressed. In this paper a solution has been proposed towards

this issue using Fuzzy Pattern Recognition approach. While using regional languages for text communication the most common way would be to roman-ize such texts using 26 Roman Alphabets and apply the existing computer and communication tools, which are based on English. This romanization can be done either following certain standards or some non-standard natural way based on phonetics. Out of these two, the second one is more common and natural but involves more vagueness and uncertainty, and that is the reason, fuzzy logic was found to be suitable to address the issue of identification of various romanized regional languages.

Fuzzy Logic, introduced by Zadeh in 1965,l provides a handy tool to deal with uncertainty (vagueness).8 Fuzzy Pattern Recognition2-5 has been one of the main application oriented research that had been pursued by many researchers.2 Most of the work on Language identification9-11 has been based on dictionary.7 For the first time Fuzzy Pattern Recognition based techniques were applied for identification of three European languages namely English, German and French even when the work-break-up was not known.6

In this paper, the problem of Language Identification for non-standard romanized plaintexts of five Indian languages - Hindi, Bengali, Manipuri, Urdu and Kashmiri has been tackled using Fuzzy Pattern Recognition (FPR) when the texts are continuous

Pratibha Yadav, Girish Mishra and P. K. Saxena 113

(without word break-up) and no dictionary is available. The problem is quite challenging as all the five languages are quite similar phonetically and moreover romanization is non-standard.

A set of 12 feature fuzzy sets has been selected for the purpose of classification and a classification algorithm has been designed based on fuzzy dissimilarity as described in the following sections.

2. Features for Classification

For the problem of identification of these five Indian languages, which are phonetic in nature, the linguistic characteristics of these languages are exploited. These characteristics are based on the occurrences of various alphabets and their affinity to combine with some alphabets. After a thorough and careful study of these languages a set of fuzzy features has been selected, based on the following set of linguistic characteristics:

1. Frequencies of alphabets 2. Variety of left contact letters of an alphabet 3. Variety of right contact letters of an alphabet 4. Variety of two-sided contact letters of an alphabet 5. Frequencies of doublets 6. Occurrences of highest diagraph starting with a letter 7. Occurrences of highest diagraph ending with a letter 8. High-medium-low-very low frequency categorization 9. Frequency of the alphabet with which highest diagraph starting with a letter is formed 10. Frequency of the alphabet with which highest diagraph ending with a letter is formed 11. Frequency of the alphabet with which the specified alphabet makes most frequent reversal 12. Frequency of the alphabet with which the specified alphabet makes least frequent reversal

Fuzzy sets corresponding to each of these characteristics have been constructed with the set of 26 alphabets as the basic (Universal) set by defining characteristic values in the interval [0,1]. The characteristic values of the fuzzy set m corresponding to the first characteristic are obtained by dividing the frequencies of alphabets by maximum frequency of alphabets. The characteristic values of the fuzzy sets /i2, /i3 and fii corresponding to characteristics 2, 3 and 4 respectively are obtained by normalization of

the entries by 23 as the maximum number of different letters contacting a given letter does not exceed 23. For construction of fuzzy sets /is, fi& and /X7 corresponding to characteristics 5, 6 and 7, corresponding scores are taken out of a text length of 10. For constructing fuzzy set for frequency categorization very high, high, medium, low and very low frequent letters, characteristic value /*8 is taken as

/i8(x) = 1.0 if fii(x)>0.7 = 0.9 if 0 . 5 < / u ( x ) < 0 . 7 = 0.7 if 0 .3</xi(x)<0.5 = 0.5 if 0.1 </xi(x)<0.3 = 0.1 if otherwise

Finally for the construction of fuzzy sets fig, /*io, / in , and /U12 corresponding to characteristics 9,10,11 and 12, normalization is done by dividing the values by maximum frequency to bring down the membership grades in the interval [0,1].

Thus, for each of the five languages considered, such fuzzy sets are constructed by choosing a large number of texts of each of languages (each text with length=400 characters). Finally, for each of these languages standard feature fuzzy sets/prototypes were constructed by taking averages. Thus five sets of prototypes say, H? , A»f, fJ-f1, Mi7 and fif (i = 1 , . . . , 12) are extracted for the languages Hindi, Bengali, Ma-nipuri, Urdu and Kashmiri respectively.

After the construction of prototypes for each of these languages, the next problem is to develop a classification criterion so that a given unknown text can be identified and classified to one of these five classes.

3. Classification Criteria

For classification of patterns, one has to use some distance measure or similarity measure to decide about the closeness of the unknown pattern with various prototypes. There are various distance measures5 like Hamming Distance, Euclidean distance, Minkowski's distance etc. which can be used for comparing two feature fuzzy sets /i and v.

All of these distance measures were tested but the classification score was not very satisfactory. Hence, in this paper, a new dissimilarity measure defined in6'11 has been used. It is defined through the following fuzzy relation fin between /i and v.

114 Identification of Indian Languages in Romanized Form

pR(xi,xj) = e-kMx*-''W (1)

Here k and I are parameters, which can be fixed imperically or through experimentation. Such a matrix R with entries coming from fuzzy relations of the form (1) is called a fuzzy relational matrix, denoted by /XR. Then the dissimilarity between fuzzy sets \x and v is defined using fuzzy relational matrix HR as

df(n, v) = a(26 - TracefiR) +

+ PJ2T,\lJ'R(xi>xj) -f*R{xj,Xi)\ (2) * 3

After lots of experimentation and learning, the parameters k and I were chosen to be 1.0 and 2.0 respectively. The values of a and (3 were fixed as 2.0 and 0.25 respectively. For any given unknown text 12 feature fuzzy sets /x i , . . . , /ii2 are calculated. For calculating the association of this unknown pattern with a class say Hindi, the following process is followed. For each i, /i, is compared with (if1 and dissimilarity measure df is calculated using (1) and (2) as

d? = dI{&,fl?) (3)

The final dissimilarity value DH of the unknown text with the Hindi language is calculated as follows:

12

DH = 5> 4 d? (4) t = i

where tUj's are weights. These weights are chosen according to the significance of the characteristics. In our case the weights were selected as

W\ W2 Ws W4 W5 WQ

0.140 0.135 0.135 0.135 0.059 0.100 W-j W% Wg Wio Wn W\2

0.100 0.056 0.035 0.035 0.035 0.035 Similarly, other dissimilarity measures DB, DM,

Du, and DK (with Bengali, Manipuri, Urdu and Kashmiri resp.) were computed. The unknown text is classified to the class with which the value of the dissimilarity measure is minimum.

4. Results

The Algorithm has been tested on a number of texts from each of the five languages. Text length has been taken as 400 characters (chars). The program has been tested on PC and takes few seconds of CPU.

Dissimilarity values of twenty-five test samples with each of the five classes have been reflected in Table 1. Out of these, text 1 to text 5 are Hindi texts, text

Table 1. Values of the Dissimilarity Measures with Prototypes of Five Languages.

Texts

Text 1 Text 2 Text 3 Text 4 Text 5

Text 6 Text 7 Text 8 Text 9 Text 10

Text 11 Text 12 Text 13 Text 14 Text 15

Text 16 Text 17 Text 18 Text 19 Text 20

Text 21 Text 22 Text 23 Text 24 Text 25

DH

03.85 03 .86 03.79 03.75 03.62

04.76 04.04 04.67 04.57 05.43

04.45 04.90 04.55 04.49 04.47

04.55 04.43 04.18 04.41 04.14

04.57 04.07 04.63 04.64 04.70

DB

05.91 05.83 05.99 06.52 06.00

02 .83 03.67 02 .88 02.91 02 .73

05.25 06.05 05.52 05.77 05.42

05.83 06.47 05.92 06.02 05.91

05.80 05.94 06.21 06.44 06.11

DM

05.63 04.83 05.12 05.34 05.01

04.68 04.75 05.33 05.11 05.57

02.92 03 .43 02 .99 02 .96 03 .49

05.14 05.24 04.84 05.37 05.16

04.87 04.72 04.99 05.03 04.90

Du

04.21 04.06 04.00 04.11 03.77

04.81 04.23 05.12 04.86 05.69

04.70 05.43 04.88 04.64 04.98

03 .83 03 .79 03 .14 03 .34 03 .34

04.90 04.63 05.19 05.07 05.01

DK

05.23 04.52 04.78 04.98 04.92

05.33 04.86 05.51 05.21 06.00

04.71 04.82 04.51 04.68 04.77

05.27 05.44 05.12 05.15 05.15

03 .35 02 .91 03 .28 0 5 . 2 3 0 3 . 3 5

6 to text 10 are Bengali texts, text 11 to text 15 are Manipuri texts, text 16 to text 20 are Urdu texts and text 21 to text 25 are Kashmiri texts. Table 2 gives a summary of a very large number of experiments and tests reflecting the over all success of the Identifier to be almost 100% (depicted through bar-chart in Fig. 1). After trying the algorithm with text length 400, efforts were made to find the optimal length required for the algorithm so that a good success rate is achieved. For this purpose the algorithm has been tried with text length 200 and 150 chars also. The summary of results with text length 200 and 150 chars has been shown in Tables 3 and 4 respectively (depicted through bar-charts in Fig. 2 and Fig. 4 respectively). Even in these cases success rate achieved is very good (88% to 98%).

Pratibha Yadav, Girish Mishra and P. K. Saxena 115

Text Length - 400 chars

Hnd Bengali Maripuri Urdu Kashmiri

• # Correctly Classified

Fig. 1. % Success for Language Identification (for texts with 400 chars).

Text Length - 200 chars

Hnd Bengali Manipuri Urdu Kashmiri

• # C orrectly C lassified

Fig. 2. % Success for Language Identification (for texts with 200 chars).

5. C o n c l u s i o n s

A five-class classification problem has been addressed using F P R . The advantage of the approach is tha t the prototypes of each of the five languages are constructed once and then any single given unknown text is identified if it belongs to any of these five languages. It is independent of dictionary and processes texts even when word-break-up is not known. The

identifier developed is working well with a very high

success rate (above 99%) for plaintexts of these five

languages for text length of 400 chars. Success ra te is

above 95% for text length of 200 chars and is slightly

less (above 88%) for text length of 150 chars. These

are depicted through bar-chart in the following Fig

ure:

Hindi Bangla Mampun Urdu Kashmiri

• 400 chars • 200 chars a 150 chars

Fig. 3. Comparison of success(%) for different text lengths considered.

Text Length -150 chars

Hnd Bengali Maripuri Urdu Kashmiri

• # Correctly Classified

Fig. 4. % Success for Language Identification (for texts with 150 chars).

References

1. L. A. Zadeh; Fuzzy Sets. Inform, and control 8, pp338-353, 1965.

2. Bezdek C. J. and Pal, S. K.; Fuzzy models for Pattern Recognition, IEEE Press, 1992.

3. C. J. Bezdek; Pattern Recognition with Fuzzy Objective Function Algorithms, Plenum, New York, 1981.

116 Identification of Indian Languages in Romanized Form

Table 2. Success rate of Language Identification (Text length = 400 chars)

Language

Hindi Bengali Manipuri Urdu Kashmiri

#Texts

8139 7502 7584 976 396

Hindi

8102 22 0 1 0

Bengali

10 7457

13 0 0

Manipuri

0 2

7570 0 1

Urdu

27 21 1

975 0

Kashmiri

0 0 0 0

395

Success Rate (%)

99.54 99.40 99.82 99.90 99.75

Table 3. Success rate of Language Identification (Text length = 200 chars)

Language

Hindi Bengali Manipuri Urdu Kashmiri

#Texts

16278 15004 15167 1952 791

Hindi

15449 398

3 25 8

Bengali

108 14402

107 42 0

Manipuri

26 32

15049 12 0

Urdu

769 183 11

1872 0

Kashmiri

20 1 0 0

783

Success Rate (%)

94.91 95.99 99.22 95.99 98.99

Table 4. Success rate of Language Identification (Text length = 150 chars)

Language

Hindi Bengali Manipuri Urdu Kashmiri

#Texts

21704 20006 20223 2602 1054

Hindi

19279 733 49 74 10

Bengali

288 18715

197 63 5

Manipuri

105 97

19906 35 1

Urdu

2183 496 78

2429 0

Kashmiri

95 9 0 0

1038

Success Rate (%)

88.83 93.55 98.43 93.35 98.48

4. H. J. Zimmermann; Fuzzy Set Theory and its Applications, 4th Ed., Kluwer Academic Publisher, Bon-ston/ Dordrecht/London, 2001.

5. G. J. Klir and B. Yuan; Fuzzy Sets and Fuzzy Logic: Theory and Applications, Prentice Hall, 1997.

6. P. K. Saxena and Uma Gupta; Fuzzy Language Identifier, Proceedings of the National Seminar on Cryp-tology (NSCR), Delhi, pp. D-ll to D-20 , 1998.

7. Kenneth R. Beesley; Language Identifier: A Computer Program for Automatic Natural-Language Identification of On-line Text, Language of Crossroads: Proceedings of the 29th Annual Conference of the American Translators Association, pp. 47-54, 12-16 Oct,1988.

8. Saxena P. K., Yadav P.and Sarvjeet K.; Fuzzy Sets in

10.

11.

Cryptology, Proceedings INFOSEC 1994, Bangalore, pp. 13 to 33 , 29-30 Oct,1994. Navneet Gaba, Sarvjeet Kaur and P. K. Saxena; Identification of Encryption Schemes for Romanized Indian Languages, Proceedings of ICAPR 2003, pp. 164 to 168 , 2003. Verma N., Khan S. S. and Shrikant; Statistical Feature Extraction to Discriminate Various Languages: Plain and Crypt, Proceedings of National Conference on Information Security (NCIS), pp. 1 to 10 , Jan, 2003. Yadav P. and Saxena P. K.; Identification of Regional Languages - A Fuzzy Theoretic Approach, Proceedings of NCIS, New Delhi pp. 11 to 19 , Jan, 2003.

117

Onl ine B a n g l a H a n d w r i t i n g R e c o g n i t i o n S y s t e m

K. Roy

Dept. of Computer Science West Bengal University of Technology

BF 142, Saltlake, Kolkata-700 064, India

N. Sharma, T. Pal and U. Pal

Computer Vision and Pattern Recognition Unit Indian Statistical Institute

Kolkata- 700 108, India

Handwriting recognition is a difficult task because of the variability involved in the writing styles of different individuals. This paper presents a scheme for the online handwriting recognition of Bangla script. Online handwriting recognition refers to the problem of interpretation of handwriting input captured as a stream of pen positions using a digitizer or other pen position sensor. The sequential and dynamical information obtained from the pen movements on the writing pads are used as features in our proposed scheme. These features are then fed to the quadratic classifier for recognition. We tested our system on 2500 Bangla numeral data and 12500 Bangla character data and obtained 98.42% accuracy on numeral data and 91.13% accuracy on character data from the proposed system.

Keywords: Online Recognition, Indian Script, Bangla, Modified quadratic discriminant function

1. Introduction

Data entry using pen-based devices is gaining popularity in recent times. This is so because machines are getting smaller in size and keyboards are becoming more difficult to use in these smaller devices. Also, data entry for scripts having large alphabet size is difficult using keyboard. Moreover, there is an attempt to mimic the pen and paper metaphor by automatic processing of online characters. However, wide variation of human writing style makes online handwriting recognition a challenging pattern recognition problem.

Work on online character recognition started gaining momentum about forty years ago. Numerous approaches have been proposed in the literature [1-5] and the existing approaches can be grouped into three classes namely, (i) structural analysis methods where each character is classified by its stroke structures, (ii) statistical approaches where various features extracted from character strokes are matched against a set of templates using statistical tools and (iii) motor function models that explicitly use trajectory information where the time evaluation of the pen co-ordinates plays an important role.

Many techniques are available for online recognition of English,Arabic, Japanese and Chinese [1-4, 6-9] characters but there are only a few pieces

of work [10-13] available towards Indian characters although India is a multi-lingual and multi-script country. Connell et al. [11] presented a preliminary study on online Devnagari character recognition. Joshi et al. [12] also proposed a work on Devnagari online character recognition. Later Joshi et al. [13] proposed an elastic matching based scheme for online recognition of Tamil character recognition. Although there are some work towards online recognition of Devnagari and Tamil scripts but the online recognition work towards other Indian languages is very few. In this paper we propose a system for the online recognition of Bangla character. Recognition of Indian character is very difficult with compare to English because of its shape variability of the characters as well as larger number of character classes. See Figure 1 where samples of four Bangla characters are shown to get an idea of handwriting variability.

There are twelve scripts in India and in most of these scripts the number of alphabets (basic and compound characters) is more than 250, which makes keyboard design and subsequent data entry a difficult job. Hence, online recognition of such scripts has a commercial demand. Although a number of studies [14-16] have been done for offline recognition of a few printed Indian scripts like Devnagari, Bangla, Gurumukhi, Oriya, etc. with

118 Online Bangla Handwriting Recognition System

commercial level accuracy, but to the best of our knowledge no system is commercially available for online recognition of any Indian script. In this paper we propose a scheme for online Bangla handwritten character recognition and the scheme is robust against stroke connections as well as shape variation while maintaining reasonable robustness against stroke order-variations. A quadratic classifier is used here for recognition purpose.

The rest of the paper is organized as follows. In Section 2 we discuss about the Bangla language and data collection. Feature extraction process is presented in Section 3. Section 4 details the classifier used for recognition. The experimental results are discussed in Section 5. Finally, conclusion of the paper is given in Section 6.

v£ <&

%

&

3> *

•ssr

vC

"*l -zX &i *t

• 5 *

3? ^ • »

" < ^

• ^

<F

a © a 0 a

<L Fig. 1. Examples of some Bangla online characters. First three columns show samples of handwritten characters and last column shows samples of a numeral.

2. Bangla Language and online data collection

Bangla, the second most popular language in India and the fifth most popular language in the world, is an ancient Indo-Aryans language. About 200 million people in the eastern part of Indian subcontinent speak in this language. Bangla script alphabets are used in texts of Bangla, Assamese and Manipuri languages. Also, Bangla is the national language of Bangladesh.

The alphabet of the modern Bangla script consists of 11 vowels and 40 consonants. These characters are called as basic characters. Writing style in Bangla is from left to right and the concept

of upper/lower case is absent in this script. It can be seen that most of the characters of Bangla have a horizontal line (Matra) at the upper part. From a statistical analysis we notice that the probability that a Bangla word will have horizontal line is 0.994 [14].

In Bangla script a vowel following a consonant takes a modified shape. Depending on the vowel, its modified shape is placed at the left, right, both left and right, or bottom of the consonant. These modified shapes are called modified characters. A consonant or a vowel following a consonant sometimes takes a compound orthographic shape, which we call as compound character. Compound characters can be combinations of two consonants as well as a consonant and a vowel. Compounding of three or four characters also exists in Bangla. There are about 280 compound characters in Bangla [15]. In this paper we consider the recognition of Bangla basic characters.

To get an idea of Bangla basic characters and their variability in handwriting, a set of handwritten Bangla basic characters are shown in Figure 2. Main difficulty of Bangla character recognition is shape similarity, stroke size and the order variation of different strokes. By stroke we mean the set of points obtained between a pen down and pen up. From the statistical analysis on our dataset we found that the minimum (maximum) number of stroke used to write a Bangla character is 1 (4). The average number of stroke per character is 2.2. We have also seen that the characters W, ' Q' and '*>' are mostly written by single stroke whereas the character 'w' is written by almost all writer by 4 strokes. It also found that the characters ' ^ , '& , '^ ' , '^ ' , '?', 'S',^' etc. are always written by 2 strokes.

Online recognition of these characters imposes several problems like stroke-number, stroke connection and shape variations. Most characters are composed of multiple strokes. Another difficult problem involves in the stroke-order variations. Writing of different stroke sequences of a character is not similar for all persons. During handwriting, some people give upper stroke of a character before giving a lower stroke of that character. Whereas to write the same character, some people give lower stroke before giving the upper stroke of that character. These stoke order variations complicates the development procedure of an online recognition system.

K. Roy, N. Sharma, T. Pal and U. Pal 119

\3f vxi *-£^> © • * sj»3>2 *5»

T *

v!r

<?r x

*4T ST

^ ^ C

V ar f*

*F 3r^r y 5

"3T

-* \

^ ~

<rT SS 1°

• B O "

^r T> *

\ &

>£ *5 0» *J-« r

•&r •»r >r T3"

0

0

• #

Fig. 2. Examples of handwritten Bangla basic characters.

To illustrate this stoke order variation in Bangla script, Figure 3 shows a Bangla character that contains four different strokes. The left-most column shows the first stroke and this stroke is same for all the three samples of three different writers. Stroke-order varies from the second column onwards and the final (complete) character is shown in the rightmost column. From the 2nd column of Figure 3, it can be noted that the three samples have different shape. This is because of stroke order variation of the writers. For upper sample of the second column of Figure 3, the writer has given the upper stroke as second stroke. For middle sample of the second column the writer has given the lower stroke as second stroke. For lower sample, the writer has given the middle stroke as second stroke. Similar situation also occurs in 3rd column for Figure 3. Matra is a very common feature in Bangla and its length varies from writer to writer. It is seen that the presence or absence of Matra is the main difference between two characters. For example the character ' ^ ' is a basic character which has Matra but ''0' is a Bangla numeral and it does not have Matra.

For online data collection, the sampling rate of the signal is considered fixed for all the samples of all the classes of character. Thus the number of points M in the series of co-ordinates for a particular sample is not fixed and depends on the time taken to write the sample on the pad. As the number of points in

actual trace of the characters are generally large and varies greatly due to high variation in writing speed, a fixed lesser number of points, regularly spaced in time are selected for further processing.

The digitizer output is represented in the format of pj € R2X{0,1} i = 1..M, where pi is the pen position having x-coordinate (n) and y-coordinate (yi) and M is the total number of sample points. Let (pi) and (pj) be two consecutive pen points. We retain both of these two consecutive pen points (pi) and (pj) if the following condition is satisfied:

x2 + y2 > m2 (1)

where x = Xi — Xj and y = yi—yj. The parameter m is empirically chosen. We have set m equal to zero; in Equation 1 to removes all consecutive repeated points.

&

-~b

• &

~&

^

$

^

*5 %

©" 3> c, &

Fig. 3. Example of different stroke order for a character having four strokes.

Analyzing a total of 15000 Bangla characters we found that, for writing Bangla characters, the number of sample points (M) varies from 14 (for the character ^5) to 176 (for the character ^F) points. The average number of sample points in a Bangla character is 72. We also computed the average number of sample points in each character class. We noted that the character class (5f) has the maximum number of sample points and its average value is 113. The character class '&' has the minimum number (46) of sample points.

3. Feature extraction

Any online feature is very much sensitive to writing stroke sequence and size variation. Also, in Bangla Matra creates a lot of problem in online recognition. To overcome it, we detect and remove the Matra present in the characters. As Matra of Bangla script

120 Online Bangla Handwriting Recognition System

is a digital straight line lies on the upper part of a character. A stroke is called Matra if : (a) The ratio of distance of 1st and last point of the stroke to sum of individual distances between constituent points of the stroke is less than 1.5. (b) The ratio of the stroke height to the character height [max(yi) — min(yi)Vi\ is less than 0.35 and all the points of the stroke lie in upper side (we consider top 40% height of the character as upper side) of the character.

These thresholds are decided based on statistical analysis of our dataset. If more than one stroke satisfies the above criteria then higher value of condition 1 is selected.

The features calculated based on Matra are (1) The ratio of average value of x coordinate of the selected stroke to the length of the character, (2) The ratio of average value of y coordinate of the selected stroke to the width of the character, (3) Ratio of the length (L = J2h i = 1—M where k = (x2 + j /2) , x = Xi - xi+i and y = yt - yi+i) of the stroke to the length of the character, (4) Ratio of the area of the stroke to the character and (5) Ratio of aspect ratio of the stroke to that of character.

A total of 5 features as discussed above are calculated based on Matra. After feature detection from Matra, it is then removed from the character and the rest of the points of the characters are first normalized. The normalization is done in two stages. First the points are re-sampled to a fixed number points (N) and then they are converted from equal time sample to equal distant points. For example see Figure 4. We have studied several local features, which include a normalized representation of the co-ordinates, a representation of the tangent slope angle, a normalized curvature, the ratio of tangents, etc.

The processed character is transformed into a sequence t = [£I,..,£JV] of feature vectors U = (Ui,U2,ti3)T. Here (1) t u = (Xi-/zx)/<7y and t i 2 = (yj-/xy)/<7y are the pen co-ordinates normalized by the sample mean fj, = ji 5Zi=i Pi a n d standard deviation, ay =

* / ^ r j J2 (A*I/ — Vi)2 °f the character's sample points,

and (2) ti3 = arg((xi+i-Xi-i) + j * (j/i+i - Jft-i)), with

j 2 = — 1 and "arg " the phase of the complex number above, is an approximation of the tangent slope angle at point i.

Thus finally, a feature vector sequence is defined as t = [ti,..,tff], each vector of it as U = (tn,ti2,ti3)T is obtained. Here we consider N = 50. So a total of 155 (50 X 3 [3 for each point] + 5 [5 features based on Matra]) features are used in our experiment.

..../

r"\ ••:-' 1 • /

;: . •'.. ; ; • • • '

^

fV 1 IA

/ / ^ / /

Fig. 4. Feature extraction from a sample of character is shown, (a) Original image, (b) its normalized point used as feature (mapped into 50 points), (c) the normalized character.

4. The Classifier

Based on these 155 dimensional features, recognition of characters in quadratic classifier [17] is carried out by using the following discriminant function:

g(X) = (N + N0-n-l)ln[l + j^[\\X-M\\2

i=l At T N &

k

+ X>(A< + ^ 2 ) - (2) i=l

where X is the feature vector of an input character; M is a mean vector of samples; <j>J is the ith eigen vector of the sample covariance matrix; A* is the ith eigen value of the sample covariance matrix; n is the feature size; a2 is the initial estimation of a variance; N is the number of learning samples; and AT0 is a confidence constant for a , and 7V0 is considered as 3N/7 for 155 dimensional feature. We do not use all eigen values and their respective eigen vectors for the classification. We sort the eigen values in descending order and take first 70 eigen values and their respective eigen vectors for classification. Rejection in the system is done if for a character the difference of 1st and 2nd value of g(X) is smaller than a threshold.

K. Roy, N. Sharma, T. Pal and U. Pal 121

5. Results and Discussion

The experimental evaluation of the above techniques was carried out using isolated Bangla character and numerals (digits). The data was collected from people of different background both by using mouse as well as Wacom tablet. The mouse is used as an input medium to facilitate the user with the flexibility of using this common input medium. The data obtained by mouse is in general poorer than Wacom tablet. Our another aim to use this mouse as one of the input devices is to test our system on these poorer data. If we get encouraging results from the data capture by mouse then there is no need to use any additional hardware like Wacom tablet for data capture. A total of 15,000 characters (2500 digits and rest are characters) are collected for the experiment. Out of them 66.7% of the characters (digits) are used for the training of the classifier for the present work and rest is used for the testing purpose. The recognition accuracy obtained from our classifier is shown in Table 1.

Table 1. Recognition results on Bangla character and numeral (here rejection is not considered).

Data Character Numeral

Recognition rate 91.13% 98.42%

Error rate 8.87% 1.58%

From the experiments we noted that the overall recognition accuracy of the proposed scheme was 91.13% for Bangla characters and 98.42% accuracy for Bangla numerals when rejection is not considered. Accuracy of 96.23% and 99.58% is obtained for Bangla characters and numerals respectively, if we consider first two top choices of the recognition results. The detail recognition results are given in Table 2.

Table 2. Recognition results based on different choices from top.

Top choices

1 2 3 4 5

Accuracy Character

91.13% 96.23% 97.68% 98.26% 98.39%

Numeral 98.42% 99.58% 99.72% 99.72% 99.86%

We also analyzed the error versus rejection rate

of the classifier and the results are presented in Table 3. Table 3 depicts that for Bangla numeral, 1.57% error occurs at a rejection of 0.13% and only 0.85% error occurs when rejection was 1.10%. Table 3 also depicts that for Bangla character 6.56% error occurs at a rejection of 3.79% and only 2.64% error occurs when rejection was 17.27%.

Table 3. Rejection versus error rate obtained for the Characters and Numerals.

Characters Rejection

3.79% 7.44% 12.28% 17.27%

Error 6.56% 5.12% 3.69% 2.64%

Numerals Rejection

0.13% 0.61% 0.85% 1.10%

Error 1.57% 1.10% 0.96% 0.85%

Recognition accuracy of about 96% (99.6%) is obtained in character (numeral) recognition, if first two choices are considered. These different choices in recognition accuracy will be very helpful in designing a complete recognition system on word/sentence level where we will be able to take the help of dictionary or we can predict alternate results. From the experiment we noted most of the errors occurred because of similar shape characters. Maximum error occurred between ' T and ' T and it is noted that about 1.067% cases they mis-recognized one as the other. Main difference between the above two characters is that there is a small loop in left bottom side of one of the characters. Some times during handwriting people do not give this loop and hence mis-recognition occurs. The next erroneous character pair in Bangla is '&', and 'l»'. They mis-recognized in 0.93% cases. The main difference between the above characters is also the presence of loop in the lower half of the character and sometimes people do not give this loop during handwriting. The five most erroneous character pairs obtained from our experiment are shown in Figure 5. Here erroneous Bangla character pairs are shown in the first row. Percentage of error as well as number of times these character pairs mis-recognized between them are given in the second row of this Figure. For example character '*T is mis-recognized as ' T in 19 cases, and the characters ' T is mis-recognized as ' T in 11 cases. So they mis-recognized between them in 30 (19+11) cases and their percentage of error is 1.067%.

122 Online Bangla Handwriting Recognition System

-V&J" 1.067%

30(19+11)

T? ~B SYT (?T <5 \& 0.93%

26(16+10) 0.32% 9(8+1)

0.14% 4 (3+1)

*4~ n 0.14% 4 (3+1)

Fig. 5. Examples of some erroneous characters.

upper part . We tested the proposed system on 15000

samples and obtained encouraging results. Not much

work has been done towards the online recognition

of Indian scripts in general and Bangla in part icular .

So this work will be helpful for the research towards

online recognition of other Indian scripts.

We compared our work with the work of Garain

et. al. [10]. For numeral recognition they have used

a dataset of 400 numerals for their work. They

captured their da t a by Wacom tablet and did not

used mouse for da ta collection. For feature extraction

they have used angular variation in 8 directional

code and Euclidian distance of each sample point.

Nearest neighbor classifier has been used by them

for recognition. A comparative result on numeral

recognition is shown in Table 4. Garain et. al.

obtained 97.43% accuracy on training da ta of size

400 and they did not report any test result. We

obtained 99.86% accuracy on training da ta and

98.42% on test data .

Table 4. Comparison result on numeral recognition.

Method

Garain et. al. [10] Proposed

Data size 400

2500

Recognition Train

97.43% 99.86%

Test Not reported

98.42 %

Comparative results of character recognition is

shown in Table 5. Garain et. al. [10] have used a

dataset of 2440 character collected using Wacom

tablet for their work and obtained 96.34% accuracy

on training data . We obtained 98.73% accuracy on

training da t a and 91.13% on test da ta .

Table 5. Comparison result on character recognition.

Method

Garain et. al. [10] Proposed

Data size 2440 12500

Recognition Train

96.34% 98.73%

Test Not reported

91.13 %

6. C on c lu s ion

This paper presents a quadratic classifier based

system for the recognition of online Bangla

handwrit ing. A modified and robust feature

extraction technique is proposed here which can be

used for the scripts with Matra/Shirorekha at the

References

[1] R. Plamondon and S. N. Srihari, IEEE PAMI 22, 63 (2000).

[2] C. C. T. C. Y. Suen and T. Wakahara, IEEE PAMI 12, 179 (1990).

[3] E. J. Bellagarda, J. R. Bellagarda, D. Nahamoo and N. S. Nathan, A probabilistic framework for online handwriting recognition, in Proc. of 3rd IWFHR, 1993.

[4] I. Guyon, M. Schenkel and J. Denker, Overview and synthesis of on-line cursive handwriting recognition techniques, in Handbook of Character Recognition and Document Image Analysis, 1997.

[5] CEDAR, Penman: Handwritten Text Recognition Project Description.

[6] C. Bahlmann and H. Burkhardt, IEEE PAMI 26, 1 (2004).

[7] S. D. Connell and A. K. Jain, PR 34, 1 (2001). [8] S. Jaeger, C. L. Liu and M. Nakagawa, IJDAR 6,

75 (2003). [9] C. L. Liu, S. Jaeger and M. Nakagawa, IEEE PAMI

26, 198 (2004). [10] U. Garain, B. B. Chaudhuri and T. Pal, Online

handwritten indian script recognition: A human motor function based framework, in Proc. of 16th ICPR, 2002.

[11] S. D. Connell, R. M. K. Sinha and A. K. Jain, Recognition of unconstrained online devnagari characters, in Proc. of 15th ICPR, 2000.

[12] N. Joshi, G. Sita, A. G. Ramakrishnan, V. Deepu and S. Madhvanath, Machine recognition of online handwritten devanagari characters, in Proc. of 8th ICDAR, 2005.

[13] N. Joshi, G. Sita, A. G. Ramakrishnan and S. Madhvanath, Elastic matching algorithms for online tamil character recognition, in Proc. ICONIP, 2004.

[14] B. B. Chaudhuri and U. Pal, PR 3 1 , 531 (1998). [15] B. B. Chaudhuri and U. Pal, An ocr system to read

two indian language .scripts: Bangla and Devnagari, in Proc. of 4th ICDAR, 1997.

[16] V. Bansal and R. M. K. Sinha, On how to describe shapes of Devanagari characters and use them for recognition, in Proc. 5th ICDAR, 1999.

[17] T. Wakabayashi, S. Tsuruoka, F. Kimura and Y. Miyake, Systems and Computers in Japan 26, 35 (1995).

123

Oriya Off-Line Handwritten Character Recognition

U. Pal, N. Sharma

Computer Vision and Pattern Recognition Unit Indian Statistical Institute

Kolkata-108, INDIA, E-mail:umapada@isical. ac. in

F. Kimura

Graduate School of Engineering Mie University

1577 Kurimamachiya-cho, TSU, Mie 514-8504, Japan

Recognition of handwritten characters is difficult because of variability involved in the writing style of different individuals. This paper deals with recognition of off-line Oriya handwritten characters using quadratic classifier based on the features obtained mainly from directional chain code information. Here, at first, the bounding box of a character is segmented into blocks and directional chain code features are computed in each of these blocks. Next, these blocks are down sampled using Gaussian filter for recognition. Finally, chain code features obtained from down sampled blocks are fed to the quadratic classifier for recognition. We used two sets of feature vectors (one feature vector has 64 dimension and the other has 400 dimension) and their corresponding results obtained from the classifier are reported. We tested our system on 9556 Oriya off-line handwritten characters and we obtained 91.11% accuracy from the proposed recognition system when 400 dimensional feature vector was considered. We have used five-fold cross-validation technique for result computation.

Keywords: Oriya script, Handwritten character recognit

1. Introduction

Recognition of handwritten characters has been a popular research area for many years because of its various application potentials. Some of its potential application areas are postal automation, bank cheque processing, automatic data entry, etc. There are many pieces of work towards handwritten recognition of Roman, Japanese, Chinese and Arabic scripts, and various approaches have been proposed by the researchers towards handwritten character recognition.1'2 One of the widely used approaches is based on Neural Network.1 Here the network architecture is, at first, trained by a set of training data and then the trained networks classify the input. Some researches used structural approach, where each pattern class is defined by structural description and the recognition is performed according to structural similarities.3 Statistical approach is also applied to character recognition.4 It is insensitive to pattern noise and distortion but modeling of statistical information is a tedious task. Combination of structural and statistical method is also used for character recognition.4 Among others, Support vector machines,5

Fourier and Wavelet description,6 Fuzzy rules,7 tolerant rough set,8 are reported in the literatures.

Indian script, Document analysis.

In this paper, we propose a system for the recognition of unconstrained off-line Oriya handwritten characters. Although some work have been done towards the recognition of Oriya printed characters9 '10

and handwritten Oriya numerals11 but to the best of our knowledge there is no work on Oriya handwritten characters and this is the first work of its kind.

In this paper a quadratic classifier based scheme is proposed for unconstrained off-line Oriya handwritten character recognition. In the proposed scheme, at first, the bounding box of a character is segmented into blocks and directional features are computed in each of these blocks. Next, these blocks are down sampled using Gaussian filter for recognition. Finally, chain code features obtained from down sampled blocks are fed to the quadratic classifier for recognition.

Rest of the paper is organized as follows. In Section 2 properties of Oriya language and data collection for the present work are discussed. Feature extractions procedure is reported in Section 3. In Section 4, we briefly explain the classifier used for the recognition purpose. The experimental results are discussed in Section 5. Finally conclusion is given in Section 6.

124 Oriya Off-Line Handwritten Character Recognition

2. Oriya Language and data collection

India is a multi-lingual and multi-script country and there are about 22 official languages. Oriya is one of the official languages in India and it is mainly used in the Indian state of Orissa. More than 31 million people in the eastern part of Indian subcontinent speak in this language. The Oriya script, by which Oriya language is written, is developed from the Kalinga script, one of the many descendents of the Brahmi script of ancient India.

The alphabet of the modern Oriya script consists of 11 vowels and 41 consonants. These characters are called basic characters and the basic characters of Oriya script are shown in Fig.l. Out of these 52 basic characters in Oriya, two characters are equal in shape. For recognition, we consider these two characters as one class. Writing style of Oriya script is from left to right and as in other Indian script, concept of upper/lower case is absent in Oriya script. From Fig.l it can be noted that most of the characters in Oriya is circular in nature and there is no horizontal line (like Matra/Shirorekha in De-vnagari script) in the characters of this script. In Oriya script a vowel following a consonant takes a modified shape. Depending on the vowel, its modified shape is placed at the left, right, both left and right, or bottom of the consonant. These modified shapes are called modifiers or matra. A consonant or a vowel following a consonant sometimes takes a compound orthographic shape, which we call as compound character. Compound characters can be combinations of two consonants as well as a consonant and a vowel. There are more than 200 compound characters in Oriya script10 and in this paper we consider the recognition of off-line handwritten Oriya basic characters.

Main difficulty of any recognition system is shape similarity. In Oriya there are many characters having similar in shape. Examples of some groups of similar shaped characters are shown in Fig.2. From the figure it can be seen that shapes of two or more characters of a group is very similar and such shape similarity makes the recognition system more complex to get higher recognition rate. Data collection for the present work has been done from different individuals of various professionals. We have collected 9556 data (at least 180 samples of each character class) for the experiment of the proposed work. We used a flatbed scanner for digitization. Digitized im

ages are in gray tone with 300 dpi and stored as TIF Format. For binaraization of the data we used the Otsu12 method and have converted the data into two-tone (0 and 1) images (Here ' 1 ' represents object point and '0' represents background point).

2 l 2 l l « f i Q Q € ) ^ ( 3 ^ l ( 3 l ( a )

<S\QQ @ £1 @ ® £J Q § o 8 ~ ( b )

Fig. 1. Basic characters of Oriya script, (a)Vowels and (b) Consonants

€161

fi?Q

Q Q Q

QQ

0 ©

QQ Fig. 2. Examples of some similar shaped characters

3. Feature selection

Histograms of direction chain code of the contour points of the characters are used as feature for recognition.13'14 We have used two sets of features for Oriya character recognition. For high-speed recognition we use 64 dimensional features, and for high accuracy recognition we use 400 dimensional features. The feature extraction procedure is described below.

3

4* *

5

r

j

t

1 I 1 k 1 J*

r . *

> | 7

X

x P X

i

X 1

Fig. 3. (a) For a point P and its four neighbors are shown by X,- (b) For a point P the direction codes for its eight neighboring points are shown

U. Pal, N. Sharma and F. Kimura 125

Q Bounding

box

Contour

7X7 block

(b)

Extraction

and

segmentation

Ch»n Cose

1

i

4

Court

2

T" 2

0 % I i f 1 1 •.

Cham code (d)

*• -t *

- f «. - w ™ ^ . H M ,fa «

(fl 11

P4

a -6 to 0>\ T—1

4H

0 4-i

<tj

u t. w

4 3 2

7 S

^ 1 2 J e )

./*tv

Gaussian Filter

: 1(f) Down sampling to 64 dim. feature

Fig. 4. Pictorial representation of the 64 dimensional feature extraction process for an Oriya character, (a) Two tone image of a Oriya character 'KA' (b) Bounding box of the character. (c) Contour of the character shown in black color and the bounding box is segmented into 7 X 7 blocks, (d) Chain code of a block shown in zoomed version and its chain code count is also shown in the neighboring table (e) 196 dimensional chain code features obtained from 7 X 7 block, (f) 64 dimensional features obtained after down sampling by a Gaussian filter.

Given a two-tone image, we first find contour points of the image by the following algorithm. For all object point in the image, consider a 3 x 3 window surrounded to the object point. If any one of the four neighboring points (as shown in Fig.3(a)) is a background point then this object point (P) is considered as a contour point. Otherwise it is a non-contour point.

3.1. 64 Dimensional feature extraction:

At first we compute the bounding box (minimum rectangle containing the character) of an input character. This bounding box is then divided into 7 x 7 blocks (as shown in Fig.4(c)). In each of these blocks the direction chain code for each contour point is noted and frequency of direction codes is computed. Here we use chain code of four directions only [directions 0 (horizontal), 1 (45 degree slanted), 2 (vertical) and 3 (135 degree slanted)]. See Fig.3(b) for illustration of four chain code directions. We assume chain code of direction 0 and 4 are same. Also, we assume direction 1 and 5, 2 and 6, 2 and 7 are same.

Thus, in each block, we get an array of four integer values representing the frequencies of chain code in these four directions. These frequencies are used as feature. Histogram of the values of these four direction codes in each block of an Oriya handwritten character ('KA') is shown in Fig.4(e). Thus, for 7 x 7 blocks we get 7 x 7 x 4 =196 features. To reduce the dimension of feature vector, after the histogram calculation in 7 x 7 blocks, the blocks are down sampled into 4 x 4 blocks by a Gaussian filter. As a result we have 4 x 4 x 4 (=64) features for recognition. Histogram of these values of all the four directions obtained after down sampling is shown in Fig.4(f). In this feature calculation we perform a height normalization and it is simply done by multiplying each feature vector component by the ratio of the standard height to the actual height of the character. For our experiment, we consider 76 as the standard height.

3.2. 400 Dimensional feature extraction:

To obtain 400 dimensional features we apply the following steps.

Step 1: At first size normalization of the input

126 Oriya Off-Line Handwritten Character Recognition

binary image is done. Here we normalize the image into 126 x 126 pixels.

Step 2: The input binary image is then converted into a gray-scale image applying a 2 x 2 mean filtering 5 times.

Step 3: The gray-scale image is normalized so that the mean gray scale becomes zero with maximum value 1.

Step 4: Normalized image is then segmented into 9 x 9 blocks.

Step 5: A Roberts filter is then applied on the image to obtain gradient image. The arc tangent of the gradient (strength of gradient) is quantised into 16 directions and the strength of the gradient is accumulated with each of the quantized direction. By strength of Gradient ( f(x,y) ) we mean f(x,y)= ^(Au)2 + (At;)2

and by direction of gradient 0(x, y) we mean 9(x,y)= t a n - 1 ^ ,

where, Au = g(x+l,y+l) - g(x,y) and , Av = g(x+l,y) - g(x,y+l) and, g(x,y) is a gray scale at (x, y) point.

Step 6: Histograms of the values of 16 quantized directions are computed in each of 9 x 9 blocks.

Step 7: 9 x 9 blocks is down sampled into 5 x 5 by a Gaussian filter. Thus, we get 5 x 5 x 16 = 400 dimensional feature.

4. Character Recognition Classifier

Recognition of characters in quadratic classifier13 is carried out by using the following discriminant function:

g(X) = (N + N0~n-1) ln[l + j^[\\X - M| |2

i = l M + N a

+ $ > ( A i + 7f(72). (1) i = l

where X is the feature vector of an input character; M is a mean vector of samples; <f>[ is the ith eigen vector of the sample covariance matrix; A; i is the ith

eigen value of the sample covariance matrix; n is the feature size; a1 is the initial estimation of a variance; N is the number of learning samples; and N0 is

a confidence constant for a , and N0 is considered as 3N/7 for 64 dimensional feature, and N0 = N/9 for 400 dimensional feature. We do not use all the eigen values and their respective eigen vectors for the classification. For 64 dimensional case, we sort the eigen values in descending order and take first 20 eigen values and their respective eigen vectors for classification. For 400 dimensional case, we take first 40 sorted eigen values and their respective eigen vectors for the classification.

Here, at first, for high speed recognition we use 64 dimensional features in the quadratic classifier. Next, to get higher accuracy we use 400 dimensional features. Rejection criteria of the proposed system depends on the difference of 1st and 2nd value of g(X).

5. Results and discussion

We applied our proposed scheme on 9556 characters obtained from individuals of different sections of population (like student, teacher, government employee, businessmen etc.) of Orissa state. From the experiment with 64 dimensional features we note that the overall recognition accuracy of the proposed scheme is 84.58%, when zero rejection rate is considered. From the experiment with 400 dimensional features we note that the overall recognition accuracy of the proposed scheme is 91.11%, when rejection rate is zero. We also notice that 92.61% (96.52%) accuracy is obtained if we consider first two top choices of the recognition results in 64 (400) dimensional features. The detail recognition results with different choices (from top) are given in Table 1. For result computation, we have used five-fold cross validation scheme. Here we divided the database into 5 subsets and testing is done on each subset using rest of the subsets for learning. The recognition rates for all the subsets are averaged to get accuracy.

Table 1. Recognition results of Oriya handwritten characters based on different choices from top(at 0% rejection).

No. of Top Choices

1 2 3 4 5

64 Dimensional Feature

84.58% 92.61% 95.98% 97.29% 98.13%

400 Dimensional Feature

91.11% 96.52% 98.19% 98.92% 99.15%

U. Pal, N. Sharma and F. Kimura 127

From the experiment we noted that in both 64 and 400 dimensional feature vector the character 'THA'(O) got the highest recognition accuracy (99.21% in 64 dimension and 100% in 400 dimension). Also from the experiment we noted that both in 64 and 400 dimensional feature vectors the character 'BHA'(W) got the lowest recognition accuracy (47.37% in 64 dimension and 61.05% in 400 dimension). This lower accuracy is due to the similarity of

the character 'BHA'(Q) with the characters 'U'(W)

and 'UU' (Q). Although from the printed version of these characters we can see some differences in the shape of these three characters, but during writing people do not give much attention and hence these characters look very similar. As a result such lower accuracy is obtained. To get an idea about the shape of some of these handwritten characters a few handwritten samples of these three characters are shown in Fig.5. From this figure it can be noted that many samples of these three characters look very similar.

Table 2. Main Confusion pair obtained from the experiment.

mis-recQjniized as the samples of the character class 'BHA'(W).

Character class

Q Q Q Q Q Q Q

Classified as

Q n

Q Q €1 ® Q ©

400 Dim. features

29.47%

23.15%

17.46%

11.64%

9.13%

7.98%

7.38%

64 Dim. features

29.54%

23.64%

20.10%

8.46%

13.44%

8.46%

13.30%

From the experiment we note main confusing character pairs and their results are given in Table 2. From the table it may be noted that both in 64 and 400 dimensional case maximum confusing character

pair is 'BHA'(® ) and ' U ' ( Q ) . About 29% samples

of the character class 'BHA'(v2«) are mis-recognized

as the samples of the character class 'U'(vJv- Also

about 23% samples of the character class 'U'

Printed characters

Q

Q

Q

Handwritten samples

<? GL(F)^<£ @<3J ^ © ^ 65 <£. *$ § < ^

Fig. 5. Hand written samples of three Oriya characters.

We also analyzed error versus rejection rate of the classifier and the details results are shown in Table 3. From the table it can be noticed that in 64 (400) dimensional feature 11.32% (5.61%) error occurs when we reject 7.02% (6.92%) characters, and only 3.42% (0.99%) error occurs when 28.09% (28.00%) characters are rejected.

There is no existing work on Oriya handwritten characters and hence we cannot compare our results.

Table 3. Rejection versus error rate obtained from the proposed classifier.

64 Dimension Rejection

0.00% 7.02% 12.22% 17.82% 28.09%

Error

15.42% 11.32% 10.0% 6.36% 3.42%

400 Dimension Rejection

0.00% 6.92% 12.42% 17.96% 28.00%

Error

8.89% 5.61% 3.77% 2.45% 0.99%

6. Conclusion

This paper deals with a scheme for the recognition of unconstrained off-line Oriya handwritten characters. To take care of variability involved in the writing style of different individuals, the features are mainly considered from the contour of the characters. We tested our scheme on 9556 data and obtained 91.11% recognition accuracy. To the best of our knowledge there is no work on Oriya handwritten character and this is the first work on Oriya handwritten character recognition.

are

128 Oriya Off-Line Handwritten Character Recognition

References

1. R. Plamondon and S. N. Srihari, IEEE Trans, on PAMI 22, 62 (2000).

2. Y. Suen, M. Berthod and S. Mori, Proceeding of the IEEE 68, 469 (1980).

3. U. Pal and B. B. Chaudhuri, Pattern Recognition 37, 1887 (2004).

4. J. Cai and Z. Q. Liu, IEEE Trans, on PAMI 21 , 263 (1999).

5. H. Byan and S. W. Lee, IJPRAI17, 459 (2003). 6. P. Wunsch and A. F. Laine, Pattern Recognition 28,

1237 (1995). 7. Z. Chi and H. Yan, Pattern Recognition 28, 56

(1995).

8. K. Kim and S. Y. Bang, IEEE Trans, on PAMI 22, 923 (2000).

9. S. Mohanty, IJPRAI 12, 1007 (1998). 10. B. B. Chaudhuri, U. Pal and M. Mitra, Sadhana 27,

23 (2002). 11. K. Roy, T. Pal, U. Pal and F. Kimura, Oriya hand

written numeral recognition system, In Proc. 8th International Conference on Document Analysis and Recognition 2005.

12. N. Otsu, IEEE Trans, on SMC 9, 62 (1979). 13. F. Kimura, K. Takashina, S. Tsuruoka and

Y. Miyake, IEEE Trans, on PAMI 9, 149 (1987). 14. F. Kimura, T.Wakabayashi, S. Tsuruoka and

Y. Miyake, Pattern Recognition 30, 1329 (1997).

Recognition of Handwritten Bangla Vowel Modifiers

S. K. Parui, U. Bhattacharya and S. K. Ghosh

Computer Vision and Pattern Recognition Unit, Indian Statistical Institute, Kolkata-700108, India E-mail: [email protected], [email protected], [email protected]

There have been a few studies on recognition of handwritten basic characters of Indian scripts including Bangla. However, for real life applications, a proper study of its vowel modifiers and their handwritten variations is essential. The only work on the recognition of handwritten vowel modifiers of an Indian scripts reported so far1 deals with vowel modifiers that appear in the lower zone of a handwritten Bangla character. In the present study, we propose a scheme for detection and recognition of vowel modifiers that appear in the middle. The study is based on shape, size and positional features extracted from the skeleton of a handwritten character image. Recognition is done on the basis of Gaussian and Dirichlet mixture distributions. The scheme has been tested on a recently developed large database of handwritten Bangla vowel modifiers and the results are satisfactory.

Keywords: Handwritten character recognition; Handwritten Bangla character recognition; Bangla vowel modifiers; Dirichlet distribution

1. Introduction

India has a large number of languages and scripts with Bangla being the second most popularly used Indian script just behind Devanagari. Also, Bangla is the official script of Bangladesh, a neighbouring country of India and it is used by more than 200 million people worldwide.

Diverse schemes for handwritten character recognition are discussed in Ref. 2. They include works on English, Chinese, Korean, Arabic and Kanji scripts. Such recognition works for Indian scripts include Refs. 3 and 4 for Devnagari characters and Refs. 5 and 6 for Bangla characters. All these recognition studies for handwritten characters of Indian scripts were done on the basis of small databases collected in laboratory environments. Moreover, research in recognition of off-line handwritten characters of Indian scripts has been limited to basic characters. Any significant research work in such an area requires availability of standard tools and data resources. The progress in handwritten character recognition research for Indian scripts particularly Bangla has not been significant due to nonavailability of standard databases of handwritten numerals/characters/words. However, situation has started to change in the recent years and a few such databases have already been reported.7

2. Background

There are 11 vowels (Fig. 1(a)) and 39 consonants (Fig. 1(b)) in modern Bangla alphabet. They are called basic characters. Unlike in English, the con

cept of upper/lower case is absent in Indian scripts. However, often a vowel in conjunction with a consonant or a cluster of consonants form different shapes and thus the total size of the alphabet of an Indian script may be as large as 500. In Bangla, a vowel other than '5T following a consonant may take a modified shape and these are called vowel modifiers (VMs). Modified shapes of 10 vowels (other than *5T) are shown in Figs. 2(a) — (j). These will henceforth (in the rest of this article) be respectively called as A, I, II, U, UU, RI, E, AI, O, and AU.

^, Sit, % t , % @, \ 3, tf, «, S (a)

^, if *t, \ «, 'rJ, % % % tf»,

% i , vs, u, «r, ^ % w, «, ^ n *p, , *i, \ *, *r, n ^ n 3, vSt U, % % ^ g, *

(b)

Fig. 1. Bangla basic characters (o) vowels; (6) consonants.

A large database of handwritten Bangla basic characters and a scheme for its recognition was reported8 recently. A similar database of Bangla VMs has also been developed.1 A major problem encountered in Bangla handwriting recognition is the presence of VMs along with basic characters. These VMs

130 Recognition of Handwritten Bangla Vowel Modifiers

1, r, i , « , *, <, &, £ , c t , c i (a) (b) (c) (d) (O </) (g) (h) ( 0 (;)

Fig. 2. Vowel modifiers of Bangla (a) A; (6) I; (c) II; (d) U; (e) UU; (/) RI; (g) E; (ft) AI; (i) O; (j) AU.

may appear either in the lower zone or in the middle zone or in both upper and middle zones of a basic character. These are explained in Fig. 3. The only entities that appear entirely in the lower zone are three

VMs, namely, «.(U), «.(UU) and <(Ri). In an earlier work,8 handwritten shapes of these three VMs were studied for their detection and recognition.

upper zone middle zone lower zone

ftfl"^C?fa Fig. 3. Three zones found in Bangla characters.

The skeletal representation of the first pattern may have an end pixel in the lower half (Figs. 5(a), (c), (e), (k), ([), (m) and (n)) or it may contain a loop that is vertically elongated (Figs. 5 (b), (d), (/), (o), (p), (q) and (r)). The skeleton of the second pattern also may have an end pixel in the lower half (Figs. 5 (i), (j), (k), (/), (o) and (p)) or it may have a loop that is not vertically elongated (Figs. 5. (g), (h), (m), (n), («), (r)).

(it) 0 ) (m) (») (o) (p) (S) (,)

Fig. 5. Variations in handwritten forms of 7 VMs of Bangla whole or part of which occur in the middle zone.

Of the remaining 7 vowel modifiers, 3 appear only in the middle zone while the other 4 appear in both the upper and middle zones in their printed forms. However, the shapes of the parts of the later four vowel modifiers that appear in the middle zone are exactly the same as those of the three VMs that appear entirely in the middle zone. In the present paper, handwritten forms of only the VMs or parts of VMs occurring in the middle zone are studied for their detection and classification. These 7 VMs occurring along with the basic character 3 are shown in Fig. 4. From this Figure, it can be seen that parts of these VMs that appear in the middle zone have only two distinct patterns, namely, 1 and '. The first of these two patterns can appear both on the left and on the right of a basic character while the second one occurs only on the left. Detection and recognition of the above are easier in case of their printed form.

^1 ft % J3'fa JM J3\ («) (4) (c) (d) («) (/) (*)

Fig. 4. Shapes of 7 VMs under consideration; (a), (d) and (J) VM appear only in the middle zone; (b), (c), (e) and (g) VM appear in both the upper and middle zones.

The shapes of the above two patterns occurring in the middle zone widely varies in their handwritten forms. Skeletal shapes of several handwritten samples consisting of major varieties are shown in Fig. 5.

3. Description of the methodology

3.1. Training and teat samples

It is an accepted fact that any effective work towards handwriting recognition requires a database of representative samples. Recently we developed such a database of bangla handwritten vowel modifiers.1

This database consists of a training set of 34000 samples and a test set of 6000 samples. There are equal numbers of training/test samples in each of the 10 classes. The present work is based on the above database of training and test sets.

3.2. Characteristics of the shapes of the VMs

A handwritten VM image is first binarized using Otsu's algorithm9 and cleaned by median filtering. The skeleton of the input image is obtained by applying the thinning algorithm of Datta and Parui.10

Simple heuristics have been applied for pruning of possible hairs in the skeleton. Three features are initially computed from the skeletal image of the input sample. These are presence or absence of (i) a skeletal end pixel, (ii) a skeletal loop that is not vertically elongated or (m) a skeletal loop that is vertically elongated. Let us now define four events described in Table 1.

S. K. Parui, U. Bhattacharya and S. K. Ghosh 131

Table 1. Four events characterizing various shapes of vowel modifiers.

Events Characteristics

L A skeletal end pixel in the bottom right quadrant

M A skeletal end pixel in the bottom left quadrant

N A nearly circular loop in the bottom half

V A vertically elongated loop

An extensive study of the skeletal shapes of the VM samples in our database shows that there are 18 handwritten variations, shown in Fig. 5, that the said 7 VMs can have and each of these can be characterized in terms of the above 4 events. For example, each of Figs. 5(a), (e) and (j) is characterized by the event L, Figs. 5(b),(f),(o),(p) and (r) by the event V and Fig. 5(g) by the event NV.

VM images satisfying event L have either a vertical line (i) on the right as shown in Figs. 5(a), (e), (k) and (Z) or a (") as shown in (Figs. 5(j), (I) and (p). We prepare two training sets TLI and TL2 of images having these two symbols respectively. We also prepare a training set TLZ of images having no VM but satisfying condition L. Similarly, sample images of VM satisfying condition M have either a vertical line (i)on the left (Fig. 5(c)) or ' (Fig. 5(i)). We prepare two training sets TMI a n d TMI of im

ages having these two symbols respectively. We also prepare a training set TMZ of images having no VM but satisfying condition M. In total we prepare 12 training sets of images given in Table 2.

3.3. Feature Vectors and Their Distributions

If there is a skeletal end pixel (Q) in the lower half of the image, we trace the skeleton from Q until a junction pixel or another end pixel (R) is encountered.

In case of a loop detected anywhere in the image, the junction pixel (J) is found and the skeleton is traced in North or West direction until another junction pixel or an end pixel(i?) is encountered. The curve (V) thus traced is divided into 5 segments of equal length.1 That is, four points PI,P2,P$,PA on V are found such that the curve distances between Pi-i and Pi(i = 1,2, . . . , 5 ) are equal where PQ = Q and P& = R. For this we have used the algorithm of Parui et al.11 Let 9i,i = 1,2,. . . ,5 be the angles that the line segments Pi-iPi make with the a;—axis. Note that the angles 0* are features that are invariant under scaling or size of the curve and represent only its shape. Finally, the feature vector of the curve is defined as 9 = (6i,e2,e3,94,e5,L,X,Y), where L,X,Y are the length and the coordinates of the centre of gravity of the curve respectively. We assume that 0 follows a mixture of 8-dimensional Gaussian distributions,12 which is defined as,

/ ( e) = YlPiUi^' where

i= l

« i ( 9 ) =

exp{-.5(9 - /XifErH© - Mi)}/{(27r)8/2|E i|1/2}

In the above, Pi are prior probabilities and K\ is the number of components in the mixture.

To distinguish between vertically elongated loops and loops with other shapes, the feature vector a = (Ui,U2,X,Y) is used, where Ui is the perimeter of the loop normalized with respect to the image height, C/2 is a measure of vertical elongatedness of the loop and is defined as the ratio of the width and height of the bounding box of the loop and X, Y indicate the coordinates of its centre of gravity, a also is assumed to follow a mixture of 4-dimensional

Table 2. Training sets and their characteristics.

Sets Description of the sets

T L I Images satisfying L and having 1 on the right

TLI Images satisfying L and having C

TL3 Images satisfying L but having no VM

T M I Images satisfying M and having 1 on the left

Tjv/2 Images satisfying M and having C

TM3 Images satisfying M but having no VM

Tjvi Images satisfying N and having C

TJV2 Images satisfying N but having no VM

T01 Images satisfying Vand having 1

T02 Images satisfying V but having no VM

Tpi Images obtained after removing " if present,

and 1 from the images in TLI U T Q I

Tp2 Images obtained after removing the vertical

line on the right from the images of the

4 basic characters *t. CT. ^T. "f

132 Recognition of Handwritten Bangla Vowel Modifiers

Gaussian distributions, denoted as

K2

g{a) = Y^QiVi{a), where i = i

Vi(a) =

exp{-.5(a - / i i f E - ^ a - /x i)}/{(27r)4/2 |E i |1/2}

In the above, qi are prior probabilities and K<i is the number of components in the mixture.

When there is a vertical line on the right, it may or may not be a VM (Fig. 6(a) and Fig. 6(6)). To distinguish between a VM and a non-VM, in such cases, we use a mixture of Dirichlet distributions,13

which is defined as

K3

h((3) = ^rjWj(/3) , where i = l

Ti are prior probabilities and K3 is the number of components in the mixture and

1*03) = !*(/?!,...,&) = / ^ i\P?'-\ where

i= l

4 4

Y^ fa = 1, 0 < fa < 1 and m = 2 J »™i> rn,i>0 i=l i=l

The feature vector /? = (/?i,..., (3A) used above is computed from the skeleton image as follows.

The bounding box (minimum enclosing rectangle) of the skeleton excluding the vertical line on the right is divided into four horizontal strips of equal height as shown in Figs. 6(a) and 6(6). j3i(i = 1,2,..., 4) is computed as the proportion of skeletal pixels present in the ith strip (i = 1,2,.. . ,4 from top to bottom).

Fig. 6. (a) A VM is present. (b,c) No VM is present

3.4. Classification

3.4.1. Training

From the training set T/,2 we learn the parame-

ters of / i ( 0 ) = yjpijUii(©)- From the combined t= i

training set TLI U TLZ, we learn the paramenia

ters of / 2 ( 9 ) = Yl P2iU2i(Q)- Similarly, from the

i= i training sets TM\,TM2 and TM3, the distributions /3(©)i /4(©) a n < i /5(®) respectively are determined. From the training sets T/vi and TV2 the distributions fe(@) and /?(@) respectively are determined.

We learn the parameters of gi(a) = 2_.Qiivu(a) i=l

from the combined training set T/viUX^ and the pa-

rameters of 32(a) = 2_]q2iV2i(a) from the combined »=i

training set To\ U To2-Note that the distributions U2i(©)'s represent

shapes of types that are present on the right side of the images shown in Fig. 6. From the mean vectors of these distributions, we identify the U2,(0)'s that represent a vertical shape (Fig. 6(a) and Fig. 6(6)). The other U2i(0)'s represent a shape that is not vertical (Fig. 6(c)).

Consider the two images in Fig. 6 in which there is a vertical shape on the right side in both the images. However, this shape is a VM in Fig. 6(a), but not a VM In Fig. 6(6). Two distributions h\{j3) and /i2(/3) are estimated from Tp\ and Tp2 respectively.

Expectation maximization (EM) algorithm is used to learn the parameters of the above mixture distributions. In order to find the optimal value of K, the Bayesian information criterion (BIC)14 is used.

3.4.2. Testing

The classification of VMs is made on the basis of the algorithms described below. ALGO-1 outputs a decision regarding VMs on the basis of skeletal end pixels in the lower half of the image while ALGO-2 outputs such a decision on the basis of skeletal loops present in the image. After the outputs of ALGO-1 and ALGO-2 are obtained, ALGO-3 is employed.

ALGO-1: Step-1. Set SW = 0. If there is no skeletal end pixel go to STOP-1.

S. K. Parui, U. Bhattacharya and S. K. Ghosh 133

Step-2. If the skeletal end pixel is in the left lower half, compute the feature vector of the curve V and then compute /3(©),/4(@) and / s (0 ) . Step-3. If /3(0) is the largest, VM T is present (Event M). Go to STOP-3. Step-4. If /4(B) is the largest, VM 'E' is present (Event M). Go to STOP-3. Step-5. If / s (0 ) is the largest, no VM is present so far. If there is another end pixel, go to Step-2. Else go to STOP-3. Step-6. If the end pixel is in the right lower half compute the feature vector 9 of the curve V and then compute /i(©) and /^(O). Step-7. If / i ( 0 ) is larger, VM 'E' is present (Event L). Go to STOP-3. Step-8. If hi®) is larger, compute u2i(©)'s. Find «2r(©) that is largest among them. If U2r(©) does not represent a vertical shape, no VM is present so far. If there is another end pixel, go to Step-2. If U2r(©) represent a vertical shape, set SW = SW + 1 . If SW = 1 , then if there is another end pixel go to Step-2. If SW=2, VM 'A' is present go to STOP-3. STOP-1.

ALGO-2: Step-1. If there is no loop, go to STOP-2. Step-2.Compute the feature vector a of the loop and then compute g\(a) and #2 (a). Step-3. If gi(a) is larger (Event AT), then first compute the feature vector O and then compute fe(Q) and / 7 ( 0 ) . Step-4. If fe(B) is larger, VM 'E' is present. Go to STOP-3. Step-5. If /?(0) is larger, no VM is present so far. If there is another loop, go to Step-2. Else go to ALGO-3. Step-6. If 52(0) is larger (event V), compute the C.G. of the loop. Step-7. If the loop is on the left, VM T is present. Go to STOP-3. Step-8. If the loop is on the right, set SW = SW + 1 . If SW =1, then if there is another loop, go to Step-2. If SW=2, VM 'A' is present. Go to STOP-3. STOP-2.

ALGO-3: Step-1. If SW = 0, no VM is present. Go to STOP-3. Step-2. If SW=1 (VM (•) is present in the right

side). All the skeletal pixels falling on the vertical shape are removed. The feature vector 0 of the modified image is computed. Then h\((3) and /i2(/3) are computed. If the former is greater, then VM (1) is present. Else, no VM is present. STOP-3.

4. Experimental Results

Simulation results of the proposed recognition scheme have been obtained using the database described in Section 3.1. The basic characters used for the training and test of the present scheme have been taken from a similar database7 also developed by us.

Experimental results corresponding to the events described in Table 1 are given below.

Table 3. Confusion matrix of the test set of event L

LI L2 L3

LI

85.84 8.75 9.7

L2

6.06 83.25 7.96

L3

8.1 8.00 82.34

Event L: A skeletal end pixel gets detected in the bottom right quadrant. In this case, the sample may have (i) a vertical line on the right (LI) or (ii) a ' (L2) or (Hi) no VM (L3). In these situations, the recognition accuracies are respectively 86.25%, 85.25% and 83.20% on the respective training sets and similar figures on the corresponding test sets are respectively 85.84%, 83.25% and 82.34%. The relevant confusion matrix for the test set is shown in Table 3.

Event M: A skeletal end pixel gets detected in the bottom left quadrant. In this case, the sample may have (i) a vertical line on the left (Ml) or (ii) a ' (M2) or (Hi) no VM (MS). In these situations, the recognition accuracies are respectively 98.2%, 98.12%, and 95.60% on the training sets and similar figures on the corresponding test sets are respectively 97.3%, 96.4%, 95.6%. The relevant confusion matrix for the test set is shown in Table 4.

Event N or V: A nearly circular loop in the bottom half or a vertically elongated loop is detected. In this case, the sample may have (i) a -7 (Nl) or (ii) no VM (N2 or 02) or (Hi) a vertical line (part or whole of a VM) (01). In these situations, the recognition accuracies are respectively 94.72%, 97.80%, 85.00%

134 Recognition of Handwritten Bangla Vowel Modifiers

Table 4. Confusion matrix of the test set of event M

Ml M2 M3

Ml

97.30 1.77 2.30

M2

1.50 96.40 2.10

M3

1.20 1.83 95.60

and 89.00% on the training sets and similar figures on

the corresponding test sets are respectively 93.66%,

96.60%, 84.00% and 88.00%. The relevant confusion

matr ix for the test set is shown in Table 5.

Table 5. Confusion matrix of the test set of event N and O

Nl N2 01 02

Nl

93.66 1.60 4.05 4.60

N2

3.00 96.60 5.60 3.30

Ol

1.30 1.00

84.00 4.10

02

2.00 0.80 6.35

88.00

5. C o n c l u s i o n s

In the present article, we have described a scheme

for detection and recognition of handwrit ten vowel

modifiers of Bangla. The proposed scheme has been

used to identify the whole or par t of a vowel modifier

tha t appears in the middle zone of a Bangla charac

ter. This is a pioneering work since to the best of

our knowledge no similar work is available in the lit

erature. Although the recognition accuracies are not

high, these are encouraging due to the above fact. In

future, we shall take care of the par t of handwrit

ten vowel modifiers of Bangla, which appears in the

upper zone.

References

1. S .K. Parui, U. Bhattacharya, A. K. Datta and B. Shaw, Proc. 3rd Workshop on Computer Vision, Graphics and Image Processing, WCVGIP 2006, Hyderabad, 204 (2006).

2. R. Plamondon, S. N. Srihari, IEEE Trans. Patt. Anal, and Mach. Inteli, 22(1), 63 (2000).

3. S. D. Connell, R. M. K. Sinha and A. K. Jain, Proc. of the 15th ICPR, Bercelona, Spain, 368 (2000).

4. I. K. Sethi and B. Chatterjee, Pattern Recognition, 9, 69(1977).

5. A. K. Dutta and S. Chaudhury, Pattern Recognition, 26,1757(1993).

6. F. R. Rahman, R. Rahman and M. C. Fairhurst, Pattern Recognition, 35, 997(2002).

7. U. Bhattacharya and B. B. Chaudhuri, In Proc. of the 8th Int. Conf. on Document Analysis and Recognition, Seoul, 2, 789(2005).

8. U. Bhattacharya, S. K. Parui, M. Sridhar and F. Kimura, Proc. of IICAI-05, Pune, India, 1357(2005).

9. N. Otsu, IEEE Trans. SMC, 9, 377(1979). 10. A. Datta and S. K. Parui, Pattern Recognition, 27

1181(1994). 11. S. K. Parui and D Dutta Majumder, Pattern Recog

nition Letters, 1(3), 129(1983). 12. K. Fukunaga, Introduction to Statistical Pattern

Recognition, Academic Press, San Diego, 2nd Edition, 1990.

13. K. Samuel, K. Balakrishman and J. Norman, Continuous Multivariate Distribution. New York : Wi-ley,2000

14. J. Bernardo and A. Smith, Bayesian Theory. John Wiley & Sons, 1994.

135

Template-Free Word Spotting in Low-Quality Manuscripts

Huaigu Cao* and Venu Govindaraju

Center for Unified Biometrics and Sensors (CUBS) Dept. of Computer Science and Engineering

University at Buffalo, Amherst, NY E-mail: {hcao3*, govind} ©buffalo, edu

As the OCR technique is not yet adequate for handwritten scripts with large lexicon, word spotting has been introduced as an alternative to OCR. This paper proposes a novel approach to word spotting that, instead of matching features of the word image to features extracted from predefined templates, uses the estimated posterior probability as the output of well trained classifier for spotting. Gabor features are extracted from gray scale image in order to yield higher performance on degraded, low quality document image.

Keywords: Keyword spotting; OCR.

1. Introduction

Nowadays, Optical Character Recognition (OCR) has advanced enough for a certain applications such as printed document, on-line handwritten document, and off-line hand-written documents with good image quality and relatively smaller lexicon. However, OCR is not adequate for the low-quality handwritten documents with large lexicon. Word spotting1 has been proposed as an alternative to OCR for indexing and retrieving the keywords of low-quality handwritten scripts.

The idea of word spotting is to find all the occurrences of a given word in the document images. Here the input is a word which is usually provided as user input, and the output are all the coordinates of the corresponding word images. Most of the existing approaches to word spotting choose a certain number of word images containing the user query as templates and find the best matches in the dataset. Thus the most important step is to determine the similarity between two word images. Among the existing approaches, the similarity can be defined on either the intensity of the raw images or features extracted from the original images. Several similarities between raw image data using different definitions of distances such as XOR, SSD, and EDM are discussed in.2 There are also some similarities defined over feature space such as SLH2 (using Scott and Longuet-Higgins algorithm), SC3 (Shape Context matching), DTW4 (Dynamic time warping), CORR5 (recovering the correspondences between points of interest in two images). The performance of all the above sim

ilarities are compared in4 which shows that DTW and CORR approaches have the best performance. B. Zhang, et al.6 proposed a method based on word shape and claimed to have better performance than the DTW approach in.4

All of the above approaches assume that for any query, there are a few word images stored as templates in a training set. However, since it is not easy to get all the word images of possible user queries in advance, this assumption limits the application of word spotting to a small set of keywords. To solve this problem, in the approach proposed in this paper, matching is performed in character image level rather than in word image level. Since the possible number of characters is very limited, i.e., 26 letters for English text, it will be much easier to build a training set of character images.

Another novelty of our approach is that we use Gabor features extracted from grayscale images. For some document image with degraded quality, e.g., the noisy carbon medical form shown in figure 1, so much information was lost after binarisation that the binarised version is not even readable by human beings, whereas the grayscale version is still readable. Although there are some literatures8'9 on grayscale feature extraction for OCR, all the existing feature-matching based word spotting methods still extract features from binary image.

The following sections in this paper are organized as follows: in section 2 we will talk about how to extract features from grayscale word image; in section 3, two similarities are discussed; in section 4, the results of experiments are presented.

136 Template-Free Word Spotting in Low-Quality Manuscripts

*

- t ~ , • • • • • • * . - : ,

t

-**.„ „

iJr ^iMMui^iiiJMTHfmfiffi J in ixrn ...uJWlir H I ,

a) a patch from the medical form (b) the binarized image of the patch

Fig. 1. An example of the carbon copy of medical form.

2. Ext rac t ing Gabor features from word image

2.1. Gabor filter and Gabor wavelet

The two dimensional special function of Gabor filter and its two dimensional Fourier transform can be written as follows:7

g(x,y,F,ax,ay)

e x p [ - | ( ^ + ^ ) + 27rjFa;] (1) 2naxay

G{u,v) = e x p { - - [ - 5 - i - + Zol) (2)

where au = l/(2nax), av = l/(2ircry), and parameter F specifies the central frequency of interest. Gabor wavelet derived from Gabor filter in equation (1) is defined as follows

9mn(x, y) - a~mg(x', y', F, ruc/K, ax, ay), (3)

where

x = a ' (xcos^ + ysinO),

and

x' = a m(—xsm6 + yco$0).

In equation (3), rn = 0,1,..., S-1 where S is the total number of scales, 9 = nir/K, and n = 0,1, . . . , K — 1 where K is the total number of orientations. The central frequency of interest is F. By varying m and n, we can apply filter gmn{x,y) to the input image (using 2-D convolution) to get features at different scale and orientation. The Gabor wavelet is non-orthogonal, thus there is redundant information in the filtered image. The following strategy is used to reduce the redundancy.7 Let Ui and Uh denote the

lower and upper center frequencies of interest. Then the filter parameters a, au and av are determined by

a=(Uh/Ut)^

u (a+l)V21n2

»»(*)K^] 2 l n 2 _i?J ! l2) !£ i"

(4) In order to make the Gabor filter be sensitive to the strokes of the characters, the central frequency of interest should be set to ^ , where W is the stroke width. This is because the stroke width is the half-period of the signal of interest, so the period is 2W, and the frequency is ^ • In PCR medical forms, W varies from 5 to 8 pixels which correspond to frequencies 0.1 and 0.0625, respectively. Our experiment is different from related work8'9 in that instead of applying Gabor filter of only one scale, we applied a Gabor wavelet of two different scales. The upper and lower frequencies of interest are Uh = 0.1, and Ui = 0.05 so the range of stroke width is completely covered. This seems more reasonable in that the stroke width is not fixed. Experiment of comparison indicates a better performance of using 2-scale Gabor wavelet over single scale Gabor filter.

2.2. Feature extraction from character image

In our dataset, the width and height of character images are between 30 and 40 pixels. So we take 64 x 64 pixels character image with the character at the center and apply a Gabor wavelet of two scales (Uh = 0.1, and Ui = 0.05) and four orientations to the images. Then a set of histogram features proposed in8'9

are taken from the 48 x 48 sub-image at the center

Huaigu Cao and Venu Govindaraju 137

of the 64 x 64 transformed image. Specifically, the 48 by 48 output of Gabor filter is divided into N by N blocks. Each block r(x, y) is of size M by M, where M=48/N. In each block, histogram features are calculated separately by positive and negative real part outputs weighted by a Gaussian function8 (See the following equations). F+ = x,y

£(m,n)€r(*,w) G(m - x,n - y) • max(0,FK(m,n))

(5) and F~ = x,y

X3(m,n)er(x,») G ( m ~x,n-y)- min(0,FK(m,n)) (6)

where G(x,y) = e x p { - ( i 2 + y2)/(2r2)}/(27r), and FK is the real part output of Gabor filter. For the scale m=0, N, M, r are set to 4, 12, 6 respectively; for the scale m=0, N, M, T are set to 2, 24, 12, respectively. As a result, totally 2 x 4 x (4 x 4+2 x 2) = 160 features are extracted.

2.3. Feature extraction from word image

%Wk

Fig. 2. The diagram indicating how a word image is split into four character images for feature extraction.

Let Iw[l : 64,1 : L] denote a 64 by L (L > 64) word image, where the word is centered vertically in /„,. Suppose Iw contains n characters. Then a simplified definition of the feature vector of Iw is VW=[V?V?...VT], where Vt(l < i < n) is the 160-dimensional Gabor feature vector of the 64 by 64 sub-image

1,4:4 : s j + 6 3 , 1 : 6 4 ] where i j = ("-«)+(<-i)^-«3) _ I n o t h e r w o r d S ) a w o r d

image is divided evenly into n square blocks of character images allowing overlap. Figure 2 illustrates this feature mapping process. However, characters within a word image are seldom distributed evenly. The solution is to generate three candidates of feature vectors of the following sub-images:

Iw[xi = 4 + 63,1:64],

7 4 ( 4 - 8) : (ij, - 8) + 63,1 : 64], and

7u ;[(4 + 8 ) : (x u + 8 ) + 6 3 , l : 6 4 ]

for the i'th character and choose the one with minimum cost using criteria discussed in section 3.

3. Similarity between a word image and the query word

Suppose the query w = ciC2...c„ is word consisting of n characters, and the n-char acter feature vector of the word image Iw is Vw = [V1

7V2T...V„T]T. We

need to measure the similarity between w and Vw. In our experiments, we tested two different similarities that are based on Euclidian distance and posterior probability, respectively.

3.1. Euclidian distance similarity

We can define the Euclidian distance based similarity between a word image and the query word. First we extract the feature vectors of all the character images in the training set, and calculate the mean feature vector /x(c) of each character c. Then the squared root of squared sum of Euclidian distances between each V^T(1 < i < n) and n(ci) is calculated as a cost function, i.e.,

CE(w,Vw)= ( £ | | V i - / i ( c i . (7) v i = l

The less the cost is, the more similar the word image is to the query. In our experiment, since the training set is not very large, we reduce the feature vector of character image to 40-dimensional by PCA and get better performance than the method using the original 160-dimensional feature vectors. All the 64 by

138 Template-Free Word Spotting in Low-Quality Manuscripts

64 sub-images are taken from the word images, and their distances to each character are calculated and stored in the dataset so that the execution of query skips the step of feature extraction and is extremely fast.

3.2. Probabilistic similarity

Suppose that the posterior probabilities for character classification, i.e., P(Ci\Vi)(l < i < n) are known, and that all the characters are independent, then the posterior probability of the word P(w\Vw) = Iir=i P(ci\Vi)- Then we can normalize the probability for different value of n by calculating the cost

CP(w,Vw)= In P(w\V,„)

(8) The less the cost is, the more similar the word image is to the query. In our experiment, the posterior probabilities of character classification are estimated using the method proposed in10 from the output of LIBSVM11 (an implementation of SVM classifier) using RBF kernel. All estimated probabilities are stored in our database to speed up the execution of queries.

3.3. WMR similarity

For the purpose of comparison, we also performed spotting test using the Word Model Recognizer (WMR),12 a word recognizer that uses chain-code features from binarized images and performs word recognition under a scheme of over-segmentation and recognition. In spotting test with WMR, the word recognition distance produced by WMR is taken as a cost function.

4. Results and discussions

Our experiments are done on 12 medical form images. We take 5295 character images from the first ten document images of our dataset as a training set, and do spotting tests on the last two images using as queries all the 101 different words occurred 127 times in the tested document image.

First, all the word images are manually segmented, which can also be implemented automatically later. Then spotting tests using cost functions CE, Cp and WMR distance are done respectively. Binarization and line removal algorithms proposed in13 are performed before running WMR .

To evaluate the performance of our algorithm the recall-precision curves under all three cost functions are drawn in figure 3. For each cost function, a series of threshold values of the cost function are taken. The values of precision and recall are calculated for each threshold. We can see that the method using cost Cp outperforms that using cost CE and WMR with gains of equal recall and precision rates of 26% and 15%, respectively. Test results show that our approach has a high performance. Unlike existing approaches,4 our approach does not require any word template.

ftecaHAtciaon

Fig. 3. Recall-precision curves of methods using WMR word recognizer (Precision=Recall=54%), Euclidian distance similarity (Precision=Recall=46%), and probabilistic similarity (Precision=Recall=69%), respectively.

References

1. R. Manmatha, C. Han, E. M. Riseman, and W. B. Croft, Indexing handwriting using word matching, in 1st ACM International Conference on Digital Libraries, Bethesda, MD, March 20-23, pp. 151-159 (1996).

2. S. Kane, A. Lehman, and E. Partridge, Indexing George Washington's handwritten manuscripts, Technical Report MM-34, Center for Intelligent Information Retrieval, University of Massachusetts Amherst (2001).

3. S. Belonge, J. Malik, and J. Puzicha, Shape matching and object recognition using shape contexts, IEEE Trans, on PAMI 24: 24 (2002) 509-522 (2002).

4. R. Manmatha and T. M. Rath, Indexing of handwritten historical documents-recent progress, in Symposium on Document Image Understanding Technolegy (SDIUT), pp. 77-85 (2003).

Huaigu Cao and Venu Govindaraju 139

5. J. L. Rothfeder, S. Feng, and T. M. Rath, Using corner feature correspondences to rank word images by similarity, CIIR Technical Report MM-44 (2003).

6. B. Zhang, S. N. Srihari, and C. Huang, Word image retriecal vising binary features, in Document Recognition and Retrieval XI, SPIE vol. 5296, pp. 45-53 (2004).

7. B. S. Manjunath, W. Y. Ma, Texture features for browsing and retrieval of image data, IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume: 18, Issue: 8, Pages: 837 - 842 (1996).

8. Xuewen Wang, Xiaoqing Ding, Changsong Liu, Optimized Gabor Filter Based Feature Extraction for Character Recognition, ICPR (4) 2002: 223-226 (2002).

9. Xuewen Wang, Xiaoqing Ding, Changsong Liu, Gabor filters-based feature extraction for character recognition, Pattern Recognition 38(3): 369-379 (2005).

10. T.-F. Wu, C.-J. Lin, and R. C. Weng, Probability estimates for multi-class classification by pairwise coupling, Journal of Machine Learning Research, 5:975-1005 (2004). URL: http://www.csie.ntu.edu.tw/ cjlin/papers/ svmprob/svmprob.pdf.

11. Chih-Chung Chang and Chih-Jen Lin, LIBSVM: a library for support vector machines, (2001). Software available at http://www.csie.ntu.edu.tw/ cjlin/libsvm.

12. G. Kim and V. Govindaraju, A lexicon driven approach to handwritten word recognition for real-time applications, IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume: 19, Pages: 366 -379 (1997).

13. Robert Milewski, Venu Govindaraju, Extraction of Handwritten Text from Carbon Copy Medical Form Images, Document Analysis Systems pp. 106-116 (2006).

Unconstrained Handwritten Digit Recognition: Experimentation on MNIST Database

V. N. Manjunath Aradhya, G. Hemantha Kumar, S. Noushath

Department of Studies in Computer Science University of Mysore

Mysore- 570006, INDIA E-mail: mukesh-mysore@rediffmail. com

The performance of a character recognition system depends heavily on what features are being used. Though many kinds of features have been developed and their test performances on standard database have been reported, there is still room to improve the recognition rate by developing improved features. In this paper, we propose a Two-dimensional Principal Component Analysis (2D-PCA) for efficient handwritten digit recognition. 2D-PCA is based on 2D image matrices rather than ID vectors so that image matrix does not need to be transform into a vector prior to feature extraction as done in PCA.1 For subsequent classification purpose we have used Generalized Regression Neural Network (GR.NN). A test performed on the MNIST handwritten numeral database showed a better recognition rate is among the best on the MNIST database.

Keywords: 2DPCA,GRNN,Handwritten Digit Recognition, MNIST database

1. Introduction

Recognition of characters from the document images is at the heart of any document image understanding system. Character Recognition has been the subject of intensive research during the last two decades. This, not only because it is very challenging scientific problem but also because it provides a solution for processing large volumes of data automatically. Research in Character Recognition is popular for its various application potentials in banks, post offices, defense organizations, reading aid for the blind, library automation, language processing and multimedia design.

Over the last two decades, Neural Networks have been widely used to solve complex classification problems.2 On the other hand, there is a consensus in machine learning community that Support Vector Machines (SVM) is most promising classifiers due to their excellent generalization performance.3 However, SVMs for mulit-classes classification problems are relatively slow and their training on a large data set is still a bottle-neck. In,4 they proposed an improved method of handwritten digit recognition based on neural classifier Limited Receptive Area (LIRA) for recognition purpose. The classifier LIRA contains three neuron layers: sensor, associative and output layers. An efficient three-stage classifier for handwritten digit recognition is proposed in.5 They propose efficient handwritten digit recognition based on Neural Networks and Support Vector Machine classifiers. Combining multiple clas

sifiers based on third-order dependency for handwritten numeral recognition is presented in.6 New approximation scheme is proposed to optimally approximate the probability distribution by the third order dependency, and then multiple classifiers are combined using such approximation scheme. Eigen-deformations for elastic matching based on handwritten character recognition are presented in.7

Selection of a feature extraction method is probably the single most important factor in achieving high recognition performance. Principal Component Analysis (PCA)1 and Fisher Linear Discriminant analysis (FLD),8 respectively known as eigenface and fisherface method, are the two state-of-the-art methods in face recognition. Using these techniques, a character image is efficiently represented as a feature vector of low dimensionality. The features in such subspace provide more salient and richer information for recognition than the raw image. It is this success, which made face recognition (FR) based on PCA and FLD very active, although they have been investigated for decades. In the PCA-based character recognition technique, the 2D character image matrices must be previously transformed into ID image vectors. The resulting image vectors of characters lead to a high dimensional image vector space. It is difficult to compute the covariance matrix accurately due to its large size and the relatively small number of training samples. Hence, in this paper, a image projection technique called two-dimensional principal component analysis (2D-PCA), is developed for

V. N. Manjunath Aradhya, G. Hemantha Kumar and S. Noushath 141

image feature extraction.9 As to conventional PCA, 2D-PCA is based on 2D matrices rather than ID vectors. Image covariance matrix can be constructed directly using the original image matrices. As a result 2D-PCA has two advantages over PCA. It is easier to compute the covariance matrix accurately. Time computations required to determine the corresponding eigenvectors is less.

The remainder of this paper is organized as follows: In section 2, the 2D-PCA method is presented. In section 3, a brief GRNN is described. In section 4, experimental results are presented for the MNIST handwritten database. Finally, conclusions are drawn at the end.

2. 2DPCA Method

Let M denote an n-dimensional unitary column vector. Project an image A, onto M by the following transformation.

Y = AM (1)

where Y is called the projected feature vector of image A The total scatter of the projected samples can be characterized by the trace of the covariance matrix of the projected feature vectors as

J(M) = trace(C)x (2)

where Cx denotes the covariance matrix of the projected feature vectors of the training samples and trace{C)x denotes the trace of Cx. The covariance matrix of Cx can be denoted by

trace{C)x = MT[E(A - EA)T{A - EA))M (3)

let us define the following matrix as

It = E[{A - EAf(A - EA)\ (4)

The matrix It is called the image covariance matrix. We can compute It directly using the training image samples. If N is the number of training image samples, jth training image is denoted by an matrix Aj(j = 1,2, ,N), and the average image is denoted by barA. Then, It can be

It = j;Jt,{Aj-A)T(Ai-A) (5)

Alternatively, (2) can be expressed by

JM = MTItM (6)

The unitary vector M that maximizes the criterion is called the optical projection axis. The optical projection axis Mopt is the unitary vector that maximizes JM- The optimal projection vectors of 2D-PCA, Mi,, Md, are used for feature extraction. For a given image sample A, let

YK = AMK,k= 1,2,-•• ,d (7)

The above equation is called the principal component (vectors) of the sample image A. It should be noted that principal component of 2D-PCA is a vector, whereas the principal component of PCA is a scalar. For classification purpose, GRNN classifier is used.

3. Generalized Regression Neural Network (GRNN)

Work on artificial neural networks, commonly referred to as neural networks, has been motivated right from its inception by the recognition that the human brain computes in an entirely different way from the conventional digital computer.10 The generalized regression neural network was introduced by Nadaraya11 and Watson12 and rediscovered by Specht13 to perform general regressions. Generalized regression neural networks are paradigms of the Radial Basis Function (RBF) used to functional approximation. To apply GRNN to classification, an input vector x (2D-PCA projection feature matrix Ft) is formed and weight vectors W are calculated. The architecture of the GRNN is shown in Fig 1. A detailed review of this method is described in.10

Fig. 1. Sheme of GRNN

142 Unconstrained Handwritten Digit Recognition: Experimentation on MNIST Database

4. T h e R e c o g n i t i o n Resu l t s

Our experiments were performed on the well-known MNIST1 4 database of handwritten digits. MNIST database consists of total 60,000 samples. In this experiment we have used 50,000 patterns of the MNIST training set for training, and the remaining 10,000 for testing purpose. All the digits have been size-normalized and centered in a 28 X 28 box. Samples of handwritten digits are shown in Figure 2. Each experiment is repeated 36 times by varying number of projection vectors t (where t = 1, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, and 100). Since t, has a considerable impact on recognition accuracy, we chose the value that corresponds to best classification result on the image set. All of our experiments are carried out on a P C machine with P4 3GHz CPU and 512 MB RAM memory under Mat-lab 7.0 platform and Windows XP.

0 6 O 0 0

Fig. 2. Samples of handwritten digits of MNIST database

The proposed 2D-PCA method is used for feature extraction, while GRNN is used for subsequent classification purpose. The proposed method achieved 99.7% recognition rate when 10,000 testing pat terns used and training samples used is 50,000. In Figure 3, a comparison of the performance of different algorithms tested on the MNIST database is given. The method (2D-PCA + GRNN) was used on the original MNIST database and provided error rate of 0.3% that is better than error rates of most of the classifiers. The best results provided by Boosted LeNET 1 5 and Virtual SVMs6 are the state-of-art results, but corresponding classifiers were trained on a perturbed MNIST database.

M j t p ' / ^rnterSy MJW

Ti i r i r II »

Bit M U ^ H UtlelS

R^F n ^ t * ! A<B i> r cm ^

F s w t s Sneaf S«s©ire n#sr

S M-3 „ > K I

^ *m

BiBLr™^™^ B'""'1' B t B I " i

ss | ^ K i

lute w ' i ^m g?

1 -* t -f

4

| -I

3 i -t

i -i _i

Fig. 3. Comparison of the error rates of different methods on the test set of MNIST database

5. Conclus ion

In this paper, we addressed the problem of handwritten digit recognition based on 2D-PCA for feature extraction and generalized regression neural network for subsequent classification purpose. As opposed to conventional PCA, 2D-PCA is based on 2D matrices rather than ID vectors. 2D-PCA is very effective and more straightforward to use for recognition. The method with the MNIST database showed the best recognition rate (99.7%) among other classifiers proved on this database. In future enhancement, we plan to extend this method for handwritten alphanumeric characters.

References

1. M. Turk and A. Pentland, Journal of Cognitive Neu-roscience 3, 71 (1991).

2. C. M. Bishop, Neural Networks for Pattern Recognition (Clarendon Press, Oxford, 1995).

3. C. Burges, Knowledge Discovery and Data Mining 2, 1 (1998).

4. E. Kussul and T. Baidyk, Image and Vision Computing 22, 971 (2004).

5. D. Gorgevik and D. Cakmakov, An efficient three-stage classifier for handwritten digit recognition, in nth Intl. Conf. on Pattern Recognition (ICPR), 2004.

6. H. Kwang, Pattern Recognition Letters 24, 3027 (2003).

7. S. Uchida and H. Sakoe, Pattern Recognition Letters 36, 2031 (2003).

8. P. N. Belhumeur, J. P. Hespanha and D. J. Krieg-man, IEEE Trans. Pattern Anal. Machine Intell 19, 711 (1997).

9. J. Yang and D. Zhang, IEEE Trans. Pattern Anal. Machine Intell 26, 131 (2004).

V. N. Manjunath Aradhya, G. Hemantha Kumar and S. Noushath 143

10. S. Haykin, Neural Networks (Pearson Education, 2003).

11. E. A. Nadaraya, Theory of Probability applications 19, 141 (1964).

12. G. Watson, Sankhya Series A 26, 359 (1964).

13. Specht, IEEE Trans. Neural Networks 2, 568 (1991). 14. ht tp: / /yann. lecun.com/exdb/mnist . 15. D. DeCoste and B. Scholkopf, Machine Learning 46,

161 (2002).

PARTF

Image Registration and Transmission

147

An Adaptive Background Model for Camshift Tracking with a Moving Camera

R. Stolkin*, I. Florescu**, G. Kamberov***

* Center for Maritime Systems, ** Dept. of Mathematical Sciences, *** Dept. of Computer Science Stevens Institute of Technology,

Castle Point on Hudson, Hoboken, NJ 07030, USA, E-mail: {*rstolkinOstevens.edu; **[email protected]; ***[email protected]}

Continuously Adaptive Mean shift (CAMSHIFT) is a popular algorithm for visual tracking, providing speed and robustness with minimal training and computational cost. While it performs well with a fixed camera and static background scene, it can fail rapidly when the camera moves or the background changes since it relies on static models of both the background and the tracked object. Furthermore it is unable to track objects passing in front of backgrounds with which they share significant colours. We describe a new algorithm, the Adaptive Background CAMSHIFT (ABCshift), which addresses both of these problems by using a background model which can be continuously relearned for every frame with minimal additional computational expense. Further, we show how adaptive background relearning can occasionally lead to a particular mode of instability which we resolve by comparing background and tracked object distributions using a metric based on the Bhattacharyya coefficient.

Keywords: CAMSHIFT; mean shift; ABCshift; tracking; adaptive; background model; robot vision

1. Introduction

Popular and effective approaches to colour based tracking include the CAMSHIFT,1'2 Mean Shift3

and particle filtering4'5 algorithms. Of these, CAMSHIFT stands out as the fastest and simplest. CAMSHIFT was designed for close range face tracking from a stationary camera but has since been modified for a variety of other tracking situations.6'7

Robust and flexible tracking algorithms, requiring minimal training and computational resources, are highly desirable for applications such as robot vision and wide area surveillance, both of which necessitate moving cameras. Unfortunately CAMSHIFT often fails with camera motion, figure 3, since it relies on a static background model which is unable to adequately represent changing scenery.

We address these difficulties by modifying the algorithm to include a flexible background representation which can be continuously relearned. The resulting algorithm tracks robustly in two situations where CAMSHIFT fails; firstly with scenery change due to camera motion and secondly when the tracked object moves across regions of background with which it shares significant colours, figure 1.

It is observed that the adaptability of this approach makes it occasionally vulnerable to a special mode of instability, in which a shrinking search window can lead to the background being relearned as object. We detect and correct this error by comparing object and background distributions to check for

convergence.

2. Bayesian mean shift tracking with colour models

For each frame of an image sequence, the CAMSHIFT algorithm looks at pixels which lie within a subset of the image defined by a search window. Each pixel in this window is assigned a probability that it belongs to the tracked object, creating a 2D distribution of object location over a local area of the image. The tracking problem is solved by mean shifting3'8 towards the centroid of this distribution to find an improved estimate of the object location. The search window is now repositioned at the new location and the process is iterated until convergence.

By summing the probabilities of all the pixels in the search window, it is also possible to estimate the size of the tracked object region (in pixels). The search window can now be resized so that its area is always in a fixed ratio to this estimated object area.

The tracked object is modelled as a class conditional colour distribution, P (C |0 ) . Depending on the application, ID Hue, 3D normalised RGB, 2D normalised RG, UV or ab histograms may all be appropriate choices of colour model, the important point being that these are all distributions which return a probability for any pixel colour, given that the pixel represents the tracked object. These object distributions can be learned offline from training images,

148 An Adaptive Background Model for Camshift Tracking with a Moving Camera

or during initialisation, e.g. from an area which has been user designated as object in the first image of the sequence.

The object location probabilities can now be computed for each pixel using Bayes' law as:

P(oio = M 2 ) (1)

where P(0\C) denotes the probability that the pixel represents the tracked object given its colour, P(C\0) is the colour model learned for the tracked object and P ( 0 ) and P(C) are the prior probabilities that the pixel represents object and posesses the colour C respectively.

The denominator of equation (1) can be expanded as:

P(C) = P(C\0)P(0) + P(C\B)P(B) (2)

where P(B) denotes the probability that the pixel represents background.

Bradski1,2 recommends values of 0.5 for both P ( 0 ) and P(B). We find this choice difficult to justify since we take these terms to denote the expected fractions of the total search window area containing object and background pixels respectively. Hence we assign values to object priors in proportion to their expected image areas. If the search window is resized to be r times bigger than the estimated tracked object area, then P(0) is assigned the value 1/r and P(B) is assigned the value (r — l ) / r .

Bradski1,2 suggests learning the expression (2) offline (presumably building a static P(C\B) histogram from an initial image). While it is often reasonable to maintain a static distribution for the tracked object (since objects are not expected to change colour), a static background model is unrealistic when the camera moves. The CAMSHIFT algorithm can rapidly fail when the background scenery changes since colours may exist in the new scene which did not exist in the original distribution, such that the expressions in Bayes law will no longer hold true and calculated probabilities no longer add up to unity.

Particular problems arise with CAMSHIFT if the tracked object moves across a region of background with which it shares a significant colour. Now a large region of background may easily become mistaken for the object, figure 1.

3. Incorporating an adaptive background model

We address these problems by using a background model which can be continuously relearned. Rather than using an explicit P(C\B) histogram, we build a P(C) histogram which is recomputed every time the search window is moved, based only on the pixels which lie within the current search window. P(C) values, looked up in this continuously relearned histogram, can now be substituted as the denominator for the Bayes' law expression, equation (1), for any pixel. Since the object distribution, P(C\0), remains static, this process becomes equivalent to implicitly relearning the background distribution, P(C\B), because P(C) is composed of a weighted combination of both these distributions, equation (2). Relearning the whole of P(C), rather than explicitly relearning P(C\B), helps ensure that probabilities add up to unity (e.g. if there are small errors in the static object model).

Adaptively relearning the background distribution helps prevent tracking failure when the background scene changes, particularly useful when tracking from a moving camera, figure 4. Additionally, it enables objects to be tracked, even when they move across regions of background which are the same colour as a significant portion of the object, figure 2. This is because, once P(C) has been relearned, the denominator of Bayes' law, equation (1), ensures that the importance of this colour will be diminished. In other words, the tracker will adaptively learn to ignore object colours which are similar to the background and instead tend to focus on those colours of the object which are most dissimilar to whatever background is currently in view.

It is interesting to note that the continual relearning of the P(C) histogram need not substantially increase computational expense. Once the histogram has been learned for the first image it is only necessary to remove from the histogram those pixels which have left the search window area, and add in those pixels which have newly been encompassed by the search window as it shifts with each iteration. Provided the object motion is reasonably slow relative to the camera frame rate, the search window motion will be small, so that at each iteration only a few lines of pixels need be removed from and added to the P(C) histogram.

R. Stolkin, I. Florescu, G. Kamberov 149

If the P(C) histogram is relearned only once every frame, the speed should be similar to that of CAMSHIFT. However, if the histogram is relearned at every iteration, some additional computational expense is incurred, since to properly exploit the new information it is necessary to recompute the P(0\C) values for every pixel, including those already analysed in previous iterations. Theoretically, updating at each iteration should produce more reliable tracking, although we have observed good tracking results with both options.

In practice, ABCshift often runs significantly faster than CAMSHIFT. Firstly, the poor background model can cause CAMSHIFT to need more iterations to converge. Secondly, the less accurate tracking of CAMSHIFT causes it to automatically grow a larger search window area, so that far greater numbers of pixels must be handled in each calculation.

4. Dealing with instability

Adaptively relearning the background model enables successful tracking in situations where CAMSHIFT fails, however it also introduces a new mode of instability which occasionally causes problems. If the search window should shrink (due to the object region being temporarily underestimated in size) to such an extent that the boundaries of the search window approach the boundaries of the true object region, then the background model will learn that background looks like object. This results in a negative feedback cycle with the estimated object region and search window gradually (and unrecover-ably) collapsing.

We solve this problem by noting that as the search window shrinks and approaches the size of the object region, the learned background distribution, P(C\B), must become increasingly similar to the static distribution for the tracked object, P(C\0). If this increasing similarity can be detected, then both the object region and search window can be easily resized, figure 5, the correct enlargement factor being r, the desired ratio of search window size to object region size.

Several statistical measures exist for comparing the similarity of two histograms. We utilise a Bhat-tacharyya metric,9 sometimes referred to as Jeffreys-Matsusita distance,10 which for two histograms, p —

{Pi}ie{i,2,...,K}. a n d q = {qi}ie{i,2,...,K} is denned as:

dip, q) = 1 5Z (VP* - Voi? (3) i = l

0 < d < \/2. Note that this metric can easily be shown to be the same, modulo a factor of \ /2, as that referred to elsewhere in the literature.3-5 '8

At each iteration we evaluate the Bhattacharyya metric between the static object distribution, P (C |0 ) , and the continuously relearned background distribution, P(C\B). If the Bhattacharyya metric approaches zero, we infer that the search window is approaching the true object region size while the estimated object region is collapsing. We therefore resize both by the factor r. In practice we resize when the Bhattacharyya metric drops below a preset threshold. Useful threshold values typically lie between 0.2 and 0.7.

We believe this is a novel application of the Bhattacharyya metric. It is common in the vision literature3-5 '8 to use this metric to evaluate the similarity between a candidate image region and an object distribution for tracking (i.e. comparing potential object with known object). In contrast, we use the metric to compare an object distribution with a background distribution, inferring an error if the two begin to converge.

5. Summary of the ABCshift tracker

The key differences between ABCshift and the conventional CAMSHIFT tracker are as follows: 1. The background is continuously relearned. In contrast CAMSHIFT uses a static background model which often fails with a moving camera. 2. Values for the prior probabilities, P ( 0 ) and P(-B), are assigned based on the ratio, r, of search window size to estimated object region size. 3. Object and background distributions are compared using a metric based on the Bhattacharyya coefficient to check for instability and resize the search window if it is shrinking.

The ABCshift algorithm is summarised as: 1. Identify an object region in the first image and train the object model, P(C\0). 2. Center the search window on the estimated object centroid and resize it to have an area r times greater than the estimated object size. 3. Learn the colour distribution, P(C), by building a histogram of the colours of all pixels within the search window. 4- Use Bayes' law, equation (1),

150 An Adaptive Background Model for Camshift Tracking with a Moving Camera

to assign object probabilities, P ( 0 | C ) , to every pixel in the search window, creating a 2D distribution of object location. 5. Estimate the new object position as the centroid of this distribution and estimate the new object size (in pixels) as the sum of all pixel probabilities within the search window. 6. Compute the Bhattacharyya metric between the distributions, P(C\0) and P(C). If this metric is less than a preset threshold then enlarge the estimated object size by a factor r. 7. Repeat steps 2-6 until the object position estimate converges. 8. Return to step 2 for the next image frame.

6. Resul ts

To test the enhanced tracking capabilities of the adaptive background model, we compare the performance of the ABCshift tracker with that of CAMSHIFT for a large number of extended video sequences, available at our website.11 Some sample images are shown here in the figures.

ABCshift has succeeded in conditions of substantial camera motion, rapidly changing scenery, dim and variable lighting and partial occlusion. The strengths of ABCshift are particularly apparent in scenarios where the tracked object passes across regions of background with which it shares significant colours.

7. Conclusions

Adaptive background models, which can be continuously relearned, significantly extend the capabilities and potential applications of simple colour based tracking algorithms to include moving cameras and changing scenery. They also provide robustness in difficult situations where other algorithms fail, such as when the tracked object moves across regions of background with which it shares significant colours.

It is observed that the resulting flexibility can make the tracker prone to a particular mode of instability, however this can be corrected by an innovative application of a metric based on the Bhattacharyya coefficient.

Future work will examine alternative adaptive

techniques for relearning models of both the background and the tracked object. We are also exploring automated initialisation of the tracker in surveillance scenarios and applications of these algorithms to visual servoing and robot vision.

References

1. G. R. Bradski, Intel Technology Journal (Q2 1998). 2. G. R. Bradski, Real time face and object tracking

as a component of a perceptual user interface, in Proc. 4th IEEE Workshop Applications of Computer Vision, 1998.

3. D. Comaniciu, V. Ramesh and P. Meer, IEEE Transactions on Pattern Analysis and Machine Intelligence 25, 564 (2003).

4. L. V. G. Katja Nummiaro, Esther Roller-Meier, Object tracking with an adaptive color-based particle filter, in Pattern Recognition : 24th DAGM Symposium, Proceedings, Lecture Notes in Computer Science Vol. 2449 (Zurich, Switzerland, 2002).

5. P. Perez, C. Hue, J. Vermaak and M. Gangnet, Color-based probabilistic tracking, in Proceedings of the 7th European Conference on Computer Vision-Part I, Lecture Notes In Computer Science Vol. 23502002.

6. J. G. Allen, R. Y. D. Xu and J. S. Jin, Object tracking using camshift algorithm and multiple quantized feature spaces, in Proceedings of the Pan-Sydney area workshop on Visual information processing, ACM International Conference Proceeding Series Vol. 100 (Australian Computer Society, Inc., Darlinghurst, Australia, 2004).

7. N. Liu and B. C. Lovell, Mmx-accelerated real-time hand tracking system, in IVCNZ 2001, (Dunedin, New Zealand, 2001).

8. D. Comaniciu and P. Meer, Mean shift: A robust approach toward feature space analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 24(5), 603-619 (May 2002).

9. A. Bhattacharayya, On a measure of divergence between two statistical populations defined by their probability distributions. Bulletin of the Calcutta Mathematical Society 35, 99-110 (1943).

10. H. Jeffreys, An invariant form for the prior probability in estimation problems. Proc. Royal Society (A) 186, 453-461 (1946).

11. www.math.Stevens.edu/~ifloresc/ABCshift.htm.

R. Stolkin, I. Florescu, G. Kamberov 151

Fig. 1. A simple blue and red checkered object, moving from a region of white background into a region of red background. CAMSHIFT fails as soon as t he object moves against a background with which it shares a common colour. Frames 350, 360, 380, 400, and 450 shown. Green and red squares indicate the search window and estimated object size respectively. This movie, RedWhitelCAMSHIFT.avi, can be viewed at our website.11

Fig. 2. ABCshift tracks successfully. Frames 350, 360, 380, 400, and 450 shown. Green and red squares indicate t he search window and estimated object size respectively. This movie, RedWhitelABCshift.avi, can be viewed at our website.11

Fig. 3. Person tracking with CAMSHIFT from a moving camera in a cluttered, outdoors environment. Frames 1, 176, 735, 1631, and 1862 shown. Since the tracked person wears a red shirt, CAMSHIFT fixates on red regions of background, including brick walls and doors, and repeatedly loses the tracked person. Green and red squares indicate the search window and estimated object size respectively. This movie, PeopleTrackinglCAMSHIPT.avi, can be viewed at our website.11

Fig. 4. ABCshift successfully tracks throughout the sequence and is not distracted by red regions of background, despite being initialised in image 1 which contains no red background. Frames 1, 176, 735, 1631, and 1862 shown. Green and red squares indicate the search window and estimated object size respectively. This movie, PeopleTrackinglABCshift.avi, can be viewed at our website.11

Fig. 5. Bhattacharyya resizing. A simple red and blue checkered object is tracked across red, white and blue background regions. Frames 180, 200, 205, 206 shown. Due to rapid, jerky motion from frames 180 to 205, the search window has shrunk until it falls within the object region, risking relearning that background looks like object. ABCshift has detected this instability using the Bhattacharyya metric, and automatically corrects the estimated object region and search window size in frame 206. Green and red squares indicate the search window and estimated object size respectively. This movie, BhattacharyyaRedWhiteBlue.avi, can be viewed at our website.11

152

Colour and Feature Based Mult iple Object Tracking Under Heavy Occlusions

Pabboju Sateesh Kumar, Prithwijit Guha and Amitabha Mukerjee

Computer Vision Group, IIT Kanpur Kanpur - 208016, UP, India

E-mail; {psateesh,pguha, amit} Giitk. ac. in

Tracking multiple objects in surveillance scenarios involves considerable difficulty because of occlusions. We report a composite tracker - based on feature tracking and colour based tracking - that demonstrates superior performance under high degrees of occlusion. Disjoint foreground blobs are extracted by using change masks obtained by combining an online-updated background model and flow information. The state of occlusion/isolation is identified by associating foreground blobs with object regions predicted using motion initialized mean-shift tracker (colour cue). The feature tracker is invoked in occluded situations to localize these with higher accuracy. We present results from dense traffic data with 5-15 objects in the scene at any instant. Overall tracking accuracy improves to 94.7% from 85.3% achieved by the colour only tracker.

Keywords: Tracking, Occlusions, Feature Correspondence

(a) (b)

Fig. 1. Results of multi-object tracking using (a) only colour cue and (b) both colour match and feature correspondences. The scene contains a total of 16 objects, out of which 10 appear in crowds, (a) The colour based tracker properly localizes only 5 objects; (b) The composite colour-feature based tracker successfully localizes 14 objects in the scene.

1. In t roduc t ion

An algorithm for tracking multiple agents in a monocular surveillance setup is reported. An early approach to this problem deals with tracking blobs obtained from the process of background subtraction.1 However, such blobs do may form a group and get detected as a single blob or an agent can be detected as multiple blobs due to occlusions. The W4 system2 differentiates people from other objects by shape and motion cues and tracks them under occlusions by constructing appearance models and detecting body parts. Several researchers3 have employed particle filtering along with prior shape and motion models for multi-person tracking in cluttered scenes. Recently, Zhao et al4 proposed a Bayesian approach for tracking multiple persons under occlusions by computing MCMC based MAP estimates with prior information about camera model and hu

man appearance along with a ground plane assumption. McKenna et al.,5 on the other hand, presents a colour based tracking algorithm that performs in relatively unconstrained environments and works at three levels of'abstraction, viz. regions, people and groups.

In the recent past, a number of approaches have distinguished the types of occlusion, treating these as a source of additional information to be used in the visual analysis.6 This work builds on this approach, and uses the occlusion type to guide the tracking. We present a hybrid approach based on feature and colour based tracking, and contrast this with other approaches involving only colour.

The algorithm works by learning a background model as a pixel-wise mixture of Gaussians, change masks on which along with inter-frame motion information segments the objects as foreground blobs.6

The object is characterized by its supporting region, weighted colour distribution, trajectory and a planar graph constructed by Delanuay triangula-tion of the feature point set extracted in its supporting region. The system maintains a set of objects to (from) which objects are added (removed) as they enter (exit) the scene. We identify the objects to be either isolated or occluded by associating motion initialized mean shift tracker predicted object regions with foreground blobs. More so, the dissociated object/foreground regions are detected to identify their disappearances / reappearances. The occluded objects are further tracked with higher accuracy by the feature-point graph structure constrained

Pabboju Sateesh Kumar, Prithwijit Guha and Amitabha Mukerjee 153

feature correspondences. The object features are selectively updated based on their occlusion states.

Some salient strengths of the proposed scheme are the following. First, the ability to identify occlusion states and using the same in selective feature updates. Second, the inherent ability of recognizing failure situations and automatic track restorations and finally, a relatively unconstrained approach that does not assume any priors on object shape, motion models and ground plane. Figure 1 shows the performance improvement achieved by the hybrid colour-feature based tracking algorithm as compared to a colour only tracker.

The colour based multi-agent tracking algorithm and its extension to invoke feature correspondences is presented in section 2. Experimental results on a dense traffic video with ground-truth validation are reported in section 3.

2. Multiple Object Tracking

The object regions are segmented as a set of disjoint foreground blobs extracted by combining the cues derived from the change masks over the learned background models (pixel-wise mixture of Gaussians) and the inter-frame motion information. The object regions predicted by motion initialized mean shift trackers7 are associated with the extracted foreground blobs6 to detect the objects in either the state of isolation from other objects and background elements or the state of occlusion arising due to crowding and partial occlusions. Additionally, the cases of entry/exit and disappearance/reappearances are also identified. These occlusion cases guide the tracking algorithm in selective object feature updates and track restoration. The system maintains a set S{t) of objects, to (from) which objects are added (removed) as they enter (exit) the scene. The individual object features are updated as they are tracked across the frames. When an unmatched foreground blob detected, It is matched with disappeared objects based on color and position matching. Search region around each disappeared object is taken into consideration while matching with unmatched foreground blob. In the following sub-sections, we detail the limitations of colour based multi-object tracking algorithm6 and the proposed extension to combine colour and feature based tracking.

2.1. Tracking With Colour Cue

The j t h object Aj(t) is characterized by the set of pixels aj(t) it occupies, the colour distribution hj(t) weighted by the Epanechnikov kernel7 supported over the minimum bounding ellipse of a.j(t) and the finite length position history of the centers {cj(t — £')}T'=o °f * n e m m i m u m bounding ellipse of a,j(t). The object features are initially learned from the foreground blob extracted at its very first appearance and are updated throughout the sequence whenever it is in isolation.

An estimate of the center cj ' (t) of the minimum bounding ellipse of a, (t) is obtained by extrapolating from the trajectory {c,(t — t')}y=i- The mean-shift iterations,7 initialized at an elliptic region centered at cj (t) further localize center of the minimum bounding ellipse of the object region at Cj(t).

The object region and foreground blob associations are computed to identify the various occlusion states. The supporting pixel set, weighted colour distribution and trajectory information are updated for isolated objects. For the occluded ones (same (foreground) object pixel in different (object regions) foreground blobs), we only update the trajectory. More so, we identify the dissociated objects (disappearance) and blobs (entry/reappearance) followed by object-blob association re-computation to restore tracks of the existing ones and log the new objects in S(t).

The colour only tracker employ's the mean-shift algorithm for object localization and is thus prone to erroneous drifts in the mean-shift iterations. The mean shift algorithm models the target as a weighted colour distribution learned over an elliptical domain. Thus, convex near-elliptic compact objects are successfully tracked with this algorithm. However, several real world objects have non-convex shapes with holes - e.g. rickshaw, cycle, man on a motorbike etc. In such cases, the mean-shift tracker learns the background colour distribution in the target model and hence drifts away in object localization iterations. More so, mean-shift trackers are also found to fail under severe occlusions, as it models the colour distribution of the whole target region and not by parts. To avoid these limitations, we extend the object characterization to include the feature points as well, which we describe in the following sub-section.

154 Colour and Feature Based Multiple Object Tracking Under Heavy Occlusions

2.2. Combining Colour and Feature Cues

Feature correspondence was proposed in the context of image registration,8 later extended9 for the selection of good feature points. Consider the consecutive images fit and flt+i? such that n$(U) = Ot+i(U + d\j). Tomasi et al.9 have shown that the displacement vector d u is sufficient during tracking feature points between successive frames approximating deformation value to zero. Feature points are tracked using' the Kanade-Lucas-Tomasi (KLT) tracker.10

The sum of squared difference between consecutive images is reduced to find the displacement vector. Tracking is based on symmetric definition for dissimilarity between two images unlike earlier approaches given as ftt(U-du/2) = 0 e + i ( U + du /2 ) . The displacement vector can be computed by solving the equation G(U)du = e(U), where the 2 x 2 symmetric matrix G and the residue vector e are obtained from,

G(U) = / gt(X)gt(Xfw(X)dX JW(U)

e(U) = 2 x / (Qt - Oe+i)(X)ge(X)W(X)cJX JW(U)

Where, the integration is performed over a certain window W(U) centered at U, g t(X) = V« t (X) is the image gradient and w(JL) is a weighing function defined over W(U).' The pixel position U is considered to host a good feature point, if both the eigen values (Ai,A$) of G(U) are sufficiently high, i.e. mm(Ai, A2) > A, where A is a predefined threshold.

Fig. 2. Delaunay triangulation based graph model for (a car, a cycle and a rickshaw).

The feature tracker is invoked for objects under occlusions. We extend the object characterization of the colour only tracker to include the set of feature points in the object region. We perform a Delau

nay triangulation (figure 2) over the feature point set forming a planar graph that represents a geometrical structure of the object. Isolated feature tracking can erroneously correspond to a) points in the background or b) points on other objects. In case (a), foreground segmentation can be used to eliminate the correspondence. In case- (b), the faulty feature correspondence is detected by ;using the Motion Consistency Hypothesis: feature points on the same rigid body exhibit consistent motion. Even where the bodies exhibit large deformation (e.g. human motions), some branches of the graph exhibit relatively stable deformations. Object feature points are tracked in the consecutive images constrained by the feature-point planar graph structure to improve the tracking performance. However, in cases of (near) complete occlusions, where neither colour match nor feature correspondences can be established, we continue the tracking with motion predicted object position. Object is said to disappear, either it is found out as disappeared using predicates described in6 or if it losses minimum threshold number of features. When object disappears it is tracked using only motion information, which is susceptible to error as number of frames increase. In the case of hybrid tracker the duration of disappearance for objects is less compared to color only tracker, as objects are tracked until the object completely disappears. Hence matching accuracy increased in hybrid tracker compared to color only tracker.

3. Resul ts and G r o u n d - t r u t h Validation

We report an experiment on a traffic surveillance video involving a wide variety of vehicles like motorbikes, bicycles, rickshaws, cars, buses, trucks and tractors, as well as people and animals (cow). Next, we compare our results in multi-object tracking based on only colour and both colour and feature (figure 3).

The proposed algorithm is tested on different data sets. Figure 4 shows the results obtained on human data set. In this data set, we do not perform neglecting of feature points based on Motion Consistency Hypothesis described in section 2.2, because the tracked objects (humans) in this video are not rigid. Tracking accuracy in this case is less for both color only tracker and hybrid tracker compared to their respective Traffic video performances 3.

Pabboju Sateesh Kumar, Prithwijit Guha and Amitabha Mukerjee 155

(<-•) M)

Fig. 3. Re-identification errors. In sequence (c,d) (Mean shift tracker), a silver SUV occludes the rickshaw at upper center; after the occlusion the rickshaw is misidentified as another object that disappeared earlier, and the SUV is wrongly matched as rickshaw. The man in the blue bounding box is matched with a car that left the scene. All these errors are overcome by the hybrid feature tracker (e,f), it is particularly good at re-identification.

1|_A b- • J J5

(b)

•flrt-* —fi»te^!ii

(c) (d)

Fig. 4. Comparison of Mean shift and Modified trackers on Human data set

We observe, in the igures 4(a) and (c), that Mean shift tracker clearly fails to track the person under severe occlusion behind another person. In the igures 4 (b) and (d), we observe that Hybrid tracker able to track person under severe occlusion with blue bounding box. M i n i m u m dis tance (MinDist): Distance between point features can be changed. While selecting the point features, If the point feature that' is going to

be selected is near to already selected features, then it is not selected. It is based on the assumption that "neighboring pixels generally have similar goodness values" . The point feature is considered to be near other point feature if the distance between them is less than minimum distance. This can be used to speed up the process. For results in Igure 4 MinDist is fixed as 0 while it is fixed as 5 in figure 3, We validate the results against a ground-truth data over 700 frames. We compute the following measures:

• Total Tracking Accuracy accuracyt = where bi denotes the number of well

tracked objects and c* is the number of track losses in the ith frame. Re-identification accuracy (re — identt): Let Si be the number of disappeared objects in the ith frame, e» be the number of erroneously tagged reappearances3,, §i the number of successfully registered object entries in the scene, f% be the number of reappearances erroneously detected as scene entries, and lit be the the number of scene entries erroneously detected as reappearances. Then we define the Re-identification accuracy as the ratio re-identt = ^ # % T T r r r f ^ -

2_rfi—% \si~r9* i »*t;

Tracking Accuracy in Crowds is the ratio T**' it

er owdt = w *7* * x, where p» is the number of well tracked objects in the crowd and qi is the number of track losses in crowd, as observed in the ith frame. Approximat ion of Object Localization Ac-curacy, localizt = 2"fef \ % where n* denotes the number of objects tracked with an ill-sized or misplaced bounding box in the ith frame. As seen in table 1, results are significant improvements in all the categories, with strikingly im~ proved re-identification.

4, Conclusion

We have proposed an algorithm for multi-object tracking under occlusion by combining multiple cues( Colour, Motion, Features ) based on their importance in particular situation. The proposed scheme

aLet the agents a and b disappear in frames say x» y respectively. In frame i ( i > (x, y)) , the tracker identified agent b as re-appeared when actually it was agent a.

156 Colour and Feature Based Multiple Object Tracking Under Heavy Occlusions

is not restricted by any prior object shape/mot ion

models or ground plane assumptions and thus per

forms satisfactorily in relatively unconstrained en

vironments; more importantly, since no camera cal

ibration is needed, it can be placed anywhere and

immediately put to work. The remaining limitations

are significantly more difficult - e.g. when an ob

ject is nearly fully occluded (motion projection is

the only option), or differentiating between multi

ple objects entering the scene together (before they

split). A cue towards the latter may be to look for

differences in deformations of the feature graph - re

sulting in several clusters of motion of point features

on single blob, an approach tha t may work when

the objects deform differently, or move with different

speeds. Eventually, it would be important to extend

these ideas to work in more general situations, e.g.

cameras tha t move (initially with pan-tilt motions),

and for dynamic backgrounds (trees, fountains).

References

1. C. R. Wren, A. Azarbayejani, T. Darell and A. Pe-naland, Pattern Analysis and Machine Intelligence 19, 780(July 1997).

2. I. Haritaoglu, D. Harwood and L. Davis, IEEE Transactions on Pattern Analysis and Machine Intelligence 22, 809(August 2000).

3. C. Needham and R. Boyle, Tracking multiple sports players through occlusion, congestion and scale, in Proceedings of the 12th British Machine Vision Conference, 2001.

4. T. Zhao and R. Nevatia, Tracking multiple humans in crowded environments, in IEEE Computer Society Conference on Computer Vision and Pattern Recognition, July 2004.

5. S. McKenna, S. Jabri, Z. Duric and A. Rosen-feld, Computer Vision and Image Understanding 80 (2000).

6. P. Guha, A. Biswas, A. Mukerjee and K. Venkatesh, Occlusion sequence mining for complex multi-agent activity discovery, in The Sixth IEEE International Workshop on Visual Surveillance, May 2006.

7. D. Comaniciu, V. Ramesh and P. Meer, Real-time tracking of non-rigid objects using mean shift, in Computer Vision and Pattern Recognition, 2000.

8. B. D. Lucas and T. Kanade, An iterative image registration technique with an application to stereo vision, in Proceedings of 7th International Conference on Artificial Intelligence (IJCAI1981), (Vancouver, British Columbia, 1981).

9. C. Tomasi and T. Kanade, Detection and Tracking of Point Features, Tech. Rep. Technical Report CMU-CS-91-132, Carnegie Mellon University (April 1991).

10. S. Birchfield, Kit an implementation of the kanade-lucas-tomasi feature tracker www.ces.clemson.edu/stb/klt/.

Table 1. Performance Comparision of colour only and colour-feature based hybrid tracker

bt ct

accuracyt

St

et ft 9t ht

re — identt

Pt It

crowdt

nt

localizt

Colour tracker

2904 500

85.31%

29 17 2 5

30

23.44%

425 357

54.35%

213

92.67%

Hybrid tracker

3246 182

94.69%

23 1 2

20 0

93.02%

592 193

75.41%

15

99.54%

157

DCT Properties as Handle for Image Compression and Cryptanalysis

Anil Kr. Yekkala

Philips Electronics India Ltd. No 1, Murphy Road, Bangalore 560008, INDIA

E-mail: anil.yekkala&philips.com

C. E. Veni Madhavan

Indian Institute of Science Bangalore 560012, INDIA

E-mail: cevm&csa.iisc.ernet.in

Narendranath Udupa

Philips Electronics India Ltd. No 1, Murphy Road, Bangalore 560008, INDIA

E-mail: narendranath.udupa&philips.com

DCT transformation has been traditionally used to exploit spatial redundancy within an image and video for purpose of compression. JPEG standard commonly used for image compression is based on DCT transformation. In this paper we analyze the linear and orthogonal properties of DCT transformation for achieving an improvement over compression provided by JPEG, and we also use the DCT properties to demonstrate vulnerability of lightweight encryption schemes based on encrypting the DC values.

Keywords: DCT transformation; JPEG; DC prediction; DC encryption

1. Introduction

Both lossy and lossless compression schemes are based on exploiting the spatial redundancy within an image. In case of lossy compression schemes the spatial redundancy is exploited by representing the image in its frequency domain using a transform coding technique, and then doing the further processing in the frequency domain. DCT transformation is one of the most popularly used transformation techniques for image compression. For instance, in case of JPEG compression1 the entire image is divided into blocks of size 8x8, and each block is represented in its frequency domain by applying 2-dimensional DCT transformation. In the frequency domain the component corresponding to zero frequency is known as DC value and the non-zero frequency components are known as the AC values. It has been generally observed7 that most of the energy of an image is contained in the DC values. Hence, in order to reduce the number of bits to represent the DC values, the spatial redundancy is further exploited by encoding the DC values using Differential DC encoding. It is to be noted that even after differential DC encoding, the DC values generally occupy about 8% of the image

space. Using the properties of DCT we propose a replacement to Differential DC encoding, hence showing further improvement in compression achieved by JPEG. Again owing to the fact that most of the energy is present in DC values, several lightweight encryption schemes3 are based on encrypting the DC values to reduce the intelligibility of the image. Even though techniques4-6 for reconstructing images are available in the case when few of the DC values are corrupted, currently we do not see many methods in literature, which focus on reconstructing an image in the event of corruption of all DC values. Again using the properties of DCT transformation we show a method to reduce the amount of noise introduced in the image due to encryption of DC values, hence showing the vulnerability of Lightweight encryption schemes based on encrypting only the DC values.

1.1. Properties of DCT Transformation

In this paper we use the Linear Property and Orthogonal Property of DCT Transformation, to achieve better compression compared to JPEG, and also to reduce the amount of noise introduced due to encryption of DC values.

158 DCT Properties as Handle for Image Compression and Cryptanalysis

1.1.1. Linear Property

If X is a square matrix of dimension n x n, and if A is a constant square matrix of same dimension with constant value a, then

DCT{X + A) = DCT(X) + K

where K is a matrix whose first element namely the DC value is equal to n x a, and remaining elements i.e. AC values are zero. In case of images the value of n is equal to 8. Hence it can be concluded that modifying or corrupting the DC value of an image block results in a constant shift in each pixel of the block and vice-versa.

1.1.2. Orthogonal Property:

2-dimensional DCT transform is an orthogonal transformation. Hence, in case of JPEG, the output of DCT transformation on each block of 8x8 pixels can be considered to be a representation of 64 orthogonal equations on a vector of 64 elements. Similarly the IDCT operation can be considered equivalent to solving 64 unknown values using 64 independent equations. Hence, based on Orthogonal Property of DCT transformation, it can be observed that decoding each block without knowing the DC value is equivalent, to solving 64 unknown pixel values, based on 63 orthogonal equations representing the equations for the AC values. Hence the problem is equivalent to replacing the equation for DC value, by an alternative equation that is orthogonal to remaining 63 known equations. In this paper we propose to replace the equation X\ + X^ + ... + X&4 = 8 * DC representing the DC value by the equation representing the top-left pixel Xi = p or equivalently by an equation representing any other pixel. Hence, it can be concluded that even if the DC value of the block is not known, it is possible to decode all the pixel values of the block if atleast one pixel value is known along with the 63 AC values. But it is to be noted that solving the pixel values based on AC values and top-left pixel of the block is considered computationally ineffective, compared to solving the pixel values based on IDCT (i.e. using DC value and the AC values). Hence in order to use the fast computational techniques available for IDCT, the top-left pixel is used to predict the DC value approximately (8 times the top-left pixel value). Then using IDCT the pixel values of the block are obtained approximately. The

error (S) due to approximation of DC value is obtained by taking the difference in top-left pixel before and after the IDCT operation. Using Linear Property the error in each pixel due to usage of approximate DC is adjusted by adding the error 5 to each pixel value of the block.

2. Application for Image Compression

In an image it can be observed that the magnitude of the difference between two consecutive DC values is relatively higher, compared to magnitude of difference between two consecutive pixels. This phenomenon can be clearly observed from Table 1, the second column of the Table 1 shows the average of absolute difference of DCs of two consecutive blocks, whereas the third column of the Table 1 shows the average of absolute difference of top-left pixel of a block with a pixel to its left present in the previous block, and the fourth column shows the ratio of second column to third column. It can be observed that ratio for all images except for baboon are above 8. The ratio for baboon image is 7.25, which is quite less. This is due to texture of the baboon image, which can be observed in Table 1. Due to the sharp texture of the baboon image the pixel-to-pixel difference is quite high, but since same type of texture is present in most of the image, the DC differences between two consecutive blocks is generally low. Hence the ratio for baboon image is comparatively low. Again it can be observed from Table 1 that the ratio is quite high for pills image, this is due to smooth texture and sharp edges of the images, resulting in small pixel-to-pixel difference but high block-to-block difference.

Fig. 1. Baboon and Pills.

Hence it can be concluded that the effect of the differential encoding scheme can be improved further by taking difference between two consecutive pixels

Anil Kr. Yekkala, C. E. Veni Madhavan and Narendranath Udupa 159

Table 1. Comparison of DC Diff with Pixel Diff

Standard Images

Lena

Peppers

Baboon

Brandyrose

Bandon

Pills

Opera

Mean of Absolute

DC Differences

(a)

137.1

136.5

106.6

94.1

34.8

184.1

70.4

Mean of Absolute

Pixel Differences

(b)

6.6

6.2

14.7

4.1

2.5

4.9

6.4

Ratio of (a) and (b)

(a/b)

20.77

22.02

7.25

22.95

13.92

37.57

11.00

i.e. with reference to Figure 2 the difference between the top-left pixel (P) of a block (Block2) and the pixel (Q) to its left from the previous block (Blockl). The differential pixel encoding can be applied to all blocks except for the first block in each row.

Q

Blockl PI Block2

Fig. 2. Pictorial view of 2 consecutive blocks of an image.

The image compression based on differential pixel encoding for gray images is explained in following sub-sections

2.1. Mathematical Notations

For explaining the encoding and decoding procedure the following notations will be used I - Denotes the entire image Iij - Represents an 8x8 pixel blocks of an image, namely the block in ith row and jth column of the image. Iij(m,n) - Represents a pixel within the block, namely the pixel in mth row and nth column of the block Iij.

2.2. Encoding procedure for Gray Images

The encoding procedure is similar to JPEG encoding, except for the fact that Differential Pixel encoding is used instead of Differential DC encoding. The blocks

are encoded in sequential order starting from first leftmost block in first row, followed by the second leftmost block in the first row and lastly the last i.e. the rightmost block in the last row. Differential Pixel encoding is not applied for the first block in each row, for these rows the entire DC value is encoded along with the AC value. For the remaining blocks where Differential Pixel encoding is applied, the procedure (say block Itj, where j > 1) is as follows

(1) DCT transformation is applied on the block Ii%j. (2) The DC value of the block U j is replaced by the

top-left pixel value of the block namely It j (1,1). (3) Differential Pixel encoding is applied for the top-

left pixel of the block Iij by taking the difference between the top-left pixel I,j(l, 1) of the block and pixel J i j_ i ( l ,8 ) to its left present in the previous block

A j = / M ( l , l ) - / y - i ( l , 8 )

(4) The pixel difference Dij, along with the AC values of the block are then quantized and encoded using run-length encoding.

2.3. Pixel Difference and quantization

In order to reduce the amount of error due to quantization on Differential pixel encoding, the difference Dij is computed by taking the difference of Iij(l, 1) from modified Ji,j_i(l ,8), i.e. the previous encoded block Jj,j-i is decoded and the modified pixel value i i j _ i ( l , 8) is used. Moreover, the quantization value used for quantizing the pixel difference is 8 times less the quantization value used for quantizing the DC difference in case of JPEG, this is due to the fact that DC value is equal to 8 times the average pixel value of the block.

2.4. Decoding procedure for Gray Images

The encoded image date is segregated into blocks by identifying the End of Block (EOB) indicators and de-quantization is applied on each block. Similar to encoding the image is decoded is sequential manner starting from first row, and within each row the decoding starts from the left most block. For the first block in each row, the DC and AC value is obtained after de-quantization, and IDCT operation is applied on these blocks to obtain the pixel values. For the remaining blocks (say block Iij, where j > 1) the pixel

160 DCT Properties as Handle for Image Compression and Cryptanalysis

difference along with the AC values are obtained, and the decoding procedure is as follows

(1) The top-left pixel of the block Iij is obtained by taking sum of the pixel difference Dij and top-right pixel of the previous block, which is already decoded

J i j ( l ) l ) = A j + / i j - i ( l , 8 )

(2) The DCij value of the block is approximately computed using Ddj = 8 * I%,j(l, 1)

(3) IDCT transformation is applied on the approximate DC value and the AC values, to compute the pixel values of the block approximately, denoted by I*j .

(4) The error in pixels, due to approximation of DC value is calculated by taking the difference in top-left pixel before and after IDCT transformation.

Sid = Itj(l,l) - ^ ( 1 , 1 )

(5) The correction is then applied on all pixels of the block Iij by adding Sij to all pixels of the block

2.5. Extension for Color Images

The encoding/decoding scheme for gray images is extended to color images by using the encoding/decoding procedure similar to gray images on all the 3 color components namely Y, Ct, and C r components of the color image.

3. Vulnerability of Lightweight DC Encryption Scheme

In this section we propose a scheme for reducing the amount of noise introduced in an image due to encryption (or corruption) of all DC values. It is assumed that atleast one pixel value of the entire image is known in addition to all AC values. The scheme is then based on first decoding a block whose pixel value is known, along with the available AC values of the block, and then using the advantage of spatial redundancy within an image, a pixel value of neighborhood blocks are predicted, then using the predicted pixel along with the available AC values the neighborhood block is decoded, hence proceeding in a similar manner all blocks of the image are decoded.

3.1. Decoding procedure for gray images

Inputs:

(1) Quantized AC values (2) One pixel of the image is assumed to be known,

for sake of simplicity it is assumed to be th top-left pixel of the image.

Procedure:

(1) AC values are de-quantized using the quantization tables.

(2) The DC value of the block 7i,i is approximately computed as DC\ti = 8 * p.

(3) Using the approximate DC value DCit\ along with the computed AC values, the approximate pixel values of the block 7i,i are obtained using IDCT.

(4) The error in pixel values of the block 7i,i due to approximation of DC\,i is adjusted, by adding delta error Siti = p — / i , i ( l , 1), to all the pixels of block.

(5) The top-left pixel of the block I\$. is predicted using linear extrapolation its neighboring pixels available from already decoded block J^i ,

/ 1 , 2 ( l , l ) = 2 * / i , 1 ( l , 8 ) - J M ( l , 7 )

(6) If the predicted pixel value /i,2(l, 1) is less then 0 it is approximated to 0, and if it is greater then 255, it is approximated to 255.

(7) Using the predicted top left pixel ii,2(l, 1) along with the AC values the entire block 7i,2 is decoded similar to 7i,i.

(8) Similar to decoding of I\$ all blocks in the first row of the image are decoded sequentially.

(9) The top-left pixel of the block 72,i is predicted using linear extrapolation on its neighboring pixels available from already decoded block 7i,i

/ 2 , 1 ( l , l ) = 2 * / l l i ( 8 , l ) - J i , i ( 7 , l )

(10) If the predicted pixel value 72,i(l, 1) is less then 0 then it is approximated to 0, and if it is greater then 255, it is approximated to 255.

(11) Using the predicted value of the top left pixel i2,i (1,1) along with the AC values the entire block 72,i is decoded similar to 7i,i.

(12) Similar to decoding of 72,i all blocks in the first column of the image are decoded sequentially.

Anil Kr. Yekkala, C. E. Veni Madhavan and Narendranath Udupa 161

(13) The top-left pixel /2,2(1,1) of the block J2,2 is computed using a linear combination of extrapolation and interpolation techniques based on pixels available from the block /1)2 and block i^i-

J2,2(l, 1) = {[2 * /i,2(8,1) - J l i2(7,1)] + [2 * 72,i(l,8) - I2 ,i(l,7)] + [/i,2(8,2) + /2,i(2,8)]}/4

(14) If the predicted pixel value /2,2(1) 1) is less then 0 it is approximated to 0, and if it is greater then 255, it is approximated to 255.

(15) Using the predicted value of the top }eft pixel 72,2(1>1) along with the AC values the entire block 72>2 is decoded similar to Ji,i.

(16) Similar to decoding of 72,2 all blocks in the second row and second column of the image are decoded sequentially, followed by third row and third column and sequentially all the blocks of the image.

3.2. Controlling error propagation due to pixel prediction

Error propagation due to usage of pixel prediction is controlled to large extent, by checking if all the pixel values of each predicted block after decoding are lying between 0 and 255. If a pixel within the block is less then 0, then all pixels in the block are increased by a constant factor, such that the minimum pixel value within the block becomes 0. Similarly, if a pixel within a predicted block is above 255, each of the pixels within the block is reduced by a constant factor, such that the maximum pixel value within the block is 255. It can be observed from Linear Property of DCT transformation, a situation when the minimum pixel value is less then 0, and maximum pixel value is greater then 255 will not arise.

3.3. Extension of decoding procedure for color images

The reconstruction algorithm is extended to colour images by using the decoding scheme for grey image on all the 3 colour components of the colour image independently. Since the pixel changes are smoother in all the three planes of RGB pixel representation, compared to other pixel representations (e.g YC(,Cr), the pixel prediction technique give better results for colour images in RGB colour format. This is confirmed based on the experiments carried out on the approach for predicting images based on AC values, in different colour representations.

4. Results

4.1. Image compression

The compression using Differential pixel encoding has been tested on more then 100 images. The results from Table 2 show 2% to 8% improvement over JPEG compression while maintaining the same noise level measured in terms of MAD (Mean Absolute Difference). The results have been verified at various bitrates and found to be consistent. The results also show that amount of gain in compression achieved for a smooth textured image like Pills (7.17%) is relatively higher to the gain in compression achieved for a highly textured image like Baboon (2.02%). Hence, it can be concluded that the proposed approach gives better results for smoothly textured images.

Table 2. Compression results compared to JPEG

Image («».)

Baboon (512x512)

Bandon (610x403)

Lena (512x512)

Opera (695x586)

Peppers (512x512)

Pills (800x519)

JPEG

68,940

24,635

32,572

60,170

33,754

54,762

Size

Prop. Method

67,406

23,544

30,460

57,871

31,586

50,833

Gain %

2.22

4.44

6.48

3.82

6.43

7.17

MAD

JPEG

7.31

2.02

2.97

3.85

3.46

2.36

Prop. Method

7.32

2.04

2.99

3.85

3.46

2.37

4.2. Cryptanalysis of DC encryption schemes

Experiments have been carried out on more then 100 images.

The results of the experiments on few of the standard images are shown in Table 3. It can be clearly observed that the reconstruction scheme improves the image quality measured using MAD. For

162 DCT Properties as Handle for Image Compression and Cryptanalysis

Original Image } Image with DC j \ values Encrypted I i using stream ciphei i

Reconstructed Image based tm

osetl scheme

.„£na

• • • i H HHENPR

ilfllRllh. -Uppers.

h " i *r":r-

Fig. 3. Cryptanalysis of DC based encryption scheme-

Table 3. Cryptanalysis results of DC encryption scheme

Image

Lena

Peppers

Baboon

Brandyrose

Bandon

Pills

Opera

MAD for DC encrypted Image

53.7

51.2

56.1

59.3

32.4

52.7

59.7

MAD for Reconstructed Image

25.5

14.7

34.9

30.5

11.8

22.3

29.8

example in case of Lena the MAD decreases from 53.7 to 25.5. The reduction of noise can be also observed from Figure 3, it clearly shows that intelligibility of the image has been improved to large extent using the proposed reconstruction scheme. It is also to be noted that the reconstruction of the image was started from the top-left pixel, but the reconstruction can be started from any arbitrary locations and the outputs can be combined to improve the intelligibility of the reconstructed image.

5. Conclusions

In this paper we used linear and orthogonal properties of DCT for achieving an improvement over compression provided by JPEG. In addition we also used the Linear and Orthogonal properties of DCT transformation to demonstrate the vulnerability of lightweight encryption schemes based on encrypting DC values, by reducing the noise considerably introduced by encryption.

References

1. Gregory K. Wallace, "The JPEG still picture compression standard", IEEE transactions on Consumer Electronics* 1991

2. Subhasis Sana, "Image Compression - from DCT to Wavelets : A Review", ACM Crossroads students magazine

3. Salah Alys "Multimedia Security: Survey and Analysis", Multimedia and Networking Research Lab. CTI, DePaul University, Chicago, h t t p : / / vw . mnlab.cs.depaul.edu

Anil Kr. Yekkala, C. E. Veni Madhavan and Narendranath Udupa 163

4. Shuiming Ye, Xinggang Lin, Qibin Sun, "Content Based Error Detection and concealment for image transmission over wireless channel", IEEE 2003

5. Yefeng Zheng, "Performance Evaluation of Spatial Domain Error Concealment of Image Recovery", Project Report of ENEE739M, spring 2002

6. Yao Wang; Qinfan Zhu, "Error control and conceal

ment for video communication: A review", Proceedings of the IEEE v 86 n 5 pt , p 974-997 (1 May 1998)

7. Nick Kingsbury, "Image Characteristics", Version 2.9: Jun 8, 2005 9:10 am GMT-5, source: http://cnx.rice.edu/content/mll085/latest/

164

Genetic Algorithm for Improvement in Detection of Hidden Data in Digital Images

Santi P. Maity* and Prasanta K. Nandi

Bengal Engineering and Science University, Shibpur Howrah, West Bengal

India E-mail: [email protected]*, [email protected]

Malay K. Kundu

Center for Soft Computing Research, Indian Statistical Institute,

Kolkata, West Bengal India

E-mail: [email protected]

The paper proposes a data hiding scheme in digital images to serve the purpose of authentication and covert communication. The algorithm is implemented in three stages. In the first stage, a region for data embedding is selected in such a way so that maximum number pixel values of the cover image differ from the message pixel values by less than a predefined threshold and a difference signal (D) is formed. The difference signal modulated by proper embedding strength is then added to the cover to achieve stego image. In the second stage, Genetic Algorithm (GA) is used to achieve a set of parameter values (used as Key) to represent optimally this difference signal. In the third stage, the parameter values at the decoder end form an approximate version of the difference signal using linear interpolation scheme and subsequently the embedded data is decoded. The novelty of the proposed scheme lies in higher payload capacity without compromising much for visual and statistical invisibility of the hidden data. The later property is verified as the proposed embedding process, unlike many other reported algorithms, causes very small change in the higher-order statistics of the wavelet coefficients for the stego data. Experiment results also show that the algorithm is robust to non-malicious operations like mean and median filtering, additive noise and moderate image compression.

Keywords: Digital data hiding; Difference signal; GA; Statistical invisibility.

1. Introduction

Data hiding is the art of invisible communication that embeds a secret message into innocuous-looking cover objects such as simple text, images, audio or video recordings. The technique may serve the purposes of unobtrusive military and intelligence communication, covert criminal communication, protection of civilian speech against repressive government. Classical steganography deals with the methods of embedding a secret message, a copyright mark or a serial number in a cover message by slightly modifying the bits in the data. After embedding, the cover object becomes known as the Stego-object. This embedding is characterized by a Key, some data, without the knowledge of which, it would be extremely difficult to detect or remove the embedded material. The essential requirements of image data hiding are visual imperceptibility of the hidden data, security against statistical analysis and robustness to non-malicious operations that a communication channel is to face. The processing may include compression for efficient storage and transmission, mean/median

filtering for the purpose of noise cleaning. However, the degree or depth of this signal processing operations should be restricted to a level so that the stego-object must preserve its commercial value. A data hiding method will be called secure if the stego-objects do not contain any detectable artifacts due to message embedding. In other words, the set of stego-objects should have the same statistical properties as the set of cover-objects. If there exists an algorithm that can guess whether or not a given object contains a secret message with a success rate better than random guessing, the data hiding system is considered broken.1

Literature of digital image staganography is quite rich. Most of these techniques utilize the LSB (least significant bit) embedding applied either directly to pixel values, to the indices in palette images (EZ Stego), or to the quantized DCT coefficients for the JPEG format (J-Steg, JPHide & Seek, Out-Guess).2 Although messages embedded into an image are often imperceptible to the human eye, they often disturb the statistical nature of the image. Farid

Santi P. Maity, Prasanta K. Nandi and Malay K. Kundu 165

studied for a broad range of natural images and reported that there exists strong higher-order statistical regularities among the wavelet coefficients.3If a message is embedded within an image, these statistics are significantly altered. So the design of image steganography algorithms, for a given auxiliary message, may be directed towards the efficient selection of the embedding-region within the cover and subsequent modulation process so that visual and statistical invisibility of the hidden data is well maintained.

Payload becomes an important requirement when information hiding is used for covert communication. However, higher embedding capacity obviously affects visual and statistical invisibility of the hidden information along with robustness performance. These requirements form a multidimensional nonlinear problem of conflicting nature. Genetic algorithm (GA), like an efficient search for optimal solutions in many image processing and pattern recognition problems, can also be used in this topic of research but in reality the usage of the tool has been explored very little. Maity et aZ.4propose an algorithm for data hiding in digital images where GA is used to find out parameter values, namely reference amplitude and modulation index. The performance of the algorithm is compared for linear, parabolic and power law functions used for modulating the auxiliary message.

In this work, Genetic Algorithm is used to achieve a set of parameter values (used as Key) to represent optimally the difference signal (D) that is obtained by subtracting the pixel values of the auxiliary images (messages) from that of the pixel values of the cover image. The difference signal with its proper embedding strength is added to the respective cover data. The parameter values at the decoder end are used to form difference signal using Linear Interpolation method. Message is recovered by doing inverse operation using this difference signal. Experiment results show that the hidden data is secured against statistical analysis and robust to various non-malicious operations.

The rest of the paper is organized as follows: Section 2 presents problem definition and scope of the work. Section 3 briefly describes steganalysis method based on higher order statistics and section 4 presents proposed data embedding method. Section 5 describes performance evaluation while conclusions are drawn in section 6.

2. Problem definition & scope of the work

Different data hiding applications require different payload, typically varying from a few bits in access control, up to at most one hundred bits in authentication and fingerprint problems, may demand much higher payload capacity for information-hiding applications. The last application can be used for covert communication and objective is to embed data in such a way without compromising much of image fidelity and decoding reliability. In additive embedding process, data hiding is accomplished by adding to the host data a scaled version of the auxiliary message. Image fidelity is degraded with the increase in pay-load capacity and embedding strength. The higher value of embedding strength, at the cost of greater visual distortion, increases reliability of decoding.

One possible solution to cope up this trade-off like problem is to map the auxiliary message signal to a difference signal. This is formed by selecting the embedding region within the cover signal so that auxiliary message forms a lower distance difference signal. Now the difference signal can then be added with higher embedding strength so that visual distortion of the cover can be set to an acceptable value. Decoding of message needs the regeneration of difference signal and inverse process will extract the message signal. The reliability of the decoding process depends on how faithfully the difference signal is regenerated. The best decoding is possible if complete knowledge of the difference signal is available at decoder end which may be treated as an overhead problem. To reiterate the problem, the higher the payload capacity, the more number of signal points are needed for regeneration of the difference signal. Thus important point arises how to select those N-signal points that regenerate the M-point difference signal faithfully where N <c M. This can be thought as an optimization problem and GA finds application to yield optimal solutions.

The above problem can be stated mathematically as follows: Given an M-point difference signal (£>), how an approximate signal (D ) be generated using N- signal points where N -C M and (D ) is close resemblance of (D). One way to regenerate better approximation signal using higher order interpolation but in that case computation cost increases

166 Genetic Algorithm for Improvement in Detection of Hidden Data in Digital Images

exponentially with order of interpolation. Linear interpolation is a good compromise between the computation cost and better approximation for regenerated signal. In such case, it is the important point to find which iV-points would generate better approximation and how TV-values affect this approximation function. This is an optimization problem and GA finds application to yield optimal solutions.

3. Steganalysis based on higher order statistics

Farid proposed a universal blind steganalytic detection method3 based on higher order statistics of natural images. We briefly describe two steps as fol-lows:The detection scheme can be separated into two parts : (1) Extraction of a set of statistics, called the feature vectors, for each investigated image. (2) Formation of a Classification Algorithm to separate untouched images from stego-images on the basis of their feature vectors.

Formation of feature vectors

The feature vector, for a certain image, is formed using the first four normalized moments namely Mean, Variance, Skewness and Kurtosis of the wavelet coefficients of vertical (vi[x, y]), horizontal (hi[x, y]), and diagonal (di[x,y]) subbands at scales i — l,2,3....n. Thus we have total 4.3. (n-1) elements of the feature vector / for the test image.

The remaining elements of / are derived from the Error Statistics of an Optimal Linear Predictor. The prediction for a specific subband coefficient is performed considering the 4 neighboring coefficients, the corresponding in the courser scale of the same orientation and coefficients of subbands of other orientations (and scales). The predicted value for the vertical subband Vi[x,y)] is given by

v[ = wiVi[x- l,y] + w2Vi\x + l,y] + w3Vi[x,y - 1]

+ w4Vi[x, y + 1] + w5vi+i [x/2 + y/2] + w6di[x, y]

+ w7di+1[x/2,y/2] (1)

where uik denotes the predictor coefficients. The other two subbands h[x, y)] and d[x, y)} can be predicted in the same way. The optimal predictor coefficients Wk,opt are determined for each subband so that the mean squared error within each is minimised. The log error in the linear predictor is then given by:

eVtlog[x,y} = \og2(vi[x,y}) - log2(v'i[x,y}) (2)

where Vi[x, y] is obtained by inserting Wk.opt into Eq. (1), yields the Log Error. The mean, variance, skewness, and kurtosis of the Log Error of each subband form the remaining 4.3.(n-l) elements of / .

Classification Algorithm

The classification algorithm called the Fisher Linear Discriminant Analysis is used to classify a new image by means of its feature vector. The FLD algorithm is first trained with feature vectors from untouched and stego-images. The algorithm determines a projection axis by means of this training set to project the 24.(n-l)-dimensional space of feature vectors into a one dimensional subspace. The projected feature vector f is referred to as the Detection Variable d. New feature vectors obtained from new images are classified by thresholding d. If d is greater than a certain value, the image is classified as stego, if not, it is classified as untouched.

4. Proposed method

The total process of data embedding and decoding consists of three stages. These are Stage 1: Selection of data embedding regions and formation of difference matrix followed by data embedding, Stage 2: Generation of set of points using GA to optimally represent the difference matrix, Stage 3: Message retrieval. Fig. 1 represents the flowchart for the main module of the proposed work.

A. Stage 1

The choice of cover images is important and influences the security in a major way. Images with a low number of colors, computer art, images with a unique semantic content, such as fonts, should be avoided. Some data hiding experts recommend grayscale images as the best cover-images. We consider gray scale image as cover and the similar type image like text information as message signal since it can preserve contextual information even after various signal processing operations. Steps for the selection of embedding region are as follows.

Step 1 : Input gray scale images as cover and message signal. Step 2: Setting of an appreciable Percentage for matching criteria (82 percent) Step 3 : Selection of a region from the Cover equal

Santi P. Maity, Prasanta K. Nandi and Malay K. Kundu 167

( S M )

Enter the Cover&le in .dat fiom

Enter the Mes sazefik in dat from

Enter the rnaxnumGenezatiDn mint e m

I - IiutiafcE veneration nurtb er e= 1

i '

Window Selection

" Initial™ Pouuktion

z Evaliate fitness of each individual

J PeiformS elect mates

I PeribimCros s o ve r

i PetfomiMutation.

1 r

Stone New Population. s=s*i

Fig. 1. Flowchart for the main module of the proposed work

in the size to that of message signal. Step 4 : Comparison of variation of the pixel values between the cover and message signal. Step 5 : Repetition of the above process by dynamically selecting windows all over the cover image. Step 6 : Once the Percentage matching criteria is satisfied, the process terminates, otherwise, it is continued till the end of the cover image.

Step 7 : Output: (1) If percentage matching criteria is satisfied, then returns the Difference Matrix to ensure a smooth image with little variations. (2) If no matching region is found, returns a null matrix denoting failure of finding the specified percentage matching region.

The difference matrix (D) is then multiplied by proper embedding strength (K), and added to the respective pixel values of the cover image (C). The

168 Genetic Algorithm for Improvement in Detection of Hidden Data in Digital Images

stego image can be obtained as follows:

S = C + K.D (3)

B. Stage 2

The main objective is to find an optimal set of points using GA5 so that an approximate version of difference signal can be generated. 1. Initialization of population: Chromosomal representation of the parameter values. The initial population is formed by taking almost equi-spaced x-y data points with small perturbations. 2. Select Mates: Objective : To select, most of the times, the best fitted pair of individuals for crossover. Step 1 : Input : Population Step 2 : Best fitted pair of Individuals are chosen by Roulette-wheel selection process by adding up fitness values of the individuals to get Sumfitness Step 3 : Then randomly select Individuals to cross 50% of the Sumfitness value in a cumulative way Step 4 : The particular Individual which crosses the 50% criteria in the Cumulative process, is chosen to be one of the mating pool pair. Step 5 : This process is again carried on to find another Individual of the mating pool Step 6 : Output : Pair of Individuals or mating pool

3. Crossover: Objective: To find the Crossover site and to perform crossover between the Mating pool pair to get new pair of more fitted Individuals. Step 1 : Input: Mating pool pair Step 2 : Finding the crossover site in a random manner Step 3 : Exchange the portions lying on one side of crossover site of those mating pool pair Step 4 : Output: New pair of Individuals 4. Mutation: Objective: To mutate or change a particular bit or allele in a Chromosome with a very small probability. Step 1 : Choose a very small mutation probability Step 2 : Depending upon that probability value, change a bit from ' 1 ' to '0' or '0' to ' 1 ' Step 3 : The bit position selected for mutation is the Crossover site 5. Objective function: Objective : To estimate the fitness value of an Individual Step 1 : Input: Population Step 2 : On each Individual of the Population apply 2-D interpolation technique to approximate the original matrix Step 3 : The absolute mean error is evaluated by

subtracting the interpolated matrix from the original matrix Step 4 : The inverse of that absolute mean error is considered to be the fitness value of that particular Individual

C. Stage 3

The final stage of the algorithm is the retrieval process of the message. An approximate version {Dapp) of difference signal (D) is obtained using Linear Interpolation technique among the N points grayscale values calculated by Genetic Algorithm. The approximate cover image matrix Capp is then calculated using the Stego-image S and Dapp as follows:

(sapp = '-' — K.Uapp (4J

The message can be retrieved from the following relation:

IJ : = : {sapp Uapp \ )

where Dapp = K.D.

5. Performance evaluation

The efficiency of the proposed algorithm is tested by embedding difference signal in several cover images. The visual quality of the watermarked image is represented by peak signal-to-noise ratio (PSNR). The watermarked image is shown in Fig. 2(a) and PSNR value for the watermarked image is 39.33 dB. As the number of generations are increased from 1000, 2000 to 4000, observation of Figs, (c), (d) and (e) reveals the fact that the retrieved watermark images are becoming more and more close to the original images. We emphasize on subjective quality for the recognizability of the extracted message rather than any objective measure. The improvement in decoding is borne out by the property of Genetic Algorithm which produces better solutions as the number of generations are increased. The number of parameter values considered here are (20 x 20). Sometimes various linear and nonlinear filtering are used to remove noise from the images. Median filtering is one non linear filtering that removes noises but preserves edge information. Fig. 3 shows the retrieved watermark messages from the median filtered version of the watermarked images. The stego image is filtered using (3 x 3) window. Figs, show how the quality of retrieved messages are improved with number of

Santi P. Matty, Prasanta K. Nandi and Malay K. Kundu 169

(d) (e)

Fig. 2. (a) Watermarked image; (b) watermark image; (c)-(e) retrieved messages after 1000, 2000 and 4000 iterations respectively

iterations although the number of parameters value

remain same.

* infe ify (a) (b) (c)

Fig. 3. Robustness performance of median filtering; (a),(b) and (c) indicate retrieved watermark messages after 1000, 2000 and 4000 iterations respectively

Fig. 4. Stego-test; Black cross-stego test image using proposed algorithm; (b)Colour circles-Sample stego images after various operations; (c)red square-untouched image

Fig. 4 shows the stego-test results. As we ob

serve, this stego-test does not place the test image

within the cluster for sample stego-images. There

fore, the test cannot decide with certainty tha t the

test image is actually a stego-image. This estab

lishes the security of the proposed algorithm against

Farid's steganalytic technique involving higher-order

statistics.

6. C o n c lu s io n s

A da ta hiding algorithm with improved payload ca

pacity is proposed for digital images. GA is used to

achieve a set of parameter values so tha t faithful de

coding of message is possible. Decoding reliability is

improved with the increase of number of iterations in

GA when the set of parameter values are fixed. The

algorithm is proven to be secured against stego-test

based on higher order statistics.

References

1. R. J. Anderson and F. A. Petitcolas , On the limits of steganography, IEEE Journal on Selected Areas in Communications, 16 (4)(1998).

2. R. Anderson, Information hiding, in Proc. of the 1st Workshop on Information Hiding, LNCS-1174, Springer Verlag, (New York, 1996).

3. H. Farid, Detecting Steganographic Message in Digital Images, Dartmouth College, Computer Science, (TR2001).

4. S. P. Maity, M. K. Kundu, and P. K. Nandi, Genetic algorithm for optimal imaperceptibility in image communication, in Proc. 11th Int. Conference on Neural Information processing, ( Kolkata, India, 22-25 Nov. 2004).

5. D. Goldberg, Genetic Algorithms: Search, Optimization and Machine Learning, (Addison-Wesley, Reading, M. A., 1989).

170

High Resolution Image Reconstruction from Multiple UAV Imagery

Jharna Majumdar, B. Vanathy and Lekshmi S.

Aerial Image Exploitation Division, Aeronautical Development Establishment (DRDO),

C. V. Raman Nagar Bangalore - 560 093, India. E-mail: [email protected], [email protected]

The objective of this work is to study and develop a suitable algorithm for High Resolution (HR) reconstruction of images from a sequence of Low Resolution (LR) images obtained from an aerial reconnaissance platform. The aerial images are motion blurred, translated and rotated with respect to each other and have perspective distortion due to the forward-looking sensor. The paper proposes frequency domain approach and feature based approach for image registration followed by a Bayesian MAP estimate for the reconstruction of the high resolution image.

1. Introduction

High Resolution reconstruction is a major area of research in allied branches of science like Artificial Intelligence, Machine Vision and Medical Imaging. With the advancement in the field of Computer Science, it has become possible to explore new algorithms for restoration, using multiple image sequences.

Image resolution depends on the physical characteristics of the imaging system viz., the optic, density and the spatial response of the detector elements. The commonly available imaging systems do not sample the scene according to the Nyquist criterion. As a result of this, high frequency contents of images are destroyed and they appear to be of lower resolution. The image features may be lost considerably if the aliasing is too much i.e. if the image is of a very low resolution.

To improve the resolution of the image, attempt is to be made to recover the lost frequency components. Interpolation is a process used to estimate the signal values in between the sampled values without any further knowledge about the signal. Any number of functions can be generated to approximate the actual function representing the signal. With only the sampled points given, the accuracy of the interpolated pixels depends on how close the estimated signal is to the actual signal.1

Interpolation techniques such as bilinear, bicubic, bicubic splines etc. have been quite common and provide better resolution. However, the number of constraints they impose upon the solution inherently limits all these methods. High Resolution techniques are superior over the interpolation techniques because of the additional constraints imposed on the

HR image after it has been simulated through the imaging process and temporally correlated frames.

2. Operational Scenario

In the present scenario, camera is mounted on the aircraft, which captures continuous scene of the terrain during flight. The images suffer from perspective distortion due to the forward-looking sensor, blurring due to ego motion and PSF. The consecutive image frames are also spatially translated and rotated with respect to one another. Random noise is also added to the down sampled image while acquisition. The aerial images obtained can thus be expressed by the following equation.

gk (m, n) = ak{h[f (x, y)} + rjk (x, y)} (1)

where g^ is the kth observed image frame, / represents the original scene, h is the blurring operator, r]k

is the additive noise term and ak represents a nonlinear function that digitizes the image into pixels and quantizes the resulting pixel values from intensities into gray levels.

3. Modeling of the problem

The HR reconstruction algorithm is described in the following steps:

(1) Estimating a HR image. (2) Simulating the HR image through the imaging

process to yield multiple LR images. (3) Developing a cost function from the estimation

and the measurements. (4) Minimizing this cost function to get an appro

priate HR image.

Jharna Majumdar, B. Vanathy and Lekshmi S. 171

A good estimation of the HR image helps in minimizing the run time of the machine and brings out the convergence in less number of iterations. For the present work, bi cubic interpolation is used as the initial estimation of the HR image.

Simulation of the imaging process forms the backbone of the proposed algorithm. The proximity between the simulation process and that occurring in real life determines the error in the final result and is a crucial factor in establishing good quality HR images. The three steps involved in the simulation of the imaging process are (a) warping the HR image, (b) blurring the HR image and (c) down sampling the HR image to get LR images. Since the process is iterative, the HR image in the above-mentioned steps changes in every iteration and is the best estimate obtained till that stage. In this paper, an algorithm using Bayesian MAP approach is proposed.

3.1. Registration

For this algorithm, it is necessary to estimate the sub pixel motion very accurately by registering the images. This paper uses frequency domain approach2

and feature based approach34 to accomplish the task.

3.1.1. Registration by frequency domain approach

If Fi and F2 are the fequency domain representations of two images that are translated by (X0, Y0) and rotated by 0O with respect to each other, then F\ and F2 are related by the following equation:

F2(u, v) = exp(-j27r(ux0 + vy0)) (2)

F\ (u cos 0O + v sin 0o, —u sin 90 + v cos 60)

If the magnitude in (u,v) domain is changed to (p,0), equation (2) is obtained in the following form:

M2(P,9) = Ml(p,9~eo) (3)

d0 can be separated from the above equation using phase correlation method. The cross power spectrum (i.e. the phase correlation) of the two Fourier transform Fi and i<2is defined as:

F{u,v)F'*{u,v) , . n , „ ,,. • P, \j,,( x| = exp027r(ua:o + vVo)) (4 \F(u,v)F'{u,v)\

Inverse Fourier transform of the right-hand side term of the above equation give a peak at 60, the rotation angle. The image is warped by the rotation angle

so that only translation component remains between the images. This is given in equation below:

F2(u,v) = exp(-j2-ir(ux0 + vy0))Fi(u,v) (5)

Again by taking phase correlation and then inverse Fourier transform, translation parameters xo and yo can be obtained. To get the motion parameters to sub pixel accuracy, the image is first interpolated using bi cubic interpolation and then the parameters are found.

3.1.2. Registration by feature based approach

Feature extraction is an important step in feature based image registration methods. Feature based methods rely on accurate detection of features. A feature is the result of an interpretation of n pixels in a window of (p x p), where p is the size of the window.

The feature based registration scheme involves extracting feature points in the pair of images to be registered, reducing the number of points to retain dominant features and matching the pair using suitable matching techniques. Usual features are edges, corners, junctions, close connected regions or segmented texture regions of the image.

Feature Extraction

In this paper, we have used Harris corner detector as feature extractor. The Harris corner detector is based on an underlying assumption that corners are associated with the maxima of the local function. It is less sensitive to noise in the image since the computations are based entirely on first derivatives.

The Harris corner detector computes a corner-ness value, C(x,y) for each pixel in the image, I(x,y). A pixel is declared as a corner if the value of C(x,y) is below a certain threshold. The value of C(x,y) is computed from the intensity gradients in the x and y direction as follows

C(x,y) = detC s t r — a(traceCg t r) (6)

The local structure matrix Cst ris computed as follows.

C3tT = WG(r;cr) x Jx Jxjy

Jxjy Jy (7)

172 High Resolution Image Reconstruction from Multiple UAV Imagery

where fx and fy denote the first derivative along the x and y direction respectively of the point f(x,y) in the image and v/o(r;a) is a Gaussian filter of selected size, a.

Feature Reduction

After feature extraction, feature reduction is done to retain only dominant features suitable for registration. A large number of feature reduction methods exist, out of which Cornerness Similarity measure is the one which is chosen in this paper. Cornerness is the characteristic property of any interest point (feature point) P and is defined as Cp=| Af + A§ |. To measure the correspondence between 2 points, P and Q in two images, the similarity measure S (P,Q) is defined as S(P,Q) = min(Cp,Cq)/max(Cp,Cq). A point is considered as a good feature if, S(P,Q) >Tc where Tc is a variable threshold.

Estimation of Homography and computation of transformation parameters

Homography is the transformation function that finds the corresponding features and mapping between the coordinate systems of the two images. In this method of matching, the feature points and determining the homography, two pairs of feature points are chosen randomly from the image pair at a time. The homography between the selected feature points is computed and the correctness of the computed homography is checked using a scoring module.

This process is iterated until a good score is obtained or for a fixed number of iterations. The homography or transformation involving two feature point sets is a combination of translation in the x-direction (Tx), translation in the y-direction (Ty), scaling factor (s) and angle of rotation (9 ). This is shown below.

H sCosO —sSinO Tx

sSinO sCosO Ty

0 0 1 (8)

For a pair of corresponding matched points, the homography computation is done to calculate the transformation parameters.

3.2. Reconstruction

3.2.1. Restoration

Restoration is the first step of reconstruction. It is an estimation process that attempts to recover an ideal high quality image from a degraded image.5'6

Restoration is applied to remove.

• Degradation such as blurring due to optical systems aberrations, atmospheric turbulence, motion and diffraction.

• Statistical degradation such as noise and measurement errors.

These two types of degradation lead to conflicting requirements on the restoration filter. To simplify the process, additive noise has been neglected in the present case and only the multiplicative motion blur has been considered. The PSF has been considered to be square Gaussian.

The Optical Transfer Function (OTF) for de-blurring can be written as:

H{u,v) = • -.— -^

v / ^ / ^ e x p (- (^fl±^l j + i (9)

where ax and cr}. are tunable parameters chosen judiciously according to the image. At this stage, we have an initial estimate of the HR image and the optical flow parameters between the LR images. The remainder of the process consists of developing a cost function using the MAP estimate and then minimizing it7.8

3.2.2. Cost Function

Given that we observe "y" (the LR images), a MAP estimate of the High Resolution image Z is formed. The estimate can be computed as shown below.

Z = a,rgmaxzPr(z/y, s) (10)

Here, we are trying to maximize the probability of occurrence of High Resolution pixels given the measured Low Resolution pixel values and the accurately determined optical flow parameters. The value of Z pixels, which maximizes this probability, forms the High Resolution image. Using Baye's Theorem this can be expressed as following:

Pr(y,s/z)Pr(z) Z — argmax0 Pr(y, s) (11)

Jharna Majumdar, B. Vanathy and Lekshmi S» 173

Size*60x60

Fi& 1 Input image

Size=l 20x120

Fig. 2 Input image pair after scaling

i) U singF ourier method R owshift = 4.00 C dumnshift = -12.0 0

(a) Iteration 1 (b)Iteratirai2 (c) Iteration 3 Fig3 Reconstructed images obtained by Fourier based method

if) UsingFeature based method Ro wiMft =4.8 C olumnshift - -15.0 4 Rotation-1.14

parameters for the cost function. The cost function is as follows:

j 2

l ^—^ j

Urn ~~ [(/ ^m1r9r

r=l,AT W = 5 E

m=l,pm + (13)

9 2lf r=l,AT

4. Resul ts

The algorithm described above is implemented in a Pentium IV PC with 512 MB RAM. The test images are the aerial images obtained during actual flight of the aircraft. The registration algorithm uses both Fourier and Feature based approach. The reconstruction algorithm is applied for some selected target regions of interest using both Fourier and Feature based approach. The test set, reconstructed images and the corresponding time required to perform the tasks are given below in Figure 1 and the time chart is given in Table 1.

( aj Iteration 1 (b) Iter ati on 2 ( c) I ieraticn 3 Fig.4 Reconstructed images obtained by Feature based method

Fig. 1. Reconstructed images obtained by Courier based and feature based methods.

Table 1. Time chart for generation of outputs.

Iteration number

Iteration 1 Iteration 2 Iteration 3

Frequency based approach time

(in sees) 337 307 307

Feature based appraoch time

(in sees) 57 58 56

The Probability Distribution Function (PDF) is assumed to be Gaussian because of its desirable property of giving a unique global minimum. The PDF of High Resolution pixels is therefore:

Pr(z) = Kexp - z? (12)

where K is a normalizing constant and of are the covariance factors which can be used as tunable

5. Conclusion

The algorithm proposed in this paper shows that aliasing reduction and resolution enhancement can be achieved by using multiple frames that are rotated and/or translated with respect to each other.

One of the applications of this work deals with the real time processing of digital video in which the primary objective is to use multiple frames to produce a video sequence of high-resolution frames. Another application of this research is the post light analysis of video scenes acquired for surveillance purposes where detailed knowledge of the structure of the scene is required.

6. Acknowledgement

The authors wish to express their heartfelt thanks to Director ADE for his kind permission to publish the paper.

References

1. Michael Irani and Shmuel Peleg? CVGIP .'Graphical Models and Image Processing 53, (1991)

2. B. SriniYas Reddy and B. N. Chatterji, IEEE Trans, on Image Processing 5, 8 (1996)

174 High Resolution Image Reconstruction from Multiple UAV Imagery

3. Jharna Majumdar et al,SPCOM (2004) . ()• 4. J. Shi and C. Tomasi,IEEE Conference on Computer

Vision and Pattern Recognition , (1994) 5. R. Tsai and T. Hua,ng,Advances in Computer Vision

and Image Processing 1, (1984)

6. Hardie et al.,Optical Engineering , (1998) 7. Hardie et al.JEEE Thins, on Image Processing , 6

(1997) 8. Hu He and L. P. Kondi, Journal of Electronic Imag

ing 13, 3 (2004)

175

Image Registration and Object Tracking via Affine Combination

Nilanjan Ray Department of Computing Science

University of Alberta 2-21 Athabasca Hall, Edmonton, Alberta T6G 2E8, Canada

Email: [email protected]

Dipti Prasad Mukherjee Electronics and Communication Sciences Unit

Indian Statistical Institute 203 B T Road Kolkata, West Bengal 700108, India

Email: [email protected]

In this paper we illustrate image registration and object tracking by the use of affine combination of pixel locations. To resolve the pixel location correspondences, we interpolate the image intensity at locations produced by the affine combinations of a set of ordered pixels. Then we show that establishing pixel correspondence between images undergoing affine transformation is a matter of point-to-point comparison of the interpolated image intensities over the affine combinations. Through the use of the affine combination we illustrate how optical flow based technique can be employed for tracking targets, the motion for which is assumed to be affine. When the target motion model is that of a rigid body, we illustrate how the rigid body constraint can be easily accommodated within the same optical flow based computational framework. The contribution of this work is to rigorously establish affine combination as a registration and tracking tool.

1. Introduction

Resolving point correspondence is one of the most fundamental problems that arise in a gamut of image analysis applications, such as, image registration, stereo pair matching, structure from motion, object tracking and so on. Often the transformation model between the images (at least locally) is granted to be an affine transformation, especially when the distance between the camera plane and the object is much more than the dimensions of the objects/structures themselves. Thus affine transformation invariant descriptors' emerge as important tools for finding point correspondences.

Typically an affine descriptor is computed as a window around an interest point (such as, detected corner point, local minimum point within a region, and so on). Examples include maximally stable extremal region2, edge-based and intensity extrema-based region3, Harris -affine detector4 among many others'.

In this paper we concentrate on analyzing a type of affine descriptor that Tell and Carlsson describe as image intensity profile between two interest points5'6. We point out here that this descriptor belongs to a class of affine descriptors characterized by the affine combination7 of interest points. The purpose of this paper is to formally establish affine combination as an affine invariant descriptor that holds certain desirable properties to qualify as a tool in establishing pixel or point correspondence. To this end we show that affine

combinations need not be limited to the intensity profile between two interest points; rather more general descriptors can be formed by more than two interest points. One important property of the affine combination descriptors proposed here (as well as in5'6) is that unlike the other aforementioned affine descriptors it allows point-to-point image intensity comparison for matching interest points. The point-to-point image intensity comparison leaves less possibility for false positive matches.

We illustrate object tracking as another application of affine combination here. The proposed tracking framework with affine combinations makes use of object templates that incorporate both object shape and texture in dense optical flow computation.

2. Affine Combination and Pixel Correspondence

Let X (2-by-l column vector) be a pixel location. The 2D affine transformation of X is given by X' = AX + f ,

where A is a 2by-2 matrix and / i s a 2-by-l column vector; together A and / represent a 2D affine transformation. For a set of points Xu X2, X3, ..., X„, a

linear combination: \aX is called affine combination

when the scalar coefficients sum up to unity7: V t t = i. ;=1

Below we prove certain properties of affine combination

176 Image Registration and Object Tracking via Affine Combination

with regard to affine transformation that will be useful in resolving pixel correspondence.

Let / be an image. Let / ' be the image formed after applying a 2D affine transformation to I. Let X\ and X2 be two pixels (2-by-l column vectors) on image /. Let the corresponding pixels for X\ and X2 after the affine transformation on / ' be respectively X[ and X'2. Then:

X\ - AXt + f and X'2 = AX2 +f, A and /characterize the 2D affine transformation between the two images. The affine combination in this case is X, + /(X2 - X , ) ,

where ?e9t This affine combination defines a straight line passing through X and X2. The following proposition establishes the invariance property of this affine combination. Propla The corresponding point of X, +t(X2 - X , )

after affine transformation is X[ + t{X\ — X[) for any given value of the parameter /, where the corresponding points of Xi arid X2 are given by X\ and X'2

respectively. Proof. To prove the proposition let us consider the identity:

X[+t(X'2-X\) =

AX)+f+t(AX2+f-AXt-f) = A{X^t{X2-X{))+f.

• The next proposition proves the converse of Propla Proplb. If the ordered point pairs <XU X2) on the first image and (X*,X2) on the second image are not

corresponding pairs in an affine transformation, then X , + / ( X 2 - X , ) and X\+t(X'2-X\) are not corresponding points for almost all te% Proof. To prove Proplb, let us assume that the corresponding points ofX, andX2 are given respectively by X[ and X'2 on the second image. By Propla, the

corresponding point of Xi+t(X2-X]) is

X[+t(X'2 — X,'). Thus we need to prove that

X' + tiX^-X',)^ X' +t(X2-X') for almost all

values of t. So, let us examine the solution set of / for

which the equality holds:

x']+t(x'2-x'l) = x"[+t(x;-x;). This equation can be rearranged as:/(X2' - X[-X\ + X') = X,* -X[. By the premise of Proplb (X'X2)*(X*X2*). So both

( X ^ - X . ' - X ' + X*) and X,*-X,' cannot be simultaneously zero vectors. Therefore the solution set of t(X2 - X; - X2* + X') = X\ - X,' for / will either be

empty, or a singleton set {i.e., a unique solution). Thus at most for a single value of /, the corresponding point of X, +t(X2 - X , ) will be X\ +t(X2 - X , ' ) . From the measure theoretic point of view a singleton set is of length zero. Therefore ( l - f )X,+/X 2 and (1-/)X* +/X* are not corresponding points for almost

all /e 91 • Let us now extend the two propositions for ordered point triplets. As before we assume that after the affine transformation the corresponding points ofXi,X2 andX3

are X\, X'2 and X3' respectively. We consider the

affine combination of the three points with two scalar parameters t and s, and prove the following propositions Prop2a and Prop2h

Prop2a The corresponding point of

X, + t(X2 - X,) + s(X3 - X,) after affine

transformation is X[ + t(X'2-X[) +s^-X{) for

every [/ ^feSR2. Proof. Similar to the proof of Propla •

The converse proposition is as follows. Prop2b. If the ordered point triplets (X\, X2, X3) on the first image and (X*,X*,X*) on the second image are

not corresponding triplets, then

X 1 + r ( X 2 - X 1 ) + s ( X 3 - X 1 ) and

X' +t(X'2 - X^) + s(X'} - X*) are not corresponding

points for almost all [/ s]T€ 912. Proof. Let us assume as before that the corresponding points of X , X2 and X3 are given by X\, X'2 and X'3

respectively on the second image. Arguing as in the proof of Proplb, we need to examine the solution set of the following equation:

x;-rt(x'2-x;)+s(x;-x;)= x;+/(x;-x;)+5(x;-x,*),

which can be rearranged as:

[(x2-x;-x;+x,*) (x;-x;-x;+x;)] Ls

x;-x;. Let us denote by

c=[(x2 -x\ -x;+x;) (x; -x ; -x\+x;>] and b = X'~X'r The equation will be consistent (i.e., will have at least one solution) if and only if the rank of the coefficient matrix C equals that of the augmented matrix [C b]1. Let us assume that the equation is

Nilanjan Ray and Dipti Prasad Mukherjee 177

consistent and since C is of size 2-by-2 let us now examine the following three cases: Case I: Rank of C is 2. In this case there will be a unique solution. Case II: Rank of C is 1. In this case the general solution

set can be written as: 4> +

t

s s eNull(C)\

where [f0 %f is a particular solution of the equation, and Null(Q is the null space of the matrix C9 Le., the set of solutions of the homogeneous equation. However, we know that dimension of Nu11(C) is number of columns of C - rank of C. Thus the dimension of Nuli(C) is 1, implying that the solution set in this case is a straight line in 9t*. Case III: .Rank of C is 0. Here both the augmented matrix [C b] and the coefficient matrix C must be zero matrices. This is possible if and only if X\ = X*, X*2 = X* and X*3 = X\. However this possibility is ruled out by the

premise of the proposition. Therefore examining all the three cases, we

conclude that the solution set of [I sf can be at most a straight line. Once again the Lebesgue measure (in this case area) of the solution space will be 0 and clearly insignificant compared to the joint parameter space 9f or a bounded plane in 3t2. •

3. Applications

We describe two applications of the affine combination in image analysis here- registration and tracking.

3.1. Image Registration

We assume as before that images /and / ' are related via an unknown 2D affine transformation, i.e., if X and X'are corresponding pixel locations on the two images respectively, then the intensity invariance holds: / (X) = //(JT/)- Let us assume that a set of pixel locations QCU X2$..., XN) on image /and another set of pixel locations (Yu Y29..., YM) on image / ' are given. We want to find out the correspondence between the two sets under the unknown affine transformation.

Given the propositions about the affine combination discussed in Section 2, we attack this problem by hierarchical matching technique. First, ordered pixel pairs

' from one set are matched against the ordered pixel pairs from the other set. In the next stage of matching, from a few best matching point pairs, the ordered point triplets are formed and matched. Note that to find out the unknown 2D affine transformation (six parameters), one needs three pixel correspondences.

To match the ordered pixel pair (Xl9X2) on the image / with the ordered pixel pair (Yu Y2) on image / ' , we compare the intensity signals point-to-point for the affine combination:

A , ,

where Kt is the number of discrete values of the parameter I. Alternatively, we may also consider normalized cross-correlation between I(Xl^t(X2-Xl)) and I\YX + /(72 - 7 , ) ) . More general match measure such as mutual information can also be considered when the intensity invariance assumption does not hold between /and f.

In the second phase of matching, the ordered triplets are to be matched. In this phase, according to the matching scores obtained in the first phase* we choose only a few feature points from both the images. Note the first phase of matching helps to reduce the computational complexity in the second phase. Next, between ordered triplets pf1?X29X3) and (X*9X*2,Xl),

we obtain MAD score: 1 toXt+tjXt-XJ+siXj-XjU

KtKs ^\ixx;+<*; -x;)+s(x; -x;yj[ where Ks is the number of discrete values of the parameter s. Then best matching ordered triplets are chosen from these scores (an example result is provided in Figure 1). If we need more than three points of correspondent second best matching triplets can also be chosen.

Figure i. Three point correspondences found by hierarchical matching to compute prevailing affine transformation between the images.

3.2. Object Tracking

The properties of affine combination can be utilized in tracking objects within the optical flow framework. Let us assume that S^ is the set of pixels (say obtained via segmentation on the initial video frame) belonging to an object that we want to track. Let Xh X2$ and X3 be three comer pixel locations for the bounding box of SAj (The choice of Xu X2, and X2 can be arbitrary so long as they are non-collinear). Let S be the set of parameter

178 Image Registration and Object Tracking via Affine Combination

associated in the affine combination of Xu X29 and X3: S = { (M) :X I +l (X 2 -X l ) + s(X 3 -X 1 )€S o b j }. If the

motion of the object can be approximated by 2D affine transformation, then tracking S^ is merely keeping track of the pixel locations Xu X29 and X3 over the image frames, because the entire object is always designated via the affine combinations over the known set S of parameter values.

To track the vectors Xu X2$ and X3 over an image sequence, we employ optical flow constraint equation-between two consecutive images /(current frame) and/p

(previous frame)8:

\lx(x>y) iy(x>y)\x yY = -it(x*y)> where (x9 y) is a pixel location, and (4, Iy) is the spatial image gradient at (x9y\ I, is the difference image between / and lp. (X, y ) is the velocity of the pixel (x, y). When

we express the optical flow constraint equation via the comer pixel vectorss it takes the following form:

[(\-t-s)Ix tl„ slx (l-t-s)Iy tlv sl\

h x, x, y, yi y-: vj = -*n where the "." above a symbol denotes its first derivative over time (Note that X\ is explicitly defined as (xb y\\X2

and X3 are defined similarly). We solve this equation for the 6 unknowns [i, x2 x3 yx y2 y3J by the

least square method* since the set of (/, s) for which the equation holds is the set S9 and the cardinality of S is typically much larger than 6.

[i, i 2 i 3 yx y2 y3J is solved and *,... etc. are

updated as follows:

[xx(b) x2(b) x3(b) yx(b) y2(b) y3(b)Y =

[xl x2 x3 yl y2 J3] r + ^[i, x2 x3 y} y2 yj,

where b is a non-negative scalar parameter, the value of which, say a9 is chosen by minimizing the energy functional: a = argmin

[/(*, (b)+ t(x2 (b) - x{ (b)) + six, (b) - xx (b)), uhsy, Q>)+Hy1 (b) - y, (b)) + s(yi (b) - j , (b)))-g(t,s)f where g(t, s) is a template function for the object and it can, for example, be the image intensity of the initial

g(t,s) = I0(x? + t(x°2 - 0 + *(*° -*,°),

The superscript 0 designates the vectors (xuy\) etc. for the target on the initial frame 70. Update of the vectors can also be done in an iterative way for better

frame:

convergence. Note that a multi-resolution approach can also be in place for this minimization process for quick convergence. This proposed tracking framework using affine combination integrates both object shape and its texture/intensity within a single template and the computation is that of solving linear equations.

3.2.1 Rigid Body Constraint

Sometimes the frame to frame motion for an object can be approximated via a rigid body motion. In this case we can impose the constraints that the lengths between any two corner points of the bounding box for the object remain constant, i.e.9

(xx-x2)2+(yx-y2)

2 = C P

(x2-x3f-¥(y2-y3f ^C29

(x 3 "»x 1 ) 2 +(^™ J | )2 -C 3 .

When we differentiate these three equations with respect to time and collect them in a matrix vector form, they become:

x 2 - x 3 x 3~x 2 0 J 2 - J 3 J3-.F2

0 x3-xx yt-y3 0 y3-yK_

x3 yx h yJ^i® 0 Of. This rigid body constraint equation can now be concatenated with the optical flow constraint equations and solved with least square method (An example of tracking is provided in Figure 2).

0

xx-x3

h

Figure 2. Tracking objects: four frames arc shown. Affine Mid rigid body models are assumed respectively for the near and the far boat. To handle occlusion, we assume there is a front-to-back order for the objects in along the viewing direction.

Summary and Future Work

Nilanjan Ray and Dipti Prasad Mukherjee 179

This paper establishes the use of affine combination in point correspondence resolution. We show that a point-to-point matching of the intensity profile can compare ordered point pairs or triplets toward correspondence. A hierarchical computational strategy is used here to reduce the search. It is also illustrated that object tracking using affine motion assumption can be performed by the use of affine combination.

In our future endeavor we like to use the affine combination (point-to-point intensity profile) matching in planar graph isomorphism. The planar graph is formed by suitably connecting the interest points (e.g., corner points). Although isomorphism/graph matching is known to be a hard problem, heuristic techniques may yield robust results with the use of the affine combination metric.

References

1. K. Mikolajczyk et al., International Journal of Computer Vision, 65,43 (2005).

2. J. Matas et al., Image and Vision Comp. 22, 761 (2004).

3. T. Tuytelaars, International Journal of Computer Vision, 59,61 (2004).

4. K. Mikolajczyk and C. Schmid, In Proc. Of7'h ECCV, Denmark (2002).

5. D. Tell and S. Carlsson, In European Conference on Computer Vision, 1, 814 (2000).

6. D. Tell and S. Carlsson, In European Conference on Computer Vision, 1,68 (2002).

7. A.R. Rao and P. Bhimasankaram, Linear Algebra, (1992).

8. B.K.P. Horn and B.G. Schunck, Artificial Intelligence, 17,185(1981).

180

Progressive Transmission Scheme for Color Images Using BTC-PF Method

Bibhas Chandra Dhara

Department of Information Technology Jadavpur University, Salt Lake Campus

Kolkata 700098, India E-mail: [email protected]

Bhabatosh Chanda

Electronics and Communication Sciences Unit Indian Statistical Institute

Kolkata 700108, India E-mail: [email protected]

In this paper, a new color image progressive transmission scheme has been proposed. In this method, a color image (RGB) first transformed into (O1O2O3) domain using a reversible transformation. Each image plane Oj (1 < t < 3) is first divided into a number of non-overlapping macroblocks. The size of macroblocks depend on visual sensitivity of image planes (O,). Then each block is decomposed like quadtree fashion based on smoothness criterion. The leaf blocks of size 4 x 4 of each plane are then coded by BTC-PF method. The transmissions of different planes are done in interleaving way. Finally, residual planes are encoded and transmitted accordingly. The experimental result shows that the proposed method gives least bit rate with a reasonably good quality.

Keywords: color image progressive transmission, block truncation coding, pattern fitting, full-search progressive transmission tree, reversible color transformation.

1. Introduction

At the present age of information sharing and communication, Internet becomes more popular. But main restriction of Internet is its bandwidth. Specially during search, transmission of large images in raster scan method take a huge time. Again after the completion of transmission, if found that the image is not the desirable one then it is total wastage of both time and bandwidth. Progressive image transmission (PIT) provides a solution for this. In PIT, images are transmitted stage by stage. In the first stage receiver reconstruct an approximation of the original image and accordingly can decide either to continue or abort the transmission. Tzou in "Ref. 1" presented a thorough review and comparisons of some PIT methods. The PIT methods can be classified into: transform domain methods, pyramid-structure methods and spatial domain methods.

Transform domain PIT methods are mainly based on DCT based methods2 and wavelet based method.3,4 Different pyramid structure based PIT methods are reviewed in "Ref. 5". A method based on Laplacian pyramid is presented in "Ref. 6".

Bit plane method (BPM)1 is the simplest and easiest spatial domain PIT method. But initial image quality is very low, an improved BPM (IBPM)7

has been proposed to enhance the image quality in the initial stage with higher data rate. The bit rate of PIT can be reduced by using vector quantization (VQ) based methods.8"10 Quadtree based PIT methods are proposed in "Ref. 11" to reduce the bit rate. Guessing by Neighbor (GBN) method12 exploits the correlations among neighbor pixels. In "Ref. 13", a PIT method using block truncation coding (PBTC) has been proposed.

Almost all the above mentioned methods are the PIT for the grayscale images. These methods can also be used for color image progressive transmission. In straight forward way, a gray scale PIT technique is employed to each color planes (R, G, B) separately and then information is transmitted from each planes with equal importance. However, these methods do not use the correlation between color planes. To achieve good results, first transform an image from RGB to some other domain XYZ. Then certain PIT methods are used on them and transmit more data from visually sensitive plane compared to other planes. These transmissions of data must be in interleaving fashion according to importance. The BPM and PBTC methods can be used in the first approach and let us call these methods as CBPM and CPBTC, respectively. As the expected bit rate be-

Bibhaa Chandra Dhara and Bhabatosh Chanda 181

comes very high. To reduce the bit rate of PBTC for color images, a method using common bit map for each of three color blocks (CBMPBTC) is presented in "Ref. 14". GBN method addresses the second approach for color image transmission.

In this paper, a new color image progressive transmission method is proposed. In this method first a color image (RGB) is transformed into (O1O2O3) domain using a reversible transformation.15 The organization of the paper is as follows. BTC-PF based PIT is described in section 2. Proposed PIT method for color images are given in section 3. Experimental results are reported in section 4. Finally, conclusions are drawn in section 5.

2. Progressive transmission of BTC-PF coded block

In BTC-PF16 method, a block (n x n) tries to fits with some codewords having Q levels. At time reconstruction of the block, the index of the selected codeword and Q different m ean values fit (1 < t < Q) are required. In this experiment image blocks of size 4 x 4 are coded by BTC-PF method using two level patterns. To reconstruct a block, gray values /ii, /X2 and index / of the selected pattern are required. Instead of m and H2 (A*2 > Mi) t w o values A and d defined as A = ' ^ ^ and d = ^ ^ are used. To transmit and reconstruct the blocks a full search progressive transmission tree (FSPTT) is used. A partial structure of FSPTT is shown in Figure 1. The first level has 8 nodes and second and third levels have 32 and 128 nodes, respectively. In this FSPTT, the intermediate nodes (except root node) are also presents at leaf level. The dominating power of the intermediate nodes ensures their occurrence as leaf nodes. (A, d, I) is transmitted progressively to the receiver. The transmission scheme is described below with an example (see Fig. 2).

186 173 158 152

171 171 171 171

167 169 171 182

172 178 182 182

(a)

171 171 171 171

171 171 171 171

(d) 162 162 162 162

181 185 186 191

171 171 171 171

180 180 180 180

180 180 162 162

170 170 170 170

180 180 180 180

(g)

162 162 180 180

180 180 180 180

(b) 170 170 170 170

172 172 172 172

(e)

180 180 180 180

180 180 162 162

180 180 180 180

172 172 172 172

162 162 180 180

160 160 160 160

162 162 162 162

180 180 180 180

(h)

160 160 160 160

160 160 160 160

(c)

162 162 162 162

180 180 180 180

(f) 180 180 180 180

160 160 160 160

180 180 180 180

Fig. 2. Illustration of progressive transmission of a block coded by BTC-PF. (a) Original image block, (b) reconstructed image by BTC-PF with A=171, d=9 and 1=14 of Fig. 1, (c) -(h) step by step refinement of the block using i*=16 and F i = 8 .

Algorithm-I

Step 1 Transmission of gray level A. From A two values Aq — A/F and Ar = A%F are obtained for some fixed value F (=2fe).

l .a Transmit Aq to the receiver and then reconstruct the block with the value Aq*F and root node of FSPTT as the pattern (see Fig. 2c).

l .b Transmit Ar, refine the reconstructed block with previous pattern and A as gray level (see Fig. 2d).

Step 2 Transmit three bits to represent the intermediate pattern (Pu) at level 1 (assuming root at level 0), along the path from root to the selected leaf node, as the intermediate pattern. Along with the pattern index, transmit d value. Like A value from d calculate dq = d/F\ and dr = d%F\ for some fixed F\.

2.a Transmit dr to the receiver and then reconstruct the block using Pi, A, and dr (see Fig. 2e).

2.b Transmit dq, and refine the reconstructed block with Pi, A, and d (see Fig. 2f).

Step 3 Two bits are transmitted to represent the pattern {Pij) at level 2, along the same path. Reconstruct the block with P2J, A and d (see Fig. 2g).

Fig. 1. A partial structure of FSPTT used in BTC-PF.

182 Progressive Transmission Scheme for Color Images Using BTC-PF Method

Step 4 Finally remaining two bits are transmitted to represent the actual pattern (Pk) at leaf level, selected by BTC-PF, and reconstruct refine the block by Pk, A and d (see Fig. 2h).

3. Proposed progressive transmission scheme of a color image

In color image high correlation exists between R, G, B planes. So RGB are transformed into less correlated reversible triplet (O1O2O3).15 This proposed method consists of two phases. In the first phase, the image planes Oi are coded in lossy manner and are transmitted progressively in inter leave format. In the second phase, the residual planes (Oi) resulted from Oi and corresponding reconstructed planes are coded and transmitted.

3.1. RGB to 0\OiO$ conversion

There are standard color conversions like RGB to YIQ, YUV, YCbCr etc. where YIQ etc. have less inter-correlation among the triplet than RGB. However, since data in RGB and as well as transformed domain are considered to be integer, the aforementioned transformations are, in a sense, lossy transformations. In this work, RGB to O1O2O3 transformation15 is used, which is known to be lossless. RGB to O1O2O3 conversion is given by

^ .R + G + B nc. Oi = [ ^ +0.5J ,R-B

Oi = Y—Y~ + 0.5J O3 = B - 2G + R

and corresponding inverse transformation is

B = Ox - 02 + L ^ + 0.5J - L ^ + 0.5J

G = O!-L^+0.5 j

R = o, + o2 + o3 - [y . + 0.5J - [ Y + 0.5J

where [ x J stands for largest integer not exceeding x. 0\ stands for intensity or luminance component, while O2 and O3 together represent chrominance at each pixel.

3.2. Phase I coding

The first phase consists of Algorithm-I (described in section 2) preceded by quadtree partition of each

macroblock.

3.2.1. Coding Scheme: 0\ plane

The 0\ plane represents the luminance component of the color image. First this plane is divided into a number of non-overlapping macroblocks of size 16 x 16. Based on smoothness criterion (say, block variance) each block is decomposed in quadtree manner till we reach blocks of size 4 x 4 . The decomposition of a block can be represented either by single bit or by five bits. An example is shown in Fig. 3. In this coding technique, the leaf nodes of quadtree are actually transmitted. The blocks of size 16 x 16 and 8 x 8 are encoded by only the block mean, while 4 x 4 blocks are encoded by BTC-PF method.

^ r internal node external node

(a) (b)

Fig. 3. Example of the quadtree partition: (a) 16 x 16 block (Big), (b) corresponding quadtree and encoding string is 11010.

3.2.2. Coding Scheme: O2 and O3 plane

Both O2 and O3 represent the chrominance components and contain less information compared to 0\ plane. Like 0\ plane, these two planes are also first divided into macroblocks of size 32 x 32. Then each of them is decomposed until 8 x 8 blocks are reached. All blocks of different sizes are initially coded only by block mean. To achieve the better quality, from each 8 x 8 block one 4 x 4 block is generated using averaging over 2 x 2 window. Then these 4 x 4 blocks are encoded by BTC-PF. Here the reconstruction levels are ^1 and H2, where \i2 is greater than mean of 8 x 8 block and \i\ is less than that. Let the difference between /*i and mean of 8 x 8 is <5i, and for fi2 is 62-

Bibhas Chandra Dhara and Bhabatosh Chanda 183

3.2.3. Progressive transmission in phase I

The positions of blocks of different sizes are transmitted by the bit streams representing the quadtree partition. Then the block mean of 16 x 16 and 8x8 blocks of Oi plane, and that of 32 x 32, 16 x 16 and 8 x 8 blocks of 0% and O3 planes are transmitted along with A value of 4 x 4 blocks of 0\ plane. The transmission of bit stream up to step l.a of Algorithm-I is the stage 1 of the proposed method. Similarly, up to step l.b, 2.a, 2.b, 3, 4 are assumed as the stage 2, 3, 4, 5 and 6, respectively. Then the bit streams consisting of 61, 5%, and seven bits for the selected patterns for 0 2 and O3 planes are transmitted. This is consider as the stage 7 of the proposed method.

3.3. Phase II coding

The second phase is used to achieve even better quality. In this phase, the residual planes Oi are obtained from Oi and their corresponding reconstructed planes. All three residual planes are encoded by BTC-PF method considering the eight patterns at level 1 (assuming root at level 0) of the FSPTT shown in Fig. 1.

3.3.1. Progressive transmission in phase II

Residual planes 0\, O2 and O3 are transmitted in order. These steps are considered as stage 8, 9 and 10 of the proposed method.

4. Experimental Results

In our experiment a number of512x512 color images are used. The performances of the proposed method are evaluated in terms of bit rate and peak signal-to-noise ratio (PSNR). The visual quality of the proposed method at different stages are shown in Figure 4. The stage by stage performance of the proposed PIT method are reported in Table 1. In phase I (up to stage 7) all bit rates are calculated straight way on raw data, but in phase II entropy coding is used. Average results of different spatial domain PIT methods and the proposed method are reported in Table 2 for comparison. Present method is compared with CBPM, CPBTC, CBMPBTC, and GBN methods. Table 2 reveals that, the average quality of reconstructed images at very first step is good enough with consuming least number of bits compared to other methods. At the first stage bit rate

of CBPM, CPBTC and GBN methods is greater or equal to 3 bpp and quality of reconstructed image are 18.106db, 27.828db, and 20.83db, respectively. Results of CBMPBTC method are 1.19bpp and 24.99db, but proposed method gives quality 22.051db with bit rate 0.157 bpp. Hence, after the first sketch of the image, if it is found that the image being transmitted is not the desired one and the transmission is terminated, the proposed method saves maximum time and bandwidth. The table also shows that both PSNR and Cbpp of other methods increase in large functions; where as in the proposed method these increase in small steps. So if the receiver wants to terminate transmission at some middle of two stages, it has to wait for long for the current stage to be completed. The termination is least time consuming for the proposed method.

5. Conclusions

In this paper a PIT for color images using BTC-PF is proposed. This method gives low bit with good qualities of initial images. In this method color images (RGB) are first transformed to (O1O2O3) domain by a reversible transformation and then each image plane is decomposed in quadtree fashion and then blocks are encoded BTC-PF method. Finally, to improve the quality of output images the residual planes are transmitted. The results show that proposed method gives least bit rate compare to other spatial domain PIT methods while achieving reasonably good quality.

References

1. K. H. Tzou, Optical Engineering 26, 581 (1987). 2. T. S. Chen and C. Y. Lin, A new imrpovement of

jpeg progressive image transmission using weight table of quantized dct coefficient bits, in Proceedings of the third IEEE pacific rim conference on multimedia, 2002.

3. J. M. Shapiro, IEEE Transactions on Signal Processing 41 , 3445 (1993).

4. P. Y. Tasi, Y. C. Hu and C. C. Chang, Signal Processing: Image Communication 19, 285 (2004).

5. M. Goldberg and L. Wang, IEEE Transactions on Communications 39, 540 (1991).

6. G. Qiu, IEEE Transactions on Image Processing 8, 109 (1999).

7. C. C. Chang, F. C. Shine and T. S. Chen, A new scheme of progressive image transmission based on

184 Progressive Transmission Scheme for Color Images Using BTC-PF Method

Table 1. The stage by stage performance of the proposed PIT method in terms of PSNR and cumulative bit rate (Cbpp), * indicates the results with entropy coding.

Images

Airplane

Lena

Peppers

Splash

Tiffany

Average

PSNR Cbpp PSNR Cbpp PSNR Cbpp PSNR Cbpp PSNR Cbpp PSNR Cbpp

Stages 1

21.314 0.164 22.597 0.159 21.538 0.200 22.521 0.126

22.285 0.139 22.051 0.157

2 23.865 0.311 25.774 0.301

24.009 0.379 25.671 0.238

25.532 0.261 24.970 0.298

3 24.246 0.460 26.208 0.446 24.308 0.531 25.933 0.318

25.756 0.370 25.290 0.425

4 25.174 0.534 27.122 0.518

24.986 0.607

26.584 0.358

26.042 0.424 25.981 0.488

5 27.217 0.583 27.745 0.566 25.913 0.657 27.973 0.384

27.567 0.460

27.283 0.530

6 27.935 0.632 28.592 0.614 26.223 0.707 28.206 0.410

27.894 0.496 27.770 0.571

7 28.798 0.720

28.978 0.676

27.807 0.886 30.025 0.527

28.650 0.568 28.851 0.675

8* 29.962 1.054

30.302 1.029

28.567 1.240

30.906 0.822

29.138 0.933 29.775 1.015

9* 30.389 1.279

31.128 1.339

29.066 1.539

31.707 1.061

29.688 1.232

30.395 1.290

10* 31.384 1.521

32.185 1.593

29.854 1.801

32.647 1.298

30.818 1.544

31.377 1.551

Table 2. Comparative results of proposed method

Methods

CBPM

CPBTC-16

CBMPBTC

GBN

Prop.

PSNR Cbpp PSNR Cbpp PSNR Cbpp PSNR Cbpp PSNR Cbpp

1 18.106

3 27.828 3.187 24.99 1.19

20.83 3.00

22.051 0.157

2 22.387

6 33.929 6.296 28.34 2.38

22.40 4.82

24.970 0.298

with some other methods.

Stages 3

28.466 9

39.737 9.468 30.67 3.76

31.38 7.82

25.290 0.425

4 34.475

12 45.729 12.577 32.04 5.51

34.46 10.03

25.981 0.488

5 40.490

15 52.328 15.307 33.01 8.00

39.71 13.03

27.283 0.530

6 46.277

18 60.569 17.440 34.32 11.97 46.40 15.79

27.770 0.571

7 51.253

21

28.851 0.675

8

29.775 1.015

9

30.395 1.290

10

31.377 1.551

bit-plane method, in Proceedings of the fifth asia pacific conference on communications and fourth optoelectronics and communications conference, 1999.

8. L. Wang and M. Goldberg, IEE Proceedings 135, 421 (1988).

9. W. J. Hwang and B. Y. Ye, IEEE Transactions on Consumer Electronics 43, 17 (1997).

10. E. A. Riskin, R. Lander, R. Y. Wang and L. E. Atlas, IEEE Transactions on Image Processing 3, 307 (1994).

11. Y. C. Hu and J. H. Jiang, Real Time Imaing 11, 59 (2005).

12. C. C. Chang, T. K. Shih and I. C. Lin, The visual computer 19, 342 (2003).

13. C. C. Chang, H. C. Hsia and T. S. Chen, A progressive image transmission scheme based on block truncation coding, in Lecture notes in computer science, Vol 2105, (Springer-Verlag Berlin Heidelberg, 2001).

14. C. C. Chen and M. N. Wu, A color image progressive transmission method by common bit map block truncation coding approach, in Proceedings of ICCT2003, 2003. K. Komatsu and K. Sezaki, In Proc. of SPIE Visual Communications and Image Progessing 2727', 1094 (1996). B. C. Dhara and B. Chanda, Pattern Recognition 37, 2131 (2004).

15.

16.

Bihhas Chandra Dham and Bhabatosh Chanda 185

\A (I)) M I'D

Fig. 4. (a) Original Image, (b)-(k) Reconstructed images by proposed method in different stages: (b) Stage 1: PSNR=22.597db, Cbpp=0.159, (c) Stage 2: PSNR=25.774db, Cbpp =0.301, (d) Stage 3: PSNR=26.208db, Cbpp-0.446, (e) Stage 4: PSNR=27.122db, Cbpp=0.518, (f) Stage 5: PSNR=27.745db, Cbpp=0.566, (g) Stage 6: PSNR=28.592db, Cbpp=0.614, (fa) Stage 7: P8NR=28.978db, Cbpp=0.676, (i) Stage 8; PSNR=30.302db, Cbpp=1.029, 0) Stage 9: PSNR=31.128db, Cbpp= 1.339, (k) Stage 10: PSNR=32.185db, Cbpp=1.593.

186

Registration Algorithm for Motion Blurred Images

K. V. Arya and P. Gupta

Department of Computer Science & Engineering, Indian Institute of Technology Kanpur,

Kanpur-208 016, India E-mail: {kvarya,pg} @cse.iitk. ac. in

This paper proposes an algorithm for restoring the motion blurred images. The restored images are then used for registering the template image. Restoration and registration of blurred images is very important problem from its application point of view. For restoration of motion blurred images, a modified Weiner filter based technique has been developed. An algorithm to register the restored image with given template image has also been proposed. The experimental results demonstrate that the images can successfully be registered in images with substantial amount of artificial and natural motion blur.

Keywords: Correlation coefficient; Image registration; Image restoration; Motion blur; Wiener filter.

1. Introduction

Due to the relative motion between the camera and scene during the image procurement, the captured images get blurred along the direction of relative motion. The noise and blur mixed with original image data put the constraints on the registration of the images. It is often required to recognize human faces or objects in such blurred images for example tracking and identification of criminals, where image of a human face or number plate of a running vehicle taken in hit and run situation gets blurred due to relative motion between the camera and object/face. This may be done through image restoration and subsequent registration of target face or object. Image registration is the process of finding point to point mapping between two images.1 One of the images is called reference or template image and the other one is called target or sensed image. The reference image is usually kept unchanged and the target image is a newly scanned image whose geometry must be changed.

The restoration of blurred and noisy image depends on the blurring system model. To restore the images from the noisy and blurred images, a number of restoration methods have been developed over the past two decades2"6 . This paper deals with two critical problems. First one is the restoration of image from motion blurred image. Second problem is to register restored target image with the original template image. The point spread function (PSF) parameters (namely blur length and blur direction) proposed in6"11 are used in solving the first problem. A

Weiner filter based image restoration algorithm has been developed in.7 Cole12 has modified Wiener filter. In this work modified Weiner filter based restoration algorithm has been developed. The primary task in restoration is to estimate the power spectrum of the blurred image which is then used to determine the PSF parameters.

To register the restored target image with the original template image, a method based on selective correlation coefficient (SCC)13 is used. The use of mask function in computation of similarity measure suppresses the pixels of noisy region from taking part in computation and hence, makes the registration process fast. In SCC, the binary mask function forces the gray scale images to be converted to binary images and hence, this additional step slows down the process of determining the similarity measure. In this paper a real valued mask function is defined to reduce the number of pixels taking part in the similarity measure computation. The mask function reduces the influence of noisy pixels of the restored image and hence, reducing the number of the pixels taking part in correlation computation which makes the process fast.

The rest of the paper is organized as follows. Section 2 presents the image degradation model and brief idea about determination of PSF parameters. The image restoration method is presented in Section 3. The proposed image registration method is discussed in Section 4. The experimental results are given in Section 5. Conclusions are given in the last section.

K. V. Arya and P. Gupta 187

2. B l u r M o d e l a n d P a r a m e t e r s E s t i m a t i o n

Images captured in uncontrolled environments in

variably represent a degraded version of an origi

nal image due to imperfections in the imaging pro

cess. The model shown in Fig. 1 describes the gen

eralized process of image degradation and restora

t ion. 1 4 ' 1 5 Given degradation function h(x,y) tha t to

gether with noise te rm n(x, y), operates on an input

image f(x,y) to produce a degraded image g(x,y).

The observed blurred/noisy image in the spatial do

main can be given by,

g(x,y) = h{x,y)®f(x,y)+n(x,y) (1)

where the symbol ® indicates the spatial convolu

tion. Since the convolution in spatial domain is the

multiplication in frequency domain, the degradation

model in (1) can be writ ten in frequency domain as,

G(m, n) = H(m, n)F(m, n) + N(m, n)

where G(m,n), H(m,n), F(m,n), and N(m,n) are

the Fourier transforms of g(x,y),h(x,y),f(x,y), and

n(x,y) respectively.

fix, y) Degradation function h(x,y)

(x, y)

Restoration filter rlx.y)

fix, y)

Noise nix, y}

DEGRADATION RESTORATION

Fig. 1. Image Degradation/Restoration Model

The blur direction is identified by applying

Hough transform on the spectrum of blurred image

and identifying the direction of line in the spectrum.

The blur length is found by rotating the Fourier spec

t rum of the blurred image in the estimated direc

tion, then observing the negative value in the inverse

Fourier transform. The detailed procedure to deter

mine these PSF parameters is given in.7

3 . B l u r r e d I m a g e R e s t o r a t i o n

The first task to restore the image from the motion

blurred image is to identify the PSF parameters blur

length and blur direction. In this work these param

eters are identified using the algorithms given in.7

The values of these parameters are then used to de

termine the power spec t rum (H(m,n)) of degrada

tion function. Many approaches to image restorat ion

have been reported in l i tera ture 1 4 but they perform

poorly in the presence of noise. Several variations of

the Wiener filter7'15'16 are used to restore the blurred

image, where image and noise both are considered as

random process.1 7 It then find an est imate f(x,y) of

the ideal image f(x,y) such tha t the mean square

error between them is minimized. In this work the Weiner filter has been modified

as suggested by Cole.12 T h e modified restorat ion filter is represented by the following transfer function,

R(m,n) = F(m, n)

\H{m, n)rF{m, n) + N(m, n) (2)

where R(m,n) is the Fourier transform of r(x,y). Then Fourier transform of the restoration filter output is given by,

F(m,n) = \R(m,n)\2G{m,n)

F{m,n)

\H(m, n)\zF{m, n) + N{m, n) G{m,n)

(3)

where G(m, n) represents the Fourier transform of observed degraded image g(x,y)a,nd given by the following expression,

G(m,n) = \H(m,n)\2F{m,n) + N(m,n)

In (3) the knowledge of power spectrum of original image is required, which is rarely known in motion blurred images. Therefore, (3) can be approximated by the following expression,

F(m,n) = 1

G{m,n) _\H(m,n)\* + K_

where K is constant and can be determined through experiments.

The restored image in the spatial domain is given by the inverse Fourier transform of the frequency domain estimate F{m, n). Therefore, restored image f(x, y) is given

by f(x,y) = Q-1{F(m,n)}

4. I m a g e R e g i s t r a t i o n

This section describes t h e process for registering a

template image in the restored target image. The

schematic block diagram of registration process is

shown in Fig. 2. It is clear tha t the registration pro

cess outputs the restored templa te image wi th regis

tered window for the input blurred templa te image

and query image. The algori thm given here computes

188 Registration Algorithm for Motion Blurred Images

the correlation coefficient as a similarity measure between the target image and sliding template image windows in the restored image.

Input Blurred Image

' , g (x ,y )

Image Restoration Module

1 *

Query Image

A f (x,y)

V Mask Function —«-Tst

Correlation Computation

1

f (x,y

s •J

Output Restored Template 1 Image with Registered

Window j

Fig. 2. Schematic Diagram of Image Registration Process

The window having highest correlation value indicates the location of query image. To compute the correlation coefficient we have modified the scheme proposed by Kaneko et o/.13in which they compute the selective correlation coefficient (SCC) by generating the a selecting mask for each pixel before calculating correlation coefficient. The mask defined is binary in nature and hence, the given image is converted to a mask-image where each pixel has the brightness value either '0' and ' 1 ' . Based on the mask-image those pixels are selected which take part in correlation computation. The binary mask function defined in SCC computation performs poorly in case of restored images as it is not possible to remove the blur and noise completely due to the limitations of debluring techniques. Therefore, in this work the SCC scheme is modified by taking a real-valued mask function. Mask function used here is defined using the concept of German and Reynolds M-estimator,18,19 which reduces the influence of noisy pixels. Hence, noisy pixels have very little contribution in the computation of similarity measure. This makes the similarity measure more effective in the presence of noise.

Here a two-dimensional image is represented as a one-dimensional list of pixel brightness values tak

ing in row major fashion. Let / = {/j}"=1 define a template/query image and / = {/»}"=1 is a restored target image of same size n as query image. The correlation coefficient for each pair of pixels between template and target images is defined as follows,

£? = 1 rrij (ft - P) (ft - J)

E?=i m* ( / i - F ) sjT.Urniih-1)*

where F and / are the average brightness values in restored target image and template image respectively and mask function rm is defined using the concept explained in,18 and is given in (4). Let X{ represents the difference in brightness values of ith pixel in both the images i.e. Xi = (/j — /i)/255. Then for all pixels « = 1,2,--- , n

n+kp if <o > 0 (4)

5. Exper imenta l Resul ts

The experiments are performed on approximately 50 gray scale images having natural as well as artificial motion blur. The four representative images are shown in Fig. 3 to 6. The experimental results with

(c)

Fig. 3. Image with natural motion blur (a) Query image, (b) Blurred target image and (c) Registered window using restored image obtained using the proposed algorithm.

natural blur images are shown in Fig. 3 and Fig. 4 and those with artificial motion blur are given in Fig. 5 and Fig. 6. In all the experimental results Fig. (a), (b) and (c) indicates the query image, blurred

target image and matched window on the restored target image respectively.

(c)

Fig. 4. Image with natural motion blur (a) Query image, (b) Blurred target image and (c) Registered window using restored image obtained using the proposed algorithm.

l l l l l l l i i t i§iks ^

mi^mmr W .ill

• I K * 00 0>)

(c)

Fig. 5. Image with artificial motion blur with blur length 20 and blur direction 35 (a) Query image, (b) Blurred target image and (c) Registered window using restored image obtained using the proposed algorithm.

It is observed that the proposed registration algorithm has correctly matched the query image in all the restored images. The algorithm has performed poorly in the images with artiicial motion blur having blur length and blur direction more than 45 and 65 respectively. But it has indicated the robust performance in cases of natural motion blur.

K. V. Arya and P. Gupta 189

(c)

Fig. 6. Image with artificial motion blur with blur length 25 and blur direction 55 (a) Query image, (b) Blurred target image and (c) Registered window using restored image obtained using the proposed algorithm.

6. Conclusions

Algorithms for automated restoration and registration of motion blurred images are presented here. It is likely to have application in identification tasks in uncontrolled environments. The image restoration algorithm uses the knowledge of PSF parameters viz. blur length and blur direction while the registration algorithm is based on mask-based correlation computation method. Experimental results demonstrate the effectiveness of the algorithms on a wide range of images with natural and varying degree of artificially created motion blurs.

The algorithm may be extended using more sophisticated registration techniques. The point spread function parameters may also be utilized in the registration process for improved correlation computation.

References

1. L. G. Brown, ACM Computing Surveys 24, 326 (1992).

2. H. C. Andrews and B. R. Hunt, Digital Image Restoration (Prentice - Hall, Englewood cliffs, NJ, 1977).

3. W. J. Woods and V. K. Ingle, IEEE Trans. Acoustic Speech and Signal Processing 20, 188 (1981).

4. R. L. Lagendijk, J. Biemond and D. E. Boekee, IEEE Trans. Acoustic Speech and Signal Processing 36, 1874 (1988).

5. R. L. Lagendijk and J. Biemoed, Basic Methods for Image Restoration and Identification, in Hand Book

190 Registration Algorithm for Motion Blurred Images

of Image and Video Processing, ed. A. Bovik (Academic Press, 2000), pp. 125-140.

6. Q. Li and Y. Yoshida, IEICE Trans. Fundamentals E80-A, 1 (1997).

7. R. Lokhande, K. V. Arya and P. Gupta, Identification of parameters and restoration of motion blurred images, in Proc. 2006 ACM Symposium on Applied Computing, (Dijon, France, 2006).

8. M. Cannon, IEEE Trans. Acoust. Speech Signal Process. 24, 56 (1976).

9. M. M. Chang, A. M. Tekalp and A. T. Erdem, IEEE Trans. Signal Processing 39, 2323 (1991).

10. R. Fabian and D. Malah, CVGIP: Graphical, Models and Image Processing 53, 403 (1991).

11. D. B. Gennery, J. Opt. Soc. Amer. 63, 1571 (1973). 12. E. R. Cole, The removal of unknown image blurs

by homomorphic filtering, PhD thesis, Department

of Electrical Engineering, University of Utah, Salt Lake City, (UT, June 1973).

13. S. Kaneko, Y. Satoh and S. Igarashi, Pattern Recognition 36, 1165 (2003).

14. M. R. Banham and A. K. Katsaggelos, IEEE Signal Processing Magazine 14, 24 (1997).

15. R. C. Gonzalez and R. E. Woods, Digital Image Processing (Pearson Education, 2003).

16. I. Pitas, Digital Image Processing Algorithms and Applications (John Wiley arid Sons, Inc., 2000).

17. K. S. Trivedi, Probability and Statistics with Reliability, Queuing and Computer Science Applications, 2nd Edition (John Wiley & Sons, 2002).

18. M. Black and A. Rangarajan, Intl. Journal of Computer Vision 19, 57 (1996).

19. D. German and G. Reynolds, IEEE Trans. Pattern Anal. Machine Intell. 14, 376 (1992).

PART G

Image Segmentation

193

Aggregation Pheromone Density Based Change Detection in Remotely Sensed Images

Megha Kothari and Susmita Ghosh

Department of Computer Science and Engineering Jadavpur University

Kolkata 700032, India

Ashish Ghosh

Machine Intelligence Unit and Center for Soft Computing Research Indian Statistical Institute

203 B. T. Road, Kolkata 700108, India E-mail: [email protected]

Ants, bees and other social insects deposit pheromone (a type of chemical) in order to communicate between the members of their community. Pheromone that causes clumping or clustering behavior in a species and brings individuals into a closer proximity is called aggregation pheromone. This article presents a novel method for change detection in remotely sensed images considering the aggregation behavior of ants. Change detection is viewed as a segmentation problem where changed and unchanged regions are segmented out via clustering. At each location of data point, representing a pixel, an ant is placed; and the ants are allowed to move in the search space to find out the points with higher pheromone density. The movement of an ant is governed by the amount of pheromone deposited at different points of the search space. More the deposited pheromone, more is the aggregation of ants. This leads to the formation of homogenous groups of data. Evaluation on two multitemporal remote sensing images establishes the effectiveness of the proposed algorithm over an existing thresholding algorithm.

Keywords: Change detection; Remote sensing; Aggregation pheromone system

1. Introduction

Automatic detection of landcover transitions in multitemporal remotely sensed images is of widespread interest due to a large number of real world applications ranging from forestry and agricultural surveys to urban studies and natural disaster management.1"3 In many real world applications of change detection, the objective is to map only one (or, few) landcover transition(s) of interest like agricultural land to urban area, forest to burned area, forest to agricultural land etc. Land-cover change detection using multi-temporal remote-sensing images is a challenging task due to lack of a-priori information about the shape of changed areas, absence of a reference background, differences in light conditions, atmospheric conditions, sensor calibration, ground moisture at the two image acquisition dates, alignment of multi-temporal images (registration) etc.4'5

These factors restrict the use of most classical multi-temporal image-analysis techniques to few remote-sensing change analysis problems. There are several techniques available in the literature 6 ~ n where Neural Networks, Markov Random Field (MRF) and Support Vector Machines (SVM) are used for de

tecting changes in remotely sensed images. In this article change detection is viewed as a segmentation problem, where changed and unchanged regions are segmented using clustering. Clustering is performed considering the aggregation behavior found in ants and ant like agents.

Numerous clustering algorithms have been developed inspired by the ability of ants to cluster their corpses into "cemeteries" in an effort to clean up their nests.12 Besides nest cleaning, many functions of aggregation behavior have been observed in ants and ant like agents.13'14 These include foraging-site marking and mating, finding shelter and defense. For example, after finding safe shelter, cockroaches produce a specific pheromone with their excrement, which attracts other members of their species.14

Tsutsui and Ghosh15 used aggregation pheromone systems for continuous function optimization where aggregation pheromone density is defined by a density function in the search space. Inspired by the aforementioned aggregation behavior found in ants and other similar agents, attempts are already made for solving clustering16 and image segmentation17

problems. In this article this metaphor is used to detect changes in remote sensing images.

194 Aggregation Pheromone Density Based Change Detection in Remotely Sensed Images

2. Aggregation Pheromone Density based Change Detection

As mentioned in the introduction, aggregation pheromone brings individuals into closer proximity. This group formation nature of aggregation pheromone is being used as the basic idea of the proposed technique. Here each ant represents one data/pixel of an image. The ants move with the aim to create homogenous groups of data. The amount of movement of an ant towards a point is governed by the intensity of aggregation pheromone deposited by all other ants at that point. This gradual movement of ants in due course of time will result in formation of groups or clusters. The proposed technique has two parts. In the first part, clusters are formed based on ants' property of depositing aggregation pheromone. The number of clusters thus formed might be more than the desired number. So, to obtain the desired number of clusters, in the second part agglomerative average linkage18 clustering algorithm is applied on these already formed clusters.

2.1. Formation of Clusters

While performing segmentation for a given image, we group similar pixels together to form a set of coherent image regions. Similarity of the pixels can be measured based on intensity, color, texture and consistency of location of different pixels. Individual features or combination of them can be used to represent the pixel of an image. For each pixel, we associate a feature vector x. Clustering is then performed on all the pixels to group them into segments.

Let us consider a data set of n patterns (xi,X2,X3,...,xn) and a population of n-ants (Ai,A2,A3,...,An) where an ant At represents the data pattern Xi. Each individual ant emits pheromone in its neighborhood. The intensity of pheromone emitted by an individual A at x decreases with its distance from it. Thus the pheromone intensity at a point closer to x is more than those at other points that are farther from it. To achieve this, the pheromone intensity emitted by A is assumed to follow a Gaussian distribution. The pheromone intensity deposited at x ' by an ant A (located at x) is given by

AT'(A,X!) = exp~' " i / , (1)

and the total aggregation pheromone density de

posited by the entire population of n ants at x ' is then given by

n . n

_—^ d (x , , x ' ) 2

Ar(x ' ) = 2 > x P 53>—, (2) i = l

where, 8 denotes the spread of Gaussian function. In the similar way, we can compute the total aggregation pheromone density Ar(x) for any point x.

Now, an ant A1 which is initially at location x' moves to the new location x" (which is computed using Eq. 3) if the total aggregation pheromone density at x" is greater than that at x'. The movement of an ant is governed by the amount of pheromone deposited at different points in the search space. It is defined as

„ , Next(A') x"=x' + n. K—t, (3)

where, n t 2

Next(A') = ^ ( x i - x'). exp~ ^ V " ' , (4) i=l

with j] as a step size. This process of finding a new location continues until an ant finds a location where the total aggregation pheromone density is more than its neighboring points. Once the ant Ai finds out such a point xj with greater pheromone density, then that point is assumed to be a new potential cluster center, say Zj (j = 1,2,..., C, C being number of clusters); and the data point with which the ant was associated earlier (i.e., x-,) is assigned to the cluster so formed with the center Zj. Also the data points that are within a distance of (5/2 from Zj are assigned to the newly formed cluster. On the other hand, if the distance between x- and the existing cluster center Zj is less than 6/2 and the ratio of their densities is greater than threshold-density (a predefined parameter), then the data point xi is allocated to the already existing cluster centering at Zj. Higher value of density ratio represents that the two points are of nearly similar density and hence should belong to the same cluster. The proposed aggregation pheromone based clustering (APC) algorithm for formation of clusters is given below.

begin I n i t i a l i z e a, threshold-density, n and C = 0 for i = 1 : n do if the data pattern X; is not already

assigned to any cluster

Megha Kothari, Susmita Ghosh and Ashish Ghosh 195

Compute Ar(xi) using Eq. 2

label 1: Compute new location xj using Eq.

3

Compute Ar(x-)

if (Ar(x'i) > Ar(xi))

Update the location of ant Ai to xj and

goto label 1

else continue

if (C == 0) //If no cluster exists

Consider xj as cluster center Zi and

increase C by one

else

for j = 1: C

if (min(Ar(x/i),Ar(Zj))/max(AT(x;),AT(Zj)) >

thresholdjdensity and d(x[, Zj) < 6/2) Assign xj to Zj

else

Assign xj as a new cluster center say,

Zc+i and increase C by one

Assign all the data points that are at a

distance of 6/2 from xj to the newly formed

cluster Zc+i

end of else

end of for

end of else

end of if (if the data pattern Xi ...)

end of for

end

2.2. Merging of Clusters

In this stage, to obtain the desired number of clusters we apply average linkage18 algorithm. The algorithm works by merging the two most similar clusters until the desired number of clusters are obtained.

3. Experimental Results

3.1. Description of Data Set

Two real multitemporal images namely Mexico and lake Mulargia have been used for experimental purpose. Mexico image is acquired on 18t/l April 2000 and 20"1 May 2002 by Thematic Mapper Plus (TM+) sensor of Landsat-7 satellite. From the entire available Landsat scene, a section of 512x512 pixels has been selected as test site. Significant portion of the vegetation in the aforesaid area between the two dates was destroyed by wildfire. Figs. 1(a) and 1(b) show the channel 4 of the April and May images, respectively. Experts generated the reference map [Fig.

1(c)] by visual analysis of the images with the help of available ground truth concerning the location of wildfire. The reference map is required to assess the change detection errors.

Image of lake Mulargia of Sardinia Island is acquired by Thematic Mapper Plus (TM+) sensor of Landsat-5 satellite on September 1995 and July 1996. Between the two acquisition dates the water level in the lake was increased. Fig. 2 shows a section of 412x300 pixels of channel 4 of September and July image along with the reference map.

3.2. Results

From given multitemporal images we compute difference image by taking the absolute difference of the two co-registered images of different dates. The difference images corresponding to Mexico and lake Mulargia are shown in Figs. 1(d) and 2(d), respectively. Feature vector corresponding to each pixel of the difference image is generated by considering its gray value and the average gray value of neighboring pixels (second order). The algorithm, described in Section 2, is then applied on this difference image to obtain the corresponding change detection map [Figs. 3(a) and 4(a)]. Values of n and threshold-density are kept to 1 and 0.9, respectively and different values of 6 in the range [0, 1] are considered. We determine a stable range of 6 for which clusters were compact. Results are shown for a 6 value taken from this range. The results obtained by the proposed A PC algorithm are compared with those obtained by ground truth based optimal Manual Trial and Error Thresholding (MTET) technique and are depicted in Table 1.

Table 1. Missed alarms, false alarms and total alarms resulting from the proposed APC algorithm and MTET.

Images used

Mexico

Lake

Technique used

APC(<5=0.23) MTET

APC((5=0.57) MTET

Missed alarms

2654 2404

1000 1015

False alarms

760 2187

599 875

Total alarms

3414 4591

1599 1890

It is seen from the table that missed alarms (changed pixels classified as unchanged) obtained by

196 Aggregation Pheromone Density Based Change Detection in Remotely Sensed Images

" -y - >

>*

Fig. 1. Mexico image (a) Acquired on April 2000 (b) Acquired on May 2002 (c) Reference map (d) Difference image.

Fig. 2. Lake Muiargia image (a) Acquired on September 1995 (b) Acquired on.July 1996 (c) Reference map (d) Difference image.

Fig. 4. Change detection map for lake Muiargia image by (a) APC algorithm (b) MTET.

Fig. 3. Change detection map for Mexico image by (a) APC algorithm (b) MTET.

APC and MTET are comparable but false alarms (unchanged pixels classified as changed) obtained by APC are significantly less than those obtained using MTET. In terms of OYerall error, APC performs better than MTET. Figs. 3 and 4 show the change detection map obtained by both the techniques for Mexico and lake Muiargia image, respectively. Comparison of these change detection maps justifies the capability of the proposed APC algorithm for exploiting spatial contextual information to extract the changed regions properly. It is worth mentioning that MTET requires ground truth information to find out the optimal threshold whereas APC is completely unsupervised in nature.

4. Conclusions

In this paper we have proposed a new algorithm for change detection based on aggregation pheromone density, which is inspired by the ants' property to accumulate around points having higher pheromone density, To evaluate the performance of the proposed algorithm experiments were carried out with two multitemporal remote sensing images. Qualitative and quantitative evaluation of the experimental results establishes the superiority of the proposed APC algorithm over existing MTET. The proposed algorithm needs fewer parameters while producing better results. Moreover unlike MTET it does not require any ground truth information.

Megha Kothari, Susmita Ghosh and Ashish Ghosh 197

A c k n o w l e d g m e n t s

Authors would like to acknowledge the Department

of Science and Technology, Government of India and

University of Trento, Italy, the sponsors of the India-

Trento Program on Advanced Research (ITPAR),

under which a project titled " A d v a n c e d Techn iques for R e m o t e S e n s i n g I m a g e P r o c e s s i n g " is being carried out at the Department of Com

puter Science and Engineering, Jadavpur Univer

sity, Kolkata. Authors would also like to acknowledge

Prof. Lorenzo Bruzzone, Italian collaborator of this

project, for providing the images.

R e f e r e n c e s

1. J. A. Richards, Remote Sensing Digital Image Analysis, 2nd edn. (Springer-Verlag, 1993).

2. R. J. Radke, S. Andra, O. Al-Kofahi and B. Roysam, IEEE Transactions on Image Processing 14, 294 (2005).

3. D. Lu, P. Mausel, E. Brondizio and E. Moran, International Journal on Remote Sensing 25, 2365 (2004).

4. J. R. G. Townshend, C. O. Justice, C. Gurney and J. McManus, IEEE Transactions on Geoscience and Remote Sensing 30, 1054 (1992).

5. P. Gong, E. F. Ledrew and J. R. Miller, International Journal on Remote Sensing 13, 773 (1992).

6. S. Patra, S. Ghosh and A. Ghosh, Unsupervised change detection in remote-sensing images using one-dimensional self-organizing feature map neural network, in Proceedings of 9 International conference on Information Technology (ICIT06), (IEEE Computer Society press, Bhubaneswar, India, 2006).

7. S. Ghosh, S. Patra, M. Kothari and A. Ghosh, AN-VESA: The Journal of F. M. University 1, 48 (2005).

8. L. Bruzzone and D. F. Prieto, IEEE Transactions on

Geoscience and Remote Sensing 38, 1171 (2000). 9. T. Kavzoglu and P. Mather, International Journal

on Remote Sensing 24, 4907 (2003). 10. P. A. Brivio, M. Maggi, E. Binagi, I. Gallo and J. M.

Gregoire, Exploiting spatial and temporal information for extracting burned areas from time series of SPOT-VGT data, in Proceedings of the 1st International Workshop on the Analysis of the Multi-Temporal Remote Sensing Images, 2001.

11. F. Bovolo and L. Bruzzone, A context-sensitive technique based on support vector machines for image classification, in Preceedings of 1st International Conference on Pattern Recognition and Machine Intelligence, eds. S. K. Pal, S. Bandyopadhyay and S. Biswas (Springer, 2006).

12. A. L. Vizine, L. N. de Castro, E. R. Hruschka and R. R. Gudwin, Informatica 29, 143 (2005).

13. M. Ono, T. Igarashi, E. Ohno and M. Sasaki, Nature 377, 334 (1995).

14. M. Sukama and H. Fukami, Journal of Chemical Ecology 19, 2521 (1993).

15. S. Tsutsui and A. Ghosh, An extension of ant colony optimization for function optimization, in Proceedings of the 5 Asia Pacific Conference on Simulated Evolution and Learning (SEAL04), Pusan, Korea, 2004.

16. M. Kothari, S. Ghosh and A. Ghosh, Aggregation pheromone density based clustering, in Proceedings of 9 International conference on Information Technology (ICIT06), (IEEE Computer Society press, Bhubaneswar, India, 2006).

17. S. Ghosh, M. Kothari and A. Ghosh, Aggregation pheromone density based image segmentation, in Proceedings of 5 Indian Conference on Computer Vision, Graphics and Image Processing (ICVGIP06), (Springer.LNCS, 2006).

18. S. Theodoridis and K. Koutroumbas, Pattern Recognition, 2nd edn. (Elsevier Academic Press, Amsterdam, 2003).

198

Automatic Brain Tumor Segmentation Using Symmetry Analysis and Deformable Models

Hassan Khotanlou, Olivier Colliot1 and Isabelle Bloch

GET-Ecole Nationale Superieure des Telecommunications Departement TSI, CNRS UMR 5141 LTCI

46 rue Barrault, 75634 Paris Cedex 13, France E-mail: {Hassan. Khotanlou, Olivier. Colliot,Isabelle.Bloch} Oenst.fr

1 Current address: LENA UPR 640 CNRS, Paris, France

We propose a new general automatic method for segmenting brain tumors in 3D MRI. Our method is applicable to different types of tumors. A first detection process is based on selecting asymmetric areas with respect to the approximate brain symmetry plane. Its result constitutes the initialization of a segmentation method based on a combination of a deformable model and spatial relations, leading to a precise segmentation of the tumors. The results obtained on different types of tumors have been evaluated by comparison with manual segmentations.

Keywords: Segmentation, symmetry plane, deformable Models, spatial relations, brain tumors, MRI.

1. Introduction

The segmentation of brain tumors in magnetic resonance images (MRI) is a challenging and difficult task because of the variety of their possible shapes, locations, image intensities. The aim of this paper is to contribute to this domain, by proposing an original method, which is automatic and general enough to address the variability issues.

Existing methods are classically divided into region based and contour based methods, and are usually dedicated to full enhanced tumors or specific types of tumors. In the first class, Clark et al.1 have proposed a method for tumor segmentation using knowledge based and fuzzy classification, where a learning process prior to segmenting a set of images is necessary. Other methods are based on statistical pattern recognition techniques such as.2"4 These methods fail in the case of large deformations in the brain. Existing contour based methods are not fully automatic and need some manual operation for initialization. Lefohn et al.5 have proposed a semiautomatic method using level sets. Another segmentation method based on level sets was introduced by Ho et al.6 that uses Tl-weighted images both with and without contrast agent for tumor detection. A method by deformable model and neural network was introduced by Zhu and Yang7 that processes the image slice by slice and is not a real 3D method.

In this paper we introduce a fully automatic method for the segmentation of different types of tumors in 3D MRI, based on a combination of region based and contour based methods. In the first step,

described in Section 2, we use the mid-sagittal approximate symmetry plane and detect tumors as an asymmetry with respect to this plane. In the second step, detailed in Section 3, a precise segmentation is obtained using an original combination of deformable models and spatial relations. Results are then presented in Section 4.

2. Tumor detection based on symmetry

In this section we detail the first step of the proposed approach, by explaining our method to compute the symmetry plane of the brain and then the method to detect tumors based on this plane.

2.1. Computation of the approximate symmetry plane

As proposed in,8 the computation of the approximate symmetry plane is expressed as a registration problem. A degree of similarity between the image and its reflection with respect to a plane is computed. The best plane is then obtained by maximizing this similarity. This optimization is performed using downhill simplex method and is initialized by the plane obtained from principal inertia axes, which proves to be close to the global optimum.

Let u be a unit vector in M3 and IIUid a plane in R3 orthogonal to the vector u and passing at the distance d from the coordinate origin. We denote by eu,d(/) the reflection of image / with respect to the plane nu,d: euM(f)(x,y,z) = f(eu,d(x,y,z)). An im-

age / is called reflection symmetrical if there exists a reflection plane IIUjrf such that eu^(f) = f.

The idea is to compute a symmetry measure A*u,d(/) of ^^e image / with respect to an arbitrary reiection plane IIUid, and to find the plane leading to the maximal symmetry degree and the corresponding value of symmetry measure / i( /) :

u€S2, «f€R+

In this case the symmetry measure /Au,d(/) c a n be defined as the similarity between images / and eu>£j(/). In this work we use the following symmetry measure:

„ m , ll/-eu,d(/)H2

lhi9dKJ) - l 2||j||2

2*2. Tumor detection

In our previous work9 we have used the fuzzy pos-sibilistic C-means (FPCM) classification algorithm for tumor detection and we obtained good results for detecting hyper intensity tumors (full enhanced tumors). However this method is difficult to generalize to any type of tumor while keeping it automatic. Therefore we suggest another approach, using the approximate symmetry plane.

(c) (d)

Fig. 1. (a) One axial slice of the original 3D image, (b) Brain mask and symmetry plane, (c) Another example, (d) Brain mask of image (c) with symmetry plane.

Pathological brains are usually not symmetric, thus the symmetry plane is computed on the seg-

Hassan Khotanlou, Olivier Colliot and IsabeUe Block 199

mented brain. In the normal brain, it has been observed that the symmetry plane of the grey level brain image and of the segmented brain are approximately equal. The segmentation of the brain is performed as in.10 The algorithm summarized in section 2.1 is then applied on the binary image of the brain. Applying this method to images containing tumors provides a good approximation of the mid-sagittal plane, despite the asymmetry induced by the tumors, thanks to the preliminary segmentation of the brain. This is illustrated in Figure 1.

Now tumors can be detected by evaluating this asymmetry with respect to the obtained plane. We assume that tumors are localized in only one hemisphere. This hemisphere is found by comparing the grey level characteristics (mean and standard deviation) of grey matter, white matter and CSF computed in the whole image on the one hand, and in each hemisphere on the other hand. Let H n denote the histogram of grey levels in the normal hemisphere and H p the histogram in the pathological hemisphere. The histogram difference H a = H p — H n provides useful information about new intensity classes induced by the tumor. In the case of a tumor without edema (as in Figure 1(c)) a positive peak can be observed in M3 that shows the tumor intensity range (see Figure 2(a)) and we can use a thresholding and morphological operations to extract the tumor (Figure 2(b)).

In the case of a tumor with edema (as in Figure 1(a)) we observe two peaks in M3 (Figure 3(a)). Because the intensity of edema is always lower than the intensity of the tumor, the i rs t peak correponds to the edema and the second peak to the tumor. We have considered the peaks with more than 300 voxels, this threshold being based on the analysis of M3

for several normal brains. The negative peaks observed in H s correspond

to normal tissues, around the tumor and the edema, since these tissues are less represented in the hemisphere containing the pathology than in the other hemisphere. These tissues can therefore been obtained automatically (Figures 2(c) and 3(c)). They will be used for introducing spatial relations in the next section.

3. Refined segmentat ion

In this section we detail the second step of the proposed approach: the previous detection of the tumor

200 Automatic Brain Tumor Segmentation Using Symmetry Analysis and Deformable Models

1 Ti&ue Around the Turner

Ettera, Tumor

Fig. 3. (a) Graph of H 3 for image (a) of Figure 1. (b) Extracted tumor after morphological operations, (c) Tissues around the tumor.

to constrain this model by spatial relations between the tumor and other tissues, as an adaptation of the method described in11 for normal brains.

3 .1 . Spatial relations constrained deformable model

The evolution of the deformable surface X is described by the following dynamic force equation:12

7 ^ r = Fint(X) + F e x t (X) , where F i n t is the internal force that specifies the regularity of the surface and Fext is the external force that drives the surface towards image edges. The chosen internal force is Fint — c*V2X-/?V2(V2X), where a and /? respectively control the surface tension and rigidity, and V2

is the Laplacian operator. It is then discretized on the simplex mesh using the finite difference method.12

In our case, the external force is not only derived from image edges but also constrains the deformable model to satisfy spatial relations to the surrounding tissues. The spatial relations are represented by fuzzy sets in the image space,13 from which a new fuzzy force is derived. The external force is then written as:11 Fext = Av + /xF/j where v is a classical external force such as gradient or balloon and FR is the force attached to the spatial relationships.

Tisam Around tb® *t*umor / s •

. > i ! : :

Fig. 2. (a) Graph of H s for image (c) of Figure 1. (b) Extracted tumor after morphological operations, (c) Tissues around the tumor.

is used to initialize a deformable model. We propose

3.2. Constrained deformable model for tumor segmentation

Spatial relations are useful to guide the recognition of objects in images since they provide an important information about the spatial organization of these objects. Two main classes of spatial relations can be considered: topological relationships, such as inclusion, exclusion and adjacency, and metric relationships such as distances and orientations. Here we use a combination of topological and distance information.

The evolution process of the deformable model can be guided by a combination of several relations, via information fusion tools. Here, two types of information are available: the initial detection and the surrounding tissues. Therefore we use (i) the distance from the initial segmented tumor, and (ii) the tissues around the tumor which were obtained in the previous step. The idea is that the contour of the tumor should be situated somewhere inbetween the boundary of the initial detection and the boundary

Hassan Khotanlou, Olivier Colliot and Isabelle Bloch 201

of the normal tissues (excluding the background). A fuzzy set representing the relation "near the tumor" is defined as an increasing function of the distance. A distance map in the normal tissues to its complement (tumor and background) is computed, and a fuzzy set is derived again using an increasing function. These two relations are represented as fuzzy sets in the image space. They are illustrated in Figure 4.

(a) (b) (c)

Fig. 4. Spatial relations used for segmentation on two examples, (a) Near the tumor, (b) Relation provided by the normal tissues, (c) Fusion of the two relations.

These relations are combined using a conjunctive fusion operator (a t-norm), leading to a fuzzy set /J,R. The resulting fuzzy set provides high values in the region where both relations are satisfied, and lower elsewhere. The fuzzy force is derived from this fusion

where d is a distance map to the kernel of fiR (i.e. points x for which /J.R(X) = 1). The classical external force is calculated by Generalized Gradient Vector Flow12 based on an edge map obtained from Canny-Deriche edge detection.

4. Resul t s and conclusion

We have applied the method to 10 different real 3D Tl-weighted MRI (of size 256 x 256 x 124). These images contain tumors with different sizes, intensities, shapes and locations. This allows us to illustrate the large field of application of our method. The evaluation of the segmentation results was performed through a quantitative comparison with the

results of a manual segmentation. Let us denote by A the manually segmented tumor and B the tumor segmented by our method. We used three measures to evaluate the results, as proposed in14 which are:

• overlap: |4S§| ; • Hausdorff distance between A and B, defined

as max.(h(A,B),h(B,A)) where h(A,B) = maxaeAminxesd(a,6), and d(a,b) denotes the Euclidean distance between a and b (a and b are points of A and B respectively);

• the signed distances from the surface of B to the surface of A are computed, and the absolute average value of distances is derived.

The segmentation results for the two cases of Figure 1 are shown in Figure 5. In the first case, the initial detection based on symmetry analysis only provides a part of the tumor. The whole tumor is successfully recovered by the second segmentation step using the deformable model and the spatial relations. Even in the second case, where the initial detection is already quite good, the second step provides a more precise boundary of the lesion.

The quantitative results obtained by comparing the automatic segmentations with the available manual segmentations are provided in Table 1 for 10 cases. For the overlap, all values are greater than 85% (note that values above 70% are generally considered as good results), and most of them are greater than 91%. The distance-based evaluations should be compared to the voxel size, which is typically 1 x 1 x 1.3 mm3 for all images. The Hausdorff distance in these 10 cases is always less than five voxels, and often even smaller. It should be noted that this measure is particularly severe since it is a maximum distance and provides the error for the worst point in the segmentation. The average distance is less than one voxel, which means that in average the automatic contour is very close to the manual one (less than one voxel distance). Due to the partial volume effect, the obtention of more precise results would require to work at sub-voxel level.

All these results show the high accuracy of the proposed method, which was also confirmed by a visual evaluation performed by medical experts.

As a conclusion, the proposed hybrid algorithm using region based and contour based methods proves to be efficient to segment brain tumors in 3D

202 Automatic Bmin Tumor Segmentation Using Symmetry Analysis and Deformable Models

in) (l>) (c)

Fig. 5. Final segmentation results on two different cases. (a) Initial detection superimposed on an axial slice, (b) Final segmentation, (c) Result superimposed on a sagittal slice.

Table 1. Evaluation of the segmentation results of tumors on a few 3D MR images for which a manual segmentation was available.

Dataset Overlap Hausdorff Average

Tumor 1 Tumor 2 Tumor 3 Tumor 4 Tumor 5 Tumor 6 Tumor 7 TumorS Tumor 9 Tumor 10

(%)

95.56 91.32 96.12 88.24 90.08 91.21 95.05 85.86 88.63 91.63

(mm)

2.66 5.20 1.32 1.51 3.14 2.40 4.02 3.62 4.91 3.83

(mm)

0.62 1.41 0.41 1.21 1.15 1.01 1.29 1.31 0.92 0.84

MR images. I ts application to several datasets with

different tumor sizes, intensities and locations shows

tha t it can automatically detect and segment very

different types of brain tumors with a good quality.

Our method can be applied as well to T2-weighted

images or FLAIR images. However it may fail in the

case of a symmetrical tumor across the mid-sagittal

plane, but this case is very rare. Future works aims

at combining several modalities such as T2-weighted

and FLAIR to develop the segmentation of edema

and in i l t ra t ion around the tumors.

A c k n o w l e d g m e n t s

Hassan Khotanlou is supported by Bu AM Sina Uni

versity, and Olivier CoUiot by a grant from Paristech

- lie de FVance.

R e f e r e n c e s

1. M. Clark, L. Lawrence, D. Golgof, R. Velthuizen, F. Murtagh and M. Siibiger, IEEE Transactions on Medical Imaging 17(April 1998).

2. M. Kaus, S. WarBeid, A. Nabavi, E. Chatzidakis, P. Black, F. Jolesz and R. Kikinis, Segmentation of meningiomas and low grade gliomas in MRI, in MIC-CAI, (Cambridge UK, 1999).

3. N. Moon, E. Bullitt, K. Leemput and G. Gerig, Model-based brain and tumor segmentation, in ICPR, (Quebec, 2002).

4. M. Prastawa, E. Bullitt, S. Ho and G. Gerig, M^edical Image Analysis 18, 217 (2004).

5. A. Lefohn, J. Gates and R. Whitaker, Interactive^ GPU-BasM Level Sets for SD Brain Tumor Segmentation, tech. rep., University of Utah (April 2003).

6. S. Ho, E. Bullitt and G. Gerig, Level set evolution with region competition: Automatic 3D segmentation of brain tumors, in ICPR, (Quebec, 2002).

7. Y. Zhu and H. Yang, IEEE Tmnsactions on Medical Imaging IS , 55 (1997).

8. A. Tuzikov, O. Colliot and I. Bloch, Pattern Recognition Letters 24, 2219(oct 2003).

9. H. Khotanlou, J. Atif, O. Colliot and I. Bloch, 3D Brain Tumor Segmentation Using Fuzzy Classification and Deformable Models, in WILF, (Crema, Italy, 2005).

10. J.-F. Mangin, O. Coulon and V. R*ouin, Robust brain segmentation using histogram scale-space analysis and mathematical morphology, in MICCAI, (Cambridge USA, 1998).

11. O. Colliot, O. Camara and I. Bloch, Pattern Recognition 39, 1401 (2006).

12. C. Xu and J. Prince, IEEE Transactions on Image Processing 7, 359 (1998).

13. I. Bloch, Image and Vision Computing 23, 89 (2005). 14. G. Gerig, M. Jomier and M. Chakos, Valmet: a new

validation tool for assessing and improving 3d object segmentation, in MICCAI, (Utrecht, Netherlands, 2001).

203

Edge Recognition in MMWave Images by Biorthogonal Wavelet Decomposition and Genetic Algorithm

C. Bhattacharya and V. P. Dutta

DEAL (DRDO) Raipur Road, Dehradun 248001, India

E-mail: [email protected], [email protected]

In this paper, we present a multiresolution approach to recognize the edges of objects in MMWave images by generating multiscale image approximations. Bounds of threshold for the multiscale edge operator are optimized by genetic algorithm (GA). Equivalence of 2-D multiscale discrete wavelet transform to Canny edge operator is exploited here to optimize the bounds of hysteresis threshold. The invariance of object boundaries to dyadic coarser change of scale is shown in the results.

Keywords: Multiresolution; Edge Recognition; Genetic Algorithm; Biorthogonal wavelet

1. Introduction

Passive sensors such as millimeterwave (MMWave) imaging sensors have the advantage that they require no radio transmission and are all-weather, day-night imaging instruments. But the images acquired are of poor resolution, low dynamic range and hence are of poor contrast; images remain blurred due to scanning antenna of the sensor. Recognition of objects of interest, therefore, is difficult in MMWave images purely by human vision. Secondly, information content in such images can be enhanced by image fusion from other passive sensors requiring automatic object recognition methods.

One important and widely accepted low-level processing method is to detect the edges of objects of interest at multiple scales.1-6 We pursue this approach for MMWave images, as object boundaries remain invariant to dyadic change of image resolution. Multiscale edge detection property of wavelet transforms has been proved to be equivalent to the multiscale nature of the edge detector devised by John Canny.3 This algorithm,1'5,6 popularly known as the Mallat-Zhong discrete wavelet transform (M-Z DWT) exploits the inherent edge extraction property of DWT by evaluating the local maxima of modulus of DWT of images. Another way of utilizing the multiscale analysis property of DWT is to generate progressively lower resolution (higher scale) detail components of image through two-dimensional (2-D) DWT and synthesizing the fine resolution edge map from these multiscale detail components.4

In this paper, we exploit the equivalence of multiscale Canny edge detector to multiresolution edge

map generated by 2-D DWT instead of using one-dimensional (1-D) edge operators. The advantage in our method from M-Z DWT implementation is that by utilizing 2-D DWT, hierarchical scale image approximations and horizontal, vertical and diagonal edge maps at dyadic coarser resolutions are simultaneously generated. We select a particular biorthogonal wavelet filterbank (CDF3.5) pair to prove this equivalence of edge detection. For a Gaussian window of normalized width CTJ = 1, smoothing the image by Canny operator and approximation by scaling operator in biorthogonal DWT remain equivalent. It is well known that final edge extraction in Canny operator is stable with respect to false edges because of hysteresis in the upper and lower bounds of threshold. Still streaking and discontinuities in edges occur because of non-optimized threshold limits.3 Selection of proper threshold bounds is crucial so that no new edge segments are created at dyadic coarser resolutions. Here, genetic algorithm (GA) is utilized to optimize the hysteresis threshold bounds by minimizing the difference in the edge map created by 2-D DWT and Canny edge operator on the dyadic scale image approximations. Finally, the results of the method for recognizing object boundaries in dyadic scale MMWave image approximations are presented.

2. Edge Detection by projection of 2-D D W T

In the M-Z DWT interpretation of wavelet edge detector, let 8(u) be the 2-D scaling function for dyadic decomposition where u = (ul, u2) G TZ?. Wavelet de-

204 Edge Recognition in MM Wave Images by Biorthogonal Wavelet Decomposition and Genetic Algorithm

tail basis functions in two dimensions are chosen as

x dO[u) 2_ d6{u) du\ ' du2

Dilation of 6{u) over 2j scales is 0j(u) = 2-id{2'ju). The dilated wavelets are

^smj.-iim. (1) Wavelet transform (WT) for image f(u) in each of two dimensions k = {1,2} is

Wf{x) = (/(«), rf(x - u)> = (/ * $ ) ( u ) (2)

where * is convolution of two continuous functions. The equivalence of multiscale Canny edge oper

ator to the M-Z DWT is then proved to be1

( ^ / ( « ) ) = 2 J V ( / ^ ) ( U ) - ( 3 )

The gradient operation over convolution of /(it) with 0j (u) in Eq. (3) is the multiscale Canny edge operator. Therefore, M-Z DWT represents the horizontal and vertical details in the image provided the gradient operation in the two sides are proportional. The modulus of WT is

Mf(u,V) = y/(W}f(u))' + (W?f(u))*

and orientation is given by

, fWff(u)\ a = arctan „ r , , . . .

Edges are co-ordinates where the modulus of M-Z DWT is maximum in an one-dimensional (1-D) neighborhood along a direction of increasing a. In its implementation 1-D DWT in both directions are performed on the scaled, smoothened approximation (/ * Oj)(u). Since dyadic decimation in DWT fil-terbanks make the edge locations shift-variant over scales, M-Z DWT is implemented in undecimated DWT filterbanks by inserting 2J zeroes in between the filter coefficients.

In our approach, 2-D projections of the approximation and detail components of image are generated instead of 1-D undecimated DWT. Using biorthogonal filterbanks for perfect reconstruction, projections of approximations and details in horizontal, vertical and diagonal directions are simultaneously available by Mallat's fast 2-D synthesis of DWT components.2

Analysis and synthesis in 2-D biorthogonal spaces are V?._ x> = V? ffi Wj. Thus, finer resolution approximations spaces are the sum of coarser resolution

biorthogonal approximation and details subspaces. Multiresolution tensor product spaces are V?-x, =

V(j_i) ® V(j-\) a n d in 1-D case, V(j_i) = Vj © Wj. Multiresolution biorthogonal projection spaces are given by the distributive property of summation over tensor product as

(4) By closure property of multiresolution spaces

^(j_i) 3 ^ 3 ^(j+i) -> ' • •' therefore, projections of coarser resolution spaces in Eq. (4) are continued it-eratively on Vj. Vj is spanned by dilated shifts in the scaling function Oj(u) and Wj is spanned by ipj(u) shown in Eq. (2). Since convolutions are commutative biorthogonal projections of f{u), that is WT components of f(u) in V,2-^ are

pv5 n/(«) = [ ((he))*e]){u)

©((/**] )*$)(«) ©((/*02)*V])(«) © ( ( / * $ ) *$)(«)] (5)

This shows that at 2^~1) scale the image is summation of approximation at coarser 2J scale and respectively the horizontal, vertical and diagonal edges represented in the projections of detail components of f(u). Modulus of detail projection is

M/(u,2>') = >/(( /*^)*^) 2 + ((/*«?)*^)2

(6) Edges are coordinates in the image where this modulus is maximum in the direction of increasing angle between the two components. Contribution of the diagonal projection is neglected in the reconstruction of image from edges.1

3. Implementation of the Edge Detector

3.1. Biorthogonal Cohen-Daubechies-Feaveau (CDF 3.5) filterbank pair

Scaling and wavelet functions used here for multi-scale edge detection are shown in Fig. 1. 6j(u) in Fig. 1(a) provides symmetric Gaussian smoothing derived from cubic spline function. tpj{u) in Fig. 1(b) is its anti-symmetric derivative that is a quadratic spline function. Biorthogonal analysis filters of CDF

C. Bhattacharya and V. P. Dutta 205

0.8

0.4

0.2

Aniljnn leafing function

> : / ; /

^ i

\ !

\ :

MlfcttNMllltfanctta

- 1 0 1 tsnptenunbtr

(a) (b)

Fig. 1. (a) Interpolated scaling function for CDF3.5; (b) interpolated analysis function for CDF3.5.

3.5 on interpolation produce the basis functions in Fig. 1. The coefficients of analysis and synthesis low pass filters respectively are2

1 3 3 1' h = [h-i ho h\ h<z] = V2

8 8 8 8

= [h

= v/2

h = \h-5 h-i /i_3 h-2 h-\ ho h\ hi hj, h^ h$ h§\ ' 5 15 19 97 26 350 512 512 512 512 512 512 " '

350 26 97 19 15 5 512 512 512 512 512 512

The highpass analysis and synthesis filters are derived as

To derive multiresolution projection components in Eq. (5) in pixel domain 2-D DWT is performed by standard row over column 1-D dyadic analysis filter-bank2 followed by interpolation in synthesis filter-bank.

We show in Fig. 2 comparative performance in detecting edges by 2-D DWT edge detector and Canny edge operator for two scales of successive dyadic coarse resolutions. A relatively simple image of printed circuit board is taken to demonstrate the detectors' performance for horizontal and vertical edges. Magnitude of edge intensity for 2-D DWT edge detector in Fig. 2(b),2(e) is derived by Eq. (6). Orientations of edges are determined by Canny's algorithm for suppression of non-maximal magnitudes in a direction normal to the edges. Smoothing in the Canny edge operator is done by dyadic scaling function shown in Fig. 1(a). The width <TO for Gaussian smoothing at fine resolution is taken to be 1 so that the smoothing operator for image approximations in Fig. 2(a),2(d) is dilated by 2K Therefore, the edge

map in Fig. 2(c),2(f) is by the derivative operation in Canny edge operator and then suppression of non-maximal local magnitude is carried out.

3.2. Threshold Optimization by Genetic Algorithm (GA)

It is well known that Canny edge operator is stable against false edge detection because of upper and lower bounds in the hysteresis threshold. A modulus maximum point is declared an edge point if and only i f T l o w < M / ( u , 2 ' ) < T h i g h .

Although there is close similarity in the edge map in Fig. 2(c),2(f) with 2-D DWT edge map, streaking occurs at higher scales. This is because of nonoptimum selection of thresholds from scale to scale. We search for the optimum bounds of threshold using GA since the two edge maps are mathematically equivalent at each scale. Application of GA for image segmentation is reported somewhere else.7 Typically, fixed-length binary strings in the search population are programmed to crossover and mutate in order to locate regions of interest in the search space. Three basic genetic operators: selection, crossover, and mutation guide this search. Lower and upper thresholds are coded in 16 bits. Probability of crossover is 0.9 and probability of mutation is 0.01 with each generation of 30 population. Here the objective function is to minimize \Mf(u,2j) - V ( / * 6j)(u)\ evaluating the hysteresis threshold at each iteration until the termination condition for genetic search is reached.

4. Results

Results of implementation of the proposed edge detection method in actual multiresolution MMWave scenes are shown in Fig. 3 and 4. The original MMWave ground scene collected by the W-band imaging sensor developed at DEAL, Dehradun is shown in Fig. 3(a). Contrast in the original scene is poor because of low dynamic range of MMWave reflectivity. The contrast-enhanced image is shown in Fig. 3(b). Fig. 4(a)-4(c) show the dyadic coarser resolution approximations of image in Fig. 3(b) derived by 2-D DWT operator.

Corresponding edgemaps by Canny edge operator are respectively in Fig. 4(d)-4(f). It is seen in the edgemaps of Fig. 4(d)-4(f) that prominent objects boundaries are located without streaking. Secondly, object boundaries remain invariant to scale and no

206 Edge Recognition in MM Wave Images by Biorihogonal Wavelet Decomposition and Genetic Algorithm

(a) scale 2 image approximation, (b) 2-D D W T edge map, scale 2. (c) Canny edge map, scale 2

Fig. 2. (a),(d) Projection of image approximation at two dyadic coarse resolutions; (b),(e) edge map at two successive coarser resolutions derived from detail projections by 2-D DWT; (c),(f) edge map at two successive coarser resolutions by Canny edge operator over image approximations in (a),(d).

"IT*

Fig. 3. (a) Ori^hud MM Wove ground scene; (b) contract enhanced MM Wave- ground scene. Table 1. Hysteresis threshold bounds optimized by GA.

Scale (2J)

21

2 2

2 3

Low threshold T\ow

0.4170

0.5847

0.5076

High threshold Thigh

2.4691

2.4044

2.4702

new edge is created as scale becomes coarser. This

is possible because of the optimized bounds of hys

teresis threshold, which are output of search by GA

over 2-D D W T and the Canny edge operator at each

scale. T h e optimized hysteresis threshold for three

edge maps in Fig. 4(d)-4(f) are given in Table 1 be

low.

5* C o n c l u s i o n

In this paper, recognition of edges of objects of in

terest is shown by multiresolution image approxima

tions. The analysis of M-Z D W T is extended to 2-

D D W T as edgemap and approximations of the im

age are available simultaneously. For this, C D F 3.5

biorthogonal wavelet filterbank is used to derive the multiresolution projections. The hysteresis threshold bounds of Canny edge operator are optimized by GA based search by minimizing the distance between the 2-D D W T edge m a p and the Canny edge m a p in MMWave images.

A c k n o w l e d g m e n t s

The authors acknowledge Shri K. Sivakumar and his

team for providing the MMWave image of the sensor

produced in DEAL, Dehradun.

R e f e r e n c e s

1. S. Mallat and S. Zhong, IEEE Trans. Pattern Analysis and Machine Intelligence 14(July 1992).

2. S. Mallat, A Wavelet Tour of Signal Processing. (Academic Press, 1999).

3. J. F. Canny, IEEE Trans. Pattern Analysis and Machine Intelligence 8, 679(November 1986).

4. J. I. Siddique and K. E. Earner, Wavelet-based multiresolution edge detection utilizing gray level edge maps, in Proc. IEEE Int. Conf; on Image Processing (ICIPf98), 1998.

C. Bhattacharya and V. P. Dutta 207

K « - * • * " * •

(a) scale 1 image approximation, (b) '2-D DW'I c-i 1 >•,» m;u>. scale 1 , •!•:• (,.,iiiii\ iiU-r in.-i|). scale 1

r 4... I--*e c •«[& i W i

!

...

t &:

S .a^. tf;

~~ „»

. U^y

.- ;~r ., ,y

K&

, „

^ ^ '"-" • s ^

V)

"?

.^-:

."-'•'

~>~.-,«v

V ?

X * <

i /

4«y

a'i

fc

' i 1 i

Ji

(d) scale 2 image approximation, (e) 2-D D W T edge map, scale 2, (f) Canny edge map, scale 2

Fig. 4. (a),(b),(c) MMWave image approximations at three dyadic resolutions; (d),(e),(f) Corresponding edge map by Canny edge operator with optimized bounds of threshold.

5. Q.-H. Lu and X.-M. Zhang, Multiresolution edge de- 50, 999 (2003). tection in noisy images using wavelet transform, in 7. B. Bhanu and S. Lee, Genetic Learning for Adaptive Proc. IEEE Int. Conf. on Machine Learning and Cy- Image Segmentation. (Kluwer Academic Publishers, bernetics (ICMLC'05), 2005. 1994).

6. K. H. Kim and S. J. Kim, IEEE Trans. Biomed. Eng.

Extended Markov Random Fields for Predictive Image Segmentation

R. Stolkin

Stevens Institute of Technology Hoboken, NJ 07030, USA.

E-mail: [email protected]

M. Hodgetts

Cambridge Research Systems Ltd. UK.

A. Greig

University College London UK.

J. Gilby

Sira Ltd.

UK

Since the 1970s, there has been increasing interest in the use of Markov Random Fields (MRFs) as models to aid in the segmentation of noisy or degraded digital images. MRFs can make up for deficiencies in observed information by adding a-priori knowledge to the image interpretation process in the form of models of spatial interaction between neighbouring pixels. In data fusion problems, interaction might also be assumed between corresponding pixels in two different kinds of image of the same scene. Alternatively, temporal interaction might be assumed between corresponding pixels in consecutive frames of a video sequence. In object tracking or robotic navigation problems, a similar relationship may exist between pixels of an observed image and those of a predicted image, derived from models of the motion and scene. In all of these cases the MRF model can be extended to incorporate this additional knowledge. This paper explains the theory of Extended-Markov Random Field (E-MRF) segmentation techniques, surveys the research which has been crucial to their development and presents results from new work in this area with an application to robotic vision in conditions of extremely poor visibility.

Keywords: MRF; EM; Markov Random Field; Expectation Maximisation; segmentation; tracking; poor visibility

1. Introduction

Fundamental problems in computer and robot vision include the recognition and tracking of viewed objects against some background. Predominantly these processes are reliant, at some level, on image segmentation,1 dividing observed images into regions of object and background.

The problems of interpreting images under conditions of extremely poor visibility have received comparatively little attention from the computer vision community, with existing research largely motivated by underwater robotics applications. In contrast, the human visual system is often able to robustly interpret images that are of such poor quality that they contain insufficient explicit information to do so. We assert that such a system must function

by utilising prior knowledge of the scene in several forms.

Extended-Markov Random Fields (E-MRFs) provide a probabilistic framework for combining observed image data with expectations of that data, based on additional knowledge or prediction, during image segmentation. The following sections briefly introduce Markov Random Field (MRF) segmentation and then survey key literature in the development of both spatio-temporal and spatio-predictive E-MRF techniques. The use of E-MRF models for data fusion is also discussed. Results are presented from recent research, using E-MRF segmentation within an Expectation Maximisation feedback algorithm for robot vision in extremely poor visibility environments.

R. Stolkin, M. Hodgetts, A. Greig and J. Gilby 209

2. Markov Random Fields

Since the 1970s, there has been increasing interest in the use of MRF models to aid in the restoration and segmentation of digital images,2-4 since they can make up for deficiencies in observed information by adding a-priori knowledge to the image interpretation process in the form of models of spatial interaction between neighbouring pixels. Hence, the classification of a particular pixel is based, not only on the intensity of that pixel, but also on the classification of neighbouring pixels. Here, we are concerned with binary segmentation for object tracking, in which pixels can take either of two discrete values, namely object or background. Simplistically, pixels are more likely to belong to the object class if their nearest neighbours are also members of the object class and similarly for background pixels. Historically, the mathematical concepts originate in the statistical mechanics and mathematics literature.5-7

When segmenting an image containing TV" pixels, we seek for the ith pixel a class label, C,, which maximises the joint probability:

P(Ci) = P(C1,C2,...Ci,...CN) (1)

Unfortunately, this implies that such a probability distribution must explicitly characterise the joint statistics of every pixel. In a binary image, this would consist of 2N permutations, an impossibly massive space to search, every time a pixel needs to be classified. This combinatorial explosion is avoided by treating the image as a Markov random field, the fundamental notion associated with Markovianity being that of conditional independence,8 meaning that the probability distribution that describes a particular pixel can be de-coupled from the classifications of all other pixels in the image, except for those in a small local neighbourhood. For the pixel at image location

P(Ci,j) = •P(Ci,j)Ci+mj+n(mi„ek)) (2)

where k denotes a small local neighbourhood around (i, j ) , typically taken to include the eight nearest neighbour pixels (see figure 1).

To evaluate this expression for specific permutations of neighbourhood class labels, the MRF is characterised by a Gibbs distribution of the form:

e-Ui,3

P(d,i) = —%- (3)

where Z is included as a normalising constant to prevent equation (3) returning probabilities greater than one. The exponential part of this equation is defined as:

Uij = 2_^ J\Ci,j, Ci+mj+n) (4) m,n£k

where J is a function defined as:

[ - 1 , if a = b Ja,b ={ (5)

\ 0 , iia^b

('./)

Fig. 1. Conventional Markovian neighborhood. Grey squares indicate those pixels whose classifications influence the classification of pixel (i,j).

3. Extending Markov dependency

The origins of E-MRF ideas can be traced to the work of Bouthemy,9'10 who is concerned with the interpretation of murky underwater image sequences for robot navigation. Crucially, Bouthemy extends the notion of Markov dependency to include, not only contributions from a given pixel's neighbourhood in the observed image, but also a contribution from the corresponding pixel in the previous frame of the image sequence. Thus Markov dependency becomes both spatial and temporal.

E-MRF image models have also been used for data fusion. Jones,11'12 uses an E-MRF model to combine a high resolution visible light image with a relatively low resolution thermal image from an infrared camera, for a surveillance application. Extended-Markov Dependency is assumed between pixels in the visible light image and corresponding pixels in the infra-red image.

Fairweather13 and Hodgetts,14 also concerned with underwater robotics, extend Markov dependency such that the local neighbourhood surrounding any particular pixel also includes a contribution from the corresponding pixel in a predicted image (figure 2).

210 Extended Markov Random Fields for Predictive Image Segmentation

Predicted image Segmentation via maximum likelihood

Fig. 2. Extended Markovian neighborhood. Grey squares indicate those pixels whose classifications influence the classification of pixel (i, j). The extended nighbourhood now includes the corresponding pixel from a predicted image.

The predicted image is projected using a 3D model of the object being tracked and an estimate of camera position based on a Kalman filtered model of the robot trajectory. The method is tested on a variety of degraded images, demonstrating superior performance to both conventional2-4'15 and also spatio-temporal9 '10 MRP segmentation.

Equations (2) to (5) describe a conventional MRF image model in which pixel class labels are considered to be spatially dependent. However, Markov dependency is now extended to include a contribution from a predicted class label, dj, derived from a previous frame,9'10 a predicted image,13'14 or a corresponding image from a different kind of imaging device.11'12 Now:

"{^i,j> = "{^i,ji ^i+m,j+"(m,T.efc)' ^i,}) = ~y

(6) The exponential part of the Gibbs distribution now consists of weighted components:

Ui,j = 2_j SiJ(Ci,j>Ci+m!j+n) + S2J(Cij,Cij) m,n€k

(7) where Si and 52 are weighting constants which adjust the relative significance of information derived from the observed image versus information derived from the predicted image.

Thus, the Extended-Markov Random Field model provides a convenient means of determining the prior probability distribution for any particular pixel class label, while incorporating additional prior knowledge in the form of temporal or predictive dependencies, or fused data from an alternative imaging system.

We wish to segment each pixel (i,j), of intensity Jy, by choosing a class label, CV,, which maximises the a-posteriori probability P(Cij\Iij). From Bayes' law we have:

P{Cij | J y ) oc P(Iij \dj) x P{Ca) (8)

where the prior probability, P(CV,), is modelled by the Extended-Markov Random Field, equation (6).

Various approaches are possible for determining the class conditional probabilities, P(Iij\dj). In general it is not possible to know the true shape of these distributions over an image without already knowing the true segmentation for that image.

Often class conditional distributions are estimated from historical data, e.g. averaged or best fitted over a training set of images for which the true class labels are known. This approach may not be appropriate if, for example, lighting conditions change significantly with time over the image sequence, as might be expected in an underwater environment where a directional light source is mounted on a moving vehicle. Different distributions may be necessary for different images.

If the class conditional distributions tend to conform to a specific shape or model (often a normal distribution is assumed), then that model can be best fitted directly to the observed image statistics.16

Other authors13 '14 assume accurate prior knowledge of class conditional distributions when computing conventional, spatio-temporal or spatio-predictive MRF probabilities.

Here we describe a novel approach, which makes additional use of prior knowledge, for predicting the shape of class conditional distributions, P{Iij\Cij). The vision system is allowed to re-learn new class-conditional models for each image frame by making the approximation:

PiliilQj) * Pil^Ca) (9)

where CV, denotes the predicted class label of the pixel (i,j). In other words, the predicted image (found by projecting the object model based on estimated camera co-ordinates) is used to define a set of provisional (predicted) class labels, C_, for the observed image, from which means and variances of pixel intensities for each image region (object and

R. Stolkin, M. Hodgetts, A. Greig and J. Gilby 211

background) can be estimated. The validity of this approximation is obviously dependent on how closely the estimated camera co-ordinates approximate the true camera co-ordinates.

We approximate the class conditional distributions with Normal distributions which are particularly useful since they are of exponential form. The prior probabilities (equation 6) are also of exponential form and so it is easy to arrive at a log-likelihood function. The overall likelihood for a particular classification of a particular pixel is now:

P ( C 4 j ) x P ( / y | C y )

Where ai.. and uc., are the variance and mean of the class conditional distribution of pixel intensities that corresponds to the choice of CV, that is currently being considered for pixel (i,j). This results in the negative log-likelihood function:

/ _, S\J{Cij, C7j+mij-)-n) + D2J(Ciyj,Cij) m,n€k

-l^ltV2 (11) Cij

Si and 52 are weights which determine the significance of the class values of nearest neighbour pixels and predicted pixels respectively. They thus effect the relative significance of observed and predicted data. It is not obvious how these values should be determined and other researchers9,10'13-15 suggest experimenting to find useful values for these constants by trial and error. In good visibility, it is desirable to rely on observed information while taking comparatively little notice of error prone predictions derived from extrapolating the previous camera trajectory. Hence Si will be large and 52 comparatively small. Conversely, given the absence of observed information in bad visibility conditions, it is necessary to make greater use of predicted information. In this case, much larger values of 52 must be used. Future work may investigate methods by which these values can be automatically adjusted in response to varying visibility conditions.

The space of all possible image interpretations contains many variables since it is necessary to consider all possible class label permutations over all pixels in the image. It is not possible to search this

space (of size 2N where N is the number of pixels in the image) exhaustively in order to locate its global minimum (minimum negative likelihood). A variety of iterative algorithms have been suggested, which attempt to optimise the set of pixel class labels with respect to a statistical criterion. For a review and comparison of several of these techniques see Dubes.15

Simulated Annealing,4 (SA), belongs to the class of stochastic relaxation algorithms. SA is theoretically guaranteed to find a globally optimal labeling, however it is relatively computationally expensive and slow. In contrast, the Iterated Conditional Modes, (ICM), algorithm2'3 is not guaranteed to find an optimum set of pixel labels, being vulnerable to convergence on local minima. It is, however, several orders of magnitude faster than simulated annealing and therefore more suitable for real time applications. Other approaches include Maximiser of Posterior Marginals,2'3'15'17 (MPM), Highest Confidence First,18-19 (HCF), and Graph Cut methods.19-20

For simplicity, we use the ICM algorithm in our work for proof of principle, as do the authors of much of the related research.

5. Mutual refinement of segmentation and pose estimates via the EM / E-MRF algorithm

We have further developed these ideas in several ways. Firstly, we relearn class models at every frame, equation (9). Secondly, we note that the use of prediction results in an iterative feedback scheme, (figure 3), which can be shown to be a variant of the Expectation Maximisation (EM) algorithm.21

An estimated camera position (derived from a motion model) is used to generate a predicted image by projecting a known model of the observed object. Predicted class labels, from the predicted image, help segment the observed image via E-MRF segmentation. The object model is then best fitted to the segmented image to extract an improved camera position estimate, which is then recycled as an input to the next iteration. Iteration can be terminated when no further changes to pixel class labels occur, or when changes in estimated position fall below a threshold. The number of iterations will vary with the accuracy of the initial position estimate.

212 Extended Markov Random Fields for Predictive Image Segmentation

The object distribution sharpens and pulls to the right as the algorithm learns that the object is relatively light coloured and consistently so. Correspondingly, the background distribution pulls to the left as the algorithm learns that background pixels are dark.

Fig. 3. An iterative segmentation and pose estimation scheme for robot navigation. An estimated camera position is used to project a predicted image which helps segment an observed image using E-MRF. A model of the observed object is then fitted to the segmented image to extract an improved estimate of camera position relative to the object. The improved camera position is used to project an improved predicted image and the process is iterated until convergence

6. Results

We demonstrate E-MRF segmentation with an image taken from our data set of real video sequences with known ground-truth.22 These images feature a scale model oil-rig structure viewed in extremely bad visibility. Figure 4 shows the results of four iterations of E-MRF segmentation. The superimposed red wire frames indicate the position estimates and corresponding predicted image segmentations at each iteration. Note that the initial predicted position (first image) is significantly erroneous, and that the algorithm iteratively homes in on the true position and corresponding correct segmentation. Also note that the model fitting process enables the algorithm to ignore the large artefact (bottom right corner of the image) caused by backscattering of spotlight illumination. Projecting a predicted image from the most recent position estimate gives a very clean secondary segmentation (defined by the red wire frame outline in the figures).

Figure 5 shows the results of thresholding the image and segmenting using a conventional MRF model. Both these segmentation methods are inadequate for such poor visibility conditions, although the thresholded image is useful for initialising the E-MRF optimisation.

Additionally, our technique for predicting class conditional distributions, equation (9), enables these distributions to be relearned with each iteration. Figure 6 shows how these distributions change over four iterations.

Fig. 4. Both camera position estimate and image interpretation are mutually refined over four iterations of E-MRF segmentation. Left column shows the E-MRF segmentations which improve with each iteration. Right column shows the resulting position estimates and corresponding predicted images. Incorrect pixel labels (in the secondary projected segmentation) at each iteration are 12%, 7.2%, 1.6% and 0.8%

7. Conclusion

MRFs have long been used as models for segmenting individual images. More recently, researchers have explored ways of extending these models to incorporate the extra knowledge inherent in video sequences. These extensions include assuming temporal relationships between pixels of successive image frames,

R. Stolkin, M. Hodgetts, A. Greig and J. Gilby 213

^^P Tl Fig. 5. Thresholded image (left), used to initialise the E-MRF segmentation and conventional MRF segmentation (right).

Fig. 6. Re-learning of class conditional distributions with each iteration.

and also relationships between pixels in an observed

image and corresponding pixels in a predicted image.

Similar techniques have also been used to fuse da ta

from two different kinds of imaging device.

E-MRF techniques are particularly useful in con

ditions of extremely poor visibility, for which con

ventional M R F models and other segmentation tech

niques are inadequate.

These ideas have often been presented in the con

text of underwater robotics. However, it is possible

tha t E-MRF techniques could be applied to many

other da ta fusion tasks which involve discretely parti

tioned spaces, e.g. 2D and 3D medical imaging (com

bining information from different kinds of imaging

device) and computat ional ocean modelling (fusing

now-casts and previously generated forecasts).

R e f e r e n c e s

1. N. Pal and S. Pal. A review on image segmentation techniques. Pattern Recognition, 26(9): 1277-1294, 1993.

2. J. Besag. Spatial interaction and the statistical analysis of lattice systems. Journal of the Royal Statistical Society, series B, 36:192-236, 1974.

3. J. Besag. On the statistical analysis of dirty pictures. Journal of the Royal Statistical Society, series B, 48:259-302, 1974.

4. S. Geman and D. Geman. Stochastic relaxation: Gibbs distributions and the bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 9:721-741, 1984.

5. W. Gibbs. Elementary Principles of Statistical Mechanics. Yale University Press, 1902.

6. A. Markov. Extension of the law of large numbers to dependent events (in russian). Bull. Soc. Phys. Math. Kazan, 2(15):155-156, 1906.

7. E. Ising. Zeitschrift Physik, 31, 1925. 8. J. Zhang, P. Fieguth, and D. Wang. Random Field

Models. Academic Press, 2000. 9. P. Bouthemy and P. Lalande. Determination of ap

parent mobile areas in an image sequence for underwater robot navigation. Proceedings of IAPR Workshop on Computer Vision: Special Hardware and Industrial Applications, pages 409-412, 1988.

10. P. Bouthemy and P. Lalande. Motion detection in an image sequence using gibbs distributions. Proceedings IEEE International Conference on Acoustics, Speech and Signal Processing, pages 1651-1654, 1989.

11. G. Jones, M. Hodgetts, R. Allsop, N. Sumpter, and M. Vicencio-Silva. A novel approach for surveillance using visual and thermal images. Proceedings of the DERA/IEE Workshop on Intelligent Sensor Processing, 2001.

12. G. Jones, R. Allsop, and J. Gilby. Bayesian analysis for fusion of data from disparate imaging systems for surveillance. Image and vision computing, 21:843-849, 2003.

13. A. Fairweather. Robust Interpretation of Underwater Image Sequences. PhD Thesis, University of London, 1997.

14. M. Hodgetts, A. Greig, and A. Fairweather. Underwater imaging using markov random fields with feed forward prediction. Journal of the Society for Underwater Technology, 23(4):157-167, 1999.

15. R. Dubes, A. Jain, S. Nadabar, and C. Chen. Mrf model-based algorithms for image segmentation. Proceedings IEEE 10th International Conference on Pattern Recognition, pages 808-814, 1990.

16. J. Kittler, J. Illingworth, and J. Foglein. Threshold selection based on a simple image statistic. Computer Vision, Graphics and Image Processing, 30:125-147, 1985.

17. J. Marroquin, S. Mitter, and T. Poggio. Probabilistic solution of ill-posed problems in computational vision. Journal of the American Statistical Association, 82:76-89, 1987.

18. P. Chou and C. Brown. The theory and practice of bayesian image labelling. International Journal of Computer Vision, 4:185-210, 1990.

19. T. Meier, K. Ngan, and G. Crebbin. A robust marko-vian segmentation based on highest confidence first. Proceedings of the international conference on image processing, 1:216-219, 1997.

214 Extended Markov Random Fields for Predictive Image Segmentation

20. Y. Boycov, O. Veksler, and R. Zabih. Fast approximate energy minimization via graph cuts. Proceedings of the International Conference on Computer Vision, 1:377-345, 1999.

21. A. Dempster, N. Laird, and D. Rubin. Maximum likelihood from incomplete data via the em algo

rithm. Journal of the Royal Statistical Society, series B, 39:1-38, 1977.

22. R. Stolkin, A. Greig, and J. Gilby. A calibration system for measuring 3d ground truth for validation and error analysis of robot vision algorithms. Journal of Measurement Science and technology, 2006.

215

External Force Modeling of Snakes Using DWT for Texture Object Segmentation

Surya Prakash and Sukhendu Das

Visualization and Perception Lab, Department of Computer Science and Engineering,

IIT Madras Chennai-600 036, India.

E-mail: [email protected], [email protected]

Snakes, also known as active contours are extensively used in computer vision and image understanding applications. They are energy minimizing deformable contours that converges at the boundary of an object in an image. Deformation in contour is caused due to internal and external forces acting on it. Internal force is derived from the contour itself and external force is invoked from the image. Traditional active contours proposed by Kass et al. only work for the normal intensity images and fail to perform segmentation task in presence of texture. This limitation comes due to the limited ability of the external force present in the traditional snake, which uses directly image pixel's intensity information for its formulation. Here, we present a new external force modeling technique for snakes, which can work in presence of texture. It uses texture features for external force modeling, which are derived using discrete wavelet transform. To demonstrate our model, we use various synthetic and natural texture images.

Keywords: Snakes, active contours, texture, object segmentation, wavelet, DWT

1. Introduction

Snakes,1 also known as active contours are extensively used in computer vision and image understanding applications. They are energy minimizing deformable contours that converge at the boundary of an object in an image. Deformation in contour is caused because of internal and external forces acting on it. Internal force is derived from the contour itself and external force is invoked from the image. The internal and external forces are defined so that the snake will conform to object boundary or other desired features with in the image. Snakes are widely used in many applications such as segmentation,2 shape modeling,3 edge detection,1 motion tracking4 etc. Active contours can be classified as either parametric active contours1'5 or geometric active contours6'7 according to their representation and implementation. In this work, we focus on parametric active contours, which synthesize parametric curves within image domain and allow them to move towards the desired image features under the influence of internal and external forces. The internal force serves to impose a piecewise continuity and smoothness constraint whereas external force pushes the snake towards salient image features like edges, lines and subjective contours. External force in the traditional snake is defined as the negative of the image gradient. In the presence of such external force, snake is attracted towards large image gradients i.e.

towards the edges in the image. So if it is applied to the textured images, it will often get stuck on local texel (micro-units or cells of a texture) edges and converge at non-object boundary.

To overcome this effect, we here present a new class of external force for textured images, which we name as texture force. The snake in presence of texture force runs over the texture image surface and detects the object boundary of a texture surface, on a background texture. Texture force does not use directly the image pixel intensity values for its modeling. It considers the texture properties of the image.

2. Background

2.1. Parametric Snake Model

A traditional active contour is denned as a parametric curve v(s) = [x(s),y(s)}, s G [0,1], which minimizes following energy functional

i^(a|v' (s)|2+/3|v (s)\2)+Eext(v(s))ds (1)

where, a and f3 are weighting constants to control the relative importance of the elastic and bending ability of snake respectively, v (s) and v (s) are the first and second order derivatives of v(s) and Eext is derived from the image so that it takes on its smaller values at the feature of interest such as edges, object boundaries etc. For an image I(x,y), where (x,y) are spatial co-ordinates, typical external energy is

216 External Force Modeling of Snakes Using DWT for Texture Object Segmentation

defined as follows to lead snake towards step edges1

Eext = -\VI(x,y)\2 (2)

where, V is gradient operator. A snake that minimizes E must satisfy following Euler equation

av"(s)-0v""(s)-VEext = O (3)

Eq. 3 can also be viewed as force balance equation

Fint + Fext = 0 (4)

where, F i n t = av"(s)-/?v""(s) andF e x t = ~VEext. Fint, the internal force, is responsible for stretching and bending and F e x t , the external force, attracts snake towards the desired features in the image.

2.2. Discrete Wavelet Transform and Scalogram

The discrete wavelet transform (DWT) analyses a signal based on its content in different frequency ranges. Therefore, it is very useful in analyzing repetitive patterns such as texture. DWT decomposes a signal into different bands (approximation and detail) with different resolution in frequency and spatial extent. Let I(x) be the image signal and ipUtS(x) be a wavelet function at a particular scale, then signal filtered at point u is obtained by taking the inner product of the two < I(x),tpUtS(x) >. This inner product is called wavelet coefficient of I(x) at position u and scale s.8 Scalogram9 of a signal I(x) is the variance of this wavelet coefficient:

«;(«,«) = E { | < ! ( * ) , ^ u , . ( x ) > | 2 } (5)

The w(u, s) has been approximated by convolving the square modulus of the filtered outputs with a Gaussian envelop of a suitable width.9 The w(u, s) gives the energy accumulated in a band with frequency bandwidth and center frequency inversely proportional to scale.

mm rjr (a) i b ;

Fig. 1. (a) Synthetic texture image, (b) Magnified view of the 21 X 21 window of texture cropped at point P (marked in RED color), shown in Fig. 1(a).

3. Texture Feature Extract ion

In this section, we explain how the wavelet transform is used to extract texture features necessary for texture force estimation. It discusses the computational framework based on multi-channel processing. We use DWT-based dyadic decomposition of the signal to obtain texture properties. A simulated texture image shown in Fig. 1(a) is used to illustrate the computational framework with the results of intermediate processing.

Modeling of texture features at a point in an image involves two steps: scalogram estimation and texture feature estimation. To obtain texture features at a particular point (pixel) in an image, a n x n window is considered around the concerned point (see Fig. 1(b)). Intensities of the pixels in this window are arranged in the form of a vector of length n2

whose elements are taken column wise from the nxn cropped intensity matrix. This intensity vector (signal), which basically represents the textural pattern around the pixel, is subsequently used in the estimation of scalogram.

(b) (c)

Fig. 2. (a) 1-D texture profile of the texture window shown in Fig. 1(b), (b) Scalogram of the signal shown in Fig. 2(a), (c) Texture feature image for the image shown in Fig. 1(a).

Scalogram estimation: An input signal, obtained after arranging the pixels of n x n window as explained above, is used for the scalogram estimation. This signal is decomposed using wavelet filter. We use orthogonal Daubechies 2-channel (with dyadic decomposition) wavelet filter. Daubechies filter with level-L dyadic decomposition, yields wavelet

Surya Prakash and Sukhendu Das 217

coefficients {AL, DL, DL-J, .., Dt} where, Ai represents approximation coefficient and Di's are detail coefficients. The steps of processing to obtain scalo-gram from the wavelet coefficients are similar to that described in.10'11 Fig. 2(b) presents an example of scalogram obtained for signal shown in Fig. 2(a) using level-4 DWT decomposition.

Texture feature estimation: Once the scalogram of the texture profile at a particular point is obtained, a post-processing step is carried out to eliminate non-significant bands of the scalogram and only significant bands are used for the further processing. This is done since only significant bands contain major texture features information. Removal of non-significant bands helps in removing the redundant information and making the computation fast. Let wavelet decomposition is done up to level-L and it gives following wavelet bands B = {AL,DL,DL-i,..,Di}. Let Bi is a ith wavelet band in set B. We use following algorithm to determine significant and non-significant bands.

for each Bi do if variance for band Bi < threshold

Bi is non-significant band else

Bi is significant band end-if

end-for

where threshold is decided empirically. Variance of band Bi is defined as var(I?j) = E[(JBJ—//J)2] where Hi is the mean of all wavelet coefficients belonging to band Bi. Once the estimation of scalogram and its significant bands is over, significant bands are used for texture feature estimation. Texture features are estimated from the "energy measure" of the wavelet coefficients of the significant bands. This texture feature is similar to the "texture energy measure" first proposed by Laws.12

Let for pixel k, Dk is the set of all significant bands and w is a wavelet coefficient in a band. Then the energy measure of pixel k is calculated to be the averaged Zi-norm as

** = ;;{ £!>} (6) VxeDkwex )

where, n is the sum of cardinalities of all the members of Dk. These energy measures for all pixels in an image constitute "texture feature image" which is further used in texture force modeling. Fig. 2(c) shows a texture feature image for the texture image shown in Fig. 1(a). Pixels belonging to the same texture region exhibit same energy level.

4. Modeling of Texture Force

This analysis is based on the gradient present in the texture feature image. Let for a given texture image I(x, y), F(x, y) be the texture feature image obtained as explained in previous section. The external energy of the snake based on the gradient present in the texture feature image can be defined as follows (similar to Eq. 2)

Et?t=-\VF(x,y)\2 (7)

As done in Eq. 4, texture force (external force), which causes the change in this energy (i.e. El^t), can be defined as follows

F% = -VElft (8)

To find the object boundary, active contour deforms so it can be represented as the time varying curve v(s,t) = [x(s, t),y(s,t)] where s € [0,1] is arc-length and t € R+ is time. Dynamics of the contour in presence of texture force can be governed by following equation

7 V t = F i n t + F * S (9)

where, Vt is the partial derivative of v w.r.t. to t, —yvt is the damping force and 7 being an arbitrary non-negative constant. Fint and F ^ are internal and texture forces respectively. The contour comes to rest when the net effect of the damping, internal, and texture force reaches to zero, which eventually happens when deforming contour reaches the texture object boundary. The texture force developed here pushes active contours towards texture object boundary.

5. Experimental Results

To get the boundary of a particular object using active contours in presence of texture, a contour is initialized near the desired object. Contour is then allowed to deform towards the object boundary until it

218 External Force Modeling of Snakes Using DWT for Texture Object Segmentation

latches around the object. In case of textured images, object boundary is identified as the point where texture property changes i.e. where two texture regions meet. In texture surface, snake in presence of texture force stops moving as it gets different texture region. For a snake to stop at the texture boundary! net effect of the dampingf internal and external forces should be zero for all snake points at the object boundary. To demonstrate the performance of the snake in the presence of texture force, various kinds of synthetic and natural textures are used. We have used Daubechies 8»tap 2-channel liter for DWT decomposition.

The first example is of a texture image (Fig. 3) composed of two textures taken from a widely used Brodatz photographic album.13 Contour is initialized around the central texture and is allowed to shrink in presence of texture force. Snake took 14 iterations to converge at the central texture boundary. Texture features at snake points are estimated by taking a 15 x 15 window at each point. DWT decomposition was done up to level-4. Texture feature image for this test image in shown in Fig. 4(a). The resulting segmentation is shown in Fig. 4(b), where the identified texture object boundary is shown in dark black color.

Fig. 3. Texture image compose*i of two Brodatz textures.

(a) (b)

Fig. 4. (a) Texture feature image for the texture image shown in Fig. 3, (b) Final Segmentation result of Fig. 3. Dark black contour shows the estimated boundary of the central texture.

We present another segmentation result for an image (Fig. 5) composed of two Brodatz textures. Contour convergence to the central texture boundary, in presence of texture force, took 11 iterations. Texture features at contour points are estimated by

taking a 13 x 13 window. DWT decomposition was done up to level-4. Texture feature image for this test image is shown in Fig. 6(a). The resulting segmentation is shown in Fig. 6(b), where the identiled texture object boundary is shown in dark black color.

Fig. 5. Texture image composed of two Brodatz textures.

(a) (1»)

Fig. 6. (a) Texture feature image for the texture image shown in Fig. 5S (b) Final Segmentation result of Pig. 5. Dark black contour shows the estimated boundary of the central texture.

Fig. 7. Natural real life test image of zebra.

Fig. 8. (a) Texture feature image of zebra (shown in Fig. 7), (b) Final Segmentation result for zebra. Dark black contour shows the estimated boundary of zebra

Fig. 7 shows a natural real life test image of zebra. Contour convergence to the boundary of zebra,

Surya Prakash and Sukhendu Das 219

in presence of texture force, took 15 iterations. Texture features at contour points are estimated by taking a 11 x 11 window. DWT decomposition was done up to level-4. Texture feature image of zebra is shown in Fig. 8(a). The resulting segmentation is shown in Fig. 8(b), where the identified boundary of the zebra is shown in dark black color.

(a) (b)

Fig. 9. Comparison of segmentation results of zebra image, (a) Segmentation result obtained using proposed technique, (b) Segmentation result obtained in.14 Dark black contour shows the estimated boundary of zebra in both the cases.

6. C o n c l u s i o n

In this paper, we have introduced a new external force for snakes, which we call as texture force. Snake, in presence of texture force, can be used for the texture object segmentation. To model texture force, first texture features are estimated using wavelet decomposition which are further used in texture force modeling. Texture force is subsequently used in parametric snakes for texture object segmentation. Main novelty of this study is in the representation of the texture features and the modeling of texture force based on them. We validate our model with a few synthetic and natural texture images. Results obtained using proposed technique are quite satisfactory. In Fig. 9, we compare our segmentation result of zebra with the segmentation result obtained by Sagiv et al. in14 for the same. The result obtained by proposed technique (Fig. 9(a)) is comparable with

the result obtained in14 (Fig. 9(b)). Since proposed segmentation technique uses parametric snake, it is computationally less expensive compare to the technique presented in14 which uses geodesic active contour.

References

1. M. Kass, A. Witkin and D. Terzopoulos, International Journal of Computer Vision 1, 321(January 1988).

2. F. Leymarie and M. D. Levine, IEEE Transactions on Pattern Analysis and Machine Intelligence 15, 617 (1993).

3. D. Terzopoulos and K. Fleischer, The Visual Computer 4, 306 (1988).

4. D. Terzopoulos and R. Szeliski, Tracking with Kalman snakes, in Active Vision, eds. A. Blake and A. L. Yuille (MIT Press, Cambridge, MA, 1992) pp. 3-20.

5. L. D. Cohen, CVGIP: Image Understanding 53, 211 (1991).

6. V. Caselles, F. Catte and T. Coll, Numerische Math-ematik 66, 1 (1993).

7. V. Caselles, R. Kimmel and G. Saprio, International Journal of Computer Vision 22, 61 (1997).

8. S. Mallat, A Wavelet Tour of Signal Processing (Academic Press, 1999).

9. M. Clerc and S. Mallat, IEEE Transactions on Pattern Analysis and Machine Intelligence 24, 536(April 2002).

10. T. Greiner and S. Das, Automatisierungstechnik (2005).

11. S. G. Rao, S. Das, T. Greiner and M. Kalra, Accepted in IEE VIE Conference, Bangalore, September 26-28 (2006).

12. K. Laws, Textured image segmentation, PhD thesis, (Dept. of Electrical Engineering, University of Southern Californiai, 1980).

13. P. Brodatz, Textures: A photographic album for artists and designers (1966).

14. C. Sagiv, N. Sochen and Y. Zeevi, IEEE Transactions on Image Processing 1, 1 (February 2004).

I-FISH: Increasing Detection Efficiency for Fluorescent Dot Counting in Cell Nuclei

Shishir Shah*

Quantitative Imaging Laboratory University of Houston

Department of Computer Science Houston, TX, U.S.A.

* E-mail: [email protected]

Fatima Merchant

Advanced Digital Imaging Research, LLC. 2450 South Shore Blvd., Suite 305

League City, TX U.S.A. E-mail: [email protected]

This paper presents a methodology and results for increased detection efficiency (IDE) in the analysis of Fluorescent in situ Hybridization (FISH) images by recovering the radiance curve of the imaging device (CCD camera) used to capture the fluorescence signals and remapping the nonlinear luminance to generate a high contrast high dynamic range image. The remapping is based on a criterion that allows for maximum detectability of the signals. This leads to images where saturated values are attenuated while the weak signals are amplified. An automated dot counting algorithm is used to process the remapped images and evaluate the dot detection efficiency. Results of dot counting on a set of 2000 images are presented and compared to those obtained without the use of the proposed methodology.

Keywords: Image Enhancement; Nonlinear Radiance Mapping; Fluorescence in situ Hybridization

1. Introduction

Chromosomal aberrations are a variation from the normal, either in structure or number of chromosomes, which result from an exchange of genetic material between two or more chromosomes or from a rearrangement of genetic sequences contained in a single chromosome. The analysis of such aberrations can be useful both in a clinical and in a toxicological context. Different staining techniques allow analysis of different kinds of abnormalities. A particularly useful cytogenetic technique for the analysis of aberrations is Fluorescence in situ Hybridization (FISH).1

FISH relies on the use of fluorescent dyes that are attached to the DNA probes and when excited by one wavelength of light, emit light at a second, longer, wavelength. In typical FISH study, three fluorescent dyes are used, one to stain the cell nuclei, and two to detect DNA sequences. FISH signals in interphase cells become visible as colored dots. The analysis of a preparation consists of detecting the dots, after which the number of dots per cell can be counted or their relative positions can be measured. With chromosome enumeration, dots are counted for a large number of cells to determine the distribution of chromosomes per cell and to be able to detect small aberrant sub-populations. In an clinical setting, the num

ber of cells to be analyzed depends on the frequency of aberrant cells and the count frequency. In practical situations, this can vary from a few to more than 10,000 cells.2"4

FISH has been applied to preimplantation genetic diagnosis of X-linked diseases,5-7 common ane-uploidies using either human blastomeres or oocyte bodies8 and, more recently, translocations.9 Although FISH is widely used for genetic analyses, its reliability and accuracy depends on the types of probes, cells and their fixation. The major disadvantage associated with the use of fluorescent dyes is the weak signal emanating from a cell or its labeled constituents. In typical studies, a background pixel consists of 5,000 photons, a DAPI-stained nucleus pixel is an additional 3,000 photons, and a labeled chromosome pixel can be an additional 10,000 photons. These numbers are at least a factor of 100 smaller than the values that are obtained for images acquired through a conventional CCD camera at room-level illumination.10,11 Several investigators have evaluated the hybridization efficiencies of interphase FISH using probes specific for chromosomes 13, 18, 21, X and Y. The results reported vary from 60% to 98% for XY detection and 42% to 88% for autosomal aneuploidy detection depending on the type

Shishir Shah and Fatima Merchant 221

of cells and the choice of probes.10 '12-16 These results can be acceptable for rapid detection of chromosome aneuploidies, however, when the cells sought represent a very low proportion of the total cell population, a very high efficiency is desired while maximizing specificity and sensitivity. Proposed approaches target either the optimization of FISH protocols including alternate selection of fluorophores or identification of different cell types for investigation.8,17

In this paper, we present an approach to improve the efficiency of detecting fluorescent signals in FISH images by recovering the radiance map of the CCD camera. In general, there exists a nonlinear relationship between the true radiance of the fluorescent probe and the detected pixel value. Although the charge collected by a CCD element is proportional to its irradiance, most cameras apply a nonlinear mapping to the CCD outputs before they are written to the storage medium. The most significant nonlinearity in the response curve of any camera is at its saturation point, where any pixel with a radiance above a certain level is mapped to the same maximum image value. Further, due to the limited dynamic range of the camera, two signals having radiance values that exceed the mapping limits of the camera cannot be sensed simultaneously. This leads to saturation on one extreme or extremely weak or missing signals on the other. We use a simple technique to recover the response curve by using a set of images taken with varying but known exposure durations. This allows us to generate a high dynamic range (HDR) image that covers the full extent of the radiance captured by all the images. We further analyze the distribution of each of the fluorescent signals and remap the HDR image values to generate a high contrast image with compressed dynamic range. The remapping is based on a criterion that allows for maximum detectability of the signals. This leads to images where saturated values are attenuated while the weak signals are amplified. By testing our approach on a large dataset of FISH images, we show that dot counting specificity and sensitivity can be improved.

Rest of this paper is organized as follows: Section 2 describes the imaging setup and the method for recovering the CCD response curve to generate a high dynamic range image. Image contrast enhancement and remapping of the computed response curve is presented in section 3. Results of the developed

methodology are presented in section 4 along with comparison of results obtained on the same dataset processed without correcting for the camera response curve. Finally, conclusions and a summary of this study are presented in section 5.

2. High Dynamic Range Imaging

Computing and acquiring images with high dynamic range can be done in various ways. In this paper, we utilize the method proposed by Debevec & Malik.18

Interphase FISH slides labeled with DAPI, FITC, and Texas Red were used for analysis. A series of images at exposures varying from 0.03 seconds to 1.4 seconds in intervals of 0.04 seconds were collected. These images formed the input for recovering the camera response curve. If Ei is the irradiance value at pixel i, Atj is the set of known exposures, Z^ is the image pixel value at spatial index i at exposure j , the camera response function / can be computed as:

Zij = /(EtAtj) (1)

Assuming / to be monotonic, the irradiance Ei and the inverse function of / can be solved in the least-square error sense by rewriting equation 1 as:

g(Zij)=\nEi + \a&tj (2)

where g = I n / - 1 . To better enhance the weak and the saturated intensities, a weighted quadratic function is used that allows the function g and the irradi-ances Ei to be recovered that best satisfy the set of equations arising from 2. Recovering g only requires recovering the finite number of values that g{Z) can take since the domain of Z, pixel brightness values, is finite. Letting Zmin and Zmax be the least and greatest pixel values, N be the number of pixel locations and P be the number of images, the problem is formulated as one of finding the (Zmax — Zmin + 1) values oig(Z) and the N values of In Ei. This is done by solving the following quadratic objective function:

JV P

° = 5ZZ)N2i i ) (» (^ - ln£i - in Atj)]3 (3) »=i j = i

+A £ MZ)g"(Z)}2

Zmin+l

where w defines the weight factor that moderates weak and saturated intensities and A is a regular-ization parameter that moderates the smoothness

222 I-FISH: Increasing Detection Efficiency for Fluorescent Dot Counting in Cell Nuclei

term given by the sum of squared values of the second derivative of g to ensure that the function g is smooth.

Once the response curve g is recovered, it can be used to quickly convert pixel values to relative radiance values, assuming the exposure Atj is known. We compute the curve based on one set of images and use the same to recover the radiance values for every other image taken while imaging all the cells. The radiance curve is independently computed for each of the three image channels used to capture the fluorescence signal based on DAPI (blue), FITC (green), and Texas Red (red) labeling of the nuclei. A representative set of images taken to recover the response curve are shown in figure 1 and the corresponding response curve shown in figure 2.

Fig. 1. Images acquired to recover the camera response curve at 0.1, 0.5, 1.0, and 1.4 seconds.

Normalized Response Curve (g(Z))

1 • « • Green Channel • Red Channel

] • Blue Channel

:f

- / ,:«?'

^ * r * ' < ^

0 50 100 150 200 250 300 pixel value (Z)

Fig. 2. Recovered response curve for the CCD for each of the three color channels.

3. Contrast Enhancement and Image Remapping

Processing an image globally based on its recovered irradiance values leads to computational complex

ities. It also generally results in loss of contrast, thereby resulting in the loss of image details. We present an irradiance profile mapping algorithm that combines local filtering and global processing based on center-surround Retinex filters to overcome dot detection problems commonly encountered such as signal blooming due to the proximity of two fluorescence dots of very different intensity. For example, if a dim dot is close to a bright dot, the bright pixels can influence the processing of the dim area and can cause a black region around the bright area. On the other hand, local filtering alone tends to make pure black and pure white low contrast areas turn gray, which can lead to problems of separatibility of two fluorescent dots.

In our approach, each of the image channels are processed independently. Luminance of each channel is treated by the Retinex-based adaptive filter method. The luminance tp is assumed to be linear with respect to scene radiance and is first globally compressed. Then, the Retinex-based adaptive filter method is applied in the log-domain to the globally corrected luminance ip' to result in the treated luminance ipnew In general, the method consists of two parts: a preliminary global irradiance mapping followed by Retinex-based local processing. The global irradiance mapping that is applied to the linear image performs a first compression of the dynamic range. The global irradiance mapping function used is a power function. The curvature of the function that determines the adaptation state depends on the mean luminance of the total field of view.19 Consequently, we compute the exponent of the power function from the average luminance of the image. Let ip be the luminance image, whose maximum value is 1. The non-linear luminance I/J' is given by

if = ip^ (4)

where the value of - is an affine function of the av-7

erage luminance of the image, tpai defined as:

where N is the number of pixels in the luminance image i\) and p is the pixel-value in ip. The coefficient of the affine function were defined experimentally based on assigning - = 1 since a high or average key image

Shishir Shah and Fatima Merchant 223

is not globally compressed. As the average luminance decreases, the exponent decreases, thereby increasing the sensitivity for dark areas.

After global processing, local adaptation is performed using a surround-based Retinex method. Traditionally, surround-based Retinex methods20-22

compute a new value for each pixel by taking the difference between the log-encoded treated pixel and the log-encoded value of a mask. The mask represents a weighted average of the treated pixel's surrounding area. A drawback of the surround-based methods is that small filters tend to make pure black or pure white low contrast areas turn gray. This is due to local normalization. To overcome this problem, we use a weighted mask function m(x,y) that is sensitive to the conservation of local details. Thus, each pixel in the new luminance image is computed as:

^new{x,y) = log(?//(:r,y))-p(x,2/)-log(ra(x,y)) (6)

The (3 factor is based on a sigmoid function and maps white to white and black to black. For a pixel of high intensity, the mask is weighted by a value close to 0. Since the mask is subtracted from the log-encoded luminance, it effectively keeps the pixel bright. Similarly, a pixel of low intensity is weighted by a value close to 1, which has the effect of maintaining black. This function lets the middle gray values change without constraint while restricting the black to remain black and the white to remain white, thereby allowing maximum spread of the weak and very bright intensities. The function can be written as:

MX' ») = l ~ 1 + e-a.(^(x,v)-0.5) (?)

where a modulates the slope of the sigmoid. To enhance the separability and detectability of

fluorescent dots in the image, we further adapt the filter to follow image edges. In this way, a bright area has less influence on the treatment of a neighboring dim area. The mask is thus computed specifically for each pixel where m{x,y) is given by:

360 r m a x _ r2

Y^ Yl ^'(x + rcos(6),y + rsm(0)) -e °o7- (8) 0=0 r=0

where 6 is the angle in the radial direction, r is the distance to the central pixel and GQ^ is set to be &high or &iow based on the presence or absence of a high contrast edge, respectively. The numerical values of (Thigh and G\OW are chosen to be fractions of

the image size. In our implementation, we use the Canny edge detector to detect high contrast edges.23

The thresholds for strong and weak edges are fixed values chosen experimentally and kept the same for all images.

4. Results

The developed approach has been tested on a large dataset of images where DAPI was used as the nucleus counterstain, FITC, and Texas Red used to mark the desired chromosomes. In collecting the dataset, each image was captured based on a preset exposure and the HDR image generated from the pre-computed radiance profile. Figure 3 shows three images acquired based on low, medium, and high exposure settings. The corresponding remapped HDR image is shown in figure 4. All images are of lymphocytes from cultured blood from normal specimens.

(a) (b) (c)

Fig. 3. Image acquired at three exposure settings (a) low; (b) mid; and (c) high

Fig. 4. Image that has increased dynamic range based on the irradiance profile of the camera.

To analyze the dataset for dot counting, an algorithm similar to one proposed by Netten et al.10

was implemented. A small set of 50 images was used to train the dot detection algorithms and 2,000 independent images used to evaluate the performance. The dot count in all 2,000 images was performed in

224 I-FISH: Increasing Detection Efficiency for Fluorescent Dot Counting in Cell Nuclei

Fig. 5. Example of dot counting efficiency based on low dynamic range image (a & b) vs. high dynamic range image (c & d ) .

are not detected or segmented and missed in the dot count, and dust or debris is identified and counted as false dots. Table 2 gives the percentage error for each type, as obtained from 3,000 nuclei that were detected in both of the automated methods. Figures 5 and 6 present two examples of the result of automated image analysis and dot detection based on the low dynamic range image and the remapped HDR image as input.

Table 1. Summarization of dot detection efficiency across manual, automated analysis of low dynamic range images, and high dynamic range images.

# of Images # of Detected Cells

Dot 1 Detected Dot 2 Detected

Truth 2000 6,482 9,076 7,778

FISH 2000

4,731(73%) 6,534(72%) 5,211(67%)

FISH-IDE 2000

6,125(94.5%) 8,440(93%) 7,544(97%)

Fig. 6. Example of dot counting efficiency based on low dynamic range image (a & b) vs. high dynamic range image (c & d ) .

three ways: manually, applying the automated algorithm to low dynamic range image, and applying the automated algorithm to the remapped high dynamic range image. Manual results are used as "ground truth". Result of the dot count is shown in table 1. To gain a better understanding of the performance increase due to the proposed approach, several types of errors that occur are also evaluated. Dot counts can be erroneous when one of the following errors is made: a single dot is split and counted as two or more, two or more overlapping dots are not segmented properly and get counted as one dot, dots

Table 2. Summarization of errors seen in dot detection using automated system to analyze low dynamic range image and the remapped high dynamic range image.

# of Nuclei Split Dots

Overlapping Dots Missed Dots False Dots

Automated FISH 3000 0.8% 3.2% 0.9% 3.4%

Automated FISH-IDE 3000 0.2% 0.1% 0.1% 0.4%

5. Conclusion

In this paper we have presented a methodology to increase the efficiency of detecting fluorescent signals in FISH images by recovering the radiance map of the CCD camera. We further analyzed the distribution of each of the fluorescent signals and remapped the HDR image values to generate a high contrast image with compressed dynamic range. The remapping was based on a criterion that allowed for maximum detectability of the signals. This resulted in images where saturated values were attenuated while the weak signals were amplified. We implemented a simple dot counting image analysis system and tested the detection efficiency based on both the original images and the remapped HDR images. Results obtained on a dataset of 2000 images clearly show that dot counting specificity and sensitivity can be improved by recovering the radiance curve of the CCD

Shishir Shah and Fatima Merchant 225

and generating a high contrast remapped HDR im

age before using any algorithm for dot counting. The

increased efficiency of detecting dots in FISH image

can have critical implications in a range of clinical

and research applications from preimplantation ge

netic diagnosis to translocation analysis.

R e f e r e n c e s

1. D. Pinkel, T. Straume and J. Gray, Proc. National Academy of Science 83, 2934 (1986).

2. A. C. Carothers, Cytometry 16, 298 (1994). 3. K. R. Castleman and B. White, Bioimaging 3, 88

(1995). 4. R. E. Kibbelaar, F. Kok, E. J. Dreef, J. K. Kleiverda,

C. J. Cornelisse and A. K. Raap, Cytometry 14, 716 (1993).

5. D. K. Griffin, L. J. Wilton, A. H. Handyside, R. M. Winston and J. D. Delhanty, Human Genetics 89, 18( April 1992).

6. S. Munne, J. Grifo, J. Cohen and H. U. Weier, American Journal of Human Genetics 55, 150(July 1994).

7. J. C. Harper, E. Coonen, F. C. S. Ramaekers, D. A. J. Delhanty, A. H. Handyside, R. M. L. Winston and A. H. N. Hopman, Human Reproduction 9, 721 (1994),

8. S. Munne, C. Mdrquez, C. Magli, P. Morton and L. Morrison, Molecular Human Reproduction 4, 863 (1998).

9. C. M. Conn, J. C. Harper, R. M. Winston and J. D. Delhanty, Human Genetics 102, 117(January 1998).

10. H. Netten, I. T. Young, L. J. van Vliet, H. J. Tanke, H. Vroljik and W. C. Sloos, Cytometry 28, p. 110 (1997).

11. J. C. M. et al., Methods for ccd camera characterization, in SPIE Proceedings: Image Acquisition and Scientific Imaging Systems, (San Jose, CA, 1994).

12. P. N. Rao, R. Hayworth, K. Cox, F. Grass and M. J. Pettenati, Prenatal Diagnostics 13, 233 (1993).

13. K. Klinger, G. Landes and D. S. D. et al., Am J Human Genetics 51 , 55 (1992).

14. K. W. Klinger, Ann NY Acad Sci 731, 48 (1994). 15. M. T. Little, S. Langlois, R. D. Wilson and P. M.

Lansdorp, Blood 89, 2347 (1997). 16. D. van Opstal, J. O. van Hemel and B. H. E. et al.,

Prenatal Diagnostics 15, 705 (1995). 17. J. Yan, E. Guilbault, J. Masse, M. Bronsard,

P. DeGrandpre, J.-C. Forest and R. Drouin, Clinical Genetics 58, 309 (2000).

18. P. Debevec and J. Malik, Recovering high dynamic range radiance maps from photographs, in Proceedings of the 24th annual conference on Computer graphics and interactive techniques., (ACM Press/Addison-Wesley Publishing Co., 1997).

19. D. Alleysson and S. Siisstrunk, On adaptive non-linearity for color discrimination and chromatic adaptation, in IS&T First European Conference on Color in Graphics, Image, and Vision, (Poitiers, France, 2002).

20. L. Meylan and S. Siisstrunk, Bio-inspired image enhancement for natural color images, in IS&T/SPIE Electronic Imaging 2004: The Human Vision and Electronic Imaging IX, (San Jose, CA, 2004).

21. E. H. Land, National Academy of Sciences of the United States of America 83, 3078 (1986).

22. Z. Rahman, D. D. Jobson and G. A. Woodell, Journal of Electronic Imaging 13, 100(January 2004).

23. J. Canny, IEEE Transactions on Pattern Analysis and Machine Intelligence 8, 679(November 1986).

226

Intuitionistic Fuzzy C Means Clustering in Medical Image Segmentation

T. Chaira

E-mail: [email protected]

A. K. Ray

Department of ECE IIT Kharagpur

Midnapore, WB-721302, INDIA E-mail: [email protected]

O. Salvetti

ISTI-CNR, Via G. Moruzzi 56124 Pisajtaly

E-mail: Ovidio. Salvetti@isti. cnr. it

This paper presents a novel intuitionistic fuzzy C means clustering method using intuitionistic fuzzy set theory. This clustering method has also been used for identification of mammogram cysts. In this method, the hesitation degree has been taken into account along with the membership degree, where the cluster centers may converge to a desirable location than the cluster centers obtained by fuzzy C means algorithm. Experimental results on mammogram images show the effectiveness of the proposed method in contrast to existing fuzzy C means algorithms.

Keywords: intuitionistic fuzzy set, fuzzy clustering, image segmentation, hesitation degree.

1. Introduction

Today most of the biomedical images arising from diverse imaging modalities such as, x-ray, magnetic resonance imaging (MRI), computed tomography (CT) and so on are interpreted visually and qualitatively by radiologists. A major area of research in automated diagnosis of such images involves the detection of the region of interest that includes segmentation of the objects embedded in the uncertainty. The uncertainty may include noise, vagueness in class definitions, imprecise gray levels. The membership degree for a given sample image depends on the distance between the image patterns to a cluster center and denotes the degree to which the pattern belongs to a particular cluster. However, since membership degrees are imprecise and it varies on person's choice, there is some kind of hesitation degree (which arises from the lack of precise knowledge).

Attanassov in 1986 introduced the concept of intuitionistic fuzzy set where the non-membership degree is not equal to 1- membership as in ordinary fuzzy set. This is due to some hesitation present in assigning the membership degree. The first fuzzy method to segment the regions of an image by clustering was introduced by Bezdek.1 The algorithm requires a priori definition of number of

classes and its result depends on the number. Relating to the past work, there has been many methods using fuzzy and crisp theory. Ferahta et al.2 had given the idea of optimal classes where the number of classed are not predefined. They used a second criterion function that relates to Shannon's entropy. Rhee and Hwang3 introduced a idea of using type 2 fuzzy set in constructing the clustering algorithm. According to them, the cluster centers using type 2 fuzzy algorithm converge more to desirable locations than in type 1 fuzzy algorithm. About the intuitionistic fuzzy set theory in image processing, as it is just beginning to develop, there has been no work relating to clustering in the literature. The objective of this paper is to develop a novel method, an intuitionistic fuzzy c means algorithm, to segment medical images. In the construction of our algorithm, the following steps required are as follows:

1 An intuitionistic fuzzy set has been created using Sugeno's intuitionistic fuzzy generator.

2 modifying the objective criterion. 3 updating the cluster center using intuitionistic

fuzzy set.

The paper is organized as follows: In section 2 the Preliminaries about the intuitionistic fuzzy set is presented while Section 3 tells about the

T. Chaira, A. K. Ray and O. Salvetti 227

construction of intuitionistic fuzzy set using Sugeno's generator. Section 4 provides the intuitionistic fuzzy c means algorithm and Section 5 outlines the introduction of a new criterion functioin. Section 6 presents the results and discussion and finally the conclusion has been drawn in section 7.

2. Preliminaries

Attanassov's4 intuitionistic fuzzy set emerges from the simultaneous consideration of membership values /x and non-membership values u of the elements of a set. An intuitionistic fuzzy set in X is given by

A = {X,HA(X),I>A(X)\,X € X)}

where HA '• X —• [0,1], VA '• X —> [0,1] with the condition 0 < HA(X) + VA(X) < 1 where HA{X) and VA{X) are the membership and non-membership degrees of an element x to the set A. When VA(X) = 1 — HA{X) for every x in set A, then the set A becomes a fuzzy set. For all intuitionistic fuzzy sets, Attanassov also indicated an intuitionistic degree, TTA(X), which arises due to lack of knowledge about the membership degree, of each element x in A and is given by: TVA(X) = 1 - HA{X) - VA{X).

Obviously, 0 < XA{X) < 1.

Due to the hesitation degree, the membership values lie in the interval [fiA(Xi), fiA(Xi) + 1VA(Xi)] .

3. Construction of intuitionistic fuzzy set

In order to construct Attanassov's intuitionistic fuzzy set (IFS) from fuzzy set, intuitionistic fuzzy generators are used. From the definition of intuitionistic fuzzy generators by Bustince et al.5 :

A continuous, decreasing, and increasing function 4> : [0,1] will be called continuous, decreasing, and increasing intuitionistic fuzzy generator if <j)(x) <{l-x) for all are [0,1].

4>(0) < 1 and 0(1) = 0.

Bustince et al. obtained intuitionistic fuzzy generator from Sugeno's generating function. Sugeno's generating function is given as:

fx(x) = \ln{l + X(x)) with limx-^ofx(x) = x

For A > 0, the Sugeno's intuitionistic fuzzy generator

Mx) = T^t with A ^ °

Then0 A ( l ) = O,0A(O) = l This Sugeno's intuitionistic fuzzy generator has

been used to create an intuitionistic fuzzy set.

To construct an intuitionistic fuzzy set from a fuzzy set, intuitionistic fuzzy generator has been used here. For determining the non-membership values, Sugeno's intuitionistic fuzzy generator </> has been used in addition to the membership function /z(x) of the elements a; of a fuzzy set. In this condition with the help of Sugeno's intuitionistic fuzzy generator, the IFS becomes:

A[FS = { i l W ( , ) , / " ^ f f j s 6 X} (1) 1 + \.HA(X)

with A > 0

The hesitation degree from (1):

nA{x) = l-nA(x)- / " ^ ^ (2) 1 + \.HA{X)

For every IFS, 0 < (J,A(X) + UA{X) < 1 holds. Since the denominator 1 -I- \.HA{X), is greater

than 1, so 1 + ^ ' ( ^ is less than 1 — ^,4(^)1 for all xeX.

So, the verification that AIFS belongs to IFS is as follows:

HA(X) + VA(X) =

M*) + i+x.^t) ^ ^ ( ^ + l ~ »AW < 1 This Sugeno's intuitionistic fuzzy generator has

been used to create an intuitionistic fuzzy image for

image clustering.

4. Intuitionistic Fuzzy C-means Clustering

Conventional fuzzy C means algorithm cluster the feature vectors by searching for local minima of the

228 Intuitionistic Fuzzy C Means Clustering in Medical Image Segmentation

following objective function:

Jm(U,v;X) = ^=iEnk=lU^Dik

where Dik is some similarity measure between Vi (cluster center) and Xk (points in the pattern) or the attributer vectors and the cluster centers of each region, c is the number of clusters, n is the number of data points. Minimization of Jm is based on suitable selection of U and V using an iterative process through the following equation:

Uik = * _1_,

v i k = ^ lp Vt,fc. 2-,k=i uik

In order to incorporate intuitionistic property in conventional fuzzy clustering algorithm, the cluster centers are to be updated. So initially the hesitation degree has been calculated using Eq. (2). Due to the hesitation degree, the membership value has been updated as: uik = Uik + 7rifc>

where Uik denotes the conventional fuzzy membership.

Now using the intuitionistic membership matrix, the cluster centers, as in the conventional fuzzy clustering, may be given as:

v.{ _ ELi "'<*"** (3)

Using Eq. (3) the cluster center has been updated and also the membership matrix is updated. At every iteration the cluster center and the membership matrix are updated. The algorithm stops when the updated membership matrix and the previous matrix i.e.

maxi,k\U*rw-U;rV\<* e is a user defined value and has been selected as e = 0.03 .

5. Introduction of new criterion function

In the proposed method, the criterian function contains two terms-the first term is the general term as in conventional clustering algorithm which is an intra class distance and the second term aims in maximizing the good points in the class. The goal is to minimize the entropy of the histogram of an image.

The second function is:

So the final criterion function that has been minimized becomes:

' = £,£,*&<& +jrt,**1* i = l fc=l i = l

Pi is the probability of the ith class as:

Pi = IiT,k=iuik, ke [1,N].

6. Results and Discussion

Tests have been performed on several medical mam-mographic images. The images and the image descriptions have been obtained from http://www. imagin is . com/breasthealth/ultrasoundJ.mages2. asp. The value of A = 2 in Eq. (2) has been chosen in experimentation. Some of the results are shown below. The features of each image consists of are energy, standard deviation and entropy. The proposed method has been compared with conventional fuzzy c means and Type 2 fuzzy clustering algorithms. The image in Fig. (1) shows a breast cyst with tiny accumulations of fluid that are the most common cause of benign (non-cancerous) breast lumps in women. Complex cysts can be filled with debris and may sometimes require aspiration to confirm that they are indeed benign cysts. Both single and multiple cysts are very common. In the figure, two adjacent breast masses: one a debris filled cyst, the other a simple cyst is shown. From the results it is observed that with intuitionistic fuzzy clustering algorithm, the debris in the cysts are clearly detected than with the conventional fuzzy c means clustering where more solid particles areas are seen. Type 2 fuzzy clustering is somewhat similar to the proposed method but the solid particles are not shown clearly and also the outer boundary is not continuous.

Fig. (2) shows a breast mass, which contains a group of breast cells that are clustered together more densely than the surrounding breast tissue. Masses can be palpable (able to be felt) or nonpalpable (unable to be felt) and may be benign or malignant. The image contains large cyst with layered debris and a solid component. From the result of the proposed method using intuitionistic fuzzy C means clustering algorithm, the solid components or the debris inside

T. Chaira, A. K. Ray and O. Salvetti 229

•**«• ,\v|ii»

(a) (b) (c)

Fig. 1. (i) image, (ii)Conventional Fuzzy Cluster, iii) type 2 cluster, (iv) intuitionistic fuzzy cluster.

(a) (b) (c) (d)

Fig. 2. (i) image, (ii) Conventional Fuzzy Cluster, iii) type 2 cluster, (iv) intuitionistic fuzzy cluster.

M

J ** **feJ t • • *

(c)

Fig. 3. (i) image, (ii) Conventional Fuzzy Cluster, iii) type 2 cluster, (iv) intuitionistic fuzzy cluster.

the cyst are properly detected whereas with conventional fuzzy c means clustering and type 2 fuzzy clustering, the solid particles in the cyst are shown with more prominence.

The image in Fig. (3) shows a solid irregular breast mass with calcification (calcium deposits). Ultrasound does not reliably show image calcifications properly in all the cases. From the result it is observed that the result with Intuitionistic fuzzy C means, the calcification area is clearly detected

whereas with conventional fuzzy c means clustering, the solid particles in the cyst are shown in more areas. In type 2 fuzzy cluster, the calcification areas are somewhat better than conventional fuzzy c means but not properly detected specially at the upper right and the lower right portion of the image. The area of micro-calcification is well delineated using the proposed approach based on intuitionistic fuzzy clustering. The spreading of calcification is also properly visible with more prominence in the proposed method.

230 Intuitionistic Fuzzy C Means Clustering in Medical Image Segmentation

7. C o n c l u s i o n

This paper provides a new approach to fuzzy clus

tering using intuitionistic fuzzy set theory. The

method takes into account the hesitation in assign

ing the membership degree. The algorithm has been

tested on mammogram images for detection of micro-

calcification and the results have been observed to be

bet ter compared to Type 2 fuzzy and conventional

fuzzy C means algorithm.

R e f e r e n c e s

1. J. C. Bezdek, L. O. Hall, and L. P. Clark, Review of MR setgmentation technique in pattern recognition,

Med Phy 10:20(1993) pp. 33-48. 2. N. Ferahta et al., New fuzzy clustering algorithm ap

plied to RMN segmentation, IEEE Trans, on engi-neerig, computing, and technology V12,(2006) pp. 9-13.

3. F. C. H. Rhee and C. Hwang, A Type-2 fuzzy c means clustering algorithm, Proc. in Joint 9th IFSA World Congress and 20th NAFIPS International Conference 4 (2001) pp. 1926-1929.

4. K. T. Atanassov, Intuitionistic fuzzy sets, Theory and Applications, Series in Fuzziness and Soft Computing (Phisica-Verlag, 1999).

5. H. Bustince et al, Intuitionistic fuzzy generators Application to intuitionistic fuzzy complementation, Fuzzy sets and systems 114(2000) pp.485-504.

231

Remote Sensing Image Classification: A Wavelet-Neuro-Fuzzy Approach

Saroj K. Meher, B. Uma Shankar and Ashish Ghosh

Machine Intelligence Unit Indian Statistical Institute

203 B. T. Road, Kolkata 700108, INDIA E-mail: {saroj.t, uma, ash}@isical.ac.in

The present article proposes a wavelet-neuro-fuzzy (WNF) system for classification of land covers of remote sensing images. This classifier incorporates a new architecture for neuro-fuzzy (NF) system that expand the input space of conventional NF (CNF) systems. The performance of this new NF classifier is compared with the CNF and the conventional multi-layer perceptron (MLP) with original multispectral features of remote sensing images. Experimental study demonstrated the superiority of this NF classifier. Incorporation of wavelet features into this classifier improved its performance. Particularly, with biorthogonal3.3 wavelet the proposed NF classifier outperformed all others. Results are evaluated qualitatively and quantitatively.

Keywords: Remote sensing; pattern classification; fuzzy set; neural networks; neuro-fuzzy; wavelet transform

1. Introduction

Classification of land covers in remote sensing images is a complex task because of low illumination, low spatial resolution of sensors, rapid changes in environmental conditions and overlapping nature of regions like vegetation, soil, water, concrete structures.1 Moreover, the gray value assigned to a pixel is an average reflectance of different types of land covers present in the corresponding pixel area. Therefore, a pixel may represent more than one class with varying degree of belonging. Thus assigning a unique class label to a pixel with certainty is one of the major problems. Conventional methods cannot deal with this imprecise representation of geological information. Fuzzy set theory introduced by Zadeh2 provides a useful technique to allow a pixel to be a member of more than one category or class with graded membership. The significance of fuzzy set theory in the realm of pattern classification3 is justified in different areas including representation of graded membership values of pixels in remote sensing imagery for various land cover classes.3'4 Neural networks (NNs)5

on the other hand, are aimed to emulate the biological nervous system to make it suitable for collective decision making. Therefore, an integration of neural and fuzzy systems known as neuro-fuzzy (NF) technique enables one to design intelligent decision making systems. Many research efforts have already been made for the design of NF systems.6""9

Further, remotely sensed images comprise information over a large range of variation of frequencies that changes over different regions. They have

both spectral features with correlated bands and spatial features correlated in the same band. An efficient utilization of these spectral and spatial (contextual) information can improve the classification performance compared to the non-contextual information based methods. Research efforts have been made to take advantages of neighboring pixels (tex-tural) information1 computed from the gray level co-occurrence matrices10 for classification purpose. These methods are computationally expensive as they require an estimation of autocorrelation parameters and transition probabilities. Also, the texture elements are not easy to quantify. Later on Gaussian Markov random fields (GMRF) and Gibbs random fields were proposed to characterize textures.1 The above-mentioned conventional statistical approaches to texture analysis are restricted to the analysis of spatial interactions over relatively small neighborhoods in a single scale. Recently wavelet transform (WT)11 has received much attention as a promising multi-scale tool for texture analysis of remote sensing images, both in spatial and frequency domains.12

In the present study we have explored the advantages of both wavelet features (WF) and a new NF architecture for classification of land covers of remote sensing images by proposing a wavelet-neuro-fuzzy classifier (WNF). The number of input nodes of NN of this NF classifier is equal to the product of the number of classes and the number of features of input patterns. Number of nodes in the output layer remains the same as the number of classes. The proposed NF classifier is different from other exist-

232 Remote Sensing Image Classification: A Wavelet-N euro-Fuzzy Approach

ing NF classifiers.7"9'13-15 In most of the existing NF methods each feature of the input pattern vector is encoded with three linguistic variables low, medium and high. Thus the number of input nodes of the network is equal to 3 x (number of input features). The output nodes of the networks are provided with fuzzy class labels (computed from the training data) for learning. On the contrary, in the proposed NF method each feature of the input pattern is encoded with a sequence of membership values where each element of the sequence represents the degree of belonging of that feature to the corresponding class. The number of input nodes of the network becomes equal to the product of the number of input features and the number of classes present in the data set. The output nodes are provided with the existing class labels of the input patterns during training.

The performance of the proposed NF classifier is compared with that of a neural network (i.e., MLP)5 and conventional NF (CNF)16 (where the input nodes of the NN is equal to the number of features of the input patterns) classifier. Use of WF to these classifiers provided better accuracy compared to original features. Among various wavelets used in the present study, biorthogonal3.3 (Bior3.3) wavelet is found to be more suitable for the present problem.

2. W T based feature extraction

To deal with the non-stationary signals (e.g., remote sensing images) many research works are carried out. Efforts are made to overcome the disadvantages of Fourier transform17 that assumes the signal to be stationary. Short time Fourier transform (STFT) and wavelet transform (WT)11 '17 '18 are examples of such transforms. However, selection of window function is a major problem in STFT which is eliminated in WT. WT tries to identify both the scale and space information of the event simultaneously which make it more useful for analysis of remote sensing images.

WT is used for feature extraction from the multi-spectral input images. In this process, each spectral (band) image is decomposed using 2-D WT which provides four sub-images (wavelet coefficients) from an input image: one in the low frequency range (LL) and three in the high frequency range (LH, HL and HH). After inverse WT (called reconstruction) of these sub-images we get wavelet features (WF) of the original image. The WF thus obtained are cascaded as shown in Fig. 1 and are used for classification.

Classifier -*\ Class map

Fig. 1. Classification procedure

3. Classification techniques

Here we describe the three classifiers used for the present investigation.

iVeuraZ classifier: Both the original and WF (generated by WT) are used as input to an MLP. The MLP used in the present experiment has three layers; namely, input, one hidden and output. The number of input-layer nodes is equal to the number of features of the data patterns and the number of output-layer nodes is equal to the number of classes. MLP uses back-propagation (BP) learning algorithm5 for weight updating. The BP algorithm reduces the sum of squared error called cost function (CF), between the actual and desired output of output-layer neurons in a gradient descent manner.

Neuro-fuzzy classifier: In the present study a conventional NF technique similar to the architecture used by Lee et o/.16 is considered. The NF classifier is based on MLP described in the previous paragraph with alterations in operation to accommodate fuzzy data. The NF system can be summarized as a three step process. At first the data pattern is prepro-cessed using appropriate membership function (MF) to obtain the corresponding degree of belonging for each feature of a pattern to a particular class. These membership values of the features are then supplied as input to the NN in the next step which produces fuzzy output. Finally, output of the NN are defuzzi-fied using a MAX (maximum) operation to assign a class label to the input pattern.

Proposed neuro-fuzzy classifier: The new architecture of NF classification system extract feature-wise fuzzy information of all classes. This

Saroj K. Meher, B. Uma Shankar and Ashish Ghosh 233

classifier (Fig. 2) works in three steps and is described briefly in the following paragraphs.

the highest output node value. In case of ties, it is resolved arbitrarily.

Fuzzification Defuzzification

input pattern iona ern

1

B

S

F

F ^ I )

• B

! F

N E U R A L

N E T W 0 R K

CxD dimensional feature space

Fig. 2. Neuro-fuzzy classification model

Fuzzification: The first step of the system performs the fuzzification of input patterns using a n-type MF2 that generates the feature-wise degree of belonging of the input pattern to different classes. The membership fd,c(

xd) of feature value Xd of a pattern x thus generated expresses the degree of belonging of dth features to cth class; where x = [x\,X2,---Xd, ...xo]T, with d = 1,2, ...D and c = 1,2,...C.

Thus for a multi-featured pattern x, the membership matrix after fuzzification process can be expressed as:

F(x)

/ i , i ( * 0 h,\(x2)

ha(xi) haix2)

fuc(xi) /2,c(z2)

JD,I(XD) fofiixD) ••• /D,C(XD).

This fuzzified pattern matrix is used as input to a NN in the second step. The NN performs the classification task.

Defuzzification operation: The output of the NN provide the degree of belonging of a pattern to different classes which can further be used for higher level image analysis. If one needs to assign a hard class label instead of fuzzy label, a MAX operation is more appropriate for defuzzification. In this step the pattern is classified to class c corresponding to

»;

I . * - * * .

(b)

Fig. 3. Original (a) IRS-1A (band-4) and (b) SPOT (band-3) images

4. Experimental results and analysis

Although experiments were performed on a number of images, we present here the results of two remote sensing images. The images cover an area around the city of Calcutta having six major land cover classes namely, pure water (PW), turbid water (TW), concrete area (CA), habitation (HAB), vegetation (VEG) and open spaces (OS). These images (Fig. 3) are obtained from IRS4 (Indian remote sensing) and SPOT4 (Systeme Pour d'Observation de la Terre) satellites. The IRS image has a spatial resolution of 36.25m x 36.25m and has four spectral bands,

234 Remote Sensing Image Classification: A Wavelet-Neum-Fuzzy Approach

Fig. 4. Classified IRS-1A image using new NF with (a) original features, and (b) Bior3.3 wavelet features

whereas the SPOT image has a resolution of 20m x 20m with three spectral bands.

Selection of training samples are made according to the ground truth of the land cover regions. After training, the classifier is used for classifying the land covers of the whole image (i.e., unknown regions).

We have used wavelets from different groups (Daubechies (Db), biorthogonal (Bior), Coiflets, Symlets).18 However, results are given for four wavelets namely, Db3, Db6, BiorS.3 and Bior3.5, as their performances are found to be better than others. These wavelets are implemented with the mul-tiresolution scheme given by Mallat.11 During the decomposition of an image by WT, four sub-sampled versions of the original image are obtained. This decomposition can be extended to more than one level

that can provide more information. To have an objective evaluation, we computed the average entropy providing a measure of information of the image. We found that the entropy value is not changing significantly after two levels of decomposition.

Two performance measures namely, 0 index4 of homogeneity and Davies-Bouldin (DB) index19 of compactness and separability are used. 0 is defined4

as the ratio of the total variation and within-class variation. For a given image and given number of classes, the higher the homogeneity within the classes, the higher would he the 0 value. The DB index is based on the evaluation of some measure of dispersion within the classes and the distance between the prototypes of the classes. The smaller the DB value, the better is the partitioning.19

4*1. Classification of images

The six land covers of both the images are classified using the above mentioned three classifiers. F¥om the visualization point of view, it' is observed that the new NF classification method is performing better in classifying the land covers (i.e., segregating different regions) compared to the rest two. Hence we have shown only the classified images using this method with original spectral features in Figs. 4a and 5a. Further, the performance of all the classifiers are improved with WFs and we found that Bior3.3 wavelet is the best for the current problem. Thus we have shown the classified images using the proposed WNF (with Bior3.3 wavelet) classifier in Figs. 4b and 5b, where various land cover classes and known objects are found to be clearly visible. These objects are more or less visible in case of other classifiers and with different WFs. But the segregation of land covers and some known structures are better for the WNF classifier. This finding is supported by a quantitative evaluation.

In Table 1 we put the values of the validity measure for different methods. FVom the table we found that individually the new NF classification method is providing the maximum 0 and minimum DB values which support the superiority of the, method. The indices are improved further for all the classifiers with WF (Table 1). Bior3.3 is found to be a more suitable wavelet for land cover classification. With the use of WFs, the land cover classes became more clear and structures are found to be more prominent. Consid-

Table 1. p and DB value for different classifiers

Class i f ica t ion m e t h o d

Training patterns MLP

Conventional NF New NF

Db3 + MLP Db6 + MLP

Bior3.3 + MLP Bior3.5 + MLP

Db3 + CNF Db6 + CNF

Bior3.3 + CNF Bior3.5 + CNF Db3 + new NF

i Db6 + new NF I Bior3.3 4- new NF

Bior3.5 + new NF

I R S i m a g e

P 9.42 7.15 7.75 8.61 7.69 7.18 7.78 7.20 8.13 7.77 8.22 7.89 8.89 8.63 8.91 8.62

DB 0,56 0.93 0.81 0.70 0.90 0.92 0.89 0.92 0.78 0.80 0.77 0.80 0.68 0.69 0.67 0.69

S P O T i m a g e

P 9.33 7.05 7.69 8.55 7.50 7.11 7.58 7.10 7.98 7.72 8 . IS 7.73 8.77 8.57 8.82 8.59

DB 1.49 3.35 2.45 1.98 3.19 3.34 3.1§ 3.34 2.34 2.40 2.82 2.41 1.91 1.97 1.90 1.97

(b)

Fig. 5. Classified SPOT image using new NF with (a) original features, and (b) Bior3.3 wavelet features

Saroj K. Mehert B. Uma Shankar and Ashish Ghosh 235

ering all cases we can infer that the proposed WNF classifier is outperforming other classifiers.

It needs to be mentioned here that the computational complexities of the proposed NF method is more compared to the existing methods if the data set is having more than 3 classes. However this increase in cost is marginal in comparison with the improved classification accuracies.

5. Conclusion

We have proposed a wavelet-neuro-fuzzy (WNF) classifier for land cover classification of remote sensing images. This has been compared with conventional NF and MLP based classifiers and found to be superior. It is observed that all the classifiers are providing better results with wavelet features compared to the original feature based classification. Bior3.3 wavelet based features are found to be more suitable. Particularly, Bior3.3 with new NF classifier (WNF) outperformed all others. The performances of the classifiers were evaluated visually and quantitatively.

Acknowledgments

Authors would like to thank D8T, Gol and University of Trento, Italy, for sponsoring the collaborative project titled Advanced Techniques for Remote Sensing Image Processing.

References

1. B. Tso and P. M. Mather, Classification Methods for Remotely Sensed Data Taylor and Rrancis (2001).

2. L. A. Zadeh, Information Control 8, 338 (1965). 3. L. I. Kuncheva, Fuzzy Classifier Design Springer-

Verlag (2000). 4. S. K. Pal et a/., International Jounal of Remote Sens

ing 21, 2269 (2000). 5. S. Haykin, Neural Networks; A Comprehensive

Foundation 2nd edn. Prentice Hall (1998). 6. S. K. Pal and A. Ghosh, International Journal of

Systems Science 27, 1179 (1996). 7. S. K. Pal and S. Mitra, Neuro-Fuzzy Pattern Recog

nition: Methods in Soft Computing John Wiley & Sons, USA (1999).

8. S. Abe. Pattern Classification : Neuro-Fuzzy Methods and Their Comparison Springer Verlag (2001).

9. J. S. R. Jang et al., Neuro-Fuzzy and Soft Computing: A Computational Approach to Learning and Machine Intelligence Pearson Education (1996).

10. R. M. Haralick and K. S. Shanmugam, Remote Sensing of Environment 3, 3 (1974).

236 Remote Sensing Image Classification: A Wavelet-Neuro-Fuzzy Approach

11. S. Mallat, A Wavelet Tour of Signal Processing 2nd edn. Academic Press (1999).

12. J. Yu and M. Ekstrom, Pattern Recognition 36, 889 (2003).

13. V. Boskovitz and H. Guterman. IEEE Transactions on Fuzzy Systems 10, 247 (2002).

14. P. Gamba and F. Dellacqua. International Journal of Remote Sensing 24, 827 (2003).

15. A. Baraldi et al, IEEE Transactions on Geoscience and Remote Sensing, 39, 994 (2001).

16. S. G. Lee et al., Proc. IEEE International Conference on Fuzzy Systems 2, 1063 (1999).

17. G. Strang and T. Nguyen, Wavelets and Filter Banks Wellesley-Cambridge Press (1996).

18. I. Daubechies, Ten Lectures on Wavelets SIAM (1992).

19. D. L. Davies and D. W. Bouldin, IEEE Transactions on Pattern Analysis and Machine Intelligence 1, 224 (1979).

PARTH

Multimedia Object Retrieval

239

An Efficient Cluster Based Image Retrieval Scheme Using Localized Texture Pattern

Saumen Mandal and Sanjoy Kumar Saha

CSE Department Jadavpur University

Kolkata - 700 032, India E-mail: [email protected], [email protected]

Amit Kumar Das

CST Department Bengal Engineering and Science University

Howrah - 711 103, India E-mail: [email protected]

Bhabatosh Chanda

Electronics and Communication Science Unit Indian Statistical Institute Kolkata - 700 108, India

E-mail: [email protected]

We have designed and developed a CBIR system based on texture features, we have worked with a database consisting of various texture images. An image is divided into number of blocks. Based on its texture pattern, each block is replaced by a texture value and 'texture image' corresponding to the original image is obtained. A set of descriptors obtained from this 'texture image' describes the image. Finally, a simple but novel cluster based retrieval scheme is presented which is free from the drawbacks of fc - means algorithm. Experiment shows the effectiveness of the texture features and the clustering scheme.

Keywords: CBIR; clustering; texture co-occurrence matrix; texture pattern moment

1. Introduction

Texture is a feature that has been extensively explored by various research groups. Haralick et al.1

proposed the co-occurrence matrix representation of texture features and then meaningful statistics are extracted from the matrix. Smith and Chang2 proposed some statistics (e.g., mean and variance) from the Wavelet sub-bands. Number of researchers have described texture using different form of wavelet transformation.3'4 Fournier et al.5 have used Gabor Filters. Kaplan et al.6 used fractal dimension as a measure of texture property. Tamura et al.7 presented visually meaningful texture properties, like coarseness, contrast, directionality, line-likeness etc. Sharing a similar view Liu and Picard8 used periodicity, directionality and randomness as texture features. Thus, researchers have tried a wide variety of texture descriptors.

In case of retrieval, precision and response time are the two major issues. Corresponding to the query image submitted by the user, similar images are retrieved from the database. As the similarity measure

is a function of the features describing the image, capability of features has an impact on precision. Apart from that, the retrieval scheme also has an important role. In case of linear search, all the images in the database are compared. Thus, in terms of precision, performance may be good. But, for a large volume of database the response time will be prohibitive. As an optimization between precision and response time, several indexing schemes have come up.

Number of tree based indexing scheme9'10 are proposed by researchers. In these schemes, the distance space is divided into number of spatial domains. But, they suffer from the curses of dimensionality. In3'11 database is organised on the basis of the distance with respect to one or more key/pivot elements. In these schemes also efficiency falls as the dimensionality of the features increases. A k nearest neighbour graph (knn graph) based scheme has been proposed by Sebastian and Kimia12 where each node represents a database element and it is connected to its k nearest neighbour. But, maintaining the graph

240 An Efficient Cluster Based Image Retrieval Scheme Using Localized Texture Pattern

for a large database is not trivial. The basic purpose of indexing is to reduce the

search space. So that, the retrieval becomes fast. Clustering technique also can serve the same. It is a tool for exploring the underlying structure of a given data set. Based on some measure of similarity, a set of rule is specified to assign the data elements to a cluster domain.13-17 Thus, the elements in the database are distributed in number of cluster domains and the search operation is carried out with in a few clusters instead of the whole data set.

Distance based, density based and hierarchical clustering are the major approaches. .RT-means algorithm17'18 is one of the widely used distance based algorithm. Initial K cluster centres are provided and iteratively data points are assigned to the closest cluster centres. Centres are recalculated. Thus, it goes on till the convergence is met. if-medoids and CLARANS19 also follow similar partitioning scheme with some modification. In density based approach, for each point of a cluster the density of the data points in the neighbourhood has to exceed a threshold. DBSCAN,20 OPTICS21 etc. are examples of density based algorithms. But, setting of density threshold is difficult and the density based clustering is comparatively slow. In hierarchical clustering algorithms, the data points are decomposed in to several levels of partitions which are represented by dendo-grams. BIRCH,22 CURE23 etc. follow this approach. The prohibitive cost of dendogram creation is a disadvantage of this approach.

This paper is organized as follows. In section 2, we have described the formation of texture cooccurrence matrix and the computation of the texture features including the texture pattern moment. It is followed by the description of the proposed clustering scheme in section 3. Experimental result is presented in section 4 and finally, concluded in section 5.

2. Texture Feature

We perceive the texture as repetitive or quasi-repetitive patterns. However, the gray-level cooccurrence matrix1 captures only the co-occurrence of the intensity values, it can not reflect the repetitive nature of the patterns in true sense. In order to overcome this limitation, we present a texture cooccurrence matrix24 for describing the texture of the image.

2.1. Texture Co-occurrence Matrix

Usually a small patch of finite area of an image is required to feel or measure local texture value. The smallest region for such purpose could be a 2 x 2 block. So in order to compute the texture cooccurrence matrix, the intensity image is divided into blocks of size 2 x 2 . Then the gray level of the block is converted to binary by thresholding at the average intensity. The 2 x 2 binary pattern obtained this way provides an idea of local texture within the block. By arranging this pattern in raster order a binary string is formed and corresponding decimal equivalent is its texture valued). Thus we get 15 such texture value as block of all l's does not occur.

A problem of this approach is that a smooth intensity block and a coarse textured block may produce same binary pattern and hence, same texture value. To solve this problem, all such smooth blocks (i.e. intensity variance is less than a threshold) are assigned texture value 0.

Texture values of the blocks that we have obtained represent the texture pattern quite effectively. But, it does not reflect the contrast information of the blocks. In order to get rid of such problem, the texture value of a block is modified to incorporate the contrast information.Thus, Tj, the new texture value of a block is computed as follows:

Ti = U x 16 + Si

where, Si is the contrast in the 2 x 2 block. In our experiment,

Si = (gn - ffio)/16

where, gn is the average value of the pixels in the block marked as 1 and gw is the same for the pixels marked as 0. Thus, gn — g^ represents the contrast of the block. As the old value is multiplied and contrast is divided by 16, it is ensured that contrast information will not dominate over the contribution of texture pattern and blocks with different texture pattern will have different values.

Thus, we get the scaled image whose height and width are half of that of the original image and the pixel values are same as texture values. This new image may be considered as the image representing the texture of the original image and referred as 'texture image'. Once the 'texture image' is obtained,

S. Mandal, S. K. Saha, A. K. Das and B. Chanda 241

co-occurrence matrix corresponding to that is computed. Care is taken to make it independent of translation, rotation and size.

2.2. Computation of Features

The texture co-occurrence matrix provides the detailed description of the image texture, but handling of such multivalued feature is always difficult. To obtain more perceivable features, based on the matrix the statistical measures like energy, entropy, contrast in texture, homogeneity in texture, texture moments are computed as it is done in case of gray-level cooccurrence matrix.

We further propose texture pattern moment as a new texture descriptor. A pixel in 'texture image' represents the texture pattern of a 2 x 2 block of the original image. Thus, the shape descriptor corresponding to the 'texture image' reflects the organisation of the block level texture patterns. On the other hand, such organisation reflects the overall texture of the original image. Thus, the shape descriptor for the 'texture image' acts as the texture descriptor for the original image. This analysis has led us to propose texture pattern moment as a feature. As moment is the most widely used shape descriptors, we also compute moments of various order for the 'texture image' and refer them as the texture pattern moment. Multiscalar Texture Co-occurrence Matrix

Depending on the size of repetitive pattern, texture of an image can be classified as micro or macro texture. The presented feature is capable of handling the micro textures. In order to enhance its power to represent the macro texture we obtain multiscalar texture co-occurrence matrix.25

It is computed as follows. In first iteration, we obtain the texture co-occurrence matrix corresponding to the original gray-scale image and features are computed from it. In subsequent iteration, we consider only the pixels of alternate row and column in the image used for computation of feature in previous iteration. Texture co-occurence matrix and subsequently the features are computed corresponding to this new image.

3. Clustering Technique

In the simplest way, clustering can be described as the grouping together of similar data elements in a cluster. -ftT-means algorithm is the most widely used

distance based clustering scheme. But, it suffers from number of disadvantages. First of all, number of clusters required is to be specified by the user and initial cluster centres are also to be provided. It is a difficult task. On the other hand, the performance is very much sensitive to these parameters. Moreover, the algorithm is very much affected by the outliers. The major advantage lies in its simplicity. It is very easy to implement.

Proposed distance based clustering scheme draws motivation from the simplicity of X-means algorithm. On the other hand, it releases the burden of specifying the set of crucial parameters from the users. In our work, we consider a new term called split point instead of the conventional cluster centres or centroids. The split points play the similar role as cluster centres at the time of splitting a cluster. But, the way split points are chosen, they are not centrally located in the clusters.

In the proposed scheme, initially, all the elements are considered to be in a single cluster. The most distant points in a cluster are considered as split centres. Elements are assigned to the nearest split points. A cluster is splitted only if the variance of the distance of the elements from the split point in the cluster exceeds a predefined threshold (th). SPL, the split point list maintains the set of split points corresponding to the clusters. The process continues iteratively till splitting is required for none of the clusters. The algorithm is as follows:

1. Assume a l l elements belongs to a single c lu s t e r .

2. Find the most d i s t a n t elements ( s p l i t points) in the c l u s t e r .

3. Assign the elements to the nearest s p l i t po in t s .

4 . I n i t i a l i z e the SPL. 5. flag=0; 6. for each cluster {

compute v, the variance of the

distance of elements from split point,

if (v > th) {

flag-1.

find split points and update SPL.

}

>

7. if (flag==l) {

242 An Efficient Cluster Based Image Retrieval Scheme Using Localized Texture Pattern

distribute all the elements to the

nearest split points and goto step 5.

}

8. Stop.

The clustering algorithm is simple enough and easy to implement. It does not require any initial guess of split points or centroids. The scheme is applicable for various distance/similarity measure. The clusters may take the polygonal shape of wide variation depending on the distance measure. But, the scheme may be affected by outliers because the splitting operation is carried out with respect to the most distant points. To minimise this, a cluster merging algorithm may be taken up after obtaining all the clusters. If the cluster size falls below a limit then the elements are reassigned to the other nearest split points. In our experiment, limit of cluster size has been specified in terms a minimum number of elements. Thus, the effect of outliers has been marginalised, particularly in the context of the retrieval of top order similar images. Like, if-means algorithm, in this scheme also, in each iteration all the elements are distributed. Moreover, finding the most distant point in the cluster has the complexity of o(n2), where, n is number of elements in the cluster. Thus, the clustering process becomes slow. But, in the context of CBIR, the cluster formation can be done offline and furthermore, it is not a frequent operation.

Table 1. Comparison of precision (in %) using different texture features.

P(10)

P(20)

P(30)

Gray-level Co-occurrence

Matrix

56.81

44.54

38.18

Texture Co-occ. Matrix (without texture pattern

moment)

70.81

61.59

52.18

Texture Co-occ. Matrix (with

texture pattern moment)

74.18

64.86

55.78

In order to insert an element, it is first placed in to the nearest cluster. In case the cluster becomes a candidate for splitting then the cluster formation algorithm, from step 4 onward is to be followed. Deletion of an element may also lead to such splitting. It may lead to merging of cluster also. To find out the similar elements against a query element, nearest cluster is searched and according to the distance

measure, ordered output may be obtained. In order to avoid the boundary problem and to improve the precision of retrieval, instead of the nearest cluster, search may be continued to few neighbouring clusters also.

4. Experimental Result

We have carried out the experiment using the images of Brodatz texture database*. The database contains 112 images of size 640 x 640. Each image is divided into sub-images of size 64 x 64, 128 x 128 and 256 x 256. Taking the samples from these sub-images of different size, we have formed a set of 50 images corresponding to the original 112 images. Thus, the experimental database contains 5600 images.

In order to study the performance of the 'texture image' based features, we have carried out Euclidean distance based linear search considering each database image as the query image. Performance is compared with Gray-level co-occurrence matrix1

based system also. To form the multiscalar version, we have considered upto 3rd iteration for both the system. Along with energy, entropy, homogeneity and contrast, we have also considered 5 texture moment (upto 2nd order only). Thus, 27-dimensional feature vector is formed. The second and third column of table 1 reflect that precision of retrieval is better for the 'texture image' based features. Finally, texture pattern moments are also included. As the high order moments are not perceivable, we have considered 3 values (only first order moment and one of the second order moments (ran)) for both - texture moment and texture pattern moment. Thus, after third iteration we obtain 30-dimensional feature vector. The last column of table 1 shows that the performance further improves. Thus, the effectiveness of the proposed texture feature is established.

Selection of th determines the cluster size. A low value may be very conservative. On the other hand, a high value is too liberal to include lot of elements in a cluster. So, it is to be chosen judiciously. We have considered th = .01 in our experiment. The database has been partitioned in to 5 clusters. Number of elements in the clusters varies from 800 to 1500. To obtain the similar images against a query, linear search is carried out in the cluster with the nearest split

a www. ux.his. n o / ~ tranden/brodatz. html

S. Mandal, S. K. Saha, A. K. Das and B. Chanda 243

Table 2. Comparison of precision (in %) using different Search strategy-

P(10)

P(20)

P(30)

Linear Search

74.18

64.86

55.78

Proposed Cluster based Search

74.06

63.89

54.94

point. Table 2 shows that P{ri) (precision after the retrieval of n images) in case of cluster based retrieval is comparable with that of linear search. On the other hand, as the number of comparison is drastically reduced, retrieval becomes faster. In our experiment, using each database image as the query image, on an average an improvement of speed by a factor of 5 has been obtained. Thus, the proposed scheme is capable to provide faster response without compromising the accuracy at large.

5. Conclus ion

In this paper, we have presented the texture features based on ' texture image' formed by dividing the original gray-scale image in to 2 x 2 blocks and replacing the block by a texture value, we have also proposed texture pat tern moment as a texture descriptor. For retrieval a simple but effective clustering scheme is proposed. Experiment shows the usefulness of the proposed feature and the cluster based retrieval strategy. Further work may be carried out to find an alternate approach for split point selection. So that , the cluster generation algorithm becomes faster. Moreover, incorporation of clustered indexing (i.e. the use of index within the cluster) may lead to further speed gain.

R e f e r e n c e s

1. R. M. Haralick, K. Shanmugam and I. Dinstein, IEEE Trans, on SMC 3(11), 610 (1973).

2. J. R. Smith and S. F. Chang, Automated binary texture feature sets for image retrieval, in Proceedings of IEEE Intl. Conf. on ASSP, (USA, 1996).

3. A. P. Berman and L. G. Shapiro, Computer Vision and Image Understanding 75, 175 (1999).

4. B. Ko, J. Peng and H. Byun, Pattern Analysis and Applications 4, 174 (2001).

5. J. Fournier, M. Cord and S. Philipp-Foliguet, Pattern Analysis and Applications 4, 153 (2001).

6. L. M. Kaplan, SPIE 3312, 162 (1998). 7. H. Tamura, S. Mori and T. Yamawaki, IEEE Trans,

on SMC8(6), 460 (1978). 8. F. Liu and R. W. Picard, IEEE Trans, on PAMI

18(7), 722 (1996). 9. A. Guttman, R-trees: a dynamic index structure for

spatial searchigng, in Proc. of ACM SIGMOD conf. on management of data, 1984.

10. S. T. Leutenegger, M. A. Lopez and J. M. Edgington, A simple and efficient algorithm for r-tree packing, in Proc. of 13th International Conf. on Data Engineering, 1997.

11. J. Barros, J. French, W. Martin, P. Kelley and M. Cannon, SPIE 2670, 392 (1996).

12. T. B. Sebastian and B. B. Kimia, Metric-based shape retrieval in large databases, in Proceedings of International Conf. on Pattern Recognition, (Canada, 2002).

13. E. Ruspini, Information Control 15(1), 22 (1969). 14. R. Duda and H. Hart, Pattern Classification and

Scene Analysis (Wiley, New York, 1992). 15. J. Hartigan, Clustering Algorithms (John wiley, New

York, 1988). 16. J. C. Bezdek, Pattern recognition with Fuzzy Objec

tive Function Algorithms (Plenum Press, New York, 1981).

17. A. K. Jain and R. C. Dubes, Algorithms for clustering (Prentice Hall, New Jersey, 1988).

18. D. Pollard, Tha Annals of Statistics 9(1), 135 (1981).

19. R. T. Ng and J. Han, IEEE Trans, on Knowledge and Data Engineering 14(5), 10003 (1996).

20. M. Ester, H. P. Kriegel, J. Sander and X. Su, A density-based algorithm for discovering clusters in large spatial database with noise, in Proc. of Intl. Conf. on Knowledge Discovery and Data Mining, 1996.

21. M. Ankerst, M. Breunig, H. P. Kriegel and J. Sander, Optics: Ordering points to identify the clustering structure, in Proceedings of ACM Conf. of Special Interest Group on Management of Data, 1999.

22. R. Ramakrishnan, T. Zhang and M. Livny, Data Mining and Knowledge Discovery (1997).

23. S. Guha, R. Rastogi and K. Shim, Cure: An efficient clustering algorithm for large databases, in Proc. of ACM SIGMOD, 1998.

24. S. K. Saha, A. K. Das and B. Chanda, CBIR using perception based texture and colour measures, in Proceedings of Intl. Conf. on Pattern Recognition, (Cambridge, UK, 2004).

25. S. K. Saha, A. K. Das and B. Chanda, Image retrieval using multiscalar texture co-occurrence matrix, in Proceedings of the 6th Intl. Workshop on Pattern Recognition in Information Systems, (Paphos, Cyprus, 2006).

244

Feature Selection Based on Human Perception of Image Similarity for Content Based Image Retrieval

P. Narayana Rao, Chakravarthy Bhagvati, R. S. Bapi, Arun K. Pujari, B. L. Deekshatulu

Dept. of Computer and Information Sciences University of Hyderabad

Hyderabad 500046. E-mail: [email protected], {chakcs,bapics,akpcs,bides} @uohyd.ernet.in

In this paper, we present an approach that enables selection of appropriate low-level features based on human perception for measuring image similarity in a Content-based image retrieval (CBIR) system. Human perceptual information is captured by three psychophysical experiments designed to explore different aspects of similarity. Their outcomes are used to formulate fitness functions in a genetic algorithm that selects a subset from an initial collection of popular low-level image features such as colour, texture and structure. The reduced subset of features is, thus, correlated to human perception and represents an attempt to bridge the semantic gap. We quantitatively validate our approach by building a CBIR system using such features and evaluating the retrieval precision.

1. Introduction

Images are usually characterized by low-level descriptors such as color, texture and shape, which do not correspond to the high-level concepts that humans perceive. The difference between machine and human perception is known as semantic gap. The extent to which it is handled determines the system's reliability and efficiency. The semantic gap may be narrowed by using methods such as Relevance Feedback (RF),1

linguistic indexing of images,2 or modelling human visual perception.3

Modelling human perception of similarity can be by developing either a computational model for early human vision or a similarity function consistent with human perception. The former is related to properties of the human eye and the latter is about extracting useful relations from the similarity ratings. Though these methods cannot capture the subjectivity of the user, they can capture human judgments of visual similarity.

Several groups have been involved in research on incorporating human perception into CBIR systems. For example, Thomas et al4 used similarity metric based on a multiscale model of the human visual system. The multiscale model includes channels which account for perceptual phenomena such as color, contrast and orientation selectivity. Mojsilovic et al.5

used multi-dimensional scaling(MDS) to extract the perceptual dimensions and used hierarchical clustering to derive rules for similarity judgment. Guyader et al.6 trained Gabor filter parameters so that the filter categorization matches categorization by hu

mans. Celebi and Aslandogan3 used human similarity ratings to develop a weighted Manhattan distance measure for focused monochrome images.

We extend the work of Celebi and Aslandogan3 by designing psychophysical experiments to model similarity in CBIR context from a clustering and classification perspective. We also experimented with several databases containing natural and remote sensing colour images to assess the generality of our approach.

The paper is organized into six sections. Section 2 is an overview of our approach. Section 3 describes the psychophysical experiments and their outcomes. Section 4 deals with the use of a genetic algorithm for finding the optimal set of low-level features based on the psychophysical experiments. Results on a content based image retrieval system (CBIR) that validate our approach to capture human perception are given in Section 5, followed by conclusions in Section 6.

2. Approach

Our approach is succinctly illustrated in Figure 1. Human perception is modelled through psychophysical experiments in which human users assess three different aspects of similarity in a CBIR context. The first, identical to that of Celebi et a/.,3 is an estimation of similarity between pairs of images. Pairwise similarity between randomly selected pairs of training images is computed and labelled into four categories on a scale of 1 (extremely similar) to 4 (not similar). The results are coded in a similarity matrix. The second aspect, capturing the CBIR spirit,

P. Narayana Rao, Chakravarthy Bhagvati, R. S. Bapi, Arun K. Pujari and B. L. Deekshatulu 245

divides the set of training images into two classes — similar and dissimilar — with respect to a specific reference (query) image. The outcome is two clusters of images. The third, a classification approach, assigns class labels to the selected set of training images.

On the other hand, from the computer perspective, we start with a set of popular low-level colour, texture and structure features. Genetic algorithm (GA) is then used to select a subset of features. The GA objective or fitness functions are designed to reward conformity with the outcomes of the psychophysical experiments. Our hypothesis is that the reduced subset of features optimally capture the human notion of similarity expressed via the psychophysical experiments. The hypothesis is then validated by building a CBIR system and measuring the precision of the retrieved images.

HUMAN _> PERCEPTION

FF1, 2, 3: Fitness Functions corresponding to Expts. 1, 2, 3

F ig. 1. Overview of our approach

3. Psychophysical Experiments

In a traditional psychophysical experiment, a variable of physical stimulation is applied to an observer and then the corresponding variable of his phenomenal (or "sensory") experience is established. Here, the physical stimulation is supplied by a set of images that require an assessment of similarity from the user, while we determine a feature subset that correlates well with the assessed image similarities.

We designed three psychophysical experiments

exploring different aspects of human visual similarity. Expt 1: Pairwise similarities3

A subset of images is selected from a large database for the experiment. This subset contains images that cover all the major visual characteristics in the database. In this experiment, each image in the subset is compared with every other image in the subset and an observer judges the perceived similarity as 1 (extremely similar), 2 (considerably similar), 3 (partially similar) and 4 (not similar). The experiment is repeated multiple times with different observers and an aggregated response matrix (size M x M, where M is the number of selected images) is constructed. The (i,j) entry in the matrix, called the similarity matrix, gives the similarity score between images i and j . The similarity matrix is considered symmetric. Expt 2: Clustering

Pairwise comparison of images is hard to do for all but small subsets. In this experiment, a randomly selected single image, called a query image, is shown and the observer selects similar and dissimilar images from a subset of images. That is, the images chosen for the experiment are clustered into two classes and most closely resembles the function of a CBIR system. Expt 3: Classification

In this experiment, the observer classifies and labels the selected images. Sufficient number of images are selected so that general properties may be inferred. This experiment is an extension of Expt 2 and provides explicit information to the system about the higher level semantics in the form of a labelled subset of images.

4. Genetic algorithm formulation

We start with a set of 15 popular low-level features. Hue, saturation and intensity averages and variances form two 3-dimensional colour features. Texture features are obtained by a bank of 12 Ga-bor filters: of four orientations at three scales. The average and variance values of the responses give twelve 2-dimensional texture feature vectors. A 120-dimensional Colour Structure Descriptor (CSD), as defined by Manjunath et al.,7 is used as a structure descriptor.

We now formulate a genetic algorithm to select a subset, based on human perception as captured

246 Feature Selection Based on Human Perception of Image Similarity for Content Based Image Retrieval

by the psychophysical experiments, from the initial 15 features that together result in a 150-dimensional feature vector ( 2 x 3 + 1 2 x 2 + l x 120). Each individual {chromosome) is a 15-bit long string with each bit representing a single feature {gene). "1" indicates that the particular feature is included in the subset and "0" otherwise. For example, Bit-0 represents the colour feature and if it is 1, then the 3-dimensional average colour feature is included in the calculations. Bit-1 represents the colour variance feature, Bits-2 — 13 represent the 12 Gabor texture features and Bit-14 stands for the CSD feature.

Sixteen individuals (chromosomes) are generated randomly in the first generation. These are ranked according to their fitness as evaluated by the objective functions defined below and the top 50% survive into the next generation. Mutation operation with a probability of 0.1 and a 1-point crossover operation with a probability of 0.5 are used to add a new set of individuals replacing the weakest 50% of the population for the subsequent generation. The fitness of the entire population is the average fitness of the 16 chromosomes. The algorithm terminates when it either runs for 500 generations or the average fitness converges.

Three objective or fitness functions that relate to the three psychophysical experiments are defined. In the first method, a similarity matrix is constructed for each 15-bit string using only the features indicated by Is. The fitness function measures the Pearson coefficient of correlation between the human-generated similarity matrix from the psychophysical experiment 1 and the one computed mathematically from the individual bit-string. A higher correlation indicates greater fitness. In the second method corresponding to psychophysical experiment 2, we compute the intra-cluster distance for the images labelled as similar and the inter-cluster distance between the two clusters. The ratio of intra-cluster to inter-cluster distance is calculated for each individual in the population and the ones with the smallest ratio are assigned the highest fitness. In the third method, we construct a binary decision tree (DT) each label assigned by the human in the psychophysical experiment 3 and for each individual in the population using only the features indicated by Is in the individual. We compute an error bound using PAC learn-ability8 from the number of leaves and height of each DT. The maximum error-bound over all the classes

is assigned as the error bound for the individual. Fitness is inversely proportional to this error bound.

5. Results and discussion

We used several databases for our experiments. Two are remote sensing databases containing high-resolution aerial view (aerial) and low-resolution thumbnail images (browse) respectively. Other databases contain flowers, a variety of objects such as buses, aeroplanes, elephants and others. In addition, we also experimented with the Brodatz texture database containing 116 texture classes each containing 16 images.

Database

Aerial

Browse

Flower Brodatz

Pearson

4,7,9,10, 11,14 0,1,3,6, 7,8,10 0,1,6 0,1,3,13

PAC

3,7,8,11, 12,13 0,1,4,6, 9,10 2,3,4,8 1,2,5 9,14

PAC Unbiased 2,4,7,8, 9,11,14 0,1,10

1,2,3,8 7,8,9 10,14

Fig. 2. Reduced feature sets selected by GA

For the aerial database, a group of 15 people participated in the psychophysical experiment that generated the similarity matrix. Each observer was shown a selected set of 20 images each from the three databases and was asked to assess similarity between each pair of images. The experiment was repeated a few times for each individual. The results were aggregated and summarized in a 20 x 20 similarity matrix. For the second experiment, 40 randomly selected images were labelled similar and dissimilar to another randomly selected query image. In the third experiment, a set of 100 images was randomly selected from the databases representing 10 samples each from 10 types of images. In the third experiment, we computed a decision tree using either all the 100 instances {PAC method) or from a few instances of each class {PAC Unbiased method). In the latter case, we selected a number of samples from outside the class that is equal in number to the samples representing the class for which the DT is being constructed.

The feature subsets selected by the GA based

P. Narayana Rao, G'hakmvarthy Bhagvati, R. S. Bapi, Arun K. PujaH and B. L. Deekahatvlu 24?

Fig. 3. Precision graph for Aerial view database using the 4 methods

on Experiments 1 and 3 for the aerial database are* shown in Figure 2. In addition^ we also show the results from limited experimentation done on browse, flower and Brodatz databases for comparison. As the psychophysical experiment 2 (clustering) is not done for the other databases, its results are not shown in the table. The numbers refer to the genes in the 15-bit string used in the GA.

In the aerial database, the images are at high resolution (lm) and show large numbers of objects such as buildings and roads. Human observers determine similarity from the presence and arrangement of such structures and it may be noted that none of the GAs select the colour features (Bit-0 and Bit-l) in the reduced subsets. Similarity is judged entirely by texture and the distribution of colour across the image given by the CSD (Bit-14) feature, which corresponds to human perception. The feature subset selected by the GA based on psychophysical experiment 2 for aerial database is {1,11,12,13}, which also shows that texture features are more important.

In browse images, the areas seen in the images span several kilometres and each image contains regions of several colours. The colours are also more pronounced with arid land being yellow; waterbod-ies, black; and vegetation, red. It may be seen that both average colour and colour variance features are in the reduced subset. Colour is seen as important in the third database with flowers too. For Brodatz database, texture features are primarily selected because these images are in grayscale and colour corresponds only to intensity.

The selected feature subsets are quantitatively validated by using them in a CBIR system and eval

uating the retrieval precision. A graph of precision as the number of retrieved images is-varied from 5 to 30 is shown in Figure 3 for the aerial database. The precision plotted is the average over 5 queries. The graphs for the three experiments reveal that highest precision is achieved for Experiment 3 with the DT constructed using all the samples. Experiment 2 (clustering) performs better than Experiment 1 (pairwise similarity) initially but then falls to the fourth place. As the similarity is based only on one query image, very little information is contained in the experiment. In fact, the precision values appear proportional to the amount of information captured in each experiment. Experiment 1 relies only on pair-wise comparison, while Experiment 3 (PAC) is an explicit representation of the semantic classes and so performs better than the others.

(a) Retrieved images (Rl) for Experiment 1

Stan i w 7 (e) 01 tor Experiment 3 (PAC)

(d) Rl for Experiment 3 (PAC Unbiased)

Fig. 4. Results for a sample query image from the aerial view-database. Query image is the Irst image on the left in each row.

A sample query and the results from all the experiments are shown in Figure 4. For each experiment, the top-5 retrieved images are shown in the figure. Only 2 of the top-5 images are relevant in Expt 2 (clustering). Three images are relevant in Expts 1 (pairwise similarity) and 3 (PAC Unbiased). However ? it may be seen that the top-3 images in the PAC Unbiased method are relevant. The best performance is for Expt 3 (PAC) where 4 out of the top-5 images are relevant.

Experiment 1 is also repeated on the Aerial

248 Feature Selection Based on Human Perception of Image Similarity for Content Based Image Retrieval

database with population sizes of 50 and 100 individuals in the genetic algorithm. Several points were observed with the increased population sizes. The first is that the convergence was faster with larger population sizes. While it took nearly 129 generations for the experiment with population of 16 to converge, it took only 55 and 27 generations for population sizes of 50 and 100 respectively. Secondly, the correlation coefficient with the human similarity matrix increased to 0.6 for 50 individuals from 0.56 for 16 individuals. The correlation decreased marginally to 0.59 for 100 individuals. Thirdly, and most importantly, the subset of features selected by the experiment with 16 individuals is substantially retained for larger population sizes too. These results are summarized in Figure 5.

Population 16

50

100

Correl.

0.5634

0.6084

0.5938

Generations (convergence)

129

55

27

Features

4,7,9,10, 11,14 0,9,10 11,14 0,1,3,4, 9 ,10-14

Fig. 5. Effects of varying population sizes on GA results

It is interesting to note that the features 9,10,11,14, i.e., three Gabor texture features and the CSD texture feature were retained in all the three experiments showing the importance of texture in human perception for high resolution aerial imagery. The inclusion of colour feature (0 feature) is intuitive if one looks at the example result shown in Figure 4 where the images show distinct colour in addition to the texture comprising buildings and streets. The number of features selected increased significantly when 100 individuals were used in the experiment. However, the correlation with human similarity matrix did not change much.

The features selected by the GA were also used for retrieving images from the aerial database. Average precision was calculated by retrieving the top-30 matching images for a random set of 5 query images. Average precision increased from approximately 45%

for 16 individuals to about 70% for 50 individuals and decreased back to 60% for 100 individuals. It is possible that the G A is overspecializing when a larger number of individuals were used in the experiment. However, the results are preliminary and more experimentation is needed to study the full impact of population sizes on (a) capturing human perception (via the correlation value) and (b) on the performance in a CBIR application (via the average precision).

6. Conclusions

In this paper, we demonstrated an approach for finding a set of low-level image features useful in evaluating image similarities that is based on human perception. Our three experiments and their quantitative validation via retrieval precision indicates that our genetic algorithm based approach is a successful attempt at bridging the semantic gap. That we are able to intuitively corroborate the features selected by GA provides additional support that our approach is in the right direction.

References

1. N. Doulamis and A. Doulamis, Signal Process-ingdmage Communication 20, 334 (2006).

2. J. Li and J. Z. Wang, IEEE Trans. Pattern Analysis and Machine Intelligence 25, 1075 (2003).

3. M. E. Celebi and Y. A. Aslandogan, Content-based image retrieval incorporating models of human perception, in ITCC, 2004.

4. T. Frese, C. A. Bouman and J. P. Allebach, A methodology for designing image similarity metrics based on human visual system models, in Proc. of SPIE/IS&T Conf. on Human Vision and Electronic Imaging II, 1997.

5. A. Mojsilovic, Matching and retrieval based on the vocabulary and grammar of color patterns, in ICMCS, 1999.

6. N. Guyader, H. Le Borgne, J. Herault and A. Guerin-Dugue, Towards the introduction of human perception in a natural scene classification system, in IEEE Int. Workshop on Neural Networks for Signal Processing, September 2003.

7. B. S. Manjunath, J.-R. Ohm, V. V. Vasudevan and A. Yamada, IEEE Trans. Circuits Systems and Video Technology 11, 703 (2001).

8. T. Mitchell, Machine Learning (McGraw Hill, 1997).

249

Identification of Team in Possession of Ball in a Soccer Video Using Static and Dynamic Segmentation

V. Pallavi, Jayanta Mukherjee and A. K. Majumdar

Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur, 721302 E-mail: {pallavi, jay, akmj}<8cse.iitkgp.ernet.in

Shamik Sural

School of Information Technology Indian Institute of Technology, Kharagpur, 721302

E-mail: [email protected]

In this paper we describe a novel approach to identify the state of ball possession by two teams in a soccer video. The proposed approach uses a combination of a few techniques to identify the presence of players and ball. We segment soccer video frames by Markov Random Field (MRF) clustering technique and use a colour-texture feature for classifying the players of the two teams. The players are detected by a combination of static analysis, optical flow analysis and difference image analysis. The classification decisions from the three techniques are combined together using the Dempster Shafer theory of evidence. The ball in the video frames is identified using Hough Transform for circle detection. The proposed approach of combining evidences from various techniques makes the analysis more robust.

Keywords: Video analysis; Segmentation; Ball possession; Dempster Shafer Theory

1. Introduction

Increasing availability and use of video has led to a demand for efficient and accurate automated analysis techniques. As the volume and complexity of video information grows, the need for more intelligent video manipulation techniques also increases. Sports videos are usually distributed to a large audience and have a wide appeal. Among various kinds of sports video, we concentrate on soccer video analysis.

Different approaches for analyzing sports video are reported in the literature. Motion based event recognition using Hidden Markov Model (HMM) was proposed by Yang et al.1 to classify basketball video. Li and Sezan2 also proposed a Hidden Markov Model using colour, motion and shape features to discriminate between play and break sequences in baseball, soccer and sumo videos. Jonker et al.3 used Hidden Markov Model for recognizing strokes in tennis videos. Some of these works are also related to the analysis of soccer video. Shum et al.4 proposed an automatic technique for extracting colour models of the field and the team uniforms. Dominant color region detection and shot classification have been used by Tekalp et al.5 for automatic soccer video analysis

and summarization. Most of the approaches use either static or dynamic features for soccer video analysis. But we use a combination of static and dynamic features for video analysis. In this paper we describe a new approach based on a state based video model to identify the team that is in possession of the ball in medium shots of a soccer video.

In the proposed scheme, shots in the soccer video are detected by a colour-texture histogram and then they are classified by a dominant region histogram shot detection technique.6 Hough Transform is applied to detect circles as probable ball candidates for ball. We also use a combination of static and motion analysis to identify the presence of players in a video shot. Static analysis uses colour and texture features while motion analysis includes difference image and optical flow techniques. The classification decisions of these individual techniques are combined together by the Dempster Shafer theory of evidence. The rest of the paper is organized as follows. Section 2 describes the state based video model for representing soccer video. Section 3 describes ball identification as well as our method for player detection. In Section 4, we identify the team in possession of the ball. We show our results in Section 5 and we conclude the paper in Section 6.

250 Identification of Team in Possession of Ball in a Soccer

2. State Based Video Model

A video data model is a means of extracting information contained in an unstructured video data and representing relevant information in order to support users queries. The major existing approaches for video modelling are temporal modelling, object-oriented modelling and algebraic modelling of video.

We adopt an object oriented approach for describing a soccer video. A video object is a spatio-temporal region that corresponds to a real-world object. In soccer video, players, referee, ball and goal area can be treated as objects. Soccer video objects can be described by one of the low level features such as colour, texture, motion, shape, etc. So the object oriented approach is best suited for describing a soccer video. In our work, we consider three objects in a soccer video, namely the two teams and the background that includes the field and the spectators.

A detailed study of soccer videos shows that dynamic properties of the video objects dictate the video content. Hence modelling the behaviour of video objects is very important. An approach to describe the video content is essential when a video is used as a medium to analyze the dynamic behaviour of a system. We take up a state chart based formalism7,8 to describe the semantics of video in terms of intrinsic behaviour of perceptible objects of interest in a video like its states and state transitions. Fig. 1 shows the state chart diagram for possession of ball between two teams in a soccer video. The two classes

Fig. 1. State Chart diagram for ball possession.

i.e. Team A and Team B try to possess the ball during the play. As the match continues, either objects of Team A or of Team B are in ball possessing state. There might be a fight state where objects of both the classes fight to come to the ball possession state.

3. Object Detection and Classification

We partition a video stream into shots. The shots are then classified into one of long, medium or close

Video Using Static and Dynamic Segmentation

shot. Since our work primarily concentrates on the problem of identifying the team in possession of the ball only for medium shots, we detect medium shots in a soccer video. We perform shot detection using a colour-texture histogram from the HSV colour space.9 The shots identified here are then classified into close shots, medium shots or long shots by a shot classification technique6 based on dominant color ratio. These medium shots are then processed to locate the ball and the players.

3.1. Ball Detection using Hough Transform

The fact that the ball is present in a shot is essential to identify the team in possession of the ball. We therefore identify all the frames in the medium shots where the ball is present. Ball is circular in shape and has significant motion. We use these shape and motion features to detect ball in the frames of a soccer video. Hough transform for detection of circles is used to identify the ball in video frames. With this technique many probable ball candidates are detected. We then find the optical flow velocity for each candidate. Candidate having the maximum velocity is identified as the ball region. It has been observed that the ball is detected with an accuracy of 86.21%. Fig. 2 shows the ball identified in a medium view. Then we detect and classify player objects us-

Fig. 2. Ball identified using Hough Transform.

ing static and dynamic segmentation in the frames containing the ball.

3.2. Identification of Objects using Static Segmentation

One of the basic requirements of soccer analysis is to group low-level pixel information into meaningful

V. Pallavi, Jayanta Mukherjee, A. K. Majumdar, Shamik Sural 251

sets or partitions so that relevant information can be abstracted from the video frames. We have used a segmentation algorithm10 based on MRF processing. This partitions a video frame into segments or regions.

Each frame from medium shot where the ball is identified by ball detection algorithm is selected and segmented. The segments thus obtained after segmentation belong to one of the three classes. Each segment is considered and its COLTEX9 feature vector, which is a colour texture histogram, is generated. 1-NN Classification rule is applied to identify the class to which the segment belongs using the Object Classifier.11 Since in our subsequent work, we want to identify only the players so we consider only the segments classified to Team A or Team B. Experiments were carried out with medium shots from the soccer video match played between Real Madrid and Manchester United (UEFA Champions League 2003). We use sensitivity and specificity as the performance measures which are denned as:

a = No of correctly classified segments belonging to a class b = No of actual segments of a class c = No of segments classified to a class

Sensitivity = | , Specificity = ^

This method gives a sensitivity of 70.83 % and specificity of 64.23 % for Team A and a sensitivity of 71.43% and specificity of 7.43% for Team B. It has been observed that some spectators wear jerseys similar to that of the players to cheer their favourite teams. As a result in some frames a few spectator segments get identified as the team causing low specificity.

3.3. Identification of Objects using Dynamic Segmentation

The disadvantage of identifying objects with static segmentation method in the above section is its low specificity. The use of only colour and texture information causes this low specificity. Players in the soccer match are usually in a moving state. So we introduce dynamic image segmentation based on motion features to obtain more legitimate segments of objects. In the literature moving object detection techniques fall under three main categories - opti

cal flow segmentation, frame differencing and background subtraction. For static cameras, background subtraction is probably the most popular method. Since soccer video depends on parameters like camera motion, camera angles and focusing parameters, we use the optical flow segmentation12 and frame differencing13 methods. In a soccer video, both player as well as the camera move, so the task here is to extract only the moving objects.

3.3.1. Optical Flow Analysis

Optical flow is the distribution of apparent velocities of movement of brightness patterns in an image. Thus, optical flow velocity can give important information about the spatial arrangement of the objects viewed and the rate of change of their arrangement. We use Horn and Schunck's method12 to compute the optical flow in the soccer video frames to identify the moving objects that is the players in the soccer video.

A frame from a medium shot is selected and its motion vectors are obtained. All the segments or the regions formed after the segmentation algorithm based on MRF processing are now considered one by one. For each region we find the ratio of the number of moving pixels to the number of pixels belonging to that region. If the ratio lies between 0.2 and 0.6, then the segment is considered to be moving. Thus, we get some player segments from the frames, which are objects of both the team classes.

We have experimented with the medium shots from the same soccer video. The algorithm identifies some segments as moving objects of the soccer frame and then Object Classifier classifies them to Team A, Team B or the background. This method gives a sensitivity of 70.83% and specificity of 89.58% for Team A while a sensitivity of 54.54% and specificity of 59.09% for Team B. This method keeps the sensitivity and specificity for Team A same as compared to the static segmentation method. The improved specificity of this technique to that of the static analysis is strong evidence that the segments identified belongs to the moving objects.

3.3.2. Difference Image Analysis

Difference image analysis13 is applied on frames within the medium shot. A negative and a positive

252 Identification of Team in Possession of Ball in a Soccer Video Using Static and Dynamic Segmentation

difference image are computed which show the moving objects in a frame. The difference image analysis gives good results for the frame in a shot where there is no camera motion. Since camera motion is present even in a single shot so we apply camera motion correction11 to the frames. All the segments formed after segmentation10 are now considered one by one for integration with difference image analysis.11 This gives some segments that are supposed to be the players. These segments are then classified into teams using Object Classifier.

This method gives a sensitivity of 70.83% and specificity of 89.58% for Team A while a sensitivity of 67.86% and specificity of 69.05% for Team B. This method increases the specificity while keeping sensitivity almost the same as compared to the static analysis method. It is also observed that the percentage of correct object coverage increases here as number of segments identified as moving objects has been reduced. Thus, moving object detection using difference image analysis gives better performance than static segmentation.

It has been observed that in some frames where both the teams are present then segments belonging to Team A or Team B are identified by more than one technique. As a result combining all the evidences becomes essential.

4. Dempster Shafer Theory for Combining Evidences

In the previous section we have used static as well as dynamic segmentation for detecting the players. In this section we use Dempster Shafer theory to combine evidences obtained from different classification techniques like static segmentation, optical flow analysis and difference image analysis. Dempster Shafer theory is based on two ideas: the idea of obtaining degrees of belief for a hypothesis and Dempsters rule for combining such degrees of belief when they are based on independent items of evidence. There could have been different approaches to combine the evidences like voting principle, Bayesian approach, logistic regression or some adhoc methods like simple averages. But, in these approaches either the decisions are treated equally or there is difficulty in accurately estimating the probabilities. So we use Dempster Shafer theory where a probability function can define the degree of beliefs.

4.1. Dempster Shafer Theory

The Dempster Shafer theory starts by assuming a Universe of Discourse '©'also called Frame of Discernment, which is a set of mutually exclusive alternatives. It associates to each subset 's' of ' 6 ' : a basic probability m(s), a belief Bel(s) and a plausibility belief Pla(s). m(s), Bel(s), Pla(s) have values in the interval [0,1] and Bel(s) is not greater than Pla(s). 'm' represents the strength of some evidence. If a rule is applied, then 'm' may represent the effect of that rule. Bel(s) summarizes all the reasons to believe 's'. Pla(s) expresses how much should be believed in 's' if all currently unknown facts were to support 's'. A map m : 2e —> [0,1] such that for each XeO we have :

m(<f>) = 0

Exrf "»(*) = !

is called the basic probability assignment (bpa) for 0 . Here $ is a null set.

Suppose we are interested in finding the combined evidence for a hypothesis C. Given two independent sources of evidences m\ and m^, Dempsters rule for combining evidences is as follows.

Here m,i^(C) is the combined Dempster Shafer's bpa for C. m\ and m,2 are the basic probabilities assigned to sets A and B respectively by two independent sources of evidence. A and B are sets that include C. A and B are not necessarily proper subsets and they may as well be equal to C. The numerator accumulates the evidence, which supports a particular hypothesis and the denominator conditions it on the total evidence for the hypothesis supported by both sources.

4.2. Identification of Team in Possession of Ball

Here we apply the Dempster Shafer theory to identify the team in possession of ball in the medium shots. We have three sources of evidence: The output of static segmentation that identifies all the regions which belonging to Team A or Team B and excludes all the regions which are classified as background. Then we have optical flow analysis technique

V. Pallavi, Jayanta Mukherjee, A. K. Majumdar, Shamik Sural 253

and difference image technique that identifies some moving regions out of a frame. Our objective is to integrate these evidences in support of our hypothesis of ball possession. We have assumed if players of both the teams exist in a frame then there is a fight for ball possession. On the other hand, a player of a single team would indicate the ball possession by that team only. Thus in a shot the ball may be in possession of Team A or Team B or in fight state. Even within a shot these states may be present. We would be verifying our hypothesis by the states of ball possession manually observed for medium shots.

4.2.1. Basic Probability Assignment

By default all the three techniques operate independently. The results of any of the three techniques do not affect each other. Hence the independent assumption of the Dempster Shafers theory holds. We designate the basic probabilities for the three techniques as mi, 7B2 and m.3 respectively. A video frame

N

J- is segmented into N regions such that T = M Rn

7 1 = 1

Each of the regions R can be classified into Team A, Team B or background. The three classes constitute a set of mutually exclusive alternatives. So, the Dempster Shafer Theory starts here by assuming © such that: 9 = { "TeamA", "TeamB", "Background" } Therefore, superset V(Q) is such that: "P(0={ "TeamA", "TeamB", "Background", "TeamA.TeamB", "TeamA,Background", "TeamB,Background", "TeamA,TeamB,Background" } Frames in a medium shot may have a player or more than one player from any of the two teams. Here we try to find the presence of a player in a frame by assigning basic probabilities to each of the three classes by a function that gives the evidence of the presence of a class in the frame. Let us denote this measure for the ith source of evidence as Ei(e),eeV(&). We have modeled the function Ei(e) as follows: Ei(e) = AeSi, (i = 1,2,3) are the three independent sources of evidence. Si is the sensitivity of e for the ith source of evidence and Ae is the area of pixels identified as e by the ith source of evidence. Let us also define the normalizing factor for assigning the basic probabilities to eeV(Q) as:Ke = 5Zvee2e AeSi Hence the basic probabilities assigned to eeV(Q) is: mj(e) = y . It may be noted that in our model we accumulate the evidences for the presence of an

object or objects by aggregating the 'areas' of segments related to those objects. At the same time, the measure is also proportional to the sensitivity factor of the classifier that classifies those segments to different objects. Let us consider the static analysis module for a frame J- which shows all the regions that are identified as Team A and Team B by the classifier.

Ei(TeamA) =

Area-of-regions-identifiedjas-TeamAxSiA Ei{TeamB) = Area-of .regions Jdentifiedjis-TeamB XS\B Ex (TeamA, TeamB) = Ei(TeamA) + Ei(TeamB) Ei(TeamA,BG) = Ei(TeamA) + EX(BG) Ei(TeamB,BG) = Ei(TeamB) + Ei(BG) Ei (TeamA, TeamB, BG) = Ei(TeamA) + Ei(TeamB) + Ei(BG) therefore mx(TeamA) = i(El(TeamA)+El(^eamB))

mi(TeamB) = i(El(TeamA)+E1(TeamB)) mi(J3G) = 0

mi(TeamA, TeamB) = ^ T m ^ S ^

Basic probabilities for "Team B", "Background" and "Team A, Team B" can be calculated similarly. Here E\ (BG) is zero because we do not consider the regions identified as background. Basic probabilities can be calculated for optical flow analysis and difference image analysis techniques in a similar way.

4.2.2. Combining Evidences

We combine m\, m^ and rri3, the basic probabilities for our three techniques by Dempster Shafer theory of evidence combination to get mi$,3(TeamA), mi,2,3(TeamB), mi,2,3(TeamA, TeamB) . Since we try to find the presence of a team in medium shots of a soccer video so we consider only the subsets "Team A", "Team B" and "Team A.Team B". Finally we decide the presence of Team A, or Team B or both in a frame. If basic probability of Team A is greater than that of Team B by TIDS then frame belongs to Team A and if basic probability of Team B is greater than Team A by T I D S then frame belongs to Team B. We conclude that both the teams are present if :

((mh2,3(TeamA)) + (mh2,3(TeamB))/2)

254 Identification of Team in Possession of Ball in a Soccer Video Using Static and Dynamic Segmentation

—mi,2.3(TeamA,TeamB) < T2DS-

Empirically, T\DS has been found to be 0.2 and

T2Ds to be 0.12.

5. R e s u l t s

We performed an experiment on all the medium

shots identified by the shot classification technique

in the soccer video played between Real Madrid

and Manchester United . Frames from these medium

shots where ball is identified by the ball detection

algorithm are now processed. Then the Dempster

Shafer theory for combining the three evidences is

applied to find the presence of Team A or Team B or

Team A and Team B in the frames to identify ball

possession states. Out of 28 medium shots 26 were

classified as medium shots by the algorithm. There

were a total of 32 ball possession states and substates

in and within the shots. Out of the 32 states and

substates 23 were correctly identified. True identifi

cation of ball possession states for Team A is 66.67%,

Team B is 80% and fight is 73.33%. Table 1 shows

the comparative results of combining the evidences ie

static analysis (SA), optical flow analysis (OF) and

difference image analysis (DI) using Dempster Shafer

theory. Fig. 3 below shows a bar chart, which plots

the ball possession states and state transitions for

the teams for medium shots in a par t of the match.

Ball Possession State of Teams for a Part of the Match

UmiawiiuA Shot

1 I • Shot (Without B»W

s

Frame Numbers

Fig. 3. Ball possession states of team for a part of the match

6. C o n c l u s i o n s

In this paper we have proposed a new approach to

identify the team in possession of ball in a soccer

video. It is shown that combining static and dynamic

analysis with Dempster Shafer theory of combination

gives good results for detecting moving objects in a

soccer video. This approach thus accurately identi

fies the ball possession states, substates and state

transitions in a soccer video.

Table 1.

Evidences combined SA+OF SA+DI DI+OF

SA+DI+OF

Comparative results

Team A

(%) 45.45 55.56 66.67 66.67

Team B

(%) 80

66.67 80 80

Fight (%)

66.67 80

66.67 73.33

References

1. G. Xu, Y. Ma, H. Zhang and S. Yang, Motion based event recognition using hmm, in Proc. ICPR, (Can-nada, 2002).

2. B. Li and I. Sezan, Event detection and summarization in sports video, in IEEE Workshop on Content-based access of Image and Video Libraries, (Hawaii, 2001).

3. M. Petkovic, Z. Zivkovic and W. Jonker, Recognizing strokes in tennis videos using hidden markov models, in IASTED International Conference Visualization, Imaging and Image Processing, (Spain, 2001).

4. L. Wang, B. Zeng, S. Lin, G. Xu and H. Shum, Automatic extraction of semantic colours in sports video, in ICASSP, (Cannada, 2004).

5. A. Ekin, A. Tekalp and R. Mehrotra, Automatic soccer video analysis and summarization (IEEE Transactions on Image Processing, 2003).

6. V. Pallavi, J. Mukhopadhyay, A. K. Majumdar and S. Sural, Shot classification in soccer videos, in ReTIS, (Kolkata, India, 2006).

7. B. Acharya, A. K. Majumdar and J. Mukherjee, State chart based approach for modelling video of dynamic objects, in SPIEs International Symposium ITCOM, (Colorado, USA, 2001).

8. B. Acharya, A. K. Majumdar and J. Mukherjee, Modelling Dynamic Objects in Video Databases : A Logic Based Approach (Springer Verlag : Lecture Notes in Computer Science, 2001).

9. A. Vadivel, M. Mohan, S. Sural and A. K. Majumdar, A colour-texture histogram from the hsv color space for video shot detection, in International Conference on Vision, Graphics and Image Processing, (Kolkata, India, 2004).

10. J. Mukherjee, MRF clustering for segmentation of colour images (Pattern Recognition Letters, 2002).

11. V. Pallavi, A. Vadivel, S. Sural, A. K. Majumdar and J. Mukhopadhyay, Identification of moving objects in a soccer video, in WCVGIP, (Hyderabad, India, 2006).

V. Pallavi, Jayanta Mukherjee, A. K. Majumdar, Shamik Sural 255

12. B. K. P. Horn and B. G. Schunck, Determining op- tures in dynamic scene analysis (Image and Vision ticalflow (Artificial IntelL, 1981). Computing, 1984).

13. R. Jain, Difference and accumulative difference pic-

256

Image Retrieval Using Color, Texture and Wavelet Transform Moments

R. S. Choras

Deparment of Telecommunications University of Technology & Agriculture

S. Kaliskiego 7, 85-796 Bydgoszcz, Poland E-mail: choras@mail. atr. bydgoszscz.pl

Content-based image retrieval systems allow the user to interactively search image databases looking for those images which are similar to a specified query image. Similarity between images is then assessed by computing similarity between feature vectors. These features are represented in the vector form, and often are combined together. This paper explores novel wavelet approach to image retrieval based on a combination of color, texture and wavelet moments features. Color moments are used as color features which increase the precision of the retrieval process. Texture features are described by the mean, variance and energy of wavelet decomposition coefficients in some subbands. For describing shapes we propose a method that combines feature vector as a set of wavelet moment invariants.

Keywords: Image retrieval; wavelet transform; texture features; pseudo Zernike moments; wavelet moments.

1. Introduction

Visual information retrieval represents a new research direction in fields information technology.1

In a typical content-based image database retrieval application, the user has an image and is interested in finding similar images from the entire database. For each image in the database, a feature vector characterizing some image properties is computed and stored in a feature database. For a query image, its feature vector is computed, compared to the feature vectors in the feature database, and images most similar to the query are returned to the user (Figure 1).

The general idea of retrieval mechanisms can be described in the following steps:

• for each image in the database, a feature vector characterizing image properties is computed and stored in a feature database,

• feature vector of a query image is computed and compared to the feature vectors in the feature database,

• most similar images to the query image are returned to the user.

The features and the similarity function of the feature vectors should be efficient to match similar images and be able to discriminate dissimilar ones.

This work has been motivated by previous results on image analysis and content description.2

This paper combines the color, texture and wavelet moments to image content representation for still image databases.

QUERY FORMATION

IMAGE DATABASE

* VISUAL CONTENT DESCRIPTION

FEATURE VECTORS

FEATURE DATABASE

SIMILARITY COMPARISON

INDEXING and RETRIEVAL

RESULTS

Fig. 1. CBIR system

Wavelet transform is a powerful analysis tool for image and signal processing. Wavelet transform decomposes image into multiresolution subbands with lower dimensionality. The wavelet transform is known for its capability of multi-resolution decomposition and coefficients decorrelation and may characterize images better than other known techniques for content-based image retrieval. Therefore, features from different subbands can form an multidimensional feature vector with different dimensions most

R. S. Choras 257

likely uncorrelated to each other. This multidimensional feature vector will be suitable to represent the image.

Feature extraction is a data mapping procedure which determines an appropriate subspace of dimensionality M from the original feature space of dimensionality N (N < M). The local energy variation of an image in different spectral bands at each scale (or frequency) provide useful information for image classification.

2. Wavelet Transform

A continuous 2-D wavelet transform is a projection of the image function f(x, y) onto a family of functions which are linear combinations (dilations and translations) of a unique function * (X ,T / ) . 3 The function ty(x,y) is called a 2-D mother wavelet, and the corresponding family of wavelet functions is given by

*»,«,»(a;) = > / I t f ( s ( x - u , y - v ) ) (1)

where s, u,v,£ R and s and (u, v) are called the scale and the translation. The 2-D wavelet transform of the function f(x, y) e Li is defined by

/

+oo /H-oo / f(x,y)ys,u,v(x,y)dxdy

-00 J — OO

(2) This transform can be interpreted as a decom

position of an image function f(x,y) such that the frequency spectrum of the transformed function T<bf(s,u,v) has the same bandwidth in a logarithmic scale s.

Let the function^ be a differential operator, then the transform T*/(s , u, v) will be a continuous scale-space representation of the image edges. If we use low-pass approximation operator $ instead of high-pass differential operator ^ , the transform will produce a continuous scale-space representation of the image approximations.

Using the 2-D continuous wavelet transform T<s/f(s,u,v) or the 2-D continuous scale transform T$f(s,u,v) we can decompose the image function f(x,y) on different levels of resolution (depending on the scale s).

We obtain discrete dyadic wavelet functions by discretizing both the scale factor s and the translation (u,v). The corresponding set of the dyadic wavelet functions and the scale functions is

Vm(x) = y/s${SX - k) (3)

where s = i and (u = k,v = I), where j,k,l € Z, and where Z denotes the set of integers. m is actually the function of j and k: m = 2j + k, k = 0 , 1 , . . . , 2n'j - 1 for j = 0 , 1 , . . . , n.

The coefficients of the wavelet function can be computed according to the formula

/

+oo /(x)tt(2>'a; - k)dx

•OO

(4) where (•, •) denotes the scalar product. The cor

responding inverse wavelet transform is

OO

f{x) = Y, Cm%^{x) (5) m=0

For multidimensional analysis of the function f(x,y) we introduce an approximation operator Aj, then Ajf will be an approximation of / on the j t h level of resolution, Ao represents the identity, Ajf € Vj holds. The scale s on the j t h level of approximation is s = JJ- . Practically, we use limited number of levels j = 0 , 1 , . . . , n, where the nth level will be the coarsest resolution with the smallest scale s = jr . Next we introduce a differential operator Dj. The term Djf will denote the difference between two approximations Ajf and Aj-if on the j t h and the (j — l)th levels of resolution, i.e.,

Djf = AJ-1f-Ajf for j = l , 2 , . . . , n (6)

Each approximation Ajf(x,y) and difference Dj:Pf(x,y) can be fully characterized with a 2-D scale function $(x, y) and its associated wavelet functions $!(x,y), p = l , 2 , 3 ,

+oo +oo

Ajf{x,y)= Yl Yl ahk,i$j,kAx,y) fc=—oo (=—oo

+oo +oo

Dj,Pf{x,y)= Y $ Z dhk,i,P®j,kAx,y) (7) fc=—oo J=—oo

where

258 Image Retrieval Using Color, Texture and Wavelet Transform Moments

» a

£-'

ib Fig. 2. Process of wavelet analysis.

Fig. 3. Wavelet pyramid.

*i,JW(a?,tf) = 5 J ^ ( ^ — 5 ™ ^ — )

•**,!*(*> y) = ^ * p ( £ ^ » ^ ) **>' € z (8)

%\M = {/(^2/)»*i,M(^y)>5

rfj,fc,i,P = (/(£, y), *j,fc,/fP(a:, y)> (9)

Suppose that the coordinates x and y are not correlated, and the 2-D scale function #(x,t/) and 2-D wavelet functions #(x, y), p = 1,2,3 are separable. Than we can write

#(x, y) = 0(x)^(y)

* i (x ,y) = ^(x)#(y)

^ J ) = # W I / ) (io) where 0(x) is a 1-D scale function and ip(x) is

a 1-D wavelet function. Apparently, # 1 ? # 2 5 ^ 3 extract the details of 2-D image function f(x, y) in the x-axis, y-axis and in the diagonal directions, respectively.

This representation is called the wavelet pyramid of 2-D image function. Given a discrete image / (x , y) with a limited support x,y = 1,2, . . . ,2 n , the actual procedure for constructing this pyramid involves computing the coefficients a^kj and djtkti,p which can be grouped into four matrices A$, Dj,pi p = 1,2,3, on each level j

Aj = (%-,fc,i), DjiP = (dj,fc,i,p)

for M = l , 2 , . . . , 2 n - ' (11)

DWT decomposes an image into a pyramid structure of subimages with various resolutions corresponding to different scales. Given N x N image, a fully decomposed wavelet transformation results in 3 x flog2 N~\ +1 subimages called subbands. The computation low of the pyramidal decomposition and the results of an image are shown in Fig. 2 and 3.

3 . Color features

Each pixel of the image can be represented as a point in a 3D color space. Commonly used color spaces for image retrieval include RGB, Munsell, CIE l*a%*, CIE L*u*v*, HSV (or HSL, MSB). The CIE L*a*b* and CIE L*u*v^ spaces are device independent and considered to be perceptually uniform. They consist of a luminance or lightness component (L) and two chromatic components a and b or u and v. HSV (or HSL, or HSB) space is widely used in computer graphics and the three color components are hue, saturation (lightness) and value (brightness). The hue is invariant to the changes in illumination and camera direction and hence more suited to object retrieval.

Color in general will not allow us to determine an object's identity and for robust retrieval needs additional information such as texture and shape.4 Color feature vector is used to retrieve false positives - images which have a similar color composition as the query image but with a completely different content.

A color image I(x, y) of size X x Y consists of three channels I = ( IR , IQ, IB)* We compute a j level 2D wavelet transform on each of three matrices. Then Aj of each transform matrix represents the lowest frequency band and is used to extract color information.

R. S. Choras 259

Thus, if we interpret the color distribution of an image as a probability distribution, then the color distribution can be characterized by its moments, as well,56 Color moments have been successfully used in many retrieval systems. Furthermore, because most of the information is concentrated on the low-order moments, only the first moment (mean), the second moment (variance), and the third central moment (skewness) were used. If the value of the i color channel at the (x,y) image pixel is Pi(x, y) and the number of pixels in the image is N = X x Y, then the index entries related to this color channel are:

1 X Y

1 x Y

I x Y

x = l y—1

Since only 9 (three moments for each Aj of the three color components) numbers are used to represent the color content of each image, color moments are a very compact representation compared to other color features.

The similarity function used for retrieval is a weighted sum of the absolute differences between the suitable moments. Each moment entry is weighted by a value wn, tUj2, wa > 0 selected by the user. The images to be compared have been quantized on the same color spaces.

4. Texture Features

Texture is another important property of images. While color is a point property, texture is a local neighborhood property. Basically, texture representation methods can be classified into two categories: structural and statistical.7 Texture refers to the visual patterns that have properties of homogeneity that do not result from the presence of only a single color.

The method of feature extraction of texture based on wavelet decomposition is as follows:

(1) Wavelet decomposition. We perform a multires-olution decomposition of the image using DWT. The discrete wavelet transform can be thought of as filtering of the image into sub-bands by an array of scale and orientation specific filters. Since wavelets have good spatial and frequency localization, the wavelet coefficients provide a good characterization of the image objects. For the DWT we used D6 orthogonal wavelet which prevent correlation between scales in the decomposition of the image. We perform a three-level wavelet decomposition of an image which results in 10 sub-bands.

(2) Define energy of wavelets coefficients in each sub-bands.

E(Aj)= Yl 4 M (14)

E(Dj<p)= Yl 4w , P ' P = 1 ' 2 > 3 (15)

(3) The feature vector is constructed as (for p = 1,2,3)

Bj = [E(Aj), E(DjtP), E{Dj-l<p), S(£>,_2,p)] (16)

(4) Compute similarity distance

S1(Bj) = \Bj-Bj{ref)\ (17)

and

S2,U) = S2(Aj)*S2{Dj,p) (18)

where

\Aj{k + r,l + c)* AUref)(k + r, I + c)\ 2[ j) \Aj(k + r,l + c)\\Aji{ref)(k + r,l + c)\

(19)

S2(Dj,p) = \Dj,p - .Bj,p(re/)l (20)

(r, c) £ N and iV is the neighbourhood of a given point.

dcoior(Q,I) = YWil\mi Tni\+Wa\(7i -°i\+Wi3\sf-s!\ (13)

260 Image Retrieval Using Color, Texture and Wavelet Transform Moments

In this approach we used information about energy wavelet coefficients in 10 subbands to construct feature vector. The discrimination of different textures may be performed on the basis of only Djti and Djt2 high frequency components (horizontal and vertical subband images). The low-pass components is not used because textures are better described through the higher frequency channel than through approximation component. In this case, steps involved in extracting the texture feature vector, are the following:

(1) At each j t h level calculate the mean and standard deviation of the detailed subimages

2 " - J 2n~>

& (Dj*') = F77 E E l^>'.p' I (21) fe=i ;= i

arj(Djy) = 2n~i 2™~i

1 k=X l=\

for k, I = 1,2,. . . , 2"-J and p' = 1,2. (2) Feature vector is defined as follows:

Bj = I/i.M'/j^'M/'^/]

(22)

(23)

where

/>> = 2[muj(Dj,i) + mUj(Dj,2)}

fj,* = ^aj(Dj'l)+aj(Dj'2^ and fif,crf are respectively mean and stan

dard deviation the original image.

In this method for the level decomposition we have 8 features.

The texture similarity of a query image Q and a image I in the database is defined as

dtextureiQ, /)(M, < ) = E E d W ' 7 ) ( 2 4 ) m n

where

d(Q,I) = (Q) U)

s ( ^ ) + <?*-<? H ( ^ )

(25)

where S (^ m n ) and E(amn) are respectively mean and the variance of the transform coefficients over the database.

5. Shape Feature Extraction Based On Wavelet Transform

Our approach for feature extraction is based on DWT, and on calculation of their moments. DWT decomposes image into several different sub-bands. Some sub-bands are fed into moment descriptor block for analysis. The output from this block are a wavelet moment based feature and is further used to dissimilarity distance computation for feature matching Fig. 4.

" : • <

Fig. 4. Feature extraction based on DWT

Basically, shape based image retrieval is the measuring of similarity between shapes represented by their features. Shape is an important visual feature and it is one of the primitive features for image content description. However, shape content description is a difficult task because it is difficult to define perceptual shape features and measure the similarity between shapes. To make the problem more complex, shape is often corrupted with noise, defection, arbitrary distortion and occlusion. To characterize the shape we used following descriptors: Pseudo -Zernike moments of Wavelet Coefficient and wavelet moments.

5.1. Feature Extraction Based on Moments of Wavelet Coefficient

Pseudo-Zernike moments are used in several pattern recognition applications as feature descriptors of the image shape, and have proven to be superior to other moment functions such as Zernike moments in terms of their feature representation capabilities and robustness in the presence of image quantization error and noise.8

Pseudo-Zernike moments offer more feature vectors than Zernike moments and are less sensitive to image noise than the conventional Zernike moments. However, pseudo-Zernike moments have not

R. S. Choras 261

been effectively used in image processing applications because their computation with present methods is more complex and lengthy compared to regular moments, Legendre moments and even Zernike moments. To reduce computational requirements of pseudo Zernike moments we used the wavelet transform. Wavelet transform decomposes image into multires-olution subbands with lower dimensionality. Then, a Aj subband is input into pseudo Zernike moments for feature description.

The 2D pseudo Zernike moments of order s with repetition q of subband Aj = (oj.r.e) are defined as

PZsq = -^— / Vsq(r,6)Ajrdrd6 (26) i" Jo Jo

where Zernike polynomials

Vaq(r,6) = Rsge-i«e; j = >/=! (27)

andr = yjx1 + y2, 9 = arctan(^); —l<x,y<l.

The real-valued radial polynomials are defined as