Texture-Based Segmentation - Inria...Motivation Method Results Future Work Texture-Based Segmentation Internship at the project team PULSAR A. Schnaars1 1University of Magdeburg Supervisor:

MotivationMethodResults

Future Work

Texture-Based SegmentationInternship at the project team PULSAR

A. Schnaars1

1University of MagdeburgSupervisor: Guillaume Charpiat PhD

Pulsar - INRIA Sophia-Antipolis

INRIA Sophia-Antipolis, 15th march 2010

A.Schnaars Texture-Based Segmentation

MotivationMethodResults

Future Work

Outline

1 Motivation

2 MethodCreating a Texture ClassifierCreating an Edge ClassifierGlobal Optimization

3 Results

4 Future Work

A.Schnaars Texture-Based Segmentation

MotivationMethodResults

Future Work

Definition

No unique definition of what a texture is:

Human Vision System[3]Information of texture to distinguish between things whereedge of objects in the enviroment are not defined by clearboundariesLarge number of symbols or simple shapes -> individualobjectsV1 neurons in the primary visual cortex: Texture elementorientation, size, contrast and color

!Consisting of repetition or quasi repetition of some fundamentalimage elements

A.Schnaars Texture-Based Segmentation

MotivationMethodResults

Future Work

State of the Art & Goal

There is no “optimal” texture featureStructrual, statistical, model-based approachSIFT and SURF most widly usedSparsly application in REAL-time processing like video-analysis

GoalCombination of several informations (scale, boundaries, ...)Fast classificationRobust classifierBasis for object segmentation

A.Schnaars Texture-Based Segmentation

MotivationMethodResults

Future Work

Creating a Texture ClassifierCreating an Edge ClassifierGlobal Optimization

Outline

1 Motivation

2 MethodCreating a Texture ClassifierCreating an Edge ClassifierGlobal Optimization

3 Results

4 Future Work

A.Schnaars Texture-Based Segmentation

MotivationMethodResults

Future Work

Creating a Texture ClassifierCreating an Edge ClassifierGlobal Optimization

Ground Truth

Problem: Mostly segmentation of objects but not of singletexture regions and no annotations informationDatabase: Texture patterns from free texture libraries !17di!erent texture classes, each pattern in 3 di!erent resolutions

Brick ...

Fabric ...Grounds ...

Hair ...

Plants ...Skin ...

A.Schnaars Texture-Based Segmentation

MotivationMethodResults

Future Work

Creating a Texture ClassifierCreating an Edge ClassifierGlobal Optimization

Texture Descriptor

Requirements:Fast processingScale invarianceIntroduction of color informationIllumination invariance

Integrative method: Combination of color and texture informationCIE L*a*b* color spaceUnser Sum- and Di!erence-Histogram Features

A.Schnaars Texture-Based Segmentation

MotivationMethodResults

Future Work

Creating a Texture ClassifierCreating an Edge ClassifierGlobal Optimization

Unser Sum- and Di!erence-Histogram FeaturesTexture Descriptor

Approximation for the two-dimensional Haralick texturefeatures on the co-occurrence matrixFrequencies of sums, respectively di!erences of pixel colorlevels with a certain displacement (dx , dy ) within a region ofinterest D

A.Schnaars Texture-Based Segmentation

MotivationMethodResults

Future Work

Creating a Texture ClassifierCreating an Edge ClassifierGlobal Optimization

Unser Sum- and Di!erence-Histogram FeaturesTexture Descriptor

hcs (i) =Card {(x1, y1) ! Dc |gx1,y1 +gx2,y2 | = i} , i ! [0; 2(G c "1)] ,

hcd (j) =Card {(x1, y1) ! Dc |gx1,y1 "gx2,y2 | = j} ,

j ! ["G c +1; G c "1] , c ! L#a #b#h3s/d (i) ={hL

s/d (i), has/d (i), hb

s/d (i)}

A.Schnaars Texture-Based Segmentation

MotivationMethodResults

Future Work

Creating a Texture ClassifierCreating an Edge ClassifierGlobal Optimization

Unser Sum- and Di!erence-Histogram FeaturesTexture Descriptor

Features

Sum/ di!erence mean: fcj = 1N !i i hc

s/d (i)

Sum/ di!erence contrast: fcj = 1N !i (i "µ)2 hc

s/d (i)

Sum/ di!erence angular second momentum: fi = !i

!h3s/d (i)

"2

Sum/ di!erence entropy: fj = !i "h3s/d (i)log(h3

s/d (i))

Plus color values of each color channel

Regions of interest: 5x5, 7x7, 9x98 directions x displacement 1, 2, 4

$1000 Features

A.Schnaars Texture-Based Segmentation

MotivationMethodResults

Future Work

Creating a Texture ClassifierCreating an Edge ClassifierGlobal Optimization

Feature Selection and Reduction

Normalisation via a sigmodial function to keep outliers

y % =y "µstd

y %% =1" e"y %

1+ e"y %

A.Schnaars Texture-Based Segmentation

MotivationMethodResults

Future Work

Creating a Texture ClassifierCreating an Edge ClassifierGlobal Optimization

PCAFeature Selection and Reduction

Assumption: Correlated featuresPCA: Uncorrelating data by mapping the data into a mostsignificant subspaceGoal: Feature selection through analysis of the most importantprinciple components!Features with same loadings have the same character

Feature selection (as defined in [2])kMeans on the reduced principle componentsChoosing feature as representants whose loadings have thesmallest distance to a cluster’s center

A.Schnaars Texture-Based Segmentation

MotivationMethodResults

Future Work

Creating a Texture ClassifierCreating an Edge ClassifierGlobal Optimization

Interim ResultsClustering of the Reduced Texture Descriptor

Problems at:BoundariesCoarse texturesSubtle color shifts (small value di!erences)

A.Schnaars Texture-Based Segmentation

MotivationMethodResults

Future Work

Creating a Texture ClassifierCreating an Edge ClassifierGlobal Optimization

Textures Classes Training

Assumption: Each class could be described by a mixture ofgaussiansProblem: Big amount of data!Online learning of the parameters

PTexture(x) =!k

p(x |µk ,!k)"k

A.Schnaars Texture-Based Segmentation

MotivationMethodResults

Future Work

Creating a Texture ClassifierCreating an Edge ClassifierGlobal Optimization

Outline

1 Motivation

2 MethodCreating a Texture ClassifierCreating an Edge ClassifierGlobal Optimization

3 Results

4 Future Work

A.Schnaars Texture-Based Segmentation

MotivationMethodResults

Future Work

Creating a Texture ClassifierCreating an Edge ClassifierGlobal Optimization

Ground Truth

Berkeley Segmentation Dataset [1]: Hand-labeled segmentationsfrom 30 human subjects, public benchmark contains 300 imagesand its segmentations

A.Schnaars Texture-Based Segmentation

MotivationMethodResults

Future Work

Creating a Texture ClassifierCreating an Edge ClassifierGlobal Optimization

Edge Descriptor

For each channel of the CIE L*a*b* color space:Sobel Operator

First and second derivativesBlocksizes 1, 2, 3, 5, 7Magnitude and orientation

A.Schnaars Texture-Based Segmentation

MotivationMethodResults

Future Work

Creating a Texture ClassifierCreating an Edge ClassifierGlobal Optimization

Edge Descriptor

Autocorrelation Matrix

A(x) =!x ,y

w(x ,y)

#I 2x (x ,y) Ix Iy (x ,y)

Ix Iy (x ,y) I 2y (x ,y)

$

Block sizes of 3, 9, 23The biggest Eigenvalue

Plus color values$70 Features

A.Schnaars Texture-Based Segmentation

MotivationMethodResults

Future Work

Creating a Texture ClassifierCreating an Edge ClassifierGlobal Optimization

Feature Selection and Reduction

A.Schnaars Texture-Based Segmentation

MotivationMethodResults

Future Work

Creating a Texture ClassifierCreating an Edge ClassifierGlobal Optimization

Edge Classifier Training

Assumption: Edges can be very noisyClassifier that seperates boundaries clearly! SVMIssue: Big amount of data!Online learning of the parameters : Neural Network

A.Schnaars Texture-Based Segmentation

MotivationMethodResults

Future Work

Creating a Texture ClassifierCreating an Edge ClassifierGlobal Optimization

Interim Results

A.Schnaars Texture-Based Segmentation

MotivationMethodResults

Future Work

Creating a Texture ClassifierCreating an Edge ClassifierGlobal Optimization

Interim Results

Results are moderate compared to the e!ortEventually overfit

A.Schnaars Texture-Based Segmentation

MotivationMethodResults

Future Work

Creating a Texture ClassifierCreating an Edge ClassifierGlobal Optimization

Outline

1 Motivation

2 MethodCreating a Texture ClassifierCreating an Edge ClassifierGlobal Optimization

3 Results

4 Future Work

A.Schnaars Texture-Based Segmentation

MotivationMethodResults

Future Work

Combination of all learned informations in a (minimised) costfunction:

Graph Cut

E (p) = !p!I

L(p)+# !p,q!I

V (L(p),L(q))

L(p) =" log(PTexture(p))

V (L(p),L(q)) ="log(min(PEdge(q),PEdge(p)))

A.Schnaars Texture-Based Segmentation

MotivationMethodResults

Future Work

Extension

Measurement of the interactions between di!erent textureclassesLocation of some texture

Frequency of certain neighboring textures in certain directions

Problems:Annotation of the data, e.g. Berkeley tool only segmentsOptimisation tool (GraphCut): No di!erent interactionsbetween two classes in di!erent directions

A.Schnaars Texture-Based Segmentation

MotivationMethodResults

Future Work

Further Applications

Object recognitionLearning shape parameters of certain textures in certain objectsLearning locations of objectsProbability to find an object in an area based on its locationand texture components

A.Schnaars Texture-Based Segmentation

Appendix

Thank you for your attention!

Questions?

A.Schnaars Texture-Based Segmentation

Appendix For Further Reading

For Further Reading I

Pablo Arbelaez, Charless Fowlkes, and David Martin.The berkeley segmentation dataset and benchmark, 2007.

Yijuan Lu, Ira Cohen, Xiang Sean Zhou, and Qi Tian.Feature selection using principal feature analysis.In MULTIMEDIA ’07: Proceedings of the 15th internationalconference on Multimedia, pages 301–304, New York, NY,USA, 2007. ACM.

Colin Ware.Visual Thinking for Design.Morgan Kaufmann, July 2008.

A.Schnaars Texture-Based Segmentation