Tutorial on Medical Image Retrieval - IRMA€¦ · •IRMA code mono-hierarchical, multi-axial •Technique, Direction, Anatomy, Biosystem (TTTT-DDD-AAA-BBB) •Example: 1121-127-720-500

Tutorial on Medical Image Retrieval- IRMA -

Medical Informatics Europe 2005

28.08.2005

2©2004 Hôpitaux Universitaires de Genève / RWTH Aachen

raw data layerraw data layer

registered data layerregistered data layer

feature layerfeature layer

scheme layerscheme layer

object layerobject layer

knowledge layerknowledge layerretrievalretrieval

imagesimages

feature extractionfeature extraction

query resultsquery results

queryquery

indexingindexing

feature vectorsfeature vectors

blob-treesblob-trees

registrationregistration

RST-parametersRST-parameters

feature selectionfeature selection

identificationidentification

categorizationcategorization

categoriescategories

Information Processing (IRMA)








imagesimages



queryquery

indexingindexing









Information Processing


•• Arbitrarily taken from routineArbitrarily taken from routine

•• authenticauthentic

•• Manually classifiedManually classified

•• reliablereliable

Images: IRMA Reference Database


•• Standardized nomenclatureStandardized nomenclature

•• Systemized Nomenclature in Medicine (SNOMED)Systemized Nomenclature in Medicine (SNOMED)

•• Medical Subject Headings (Medical Subject Headings (MeSHMeSH))

•• Unified Medical Language System (UMLS)Unified Medical Language System (UMLS)

•• Digital Imaging Communications in Medicine (DICOM)Digital Imaging Communications in Medicine (DICOM)

•• Example DICOMExample DICOM

•• Tag (0018,0015) Tag (0018,0015) Body Part ExaminedBody Part Examined, 26 valid entries, 26 valid entries

•• ExtremityExtremity, , HandHand, , ArmArm, ..., ...

•• ProblemsProblems

•• coarsecoarse, , ambiguousambiguous,, non non--hierachicalhierachical

Images: Consistent Categorization


•• IRMA codeIRMA code

mono-hierarchical, multi-axialmono-hierarchical, multi-axial

•• Technique, Direction, Anatomy, Technique, Direction, Anatomy, BiosystemBiosystem

(TTTT-DDD-AAA-BBB)(TTTT-DDD-AAA-BBB)

•• Example: 1121-127-720-500Example: 1121-127-720-500

•• rradiographyadiography, plain, analog, overview, plain, analog, overview

•• ccoronaloronal, AP, supine, AP, supine

•• aabdomenbdomen, middle, middle

•• uuropoetic ropoetic systemsystem

Images: Consistent Categorization








imagesimages



queryquery

indexingindexing

















imagesimages



queryquery

indexingindexing











Categorization of Medical Images

•• Pixel data Pixel data !! Semantical Semantical meaningmeaning

•• X-X-ray imageray image

of skullof skull

•• laterallateral view view

•• good qualitygood quality

•• no obviousno obvious

pathologypathology

•• ......

•• ApplicationsApplications

•• ddigital igital radiologyradiology & PACS & PACS

•• ccomputeromputer-aided diagnosis (CAD)-aided diagnosis (CAD)

•• ccontentontent-based image retrieval (CBIR)-based image retrieval (CBIR)


•• Pixel data Pixel data !! Few numerical values Few numerical values

3.2783.278

1.4261.426

0.4500.450

......

3.263.26

•• FeaturesFeatures

•• colourcolour

•• texturetexture

•• shapeshape

Categorization: Global Features


•• SemanticalSemantical meaning meaning "" Few numerical values Few numerical values

3.2783.278

1.4261.426

0.4500.450

......

3.263.26

•• X-ray imageX-ray imageof skullof skull

•• lateral viewlateral view

•• good qualitygood quality

•• no obviousno obviouspathologypathology

•• ......

•• Common approachesCommon approaches

•• low number of categorieslow number of categories

•• IRMAIRMA approach approach

•• monomono--hierarchical coding schemehierarchical coding scheme

Semantical Gap


•• Classified images at study dateClassified images at study date

•• 6,335 images, 401 different codes6,335 images, 401 different codes

•• MinimumMinimum class size class size

•• 1010 images images

•• Hierarchical code Hierarchical code ((TTxxTTxx--DxxDxx--AAxAAx--BxxBxx))

•• 6,231 images6,231 images, , 81 categories81 categories

•• Radiographs only Radiographs only (11xx-(11xx-DxxDxx--AAxAAx--BxxBxx))

•• 5,776 5,776 imagesimages, , 57 categories57 categories

•• Newer version of the database contains 10,000Newer version of the database contains 10,000

radiographs and was used for the ImageCLEFradiographs and was used for the ImageCLEF

2005 Evaluation2005 Evaluation

Categorization: Evaluation


•• Texture: Tamura FeaturesTexture: Tamura Features

•• local features (multi-spectral)local features (multi-spectral)

•• histogram (384 bins)histogram (384 bins)

coarsenesscoarseness contrastcontrast directionalitydirectionality

•• DistanceDistance

•• Jensen-Shannon divergenceJensen-Shannon divergence

Texture


•• Image FeaturesImage Features

•• rescaled imagesrescaled images

squaredsquared 32 x 3232 x 32 16 x 1616 x 16 8 x 88 x 8 4 x 44 x 4

•• DistancesDistances

•• ccrossross correlation functioncorrelation function

•• iimage distortion modelmage distortion model

•• tangent distancetangent distance

Images


•• Pixel mappingPixel mapping

•• scans local neighborhood for best correspondencescans local neighborhood for best correspondence

•• allows local deformationsallows local deformations

•• ImprovementsImprovements

•• uuse image gradientsse image gradients

•• eevaluate local contextvaluate local context

Image Distortion Model


•• Leaving-one-outLeaving-one-out

•• 6,231 images (all imaging techniques)6,231 images (all imaging techniques)

•• 81 categories81 categories

97.0 %90.1 %80.7 %82.3 %IDM

97.7 %86.6 %76.4 %76.1 %CCF

96.6 %80.4 %66.3 %66.4 %Tamura

within 10within 55-NN1-NNRepresentation

Results (Single Classifiers)


•• Leaving-one-outLeaving-one-out

•• 5,776 images (radiographs only)5,776 images (radiographs only)

•• 57 categories57 categories

97.1 %90.0 %80.6 %81.8 %IDM

97.9 %86.4 %76.1 %75.4 %CCF

96.4 %79.3 %64.6 %64.5 %Tamura

within 10within 55-NN1-NNRepresentation

Results (Single Classifiers)


•• IDM + TamuraIDM + Tamura

•• Improvement (radiographs)Improvement (radiographs)

•• Correctness:Correctness: 81.8% 81.8% !! 85.0%85.0%

•• Within 5:Within 5: 90.0% 90.0% !! 92.8% 92.8%

95.2 %92.8 %85.0 %85.0 %Radiographs

(57 categories)

95.3 %93.0 %85.4 %85.5 %All images

(81 categories)

within 10within 55-NN1-NNSetup

Results (Parallel Combination)


•• Absolute distance not evaluatedAbsolute distance not evaluated

•• Inhomogeneous category sizeInhomogeneous category size

•• High intra-category variabilityHigh intra-category variability

•• grouped codegrouped code

•• Low inter-category variabilityLow inter-category variability

•• misclassified body regionmisclassified body region

•• misclassified directionmisclassified direction

•• Collimation fields and shuttersCollimation fields and shutters

Reasons for Misclassifications


•• e.g.: radiographs only, min. 5 imagese.g.: radiographs only, min. 5 images

00

200200

400400

600600

800800

10001000

12001200

14001400

11 55 99 1313 1717 2121 2525 2929 3333 3737 4141 4545 4949 5353 5757

images images inin category category

category numbercategory number

recognition recognition raterate99.45 %99.45 %

99.84 %99.84 %

14.29 %14.29 %

0.00 %0.00 %

Inhomogeneous Category Size


•• e.g.: IRMA-code 1121-120-800-700e.g.: IRMA-code 1121-120-800-700

•• plain radiographyplain radiography

•• coronal, coronal, posterioanterior posterioanterior directiondirection

•• abdomenabdomen

•• musculoskeletal systemmusculoskeletal system

Intra-Category Variability


•• Axial, Axial, craniocaudalcraniocaudal ((xxxxxxxx-310-xxx-xxx)-310-xxx-xxx)

•• OtherOther, , obliqueoblique ( (xxxxxxxx-410--410-xxxxxx--xxxxxx))

Inter-Category Similarity (Direction)


•• Elbow (Elbow (xxxxxxxx-xxx-44x-xxx)-xxx-44x-xxx)

•• Knee (Knee (xxxxxxxx--xxxxxx--9494x-x-xxxxxx))

Inter-Category Similarity (Anatomy)


•• Query imageQuery image

•• SystemSystem response response

Collimation Fields and Shutter


•• 85% correctness85% correctness

•• ~6,000 radiographs, ~80 categories~6,000 radiographs, ~80 categories

•• scaled representations, parallel combined classifierscaled representations, parallel combined classifier

•• 98 % correctness98 % correctness

•• within 10 most similar referenceswithin 10 most similar references

•• GlobalGlobal features applicable for many categories features applicable for many categories

if sufficient references are availableif sufficient references are available

Conclusion


•• Thorax images in frontal view (1,278 images)Thorax images in frontal view (1,278 images)

•• Nearest neighborsNearest neighbors

Example: 99.5% Recognition








imagesimages



queryquery

indexingindexing

















imagesimages



queryquery









indexingindexing



•• PreprocessingPreprocessing

•• categorization based on global featurescategorization based on global features

•• registration onto category prototypesregistration onto category prototypes

•• local feature extractionlocal feature extraction

(explosion of data volume)(explosion of data volume)

•• AbstractionAbstraction

•• significant reduction of informationsignificant reduction of information

Indexing


•• Local FeaturesLocal Features

•• Each pixel Each pixel !! One feature vector One feature vector

•• Example: Example: Blobworld Blobworld featuresfeatures

polaritypolarity anisotropyanisotropy contrastcontrastgray valuegray value

Structured Analysis of Image Content


•• Information abstractionInformation abstraction

•• image image pixels pixels regions regions blobs blobs



•• Hierarchical modelingHierarchical modeling

•• blobs blobs !! graph graph

(edge preserving) region growing(edge preserving) region growing



•• ExampleExample

1

2

3

4

5

6

78

9

11

10

12

1

0

2

3

4

6

59

7

10

8

11

•• image image !! pixel pixel !! regions regions !! blobs blobs !! graphgraph









imagesimages



queryquery









indexingindexing








knowledge layerknowledge layer

imagesimages


queryquery








indexingindexing


retrievalretrieval


System Design


•• IRMA Database BrowserIRMA Database Browser

•• visual overview of all imagesvisual overview of all images

•• IRMA QueryIRMA Query

•• extended query refinementextended query refinement

•• IRMA Code DefinitionIRMA Code Definition

•• define IRMA code itemsdefine IRMA code items

•• IRMA Code EditorIRMA Code Editor

•• manual reference coding of imagesmanual reference coding of images

•• IRMA Code BrowserIRMA Code Browser

•• access images by IRMA codeaccess images by IRMA code

Exemplary Applications


•• German / English interface to code databaseGerman / English interface to code database

•• 738 entities in four axes738 entities in four axes

•• history logginghistory logging

Code Definition


•• Reference categorization of imagesReference categorization of images

•• history logging of changeshistory logging of changes

currently over 15,000 history entriescurrently over 15,000 history entries

Code Editor


•• Browsing the reference database by IRMA codeBrowsing the reference database by IRMA code

•• e.g., e.g., anteroposterior anteroposterior chest radiographschest radiographs

(1123-111-500-000)(1123-111-500-000)

Code Browser


•• IRMAIRMA

•• general concept for medical image retrievalgeneral concept for medical image retrieval

•• any modality or body region (categorization)any modality or body region (categorization)

•• any level of details (hierarchical representation)any level of details (hierarchical representation)

•• successful evaluation of categorizationsuccessful evaluation of categorization

•• good results in the good results in the ImageCLEF ImageCLEF evaluationevaluation

Summary


•• IRMA is a joint ventureIRMA is a joint venture

•• Department of Medical InformaticsDepartment of Medical Informatics

•• Department of Diagnostic RadiologyDepartment of Diagnostic Radiology

•• Chair of Computer Science VIChair of Computer Science VI

•• FundsFunds

•• RWTH Aachen UniversityRWTH Aachen University

grant START 81/98grant START 81/98

•• German Research Foundation (DFG)German Research Foundation (DFG)

grants Le 1108/4-1, 1108/4-2, 1108/6grants Le 1108/4-1, 1108/4-2, 1108/6

!!http://http://irmairma-project.org-project.org

Acknowledgments








imagesimages



queryquery









indexingindexing

retrievalretrieval



Retrieval = Similarity of graphsRetrieval = Similarity of graphs

•• QualitativelyQualitatively:: Visualization Visualization

•• QuantitativelyQuantitatively:: Graphmatching Graphmatching









imagesimages



queryquery









indexingindexing

retrievalretrieval


Tutorial on Medical Image Retrieval - IRMA€¦ · •IRMA code mono-hierarchical, multi-axial •Technique, Direction, Anatomy, Biosystem (TTTT-DDD-AAA-BBB) •Example: 1121-127-720-500

Documents