A Graphical User A Graphical User Interface for a Fine-Interface for a Fine-Art Painting Image Art Painting Image Retrieval SystemRetrieval System
October 15-16, 2004October 15-16, 2004
Multimedia Information Retrieval 2004
IntroductionIntroductionStudents of art history learnStudents of art history learn
three primary skills:three primary skills:
Formal analysisFormal analysis ComparisonComparison ClassificationClassification
How can computer science How can computer science
contribute to the contribute to the developmentdevelopment
of these skills?of these skills? Girl with a Pearl Earring, Jan Vermeer, 1665
Multimedia Information Retrieval 2004
Working HypothesisWorking Hypothesis
An Interactive Indexing and Image An Interactive Indexing and Image Retrieval System (IIR) for fine-art Retrieval System (IIR) for fine-art paintings can aid students in these paintings can aid students in these endeavors by providing:endeavors by providing: a mathematical summarization of an imagea mathematical summarization of an image a measurable basis for comparing two a measurable basis for comparing two
imagesimages an elementary way to classify an image an elementary way to classify an image
relative to those in a database relative to those in a database
Multimedia Information Retrieval 2004
Previous WorkPrevious Work
We synthesize the goals of two research areas:We synthesize the goals of two research areas: Classification of paintings which often requires Classification of paintings which often requires
special images (brush stroke detection) or special images (brush stroke detection) or features with little semantic relevance to art features with little semantic relevance to art studentsstudents
Image retrieval which aims to bridge the semantic Image retrieval which aims to bridge the semantic gapgap
Can we find a feature set that satisfies the Can we find a feature set that satisfies the objectives of objectives of
both areas while providing analytically relevant data both areas while providing analytically relevant data to students?to students?
Multimedia Information Retrieval 2004
System OverviewSystem Overview
The system consists of two major The system consists of two major components:components:
Image Database Image Database stores images, thumbnail images, and stores images, thumbnail images, and
extracted features for later retrieval extracted features for later retrieval and analysis.and analysis.
Graphical User Interface Graphical User Interface provides interactive query capabilities provides interactive query capabilities
to the end userto the end user
Multimedia Information Retrieval 2004
Database ConstructionDatabase Construction
An XML index file stores extracted An XML index file stores extracted features and control informationfeatures and control information
A file system stores images and A file system stores images and thumbnail imagesthumbnail images
Multimedia Information Retrieval 2004
Database Construction – Database Construction – Cont.Cont.
XML Index File File System
Multimedia Information Retrieval 2004
Global Feature Global Feature ExtractionExtraction
Two different kinds of features are extracted:Two different kinds of features are extracted: Palette features Palette features
concern the set of colors in an image (color concern the set of colors in an image (color map)map)
examples: palette scopeexamples: palette scope Canvas features Canvas features
concern the spatial and frequency distribution concern the spatial and frequency distribution of colors in an image (image index)of colors in an image (image index)
examples: max, min, median, mean (for each examples: max, min, median, mean (for each color channel)color channel)
Multimedia Information Retrieval 2004
Example: Palette ScopeExample: Palette Scope
Palette ScopePalette Scope -- the total number of unique colors -- the total number of unique colors used in an image.used in an image.
We expect Dali’s piece to have a higher palette We expect Dali’s piece to have a higher palette depth than Mondrian’s work.depth than Mondrian’s work.
Hallucinogenic ToreadorSalvador Dali, 1970
Composition with Large Blue Plane,Red, Black, Yellow, and GrayPiet Mondrian, 1921
Multimedia Information Retrieval 2004
Example: Palette Scope – Example: Palette Scope – Cont.Cont.
ArtistArtist RGB Raw Palette ScopeRGB Raw Palette Scope NormalizedNormalized
MondrianMondrian 3000530005 0.001788430.00178843
DaliDali 7661376613 0.004566490.00456649
We see that Dali uses twice as much of the color spectrum as Mondrian.
Palette scope is an important feature for artist and period style identification because many styles are defined by color, i.e. Picasso’s Blue Period and fauvism.
Multimedia Information Retrieval 2004
Graphical User InterfaceGraphical User Interface
Multimedia Information Retrieval 2004
Test ResultsTest Results
Two types of tests were Two types of tests were conducted:conducted:
Feature testsFeature tests Interactive testsInteractive tests
Multimedia Information Retrieval 2004
Test Results – Cont.Test Results – Cont.
Training SetTraining Set Test SetTest Set Percent Percent CorrectCorrect
3636 3636 9494
200200 200200 8888
200200 200200 8383
Les Demoiselles d’Avignon,Pablo Picasso, 1907.
Road with Cypress and Star,Vincent Van Gogh, 1890.
Feature test to distinguish the work of Picasso and Van Gogh.
Multimedia Information Retrieval 2004
Initial Interactive TestInitial Interactive Test
Database of 10 works of each of the following Database of 10 works of each of the following ten artists: ten artists:
Braque, Cezanne, De Chirico, El Greco, Braque, Cezanne, De Chirico, El Greco, Gauguin, Gauguin,
Modigliani, Mondrian, Picasso, Rembrandt, Modigliani, Mondrian, Picasso, Rembrandt, and Van and Van
Gogh.Gogh.Training SetTraining Set Testing SetTesting Set Percent Percent CorrectCorrect
100100 9090 8181
Multimedia Information Retrieval 2004
Interactive Test on Web Museum DatabaseArtistArtist Training Training
SetSetQueriesQueries SuccessSuccess PercentPercent
AertsenAertsen 99 88 77 87.587.5
El GrecoEl Greco 1010 88 44 50.050.0
HopperHopper 1010 88 11 12.512.5
MalevichMalevich 1010 1111 66 54.554.5
MonetMonet 1010 1010 66 60.060.0
MorisotMorisot 1010 1111 55 45.545.5
RembrandtRembrandt 1010 3232 2323 71.971.9
RenoirRenoir 1010 3838 1212 31.631.6
TurnerTurner 1010 1010 33 30.030.0
VelazquezVelazquez 1010 88 77 87.587.5
OverallOverall 500500 299299 147147 49.249.2
Multimedia Information Retrieval 2004
EvaluationEvaluation ofWeb Museum Test ResultsTest Results
Overall result: 49.2% accuracy Overall result: 49.2% accuracy 29.2% better than blind guessing (10 29.2% better than blind guessing (10
guesses/50 artists = 20%)guesses/50 artists = 20%) Dissecting the classification Dissecting the classification
mistakes reveals some intelligent mistakes reveals some intelligent mistakesmistakes Rembrandt is most often confused with Rembrandt is most often confused with
Caravaggio, Ast, and VermeerCaravaggio, Ast, and Vermeer
Multimedia Information Retrieval 2004
ConclusionsConclusions Simple palette and canvas features are Simple palette and canvas features are
sufficient for an interactive classification sufficient for an interactive classification systemsystem
A single feature set can serve for A single feature set can serve for classification and image retrieval classification and image retrieval applicationsapplications
A general feature set can adequately serve A general feature set can adequately serve for educational applicationsfor educational applications
Although showing promise, we currently Although showing promise, we currently have a low confidence system have a low confidence system
Multimedia Information Retrieval 2004
Future WorkFuture Work
Can computer science provide an Can computer science provide an empirical framework for the empirical framework for the study of painting?study of painting?
Quantitative descriptionQuantitative description Falsifiable statementsFalsifiable statements Hypothesis verificationHypothesis verification