Descriptive Semantic Image Retrieval David Norton Derral Heath
Feb 21, 2016
Descriptive Semantic Image Retrieval
David NortonDerral Heath
Motivation Retrieve an image based on a descriptive
query: “Find me an image that is red, dark, scary, and
beautiful”
Content-Based Image Retrieval Retrieve an image strictly from image features
color texture shape
General semantic based image retrieval is hard “Find me a picture of a piranha”
Emotional Semantic Image Retrieval Query images matching emotional words or
word-pairs “Find me a happy picture”
Usually adjectives
Usually a small orthogonal subset of terms
Query via single words (or pairs)
Descriptive Semantic Image Retrieval Open to all descriptive words
Query via any number of words “Find me an image that is red, dark, scary, and
beautiful”
Three Components Extraction of image features
Semantic representation of image
Mapping between visuals and language
Extraction of Image Features Color (12)
Average RGB values Color count
Texture (50) Entropy
Shape (10) Eccentricity
Extraction of Image Features
Average R Average G Average B Hue Count Hue Percent Entropy Eccentricity
114.0887 99.0219 118.648 20 0.1332 8.924632 0.109940296
113.5899 98.5232 118.145 18 0.1411 8.200488 0.101902564
113.6192 98.5358 118.153 20 0.1511 8.015762 0.082569348
Extraction of Image Features
Average R Average G Average B Hue Count Hue Percent Entropy Eccentricity
146.4833 75.8135 1.547 5 0.5259 7.805696 0.064152
145.9641 75.2931 1 4 0.5317 7.654775 0.073264
146.0377 75.3628 1.054 4 0.5496 7.502806 0.054083
Semantic Representation of Image How do we obtain a description?
What is a descriptive word?
What are the features?
User Input Interface
Bright has 11 Senses 1. (17) bright -- (emitting or reflecting light readily or in large amounts; "the sun was bright and hot"; "a
bright sunlit room") 2. (6) bright, brilliant, vivid -- (having striking color; "bright dress"; "brilliant tapestries"; "a bird with vivid
plumage") 3. (5) bright, smart -- (characterized by quickness and ease in learning; "some children are brighter in one
subject than another"; "smart children talk earlier than the average") 4. (3) bright -- (having lots of light either natural or artificial; "the room was bright and airy"; "a stage bright
with spotlights") 5. (1) bright, burnished, lustrous, shining, shiny -- (made smooth and bright by or as if by rubbing;
reflecting a sheen or glow; "bright silver candlesticks"; "a burnished brass knocker"; "she brushed her hair until it fell in lustrous auburn waves"; "rows of shining glasses"; "shiny black patents")
6. (1) bright -- (splendid; "the bright stars of stage and screen"; "a bright moment in history"; "the bright pageantry of court")
7. undimmed, bright -- (not made dim or less bright; "undimmed headlights"; "surprisingly the curtain started to rise while the houselights were still undimmed")
8. bright, brilliant -- (clear and sharp and ringing; "the bright sound of the trumpet section"; "the brilliant sound of the trumpets")
9. bright -- (characterized by happiness or gladness; "bright faces"; "all the world seems bright and gay") 10. bright, shining, shiny, sunshiny, sunny -- (abounding with sunlight; "a bright sunny day"; "one shining
morning"- John Muir; "when it is warm and shiny") 11. bright, promising -- (full or promise; "had a bright future in publishing"; "the scandal threatened an
abrupt end to a promising political career")
Narrowing Down the Feature Space Interface:
Adjectives from WordNet
Restrict characters
Reduce available senses
Narrowing Down the Feature Space Post Processing:
Use Synsets
Frequent synsets
Fit ORM ontology lexicons
Image ORM Ontology
Mapping between visuals and language Series of Neural Networks
Bayes Net
Fuzzy Logic
Evaluation Let machine label images
Let humans label images
Let different humans evaluate machine and human labels
Compare evaluations
Related Work Aesthetic Visual Quality Assessment of
Paintings (2009) Congcong Li and Tsuhan Chen
Labeled impressionistic style landscape paintings as ‘high’ or ‘low’ quality using machine learning.
Algorithmic Inferencing of Aesthetics and Emotion in Natural Images: An Exposition (October 2008) Ritendra Datta, Jia Li, and James Z. Wang
Overview of research involving predicting the quality class, score, and emotional label of photographs.
Related Work A Survey on Emotional Semantic Image
Retrieval (2008) Weining Wang and Qianhua He
Surveys ongoing Emotional Semantic Image Retrieval research.
Image Retrieval by Emotional Semantics: A Study of Emotional Space and Feature Extraction (October 2006) Wang Wei-ning, Yu Ying-lin, and Jiang Sheng-ming
Labeled paintings with 12 emotional word pairs. Psychological research involved in choosing word pairs.
Further Motivation Augment the study of human perception and
cognition.
Establish a linguistic-visual foundation for an artificially creative artist.
Questions?