T.Sharon 1 Internet Resources Discovery (IRD) Introduction to MMIR
Jan 08, 2016
T.Sharon1
Internet Resources Discovery (IRD)
Introduction to MMIR
T.Sharon2
Contents
• Visual Information Retrieval (VIR)– Images– Video
• Video Information Retrieval (VIR)
• Music Information Retrieval (MIR)
T.Sharon3
Visual Information Retrieval
• Introduction
• VIR system
• VIR information domains
• Querying video
• Advanced topics
T.Sharon4
Introduction• What is VIR?
• Who needs it?
• Questions and problems
T.Sharon5
What is VIR?
Query
VIRSystem
VIR allows users toquery and retrieve visual information.
Queries will be doneaccording to informationcontent.
T.Sharon6
Who needs VIR?• Libraries
• Museums
• Scientific Archives
• Image Warehouses
T.Sharon7
Questions and Problems• How can we search visual information?
• How can visual and non-visual information can be searched together?
• Problems:– visual information is subjectively interpreted.– few representations: images, graphics, video,
animations, stereoscopic images.– requires substantial amount of resources.
Dog??
T.Sharon8
VIR System
• Architecture
• Query formulation
• Match and ranking
• Query answer
• Refinement (relevance feedback)
T.Sharon9
Architecture of VIR System
T.Sharon10
Query Formulation
• Query by example:– sketch an example– give an image
• Query by giving values to visual features:– % colors– texture– describe textually
but use visual tools to define values.
T.Sharon11
Query matching and ranking
• Similarity test• Using combination of features, for example:
– colors– texture– shape– motion– additional information
• Actions in feature space can be:– maximal distance– K nearest neighbors
........ .....
Feature 1
Feature 2
Feature 3
T.Sharon12
Query Answer
Thumbnails:
• Images – DC images
• Video– built from
selected DC images (key frames)
T.Sharon13
Query Refinement
• Using a result image from previous query.
• Launch a new query.
• Modifying a result image with an image processing tool to specify an an additional criteria.
• Changing relative weights of visual features and get a new ranking to the previous results.
T.Sharon14
VIR Information Domains
• Information domain
• Queries at Pixels Level– examples– problems
• Implementations– color– color complex– shape
T.Sharon15
Information Domains
• Metadata information– alphanumeric, database scheme
• Visual characteristic– contained in the object– achieved by using computational process, usually
image processing
T.Sharon16
Queries at Pixels Level - Examples
Find all objects for which the 100th to 200th pixels are orange (RGB=255,130,0).
Find all the images that have about the same color (certain RGB) in the central region (relative or absolute).
Find all images that are a shifted version of this particular image, in which the maximum allowable shift is D.
T.Sharon17
Queries at Pixels Level -Problems
Pixel queries are noise sensitivecouple of noise pixels can cause to discard a good
image.
Do not work on rotations.Changes in lightning and imaging conditions
effect pixels significantly and bias queries.
T.Sharon18
Implementations
• Pixels location combined with
• Database scheme built by humans
• Example techniques:– Color– Color complex– Texture– Shape
T.Sharon19
Color• Method:
– Color definition• Hue (color spectrum)• Saturation (gray)
– Calculate histograms
• Enables queries:– Find all images for which more than 30% is sky blue and
more than 25% is grass green– Sort histogram drawers, find 5 most frequent colors, find all
other images with these color features– Find all images far from this image only D
T.Sharon20
Color Complex• Method: Create histograms quad-tree:
• Calculate image histogram
• Divide image to quarters and calculate histogram for each quarter
• Continue recursively till 16x16 squares
• Enables queries:– Find images for which:
• more than 20% red-orange pixels in the right upper quarter
• more than 20% yellow pixels in the left upper quarter
• about 30% brown pixels in the bottom half of the picture
– Find all images with red patch in the middle and blue patch around.
T.Sharon21
Shape• Method:
– Suppose we have graphics collection (clip arts)• contain pure colors (little hue changes, no saturation)
– Divide image to color areas so that each area contains pixels with the same pure color
– Calculate features for each area:• color, area, elongation (sqrt(perimeter)/area),
centrality (distance of shape centroid to image center)
• Enables queries:– Find all images containing white squares in the center
– Find all images containing 2 blue circles close to the center
T.Sharon22
Examples: Existing Systems• SaFe http://disney.ctr.columbia.edu/SaFe/
• Virage http://www.virage.com/virdemo.html
• QBIC http://wwwqbic.almaden.ibm.com/ (stamps) download!
http://wwwqbic.almaden.ibm.com/cgi-bin/pcd-demo/drawpicker (photos)
• MetaSEEK http://mahler.ctr.columbia.edu:8080/cgi-bin/MetaSEEk_cate
• WebSEEK• VisualSEEK• MELDEX http://www.nzdl.org/cgi-bin/gw?
c=meldex&a=page&p=coltitle
T.Sharon23
SaFe
T.Sharon24
QBIC - Histogram Query
T.Sharon25
QBIC - Color Layout Query