Fast multiresolution image querying CS474/674 – Prof. Bebis.

Fast multiresolution image querying

CS474/674 – Prof. Bebis

Paper

• C. Jacobs, A. Finkelstein, and D. Salesin, “Fast multiresolution image quering”, Proceedings of SIGGRAPH, pp. 277-286, 1995

Problem

• Search an image database to retrieve images that are similar to a query image.

“query by content” or

“query by example”

Typically, the K best matches are returned.

Challenges

• What features to use?

• How to tolerate image distortions?

• How to organize the data?

• How to search fast?

• How to reduce storage requirements?

Image Distortions

• This study considers two types of image distortion:– A low-resolution image from a scanner or video camera.

– A rough sketch of the image painted by the user.

painted low resolution target

Traditional Metrics

• Traditional metrics based on the L1 and L2 norms cannot handle inexact matching and are time consuming.

L1

L2

Q: query

T: target

Experiments using these metrics have shown that the target image

is in the highest 1% of the retrieved images only 3% of the time.

Fast Retrieval

• Retrieval should be fast enough to handle tens of thousands of images at interactive rates.

Efficient image representation

Efficient database organization

Fast metric computation

User Interface

Can process a 128 x 128 image query on a database of 20,000 images in under 0.5 seconds*.

Returns 20 highest-ranked targets at interactive rates!

*Faster processing times should bepossible using current technology!

Proposed Method: Key Ideas

• Multi-resolution image decomposition using Haar wavelets.

• Image similarity is based on a metric that compares how many significant wavelet coefficients the query has in common with potential targets.

• Organize data efficiently to facilitate fast computation of the metric and speed-up search.

Why using wavelets?

• The use of wavelets allows the resolutions of the query and target images to be different.

• Wavelet decompositions are fast to compute and yield a small number of coefficients.

• The signature can be extracted from a wavelet-compressed version of the image directly.

Color space

• Experimented with RGB, HSV, and YIQ color spaces.

• Wavelet transform was applied on each color channel separately.

• YIQ gave the best performance (i.e., for their data).

Wavelet Type

• Haar wavelets are the fastest to compute and simplest to implement.

• Other types of wavelets might give better results but at a higher cost.

Decomposition Type

• Experimented both with standard and non-standard decompositions for all three color spaces.

• Standard decomposition worked best (i.e., both for scanned and painted queries).

• Basis functions are normalized so they become orthonormal to each other (see lecture slides on wavelets).

Coefficient Truncation

• Keep only the coefficients with largest magnitude.– Improves speed and reduces storage requirements.

– Improves discriminatory power of metric.

• 128 x 128 image 1282 = 16,384 wavelet coefficients for each color channel. – The 60 largest coefficients in each channel worked best for

painted queries.

– The 40 largest coefficients in each channel worked best for scanned queries.

Coefficient Truncation (cont’d)

wavelet decomposition

truncated coefficients

Coefficient Quantization

• The mere presence or absence of these coefficients appears to be more important than their precise magnitudes.

– Improves speed and reduces storage requirements.

– Improves discriminatory power of metric.

• Quantize coefficients into three levels: +1, 0 and -1– Large positive coefficients are quantized to +1

– Large negative coefficients are quantized to -1

Components of the metric (cont’d)

truncated coefficients

truncated and quantized coefficients

Wavelet-based metric

• Q and T represent a single color channel of the wavelet decomposition of the query and target images.

• Let Q[0, 0] and T[0, 0] be the scaling function coefficients (i.e., average intensity of that channel).

• Let and represent the truncated, quantized wavelet coefficients of Q and T (i.e., -1,0,1).

wi,j : weights (determined statistically; allow for tuning the metric)

ˆ[ , ]Q i j ˆ[ , ]T i j

Metric:

Simplifying the metric

(1) Replace with ˆ ˆ( [ , ] [ , ])Q i j T i jˆ ˆ( [ , ] [ , ])Q i j T i j

(as effective as the original metric, faster to compute)

(1 if true; 0 otherwise)

Simplifying the metric (cont’d)

(2) Group terms together into "buckets" so that only a smaller number of weights wi,j needs to be determined.

i,j

Simplifying the metric (cont’d)

• The function bin(i, j) groups different coefficients into a small number of bins (i.e., 6 bins per color channel):

bin(i, j) = min(max(i, j), 5)• Each bin is weighted by some constant w[b]

– Weights were determined using a statistical test. 0 1 2 3 4 5

0 1 2 3 4 5

Simplifying the metric (cont’d)

(3) Consider only the terms for which – Leads to even faster computations.

– Allows for a query without much detail to match a very detailed target image.

ˆ[ , ] 0Q i j

i,j

Simplifying the metric (cont’d)

(4) Count the number of matching coefficients than the number of mismatching coefficients.

–Use “search arrays” to implement this efficiently since the majority of database images will not match the query.

(1 if true; 0 otherwise)

Simplifying the metric (cont’d)

Simplifying the metric (cont’d)

• The term does not depend on the target image and can be ignored:

Final Metric:

Example

Search Arrays

• Use a set of six 2D arrays (i.e. search arrays) to organize the m coefficients from every target image T.

• There is an array for every combination of sign (+ or -) and color channel (Y, I, and Q):

contains a list of all target images T having a large positive wavelet coefficient at [i,j] location, in color channel c.

[ , ]cD i je.g.,

Search Arrays (cont’d)

[0,0]cD

[0,1]cD

[ , ]cD i j

…

T1, T9, T22, …

T3, T9, T40, …

…

Compute a score for each target image found in

T1, T22, T98, …

/ [ , ]cD i j

Querying Using Search Arrays

• Steps (for each color channel c)

(1) Compute the difference between the query’s average intensity Qc[0, 0] and those in the database.

(2) For each of the m nonzero, truncated wavelet coefficients Qc[i, j], go through the list corresponding to Dc

+[i, j] or

Dc- [i, j] (i.e., depending on the sign of Qc[i, j]).

(3) Update the score of each target image found in those lists.

Algorithm

• Preprocessing

(1) Perform a standard 2D Haar wavelet decomposition of every image in the database.

(2) Store T[0,0] for each color channel as well as the indices and signs of the m largest wavelet coefficients.

Algorithm (cont’d)

• Querying

(1) Perform a standard 2D Haar wavelet decomposition on the query image.

(2) Compute T[0,0] for each color channel as well as the indices and signs of the m largest wavelet coefficients

(3) Compute the score of each target image using:

(4) Return 20 highest-ranked target images

Examples

• Query examples using painted/scanned queries

(ranks for database sizes: 1093 | 20,558)

Examples (cont’d)

(ranks for database sizes: 1093 | 20,558)

Interactive query examples using painted queries:

Success Rate

Lq : proposed metric

Percentage of querieswhose correct target was ranked among thetop 1% of images ina database of 1093images.

Time requirements

Lq : proposed metric

Average times to matcha single query in a database of 1093/20,558images.

Extension

• V. Nikulin and G. Bebis, "Multiresolution Image Retrieval Through Fusion", SPIE Electronic Imaging (Storage and Retrieval Methods and Applications for Multimedia), San Jose, January 2004.