11 Scale Invariant Feature Transform (SIFT) David G. Lowe University of British Columbia.

11

Scale Invariant Feature Transform (SIFT)

David G. Lowe

University of British Columbia

2

References

[1] D.G. Lowe, “Object recognition from local scale-invariant features,” in Proc. Seventh IEEE Int’l Conf. Computer Vision, vol. 2, pp. 1150 -1157, 1999.

[2] D.G. Lowe, “Distinctive Image Features from Scale-Invariant Keypoints,” Int’l J. Computer Vision, vol. 2, no. 60, pp. 91-110, 2004.

2

33

Outline

• Introduction• Algorithm• Applications• Conclusions

44

Outline

• Introduction• Algorithm• Applications• Conclusions

55

An Example: Build a Panorama

66

How Do We Build Panorama?

• We need to match (align) images

77

Matching with Features

• Detect feature points in both images

• Find corresponding pairs

• Use these pairs to align images

8

An Example: Harris Detector

• It is non-invariant to image scale

8

All points will be classified as edges

Corner

99

Scale Invariant Detection

• Consider regions (e.g. circles) of different sizes around a point

• Regions of corresponding sizes will look the same in both images

1010

Scale Invariant Detection

• The problem: how do we choose corresponding circles independently in each image?

11

Laplacian of Gaussian Operator

• Maxima of Laplacian gives best notion of scale:

• Characteristic scale of a local structure

scale

bad

scale

Good !

1212

Introduction

• Generates image features, “keypoints”– invariant to image scaling and rotation– partially invariant to change in illumination

and 3D camera viewpoint– many can be extracted from typical images– highly distinctive

1313

Outline

• Introduction• Algorithm• Applications• Conclusions

1414

Algorithm

• Scale-space extrema detection• Keypoint localization• Orientation assignment• Keypoint descriptor

( )local descriptor

detector

descriptor

1515

Outline

• Introduction• Algorithm

– Scale-space extrema detection– Keypoint localization– Orientation assignment– Keypoint descriptor

• Applications• Conclusions

1616

Detection of Scale-space Extrema

• Detect locations that are invariant to scale change of the image

• SIFT uses DoG filter for scale space because it is efficient and as stable as scale-normalized Laplacian of Gaussian

1717

DoG Filtering

• Convolution with a variable-scale Gaussian

• Convolution with the DoG filter

( , , ) ( , , ) ( , )L x y G x y I x y 2 2 2( ) / 2

2

1( , , )

2x yG x y e

( , , ) ( ( , , ) ( , , )) ( , )

( , , ) ( , , )

D x y G x y k G x y I x y

L x y k L x y

1818

Scale Space

1919

Scale Space: an Example

( , , ) :L x y

( , , ) :D x y

2020

Outline

• Introduction• Algorithm

– Scale-space extrema detection– Keypoint localization– Orientation assignment– Keypoint descriptor

• Applications• Conclusions

2121

Keypoint Localization

• X is selected if it is larger or smaller than all 26 neighbors

2222

Accurate Keypoint Localization

• Reject the points that have low contrast (and are therefore sensitive to noise) or are poorly localized along and edge

233x189 original image

832 keypoints at extrema

729 keypoints after contrast filtering

536 keypoints after curvature filtering

2323

Outline

• Introduction• Algorithm

– Scale-space extrema detection– Keypoint localization– Orientation assignment– Keypoint descriptor

• Applications• Conclusions

24

Orientation Assignment

• By assigning a consistent orientation, the keypoint descriptor can be orientation invariant

Choose a region around the keypoint

Calculate the magnitude of the gradient

Calculate the magnitude of the gradient

Gaussian kernel

Orientation histogram

2 2

1

( , ) ( ( 1, ) ( 1, )) ( ( , 1) ( , 1))

( , ) tan (( ( , 1) ( , 1)) /( ( 1, ) ( 1, )))

m x y L x y L x y L x y L x y

x y L x y L x y L x y L x y

2525

Orientation Assignment: an Example (1)

2626

Orientation Assignment: an Example (2)

2727

Orientation Assignment: an Example (3)

2828

Orientation Assignment: an Example (4)

approximately 25 degrees80% of peak value

2929

Outline

• Introduction• Algorithm

– Scale-space extrema detection– Keypoint localization– Orientation assignment– Keypoint descriptor

• Applications• Conclusions

30

Descriptor

• It computes a descriptor in order to get highly distinctive description

• The descriptors are as invariant as possible to remaining variations

30

3131

Keypoint Descriptor (1)

• The best result: 8 orientations x 4x4 histogram array = 128 dimensions

3232

Keypoint Descriptor (2)

• The 128D vectors are normalized to unit length in order to reduces effect of contrast change

• Because of using gradients, the discriptor is invariant to the changes in illumination

3333

Outline

• Introduction• Algorithm• Applications• Conclusion

3434

Keypoint Matching

• The best candidate match for each keypoint is found by identifying its nearest neighbor in the database

• Using the ratio of distance (closest/next closest) chooses the correct matches– reject the distance ratio that is greater than

0.8

35

Results

3636

Object Recognition (1)

• It creates a Hough transform entry predicting the model location, orientation, and scale

• Solution for affine parameters:

1 2

3 4

x

y

tm mu x

tm mv y

1

2

3

4

0 0 1 0

0 0 0 1

x

y

m

mx yu

mx yv

m

t

t

Ax b

3737

Object Recognition (2)

• The LS solution can be determined by solving the corresponding normal equations

-1T Tx A A A b

3838

Recognition examples (1)

3939

Recognition examples (2)

4040

Outline

• Introduction• Algorithm• Applications• Conclusions

4141

Conclusions

• The SIFT keypoints are useful due to their distinctiveness

• This approach transforms an image into a large collection of local feature vectors

• It is invariant to image translation, scaling, and rotation, and partially invariant to illumination changes and affine or 3D projection