Widder

7/27/2019 Widder

1/20

Automatic License Plate Location UsingTemplate Matching

University of Wisconsin - Madison

ECE 533 Image ProcessingFall 2004

Kerry Widder

7/27/2019 Widder

2/20

Introduction

The task of recognizing some object in an image is one of the mostdifficult in the field of digital image processing. Many factors contribute to thisdifficulty, including variations in lighting and contrast, differences in orientation or

size, variations in the surroundings of the object and other objects covering partof the object. This is also true for the specific recognition task of automaticallyfinding a license plate in the image of a vehicle. The color of the vehicle may bethe same as the plate, making the boundary between the plate and the vehicledifficult to detect. The vehicle may have other rectangular shapes such as lights,grills or bumpers that make it harder to distinguish the license from other parts ofthe vehicle. The plate may also be surrounded by a holder that covers part of itand alters its proportions. The plates vary greatly in size, color and layout fromcountry to country. Some have extra symbols or pictures in the background thatmake it harder to identify the characters on the plate. The quantity of thecharacters may vary greatly, and some have all letters, others have all numerals,

and the rest have some mixture of the two.

The automatic identification of license plates provides benefits in manydifferent applications. Tollbooth violators can be photographed and a ticketissued automatically based on the image of the vehicles license. Lawenforcement officials can use it to identify speed limit violators based on imagestaken by cameras mounted in remote locations, or to automatically catch vehiclesthat run a red light. The operators of a parking structure can take an image of avehicle when it enters and again at exit and correlate the ticket number with thevehicle to prevent fraud.

Human operators reading the images and taking the appropriate actions

could handle these applications. However, many organizations dont haveenough personnel to handle these tasks manually. It is also more efficient andless costly to handle these tasks automatically. These factors make theautomatic detection of license plates a topic that has generated a lot of interest.This project will attempt to find the location of the license plates in images ofvehicles using template matching.

Approach

The task of automatically locating license plates in images has beenapproached in many different ways. One approach utilizes distance sets, a formof local descriptor that uses the spatial arrangement of features to identify anobject [1]. Another uses morphological operations, such as smoothing, closing,opening and threshold, to attempt to isolate the features of a license plate [2].The high contrast between the characters and the plate is exploited in anothermethod that searches for the greatest gray-level contrast in an area [3]. Anothermethod finds rows in the image with the most light to dark transitions, then

2

7/27/2019 Widder

3/20

examines the columns in each row for the highest concentration of dark pixels[4]. Genetic algorithms have also been used in the decision making process ofidentifying the license plates [5]. This project will explore two mechanismsutilizing template matching, correlation and moment invariants.

The correlation between two functions (or images) f(x,y) and h(x,y) is

defined as

),(),(*1

),(),(1

0

1

0

nymxhnmfMN

yxhyxfM

m

N

n

++=

=

=

(1)

where f*denotes the complex conjugate off[6]. Since the image functions dealtwith in this project are real functions, f*= f. The results of applying the correlationfunction can be displayed as an image the size of f(assuming the size offisgreater than or equal to the size ofh) with brighter pixels indicating a closermatch between fand h at those points. Ifh is a template of the object to befound in image f, then applying the correlation function between fand h will resultin image g(x,y), where the brightest spot will indicate the best match in image foftemplate h. Thus, correlation provides a simple way to search for the presence ofa given object in an image. A sample license plate will be isolated from its vehicleimage and used as the template. Each vehicle image will be correlated with thetemplate to attempt to find the license in the image.

The moment invariants of an image are statistical descriptors of the imagethat are invariant to translation, rotation and scale changes [6]. A set of seven ofthem is defined by

02201 += (2)

( ) 211

2

022024 += (3)

( ) ( ) 20321

2

1230333 += (4)

( ) ( ) 20321

2

12304 +++= (5)

( ) ( ) ( ) ( )

( ) ( ) ( ) ( )[ ]20321

2

123003210321

2

0321

2

1230123012305

33

33

+++

++++=(6)

( ) ( ) ( )

( ) ( )0321123011

2

0321

2

123002206

4

++

+++=(7)

( ) ( ) ( ) ( )( ) ( ) ( ) ( )[ ]2

0321

2

123003213012

2

0321

2

1230123003217

33

33

+++

++++=(8)

where

00

pq

pq= (9)

3

7/27/2019 Widder

4/20

and

12

++

=qp

(10)

for p + q = 2,3,.

and where

( ) ( ) ( ) =x y

qp

pq yxfyyxx , (11)

and

00

10

m

mx = and

00

01

m

my = (12)

and

( )

= dxdyyxfyxm qppq , (13)

Since images of vehicle license plates will be of varying size andorientation (unless the images are taken in a very controlled setting), theimmunity of these descriptors to variations in orientation and scale makes theman attractive candidate for this task. The moment invariants of the template willbe calculated and then compared with the calculated moment invariants of eachobject in the image. The Euclidean distance, defined as

( ) jj mxxD = for j = 1, 2,, W (14)

(where W is the number of objects) will be used as the measure of the closenessof the match between each object and the template, with the minimum valueconsidered to be the best match.

Each image must be segmented into objects before the moment invariantscan be calculated. The segmentation will be performed by first finding the edgesusing a Sobel operator, defined by the masks

-1 -2 -1

0 0 0

1 2 1

-1 0 1

-2 0 2

-1 0 1

Figure 1 Sobel operator masks.

and then calculating the exterior boundaries of each object in the resulting edgeimage. The boundaries will then be processed to define a sub-image enclosingeach object. The moment invariants will be calculated for each sub-image andcompared to the template. The algorithm will propose the boundary image with thesmallest distance to the template as the license plate match.

4

7/27/2019 Widder

5/20

Work Performed

A set of ten vehicle images was collected. The images were 640 x 480pixels. Care was taken to keep the license plate size relatively consistent to keep

the task more manageable. One of the sample images is shown in figure 2. Theimages were converted from RGB to gray scale. One of the license plates wasisolated, cropped and stored for use as the template, as seen in figure 3.

Figure 2 Sample vehicle image.

(a) (b)

Figure 3 a) License plate template, b) license plate correlation template

5

7/27/2019 Widder

6/20

The image processing was done using MATLAB [7]. Several MATLABfunctions were used that came from the book Digital Image Processing UsingMATLAB (DIPUM) [8]. The high-level algorithm is given in figure 4. A sample ofthe output is shown in figure 5.

________________________________________________________

Load template and vehicle image

Create Gaussian filters for template and image

Perform Gaussian filtering on template and image

Perform Sobel filtering on filtered image and template to find edges

Perform correlation between template and image using edge images

Load correlation template

Perform correlation between correlation template and correlation image

Locate both correlation results in copy of original image and store

Calculate moment invariants of original template

Convert Sobel of image to binary for boundary location

Perform opening to reduce stray protrusions

Locate boundaries of objects in the binary image

Process each boundary to find rectangle enclosing it

For each such image, calculate moment invariants and store

Calculate norm with template moment invariants and store

Find best matchCreate sub-image of best match, store and display

_________________________________________________________

Figure 4 License Plate Recognition Algorithm

6

7/27/2019 Widder

7/20

(a) (b)

(c) (d)

(e) (f)

(g)

7

7/27/2019 Widder

8/20

Figure 5 Results for one vehicle: a) Original image, b) Results of Sobel operator, c) Correlationwith template, d) Correlation on correlation, e) Results of correlation methods, f) Image ofcalculated boundaries, g) Result of moment invariant method.

Results

The template matching by correlation method was successful for six of theten vehicle images (60%), with one other that was close and three that completelymissed. The correlation images had a distinctive cross-shaped pattern where thelicense plate was (see fig. 5(c)). In an effort to capitalize on that and to try to getbetter results, a template was created from the correlation of the original templatewith the image it was taken from, and this correlation template was then used asthe template for doing a correlation between it and each correlation result(correlation squared). The results of this correlation on the correlation were not asgood as the original correlation results, with success for only two images and threeothers that were close. A successful correlation is shown in figure 6, where thesquare target shows the upper left corner of the plate as determined by thecorrelation on correlation and the rectangular target shows the upper left corner ofthe plate as determined by the correlation results. The grill and headlight area ofthis vehicle yields high readings, but the plate provided a better match in this case.Two unsuccessful results are shown in figure 7. The first example shows highervalues for the grill area than the license plate and the distinctive cross-shapedpattern is not evident by the license plate. In the second example, the cross-shaped pattern is there, but the grill and headlight area yields higher readings.

(a)

8

7/27/2019 Widder

9/20

(b) (c)

Figure 6 Successful correlation result: a) original image with correlation matches indicated, b)correlation image, c) correlation on correlation image.

(a) (b)

(c) (d)

Figure 7 Unsuccessful correlation results: a) & c) original images with correlation matchesindicated, b) & d) correlation images.

9

7/27/2019 Widder

10/20

The moment invariant method resulted in the successful location of thelicense plate for only two of ten images (20%). Six of the eight failures were due toproblems in getting a good boundary of the plate, the other two had enough intactboundary to select the plate as one of the objects, but the moment invariantmeasure failed to select the plates. The comparison to the template is done by

taking the minimum and maximum x and y values of each boundary and usingthem to define a sub-image of the original image. These sub-images are thencompared to the template. Therefore, if at least two full connected edges of a plateresult from the boundary selection process, the plate will be one of the candidateobjects.The first set of images in figure 8 shows a successful match. The plateboundary is almost complete. This ensures that the plate will be one of the objectscompared to the template. The second set shows an image where the boundaryimage has two complete, connected sides of the plate, which will result in thecomplete license plate being an object to be compared with the template.However, the comparison failed to correctly select the license plate. The third setof images shows a case where the boundary of the plate is insufficient to select the

plate as an object. In this case, the largest intact section of the plate boundary isone of the vertical edges. In the preliminary work on this project, this image hadsignificant protrusions off the boundary of the license plate. Adding themorphological opening step in the process removed these protrusions in this case,but resulted in the boundary of the plate being lost. The fourth set shows a casewhere the boundary of the plate has a significant protrusion caused by an edge ofthe bumper that blends into the plate. This protrusion will result in the license platebeing part of a larger object for comparison with the template. Even if this regionhad the closest match, it still would not be an exact match because of the extraarea selected with the license plate. Finally, an example of a license holdercomplicating the process is seen in the final set of images. The holder in this image

has letters at the top and blocks the bottom of the plate. The region selected by theboundary will include the letters on the top, but not the bottom of the holder. Thisregion was not successfully matched to the template.

(a) (b)

10

7/27/2019 Widder

11/20

(c) (d)

(e) (f)

(g) (h)

11

7/27/2019 Widder

12/20

(i) (j)

(k) (l) (m) (n) (o)

Figure 8 Moment invariant results: a), c), e), g) & i) Original images, b), d), f), h) & j) boundaryimages illustrating various conditions, k) o) best match assigned by the algorithm for eachrespective image.

Discussion

The correlation method of template matching was moderately successful,

but not reliable enough for commercial use. One difficulty with this method iscreating a generic template. This work just used a sample plate as the template.Careful creation of the template may produce better results and would be an areafor further study. This method also would probably be less successful if the imageshad greater variation in size or orientation of the plates. In general, correlationdoes not seem to be a good candidate for a robust license plate location method.

In the moment invariant method of template matching, the identification ofthe object boundaries proved to be troublesome. Initially, some of the edge imageshad protrusions off the plates caused by nearby features of the vehicles, as seen infigure 8(h). The morphological opening step was added to try to eliminate theseprotrusions. This added step was partially successful at doing so, but it had the

side affect of also sometimes chopping up the boundary of the license plate andmaking it impossible to select it as a region of interest for the moment invarianttest, as seen in figure 8(f). Also, variations in contrast levels and the sharpness ofthe edges in the images made the task of applying a threshold to the edge imageto convert it to a binary image difficult to do reliably for all cases. If the threshold istoo low, there will be too many edges and the boundaries will run together, makingthe object separation difficult. If the threshold is too high, important features maybe missing and the object of interest may not be identifiable.

12

7/27/2019 Widder

13/20

The sample size for this project was not very big. To get numbers that wouldbe statistically meaningful, more data would need to be collected. Further workcould be done in several areas. The refinement of the template could have animpact on the success of these methods. The use of other methods of edgedetection could be explored, which could especially benefit the moment invariant

method, where difficulties were encountered in isolating the boundaries of thelicense plates.

The many variations in the images make the location task difficult to achievewith simple template matching. To get better results, some type of adaptivemethod may be necessary to handle the wide variations in vehicle images.

13

7/27/2019 Widder

14/20

References

[1] C. Grigorescu and N. Petkov, Distance Sets for Shape Filters and Shape

Recognition, IEEE Trans. Image Processing, vol. 12, pp. 1274-1286, Oct.2003.

[2] J. Hsieh, S. Yu and Y. Chen, Morphology-based license plate detection inimages of differently illuminated and oriented cars, J. Electronic Imaging, vol.11(4), pp. 507-516, Oct. 2002.

[3] T. Kula and J. Cicero, Image Processing Experiments, in Proc. 21stSIGCSETech. Symp. On Computer Science Education, Washington, D.C., UnitedStates, 1990, pp.167-170.

[4] N. Ketelaars, Final Project: Automated License Plate Recognition,AIMeMagazine, vol. 1, pp. 9-12, 2002.

[5] J. Parker and P. Federl, An Approach to License Plate Recognition,Computer Science Technical Reports, Dept. Computer Science, University ofCalgary, Calgary, Alberta Canada, Oct. 1, 1996.

[6] R. Gonzalez and R. Woods, Digital Image Processing, 2nd ed., Prentice-Hall,New Jersey, 2001.

[7] MATLAB 7.0, The Mathworks, Inc., Natick, MA, 2004.

[8] R. Gonzalez, R. Woods and S. Eddins, Digital Image Processing UsingMATLAB, Pearson Education, Inc., New Jersey, 2004.

14

7/27/2019 Widder

15/20

Appendix

MATLAB m-file listing

%UW Madison Dept. of Electrical and Computer Engineering

%ECE-533 Image Processing

%Fall 2004 Project

%Kerry Widder

%

%License Plate Recognition

%

%This program attempts to find a license plate in the image of a

%vehicle through two approaches:

% 1. Correlation between a license template and the image.

% 2. Create a binary image from the Sobel filtered image using a

% threshold value, calculate the boundaries of the

% objects in the image, calculate the invariant moments of each

% object's area in the image and compare to the invariant

% moments of the template, using the smallest difference as

% the determination of the best match.

%

%This program utilizes Matlab 7.0. Some of the functions used came%from the book "Digital Image Processing Using Matlab", Gonzalez,

%Woods, Eddins, Pearson Education, Inc., 2004. These functions are available

%from the book's web site in the form of p-files. They will be

%identified when first used by a comment '%DIPUM'.

%

%The program assumes a gray-scale image file 'car_input.bmp' and

% a template license plate file 'template.bmp' and a correlation

% template file 'corr_template.bmp' residing in the

% working directory of the Matlab session. The output will be the

%following files:

% template_gaus.bmp - gaussian LP filtered template

% template_sobel.bmp - Sobel filtered template for edge detection

% car_gaus.bmp - gaussian LP filtered image

% car_sobel.bmp - Sobel filtered image for edge detection

15

7/27/2019 Widder

16/20

% car_correl.bmp - correlation of image with template

% car_corr2.bmp - correlation of correlation image with template

% car_corresult.bmp - image showing correlation matches

% boundaries.bmp - image showing the calculated boundaries

% lic_match.bmp - image showing the object in the image that% has the best match to the invariant moments

% of the template

clear all

% load image

tem=imread('template.bmp');

fim=imread('car_input.bmp');

%

[m,n]=size(fim);

mp=2*m;

np=2*n;

Do=150; %cutoff for gaussian filter - image

[tm,tn]=size(tem);

tmp=2*tm;

tnp=2*tn;

Dot=15; %cutoff for gaussian filter - template

%

% perform gaussian low pass filtering to reduce detail of image

%lpfilter() from DIPUM

Hgaus=lpfilter('gaussian',mp,np,Do);

%dftfilt() from DIPUM

fimg1=dftfilt(im2double(fim),Hgaus);

fimg=im2uint8(mat2gray(fimg1));

imwrite(fimg,'car_gaus.bmp');

%

%perform gaussian low pass filtering to reduce detail of template

Hgaust=lpfilter('gaussian',tmp,tnp,Dot);

temg1=dftfilt(im2double(tem),Hgaust);

temg=im2uint8(mat2gray(temg1));

imwrite(temg,'template_gaus.bmp');

%

% perform sobel filtering to find edges of image

16

7/27/2019 Widder

17/20

sobel1=[-1,-2,-1;0,0,0;1,2,1];

sobel2=[-1,0,1;-2,0,2;-1,0,1];

fsob1=imfilter(fimg1,sobel1,'conv','symmetric');

fsob2=imfilter(fimg1,sobel2,'conv','symmetric');

f2sob=fsob1.^2+(fsob2.^2);fimsob=sqrt(f2sob);

fimsobel=im2uint8(mat2gray(fimsob));

figure(1),clf,colormap('gray');

subplot(2,3,1);

imshow(fimsobel);

imwrite(fimsobel,'car_sobel.bmp');

%

% perform sobel filtering to find edges of template

tsob1=imfilter(temg1,sobel1,'conv','symmetric');

tsob2=imfilter(temg1,sobel2,'conv','symmetric');

t2sob=tsob1.^2+(tsob2.^2);

temsob=sqrt(t2sob);

temsobel=im2uint8(mat2gray(temsob));

imwrite(temsobel,'template_sobel.bmp');

%

%perform correlation between the image and the template

%dftcorr() from DIPUM

fcor=dftcorr(fimsob,temsob);

fcorr=im2uint8(mat2gray(fcor));

subplot(2,3,2);

imshow(fcorr);

imwrite(fcorr,'car_correl.bmp');

%%

%%perform correlation between the correlation and the correlation template

%%load correlation template

temcor=imread('corr_template.bmp');

fcor2=dftcorr(fcor,im2double(temcor));

fcorr2=im2uint8(mat2gray(fcor2));

subplot(2,3,3);

imshow(fcorr2);

imwrite(fcorr2,'car_corr2.bmp');

%%

17

7/27/2019 Widder

18/20

%find the pixel of highest intensity in the correlation

%and mark that spot in a new copy of the image -

%will be the top left corner of the plate

%

%find maximum intensity location%vector of max value in each column

Colmax=max(fcor);

%Col will be index of column containing max value

[Max,Col1]=max(Colmax);

%create vector containing column with max value

Maxcol=fcor(:,Col1);

%Row is row index of max value

[Max2,Row1]=max(Maxcol);

%translate

Row=Row1;

Col=Col1;

%create 'target' to identify this location in a copy of the image

% This target is a 6 x 20 pixel rectangle with the top right quarter and

% the bottom left quarter white, the rest black

target=[0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,255,255,255,255,255;

0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,255,255,255,255,255;

0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,255,255,255,255,255;

255,255,255,255,255,255,255,255,255,255,0,0,0,0,0,0,0,0,0,0;

255,255,255,255,255,255,255,255,255,255,0,0,0,0,0,0,0,0,0,0;

255,255,255,255,255,255,255,255,255,255,0,0,0,0,0,0,0,0,0,0];

%%

%find the pixel of highest intensity in the correlation of correlation

%and mark that spot in a new copy of the image - must translate by 1/2

%the size of the correlation template, which is 59,123

%

%find maximum intensity location

%vector of max value in each column

Colmax=max(fcor2);

%Col will be index of column containing max value

[Max,Col1]=max(Colmax);

%create vector containing column with max value

Maxcol=fcor2(:,Col1);

18

7/27/2019 Widder

19/20

%Row is row index of max value

[Max2,Row1]=max(Maxcol);

%translate

Row2=Row1+59;

Col2=Col1+123;%create 'target2' to identify this location in a copy of the image

% This target is a 10 x 10 pixel rectangle with the top left quarter and

% the bottom right quarter white, the rest black

target2=[255,255,255,255,255,0,0,0,0,0;255,255,255,255,255,0,0,0,0,0;

255,255,255,255,255,0,0,0,0,0;255,255,255,255,255,0,0,0,0,0;

255,255,255,255,255,0,0,0,0,0;0,0,0,0,0,255,255,255,255,255;

0,0,0,0,0,255,255,255,255,255;0,0,0,0,0,255,255,255,255,255;

0,0,0,0,0,255,255,255,255,255;0,0,0,0,0,255,255,255,255,255];

%create copy of image and insert targets

fimcor=fim;

fimcor(Row-3:Row+2,Col-10:Col+9)=target;

fimcor(Row2-5:Row2+4,Col2-5:Col2+4)=target2;

subplot(2,3,4);

imshow(fimcor);

imwrite(fimcor,'car_corresult.bmp');

%

%calculate moment invariants of template

tem2=im2double(tem);

%invmoments() from DIPUM

temp_phi=abs(invmoments(tem2));

%

%convert sobel of input to binary for boundary location - use a

% threshold of 0.25 (corresponding to 64 for 8-bit)

fbin=im2bw(fimsobel,0.25);

%

%perform erosion, then dilation to try to eliminate protrusions from

%the license plate

SE=[0,1,0;1,1,1;0,1,0];

fbinopen=imopen(fbin,SE);

%

%find boundaries of objects in image

%boundaries() from DIPUM

19

7/27/2019 Widder

20/20

B=boundaries(fbinopen);

%

%create image of all boundaries

b=cat(1,B{:});

%bound2im() from DIPUMbim=bound2im(b);

bims=im2uint8(bim);

imwrite(bims,'boundaries.bmp');

subplot(2,3,5);

imshow(bims,[]);

%

%find location of rectangles containing each boundary

NumB=size(B);

for i=1:NumB

%find and store the max and min for x and y of each boundary

rcmax(i,:)=max(B{i});

rcmin(i,:)=min(B{i});

%create image containing boundary

fsub=fim(rcmin(i,1):rcmax(i,1),rcmin(i,2):rcmax(i,2));

%calculate and store the moment invariants for subimage

fsub2=im2double(fsub);

fsub_phi(i,:)=abs(invmoments(fsub2));

%compare to template values and store difference

fsubdiff(i)=norm(fsub_phi(i,:)-temp_phi);

end

%

%find best match

[C,I]=min(fsubdiff);

%

%create subimage of best match

fimloc=fim(rcmin(I,1):rcmax(I,1),rcmin(I,2):rcmax(I,2));

%save and display image

imwrite(fimloc,'lic_match.bmp');

subplot(2,3,6);

imshow(fimloc,[]);

20