MVA '90 IAPR Workshop on Machine Vision Applicat ions Nov. 28-30,1990, Tokyo A Pattern Classifier Integrating Multilayer Perceptron and Error-Correcting Code Haibo L i , Torbjorn Kronander, and Ingemar Ingemarsson Department of Electrical Engineering, Linkisping University, S-58183 Linkoping, Sweden e-mail: [email protected], tobbe@ isy.liu.se Abstract In this paper we present a novel classifier which integrates a multilayer perceptron an d a error-correcting decoder. There are two stages in the classifier, in the first stage, mapping feature vectors from feature space to code space is achieved by a multilayer perceptron; in the second stage, error correcting decoding is done on code space, by which the index of the noisy codeword can be obtained. Hence we can get classifications of original feature vectors. The classifier ha s better classification performances than the conventional multilayer perceptrons. 1 Introduction Multilayer perceptrons trained with backpropagation[4] have been successfully applied in many areas[4][5][6][7], especially in pattern recognition[5]. A number of theoretical analyses have shown that any continues nonlinear mapping can be closely approximated using sigmoid nonlinearities and muitilayer perceptron that implies tnat arbitrary uecision regions can be formed by multilayer perceptrons. For some real-world problems, however, it is very difficult t o train a multilayer perceptron for forming mapping needed within given accurate, especially when the decision regions required are more complex and irregular, even if neural networks have more hidden layers and enough time be provided to train it. Meanwhile Error-Correcting Code theory has been well established a long time. The error correcting decoding can be viewed as a kind of classifying problem in which the noisy codewords are decoded into the correct codewords based on m inimum Hamming distance in code space. With minimum Hamming distance criterion, the decision regions in code space are very regular, for example are circle. The mapping from feature space to code space is easier achieved because such mapping is from region to region instead of from region to point. The classifier has better classification performancs than the conventional multilayer perceptrons. In this paper, we focus on how to form mapping from feature space to code space with a multilayer perceptron and how to seek good error correcting codes whose codeword dis- tributions are uniform and codeword distance is large. We will below first describe the pattern Classifier architecture. The n we discuss how to train a multilayer perceptron and outline the error-correcting code. We conclude with remarks on our pattern Classifier and future work. 2 Classi fier A rchitecture Before we describe our new classifier, let's first examine the pattern classifier problem.
5
Embed
Haibo Li, Torbjorn Kronander and Ingemar Ingemarsson- A Pattern Classifier Integrating Multilayer Perceptron and Error-Correcting Code
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
8/3/2019 Haibo Li, Torbjorn Kronander and Ingemar Ingemarsson- A Pattern Classifier Integrating Multilayer Perceptron and Er…
MVA '90 IAPR Workshop on Machine Vision Applications Nov. 28-30,1990, Tokyo
A Patte rn Classifier Integra ting Multilayer Perceptron
and Error-Correcting Code
Haibo Li, Torbjorn Kronander, and Ingemar IngemarssonDepartment of Electrical Engineering,
Linkisping University, S-58183 Linkoping, Sweden
e-mail: Haibo@ isy.liu.se, tobbe@ isy.liu.se
A b s t r a c t
In this paper we present a novel classifier which inte grates a multilayer perceptron
and a error-correcting decoder. There are two stages in th e classifier, in th e first stage,mapping feature vectors from feature space to code space is achieved by a multilayerperceptron; in the second stage, error correcting decoding is done on code space, bywhich the index of the noisy codeword can be obta ined . Hence we can get classificationsof original feature vectors. Th e classifier has better classification performances thanthe conventional multilayer perceptrons.
1 Introduction
Multilayer perceptrons trained with backpropagation[4] have been successfully applied in
m an y areas[4][5][6][7], especia lly in p at te rn recognition[5]. A num ber of theoretical analyses
have shown th at any continues nonlinear m apping can be closely approxim ated using sigmoidnonlinearities and muitilayer perceptron that implies tnat arbitrary uecision regions can be
formed by m ultilayer pe rceptrons. For some real-world problem s, however, it is very difficult
t o train a multilayer perceptron for forming map ping needed within given accurate, especially
when the decision regions required are more complex and irregular, even if neural networks
have more h idden layers and enough t ime b e provided t o t ra in i t .Meanwhile Error-Correcting Code theory has been well established a long t ime . T he
error correcting decoding can be viewed as a kind of classifying problem in which the noisy
codewords are decoded in to th e correct codewords based on m inimum Hamm ing d is tance in
code space. W ith minim um H amm ing distance criterion, the decision regions in code space
are very regular , for example are c ircle . T he mapping f rom feature space to code space
is easier achieved because su ch map ping is from region t o region instea d of from region t opoint. The classif ier has better classif ication performancs than the conventional multilayer
perceptrons.
In th is paper , we focus on how t o form mapping f rom feature space t o code space with
a m ultilayer perceptron and how to seek good error correcting codes whose codeword dis-
tr ibu tion s are uniform a nd codeword distance is large.
We will below first describe the pat ter n Classifier arch itec ture . T he n we discuss howt o train a multilayer perceptron a nd outline the error-correcting code. We conclude with
remarks on our pattern Classif ier and future work.
2 Classifier Architecture
Before we describe our new classifier, let's first examine the pattern classifier problem.
8/3/2019 Haibo Li, Torbjorn Kronander and Ingemar Ingemarsson- A Pattern Classifier Integrating Multilayer Perceptron and Er…
For an N-dimensional feature space BN, assuming there are M categories pattern in
th is space, th is is, th is feature space can be par t i t ioned in to M cel ls Ci, i = 1 , 2 , ....,M a n d
associates t o each cell Ci a exem plar vector 5.For a given feature vector f E B N , we say i t belongs t o i th category Ci, if it meets
following conditio n
where 1.1 denote s some kind of distanc e measure in B ~ ,amming distance is used here.
This pattern classif ication problem can be solved by a multilayer perceptron with the
backpropagation algorithm a s its learning algorithm[5]. T he inpu t of the m ultilayer per-
ceptron is 2 € B N , t h e o u t pu t i s ?. Supposing the desired exemplar is d: then the er ror
function can be defined as following
then the weights of the multilayer perceptron are changed in the sequence by an amountpropor t ional t o th e par t ia l der ivat ives of E with respect to t he weights until a error principle
is met . T he general er ror pr incip le is
E < E (3 )
where 6 is a fault tolerant threshold.
For our pattern classif ication problem, Band ? are b inary vector , so the c used here is
less than the m inim um H amm ing distance[l][2] . T he desired exem plar vector B is generally
given with th e miilimum Ha mm ing distance 1. If the minimum Ham ming distance is great
tha n 1, th e classif ication results will be in confusion provided t ha t an y postprocessing is not
used for the results . T h e min imum H amm ing distance is less th an 1 which means e < 1, a
pattern classif ication with this method can be viewed as a kind mapping from a region to
a point. This may be the reason why the multilayer perceptrons are especially diff icult totrain for some reai-world problems.
Based on the above analysis, we present a novel pattern classifier which integrates a mul-
tilayer percep tron and a error-correcting decode. T h e idea behind t he classif ier is achieving
classif ication th roug h, f irst, region space mapping, th at is , map ping from a region of feature
space to a region of code space by a multilayer perceptron, then error-correcting decoding
is done in t he code space, we can obt ain the index of classification. T h e m ain difference be-
tween t he new classifier and multilayer perceptron is map ping from region t o region instead
of ma ppin g from region t o point in ou r new classifier.
For a pattern classification problem with M categories, we can find a set error-correcting
code with M codewords , D , , = 1 , 2 ,....,M , whose word length are L when the minimum
Hamm ing d is tance H,jn is predetermined. T he wordlength L is determined by M and
Hmin .
T h e desired signals used for trai ning classifier are codewords D i , i = 1 , 2 , ....,M , h e error
function is defined as follows
E = ID -P I (4 )
where ? is the o utp ut of th e mult ilayer perceptron .
T h e error principle used here is also described by (3 ) , bu t we can f ind th at th e minimum
Ham ming d is tance is f ree to choose, i t can be great than 1. Clear ly, i t is eas ier to t ra in the
mult i layer perceptron th an the one without mapping in code space. Because i t is eas ier to
meet the er ror pr inciple , i t no t only improve the acc urate but speed th e t ra in ing rate .
Af ter th e p at terns are mapped on to code space, we can make er ror-correct ing decoding inth e code space, we can obtain th e index of patterns. T h e process of a pat ter n classification
by the classifier is shown in Fig.1.
T h e overall classifier stru ctu re consists of two levels: the first level maps pattern vectors
from fea ture space on to code space by a multilayer perceptron; the second level is a error
8/3/2019 Haibo Li, Torbjorn Kronander and Ingemar Ingemarsson- A Pattern Classifier Integrating Multilayer Perceptron and Er…
the m ul ti layer percept ron i s trained by t he backpropagat ion algor i thm. W hen t he maxi-
m um train e rror is less than the minim um Ham ming distance, the t raining of th e mult ilayer
percep tron finished.
If the exemplars, fi of pat tern vectors are known, a bet ter t rain procedure be designed
as follows
1) th e mult i layer perceptron is f i rst t rained w ith inp ut f;2) after the network is stable, the input 2 are chosen in l ine according t o th e sh ortest
distance t o their corresponding exemplars ste p by step for t raining th e multi layer pe rceptron.
Th is kind training procedure similar to th e cooling procedure used in th e simulated
annea ling can avoid sinking in local minimum pitfall . Hence a b et ter map ping performances
can be obtained.
4 Error-correcting code
Error-corre cting c ode has been found in a wide variety of applications, including da ta trans-
mission over a com mun ication chan nel, codes for high-speed a nd m ass memories of com puter
ect .
T h e error-correc ting code we need should mee t th e following properties 1) the distance
between each two codewords mu st be the sam e; 2) the distance between each two codewords
must be as large as possible.
Such codes as IIadam ard matrix codes can be found in [2], due t o the l imitat ions of
space , we don' t discuss this content here.
5 Conclusion remarks
Integrat ing error-correct ing code, the mult ilayer perceptron can give a bet te r performance
than wi thout such codes . Of course, the improvement in th e performance is at th e cost of the
complex system st ructure. T he m ore the er ror we want to lerant , the larger the codewordsare, and t he m ore complex the system is. How to choose the trade-off between the complex
an d fault tolerance is difference from case to case. Bu t we can say error-correcting code has
a wide application foreground in neural networks. We are in the beginning of applying error-
correcting codes for neural networks, we will work more with such object, since integrating
neural networks an d error-correcting codes seems to promise a good pa th in this aspect .
References[I] R.Blahut , Theory and Pract ice of Error Control Codes, 1983
[2] F.Macwill iams and N .Sloane, T he Theo ry of E rror-correcting Code s, 1988
[3] T.C hiue h and R.G oodm an, A Neural Network Classif ier Based on Coding Theory, AIP