Multi-Valued Neurons and Multilayer Neural Network based on Multi- Valued Neurons MVN and MLMVN 1
Multi-Valued Neurons and Multilayer Neural Network based on Multi-Valued Neurons
MVN and MLMVN
1
A threshold function is a linearly separable function
1
1
-1
-1
f(1,1)= 1
f(1,-1)= -1
f(-1,1)= -1
f(-1,-1)= -1
f (x1, x2) is the OR function
Linear separability means that it is possible to separate “1”s and “-1”s by a hyperplane
2
Threshold Boolean Functions• The threshold (linearly separable) function can be
learned by a single neuron• The number of threshold functions is very small in
comparison to the number of all functions (104 of 256 for n=3, about 2000 of 65536 for n=4, etc.)
• Non-threshold (nonlinearly separable) functions can not be learned by a single neuron (Minsky-Papert, 1969), they can be learned only by a neural network
3
XOR – a classical non-threshold (non-linearly separable) function
1
1
-1
-1
f(1,1)=1
f(1,-1)= -1
f(-1,1)= -1
f(-1,-1)=1
Non-linear separability means that it is impossible to separate “1”s and “-1”s by a hyperplane
4
Multi-valued mappings
• The first artificial neurons could learn only Boolean functions.
• However, the Boolean functions can describe only very limited class of problems.
• Thus, the ability to learn and implement not only Boolean, but also multiple-valued and continuous functions is very important for solving pattern recognition, classification and approximation problems.
• This determines the importance of those neurons that can learn and implement multiple-valued and continuous mappings
5
Traditional approach to learn the multiple-valued mappings by a neuron:
•Sigmoid activation function (the most popular):
zezF
1
1)(
-1 1
0.5
6
Sigmoidal neurons: limitations
• Sigmoid activation function has a limited plasticity and a limited flexibility.
• Thus, to learn those functions whose behavior is quite different in comparison with the one of the sigmoid function, it is necessary to create a network, because a single sigmoidal neuron is not able to learn such functions.
7
Is it possible to overcome
the Minsky’s-Papert’s
limitation for the classical perceptron?
Yes !!!8
We can overcome the Minsky’s-Papert’s limitation using the complex-valued weights and the complex activation function
9
Is it possible to learn XOR and Parity n functions using a single neuron?
• Any classical monograph/text book on neural networks claims that to learn the XOR function a network from at least three neurons is needed.
• This is true for the real-valued neurons and real-valued neural networks.
• However, this is not true for the complex-valued neurons !!!
• A jump to the complex domain is a right way to overcome the Misky-Papert’s limitation and to learn multiple-valued and Boolean nonlinearly separable functions using a single neuron.
10
NEURAL NETWORKS
Traditional Neurons
Neuro-Fuzzy Networks
Complex-Valued Neurons
Generalizations of SigmoidalNeurons
Multi-Valued andUniversal Binary Neurons
Multi-Valued andUniversal Binary Neurons
11
Complex numbers
Unlike a real number, which is geometrically a point on a line, a complex number is a point on a plane.
Its coordinates are called a real (Re, horizontal) and an imaginary (Im, vertical) parts of the number
i is an imaginary unityr is the modulo (absolute value) of
the numberAlgebraic form of a complex number
r
12
Complex numbers
Trigonometric and exponential (Euler’s) forms of a complex number
A unit circle
φ is the argument (phase in terms of physics) of a complex number
13
Complex numbers
Complex-conjugated numbers
(cos sin )
(cos sin )
i
i
z x y re r
z x y re r
i i
i i
14
XOR problem
x1 x2 22110 xwxww
z
)(zPB f(x1, x2)
1 1 1+i 1 1
1 -1 1-i -1 -1
-1 1 -1+i -1 -1
-1 -1 -1-i 1 1
1BP
1BP
n=2, m=4 – four sectors
W=(0, 1, i) – the weighting vector
1
i
-i-1
1BP
1BP
15
Parity 3 problem
n=3, m=6 : 6 sectors
W=(0, ε, 1, 1) – the weighting vector
1
1-1
1
-1 -1
ε
1-1
1X 2X 3X 3322
110
xwxw
xwwZ
)(ZPB ),,( 321 xxxf
1 1 1 2 1 1 1 1 -1 -1 -1 1 -1 1 -1 -1 1 -1 -1 2 1 1 -1 1 1 2 -1 -1 -1 1 -1 1 1 -1 -1 1 1 1 -1 -1 -1 2 -1 -1
16
Multi-Valued Neuron (MVN)
• A Multi-Valued Neuron is a neural element with n inputs and one output lying on the unit circle, and with the complex-valued weights.
• The theoretical background behind the MVN is the Multiple-Valued (k-valued) Threshold Logic over the field of complex numbers
17
Multi-valued mappings and multiple-valued logic
• We traditionally use Boolean functions and Boolean (two-valued) logic, to present two-valued mappings:
• To present multi-valued mappings, we should use multiple-valued logic
1 1,..., 0,1 ; ,..., 0,1n nx x f x x
1 1,..., 1, 1 ; ,..., 1, 1n nx x f x x
18
Multiple-Valued Logic: classical view
• The values of multiple-valued (k-valued) logic are traditionally encoded by the integers {0,1, …, k-1}
• On the one hand, this approach looks natural.
• On the other hand, it presents only the quantitative properties, while it can not present the qualitative properties.
19
Multiple-Valued Logic: classical view
• For example, we need to present different colors in terms of multiple-valued logic. Let Red=0, Orange=1, Yellow=2, Green=3, etc.
• What does it mean? • Is it true that
Red<Orange<Yellow<Green ??!
20
Multiple-Valued (k-valued) logic over the field of complex numbers
• To represent and handle both the quantitative properties and the qualitative properties, it is possible to move to the field of complex numbers.
• In this case, the argument (phase) may be used to represent the quality and the amplitude may be used to represent the quantity
21
Multiple-Valued (k-valued) logic over the field of complex numbers
/ exp( 2 )i k
{0, 1, ..., 1}j k
exp( 2 / )jj i j k
primitive kth root of unity
regular values of k-valued logici
0
1
k-1
k-2
2
k-1
1one-to-one correspondence
0 2 1, , ,..., kj The kth roots of unity are values of k-valued logic over the field of complex numbers
22
Important advantage
• In multiple-valued logic over the field of complex numbers all values of this logic are algebraically (arithmetically) equitable: they are normalized and their absolute values are equal to 1
• In the example with the colors, in terms of multiple-valued logic over the field of complex numbers they are coded by the different phases. Hence, their quality is presented by the phase.
• Since the phase determines the corresponding frequency, this representation meats the physical nature of the colors.
23
Discrete-Valued (k-valued)Activation Function
= exp 2 / ,
if
( )
/ ( ) ( ) /
ji πj k
2 j k arg z 2π j +1 k
P z
i
0
1
k-2Z
j-1
Jj+1
k-1
Function P maps the complex plane into the set of the kth roots of unity
j
1j
24
Discrete-Valued (k-valued)Activation Function
k=16
25
Multi-Valued Neuron (MVN)
)...(),...,( 1101 nnn xwxwwPxxf f is a function of k-valued logic
(k-valued threshold function)
1x
nx
),...,( 1 nxxf . . .
P ( z )
nn xwxwwz ...110
26
MVN: main properties
• The key properties of MVN: – Complex-valued weights– The activation function is a function of the
argument of the weighted sum– Complex-valued inputs and output that are lying
on the unit circle (kth roots of unity)– Higher functionality than the one for the
traditional neurons (e.g., sigmoidal)– Simplicity of learning
27
MVN Learning
q s
q
•Learning is reduced to movement along the unit circle•No derivative is needed, learning is based on the error-correction rule
- error, which completely determines the weights adjustment
s- Desired output
- Actual output
i
qs
sε
qε
28
Learning Algorithm for the Discrete MVN with
the Error-Correction Learning Rule
X
q
W – weighting vector; X - input vector
is a complex conjugated to X
αr – learning rate (should be always equal to 1)
r - current iteration; r+1 – the next iteration
is a desired output (sector)
is an actual output (sector)
1 ( )( 1)
q srr+ r + ε - ε X
n+W W
i
qs
sε
qε
s29
Continuous-Valued Activation Function
Continuous-valued case (k):
Arg exp( (arg ))
(
)
/ | |
i zP z e
z
i z
z
Function P maps the complex plane into the unit circle
Z
/( |) |zP z z
i
1
30
Continuous-Valued Activation Function
31
Continuous-Valued Activation Function
32
Learning Algorithm for the Continuous MVN with
the Error Correction Learning Rule
XW – weighting vector; X - input vector
is a complex conjugated to X
αr – a learning rate (should be always equal to 1)
r - current iteration; r+1 – the next iterationZ – the weighted sum
1 ( 1) | |qr r
rr rW
z+ ε - XW
n+ z
i
|| z
zqε|| z
zq
qε is a desired output
is an actual output| |
z
z | |q z
z - neuron’s error
33
Learning Algorithm for the Continuous MVN with
the Error Correction Learning Rule
XW – weighting vector; X - input vector
is a complex conjugated to X
αr – a learning rate (should be always equal to 1)
r - current iteration; r+1 – the next iterationZ – the weighted sum
1 ( 1) | |qr r
rr rW
z+ ε - XW
n+ z
i
|| z
zqε|| z
zq
qε is a desired output
is an actual output| |
z
z | |q z
z - neuron’s error
34
A role of the factor 1/(n+1) in the Learning Rule
| |q z
z - neuron’s error
nnn xn
wwxn
wwn
ww)1(
~ ; ... ;)1(
~ ;)1(
~11100
0 1 1
0 1 1 1 1
0 1 1
0 1 1
...
( ) ( ) ... ( )( 1) ( 1) ( 1)
...( 1) ( 1) ( 1)
...
n n
n n
n n
n n
w w x w x
w w x x w x xn n n
w w x w xn n n
w x w zw x
z
The weights after the correction:
The weighted sum after the correction:
- exactly what we are looking for- exactly what we are looking for35
Self-Adaptation of the Learning Rate
-is the absolute value of the weighted sum on theprevious (rth) iteration.
rz
is a self-adaptive part of the learning rate1
rz
i
|z| <1 |z| >1
1
1
( 1) | |r+
qr rr
r r
zW W + ε - X
n+ z z
1/|zr| is a self-adaptive part of the learning rate
36
Modified Learning Rules with the Self-Adaptive Learning Rate
1 ( )( 1) | |
q srr r
r
W W + ε - ε Xn+ z
1/|zr| is a self-adaptive part of the learning rate
1 ( 1) | |qr
r rr
zW W + ε - X
n+ z z
Discrete MVN
Continuous MVN 37
Convergence of the learning algorithm
• It is proven that the MVN learning algorithm converges after not more than k! iterations for the k -valued activation function
• For the continuous MVN the learning algorithm converges with the precision λ after not more than (π/λ)! iterations because in this case it is reduced to learning in π/λ –valued logic.
38
MVN as a model of a biological neuron
No impulses Inhibition Zero frequency
Intermediate State Medium frequency
Excitation High frequency
The State of a biological neuron is determined by the frequency of the generated impulses
The amplitude of impulses is always a constant
39
MVN as a model of a biological neuron
40
MVN as a model of a biological neuron
Maximal inhibition
Maximal excitation
2π
0π
Intermediate State
Intermediate State
41
MVN as a model of a biological neuron
Maximal inhibition
Maximal excitation
2π
0π
42
MVN:
• Learns faster• Adapts better• Learns even highly nonlinear functions• Opens new very promising opportunities for the
network design• Is much closer to the biological neuron• Allows to use the Fourier Phase Spectrum as a
source of the features for solving different recognition/classification problems
• Allows to use hybrid (discrete/continuous) inputs/output
43