PERCEPTRON. Chapter 3: The Basic Neuron The structure of the brain can be viewed as a highly interconnected network of relatively simple processing.

PERCEPTRON

Chapter 3: The Basic Neuron

The structure of the brain can be viewed as a highly interconnected network of relatively simple processing elements/ neuron.

The brain has at least 1010 neurons, each connected to 104 others We are not attempting to build computer brains – extremely simplified

versions of natural neural systems- rather we are aiming to discover the properties of models.

The idea behind neural computing - by modeling the major features of the brain- can produce computers that exhibit many of the useful properties of the brain.

Whereas, we are concerned here with maybe a few hundred neurons at most, connected to a few thousand input lines

The aim of a model is to produce a simplified version of a system.

Biological Neural Networks

Neuron - three components: its dendrites, soma, and axon (Fig. 1.3).

Dendrites receive signals from other neurons. The signals are electric impulses that are transmitted

across a synaptic gap. The soma/ cell body, sums the incoming signals. When sufficient input is received, the cell fires (transmits a

signal to other cells.) However, the frequency of firing varies a- either greater or

lesser magnitude.

human brain…

The human brain contains about 10 billion nerve cells,

or neurons. On average, each neuron is connected to other neurons through about 10 000 synapses. (The actual

figures vary greatly, depending on the local

neuroanatomy.) The brain's network of neurons forms a

massively parallel information processing

system. This contrasts with conventional computers, in

which a single processor executes a single series of

instructions. http://www.idsia.ch/NNcourse/brain.html

…

the brain has quite remarkable capabilities: 1. its performance tends to degrade gracefully

under partial damage. In contrast, most programs and engineered systems are brittle: if you remove some arbitrary parts, very likely the whole will cease to function.

2. it can learn (reorganize itself) from experience. this means that partial recovery from damage is possible if healthy units can learn to take over the functions previously carried out by the damaged areas.

3. it performs massively parallel computations extremely efficiently. For example, complex visual perception occurs within less than 100 ms, that is, 10 processing steps!

4. it supports our intelligence and self-awareness. (Nobody knows yet how this occurs.)

As a discipline of Artificial Intelligence, Neural Networks attempt to bring computers a little closer to the brain's capabilities by imitating certain aspects of information processing in the brain, in a highly simplified way.

MODELLING THE SINGLE NEURON

The basic function of a biological neuron is to add up its inputs, and to produce an output if this sum is greater than some value, known as the threshold value.

The inputs to the neuron arrive along the dendrites, which are connected to the outputs from other neurons by specialized junctions called synapses.

The junctions pass a large signal across, whilst others are very poor. The cell body receives all inputs, and fires if the total input exceeds

the threshold. Our model of the neuron must capture these important features:

The output from a neuron is either on or off. The output depends only on the inputs. A certain number must be on

(threshold value) at any one time in order to make the neuron fire. The synapses can be modeled by having a multiplicative factor on the input.

MODELLING THE SINGLE NEURON

Artificial Neural Network (ANN) is an information processing system that have similar characteristic with biological neural network.

It has been developed as a general representation for human mathematical model.

With assumption

1. Information processing occurs within the simple element called neuron.

2. A neuron consist of a cell body, soma, fibres, dendrites and a long fibre called axon.

3. Signals are transferred within the neurons through connections.

4. Each connection has its weight.

5. Each neuron uses an activation function to produce output.

MODELLING THE SINGLE NEURON

BRAIN ANALOGY AND NN

Biological Neuron

LEARNING IN SIMPLE NEURONS

We need a mechanism for achieving learning in our model neuron. Connecting these neurons together then train them in order to do

useful task. Example in Classification problem: Figure 3.5 - Two groups - one group of several differently written A’s,

and the other of B’s, we may want our neuron to output a 1 when an A is presented and a 0 when it sees a B.

The guiding principle is to allow the neuron to learn from its mistakes:

LEARNING IN SIMPLE NEURONS

If it produces an incorrect output, we want to reduce the chances of that happening again; if it comes up with correct output, then we need do nothing.

If the neuron produces a 0 when we show it an A, then increase the weighted sum so that next time it will exceed the threshold and so produces the correct output 1.

If the neuron produces a 1 when we show it an B, then decrease the weighted sum so that next time it will less than threshold and so produces the correct output 0.

Learning strategy increase the weights on the active inputs when we want the

output to be active, decrease them when we want the output to be inactive.

To achieve - add the input values to the weights when we the output to be on, and subtracting the input values from the weights when we want the output to be off.

This defines our learning rule. This learning rule is a variant on that proposed in 1949 by Donald

Hebb, and is therefore called Hebbian learning. Since the learning is guided by knowing what we want to achieve,

it is known as supervised learning.

Learning strategy Our learning paradigm can be summarized as follows:

Set the weights and thresholds randomly Present an input

Calculate the actual output - thresholding the weighted sum of the inputs. (0 or 1)

Alter the weights to reinforce correct decisions – i.e, reduce the error.

The PerceptronThe Perceptron

The operation of Rosenblatt’s perceptron is based on the The operation of Rosenblatt’s perceptron is based on the McCulloch and Pitts neuron modelMcCulloch and Pitts neuron model. The model . The model consists of a linear combiner followed by a hard limiter. consists of a linear combiner followed by a hard limiter.

The weighted sum of the inputs is applied to the hard The weighted sum of the inputs is applied to the hard limiter, which produces an output equal to +1 if its input limiter, which produces an output equal to +1 if its input is positive and is positive and 1 if it is negative. 1 if it is negative.

The Perceptron

w1

1

X1

X2

Y

b

w2

Single-layer net for pattern classification

Output UnitInput Unit

The Perceptron Negative and Positive Response

The AlgorithmThe Algorithm

1 if y_in >

0 if - ≤ y_in <

-1 if y_in < -

Step 0: Initialize all weights and bias: wi = 0 (i= 1 to n), b=0Set learning rate (0 < ≤ 1) = 0

Step 1: While stopping condition is false, do steps 2-6.

Step2: For each training pair s:t, do steps 3-5Step 3. Set activations for input units:xi = si

Step 4.Compute response of output unit:

y_in = b + xi wi ;

Step 5. Update weights and bias if an error occurred for this pattern

If y t,wi(new) = wi(old) + txi b(new) = b(old) + t else wi(new) = wi(old)

b(new) = b(old)

Step 6. Test stopping condition:If no weights changed in Step

2, stop; else, continue

y =

The AlgorithmThe Algorithm

Step 0: Initialize all weights and bias: wi = 0 (i= 1 to n), b=0Set learning rate (0 < ≤ 1) = 0

The AlgorithmThe AlgorithmStep 1: While stopping condition is false, do steps 2-6.

Step2: For each training pair s:t, do steps 3-5

Step 3. Set activations for input units:xi = si

Step 4. Compute response of output unit:

NET = y_in = bb + + xi wi ;

OUT= y =

1 if y_in >

0 if - ≤ y_in <

-1 if y_in < -

The AlgorithmThe AlgorithmStep 5. Update weights and bias if an error occurred for this pattern

If y t,wi(new) = wi(old) + txi (i = 1 to n). b(new) = b(old) + t

else wi(new) = wi(old)

b(new) = b(old)

Step 6. Test stopping condition:If no weights changed in Step 2,

stop; else, continue

Input Target

(x1 x2 1)

(1 1 1)(1 0 1)(0 1 1)(0 0 1)

1-1-1-1

=1, = 0.2wi=0b=0

Input Net Out Target Weight Changes Weights

(x1 x2 1)

(1 1 1) (1 0 1) (0 1 1) (0 0 1)

0 0 12 1 -11 1 -1-1 -1 -1

( w1 w2 b)

(1 1 1) (-1 0 -1)(0 -1 -1)(0 0 -1)

(w1 w2 b)

(0 0 0)(1 1 1) (0 1 0) (0 0 -1)(0 0 -2)

Perceptron net for Perceptron net for AndAnd function: function: binarybinary inputs and inputs and bipolarbipolar targets targets

1st Epoch

Separating lines for 1Separating lines for 1stst training input training input

x2

x1--

- +

1 + x1(1)+ x2(1)=0.2 and1 + x1(1)+ x2(1)=-0.2

Formula asas lukis grafbb + + xi wi >

Separating lines for 2Separating lines for 2ndnd training input training input

x2

x1--

- +

0 + x1(0)+ x2(1)= 0.2 and0 + x1(0)+ x2(1)= -0.2

Formula asas lukis grafbb + + xi wi >

Separating lines for 3Separating lines for 3rdrd and 4 and 4thth training input training input

For 3rd input the weight derived is –ve

For the 4th input – no weight changes

Decision boundary is still not correct for 1st input

We are not finished training

Input Target

(x1 x2 1)

(1 1 1)(1 0 1)(0 1 1)(0 0 1)

1-1-1-1

=1, = 0.2wi=0b=0

Input Net Out Target Weight Changes Weights

(x1 x2 1)

(1 1 1) (1 0 1) (0 1 1) (0 0 1)

-1 -1 11 1 -10 0 -1-2 -1 -1

( w1 w2 b)

(1 1 1) (-1 0 -1)(0 -1 -1)(0 0 0)

(w1 w2 b)

(0 0 -2)(1 1 0) (0 1 -1) (0 0 -2)(0 0 -2)

Perceptron net for Perceptron net for AndAnd function: function: binarybinary inputs and inputs and bipolarbipolar targets targets

2nd Epoch

Separating lines for 1Separating lines for 1stst training input, 2 training input, 2ndnd epoch epoch

x2

x1--

- +

0 + x1(1)+ x2(1)= 0.2 and0 + x1(1)+ x2(1)= -0.2

Formula asas lukis grafbb + + xi wi >

Separating lines for 2Separating lines for 2ndnd training input 2 training input 2ndnd epoch epoch

x2

x1--

- +

-1 + x1(0)+ x2(1)= 0.2 and-1 + x1(0)+ x2(1)= -0.2

Formula asas lukis grafbb + + xi wi >

Input Target

(x1 x2 1)

(1 1 1)(1 0 1)(0 1 1)(0 0 1)

1-1-1-1

=1, = 0.2wi=0b=0

Input Net Out Target Weight Changes Weights

(x1 x2 1)

(1 1 1) (1 0 1) (0 1 1) (0 0 1)

-2 -1 10 0 -1-1 -1 -1-2 -1 -1

( w1 w2 b)

(1 1 1) (-1 0 -1)(0 0 0)(0 0 0)

(w1 w2 b)

(0 0 -2)(1 1 -1) (0 1 -2) (0 1 -2)(0 1 -2)

Perceptron net for Perceptron net for AndAnd function: function: binarybinary inputs and inputs and bipolarbipolar targets targets

3rd Epoch

Input Target

(x1 x2 1)

(1 1 1)(1 0 1)(0 1 1)(0 0 1)

1-1-1-1

=1, = 0.2wi=0b=0

Input Net Out Target Weight Changes Weights

(x1 x2 1)

(1 1 1) (1 0 1) (0 1 1) (0 0 1)

1 1 1-2 -1 -1-1 -1 -1-4 -1 -1

( w1 w2 b)

(0 0 0) (0 0 0)(0 0 0)(0 0 0)

(w1 w2 b)

(2 3 -4) (2 3 -4) (2 3 -4)(2 3 -4)

Perceptron net for Perceptron net for AndAnd function: function: binarybinary inputs and inputs and bipolarbipolar targets targets

10th Epoch

Separating lines for Final decision BoundariesSeparating lines for Final decision Boundariesx2

x1--

- +

-4 + 2x1+ 3x2> 0.2 and-4 + 2x1+ 3x2< -0.2Formula asas lukis graf

bb + + xi wi >

Input Net Out Target

(x1 x2 1)

(1 1 1)(1 -1 1)(-1 1 1)(-1 -1 1)

0 0 11 1 -12 1 -1-3 -1 -1

( w1 w2 b) (1 1 1) (-1 1 -1)(1 -1 -1)(0 0 0)

(w1 w2 b)(0 0 0 )(1 1 1) (0 2 0) (1 1 -1)(1 1 -1)

Input Net Out Target Weight Changes Weights

(x1 x2 1)

(1 1 1)(1 -1 1)(-1 1 1)(-1 -1 1)

1 1 1-1 -1 -1-1 -1 -1-3 -1 -1

( w1 w2 b)

(0 0 0) (0 0 0)(0 0 0)(0 0 0)

(w1 w2 b)

(1 1 -1) (1 1 -1) (1 1 -1)(1 1 -1)

Perceptron net for Perceptron net for AndAnd function: function: bipolarbipolar inputs and inputs and bipolarbipolar targets targets

1st and 2nd epoch=1, = 0.2wi=0b=0

WeightsWeight Changes

LIMITATIONS OF PERCEPTRONS

The perceptron is trying to find the straight line that separates classes.

It can separate classes that lie on either side of a straight line easily enough,

but there are many situations where the division between classes is much more complex. Consider the case of the exclusive-or (XOR) problem.

LIMITATIONS OF PERCEPTRONS

The XOR logic function has two inputs and one output

It produces an output as shown in table 3.1.

Such patterns are known as linearly inseparable since no straight line can divide them up.

The single-layer perceptron has shown great success for such a simple model.

Perceptron

Perceptron learning applet http://diwww.epfl.ch/mantra/tutorial/english/perceptron/html/