Neural networks 1.ppt - FITstaff.fit.ac.cy/com.ke/files/acoe402/Neural_networks_1.pdf · ACOE 402 Neural Networks and Fuzzy Logic Artificial Neural Networks Efthyvoulos C. Kyriacou

ACOE 402 Neural Networks and Fuzzy LogicACOE 402 Neural Networks and Fuzzy LogicArtificial Neural Networks

Efthyvoulos C. Kyriacou (PhD) Assoc. Prof. 

Computer Science and Engineering Department

Resources:  Dr. Neocleous Costas, Cyprus Univ. of Technology 

Neural Networks, S. Haykin, Prentice hall 1999Fundamentals of Neural Networks, L. Fausett, Prentice hall 1994

Computational Intelligence (CI) is a special case of A ifi i l I lli (AI)

What is intelligence?

Artificial Intelligence (AI)

What is intelligence?

It is difficult to be clearly definedIt is difficult to be clearly defined.

It may not be needed to be defined to be understoodIt may not be needed to be defined to be understood.

Terman* suggests that it is  “the capacity for abstract thought”.gg p y g

Most psychologists though, agree that it is “the capacity for effective adaptation to an environment, which is done throughchange in the organism, or change in the environment, or even 

ti f i t”creation of a new environment”.__________________________________________________________________________________________________* Terman L. (1916). The uses of intelligence tests. Ιn “The measurement of intelligence”. Boston, Houghton Mifflin. 2

Here is a more practical suggestion:

Intelligence is the general mental ability involved in h l l ti iprocesses such as calculation, reasoning, 

classification, learning, using language, d d h lfunderstanding the environment, self‐correcting, 

inventing and adjusting to new situations.

Artificial Intelligence then is human‐created, non‐g ,biological intelligence.

Computational Intelligence is intelligence that emerges through some form of computationthrough some form of computation.

3

AI and CI may lead to solutions of difficult yproblems in science and engineering. 

They have also instigated a revitalization of fundamental philosophical questionsfundamental philosophical questions.

They also generated new frontiers of explorationsThey also generated new frontiers of explorations and introduced new questions.

4

Fundamental questions in relation to AIFundamental questions in relation to AICan a biological brain generate an artificial brain and i t lli ?intelligence?

C th h b i “ d t d” it f ti i ?Can the human brain “understand” its own functioning?

Will h d t di b b l t ?Will such understanding be absolute?

C b i ith i l ll ti l tCan our brain, either singly or collectively generate a new kind of intelligence?

How is WILL generated?

5

What is consciousness?What is consciousness?

Can universal consciousness exist?Can universal consciousness exist?

How does creativity appear?How does creativity appear?

How are illusions love hate generated?How are illusions, love, hate, … generated?

Can matter generate cognition ? And how?Can matter generate cognition ? And how?

6

Wh t i ?What is memory? 

Can there be olle ti e memor (or national) memor ?Can there be collective memory (or national) memory?

How are numbers words meaning codified?How are numbers, words, meaning codified?

How does the brain process logic?How does the brain process logic?

How can the brain handle with ease both crisp and fuzzyHow can the brain handle with ease both crisp and fuzzy logic?

7

Computational Intelligence ToolsComputational Intelligence ToolsExpert systems (ExS)

Artificial neural networks (ANN)

Evolutionary systems (ES)Genetic algorithms (GA)Artificial life (AL)Coevolutionary roboticsEvolvable hardware

Fuzzy systems (FS)…

8

I believe that the Artificial Neural Networks are presently the most promising technology that may lead to non‐programmed, intelligent artificiallead to non programmed, intelligent artificial systems.

ANNs will be presented here in some detailANNs will be presented here in some detail.

9

Artificial IntelligenceArtificial Intelligence

Weak Strongapproach approach

Human knowledge and specific expressions of it can be simulated through 

Human knowledge and specific expressions of it, emerge naturally in 

computational systems computational systems

Classic AI  (symbolic AI)It is expressed through symbolic entitiesthat may be properly coded

Connectionist AIExpressed through non‐symbolic methods such as the ANN. The various cognitive processes 

Some important proponents are:Dennet, Newell and Simon, Chomski, Minsky, Fodor and Pylyshyn

emerge naturally from the dynamically connected cells, as they evolve when influenced from an environment. 

10

Fodor and PylyshynSome important proponents are:Smolensky and Hameroff.

Brief explanation of the methodsBrief explanation of the methods

E t tExpert systemsAlso known as Knowledge Based Systems

It is a system that describes the behaviour  of one or more experts in some field, by using symbols and rules.

Th f l i li d k l d b i iThe set of rules is applied to a knowledge base, aiming at extracting useful information.

There are limitations because of the need to have rules for the rules of the exponentially growing size and the difficultythe rules, of the exponentially growing size, and the difficulty in finding the rules.

11

Main characteristics of Expert Systems:p y

The programming and the conclusions are based on rules.

They are not easily made tolerant to mistakes.

If the knowledge base changes reprogramming may beIf the knowledge base changes, reprogramming may be needed.

Explanation of results is easier to implement with these systems.

12

Brief explanation of Artificial Neural NetworksBrief explanation of Artificial Neural Networks

IntroductionIntroduction

Biological neuronsg

Biological neural networks

Artificial neurons

Artificial neural networks

LearningLearning

Applicationspp

13

Other names for the Artificial Neural Networks

A tifi i l N l S tArtificial Neural Systems

Parallel Distribution Processing Systems

Connectionist Systems

Neurocomputing SystemsNeurocomputing Systems

Adaptive Networks

Associative Networks

Collective Computation SystemsCollective Computation Systems

Neuromorphic Systems

14

The ANNs are based on the effort to mimic the ( )operational properties of the natural (biological) neural 

networks.

They are composed of many artificial neuronsconnected in an organized system in which there is direct or indirect communication and interaction among all its members. 

There is usually provision for information input and for the desired outputthe desired output.

Groups of neurons may organized into layers or slabsGroups of neurons may organized into layers or slabs.15

A i l b l b h i th h th f

An Artificial Neural Network

Artificial neural network

An emerging global behavior may appear through the use of simple, and local learning and evolution rules.

SYSTEM SYSTEM

Processing elementArtificial neural network

SYSTEM     INPUT

SYSTEM     OUTPUT

N 3N 2N1

N 5N 4 54

N 8N 7N 6

16Signal flow

To gain a better understanding of their functioning, let us have a brief encyclopaedic look on the biological neural networks.

Typical  Αxon of sending neuron

SynapseDendrite of receiving neuron

biological neuron Nucleus

Cell body somaInternal voltage ≈ ‐ 60 to ‐ 80 mV

INITIAL SEGMENT

AXON HILLOCK

MembraneThickness ≈ 5 to 10 nm = 0.000005 to 0.00001 mmCapacitance ≈ 0.944 μF/cm2

Field intensity ≈ 12000000 V/m = 120 kV/cm

NODE OF RANVIER

TRIGGER ZONE

Αxon or nerve fiber

Μyelin sheath

Diameter:  0.5 ‐ 22 μm in vertebrate500 ‐ 1000 μm in the squid giant axon

DENDRITE OF RECEIVING NEURON17

INFORMATION TRANSMISSION(Generation of the Action Potential)(Generation of the Action Potential)

Excitation from the environment

The receptive sensors convert changes in the environment(light pressure chemical

Habituation

Stimulus

(light, pressure, chemical constituency, ...) in Graded Potential. 

Sensory receptor

Synapse Graded Potential

Time The graded potential is gradually weakened as it is propagated towards the somaSy apse

mV

Time

propagated towards the soma.

Many graded potentials from different dendrites are 

lmV

Neuron

accumulated to create a bigger or smaller one, which, once it reaches a threshold  (–40mV), it becomes an Action PotentialAction Potential

Time

Neuron it becomes an Action Potential– (a train of pulsed voltages, nerve impulses or spikes)

18Another neuron

ΣύναψηSynapse ΣύναψηSynapse

Diameter ≈ 1 μm = 0.001mmGap ≈20 to 40 nm 

= 0.00002 to 0.00004 mmDiameter/Gap ratio ≈ 100

Delay in the transmission of the pre‐synaptic potential to a post‐synaptic potential ≈ 0.3 to 1.0 ms

Velocity ≈ 0.2 cm/minute

19

The medium for the signal transmission is basically of electrochemical nature.

There exists a great variety of synapses.

Even the position of a synapse makes a difference in signal transmissiontransmission.

The transmission of coded information at the synapses is y pprimarily done with chemical substances known as neurotransmitters [e.g. acetylcholine, norepinephrine, d h d b ddopamine, 5‐hydroxytryptamine serotonin, aminobutyric acidGABA].

These phenomena are mainly occurring in mammal nervoussystems.y

20

ComparisonsComparisonsThere are about 1010 – 1012 neurons in the brain

and about 1013 – 1016 synapses.

Organism leech Worm Fly Cockroach Bee Man

Number of Synapses >104 >105 ~109 <1011 >1011 ~1014

21

Synapses

There also exist electrical synapses, but these occur mainly in lowerThere also exist electrical synapses, but these occur mainly in lower animals.

The systematic use of a synapse is believed to improve its efficacyimprove its efficacy.

learning, memoryg, y

Hebb’s Rule

22

Mathematical models of the biological neuronMathematical models of the biological neuron

The Hodgkin – Huxley model, 1952

C (dV/dt) I g m3h(V V ) g n4(V V ) g (V V )

g y ,

Cm(dV/dt)=Im–gNam3h(V–VNa)–gKn

4(V–VK)–gL(V–VL)dm/dt = αm(1 –m) – βmm

dn/dt = αn(1 – n) – βnn

dh/dt = αh(1 – h) – βhh

23

The Hodgkin – Huxley modelThe Hodgkin – Huxley modelWhere,

Vrest = – 60 mVVna =  50mVVK = – 77 mVV 54 402 VVL = – 54.402 mVgNa =  120 mmho/cm2

gK =  36 mmho/cm2

g = 0 3 mmho/cm2gL =  0.3 mmho/cm2

E = V – Vrest

/ 25 E/10 β E/18αm = 0.125 – E/e 25‐E/10 – 1 ,   βm =  4e‐E/18

αn = 0.0110 – E/e10‐E/10 – 1,   βn = 0.125e

‐E/80

αh = 0.07e‐E/20,   βh = 1/1+ e

30‐E / 10

24

The FitzHugh Nagumo modelThe FitzHugh‐Nagumo model

This is a simpler form of the Hodgkin – Huxley model

duj/dt = uj ‐ uj3/3 ‐ zj + xduj/dt uj uj /3 zj x

d /dt k + k + kdzj/dt = ‐ k1j zj + k2j uj + k3j

25

BRAIN – NERVOUS SYSTEM

26

BRAIN – NERVOUS SYSTEM

The biological neural networks are non‐linear computational systems that are characterized by a highcomputational systems that are characterized by a high degree of parallelism, robustness and fault‐tolerance.

They show a capacity for learning, generalization, and handling vagueness.

It i ti t d th t th t t l l th f th b iIt is estimated that the total length of the brain connections are about 109 meters, which is about 25 times the earth perimeter!times the earth perimeter!

27

Even though the brain is only about 2% of the total weight of a person, it needs about 15% of the total blood and about 25% of the inhaled oxygen.

h d hLiving organisms that do not have nervous systems (bacteria, protozoa, invertebrate) can exhibit some form of behavior memory and even learningof behavior, memory and even learning.

For instance, the paramecium (a single‐cell organism)For instance, the paramecium (a single cell organism) can demonstrate learning behavior, even though it does not have brain and synapses.

28

Every neuron communicates locally and directly with many other neurons and indirectly

LAYER

other neurons and indirectly with all others.

LAYER

Groups of neurons form subsystems, layers, slabs, …

LAYER

y , y , ,

LAYER

The information transmission in synapses is done in parallel. Because of this, the frequency of changes is about 1016 per second. Taking into consideration the fact that at each synapse many neurotransmitters are also transferred, the information transmission is impressive.

29

Comparison of biological and artificial neurons and networksand artificial neurons and networks

BIOLOGICAL NEURONS

AND NETWORKS

ARTIFICIAL NEURONS 

AND NETWORKS

Dense connections

~ (1012 )(104 )

Few connections

~ (1012 neurons)(104 synapses) =

= 1016 connections  

Single neurons are different to one another Mostly similar to one anotherSingle neurons are different to one another

Modular structures

Autonomous local interaction

Mostly similar to one another

Partly modular

Non‐autonomous

Usually supervision is needed

Parallel processing

Very little energy consumption

Non‐mathematical or algorithmic operation

Usually supervision is needed

Mostly serial processing 

Much energy consumption

Mostly mathematical or algorithmic description

30

Non mathematical or algorithmic operation y g p

Artificial Neural Networks

•They are flexible non linear systems composed ofmany•They are flexible, non‐linear systems composed of many cooperating processors, that, among others, help in:

•Study of biological neural systems.•Data processing and knowledge mining.M i d l ti t ti•Mapping and relation extraction.

•Classification, Pattern recognition.•Forecasting.Forecasting.•Study of dynamical systems.•Adaptive signal processing.•Automatic control of systems.•…

31

•It is hoped that through an autonomous evolution of•It is hoped that through an autonomous evolution of such connectionist systems, some form of intelligence may emergemay emerge.

•Will it lead to artificial brain?•Will it lead to artificial brain?

•Artificial mind?•Artificial mind?

•Artificial consciousness?Artificial consciousness?

32

Artificial neurons

•They are the basic building blocks of ANNs•They are the basic building blocks of ANNs.•They are also known as:• Units or Processing Elements

Their main characteristics are:

They have multiple inputs – one output (MISO).

They are non‐linear.

They exhibit adaptivity33

They exhibit adaptivity.

They can be implemented through appropriateThey can be implemented through appropriate software or hardware.

The hardware neurons can be:

Electronic, chemical, optical, micromechanical, nanomechanicalnanomechanical, ...

F ti ll th b i l t d ll lFunctionally, they can be simulated as cellularautomata.

Here is a generalized form of an artificial neuron.

34

General form of a single‐neuron modelInput information from the environment or from other neurons

Output to the environment or to other neurons

Learning system that adapts the various parametersp

POST‐ACCUMULATORPROCESSING

(Subsystem of functional 

PRE‐ACCUMULATORPROCESSING

(Subsystem of functional and dynamical processors)

(including cross‐

MainAccumulator

TRIBUTO

R

F df d/F db k

and dynamical processors) , p2()

(including crosscorrelations), p1() DIST

Feedforward/Feedback subsystem

h()

35

Some useful properties of the artificial neural networks

•They can generalize in the sense that they conclude about an area of interest in which they hadn’t been previously taughtof interest in which they hadn t been previously taught.

•They show robustness. They continue to behave acceptably well if d l d difi deven if some neurons are deleted, or some parameters modified.

•They are fault tolerant.y

•The rules are expressed in a distributed processing and representationmannerrepresentationmanner.

•They usually learn through appropriate training, which can be id d ( i d) id d ( i d)guided (supervised) or unguided (unsupervised).

36

Structuring an ANNStructuring an ANNProcessing element

Artificial neural network

SYSTEM     INPUT

SYSTEM     OUTPUT

TN 3TN 2TN1

TN 5TN 4

TN 8TN 7TN 6

37

Signal flow

ANN Topologies, ArchitecturesNETWORK ARCHITECTURES

STATIC ARCHITECTURES DYNAMIC ARCHITECTURES

AUTONOMOUSLY ADAPTED

EXTERNALLY GUIDED

RANDOM CONNECTIONS

ORGANIΖED CONNECTIONS

ORGANIΖED FEEDFORWARD CONNECTIONS

(Without feedback)

SINGLE LAYER

TWO LAYER

THREE LAYER

MULTI‐ LAYER …

ORGANIΖED RECURRENT CONNECTIONS(With feedback)

SINGLE LAYER

TWO LAYER

THREE LAYER

(With feedback)

38

MULTI‐ LAYER …

CELLULAR NETWORKS

Some examples of ANN architecturesMultilayer perceptron (MLP)

x1 Σ

y = sgn(u )

x2 Σ

y1 = sgn(u1)

.

.Σ

. …Σ

y2 = sgn(u2)

xN ΣWeight, w

39For clarity, the rest of the weights are not shown

Multilayer, multi‐slab network

(hidden)

OUTPUTINPUT

(hidden)SLAB 2

SLAB 5SLAB 4 

(hidden)

SLAB 5 SLAB 1

(hidden)

SLAB 3 

40

Input ReceptiveWeightMatrix

p(1) α(1) WeightMatrix

A(1)

x1.

f1.

putVector

pField

IxK

Matrixw(1a) p1

(1)

.

.

.(1)

α1(1)

.

.

.(1) OxI

MatrixW(1b) A1

(1)

.

.

.(1)

A y...xk

.

.

.fk

IxK pi(1)

.

.

.

p (1)

αi(1) = 

αi(1)

.

.

.

α (1)

OxI Ao(1)

.

.

.

A (1) +

A1

.

.

.

y1...

k...

fk...

JxK

pI( )

p1(2)

.

.

αI( )

α1(2)

.

.

OxJ

AO( )

A1(2)

.

.

+Ao

.

.

.

AO

yo...

yOxK fK

JxK

Weight

.

pj(2)

.

.

.

αj(2) = [ pj(2)]

.

αj(2)

.

.

.

OxJ .

Ao(2)

.

.

.Weight

AO yO

f

WeightMatrixw(2a)

pJ(2)

p(2)αJ(2)

α(2)AO

(2)

A(2)

WeightMatrixW(2b)

41

fk= akep(L) = w(La)f A(L) =W(Lb)α(L)

Kohonen architectures

42

Recurrent architecture

43

Cellular neural network (CNN) architecture

……

……

The artificial neurons… … … The artificial neurons communicate only with their peripheral

44

peripheral.

General comments on neural architectures

It is not necessary to code the information processing in the y p gformalism shown in the previous figures/structures.

We could use an algebraic/differential formalism or other kindWe could use an algebraic/differential formalism, or other kind of graphs such as:

INPUT OUTPUTHIDDEN UNITS

45

INPUT OUTPUT

etc

Learning in ANNs

x1(κ) w1(κ)Desired outputWeight adaptation

x2(κ) w2(κ)

d(κ)

+Comparison

x2(κ)

.

w2(κ)

Σ Σ‐

N

1) i i

iu(κ f x w

=

⎛ ⎞= ⎜ ⎟

⎝ ⎠∑.

. e(κ) = d(κ) – u(κ)Error

xN(κ) wN(κ)

The learning algorithm governs the way that the weights adapt so that the error gradually

N N( )

46

The learning algorithm governs the way that the weights adapt so that the error gradually becomes acceptably small. LEARNING ALGORITHM

Learning in ANNs

Haykin (1994):

Learning is a process by which the free parameters (weights) of a neural network are adapted through a continuing process ofneural network are adapted through a continuing process of stimulation by the environment in which the network is embedded.embedded. 

A more general proposition is:

Learning is achieved through any change, in any g g y g , ycharacteristic of a network, so that meaningful results are achieved.

47

results are achieved.

This definition leads to the possibility of achieving learning through:learning through:

Synaptic weight modification.Network structure (topology) modifications.By creating or deleting neurons or connectionsBy creating or deleting neurons or connections.Through the use of suitable attractors or other it bl t bl i tsuitable stable points.

Learning through forgetting.Through appropriate choice of activation functions.Through modifying controllable parameters in aThrough modifying controllable parameters in a look‐up table defining an activation transformationtransformation.Combination of above. 48

Learning is usually a process of minimizing an error function or maximizing a benefit function.

In this respect, learning resembles the optimization, where we seek to extremize a desired criterion function (Error, Loss, Penalty, Computational energy Lyapunov ή Hamilton function)energy, Lyapunov, ή Hamilton function)

49

Learning rulesSome of the learning rules are significantly different than others, whilethere are many that are of a minor variation, or are special cases ofmore general ones usually identified by a different namemore general ones, usually identified by a different name.

Some important rules are:p

HebbDelta and its variations such as:

Backpropagation (the most common)AdalineAdalinePerceptron…

Competitive learningHopfieldpKohonenART 50

Applications of artificial neural networksIf the objective is to find a mapping such that the system captures the dynamics relating an input data set to an output captures the dynamics relating an input data set to an output data set, the problem is called function approximation or system identification in control engineering applications.

If the objective is to find a mapping such the system loosely i l i hi i d h bl iinterpolates within an input‐output data set, the problem is called generalization.

If the objective is to find a mapping such that the system generalizes to discrete output classes, the problem is called generalizes to discrete output classes, the problem is called classification. In some fields this problem is given different names such as associative memories, pattern association or 

51

pattern recognition. 

Applications of artificial neural networksIn Engineering

• Modeling manufacturing processesprocesses.

• Monitoring and diagnosis of machinery failures.

2x=θ&• Robotics and 

autonomous vehicles.1x=θ

l

2x=θ

m

• Automatic control systems.

u cm

• Quality control. 52x=x3 0 4x=x&

In Medicine and Health Sciences

• Diagnostic applications in EEG, ECG, EMG and other medical signaturesmedical signatures.

• Artificial vision.

• Artificial hearing.

• Artificial nose• Artificial nose.

• Brain controlled telekinesis.

• Identifying RNA and DNA in proteins.

53

Neuromuscular disease diagnosis (UCY, ING)

54

Quantitative Assessment of the Prognostic Factors in Breast Cancer (UCY, ING)

Manual, semi–quantitative, immunohistochemical score

% of CellsPositive

Score StainingIntensity

Score TotalScore

Diagnostic

Index

0 0 Negative 0 0 0

0 25% 1 Weak 1 1 4 1+0 – 25 % 1 Weak 1 1 – 4 1+

26 – 50 % 2 Moderate 2 5 – 8 2+

51 – 75 % 3 Strong 3 9 – 12 3+

≥ 76 % 4 VeryStrong

4 > 13 4+Strong

55

In Signal Processing

• Classification – pattern recognition.

Si l id ifi i• Signal identification, speech recognition, i t itisignature recognition, image recognition, 

k id tifi tispeaker identification.

• Seismic image analysis.

• Identification of explosives and militaryexplosives and military targets.

• Optical character recognition. 56

In Meteorology

• Modeling meteorological processes.

Filli i i i i f ll d (HTI UCY)Filling‐in missing rainfall data (HTI, UCY).

Creation of iso‐hyets.y

In Cognitive studies

Aoccdrnig to a rscheearch at Cmabrigde Uinervtisy it deosn't

In Cognitive studies

Aoccdrnig to a rscheearch at Cmabrigde Uinervtisy, it deosn t mttaer in waht oredr the ltteers in a wrod are, the olny iprmoetnt tihng is taht the frist and lsat ltteer be at the rghitiprmoetnt tihng is taht the frist and lsat ltteer be at the rghit pclae. The rset can be a total mses and you can sitll raed it wouthit porbelm. Tihs is bcuseae the huamn mnid deos not praed ervey lteter by istlef, but the wrod as a wlohe.

57

In Financial Engineering

• In decision‐making.

F i f k i di (UCY HTI)• Forecasting of market indices. (UCY, HTI)

• Stock exchange trends. (HTI, UCY)g ( )

• Bankrupsy prediction. (UCY)

• Optimization. (HTI, UCY)

58

In Prediction and forecasting

• Fraud detection.

• Power plant load prediction. (HTI, UCY)

• Potential mineral exploration sites. p

In Optimization

• In product distribution (traveling salesman problems).

I d i i ki• In decision‐making.

In Synthesis

• Synthesis of music and speech.

S h i f i l• Synthesis of new materials.59

In Data clustering and compression

In Telecommunications, TelemedicineTelemedicine, Geology, Meteorology

60

Possible research areas in ANNs

Basic Researchl h dNew learning methods

New architecturesGenetic algorithm learningGenetic algorithm learningNew single‐neuron models

MedicineDisease diagnosis

Engineeringl d fPower load forecasting

Meteorological applicationsRobotic and other complicated system control

61

Robotic and other complicated system controlShip resistance calculations

Dynamical systemsConflict resolution, conflict avoidance (Fuzzy Cognitive 

Maps ?)Dynamic analysis  (Fuzzy Cognitive Maps ?)D i i ki d i i th C F d l St tDecision‐making dynamics in the Cyprus Federal State 

(Fuzzy Cognitive Maps ?)

MiscellaneousMiscellaneousEarthquake prognosisAthletic capability prognosisAthletic capability prognosis

62

ANN and Statistical methods

•There exists some conflict among neuroscientists and statisticians on who invented first a particularand statisticians on who invented first a particular methodology !Many of the non‐dynamical models used in ANNs haveMany of the non dynamical models used in ANNs have their roots in known statistical methods such as:

Generalized linear models

P l i l iPolynomial regression

Non‐parametric regression

Discriminant analysis

Principal components

63

Cluster analysis

Comparison of terminology

•STATISTICS ANN•••Variables Features•Independent variables  Inputs•Predicted values  Outputs

d i bl i i l•Dependent variables  Targets or training values•Residuals  Errors•Estimation Training, learning, adaptation, or self‐organization.•Estimation criterion  Error function, Cost function, or Lyapunov function•Observations  Patterns or training pairs•Parameter estimates  (Synaptic) weights•Interactions  Higher‐order neurons•Transformations  Functional links•Regression and discriminant analysis Supervised learning or heteroassociation•Data reduction  Unsupervised learning, encoding, or autoassociation•Cluster analysis  Competitive learning or adaptive vector q anti ationquantization•Interpolation and extrapolation  Generalization

64

Evolutionary Systems – ESEvolutionary Systems – ES

Lately has been given great momentumLately has been given great momentum.

They ar usually known as Genetic Algorithms (GA)

Th l ith f ti i ti d hThey are algorithms for optimization and search.

Initially were studied by John Holland of the University of Michigan back in the 70’s.University of Michigan back in the 70 s.

They resemble the natural evolution65

They resemble the natural evolution.

They may be used in:

Function optimization.

Operations research.

f l l kLearning in artificial neural networks .

Evolution of new artificial neural network architectures.

Evolution of fuzzy rules.

…

They are relatively simple to apply, especially in complicated y y p pp y, p y poptimization problems.

They can find global optimization solutions

66

They can find global optimization solutions.

Some more modern terms that are being used are:

Evolutionary Computation or Evolutionary Algorithms.

These include:

Genetic Algorithms ‐ GA

Genetic Programming GPGenetic Programming ‐ GP

Evolutionary Programming ‐ EP

Evolutionary hardware ‐ EHW

Evolutionary Strategies ‐ ESEvolutionary Strategies  ES

Learning Classifier Systems ‐ LCS

67

Artificial Life Systems – ALSArtificial Life Systems  ALS

It is a highly simplified simulation of some dynamical features of living organismsliving organisms .

They have a lot with the well established science of cellular t tautomata.

They use simple rules for self‐organization.

e.g Conway’s “Game of Life”

It is common to experience Complexity and Chaotic problems.

68

They can show an emerging behaviour.

Can be used in:

Optimization problemsOptimization problems.

Neural network learningNeural network learning.

. . .

Modern Particle Swarm Systems is an extension of ALS.

69

Fuzzy Systems (FS)Fuzzy Systems (FS)

The fuzzy logic is a superset of the bi‐state Boolean l ilogic. 

It has been devised in order to tackle the so calledIt has been devised in order to tackle the so called “half‐truths” and the paradoxes, as well as our need to have partly truth valuesto have partly truth values.

70

Some paradoxes:

“I am a liar”! (Epimenides)

Am I saying the truth?

“The barber who shaves all those who don’t shave themselves” (Bertrand Russel)themselves” (Bertrand Russel)

Wh h h b b ?Who shaves the barber?

71

Fuzzy SystemsFuzzy SystemsExample:   When is someone tall?

If P is the set of all humans and h their height, a fuzzy subset μ(h) that could answer the above question is:q

0 if 1,5 m ( 1,5)

if 1,5 2,2 m 0,7

1 if 2,2 m

hh

h

h

(h)μ→ <

−→ ≤ ≤

→ >

=

⎧⎪⎨⎪⎩

Where μ(h) is the established membership function

72

Difference between fuzzy and conventional:

In binary logic, the result can be true or false:

e g 1+1=2e.g. 1+1=2

The world is mostly fuzzy.

e.g.  Its cold, 

It is greenIt is green, ...

There are vagueness because of  “probable events” and b i i diffi l fi d di idi li dbecause it is difficult to find dividing lines among events and objects.

73

Fuzzy SystemsMembership functions

1

μ(h)

0.5

00.5 Height, h [m]2.21.85

S h f i b d f hSuch functions may be constructed for other parameters, and generate a fuzzy logic system with rules:

74

Fuzzy Systems

Some membership functions from the MATLAB library:

1trapmf gbellmf trimf gaussmf gauss2mf smf

0 2

0.4

0.6

0.8

0

0.2

0.6

0.8

1zmf psigmf dsigmf pimf sigmf

0

0.2

0.4

75

Fuzzy Systemsy yAND fuzzy rule:

AND

76

Most applications are in the fields of Automatic Control, in Information Systems and in supporting , y pp gDecision‐making.

77

Fuzzy Cognitive Maps ‐ FCMy g p

It is a kind of combination between ANNs and fuzzy logic.

They are made of a set of junctions that represent concepts and from arrows that connect these 

Can be used in complicated problems in order to simulate the dynamical interrelationships as in Decision Making and in

junctions. 

dynamical interrelationships as in Decision Making and in Strategic Planning.

78Essentially are recurrent diagrams that use fuzzy values.

Concepts that have been considered

An attempt to study interrelations in the Cyprus Problem

Concepts that have been considered30 influencing parameters (concepts) have been explored. These are:

INFLUENCING PARAMETERS (Concepts)C1 Welfare of the Federal State of Cyprus

These are:

C1 Welfare of the Federal State of CyprusC2 Welfare of the Greek Cypriot StateC3 Welfare of the Turkish Cypriot StateypC4 Greek Cypriot nationalismC5 Christian religiousnessC6 Knowledge of Turkish language by the Greek CypriotsC7 Knowledge of Turkish history by the Greek CypriotsC8 Educational level of the Greek CypriotsC9 Turkish Cypriot nationalism

79

C10 Islamic religiousness

Concepts that have been considered

C11 Knowledge of Greek language by the Turkish CypriotsC12 Knowledge of Greek history by the Turkish CypriotsC13 Educational level of the Turkish CypriotsC14 Political interests of EuropeC15 Political interests of USAC16 Political interests of RussiaC1 li i l i fC17 Political interests of UKC18 Political interests of IsraelC19 Political interests of GreeceC19 Political interests of GreeceC20 Political interests of Turkey

80

Concepts that have been considered

C21 Military interests of IsraelC22 Military interests of GreeceC23 Military interests of TurkeyC24 Military interests of UKC25 Mili i f USAC25 Military interests of USAC26 Interests of Anatolian settlersC27 Level of tourism in the federated stateC27 Level of tourism in the federated stateC28 Oil fieldsC29 Quality of environmentC29 Quality of environmentC30 Other natural resources

81

Effects of changes in Islamic religiousness European and USA interests300%Effects of changes in Islamic religiousness, European and USA interests

250%

300%

Percentage change in parameters due to major increase in Islamic Religiousness

200%

g g p j g

Percentage change in parameters due to major increase in European interest

P h i d j i i USA i

150%

Percentage change in parameters due to major increase in USA interest

50%

100%

0%

WelfWelfWelfGC NChrisKnowKnowEducTC NIslamKnowKnowEducPolitiPolitiPolitiPolitiPolitiPolitiPoliti Milit Milit Milit Milit MilitInterLevelOil fiQualiOthe

-50%

lfare of the Federal State

lfare of the GC State

lfare of the TC State

Nationalismristian religiousness

owledge of Turkish langu

owledge of Turkish histo

ucational level of the GC

Nationalismamic religiousness

owledge of Greek langua

owledge of Greek history

ucational level of the TC

itical interests of Europe

itical interests of USA

itical interests of Russia

itical interests of UK

itical interests of Israel

itical interests of Greece

itical interests of Turkey

litary interests of Israel

litary interests of Greece

litary interests of Turkey

litary interests of UK

litary interests of USA

erests of anatolian settlers

vel of tourism in the Fede

fieldsality of environment

her natural resources

82

te of Cyprus

guage by the GC

tory by the GC

C uage by the TC

ry by the TCC pe a e y ce ey ers derated state

Case of increase in the Islamic religiousnessThis results in significant increase in the Greek Cypriot nationalism (+36%) and most importantly a very high increase in Christian religiousness (+283%). 

Al it ill lt i i i th ilit i t t fAlso, it will result in an increase in the military interests of Turkey, Greece and Israel.

Of significance is the observation that the welfare of the Federal State of Cyprus will reduce by 7% and the welfare of yp ythe Turkish Cypriot community by 4%. 

Some reduction in the level of tourism and the quality of the environment is also observed.

83

Case of increase in the European political interests in Cyprus

The model predicts a significant increase of the knowledge of Greek language and history by the Turkish Cypriots and vice versa. 

Thi i d i th f k l d f thThis is more pronounced in the case of knowledge of the Turkish language and history by the Greek Cypriots. 

It is however very interesting that the model suggests a significant reduction in Christian religiousness (‐28%) and g g ( )somewhat less (‐5%) in the Islamic religiousness. 

Most interesting though, is the suggestion that the interests of the Anatolian settlers will reduce by 20%.

84

Case of increase in the political interests of USA in Cyprus

The model predicts a 12% increase in the political interests of Israel and a 12% increase in the welfare of the Turkish Cypriots. 

What is most interesting though, and somewhat unusual, is the d ti f th Ch i ti li i b 16% f th T ki hreduction of the Christian religiousness by 16%, of the Turkish 

Cypriot nationalism by 18%, of the Islamic religiousness by 8%, of the political interests of Greece and Turkey by 11% and 16%of the political interests of Greece and Turkey by 11% and 16% respectively, and of the military interests of UK by 18%.

85