Mario Pavone

7/23/2019 Mario Pavone

http://slidepdf.com/reader/full/mario-pavone 1/46

OMICS Group

Contact us at: [email protected]

OMICS Group International through its Open Access Initiative iscommitted to make genuine and reliable contributions to thescientifc communit. OMICS Group hosts over !"" leading#edge peer revie$ed Open Access %ournals and organi&es over'"" International Con(erences annuall all over the $orld.OMICS )ublishing Group *ournals have over ' million readers

and the (ame and success o( the same can be attributed to thestrong editorial board $hich contains over '"""" eminentpersonalities that ensure a rapid+ ,ualit and ,uick revie$process. OMICS Group signed an agreement $ith more than-""" International Societies to make healthcare in(ormation

Open Access.



OMICS Group $elcomes submissions that areoriginal and technicall so as to serve both

the developing $orld and developedcountries in the best possible $a.

OMICS %ournals are poised in ecellence bpublishing high ,ualit research. OMICSGroup (ollo$s an /ditorial Manager0 Sstempeer revie$ process and boasts o( a strong

and active editorial board./ditors and revie$ers are eperts in their

feld and provide anonmous+ unbiased anddetailed revie$s o( all submissions.

1he *ournal gives the options o( multiplelanguage translations (or all the articles andall archived articles are available in 21M3+

4M3+ )56 and audio (ormats. Also+ all thepublished articles are archived in repositoriesand indeing services like 5OA%+ CAS+ Google

Scholar+ Scientifc Commons+ IndeCopernicus+ /7SCO+ 2I8A9I and GA3/.

6or more details please visit our $ebsite:

http:omicsonline.orgSubmitmanuscript.php

OMICS %ournals are $elcoming Submissions

http://omicsonline.org/Submitmanuscript.php

http://omicsonline.org/Submitmanuscript.php



8eural 8et$orks



Mario Pavone

)ro(esssor;niversit o( CataniaItal



9ecentl publishedarticles

• Clonal Selection - An Immunological Algorithm for Global Optimization over

Continuous Spaces• Swarm Intelligence Heuristics for Graph Coloring Problem

•O-B-CO!" Optimal Bs for CO!oring Graphs

• scaping !ocal Optima via Parallelization an# $igrationProtein $ultiple

Se%uence Alignment b& H&bri# Bio-Inspire# Algorithms

• ffective Calibration of Artificial Gene 'egulator& (etwor)s• !arge scale agent-base# mo#eling of the humoral an# cellular immune

response

• A $emetic Immunological Algorithm for 'esource Allocation Problem*



7iological inspirations

• Some numbers< – 1he human brain contains about -" billion

nerve cells =neurons> – /ach neuron is connected to the others

through -"""" snapses

• )roperties o( the brain – It can learn+ reorgani&e itsel( (rom eperience

– It adapts to the environment – It is robust and (ault tolerant



7iological neuron

• A neuron has – A branching input =dendrites> – A branching output =the aon>

• 1he in(ormation circulates (rom the dendrites tothe aon via the cell bod

• Aon connects to dendrites via snapses – Snapses var in strength – Snapses ma be ecitator or inhibitor

axon

cell body

synapse

nucleus

dendrites



?hat is an artifcial neuron

• 5efnition : 8on linear+ parameteri&ed(unction $ith restricted output range

- '

$"

+= ∑

−

=

+

+

,

n

i

ii xww f y



Activation (unctions

0 2 4 6 8 10 12 14 16 18 200

2

4

6

8

10

12

14

16

18

20

-10 -8 -6 -4 -2 0 2 4 6 8 10

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

-10 -8 -6 -4 -2 0 2 4 6 8 10-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

3inear

3ogistic

2perbolic tangent

x y =

-e.p/+

+

x y

−+=

-e.p/-e.p/

-e.p/-e.p/

x x

x x y

−+−−

=



8eural 8et$orks

• A mathematical model to solve engineeringproblems

– Group o( highl connected neurons to reali&ecompositions o( non linear (unctions

• 1asks – Classifcation

– 5iscrimination

– /stimation

• tpes o( net$orks

– 6eed (or$ard 8eural 8et$orks – 9ecurrent 8eural 8et$orks



6eed 6or$ard 8eural8et$orks

• 1he in(ormation ispropagated (rom theinputs to the outputs

• Computations o( 8o non linear (unctions(rom n input variablesb compositions o( 8c algebraic (unctions

• 1ime has no role =8Occle bet$een outputsand inputs>

- n<..

-st hiddenlaer

nd hiddenlaer

Outputlaer



9ecurrent 8eural 8et$orks

• Can have arbitrartopologies

• Can model sstems $ithinternal states =dnamic

ones>• 5elas are associated to a

specifc $eight

• 1raining is more diBcult

• )er(ormance ma beproblematic – Stable Outputs ma be

more diBcult to evaluate

– ;nepected behavior=oscillation+ chaos+ <>

-

-

"-"

-"

""



3earning

• 1he procedure that consists in estimating the parameters o(neurons so that the $hole net$ork can per(orm a specifctask

• tpes o( learning

– 1he supervised learning – 1he unsupervised learning

• 1he 3earning process =supervised> – )resent the net$ork a number o( inputs and their

corresponding outputs

– See ho$ closel the actual outputs match the desired ones – Modi( the parameters to better approimate the desired

outputs



Supervised learning

• 1he desired response o( the neuralnet$ork in (unction o( particularinputs is $ell kno$n.

• A )ro(essorD ma provide eamplesand teach the neural net$ork ho$ to(ulfll a certain task



;nsupervised learning

• Idea : group tpical input data in (unctiono( resemblance criteria un#kno$n a priori

• 5ata clustering

• 8o need o( a pro(essor – 1he net$ork fnds itsel( the correlations

bet$een the data

– /amples o( such net$orks :• Eohonen (eature maps



)roperties o( 8eural8et$orks

• Supervised net$orks are universal approimators=8on recurrent net$orks>

• 1heorem : An limited (unction can beapproimated b a neural net$ork $ith a fnite

number o( hidden neurons to an arbitrarprecision

• 1pe o( Approimators – 3inear approimators : (or a given precision+ the number

o( parameters gro$s eponentiall $ith the number o(

variables =polnomials> – 8on#linear approimators =88>+ the number o(

parameters gro$s linearl $ith the number o( variables



Other properties

• Adaptivit – Adapt $eights to environment and retrained

easil

• Generali&ation abilit – Ma provide against lack o( data

• 6ault tolerance – Grace(ul degradation o( per(ormances i(

damaged F 1he in(ormation is distributed$ithin the entire net.



• In practice+ it is rare to approimate akno$n (unction b a uni(orm (unction

• black boD modeling : model o( a process• 1he output variable depends on the input

variable $ith kF- to 8

• Goal : /press this dependenc b a(unction+ (or eample a neural net$ork

Static modeling

{ }k pk y x 0



• I( the learning ensemble results (rom measures+the noise intervenes

• 8ot an approimation but a ftting problem

• 9egression (unction• Approimation o( the regression (unction :

/stimate the more probable value o( p (or agiven input

• Cost (unction:

• Goal: Minimi&e the cost (unction b determiningthe right (unction g

[ ]1

+

-0/-/

1

+-/

∑=−=

N

k

k k

p w x g x yw J



/ample



Classifcation=5iscrimination>

• Class ob*ects in defned categories

• 9ough decision O9

• /stimation o( the probabilit (or acertain ob*ect to belong to a specifcclass

/ample : 5ata mining

• Applications : /conom+ speech andpatterns recognition+ sociolog+ etc.



/ample

/amples o( hand$ritten postal codesdra$n (rom a database available (rom the ;S )ostal service



?hat do $e need to use88

• 5etermination o( pertinent inputs

• Collection o( data (or the learning andtesting phase o( the neural net$ork

• 6inding the optimum number o( hiddennodes

• /stimate the parameters =3earning>

• /valuate the per(ormances o( the net$ork

• I6 per(ormances are not satis(actor thenrevie$ all the precedent points



Classical neuralarchitectures

• )erceptron

• Multi#3aer )erceptron

•

9adial 7asis 6unction =976>• Eohonen 6eatures maps

• Other architectures –

An eample : Shared $eights neuralnet$orks



)erceptron

• 9osenblatt =-H>

• 3inear separation

• Inputs :Jector o( real

values

• Outputs :- or #-

KKKK

KK

KK

KK K K

K

K K

K

KKK

KK

K

KK

KK

KKK

K

KK

K

K

K

K

,11++, =++ xc xcc

++= y

+−= y

,c+c 1c

∑

+ x

1 x+

11++, xc xccv ++=

-/v sign y =



3earning =1he perceptronrule>• Minimi&ation o( the cost (unction :

• %=c> is al$as F " =M is the ensemble o( badclassifed eamples>

• is the target value

• )artial cost – I( is not $ell classifed :

– I( is $ell classifed

• )artial cost gradient

• )erceptron algorithm

k x

∑ ∈ −=

M k

k k

p

v yc J /

k

p

y

k k

p

k k

p

k k

p

x yv y

v y

+=<

=>

+-c/) c/)"classifie#not wellis./ ,if

+-c/) c/)"classifie#wellis/. ,if

)

)

k x

k k

p

k v yc J −=-/

,-/ =c J k

k k

p

k

x y

c

c J −=

∂

∂ -/



• 1he perceptron algorithm convergesi( eamples are linearl separable



Multi#3aer )erceptron

• One or morehidden laers

• Sigmoid activations(unctions-st hidden

laer

nd hiddenlaer

Outputlaer

Input data



3earning•

7ack#propagation algorithm

I( the *th node is an output unit

Credit assignment

( )

-/2-/

-/-3/1

+

-/

,

j j j j

j j

j

j j

j

j j

j

j

j

i j

ji

j

j ji

ji

j j j

n

i

i ji j j

net f ot

ot o

E ot E

net f

o

E

net

o

o

E

ow

net

net

E

w

E w

net f o

owwnet

−=

−−=∂

∂=>−=

′

∂

∂−=

∂

∂

∂

∂−=

=∂

∂

∂

∂−=

∂

∂−=∆

=

+= ∑

δ

δ

αδ α α

j

jnet

E

∂∂

−=δ



Momentum term to smooth 1he $eight changes over tim

-/-+/-/

-+/-/-/-/

-/2

t wt wt w

t wt ot t w

wnet f

wo

net

net

E

o

E

ji ji ji

jii j ji

k kjk j j j

k k kjk

j j

∆+−=

−∆+=∆

=

−=∂

∂∂

∂=∂∂

∑

∑ ∑

γ αδ

δ δ

δ

κ

κ κ κ

κ



Structure Types of

Decision Regions

Exclusive-OR

Prole!

"l#sses wit$

Mes$e% regions

Most &ener#l

Region S$#pes

Single-'#yer

Two-'#yer

T$ree-'#yer

(#lf Pl#ne

)oun%e% )y

(yperpl#ne

"onvex Open

Or

"lose% Regions

Abitrar&

/Comple.it&

!imite# b& (o*

of (o#es

A

AB

B

A

AB

B

A

AB

B

B

A

BA

BA

5iLerent non linearl separableproblems

Neural Networks – An Introduction Dr. Andrew Hunter



9adial 7asis 6unctions=976s>

• 6eatures – One hidden laer – 1he activation o( a hidden unit is determined b the distance

bet$een the input vector and a prototpe vector

9adial units

Outputs

Inputs



•

976 hidden laer units have areceptive feld $hich has a centre

• Generall+ the hidden unit (unctionis Gaussian

• 1he output 3aer is linear• 9eali&ed (unction

( )∑

= −Φ=

*

j j j c x+ x s+

/

( )1

e.p

−−=−Φ

j

j

j

c xc x

σ



3earning

• 1he training is per(ormed b deciding on – 2o$ man hidden nodes there should be

– 1he centers and the sharpness o( the

Gaussians• steps

– In the -st stage+ the input data set is used todetermine the parameters o( the basis

(unctions – In the nd stage+ (unctions are kept fed $hile

the second laer $eights are estimated= Simple 7) algorithm like (or M3)s>



M3)s versus 976s

• Classifcation – M3)s separate classes via

hperplanes

– 976s separate classes viahperspheres

• Learning – M3)s use distributed

learning

– 976s use locali&edlearning

– 976s train (aster

• Structure – M3)s have one or more

hidden laers

– 976s have onl one laer

– 976s re,uire more hiddenneurons F curse o(dimensionalit

41

4+

MLP

41

4+

RBF



Sel( organi&ing maps

• 1he purpose o( SOM is to map a multidimensionalinput space onto a topolog preserving map o(neurons – )reserve a topological so that neighboring neurons

respond to similar Ninput patterns

– 1he topological structure is o(ten a or ' dimensionalspace

• /ach neuron is assigned a $eight vector $ith thesame dimensionalit o( the input space

• Input patterns are compared to each $eightvector and the closest $ins =/uclidean 5istance>



• 1he activation o( theneuron is spread in itsdirect neighborhoodFneighbors becomesensitive to the same

input patterns• 7lock distance• 1he si&e o( the

neighborhood isinitiall large but

reduce over time FSpeciali&ation o( thenet$ork

6irst neighborhood

nd neighborhood



Adaptation

• 5uring training+ the$innerD neuron andits neighborhoodadapts to make their$eight vector moresimilar to the inputpattern that causedthe activation

• 1he neurons are

moved closer to theinput pattern

• 1he magnitude o( theadaptation iscontrolled via a

learning parameter$hich deca s over

Shared $eights neural net$orks:



Shared $eights neural net$orks: 1ime 5ela 8eural 8et$orks

=1588s>• Introduced b ?aibel in -HH

• )roperties – 3ocal+ shi(t invariant (eature etraction

–

8otion o( receptive felds combining localin(ormation into more abstract patterns at ahigher level

– ?eight sharing concept =All neurons in a(eature share the same $eights>

• All neurons detect the same (eature but in diLerentposition

• )rincipal Applications – Speech recognition

–

Image analsis



1588s =contPd>

• Ob*ects recognition inan image

• /ach hidden unitreceive inputs onl(rom a small region o(

the input space :receptive feld

• Shared $eights (or allreceptive felds Ftranslation invariancein the response o( thenet$ork

Inputs

2idden3aer -

2idden3aer



• Advantages – 9educed number o( $eights

• 9e,uire (e$er eamples in the training set

• 6aster learning

– Invariance under time or spacetranslation

–

6aster eecution o( the net =incomparison o( (ull connected M3)>

8 l 8 k



8eural 8et$orks=Applications>

• 6ace recognition

• 1ime series prediction

• )rocess identifcation

• )rocess control

• Optical character recognition

• Adaptative fltering

• /tc<

C l i 8 l



Conclusion on 8eural8et$orks

• 8eural net$orks are utili&ed as statistical tools – Ad*ust non linear (unctions to (ulfll a task – 8eed o( multiple and representative eamples but (e$er than

in other methods• 8eural net$orks enable to model comple static

phenomena =66> as $ell as dnamic ones =988>• 88 are good classifers 7;1

– Good representations o( data have to be (ormulated – 1raining vectors must be statisticall representative o( the

entire input space – ;nsupervised techni,ues can help

• 1he use o( 88 needs a good comprehension o( the problem



International 5ournal of Swarm Intelligence an# volutionar&

Computation

International 5ournal of Swarm

Intelligence an# volutionar&

Computation

i l l f S lli #



A Global Collo%uium on Artificial

Intelligence

International 5ournal of Swarm Intelligence an#

volutionar& Computation

OMICS Group Open Access Membership



OMICS Group Open Access Membership

OMICS publishing Group Open Access

Membership enables academic and researchinstitutions+ (unders and corporations toactivel encourage open access in scholarlcommunication and the dissemination o(research published b their authors.6or more details and benefts+ click on the

link belo$:http:omicsonline.orgmembership.php

http://omicsonline.org/membership.php



Mario Pavone

Documents