Computer Science Dept, UNC Charlotte Copyright 2002 Kayvan Najarian1 Neural Networks Outline –Introduction –From biological to artificial neurons –Self.

Computer Science Dept, UNC Charlotte

Copyright 2002 Kayvan Najarian 1

Neural Networks

• Outline– Introduction

– From biological to artificial neurons

– Self organizing maps

– Backpropagation network

– Radial basis functions

– Associative memories Hopfield networks

– Other applications of neural networks



Introduction

• Why neural networks?– Algorithms developed over centuries do not fit the complexity of

real world problem

– The human brain: most sophisticated computer suitable for solving extremely complex problems

• Historical knowledge on human brain– Greeks thought that the brain was where the blood is cooled off!

– Even till late 19th century not much was known about the brain and it was assumed to be a continuum of non-structured cells

– Phineas Gage’s Story• In a rail accident, a metal bar was shot

through the head of Mr. Phineas P.

Gage at Cavendish, Vermont, Sept 14, 1848 – Iron bar was 3 feet 7 inches long and weighed 13 1/2 pounds. It was 1 1/4

inches in diameter at one end



Introduction (cont’d)

• He survived the accident!– Originally he seemed to have fully recovered with no clear effect(s)

• After a few weeks, Phineas exhibited profound personality changes

– This is the first time, researchers have a clear evidence that the brain is not a continuum of cell mass and rather each region has relatively independent task



Introduction (cont’d)

• Biological neural networks– 1011 neurons (neural cells)

– Only a small portion of these

cells are used

– Main features• distributed nature,

parallel processing

• each region of the brain

controls specialized task(s)

• no cell contains too

much information: simple

and small processors

• information is saved mainly in the connections among neurons



• learning and generalization through examples

• simple building block: neuron– Dendrites: collecting

signals from other neurons

– Soma (cell body): spatial

summation and processing

– Axon: transmitting signals to

dendrites of other cells

Introduction (Continued)



Introduction (Continued)

• biological neural networks:

formation of neurons with different connection strengths



• Biological vs. artificial neurons– From biological neuron to schematic structure of artificial

neuron• biological:

– Inputs

– Summation

of inputs

– Processing

unit

– Output

• artificial:

From biological to artificial neural nets

1x

Nx

1w

Nw

)...( 11 nn xwxwfy



– Artificial neural nets:• Formation of artificial neurons

From biological to artificial neural net (continued)

neuron 1

1x

Nx

NMw

neuron 2

neuron M

iw1

11w

Niw

neuron i

neuron M-1

1y

2y

iy

1My

My



– Multi-layer neural nets:• Serial connection of single layers:

– Training: finding the best values of weights wij

• Training happens iteratively and through exposing the network to examples:

From biological to artificial neural nets (continued)

1x

Nx

NMw

iw1

11w

Niw

1y

2y

iy

1My

My

ijijij www )old()new(



– Activation functions:

• Hard limiter (binary step):

– Role of threshold

– Biologically supported

– Non-differentiable


x

x

x

xf

if1

if0

if1

)(

1

1

x

)(xf



• Binary sigmoid (exponential sigmoid)

– Differentiable

– Biologically supported

– Saturation curve is controlled

by

– In limit when , hard limiter is achieved

• Bipolar sigmoid (atan)

– As popular as binary sigmoid


)(tan)( 1 xxf



• Supervised Vs Unsupervised Learning– Supervised learning (classification)

• Training data are labeled, i.e the output class of all training data are given

• Example: recognition of birds and insects– Training set:

– Classification:

– Unsupervised learning (clustering)• Training data are not labeled• Output classes must be generated during training• Similarity between features of training example creates different

classes

birdowlinsectantinsectbeebirdeagle ,,,

?sparrow




• Example : types of companies– Features: Number

of employees &

rate of growth

– Training data

create natural

clusters

– From graph:

Class #1: small size companies with small rate of growth

Class #2: small size companies with large rate of growth

Class #3: medium size companies with medium rate of growth

Class #4: large size companies with small rate of growth– Classification: a company with NOE=600 & ROG=12%

is mapped to Class #3

Number of Employees

Rat

e of

Gro

wt h

100 500 1000

5%20

%10

%





– Artificial neural networks as classification tools:• If training data are labeled, supervised neural nets are used• Supervised leaning normally results to better performance• Most successful types of supervised ANNs:

– Multi-layer perceptrons– Radial basis function networks

– Artificial neural networks as clustering tools:• If training data are not labeled, unsupervised neural nets are used• Most successful types of supervised ANNs:

– Kohonen network

– Some networks can be trained both in supervised and unsupervised modes



Perceptron

• A more advanced version of

simple neuron

• Structure (architecture):– Very similar to simple neuron

– The only difference:

activation function is

bipolar hard limiter

iny

iny

iny

y

_if1

_if0

_if1

1

1

iny

y

1x

ix

nx

n

iiij wxbiny

1

_



Competitive Self-Organizing Networks

• Biological procedure:– Each neuron (group of neurons)

stores a pattern– Neurons in a neighborhood store

similar patterns– During classification the similarity

of new pattern with all the patterns

in all neurons is calculated– The neurons (or neighborhoods)

with highest similarity are the winners– The new pattern is attributed to the

class of the winner neighborhood

Rocking Chair

Study Chair Desk

Dining Table

DishesFood

Forest

Books

Trees

Rocking Chair

Study Chair Desk

Dining Table

DishesFood

Forest

Books

Trees

Winner: Desk (and its neighborhood)

New pattern: Conference Table



Kohonen Self-Organizing Maps

• Main idea: placing similar objects close to each other– Example: object classification using one-dimensional array

– New pattern: hovercraft • is mapped to the nationhood of truck or airplane

weight (0 to 1) texture (wood=0, metal =1)

size (0 to 1) flies (1) or not (0)

pencil

pencil

airplane

airplane

chair

chair

book

book

wooden house

wooden house

stapler

stapler

metal desk

metal desktruck

truck

kite

kite

Inputs



Kohonen Self-Organizing Maps (continued)

• Two-dimensional arrays– Proximity exists on two dimensions– Better chance of positioning similar

objects in the same vicinity– The most popular

type of Kohonen network– How to define neighborhoods

RectangularHexagonal

• Radius of neighborhood

(R)

• R = 0 means that each

neighborhood has only

one member



Kohonen Self-Organizing Maps (continued)

• Example: two-dimensional array– Classification of objects

• Objective: clustering of

a number of objects • 10 input features: # of lines,

thickness of lines, angles, ...• Competitive layer is

a 2D-grid of neurons• Trained network clusters

objects rather successfully• Proximity of some objects

is not optimal



Backpropagation neural networks

• Idea: not as biologically-supported as Kohonen • Architecture:

– Number

of layers &

number of

neurons in

each layer– Most popular structure

• Activation functions: – sigmoid

1x

Nx

NLw

iw1

11w

Niw

1y

2y

iy

1Ly

My

jkvjz



Backpropagation neural networks (continued)

• Updating weights:– Based on Delta Rule – Function to be minimized:

– Best updating of

wJK is toward

the gradient– Error of output layer is propagated back towards the input layer– Error is calculated at output layer and propagated back towards

the input layer– As layers receive the backpropagated error, they adjust their

weights according to the negative direction of gradient

k

kk ytE 25.0




– Define:

– Then:

and:

– Now:

)_( KKKK inyfyt

JKJK

zw

E

IJk

JkkIJ

xinzfwv

E_

jkjk

jk zw

Ew




• Applications:– Time-series analysis and prediction

• Problem statement (simplest case): – The future value of a signal depends on the previous values of

the same signal, i.e.• Objective: to use a neural net to estimate function “g”• Procedure:

– Form a number of training points as:

– Train a backpropagation net to learn the input-output relation• Advanced cases:

– Procedure is similar to the simple case

)(,...),2(),1()( ptytytygty

1....,,1,

)()(,...),2(),1(

pnppi

iypiyiyiy

)(,...),2(),1(),(,...),2(),1()( rtxtxtxptytytygty




• Feedforward neural time series models are used in many fields including:

– Stock market prediction– Weather prediction– Control– System identification– Signal and image processing– Signal and image compression

• Classification:– When a hard limiter is added to the output neurons

(only in classification and not during the training phase), backpropagation network is used to classify complex data sets



Radial Basis Function networks (RBFN’s)

• Architecture:– Weights of the input

layer are all “1”, i.e.:

– Based on the exact definition

of radial basis functions ,

many different families of

RBFN’s are defined– Main properties of all “radial” basis functions: “radial”

• Example: is a radial function because the value of

is the same for

all points with equal

distance from point C.

1x

2

1)(

Cxxx

…..

2x

nx

(x)1

(x)2

(x)m

…... y

nxxx ...,,,x 21

(.)i

C

m

iiiy

1

x

1

2

m

1all



RBFN’s (continued)

• Basis functions– Gaussian basis functions:

• Coordinates of center:• Width parameter:

– Reciprocal Multi-Quadratic (RMQ) functions:

• Width parameter:

• Training: Least Mean Square Techniques

2

2xx

exp(x)i

C

ii

iCx

i

2

Cxx1

1(x)

ii

i

b

ib



RBFN’s (continued)• Example:

– Trying to approximate a no-linear function using RMQ-RBFN’s– Function to be estimated:

– 135 Training points– 19 Basis functions are

uniformly centered

between -10 and 10– All basis functions have: b = 0.5– Training method: batch– Estimation is rather successful

-10 -8 -6 -4 -2 0 2 4 6 8 10-2.5

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

2.5

Input

Actu

al and E

stim

ate

d O

uto

ut

RBFNs (batch method)

Red: Actual

Blue: Estimated

xxxg 1ln).sin()(



Associative Memories

• Concept:– Object or pattern A (input) reminds the network of object or pattern

B (output)

• Heteroassociative Vs. autoassociative memories– If A and B are different, the system is called heteroassociative net

• Example: you see a large lake (A) and that reminds you of the Pacific (B) ocean you visited last year

– If A and B are the same, the system is called autoassociative net• Example: you see the Pacific ocean for the second time (A) and that

reminds you of the Pacific (B) ocean you visited last year



Associative Memories (continued)

• Recognizing new or incomplete patterns– Recognizing patterns that are similar to one of the patterns stored in

memory (generalization)• Example: recognizing a football player you haven’t seen before from

his clothes

– Recognizing incomplete or noisy patterns whose complete (correct) forms were previously stored in memory

• Example: recognizing somebody’s face from a picture that is partially torn

• Unidirectional Vs. bidirectional memories– Unidirectional: A reminds you of B

– Bidirectional: A reminds you of B and B reminds you of A

• Many biological neural nets are associative memories



Hopfield Network

• Concept:– A more advance type of autoassociative memory

– Is almost fully connected

• Architecture– Symmetric weights

– No feedback from a

cell to itself

– Notice the “feedback” in

the network structure

jiij ww

0iiw



• Concept:– Bidirectional memory: pattern A reminds you of pattern B and pattern B

reminds you of pattern A– Is almost fully connected

• Architecture– Symmetric weights

– No feedback from a

cell to itself

– Notice the “feedback” in

the network structure

Bidirectional Associative Memories (BAM)

jiij ww

0iiw

YmYjY1

XnXiX1

… …

……

nmw11wijw



Other Applications of NNs

• Control– Structure:

– Example: Robotic manipulation

SystemNeuro-controller

Actualbehavior

ControldecisionDesired

behavior



• Finance and Marketing– Stock market prediction

– Fraud detection

– Loan approval

– Product bundling

– Strategic planning

• Signal and image processing– Signal prediction (e.g. weather prediction)

– Adaptive noise cancellation

– Satellite image analysis

– Multimedia processing

Applications of NNs (continued)



• Bioinformatics– Functional classification of protein

– Functional classification of genes

– Clustering of genes based on their expression (using DNA microarray data)

• Astronomy– Classification of objects (into stars and galaxies ad so on)

– Compression of astronomical data

• Function estimation




• Biomedical engineering– Modeling and control of complex biological system (e.g. modeling

of human respiratory system)

– Automated drug-delivery

– Biomedical image processing and diagnostics

– Treatment planning

• Clustering, classification, and recognition– Handwriting recognition

– Speech recognition

– Face and gesture recognition


Computer Science Dept, UNC Charlotte Copyright 2002 Kayvan Najarian1 Neural Networks Outline –Introduction –From biological to artificial neurons –Self.

Documents

computer science dept

unc charlotte copyright

biological neuron

neurons neural cells

kayvan najarian7 biological

kayvan najarian6 introduction

kayvan najarian2 introduction

cells introduction