Neural Networks Primer Dr Bernie Domanski The City University of New York / CSI 2800 Victory Blvd 1N-215 Staten Island, New York 10314 [email protected].

Neural Networks Primer

Dr Bernie DomanskiThe City University of New York / CSI

2800 Victory Blvd 1N-215Staten Island, New York 10314

[email protected]://domanski.cs.csi.cuny.edu

mailto:[email protected]




© B. Domanski, 2000-2001. All Rights Reserved. Slide 2

What is a Neural Network?

Artificial Neural Networks – (ANN) Provide a general, practical method for

learninglearning – real valued functions– discrete valued functions– vector valued functionsfrom examples

Algorithms “tune” input parameters to best fit a training set of input-output pairs


What is a Neural Network?

ANN Learning is robust to errors in the training set data

ANN have been applied to problems like Interpreting visual scenes Speech recognition Learning robot control strategies Recoginizing handwriting Face recognition


Biological Motivation

ANNs are built out of a densely interconnected set of simple units (neuronsneurons) Each neuron takes a number of real-

valued inputs Produces a single real-valued output Inputs to a neuron may be the outputs of

other neurons. A neuron’s output may be used as input

to many other neurons


Biological Analogy

Human Brain: 1011 neurons Each neuron is connected to 104 neurons Neuron Activity is inhibitedinhibited or excitedexcited

through interconnections with other neurons

Neuron switching times = 10-3 (human) Time to recognize mom = 10-1 seconds Implies only several hundred neuron neuron

firingsfirings


Complexity of the Biological System

Speculation: Highly parallelparallel processes must be

operating on representations that are distributed over many neurons.

Human neuron switching speeds are slowslow

Motivation is for ANN to capture this Motivation is for ANN to capture this highly parallel computation based on a highly parallel computation based on a distributed representationdistributed representation


A Simple Neural Net Example

Input

Nodes

Input

Nodes

OutputOutput

NeuronsNeurons

LINKLINK

WeightWeight

How Does the Network Work?

Assign weights weights to each input-linkto each input-linkMultiplyMultiply each weight by the input value (0 or 1) each weight by the input value (0 or 1) SumSum all the weight-firing input combinations all the weight-firing input combinationsIf If Sum >Sum > ThresholdThreshold for the Neuron then for the Neuron then Output = +1Output = +1 Else Output = -1Else Output = -1

So for the X=1, Y=1 case –So for the X=1, Y=1 case – IF w1*X+w2*Y > 99 THEN OUTPUT =Z= +1 IF w1*X+w2*Y > 99 THEN OUTPUT =Z= +1 50*1+50*1 > 99 50*1+50*1 > 99 IF w3*X+w4*Y+w5*Z > 59 THEN OUTPUT = IF w3*X+w4*Y+w5*Z > 59 THEN OUTPUT =

+1+1 30*1+30*1+(-30)*1 > 59 30*1+30*1+(-30)*1 > 59 ELSE OUTPUT = -1ELSE OUTPUT = -1

OR

100

10099

X Youtput

0 0 -10 1 11 0 11 1 1

Exclusive OR

X

Y

Output

Neurons

LINK

W1

X Youtput

0 0 -10 1 11 0 11 1 -1

50

50 99

-30

30

30 59

Exclusive-OR

W2

W3

W4

W5


Appropriate Problems for Neural Networks

Instances where there are vectorsvectors of many defined features (eg. meaurements)

Output may be a discrete value or a vector of discrete values

Training examples may contain errors Non-trivial training sets imply non-trivial

time for training Very fast application of the learned network

to a subsequent instance We don’t have to understand the learned

function – only the learned rules


How Are ANNs Trained?

Initially choose small random weights (wi) Set threshold = 1 Choose small learning rate (r)

Apply each member of the training set to the neural net model using the training rule to adjust the weights


The Training Rule Explained

Modify the weights (wi) according to the Training Rule:

Here – r is the learning rate (eg. 0.2) t = target output a = actual output xi = i-th input value

wi = wi + wi wherewi = r * (t – a) * xi


Training for ‘OR’

Training Set:X1 X2

target 0 0 -1 0 1 1 1 0 1 1 1 1

Initial Random Weights

W1 = .3W2 = .7

Learning Rate

r = .2


Applying the Training Set for OR - 1

X1

X21 a

0 0 = -10 1 = -1 X

w1 = r * (t – a) * x1= .2 * (1-(-1)) * x1

= .2 * (2) * 0= 0

w2 = .2 * (1-(-1)) * x2

= .2 * (2) * 1= .4

w1 = w1 + w1

= .3 + 0 = .3

w2 = w2 + w2

= .7 +.4 = 1.1

.3

.7



X1

X21 a

0 0 = -10 1 = +11 0 = -1 X

w1 = r * (t – a) * x1= .2 * (1-(-1)) * x1

= .2 * (2) * 1= .4

w2 = .2 * (1-(-1)) * x2

= .2 * (2) * 0= 0

w1 = w1 + w1

= .3 + .4 = .7

w2 = w2 + w2

= 1.1+0 = 1.1

.3

1.1



X1

X21 a

0 0 = -10 1 = +11 0 = -1 X

w1 = r * (t – a) * x1= .2 * (1-(-1)) * x1

= .2 * (2) * 1= .4

w2 = .2 * (1-(-1)) * x2

= .2 * (2) * 0= 0

w1 = w1 + w1

= .7+.4 = 1.1

w2 = w2 + w2

= 1.1+0 = 1.1

.7

1.1



X1

X21 a

0 0 = -10 1 = +11 0 = +11 1 = +1

1.1

1.1


Training for ‘AND’

Training Set:X1 X2

target 0 0 -1 0 1 -1 1 0 -1 1 1 1

Initial Random Weights

W1 = .3W2 = .7

Learning Rate

r = .2


Applying the Training Set for AND - 1

X1

X21 a

0 0 = -10 1 = -11 0 = -11 1 = -1 X

w1 = r * (t – a) * x1= .2 * (1-(-1)) * 1= .4

w2 = .2 * (1-(-1)) * 1= .4

w1 = w1 + w1

= .3 + .4 = .7

w2 = w2 + w2

= .7 +.4 = 1.1

.3

.7



X1

X21 a

0 0 = -10 1 = +1 X

w1 = r * (t – a) * x1= .2 * (-1-(+1)) * 0= 0

w2 = .2 * (-1-(+1)) * 1= -.4

w1 = w1 + w1

= .7 + 0 = .7

w2 = w2 + w2

= 1.1 -.4 = .7

.7

1.1



X1

X21 a

0 0 = -10 1 = -11 0 = -11 1 = +1

.7

.7


Applying the Technology

Date #Trans CPUBusy RespTime DiskBusy NetBusy01-Oct-93 28 3 9 71 302-Oct-93 140 80 6 90 403-Oct-93 156 87 4 12 504-Oct-93 187 95 7 69 505-Oct-93 226 40 0 16 506-Oct-93 288 16 5 40 607-Oct-93 309 10 2 64 608-Oct-93 449 84 4 18 809-Oct-93 453 89 3 32 810-Oct-93 481 77 2 44 811-Oct-93 535 23 8 61 812-Oct-93 609 37 3 86 913-Oct-93 658 58 9 51 914-Oct-93 739 33 8 25 915-Oct-93 776 25 1 34 10


Select The Data Set

Choose data for the Neugent


Select The Output That You Want to Predict

Choose Inputs

Identify the Outputs


Train And Validate the Neugent

Choose Action to be performed:

• Create the model (Quick Train)

• Train & Validate (to understand the predictive capability)

• Investigate the data (Export to Excel or Data Analysis)


Validate the Neugent With the Data Set

Selecting Training Data –

• Select a random sample percentage

• Use the entire data set


Neugent Model is Trained, Tested, and Validated

Training Results –

•Model Fit: 99.598%(trained model quality)

•Predictive Capability: 99.598%(tested model quality)


View The Results in Excel

Consult trained Neugent for prediction

Save results using Excel


Data Analysis

Stats & Filtering: mean, min, max, std dev, filtering constraints

Ranking: input significance

Correlation Matrix: corr. between all fields


Correlation Matrix

The closer to 1, the stronger the indication that the information represented by the two fields is the same

NetBusy vs #Trans = .9966


Actual Vs Predicted

Net Busy: Actual Vs Predicted

0

20

40

60

80

100

120

10

/1/9

3

10

/15

/93

10

/29

/93

11

/12

/93

11

/26

/93

12

/10

/93

12

/24

/93

1/7

/94

1/2

1/9

4

2/4

/94

2/1

8/9

4

Date

Us

ag

e

NetBusy_actual

NetBusy_predicted


Actual Vs Predicted

Test results:label NetBusy_actual NetBusy_predicted

01-Oct-93 3 3.9391602-Oct-93 4 3.5574803-Oct-93 5 6.0737704-Oct-93 5 5.2292805-Oct-93 5 4.6911606-Oct-93 6 4.799707-Oct-93 6 7.0891208-Oct-93 8 7.3507309-Oct-93 8 5.742410-Oct-93 8 7.9224611-Oct-93 8 7.9441212-Oct-93 9 9.0207813-Oct-93 9 9.5198914-Oct-93 9 8.7129


Summary

Neural NetworksModeled after neurons in the brainArtificial neurons are simpleNeurons can be trainedNetworks of neurons can be taught

how to respond to inputModels can be built quicklyAccurate predictions can be made


Questions?

Questions, comments, … ??

Finding me –Dr Bernie Domanski Email: [email protected] Website: http://domanski.cs.csi.cuny.edu Phone: (718) 982-2850 Fax: 2356

Thanks for coming and listening !

Neural Networks Primer Dr Bernie Domanski The City University of New York / CSI 2800 Victory Blvd 1N-215 Staten Island, New York 10314 [email protected].

Documents