Top Banner
Machine Learning Neural Networks: Introduction 1 Based on slides and material from Geoffrey Hinton, Richard Socher, Dan Roth, Yoav Goldberg, Shai Shalev-Shwartz and Shai Ben-David, and others
30

Neural Networks: Introduction - svivek · Neural networks A robust approach for approximating real-valued, discrete-valued or vector valued functions Among the most effective general

May 20, 2020

Download

Documents

dariahiddleston
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Neural Networks: Introduction - svivek · Neural networks A robust approach for approximating real-valued, discrete-valued or vector valued functions Among the most effective general

MachineLearning

NeuralNetworks:Introduction

1BasedonslidesandmaterialfromGeoffreyHinton,RichardSocher,DanRoth,Yoav Goldberg,ShaiShalev-Shwartz andShaiBen-David,andothers

Page 2: Neural Networks: Introduction - svivek · Neural networks A robust approach for approximating real-valued, discrete-valued or vector valued functions Among the most effective general

Wherearewe?

Generallearningprinciples• Overfitting• Mistake-boundlearning• PAClearning,samplecomplexity• Hypothesischoice&VCdimensions• Trainingandgeneralizationerrors• RegularizedEmpiricalLoss

Minimization• BayesianLearning

Learningalgorithms• DecisionTrees• Perceptron• AdaBoost• SupportVectorMachines• NaïveBayes• LogisticRegression

4

Producelinearclassifiers

Page 3: Neural Networks: Introduction - svivek · Neural networks A robust approach for approximating real-valued, discrete-valued or vector valued functions Among the most effective general

NeuralNetworks

• Whatisaneuralnetwork?

• Predictingwithaneuralnetwork

• Trainingneuralnetworks

• Practicalconcerns

6

Page 4: Neural Networks: Introduction - svivek · Neural networks A robust approach for approximating real-valued, discrete-valued or vector valued functions Among the most effective general

Thislecture

• Whatisaneuralnetwork?– Thehypothesisclass– Structure,expressiveness

• Predictingwithaneuralnetwork

• Trainingneuralnetworks

• Practicalconcerns

7

Page 5: Neural Networks: Introduction - svivek · Neural networks A robust approach for approximating real-valued, discrete-valued or vector valued functions Among the most effective general

Wehaveseenlinearthresholdunits

11

features

dotproduct

threshold

Predictionsgn(&'( + *) = sgn(∑./0/ + *)

Learningvariousalgorithmsperceptron,SVM,logisticregression,…

ingeneral,minimizeloss

Butwheredotheseinputfeaturescomefrom?

Whatifthefeatureswereoutputsofanotherclassifier?

Page 6: Neural Networks: Introduction - svivek · Neural networks A robust approach for approximating real-valued, discrete-valued or vector valued functions Among the most effective general

Featuresfromclassifiers

12

Page 7: Neural Networks: Introduction - svivek · Neural networks A robust approach for approximating real-valued, discrete-valued or vector valued functions Among the most effective general

Featuresfromclassifiers

13

Page 8: Neural Networks: Introduction - svivek · Neural networks A robust approach for approximating real-valued, discrete-valued or vector valued functions Among the most effective general

Featuresfromclassifiers

14

Eachoftheseconnectionshavetheirownweightsaswell

Page 9: Neural Networks: Introduction - svivek · Neural networks A robust approach for approximating real-valued, discrete-valued or vector valued functions Among the most effective general

Featuresfromclassifiers

15

Page 10: Neural Networks: Introduction - svivek · Neural networks A robust approach for approximating real-valued, discrete-valued or vector valued functions Among the most effective general

Featuresfromclassifiers

16

Thisisatwolayerfeedforwardneuralnetwork

Page 11: Neural Networks: Introduction - svivek · Neural networks A robust approach for approximating real-valued, discrete-valued or vector valued functions Among the most effective general

Featuresfromclassifiers

17

Theoutputlayer

ThehiddenlayerTheinputlayer

Thisisatwolayerfeedforwardneuralnetwork

Thinkofthehiddenlayeraslearningagoodrepresentationoftheinputs

Page 12: Neural Networks: Introduction - svivek · Neural networks A robust approach for approximating real-valued, discrete-valued or vector valued functions Among the most effective general

Featuresfromclassifiers

19

Thedotproductfollowedbythethresholdconstitutesaneuron

Fiveneuronsinthispicture(fourinhiddenlayerandoneoutput)

Thisisatwolayerfeedforwardneuralnetwork

Page 13: Neural Networks: Introduction - svivek · Neural networks A robust approach for approximating real-valued, discrete-valued or vector valued functions Among the most effective general

Butwheredotheinputscomefrom?

20

Whatiftheinputsweretheoutputsofaclassifier?Theinputlayer

Wecanmakeathree layernetwork….Andsoon.

Page 14: Neural Networks: Introduction - svivek · Neural networks A robust approach for approximating real-valued, discrete-valued or vector valued functions Among the most effective general

Letustrytoformalizethis

21

Page 15: Neural Networks: Introduction - svivek · Neural networks A robust approach for approximating real-valued, discrete-valued or vector valued functions Among the most effective general

Neuralnetworks

Arobustapproachforapproximatingreal-valued,discrete-valuedorvectorvaluedfunctions

Amongthemosteffectivegeneralpurpose supervisedlearningmethodscurrentlyknown

Especiallyforcomplexandhardtointerpretdatasuchasreal-worldsensorydata

TheBackpropagationalgorithmforneuralnetworkshasbeenshownsuccessfulinmanypracticalproblems

Acrossvariousapplicationdomains

22

Page 16: Neural Networks: Introduction - svivek · Neural networks A robust approach for approximating real-valued, discrete-valued or vector valued functions Among the most effective general

Artificialneurons

Functionsthatverylooselymimicabiologicalneuron

Aneuronacceptsacollectionofinputs(avectorx)andproducesanoutputby:

1. Applyingadotproductwithweightsw andaddingabiasb2. Applyinga(possiblynon-linear)transformationcalledanactivation

25

123423 = activation(&'( + *)

Page 17: Neural Networks: Introduction - svivek · Neural networks A robust approach for approximating real-valued, discrete-valued or vector valued functions Among the most effective general

Artificialneurons

Functionsthatverylooselymimicabiologicalneuron

Aneuronacceptsacollectionofinputs(avectorx)andproducesanoutputby:

1. Applyingadotproductwithweightsw andaddingabiasb2. Applyinga(possiblynon-linear)transformationcalledanactivation

27

Dotproduct

Thresholdactivation

Otheractivationsarepossible

123423 = activation(&'( + *)

Page 18: Neural Networks: Introduction - svivek · Neural networks A robust approach for approximating real-valued, discrete-valued or vector valued functions Among the most effective general

Activationfunctions

Nameoftheneuron Activationfunction:activation(;)Linearunit ;Threshold/sign unit sgn(;)

Sigmoidunit1

1 + exp(−;)Rectifiedlinearunit(ReLU) max(0, ;)Tanh unit tanh(;)

28

123423 = activation(&'( + *)

Manymoreactivationfunctionsexist(sinusoid,sinc,gaussian,polynomial…)

Alsocalledtransferfunctions

Page 19: Neural Networks: Introduction - svivek · Neural networks A robust approach for approximating real-valued, discrete-valued or vector valued functions Among the most effective general

Aneuralnetwork

Afunctionthatconvertsinputstooutputsdefinedbyadirectedacyclicgraph

– Nodesorganizedinlayers,correspondtoneurons

– Edgescarryoutputofoneneurontoanother,associatedwithweights

• Todefineaneuralnetwork,weneedtospecify:– Thestructureofthegraph

• Howmanynodes,theconnectivity– Theactivationfunctiononeachnode– Theedgeweights

30

Input

Hidden

Output

wFGH

wFGI

Page 20: Neural Networks: Introduction - svivek · Neural networks A robust approach for approximating real-valued, discrete-valued or vector valued functions Among the most effective general

Aneuralnetwork

Afunctionthatconvertsinputstooutputsdefinedbyadirectedacyclicgraph

– Nodesorganizedinlayers,correspondtoneurons

– Edgescarryoutputofoneneurontoanother,associatedwithweights

• Todefineaneuralnetwork,weneedtospecify:– Thestructureofthegraph

• Howmanynodes,theconnectivity– Theactivationfunctiononeachnode– Theedgeweights

31

Input

Hidden

Output

wFGH

wFGI

Page 21: Neural Networks: Introduction - svivek · Neural networks A robust approach for approximating real-valued, discrete-valued or vector valued functions Among the most effective general

Aneuralnetwork

Afunctionthatconvertsinputstooutputsdefinedbyadirectedacyclicgraph

– Nodesorganizedinlayers,correspondtoneurons

– Edgescarryoutputofoneneurontoanother,associatedwithweights

• Todefineaneuralnetwork,weneedtospecify:– Thestructureofthegraph

• Howmanynodes,theconnectivity– Theactivationfunctiononeachnode– Theedgeweights

32

CalledthearchitectureofthenetworkTypicallypredefined,partofthedesignoftheclassifier

Input

Hidden

Output

wFGH

wFGI

Page 22: Neural Networks: Introduction - svivek · Neural networks A robust approach for approximating real-valued, discrete-valued or vector valued functions Among the most effective general

Aneuralnetwork

Afunctionthatconvertsinputstooutputsdefinedbyadirectedacyclicgraph

– Nodesorganizedinlayers,correspondtoneurons

– Edgescarryoutputofoneneurontoanother,associatedwithweights

• Todefineaneuralnetwork,weneedtospecify:– Thestructureofthegraph

• Howmanynodes,theconnectivity– Theactivationfunctiononeachnode– Theedgeweights

33

CalledthearchitectureofthenetworkTypicallypredefined,partofthedesignoftheclassifier

Learnedfromdata

Input

Hidden

Output

wFGH

wFGI

Page 23: Neural Networks: Introduction - svivek · Neural networks A robust approach for approximating real-valued, discrete-valued or vector valued functions Among the most effective general

Abriefhistoryofneuralnetworks

• 1943:McCulloughandPittsshowedhowlinearthresholdunitscancomputelogicalfunctions

• 1949:Hebbsuggestedalearningrulethathassomephysiologicalplausibility

• 1950s:Rosenblatt,thePeceptron algorithmforasinglethresholdneuron

• 1969:MinskyandPapert studiedtheneuronfromageometricalperspective

• 1980s:Convolutionalneuralnetworks(Fukushima,LeCun),thebackpropagationalgorithm(various)

• Early2000s-today:Morecompute,moredata,deepernetworks

34Seealso:http://people.idsia.ch/~juergen/deep-learning-overview.html

very

Page 24: Neural Networks: Introduction - svivek · Neural networks A robust approach for approximating real-valued, discrete-valued or vector valued functions Among the most effective general

Whatfunctionsdoneuralnetworksexpress?

35

Page 25: Neural Networks: Introduction - svivek · Neural networks A robust approach for approximating real-valued, discrete-valued or vector valued functions Among the most effective general

Asingleneuronwiththresholdactivation

36

Prediction=sgn(b+w1 x1 +w2x2)

++

++

+ +++

-- --

-- -- --

---- --

--

b+w1 x1 +w2x2=0

Page 26: Neural Networks: Introduction - svivek · Neural networks A robust approach for approximating real-valued, discrete-valued or vector valued functions Among the most effective general

Twolayers,withthresholdactivations

37

Ingeneral,convexpolygons

FigurefromShaiShalev-Shwartz andShaiBen-David,2014

Page 27: Neural Networks: Introduction - svivek · Neural networks A robust approach for approximating real-valued, discrete-valued or vector valued functions Among the most effective general

Threelayerswiththresholdactivations

38

Ingeneral,unionsofconvexpolygons

FigurefromShaiShalev-Shwartz andShaiBen-David,2014

Page 28: Neural Networks: Introduction - svivek · Neural networks A robust approach for approximating real-valued, discrete-valued or vector valued functions Among the most effective general

Neuralnetworksareuniversalfunctionapproximators

• Anycontinuousfunctioncanbeapproximatedtoarbitraryaccuracyusingonehiddenlayerofsigmoidunits[Cybenko 1989]

• Approximationerrorisinsensitivetothechoiceofactivationfunctions[DasGupta etal1993]

• Twolayerthreshold networkscanexpressanyBooleanfunction– Exercise:Provethis

• VCdimensionofthresholdnetworkwithedgesE:JK = L(|N|log|N|)

• VCdimensionofsigmoidnetworkswithnodesVandedgesE:– Upperbound:Ο J H N H

– Lowerbound:Ω N H

39

Exercise:Showthatifwehaveonlylinearunits,thenmultiplelayersdoesnotchangetheexpressiveness

Page 29: Neural Networks: Introduction - svivek · Neural networks A robust approach for approximating real-valued, discrete-valued or vector valued functions Among the most effective general

Neuralnetworksareuniversalfunctionapproximators

• Anycontinuousfunctioncanbeapproximatedtoarbitraryaccuracyusingonehiddenlayerofsigmoidunits[Cybenko 1989]

• Approximationerrorisinsensitivetothechoiceofactivationfunctions[DasGupta etal1993]

• Twolayerthreshold networkscanexpressanyBooleanfunction– Exercise:Provethis

• VCdimensionofthresholdnetworkwithedgesE:JK = L(|N|log|N|)

• VCdimensionofsigmoidnetworkswithnodesVandedgesE:– Upperbound:Ο J H N H

– Lowerbound:Ω N H

40

Page 30: Neural Networks: Introduction - svivek · Neural networks A robust approach for approximating real-valued, discrete-valued or vector valued functions Among the most effective general

Neuralnetworksareuniversalfunctionapproximators

• Anycontinuousfunctioncanbeapproximatedtoarbitraryaccuracyusingonehiddenlayerofsigmoidunits[Cybenko 1989]

• Approximationerrorisinsensitivetothechoiceofactivationfunctions[DasGupta etal1993]

• Twolayerthreshold networkscanexpressanyBooleanfunction– Exercise:Provethis

• VCdimensionofthresholdnetworkwithedgesE:JK = L(|N|log|N|)

• VCdimensionofsigmoidnetworkswithnodesVandedgesE:– Upperbound:Ο J H N H

– Lowerbound:Ω N H

41

Exercise:Showthatifwehaveonlylinearunits,thenmultiplelayersdoesnotchangetheexpressiveness


Related Documents
Research Article A Real Valued Neural Network Based …downloads.hindawi.com/journals/isrn/2014/579125.pdf · 2017. 12. 4. · Research Article A Real Valued Neural Network Based

Research Article A Real Valued Neural Network Based...

Category: Documents
Application: Semantic Role Labeling - svivek

Application: Semantic Role Labeling - svivek

Category: Documents
Discrete-Valued Neural Communication

Discrete-Valued Neural Communication

Category: Documents
valued neutrosophic graphs Introduction of some …fs.unm.edu/neut/IntroductionOfSomeNewResults.pdfintuitionistic fuzzy, interval-valued fuzzy, interval-valued intuitionistic fuzzy

valued neutrosophic graphs Introduction of some...

Category: Documents
ARTIFICIAL NEURAL NETWORKS - McGill CIMjer/courses/ai/readings/MitchellCh4.pdf · 4.1 INTRODUCTION Neural network learning methods provide a robust approach to approximating real-valued,

ARTIFICIAL NEURAL NETWORKS - McGill...

Category: Documents
CONTINGENCY ANALYSIS USING COMPLEX …shodhganga.inflibnet.ac.in/bitstream/10603/2277/11/11...50 CHAPTER 3 CONTINGENCY ANALYSIS USING COMPLEX VALUED NEURAL NETWORK 3.1 INTRODUCTION

CONTINGENCY ANALYSIS USING COMPLEX...

Category: Documents
KNOWLEDGE REDUCTION IN LATTICE-VALUED …pdfs.semanticscholar.org/2be6/40b1b32ec320f9f0b3ac1e822d52b4a30f2d.pdfKNOWLEDGE REDUCTION IN LATTICE-VALUED INFORMATION SYSTEMS WITH INTERVAL-VALUED

KNOWLEDGE REDUCTION IN LATTICE-VALUED...

Category: Documents
Basics of Neural Network ProgrammingNeural network programming guideline Whenever possible, avoid explicit for-loops. Andrew Ng Vectors and matrix valued functions Say you need to

Basics of Neural Network ProgrammingNeural network...

Category: Documents
Valued connections

Valued connections

Category: Business
Nonlinear System Identification Using Takagi-Sugeno … · Abstract— In this paper, we propose a stable learning mechanism for novel Takagi-Sugeno-Kang type interval-valued neural

Nonlinear System Identification Using Takagi-Sugeno … ·....

Category: Documents
Recurrent Neural Networks - UVicleizhao/Reading/chapter/07.pdf · 272 CHAPTER 7. RECURRENT NEURAL NETWORKS The individual values in a sequence can be either real-valued or symbolic.

Recurrent Neural Networks -...

Category: Documents
MODULE 3 - Deepak D. · 2019. 9. 16. · MODULE 3 ARTIFICIAL NEURAL NETWORKS INTRODUCTION Artificial neural networks (ANNs) provide a general, practical method for learning real-valued,

MODULE 3 - Deepak D. · 2019. 9. 16. · MODULE 3...

Category: Documents