06logisticregression 150930040919 Lva1 App6891

7/24/2019 06logisticregression 150930040919 Lva1 App6891

http://slidepdf.com/reader/full/06logisticregression-150930040919-lva1-app6891 1/44

Logistic RegressionA Classifcation Algorithm

One o the most popular and mostwidely used learning algorithm tod



0: “Negatie Class! "e#g#$ %tumor&

1: “'ositie Class 1! "e#g#$

tumor&(: “'ositie Class (! e# #$

Classifcation

*mail: +pam , Not +pam-

Online .ransactions: /raudulent "es ,No&- .umor: alignant , 2enign -

0: “Negatie Class! "e#g#$ %tumor&

1: “'ositie Class! "e#g#$ matumor&



.hreshold classifer outputat 0#4:

5 $ predict “y 6 1!




.hreshold classifer outputat 0#4:





Bad thing to do or linear regression

2eore we 9ust got lucy;



Classifcation: y 6 0or 1

can %e < 1 or= 0

Logistic Regression:

Classifcation task





+igmoid unctionLogistic unction

Logistic Regression Model

ant

1

0#

4

0

Need to select parameters so tha

o it with an algorithm later



Interpretation o HypothesisOutput

6 estimated pro%a%ility that y 6 1 on new input D

.ell patient that @0E chance o tumor %ein

malignant

*Dample:5

“pro%a%ility that y 6 gien D$ parameteriFed %y



ecision %oundary



Logistic regression

5

5

or

5

H





eDample with eatures D1$ D( that satisy this eIuation p



D1

D(

Decision Boundary

1 ( 7

1

(

7

'redict “ “ i



Non-linear decisionoundaries

D1

D(

D1

D(

'redict “ “ i

1J1

J1

1



Cost unction

.o ft the parameters



.rainingset:

?ow to choose

parameters -

m eDamples



Cost unction

Linear regression:

“nonJconeDunction!

“coneDunction!

Logistic

KKKKK

L i ti i t



Logistic regression costunction

5 y 6 1

10

Cost




5 y 6 1

10

Cost




5 y 6 0

10

Cost





+implifed cost unction andgradient descent





hy do we chose this unction whe

other cost unctions eDist-• .his cost unction can %e deried

statistics using the principle o

!a"i!u! likelihood esti!ati – An ecient method to fnd parame

data or diMerent models

– 5t is a coneD unction




Output


.o ft parameters :

.o mae a prediction gien new :

?ypothesis estimapro%a%ility that y6



#radient Descent

ant :

Repeat

"simultaneously update all &



#radient Descent

ant :

"simultaneously update all &

Repeat

Algorithm loos identical to linear regression

2ut actually they are ery diMerent rom each



Hypothesis

Cost unction

#radient Descent



Adanced optimiFatio



Opti!i$ation algorith!

Cost unction # ant #

ien $ we hae code that cancompute- -

"or&

Repeat

radient descent:



Opti!i$ation algorith!

ien $ we hae code that cancompute

-

- "or

&Opti!i$ationalgorith!s%

-

radient descent-

NewtonJRaphsons method- Con9ugate gradient- 2/+ "2roydenJ/letcherJoldar%J+hann- LJ2/+ "Limited memory J 2/+&

P N d t ll i l h "l ii$ation algorith!% Con9ugate gradient$ 2/+



P No need to manually pic alpha "learning raP ?ae a cleer inner loop "line search algo

which tries a %unch o alpha alues and p

good oneP Oten aster than gradient descentP Can %e used successully without understan

compleDity

Q ery complicated Q Could mae de%ugging more dicult Q +hould not %e implemented themseles "im

only i you are an eDpert in numerical compu Q iMerent li%raries may use diMerent implem

J

i$ation algorith!% Con9ugate gradient$ 2/+

* l



*Dample: function [jVal, grad

= costFunc

jVal = (theta(1)-5

(theta(2)-

gradient = "eros(

gradient(1) = 2#(

gradient(2) = 2#(

o$tions = o$ti%set(&'radj*, &on*, &ater*, &initial/heta = "eros(2,1)!

[o$t/heta, functionVal, eitFlag]

= f%inunc(0costFunction, initial/heta, o$tio

ction !ini!i$ation unconstrained







gradient(1) = [ ]!

function [jVal, gradient] = costFunction(thet

theta =

jVal = [ ]!

gradient(2) = [ ]!

gradient(n+1) = [ ]

code tocompute

code to

computecode tocompute

code to

compute

pti!i$ation algorith!% !inunc



• Notice that %y using minunc$ you did not h

write any loops yoursel • or set a learning rate lie you did or gradiedescent#

• .his is all done %y minunc• you only needed to proide a unction calc

the cost and the gradient#

pti!i$ation algorith!% !inunc

'rediction



Once you hae optimiFed $ compute:

5 then

else

'rediction



ultiJclass classifcatio

OneJsJall algorithm

Multiclass classifcation



Multiclass classifcation

*mail oldering,tagging: or$ /riends$ /amil?o%%y

edical diagrams: Not ill$ Cold$ /lu

eather: +unny$ Cloudy$ Rain$ +now



D1

D(

D1

D(

2inaryclassifcation:

ultiJclassclassifcation:

One-(s-all )one-



D1

D(

One-(s-all )one-(s-rest*%

Class 1:Class (:Class 7:

One (s all



One-(s-all

.rain a logistic regression classifer

or each class to predict the pro%a%ilthat #On a new input $ to mae aprediction$ pic the class that

maDimiFes

06logisticregression 150930040919 Lva1 App6891

Documents