[Lecture 2] AI and Deep Learning: Logistic Regression (Theory)

Introduction toArtificial Intelligence & Deep Learning

Lecture 2: Logistic Regression

Dr. Kobkrit Viriyayudhakorn

Following slides are based on

• Andrew Ng’s Coursera Deep Learning course.

• Stanford CS231n: Convolutional Neural Networks for Visual Recognition course.

• Goodwill’s Deep Learning Books

• Prof. Thanaruk’s Introduction to Concepts and Techniques in Data Mining and Application to Text Mining Book.

• Andreas’s Introduction to Machine Learning with Python Book.

• Giancarlo Zaccone’s Getting Start with Tensorflow Book.

• Justine Johnson’s Python Numpy Tutorial

• ETC…

What is Neural Network (Regression)?P

rice

Size of House

What is Neural Network (Classification)?R

ed-d

ish

Round-dish

House Price Prediction with 4-3-1 NN

Size (x1)

#Bed Room (x2)

Wealth (x4)

Zip Code (x3)Price

Supervised Learning in Neural Network

Input (X) Output (Y) Application

House Features Price Real Estate Agent

Patient conditions Disease Physician Assistant

Image 10,000 Objects Photo Recognition

CCTV Camera Footage Person Name / Car License Number

Security / Robot

Audio Text Transcript Speech Recognition, Subtitle Generation

Text of Thai Language Text of English Language Machine Translation

Radar Signal, Images Position of Obstacle Autonomous Driving

Neural Network Example

Feed Forward Neural Network Convolutional Neural Network (CNN)

Recurrent Neural Network (RNN)

Supervised Learning

• Structure Data • Unstructured Data

Sepal Length

SepalWidth

… Species

5.1 3.5 I. setosa

6.3 3.3 I. virginica

Iris Flower Dataset

Why Deep Learning is so popular?

Labelled data (m)

Perf

orm

ance

• Data

• Computation

• Algorithm

Why Deep Learning is taking off?

Logistic Regression

Binary Classification

1 (Flower) or 0 (Non-Flower)

65 10 14 78

43 21 4 25

5 20 130 60

90 120 90 35

78 28 120 30

35 25 1 18

33 12 24 250

65 0 120 45

85 123 4 50

78 88 123 33

Binary Classification

1 (Flower) or 0 (Non-Flower)

255 245 91 128

123 1 244 255

55 80 120 45

85 123 44 39

88 88 123 33

Red

Blue

Green

64

64

Notations

Logistic Regression

Logistic Regression Cost Function

Loss (error) function:

𝑦 = 𝜎 𝑤𝑇𝑥 + 𝑏 𝜎 𝑧 =1

1 + ⅇ−𝑧

𝑥 ⅈ , 𝑦 ⅈ … , 𝑥1𝑚

𝑦 𝑚 𝑦 ⅈ ≈ 𝑦 ⅈ

, where

Given want

Gradient Descent

Want to find W, b that minimize 𝐽 W, 𝑏

𝑦 = 𝜎 𝑊𝑇𝑥 + 𝑏 𝜎 𝑧 =1

1 + ⅇ−𝑧, where

𝐽 W, 𝑏 =1

𝑚

𝑖=1

𝑚

ℒ 𝑦 𝑖 , 𝑦 𝑖 =−1

𝑚

𝑖=1

𝑚

𝑦 𝑖 log 𝑦+𝑖

1 − 𝑦 𝑖 log 1 − 𝑦 𝑖

𝐽 W, 𝑏

𝑊

𝑏

Gradient Descent (1D)

w

Computation Graph

Computing derivatives

b = 3

a = 5

c = 2u = bc

v = a + u J = 3v

11

6

33

Computing derivatives

b = 3

a = 5

c = 2u = bc

v = a + u J = 3v

11

6

33

Logistic Regression Gradient Descent

𝑧 = 𝑤𝑇𝑥 + 𝑏

𝑦 = 𝑎 = 𝜎 𝑧

ℒ 𝑎, 𝑦 = − 𝑦 log 𝑎 + 1 − 𝑦 log 𝑎

Logistic Regression Gradient Descent

𝑧 = 𝑤1𝑥1 + 𝑤2𝑥2 + 𝑏

𝑤1

𝑥1

𝑤2

𝑥2

𝑏

𝑎 = 𝜎 𝑧 ℒ 𝑎, 𝑦

Gradient Descent in 𝑚 examples

What is Vectorization?

Algorithm for Logistic Regression