Intro deep learning

M. R. Avendi CPCC, UC Irvine

Nov. 2014

Deep Learning, Trends, and Advances

Outline Introduction and Motivations Machine Learning and Challenges Neuroscience Experiments Neural Networks and Optimization Deep Networks and Advances Summary

Machine Learning Supervised: labeled data, eg. spam filtering Unsupervised: unlabeled data, eg. clustering

Typical Applications

Images, Behind the Scene

Image Classification

Image Classification

What are features!?

Feature Extraction Hard , time consuming, requires knowledge Human brain does feature extraction

Neuroscience Experiment, (1992)

Auditory cortex learns to see!

Roe, Anna W., et al. "Visual projections routed to the auditory pathway in ferrets: receptive fields of

visual neurons in primary auditory cortex." The Journal of neuroscience 12.9 (1992): 3651-3664.

Seeing With Tongue Blind people can see using tongue http://www.wicab.com/en_us/press.html

One Learning Algorithm Hypothesis

We want: Automatic feature learning Training data Unlabeled: whatever, we have a lot! Labeled: small!

Mimicking Brain: Neural Networks Perceptron: one-layer NN Parameters : w, not known, training Activation function: f(x) =f(wi xi +w0)

Training One Layer Network Training data: input x, output y

• Reference: Deep Learning and Neural Networks, by Kevin Duh, http://cl.naist.jp/~kevinduh/a/deep2014/

Training: Gradient Descent

• Reference: Deep Learning and Neural Networks, by Kevin Duh, http://cl.naist.jp/~kevinduh/a/deep2014/

Two-Layer Network

• Reference: Deep Learning and Neural Networks, by Kevin Duh, http://cl.naist.jp/~kevinduh/a/deep2014/

Training: Backpropagation

• Reference: Deep Learning and Neural Networks, by Kevin Duh, http://cl.naist.jp/~kevinduh/a/deep2014/

Training: Backpropagation

• Reference: Deep Learning and Neural Networks, by Kevin Duh, http://cl.naist.jp/~kevinduh/a/deep2014/

Multi-Layer Network: Dark Ages

• Reference: Deep Learning and Neural Networks, by Kevin Duh, http://cl.naist.jp/~kevinduh/a/deep2014/

Breakthrough, [Hinton, et al., 2006] Layer-Wise Pre-Training, unsupervised Optimize likelihood of data, P(x)

• Reference: Deep Learning and Neural Networks, by Kevin Duh, http://cl.naist.jp/~kevinduh/a/deep2014/

Breakthrough, cnt. Fine-tune using labeled data, supervised

• Reference: Deep Learning and Neural Networks, by Kevin Duh, http://cl.naist.jp/~kevinduh/a/deep2014/

Deep Learning Approaches Deep Belief Networks RBM: learns data likelihood Stacked RBMs

Deep Learning Approaches Stacked Autoencoders Autoencoders: learns to reconstruct input data Easier to train

• Reference: Deep Learning tutorial, Andrew Ng

Convolutional Networks

• Reference: Deep Learning tutorial, Andrew Ng

Extracted Features Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hierarchical Representations[Lee et

al., 2009]

Deep Learning: advances Microsoft real-time speech translation https://www.youtube.com/watch?v=NhxCg2PA3ZI

Deep Learning: advances Google artificial brain learns to find cat and face NN, 1 billion connection, 16000 computers, browse YouTube

for 3 days

Others Google+ Image Search, no-tag image search Handwriting recognition Android speech to text Medical Diagnosis

Summary

• Reference: Deep Learning and Neural Networks, by Kevin Duh, http://cl.naist.jp/~kevinduh/a/deep2014/