René Donner Deep Learning
Overview
3
The (amazing) things Deep Learning can do
How does it work?
How can you start with DL?
René Donner Deep Learning
Roughly …
4
Deep learning finds patternsin data corresponding tohigh-level, abstract concepts
What can it do?
René Donner Deep Learning
What it can be used for
6
Image recognition
Text understanding, translation
Voice recognition
Playing video games
Driving cars
…
René Donner Deep Learning
Image recognition
7
René Donner Deep Learning
Scene labeling
8
http://www.purdue.edu/newsroom/releases/2014/Q1/smartphone-to-become-smarter-with-deep-learning-innovation.html
René Donner Deep Learning
Text recognition
9
http://www.pyimagesearch.com/2014/09/22/getting-started-deep-learning-python/
Large-Scale Deep Learning for Intelligent Computer Systems, Jeff Dean, Google, BayLearn 2015
René Donner Deep Learning
Text understanding
10
2013 Glove: Global Vectors for Word Representation, Jeffrey Pennington, Richard Socher and Christopher D. Manning
René Donner Deep Learning
Word embeddings
11
René Donner Deep Learning
Information extraction / Reasoning
12
MetaMind
René Donner Deep Learning
Some well know research groups
13
Stanford / BaiduAndrew Ng
NYU / FacebookYann LeCun
UToronto / GoogleGeoffrey Hinton
René Donner Deep Learning
NVIDIA
14
brand new: M40(same as Geforce GTX Titan X)
Images: NVIDIA website
How does it work?
René Donner Deep Learning
Difference to classic ML
16
http://rinuboney.github.io/2015/10/18/theoretical-motivations-deep-learning.html
René Donner Deep Learning
Deep learning
17
http://theanalyticsstore.ie/deep-learning/
René Donner Deep Learning
Visualization
18
1. Layer
higher Layers
Emergence of Object-Selective Features in Unsupervised Feature Learning, Adam Coates, NIPS 2012
René Donner Deep Learning
Deep learning
19
How does it work?
http://theanalyticsstore.ie/deep-learning/ http://stats.stackexchange.com/questions/114385/what-is-the-difference-between-convolutional-neural-networks-restricted-boltzma
René Donner Deep Learning
Optimization
20
Stochastic gradient descent
Automatic differentiation
blog.datumbox.com
René Donner Deep Learning
Local minima
21
Less problematic than thought - saddle points
https://ganguli-gang.stanford.edu/figures/14.Saddlepoint.jpg
René Donner Deep Learning
Deep learning
22
Low level features of color images
https://www.coursera.org/course/neuralnets
René Donner Deep Learning
Deep learning
23
http://www.pamitc.org/cvpr15/files/lecun-20150610-cvpr-keynote.pdf
René Donner Deep Learning
ImageNet topologies
24
ImageNet Classification with Deep Convolutional Neural Networks", Alex Krizhevsky
“Inception” deep neural network architecture. Source: Christian Szegedy et. al. Going deeper with convolutions. CVPR 2015
René Donner Deep Learning
MNIST - Demo
25
René Donner Deep Learning
MNIST
26
http://deeplearning4j.org/rbm-mnist-tutorial.html
René Donner Deep Learning
Deep learning - why does it work?
27
Can cope with huge amounts of data
Learns small invariances
Overcomplete, sparse, representations
Learn Embedding
Lots of data
Recent advance: it is actually computable!
René Donner Deep Learning
Deep learning - pros
28
Not-domain specific
Supervised / Semi-supervised / Unsupervised
Classification / regression in last layer
Simple math
Hip
René Donner Deep Learning
Deep learning - cons
29
Lots of meta-parameters
Needs a lot of data
Very compute intensive
Hip
Getting started with DL
René Donner Deep Learning
Frameworks
31
Many different DL toolboxes
Efficiency important (GPU)
Attention to numerical issues
René Donner Deep Learning
Frameworks
32
Caffehttp://caffe.berkeleyvision.org/Plain text filesFastest CNN, GPU
Kerashttps://github.com/fchollet/kerasPython, on top of Theano
TensorFlowhttp://tensorflow.org/Python, by Google
MXNethttps://github.com/dmlc/mxnetPython, R, Julia
Slid
e fro
m c
affe
tuto
rial
René Donner Deep Learning
Tensorflow
33
General gradient descent library
René Donner Deep Learning
Tutorials
34
Stanford tutorialhttps://deeplearning.stanford.edu/wiki/index.php/UFLDL_TutorialMatlab code snippets
videolectures.nethttp://videolectures.net/deeplearning2015_montreal/
courserahttps://www.coursera.org/course/neuralnets
René Donner Deep Learning
Practical hints
35
Bengio ArxivPractical Recommendations for Gradient-Based Training of Deep Architectureshttp://arxiv.org/abs/1206.5533http://rinuboney.github.io/2015/10/18/theoretical-motivations-deep-learning.html
Kaggle http://www.kaggle.com/c/galaxy-zoo-the-galaxy-challengehttp://benanne.github.io/2014/04/05/galaxy-zoo.html
Relevant conferences NIPS (https://sites.google.com/site/deeplearningworkshopnips2013/accepted-papers)CVPR, ICMLMany interesting papers on arxiv.org
René Donner Deep Learning
Current research topics
36
Parallelization
What is deep learning, actually?
Alternative, faster, simpler methods
Multi-domain, transfer learning
