Convolutional Neural Networks at scale in Spark MLlib
Post on 28-Jan-2018
974 Views
Preview:
Transcript
Spark Technology Center
1. Machine Learning Engineer at the Spark Technology Center
2. Contributor to MLlib, dedicated to scalable deep learning.
3. Previously, studied Applied Mathematics to Computer Science and Economics at Harvard
Jeremy Nixon
Future Work1. Convolutional Neural Networks
a. Convolutional Layer Typeb. Max Pooling Layer Type
2. Flexible Deep Learning API3. More Modern Optimizers
a. Adamb. Adadelta + Nesterov Momentum
4. More Modern activations5. Dropout / L2 Regularization6. Batch Normalization7. Tensor Support8. Recurrent Neural Networks (LSTM)
Spark Technology Center
1. Framing Deep Learning2. MLlib Deep Learning API3. Optimization4. Performance5. Future Work
Structure
Spark Technology Center
1. Structural Assumptions2. Automated Feature Engineering3. Learning Representations4. Applications
Framing Convolutional Neural Networks
Spark Technology Center
- Network depth creates an extraordinary range of possible models.
- That flexibility creates value in large datasets to reduce variance.
Structural Assumptions:
Combinatorial Flexibility
Spark Technology Center
X = Normalized Data, W1, W2 = Weights
Forward:
1. Multiply data by first layer weights | (X*W1)2. Put output through non-linear activation | max(0, X*W1)3. Multiply output by second layer weights | max(0, X*W1) *
W24. Return predicted output
Structural Assumption:The Model
Spark Technology Center
- Pixels - Edges - Shapes - Parts - Objects- Learn features that are optimized for the
data- Makes transfer learning feasible
Structural Assumptions:
Hierarchical Abstraction
Spark Technology Center
Structural Assumptions:
Location Invariance
- Convolution is a restriction on the features that can be combined.
- Location Invariance leads to strong accuracy in vision, audio, and language.
colah.github.io
Spark Technology Center
Learning Representations
Hidden Layer+
Nonlinearity
http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/
Spark Technology Center
1. CNNs - State of the arta. Object Recognitionb. Object Localizationc. Image Segmentationd. Image Restoratione. Music Recommendation
2. RNNs (LSTM) - State of the Arta. Speech Recognitionb. Question Answeringc. Machine Translationd. Text Summarizatione. Named Entity Recognitionf. Natural Language Generation
g. Word Sense Disambiguationh. Image / Video Captioningi. Sentiment Analysis
Applications
Spark Technology Center
Flexibility. High level enough to be efficient. Low level enough to be expressive.
MLlib Flexible Deep Learning API
Spark Technology Center
Flexibility. High level enough to be efficient. Low level enough to be expressive.
MLlib Flexible Deep Learning API
Spark Technology Center
Modularity enables Logistic Regression, Feedforward Networks.
MLlib Flexible Deep Learning API
Spark Technology Center
Introducing Convolutional and Max-Pooling Layer types.
MLlib Convolutional Neural Network
Spark Technology Center
Parallel implementation of backpropagation:
1. Each worker gets weights from master node.
2. Each worker computes a gradient on its data.
3. Each worker sends gradient to master.4. Master averages the gradients and
updates the weights.
Distributed Optimization
Spark Technology Center
● Parallel MLP on Spark with 7 nodes ~= Caffe w/GPU (single node).
● Advantages to parallelism diminish with additional nodes due to communication costs.
● Additional workers are valuable up to ~20 workers.
● See https://github.com/avulanov/ann-benchmark for more details
Performance
Spark Technology Center
Github: https://github.com/JeremyNixon/sparkdl
Spark Package: https://spark-packages.org/package/JeremyNixon/sparkdl
Access
Spark Technology Center
1. GPU Acceleration (External)2. Keras Integration3. Residual Layers4. Hardening5. Regularization6. Batch Normalization7. Tensor Support
Future Work
top related