Convolutional Neural Networks at scale in Spark MLlib

Spark Technology Center

Convolutional Neural Networks at Scale in MLlib

Jeremy Nixon


1. Machine Learning Engineer at the Spark Technology Center

2. Contributor to MLlib, dedicated to scalable deep learning.

3. Previously, studied Applied Mathematics to Computer Science and Economics at Harvard

Jeremy Nixon

Future Work1. Convolutional Neural Networks

a. Convolutional Layer Typeb. Max Pooling Layer Type

2. Flexible Deep Learning API3. More Modern Optimizers

a. Adamb. Adadelta + Nesterov Momentum

4. More Modern activations5. Dropout / L2 Regularization6. Batch Normalization7. Tensor Support8. Recurrent Neural Networks (LSTM)


1. Framing Deep Learning2. MLlib Deep Learning API3. Optimization4. Performance5. Future Work

Structure


1. Structural Assumptions2. Automated Feature Engineering3. Learning Representations4. Applications

Framing Convolutional Neural Networks


- Network depth creates an extraordinary range of possible models.

- That flexibility creates value in large datasets to reduce variance.

Structural Assumptions:

Combinatorial Flexibility


X = Normalized Data, W1, W2 = Weights

Forward:

1. Multiply data by first layer weights | (X*W1)2. Put output through non-linear activation | max(0, X*W1)3. Multiply output by second layer weights | max(0, X*W1) *

W24. Return predicted output

Structural Assumption:The Model


- Pixels - Edges - Shapes - Parts - Objects- Learn features that are optimized for the

data- Makes transfer learning feasible


Hierarchical Abstraction



Location Invariance

- Convolution is a restriction on the features that can be combined.

- Location Invariance leads to strong accuracy in vision, audio, and language.

colah.github.io


Automated Feature Engineering


Learning Representations

Hidden Layer+

Nonlinearity

http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/


1. CNNs - State of the arta. Object Recognitionb. Object Localizationc. Image Segmentationd. Image Restoratione. Music Recommendation

2. RNNs (LSTM) - State of the Arta. Speech Recognitionb. Question Answeringc. Machine Translationd. Text Summarizatione. Named Entity Recognitionf. Natural Language Generation

g. Word Sense Disambiguationh. Image / Video Captioningi. Sentiment Analysis

Applications


Flexibility. High level enough to be efficient. Low level enough to be expressive.

MLlib Flexible Deep Learning API


Flexibility. High level enough to be efficient. Low level enough to be expressive.



Modularity enables Logistic Regression, Feedforward Networks.



Introducing Convolutional and Max-Pooling Layer types.

MLlib Convolutional Neural Network


Optimization


Optimization


Parallel implementation of backpropagation:

1. Each worker gets weights from master node.

2. Each worker computes a gradient on its data.

3. Each worker sends gradient to master.4. Master averages the gradients and

updates the weights.

Distributed Optimization


● Parallel MLP on Spark with 7 nodes ~= Caffe w/GPU (single node).

● Advantages to parallelism diminish with additional nodes due to communication costs.

● Additional workers are valuable up to ~20 workers.

● See https://github.com/avulanov/ann-benchmark for more details

Performance

https://github.com/avulanov/ann-benchmark




Github: https://github.com/JeremyNixon/sparkdl

Spark Package: https://spark-packages.org/package/JeremyNixon/sparkdl

Access

https://github.com/JeremyNixon/sparkdl


1. GPU Acceleration (External)2. Keras Integration3. Residual Layers4. Hardening5. Regularization6. Batch Normalization7. Tensor Support

Future Work


Thank you for your attention!

Questions?