SMART MANUFACTURING With Apache Spark Streaming and Deep Leaning ApacheCon Big Data Europe Nov 2016, Seville, Spain Prajod Vettiyattil, Wipro @prajods https://in.linkedin.com/in/prajod
SMART MANUFACTURINGWith Apache Spark Streaming and Deep Leaning
ApacheCon Big Data Europe
Nov 2016, Seville, Spain
Prajod Vettiyattil, Wipro
@prajods
https://in.linkedin.com/in/prajod
AGENDA
• Introduction
• Smart Manufacturing
• Applications
• Deep learning solution
• CNN
• Deeplearning4j (DL4J)
• Apache Spark
• Deploying machine learning
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
2
INTRODUCTION
• Industrialization
• Automation
• Scale
• Commoditization
• Accessibility
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
3
MEN AND MACHINES
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
4
KEY POINT
Machines can do what humans can’t
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
5
SMART MANUFACTURING
• Holistic view of entire process
• Sourcing to product sale
• Efficiency
• Eco friendliness
• Predict over react
• Autonomous machines
• Learning and AI
• Use data and analytics
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
6
APPLICATIONS
• Detecting overheating
• Equipment
• Finished products
• Glass defect detection
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
7
OVERHEATING
• Detect temperatures above
defined boundaries
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
8
COOLING TOWER
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
9
ELECTRIC MOTOR
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
10
ELECTRIC MOTOR: OVERHEATING
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
11
OVERHEATING: LOWER RESOLUTION
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
12
ELECTRIC MOTOR: NORMAL VIEW
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
13
GLASS DEFECT
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
14
ZOOMING IN
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
15
UNDER A DIFFERENT LIGHT
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
16
AUTOMATING DETECTION
• Machine learning
• Multivariate Analysis
• Multiple categories of defects
• Deep Learning
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
17
DEEP LEARNING: WHAT
• Neural Network based Machine Learning
• Neural Network• 1 Input layer
• 1 or more hidden layers
• 1 Output layer
• Basic unit of NN• Neuron or Perceptron
• Combine perceptrons in many ways using multiple parameters
• Model parameters
• Hyperparameters
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
18
Image from: ucalgary.ca
DEEP LEARNING: WHY
• Can discern complex input patterns
• Higher accuracy than most other ML methods
• for image analysis
• Ideally suited for spatial data analysis
• Higher time needed for training
• Parallel execution
• Higher level architectures for evolving needs
• CNN, RNN, LSTM
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
19
CONVOLUTIONALNEURAL
NETWORKS
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
20
CONVOLUTIONAL NEURAL NETWORK
• Convolutional Neural Network (CNN)
• Convolve: mix two functions
• Matrix multiplication and addition
• Dot product
• Pass many filters over an image
• Image features
• Images are 3D features
• Discern the important features/signals
• Filter = kernel
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
21
CNN: APPLICATION
• Self driving cars
• Robotics
• Drones
• Industrial automation
• Physical security
• Medical labs
• Wherever images or videos are used
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
22
IMAGE FOR CONVOLUTION
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
23
CONVOLUTIONFilter matrix Image
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
24
CNN ARCHITECTURE
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
25
1st
Co
nv
olu
tio
n
No
nlin
ea
r tr
an
sfo
rm(R
ELU
, ta
nh
etc
)
Po
olin
g
Nth
Co
nv
olu
tio
n
No
nlin
ea
r tr
an
sfo
rm(R
ELU
, ta
nh
etc
)
Po
olin
g
Fu
lly c
on
ne
cte
d la
ye
rs
Ou
tpu
t p
rob
ab
iitie
s
Feature extraction + dimension reduction Classification
…..
CNN PARAMETERS
• Model parameters
• Filter values
• Weights
• Biases
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
26
• Hyperparameters
• Activation function
• # layers
• # nodes in each layer
• Filter size and count
• Regularization
• Convolution layers
• Maxpooling layers
Learned by NN Experience of designer
CNN: SAMPLE CODE• MultiLayerConfiguration conf = new NeuralNetConfiguration.Builder()
.seed(seed).iterations(iterations).regularization(false).l2(0.003).activation(“relu")
.learningRate(0.0007).weightInit(WeightInit.XAVIER)
.optimizationAlgo(OptimizationAlgorithm.STOCHASTIC_GRADIENT_DESCENT)
.updater(Updater.RMSPROP).momentum(0.9).list()
.layer(0, convInit("cnn1", channels, 50 , new int[]{5, 5}, new int[]{1, 1}, new int[]{0, 0}, 0))
.layer(1, maxPool("maxpool1", new int[]{2,2}))
….
.layer(4, new DenseLayer.Builder().nOut(500).build())
.layer(5, new OutputLayer.Builder(LossFunctions.LossFunction.NEGATIVELOGLIKELIHOOD)
.nOut(numLabels).activation("softmax").build())
.backprop(true).pretrain(false).cnnInputSize(height, width, channels).build();
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
27
DEEPLEARNING4J (DL4J)
• Deep learning library
• Open source
• Apache 2.0 license
• Java based
• Distributed execution
• Runs on Spark and Hadoop
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
28
DEEPLEARNING4J: FEATURES
• ND4J
• N Dimensional Arrays/Tensors
• Like numpy
• Canova/DataVec
• Data extraction, vectorization
• Arbitrer
• Evaluate and tune models
• CPU or GPU
• OpenBLAS, Intel MKL, Nvidia CUDA
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
29
DEEPLEARNING4J: SUPPORT
• Community support
• https://gitter.im/deeplearning4j/deeplearning4j
• Commercial support
• www.skymind.io
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
30
APACHE SPARK
• Incubated in 2009 at Berkeley University
• 100s of contributors
• Yahoo, Intel, UC Berkeley,…,50+ orgs
• 10 to 100 times faster than Hadoop MR• https://databricks.com/blog/2014/10/10/spark-petabyte-sort.html
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
31
APACHE SPARK
• Complements Hadoop
• Replaces Hadoop MR
• Adds• In memory processing
• Stream processing
• Interactive queries
• YARN or Mesos for clustering
• Java, Scala, Python, R
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
32
SPARK: LOGICAL VIEW
Apache Spark Core
Spark SQL Spark Streaming MLlib GraphX
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
33
SPARK: DEPLOYMENT VIEW
Worker Node
Executor
Master Node
Executor
Task CacheTaskTask
Worker Node
ExecutorExecutorExecutor
TaskTaskTaskCache
Spark’s
Cluster
Manager
Spark Driver
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
34
PARALLEL TRAINING: SPARK + DL4J
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
35
Spark Worker Node 1
Spark
Master Node
Spark Worker Node 2
Da
ta
Da
ta S
plit
1
Da
ta S
plit
2
Min
i
Ba
tch
1M
ini
Ba
tch
2Calculate
model
parameters
Calculate
model
parametersAveraged
parameters
from workers
THE MACHINE LEARNING PROCESS
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
36
A LONG JOURNEY
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
37
Select problem
Non ML solution ?
Analyze process
Data available ?
Define success KPI
Classify ML type
Select 2 or 3 algo
Pick one
Gather data
Analyze data
Filter data
Convert data
Vectorize
Select hyperparams
Iterate hyperparams
Iterate data sets
Evaluate perform
Validation set
Deploy
Continuous eval
SELECT PROBLEM
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
38
GATHER AND FORMAT DATA
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
39
PROGRAM
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
40
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
41
ITERATE WITH HYPERPARAMETERS AND
DATA
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
42
CONTINUOUS EVALUATION
SUMMARY
• Automation and commoditization
• Humans and machines
• Beyond human capacity
• Application to product quality inspection
• Deep leaning and images: higher accuracy
• CNN
• DL4J + Spark
• The Machine Learning process
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
43
REFERENCES
• https://en.wikipedia.org/wiki/Smart_manufacturing
• http://karpathy.github.io/neuralnets/
• http://cs231n.github.io/convolutional-networks/
• https://deeplearning4j.org/convolutionalnets.html
• https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/
• https://en.wikipedia.org/wiki/Glass_production
• http://www.qorpak.com/pages/lab-glass-defects-inspect-packaging
• https://en.wikipedia.org/wiki/Automated_analyser
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
44
QUESTIONS
Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods
45
Prajod Vettiyattil, Wipro
@prajods
https://in.linkedin.com/in/prajod