Smart Manufacturing with Apache Spark Streaming and Deep

SMART MANUFACTURINGWith Apache Spark Streaming and Deep Leaning

ApacheCon Big Data Europe

Nov 2016, Seville, Spain

Prajod Vettiyattil, Wipro

@prajods

https://in.linkedin.com/in/prajod

AGENDA

• Introduction

• Smart Manufacturing

• Applications

• Deep learning solution

• CNN

• Deeplearning4j (DL4J)

• Apache Spark

• Deploying machine learning

Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods

2

INTRODUCTION

• Industrialization

• Automation

• Scale

• Commoditization

• Accessibility


3

MEN AND MACHINES


4

KEY POINT

Machines can do what humans can’t


5

SMART MANUFACTURING

• Holistic view of entire process

• Sourcing to product sale

• Efficiency

• Eco friendliness

• Predict over react

• Autonomous machines

• Learning and AI

• Use data and analytics


6

APPLICATIONS

• Detecting overheating

• Equipment

• Finished products

• Glass defect detection


7

OVERHEATING

• Detect temperatures above

defined boundaries


8

COOLING TOWER


9

ELECTRIC MOTOR


10

ELECTRIC MOTOR: OVERHEATING


11

OVERHEATING: LOWER RESOLUTION


12

ELECTRIC MOTOR: NORMAL VIEW


13

GLASS DEFECT


14

ZOOMING IN


15

UNDER A DIFFERENT LIGHT


16

AUTOMATING DETECTION

• Machine learning

• Multivariate Analysis

• Multiple categories of defects

• Deep Learning


17

DEEP LEARNING: WHAT

• Neural Network based Machine Learning

• Neural Network• 1 Input layer

• 1 or more hidden layers

• 1 Output layer

• Basic unit of NN• Neuron or Perceptron

• Combine perceptrons in many ways using multiple parameters

• Model parameters

• Hyperparameters


18

Image from: ucalgary.ca

DEEP LEARNING: WHY

• Can discern complex input patterns

• Higher accuracy than most other ML methods

• for image analysis

• Ideally suited for spatial data analysis

• Higher time needed for training

• Parallel execution

• Higher level architectures for evolving needs

• CNN, RNN, LSTM


19

CONVOLUTIONALNEURAL

NETWORKS


20

CONVOLUTIONAL NEURAL NETWORK

• Convolutional Neural Network (CNN)

• Convolve: mix two functions

• Matrix multiplication and addition

• Dot product

• Pass many filters over an image

• Image features

• Images are 3D features

• Discern the important features/signals

• Filter = kernel


21

CNN: APPLICATION

• Self driving cars

• Robotics

• Drones

• Industrial automation

• Physical security

• Medical labs

• Wherever images or videos are used


22

IMAGE FOR CONVOLUTION


23

CONVOLUTIONFilter matrix Image


24

CNN ARCHITECTURE


25

1st

Co

nv

olu

tio

n

No

nlin

ea

r tr

an

sfo

rm(R

ELU

, ta

nh

etc

)

Po

olin

g

Nth

Co

nv

olu

tio

n

No

nlin

ea

r tr

an

sfo

rm(R

ELU

, ta

nh

etc

)

Po

olin

g

Fu

lly c

on

ne

cte

d la

ye

rs

Ou

tpu

t p

rob

ab

iitie

s

Feature extraction + dimension reduction Classification

…..

CNN PARAMETERS

• Model parameters

• Filter values

• Weights

• Biases


26

• Hyperparameters

• Activation function

• # layers

• # nodes in each layer

• Filter size and count

• Regularization

• Convolution layers

• Maxpooling layers

Learned by NN Experience of designer

CNN: SAMPLE CODE• MultiLayerConfiguration conf = new NeuralNetConfiguration.Builder()

.seed(seed).iterations(iterations).regularization(false).l2(0.003).activation(“relu")

.learningRate(0.0007).weightInit(WeightInit.XAVIER)

.optimizationAlgo(OptimizationAlgorithm.STOCHASTIC_GRADIENT_DESCENT)

.updater(Updater.RMSPROP).momentum(0.9).list()

.layer(0, convInit("cnn1", channels, 50 , new int[]{5, 5}, new int[]{1, 1}, new int[]{0, 0}, 0))

.layer(1, maxPool("maxpool1", new int[]{2,2}))

….

.layer(4, new DenseLayer.Builder().nOut(500).build())

.layer(5, new OutputLayer.Builder(LossFunctions.LossFunction.NEGATIVELOGLIKELIHOOD)

.nOut(numLabels).activation("softmax").build())

.backprop(true).pretrain(false).cnnInputSize(height, width, channels).build();


27

DEEPLEARNING4J (DL4J)

• Deep learning library

• Open source

• Apache 2.0 license

• Java based

• Distributed execution

• Runs on Spark and Hadoop


28

DEEPLEARNING4J: FEATURES

• ND4J

• N Dimensional Arrays/Tensors

• Like numpy

• Canova/DataVec

• Data extraction, vectorization

• Arbitrer

• Evaluate and tune models

• CPU or GPU

• OpenBLAS, Intel MKL, Nvidia CUDA


29

DEEPLEARNING4J: SUPPORT

• Community support

• https://gitter.im/deeplearning4j/deeplearning4j

• Commercial support

• www.skymind.io


30

https://gitter.im/deeplearning4j/deeplearning4j

http://www.skymind.io/

APACHE SPARK

• Incubated in 2009 at Berkeley University

• 100s of contributors

• Yahoo, Intel, UC Berkeley,…,50+ orgs

• 10 to 100 times faster than Hadoop MR• https://databricks.com/blog/2014/10/10/spark-petabyte-sort.html


31

https://databricks.com/blog/2014/10/10/spark-petabyte-sort.html

APACHE SPARK

• Complements Hadoop

• Replaces Hadoop MR

• Adds• In memory processing

• Stream processing

• Interactive queries

• YARN or Mesos for clustering

• Java, Scala, Python, R


32

SPARK: LOGICAL VIEW

Apache Spark Core

Spark SQL Spark Streaming MLlib GraphX


33

SPARK: DEPLOYMENT VIEW

Worker Node

Executor

Master Node

Executor

Task CacheTaskTask

Worker Node

ExecutorExecutorExecutor

TaskTaskTaskCache

Spark’s

Cluster

Manager

Spark Driver


34

PARALLEL TRAINING: SPARK + DL4J


35

Spark Worker Node 1

Spark

Master Node

Spark Worker Node 2

Da

ta

Da

ta S

plit

1

Da

ta S

plit

2

Min

i

Ba

tch

1M

ini

Ba

tch

2Calculate

model

parameters

Calculate

model

parametersAveraged

parameters

from workers

THE MACHINE LEARNING PROCESS


36

A LONG JOURNEY


37

Select problem

Non ML solution ?

Analyze process

Data available ?

Define success KPI

Classify ML type

Select 2 or 3 algo

Pick one

Gather data

Analyze data

Filter data

Convert data

Vectorize

Select hyperparams

Iterate hyperparams

Iterate data sets

Evaluate perform

Validation set

Deploy

Continuous eval

SELECT PROBLEM


38

GATHER AND FORMAT DATA


39

PROGRAM


40


41

ITERATE WITH HYPERPARAMETERS AND

DATA


42

CONTINUOUS EVALUATION

SUMMARY

• Automation and commoditization

• Humans and machines

• Beyond human capacity

• Application to product quality inspection

• Deep leaning and images: higher accuracy

• CNN

• DL4J + Spark

• The Machine Learning process


43

REFERENCES

• https://en.wikipedia.org/wiki/Smart_manufacturing

• http://karpathy.github.io/neuralnets/

• http://cs231n.github.io/convolutional-networks/

• https://deeplearning4j.org/convolutionalnets.html

• https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/

• https://en.wikipedia.org/wiki/Glass_production

• http://www.qorpak.com/pages/lab-glass-defects-inspect-packaging

• https://en.wikipedia.org/wiki/Automated_analyser


44

http://karpathy.github.io/neuralnets/

http://karpathy.github.io/neuralnets/

https://deeplearning4j.org/convolutionalnets.html

https://deeplearning4j.org/convolutionalnets.html

https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/

https://en.wikipedia.org/wiki/Glass_production

https://en.wikipedia.org/wiki/Automated_analyser

https://en.wikipedia.org/wiki/Automated_analyser

QUESTIONS


45

Prajod Vettiyattil, Wipro

@prajods

https://in.linkedin.com/in/prajod

Smart Manufacturing with Apache Spark Streaming and Deep

Documents