SINGA: Putting Deep Learning into the Hands of Multimedia Users SINGA: Putting Deep Learning into the Hands of Multimedia Users http://singa.apache.org/ Wei Wang, Gang Chen, Tien Tuan Anh Dinh, Jinyang Gao, Beng Chin Ooi, Kian-Lee Tan, and Sheng Wang 1
27
Embed
SINGA: Putting Deep Learning into the Hands of Multimedia Users Wei Wang, Gang Chen, Tien Tuan Anh Dinh, Jinyang Gao, Beng Chin.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
SINGA: Putting Deep Learning into the Hands of Multimedia Users
SINGA: Putting Deep Learning into the Hands of
Multimedia Usershttp://singa.apache.org/
Wei Wang, Gang Chen, Tien Tuan Anh Dinh, Jinyang Gao, Beng Chin Ooi, Kian-Lee Tan, and Sheng
Wang
1
SINGA: Putting Deep Learning into the Hands of Multimedia Users
• Introduction
• Multimedia data and application• Motivations
• Deep learning models and training, and design principles• SINGA
• Usability
• Scalability
• Implementation
• Experiment
2
SINGA: Putting Deep Learning into the Hands of Multimedia Users
Introduction
Image/video
Social Media
E-commerce
Health-care Text
AudioMadbits (acquired by Twitter)
Perceptio (acquired by Apple)
LookFlow (acquired by Yahoo! Flickr)
Deepomatic (e-commerce product search)
Descartes Labs (satellite images)
Clarifai (tagging)
ParallelDots
Semantria (NLP tasks >10 languages)
Ldibon
AlchemyAPI (acquired by IBM)
VocallIQ (acquired by Apple)
Multimedia Data
Multimedia Data
Deep Learning has been noted for its effectiveness for multimedia applications!
3
SINGA: Putting Deep Learning into the Hands of Multimedia Users
Motivations
Model Categories
CNN, MLP, Auto-encoderImage/video classification
Feedforward Models
Krizhevsky, Sutskever, and Hinton, 2012; Szegedy et al., 2014; Simonyan and Zisserman, 2014a
CNN
4
SINGA: Putting Deep Learning into the Hands of Multimedia Users
Motivations
Feedforward Models
Energy models
RBM
DBN
Model Categories
CNN, MLP, Auto-encoderImage/video classification
DBN, RBM, DBMSpeech recognition
Dahl et al., 20125
SINGA: Putting Deep Learning into the Hands of Multimedia Users
Motivations
Feedforward Models
Energy models
Recurrent Neural
Networks
Model Categories
CNN, MLP, Auto-encoderImage/video classification
DBN, RBM, DBMSpeech recognition
RNN, LSTM, GRUNatural language processing
Mikolov et al., 2010; Cho et al., 20146
SINGA: Putting Deep Learning into the Hands of Multimedia Users
Motivations
Feedforward Models
Energy models
Recurrent Neural
Networks
Model Categories
CNN, MLP, Auto-encoderImage/video classification
DBN, RBM, DBMSpeech recognition
RNN, LSTM, GRUNatural language processing
Design Goal IUsability: easy to implement various models
7
SINGA: Putting Deep Learning into the Hands of Multimedia Users
Motivations: Training Process
• Training process• Update model parameters to minimize prediction error
• Training algorithm• Mini-batch Stochastic Gradient Descent (SGD)
• Training time• (time per SGD iteration) x (number of SGD iterations)• Long time to train large models over large datasets, e.g., 2 weeks
for training Overfeat (Pierre, et al.) reported by Intel (https://software.intel.com/sites/default/files/managed/74/15/SPCS008.pdf).
Back-propagation (BP) Contrastive Divergence (CD)
8
SINGA: Putting Deep Learning into the Hands of Multimedia Users
Motivations: Distributed Training Frameworks• Synchronous training (Google Sandblaster, Dean et al., 2012; Baidu AllReduce, Wu et al., 2015)
• Reduce time per iteration
• Scalable for single-node with multiple GPUs
• Cannot scale to large cluster
• Asynchronous training (Google Downpour, Dean et al., 2012, Hogwild!, Recht et al., 2011)
• Reduce number of iterations per machine
• Scalable for big cluster with commodity machine(CPU)
• Not stable
• Hybrid frameworks
Design Goal IIScalability: not just flexible, but also efficient and
adaptive to run different training frameworks
9
SINGA: Putting Deep Learning into the Hands of Multimedia Users
SINGA:
A Distributed Deep Learning Platform
10
SINGA: Putting Deep Learning into the Hands of Multimedia Users
SINGA: Putting Deep Learning into the Hands of Multimedia Users
Implementation
Driver::Train()
Main Thread
Stub::Run()
Worker thread
While(not stop): Worker::TrainOneBatch()
Server thread
While(not stop): Server::Update()
Remote NodesHDFS
Ubuntu
Docker
CentOS MacOS
DiskFile
Mes
os
Zoo
keep
er
Worker Stub Server
Driver
CNN RBM RNN
OptionalComponent
SINGA Component
Legend:
SINGA Software StackSINGA Software Stack
18
SINGA: Putting Deep Learning into the Hands of Multimedia Users
Deep learning as a Service (DLaaS)Third party APPs(Web app, Mobile,..)----------------------
API
Developers(Browser)
----------------------GUI
Rafiki ServerRafiki Server
Routing(Load balancing)
Rafiki AgentRafiki Agent
User, Job, Model, Node Management
Timon(c++ wrapper)
SINGA
Timon(c++ wrapper)
SINGA
DataBaseDataBase
File Storage System
(e.g. HDFS)
File Storage System
(e.g. HDFS)
…
Rafiki AgentRafiki AgentTimon
(c++ wrapper)
SINGA
Timon(c++ wrapper)
SINGA ……
http request
http request http request
http request
SINGA’s RAFIKI
1. To improve the Usability of SINGA; 2. To “level” the playing field by taking care of complex system plumbing work, its reliability, efficiency and scalability.
19
SINGA: Putting Deep Learning into the Hands of Multimedia Users
Comparison:Features of the Systems
Comparison with other open source projects
Feature SINGA Caffe CXXNET cuda-convnet H2O
Deep LearningModels
Feed-forward (CNN) ✔ ✔ ✔ ✔ MLP
Energy model (RBM) ✔ x x x x
Recurrent networks (RNN) ✔ ✔ x x x
DistributedTrainingFrameworks
Synchronous ✔ ✔ ✔ ✔ ✔
Asynchronous ✔ ✔ x x x
Hybrid ✔ x x x x
Hardware CPU ✔ ✔ ✔ x ✔
GPU V0.2.0 ✔ ✔ ✔ x
Cloud Software
HDFS ✔ x x x ✔
Resource management ✔ x x x ✔
Virtualization ✔ x x x ✔
Binding Python (P), Matlab(M), R ongoing (P) P+M P P P+R
MXNet on 28/09/15
20
SINGA: Putting Deep Learning into the Hands of Multimedia Users
Experiment --- Usability
Hinton, G. E. and Salakhutdinov, R. R. (2006)Reducing the dimensionality of data with neural networks.Science, Vol. 313. no. 5786, pp. 504 - 507, 28 July 2006.
…
Deep Auto-EncodersRBM
• Used SINGA to train three known models and verify the results
21
SINGA: Putting Deep Learning into the Hands of Multimedia Users
Experiment --- UsabilityW. Wang, X. Yang, B. C. Ooi, D. Zhang, Y. Zhuang: Effective Deep Learning Based Multi-Modal Retrieval. VLDB Journal - Special issue of VLDB'14 best papers, 2015. W. Wang, B.C. Ooi, X. Yang, D. Zhang, Y. Zhuang: Effective MultiModal Retrieval based on Stacked AutoEncoders. Int'l Conference on Very Large Data Bases (VLDB), 2014.
Deep Multi-Model Neural Network
CNN MLP
22
SINGA: Putting Deep Learning into the Hands of Multimedia Users