Evaluation of Container Virtualized MEGADOCK System in Distributed Computing Environment March 23th, 2017 SIG BIO 49@Japan Advanced Institute of Science and Technology Kento Aoyama 1,2 , Yuki Yamamoto 1,2 , Masahito Ohue 1,3 , Yutaka Akiyama 1,2,3 1) Department of Computer Science, School of Computing Tokyo Institute of Technology 2) Education Academy of Computational Life Sciences (ACLS) Tokyo Institute of Technology 3) Advanced Computational Drug Discovery Unit, Institute of Innovative Research Tokyo Institute of Technology
32
Embed
Evaluation of Container Virtualized MEGADOCK System in Distributed Computing Environment
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Evaluation of Container Virtualized MEGADOCK System
in Distributed Computing Environment
March 23th, 2017SIG BIO 49@Japan Advanced Institute of Science and Technology
OS (image) CentOS 7.2.1511 ubuntu:14.04 nvidia/cuda8.0-devel
Linux Kernel 3.10.0 3.10.0 3.10.0
GCC 4.8.5 4.8.4 4.8.4
FFTW 3.3.5 3.3.5 3.3.5
OpenMPI 1.10.0 1.6.5 N/A
Docker Engine 1.12.3 N/A N/A
NVCC 8.0.44 N/A 8.0.44
NVIDIA Docker 1.0.0 rc.3 N/A N/A
NVIDIA Driver 367.48 N/A 367.48
CPU Intel Xeon E5-1630, 3.7 [GHz] ×8 [core]
Memory 32 [GB]
Local SSD 128 [GB]
GPU NVIDIA Tesla K40
Execution time 23
7353.80
1646.09
7850.57
1638.05
0
1500
3000
4500
6000
7500
9000
CPU (MPI) GPU
Tim
e [
sec]
Native Docker
+6.32 % slower
Profile Result (CPU time) 24
Process native [sec] docker [sec] diff Ratio (all)
FFT3D 7.40E+04 7.63E+04 +3.01% 76.84%
MPIDP-Master 8010.98 8325.9 +3.78% 8.38%
Create Voxel 3743.7 3993.29 +6.25% 4.02%
FFT Convolution 3551.08 3576.43 +0.71% 3.60%
Score Sort 2462.61 2459.7 -0.12% 2.48%
Output Detail 2139.94 2225.96 +3.86% 2.24%
Ligand Preparation 1035.51 1849.11 +44.00% 1.86%
MPI_Barrier 236.95 231.05 -2.55% 0.23%
MPI_Init 0.94 4.54 79.30% 0.00%
… … … … …
(a) MEGADOCK-Azure[2]
Measurement
• megadock-dp exec. time
• time command (3 times, median)
Dataset
• ZDOCK benchmark 1.0 [1]
(59 * 59 = 3481 pairs)
Options (OpenMP, OpenMPI)
• MPI : 12 threads / 4 MPI process / 1 node
All file input/output in Local SSD
Overview of Experiment II-(a) 25
Virtual
Machine
MPI
MPI
MPI
MPI
VM
MPI
MPI
MPI
MPI
VM
MPI
MPI
MPI
MPI
VM
MPI
MPI
MPI
MPI
VM
MPI
MPI
MPI
MPI
VM
MPI
MPI
MPI
MPI
VM
MPI
MPI
MPI
MPI
Master Process
Worker Process
(Other)
[1] R. Chen, et al. “A protein-protein docking benchmark,” Proteins: Structure,
Function and Genetics, vol. 52, no. 1, pp. 88-91, 2003.
[2] Masahito Ohue, et al. ”MEGADOCK-Azure: High-performance protein-protein interaction prediction system on Microsoft Azure HPC”, IIBMP2016.
(b) MEGADOCK + Docker on Microsoft Azure
Measurement
• megadock-dp exec. time
• time command (3 times, median)
Dataset
• ZDOCK benchmark 1.0(59 * 59 = 3481 pairs)
Options (OpenMP, OpenMPI)
• MPI : 12 threads / 4 MPI process / 1 node
All file input/output in Local SSD
Docker Swarm
• All Containers in 1 overlay network
Overview of Experiment II-(b) 26
Virtual Machine
Docker
MPI
MPI
MPI
MPI
DockerMPI
MPI
MPI
MPI
DockerMPI
MPI
MPI
MPI
DockerMPI
MPI
MPI
MPI
DockerMPI
MPI
MPI
MPI
DockerMPI
MPI
MPI
MPI
DockerMPI
MPI
MPI
MPI
Docker Swarm
(Docker Network)
Master Process
Worker Process
(Other)
[1] R. Chen, J. Mintseris, J. Janin, and Z. Weng, “A protein-protein docking benchmark,”Proteins: Structure, Function and Genetics, vol. 52, no. 1, pp. 88-91, 2003.
VM Instance/Software Specification 27
Software Env. Virtual Machine Docker
OS (image) SUSE Linux Enterprise Server 12 ubuntu:14.04
Linux Kernel 3.12.43 3.12.43
GCC 4.8.3 4.8.4
FFTW 3.3.4 3.3.5
OpenMPI 1.10.2 1.6.5
Docker Engine 1.12.6 N/A
VM Instance Standard_D14_v2
CPU Intel Xeon E5-2673, 2.40 [GHz] × 16 [core]
Memory 112 [GB]
Local SSD 800 [GB]
Execution time 28
145,534
25,515
13,132
6,006 4,098
117,219
25,145
12,331
6,344 3,971
0
25,000
50,000
75,000
100,000
125,000
150,000
1 5 10 20 30
Tim
e [
se
c]
# of VMs
VM Docker on VM
May be a measurement mistake
Scalability (Strong Scaling, based VM=1) 29
0
5
10
15
20
25
30
35
40
45
0 100 200 300 400 500
Sp
ee
d-u
p
# of worker cores
Ideal VM Docker on VM
VM=5
VM=1
VM=10
VM=20
VM=30
comparable scalability
Experiment I• MEGADOCK + Docker on Physical Machine
showed 6.32% lower performance.
• Docker can cause 0-4% compute-performance down[1]
• Communications via Docker NAT (Network Address Translation)
• MEGADOCK (GPU) + NVIDIA-Docker on Physical Machineshowed comparable performance to native.• GPU calc. is independent from container virtualization
• Container virtualization has few overhead on memory bandwidth
Experiment II• MEGADOCK + Docker on Microsoft Azure
performed comparable scalability.• Container virtualization overhead is smaller than other cloud environment factor
Result & Discussion 30
[1] W. Felter, A. Ferreira, R. Rajamony, and J. Rubio, “An updated performance comparison of virtual
machines and Linux containers”, IEEE International Symposium on Performance Analysis of Systems
and Software, pp.171-172, 2015. (IBM Research Report, RC25482 (AUS1407-001), 2014.)
• Performance overhead of Docker container-virtualization is small.• suitable for GPU-accelerated-App and Cloud Environment
• Container-Virtualization can isolate application environment from host environment.• same container image can be used on various machines
• Physical machine on local environment
• Virtual machine on cloud environment
• Docker is useful for computational research work
Conclusion 31
Multi-Node & Multi-GPU Evaluation on Cloud• NVIDIA-Docker is not available on Docker Swarm mode
• Kubernetes[1] officially support 1GPU/1node
• (experimental-feature: multi-GPU support)
Container-based Task Distribution• Web-Service-Application like container-based distribution
• easy to scale computing resource
• easy to extends multiple task (e.g. GHOST-MP, MEGADOCK)
Future Work 32
[1] B. Burns, B. Grant, D. Oppenheimer, E. Brewer, and J. Wilkes, “Borg, Omega, and
Kubernetes,” acmqueue, vol. 14, no. 1, p. 24, 2016.