Efficient Non-contiguous Data Transfer using MVAPICH2-GDR ... · High Performance Interconnects ... “A PCIe congestion -aware performance model for densely populated accelerator

Efficient Non-contiguous Data Transfer using MVAPICH2-GDR for GPU-enabled

HPC ApplicationsChing-Hsiang Chu

[email protected]. Candidate

Department of Computer Science and EngineeringThe Ohio State University

OSU Booth - SC19 2Network Based Computing Laboratory

• Introduction

• Advanced Designs in MVAPICH2-GDR

• Concluding Remarks

Outline

OSU Booth - SC19 3Network Based Computing Laboratory

Trends in Modern HPC Architecture: Heterogeneous

• Multi-core/many-core technologies• High Performance Interconnects

• High Performance Storage and Compute devices• Variety of programming models (MPI, PGAS,

MPI+X)

Accelerators / Coprocessors high compute density,high performance/watt

High Performance Interconnects InfiniBand, Omni-Path, EFA

<1usec latency, 200Gbps+ Bandwidth

Multi/ Many-core Processors

SSD, NVMe-SSD, NVRAM

Node local storage

#1 Summit(27,648 GPUs)

#2 Sierra (17,280 GPUs)#10 Lassen (2,664 GPUs)

#8 ABCI(4,352 GPUs)

#22 DGX SuperPOD(1,536 GPUs)

OSU Booth - SC19 4Network Based Computing Laboratory

Architectures: Past, Current, and Future

Multi-core CPUs across nodes Multi-core CPUs + Single GPU across nodes

Multi-core CPUs within a node Multi-core CPUs + Multi-GPU within a node

Multi-core CPUs + Multi-GPU across nodes

(E.g., Sierra/Summit, Frontier)

IBNetworks

IB Networks

IB Networks

SC 19 Doctoral Showcase 5Network Based Computing Laboratory

• Wide usages of MPI derived datatype for Non-contiguous Data Transfer

– Requires Low-latency and high overlap processing

Motivated Example – Non-contiguous Data Transfer

M. Martinasso, G. Kwasniewski, S. R. Alam, Thomas C. Schulthess, and T. Hoefler. “A PCIe congestion-aware performance model for densely populated accelerator servers. “ SC 2016

Weather Simulation: COSMO model

Mike Clark. “GPU Computing with QUDA, “Developer Technology Group, https://www.olcf.ornl.gov/wp-content/uploads/2013/02/Clark_M_LQCD.pdf

Quantum Chromodynamics: MILC with QUDA

OSU Booth - SC19 6Network Based Computing Laboratory

• Introduction

• Advanced Designs in MVAPICH2-GDR– Asynchronous designs for Maximizing Overlap

– Zero-copy (Pack-free) on Dense-GPU systems

• Concluding Remarks

Outline

SC 19 Doctoral Showcase 7Network Based Computing Laboratory

Existing GPU-enabled MPI Datatype Processing

Waste of computing resources on CPU and GPUCommon Scenario

*A, B…contain non-contiguous MPI Datatype

MPI_Isend (A,.. Datatype,…)MPI_Isend (B,.. Datatype,…)MPI_Isend (C,.. Datatype,…)MPI_Isend (D,.. Datatype,…)…

MPI_Waitall (…);

Ching-Hsiang Chu et al., "Exploiting Maximal Overlap for Non-Contiguous Data Movement Processing on Modern GPU-enabled Systems, " IEEE IPDPS 2016.

SC 19 Doctoral Showcase 8Network Based Computing Laboratory

Proposed Event-based Design – Low Latency

MPI_Isend()cudaEventRecord()

HCA CPU GPU

MPI_Waitall()Query / Progress

Send

CompletionRequest

Complete

pack_kernel1<<< >>>

pack_kernel2<<< >>>

pack_kernel3<<< >>>cudaEventRecord()

cudaEventRecord()

MPI_Isend()

MPI_Isend()

SC 19 Doctoral Showcase 9Network Based Computing Laboratory

Proposed Callback-based Design – High Overlap

MPI_Isend()addCallback()

HCA CPU GPU

Send

CompletionRequestComplete

main helper callback

pack_kernel1<<< >>>

Callback

MPI_Waitall()

addCallback()

addCallback()CPU

Computations

pack_kernel2<<< >>>

pack_kernel3<<< >>>

MPI_Isend()

MPI_Isend()

CallbackCallback

SC 19 Doctoral Showcase 10Network Based Computing Laboratory

Application-Level Evaluation (Cosmo) and Weather Forecasting in Switzerland

0

0.2

0.4

0.6

0.8

1

1.2

16 32 64 96N

orm

alize

d Ex

ecut

ion

Tim

e

Number of GPUs

CSCS GPU cluster

Default Callback-based Event-based

0

0.2

0.4

0.6

0.8

1

1.2

4 8 16 32

Nor

mal

ized

Exec

utio

n Ti

me

Number of GPUs

Wilkes GPU Cluster

Default Callback-based Event-based

• 2X improvement on 32 GPUs nodes• 30% improvement on 96 GPU nodes (8 GPUs/node)

C. Chu, K. Hamidouche, A. Venkatesh, D. Banerjee , H. Subramoni, and D. K. Panda, Exploiting Maximal Overlap for Non-Contiguous Data Movement Processing on Modern GPU-enabled Systems, IPDPS’16

On-going collaboration with CSCS and MeteoSwiss (Switzerland) in co-designing MV2-GDR and Cosmo Application

Cosmo model: http://www2.cosmo-model.org/content/tasks/operational/meteoSwiss/

2X

OSU Booth - SC19 11Network Based Computing Laboratory

• Introduction

• Advanced Designs in MVAPICH2-GDR– Asynchronous designs for Maximizing Overlap

– Zero-copy (Pack-free) on Dense-GPU systems

• Concluding Remarks

Outline

SC 19 Doctoral Showcase 12Network Based Computing Laboratory

• Exploiting load-store capability of modern interconnects– Eliminate extra data copies and expensive packing/unpacking processing

Proposed Zero-copy (packing-free) Datatype Transfer

Sour

ce G

PU M

emor

y

Dest

inat

ion

GPU

Mem

ory

PCIe

/NVL

ink

Sour

ce G

PU M

emor

y

Dest

inat

ion

GPU

Mem

ory

PCIe

/NVL

ink

Load-Store

Copy

Existing Schem Proposed Packing-free Schem

Ching-Hsiang Chu et al., “High-Performance Adaptive MPI Derived Datatype Communication for Modern Multi-GPU Systems”, to appear in HiPC 2019.

SC 19 Doctoral Showcase 13Network Based Computing Laboratory

Performance Evaluation• Zero-copy (packing-free) for GPUs with peer-to-peer direct access over PCIe/NVLink

0

5

10

15

20

25

[6, 8,8,8,8] [6, 8,8,8,16] [6, 8,8,16,16] [6, 16,16,16,16]

MILC

Spee

dup

Problem size

GPU-based DDTBench mimics MILC communication kernel

OpenMPI 4.0.0 MVAPICH2-GDR 2.3.1 ProposedPlatform: Nvidia DGX-2 system

0

5

10

15

20

25

16 32 64

Exec

utio

n Ti

me

(s)

Number of GPUs

Communication Kernel of COSMO Model

MVAPICH2-GDR 2.3.1 ProposedPlatform: Cray CS-Storm

Improved 3.4X

(https://github.com/cosunae/HaloExchangeBenchmarks)

Improved 15X

Ching-Hsiang Chu et al., “High-Performance Adaptive MPI Derived Datatype Communication for Modern Multi-GPU Systems”, to appear in HiPC 2019.

OSU Booth - SC19 14Network Based Computing Laboratory

• Introduction

• Advanced Broadcast Designs in MVAPICH2-GDR

• Concluding Remarks

Outline

OSU Booth - SC19 15Network Based Computing Laboratory

Efficient MPI derived datatype processing for GPU-resident data

Asynchronous GPU kernels to achieve high overlap between

communication and computation

Zero-copy schemes for Dense-GPU with high-speed interconnects like PCIe

and NVLink

These features are included since MVAPICH2-GDR 2.3.2 http://mvapich.cse.ohio-state.edu/ http://mvapich.cse.ohio-state.edu/userguide/gdr/

Concluding Remarks

OSU Booth - SC19 16Network Based Computing Laboratory

Thank You!

Network-Based Computing Laboratoryhttp://nowlab.cse.ohio-state.edu/

The MVAPICH2 Projecthttp://mvapich.cse.ohio-state.edu/

• Join us for more tech talks from MVAPICH2 team– http://mvapich.cse.ohio-state.edu/talks/