Page 1
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
Large-Scale Granular and Fluid (DEM/SPH) Simulations using Particles
Large-Scale Granular and Fluid (DEM/SPH) Simulations using Particles
1
Takayuki Aoki
Global Scientific Information and Computing CenterTokyo Institute of Technology
SC14 NVIDIA booth talk, November 19, 2014, New Orleans
Page 2
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
Compute Node(3 Tesla K20X GPUs)
Performance: 4.08 TFLOPSMemory: 58.0GB(CPU)
+18GB(GPU)
Rack (30 nodes)
Performance: 122 TFLOPSMemory: 2.28 TB
System (58 racks)1442 nodes: 2952 CPU sockets,
4264 GPUs
Performance: 224.7 TFLOPS (CPU) ※ Turbo boost5.562 PFLOPS (GPU)
Total: 17.1 PFLOPS
TSUBAME 2.5TSUBAME 2.5
Page 3
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
3
TSUBAME SupercomputerTSUBAME Supercomputer
Graph 500No. 3 (2011)
wire
文部科学大臣表彰
(2012)
Gordon Bell Prize (2011)
Tesla S1070X170(680GPU)
Tesla K20XTesla M2050
CUDACOE
Page 4
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
4
Page 6
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
Weak Scalability: 2.0000 PFLOPS on 4,000 TSUBAME2.0, 330 billion cells44.5 % the peak performance
Page 7
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
Granular Material Simulationsusing Discrete Element Method
Granular Material Simulationsusing Discrete Element Method
7
Page 8
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
8
Golf Bunker ShotsGolf Bunker Shots
Page 9
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
Contact interaction
Normal direction
Tangential direction
SpringViscosity
Viscosity
Friction
Spring
ijijij xkxF
Simulation for Granular MaterialsSimulation for Granular MaterialsDEM (Discrete Element Method)DEM (Discrete Element Method)
Page 10
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
10
In 2005In 2005
■ kn = 5×108 dyn/cm■ Time Integration:
2-stage Ruge-Kutta
■ = 8×104 dyn・sec/cm■ t = 4×10-7 sec
DEM (Discrete Element Method) 76,000 Particles: 48 hours
Future work:
CPU 0 CPU 1 CPU 2
Page 11
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
2 dimensional slice-grid method
Dynamic Load Balance Dynamic Load Balance
Many particles
no particle
2. Move boundary
1. Move boundary
Page 12
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
2 dimensional slice-grid method
Dynamic Load Balance Dynamic Load Balance
Many particles
no particle
2. Move boundary
1. Move boundary
Page 13
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
Computational domain is dynamically decomposed into 64 sub-domains.
Slice grid
Dynamic Domain DecompositionDynamic Domain Decomposition
KD-tree Octree
Page 14
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
• Particle Collision detection of particles with complex shapes described by CAD data is efficiently carried out by using Level Set Function.
Collision Detection using Level Set FunctionCollision Detection using Level Set Function
Particle
Polygon ofCAD data Φ > 0 Φ < 0Negative areaPositive area
Page 15
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPULevel Set Function describing CAD surfaceLevel Set Function describing CAD surface
Surface patches of CAD data Level Set Function
negative distance area far from the surface
positive distance area far from the surface
• Generation from 3D CAD data on the uniform mesh• Fast generation algorithm and inside/outside judgment
Page 16
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
Neighbor Particle ListNeighbor Particle List
Local domain0 6
3
0
0 6 3
NULL
87 percent of memory usage is reduced compared to regular neighbor list.
Linked-list methodLinked-list method
Page 18
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
Page 20
AOKI Lab.バンカーショット計算16.7 millions particles
with 64 GPUs
Page 21
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
Page 22
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPUDEM using non-spherical particles
Considering more realistic shapes of rocks, non-spherical particles are used in DEM.
Many spherical particles with rigid body connections
Page 23
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
Using spherical particles,
Page 24
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
Using non‐spherical tetrapod particles,
Page 25
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPUMultiple GPU ScalabilityMultiple GPU Scalability
• Conditions Particles : 2 × 106, 1.6 × 107, 1.29 × 108
Domain Decomposition: Dynamic load Balance using Slice Grid Method Time-Integration : 2-stage
Runge-Kutta
Page 26
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPUSPH for Fluid DynamicsSPH for Fluid Dynamics
: Kernel function
First derivatives
h
h : Kernel radius
Particle interaction within a kernel radis
Page 27
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
A list of Particle Difference Operators
Improved SPHImproved SPH
Interpolation
Gradient
Divergence
Laplacian
2nd polynomial function (Spiky shaped):
r
Generalization of Finite Difference Operators (Imoto, Tagami 2014)
Page 28
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
• Explicit Time-integration using Predictor-corrector Method
Improved SPHImproved SPH
Predicator
Collector
Temporary pressures are calculated from Birch-Murnaghan’s equations:
Positions are computed as follows:
Pressures are computed as follows:
(1)
(2)
(3)
(4)
(5)
Page 29
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPUA Dam Break SimulationA Dam Break Simulation
• Initial setting and Parameters
12 m
2.2 m
0.8 m
Water
Object 6 m10 m
4.8 m
Page 30
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPUDescription of the Object ShapeDescription of the Object Shape
• A object is represented by particles arrangement generated from CAD data
Page 31
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPUA Dam Break SimulationA Dam Break Simulation
72 M particles with 80 GPUs
Page 32
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
Fluid-Structure Interaction
Page 33
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
33
SUMMARYSUMMARY
Particle Method (DEM/SPH) based on short-range interaction are also suitable for GPU computing as well as stencil computation.
Successful many granular simulations GPU-based supercomputer TSUBAME 2.0/2.5 have been shown.
Fluid simulations using SPH is suitable to describe free-surface flows.
Particle methods can be applied to Fluid-Structure Interaction easily.