-
Copyright © Takayuki Aoki / Global Scientific Information and
Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
AMR based on Space-filling Curvefor Stencil Applications
AMR based on Space-filling Curvefor Stencil Applications
1
Takayuki Aoki
Global Scientific Information and Computing CenterTokyo
Institute of Technology
Copyright © Takayuki Aoki / Global Scientific Information and
Computing Center, Tokyo Institute of Technology
GP GPUGP GPUMotivationMotivation
2
Prof. Nakahashi
Prof. Yoshimura
Unstructured Grid Structured Grid
Coalesced Memory Access High accuracy Long stencil
High Performance
Copyright © Takayuki Aoki / Global Scientific Information and
Computing Center, Tokyo Institute of Technology
GP GPUGP GPUFlows with sharp interfacesFlows with sharp
interfaces
■ Navier-Stokes solver:Fractional Step■ Time integration:3rd TVD
Runge-Kutta■ Advection term:5th WENO■ Diffusion term:4th FD■
Poisson:MG-BiCGstab■ Surface tension:CSF model■ Surface
capture:CLSVOF(THINC + Level-Set)
Particle Methodex. SPH
Mesh Method (Surface Capture)
Low accuracy< 106-7 particles
High accuracy > 108-9 mesh points
not splash
Numerical noise and unphysical oscillation
Gas Liquid Two-phase Flows
-
A drop on the dry floor
6
Copyright © Global Scientific Information and Computing Center,
Tokyo Institute of Technology
GP GPUGP GPUIndustrial Appl. Steering OilIndustrial Appl.
Steering Oil
7 Copyright © Global Scientific Information and Computing
Center, Tokyo Institute of Technology
GP GPUGP GPUDevelopment New MaterialsDevelopment New
Materials
Material Microstructure
Dendritic Growth
Mechanical Structure
Improvement of fuel efficiency by reducing the weight of
transportation
Developing lightweight strengthening material by controlling
microstructure
Low-carbon society
-
Copyright © Global Scientific Information and Computing Center,
Tokyo Institute of Technology
GP GPUGP GPUInterface between Solid and LiquidInterface between
Solid and Liquid
Phase-field
0
1
Phase A
diffusive interfacewith finite thickness
Phase B
Mesh Adaption
Copyright © Global Scientific Information and Computing Center,
Tokyo Institute of Technology
GP GPUGP GPU
11
AMR(Adaptive Mesh Refinement)AMR(Adaptive Mesh
Refinement)Octrees and Space Filling Curves
Copyright © Global Scientific Information and Computing Center,
Tokyo Institute of Technology
GP GPUGP GPU
FDM & FVMFDM & FVM
12
FDM(Finite Difference Method)
Node Center AMR
FVM(Finite Volume Method)
Cell Center AMR
-
Copyright © Global Scientific Information and Computing Center,
Tokyo Institute of Technology
GP GPUGP GPU
13
Compressible Fluid SimulationCompressible Fluid Simulation
Heavy fluid lays on light fluid and unstable.
Euler equation:
0
yxtFEQ
evu
Q
pueuuv
puu
2
E
pvevpv
uvv
2F
IDO-CF Scheme512 x 512
Y. Imai, T. Aoki and K. Takizawa, J. Comp. Phys., Vol. 227,
Issue 4, 2263-2285 (2008)
Copyright © Global Scientific Information and Computing Center,
Tokyo Institute of Technology
GP GPUGP GPU
IDO-CF schemefor Compressible CFD
IDO-CF schemefor Compressible CFD
・Combination of FDM and FVM・Multi-Moment
interpolation・Higher-order accuracy・Less dispersive and
dissipative
Y. Imai, T. Aoki and K. Takizawa, J. Comp. Phys., Vol. 227, Issue 4, 2263‐2285 (2008)
Copyright © Global Scientific Information and Computing Center,
Tokyo Institute of Technology
GP GPUGP GPUStencil of continuum eq.Stencil of continuum eq.
PV XI
YI XYI
51 flop
83 flop
83 flop
55 flop
Copyright © Global Scientific Information and Computing Center,
Tokyo Institute of Technology
GP GPUGP GPU
16
Directional Splitting (1)Directional Splitting (1)Euler
equation:
0
yv
xu
t
02
yuv
xpu
tu
02
ypv
xuv
tv
0
ypvev
xpueu
te
0
xu
t0
2
xpu
tu
0
xuv
tv 0
xpueu
te
0
yv
t0
yuv
tu
02
ypv
tv 0
ypvev
te
-
Copyright © Global Scientific Information and Computing Center,
Tokyo Institute of Technology
GP GPUGP GPUDirectional Splitting (2)Directional Splitting
(2)
Available for explicit time integration of hyperbolic
equations:
0
)(
,0
)(
,0
)(
wpepww
vwuww
z
ewvu
t
vpewv
pvvuvv
y
ewvu
t
upewuvu
puuu
x
ewvu
t
1***
nn
ewvu
ewvu
ewvu
ewvu x-directionalintegration
y-directionalintegration
z-directionalintegration
Copyright © Global Scientific Information and Computing Center,
Tokyo Institute of Technology
GP GPUGP GPUStencil of continuum eq.Stencil of continuum eq.
PV XI
YI XYI
0 fuuff xxt xufuff iit
x
)()( 1
0 fufuf yxxy
ty
xufuff i
yi
y
txy
)()( 1
Copyright © Global Scientific Information and Computing Center,
Tokyo Institute of Technology
GP GPUGP GPUStencil of continuum eq.Stencil of continuum eq.
PV XI
YI XYI
0 fvvff yyt
yvfvf
f jjty
)()( 1y
vfvff j
xj
x
txy
)()( 1
0 fvfvf xyyx
tx
Copyright © Global Scientific Information and Computing Center,
Tokyo Institute of Technology
GP GPUGP GPUStencil of continuum eq.Stencil of continuum eq.
Direct Method Directional Splitting Method
-
Copyright © Global Scientific Information and Computing Center,
Tokyo Institute of Technology
GP GPUGP GPUStencil of continuum eq.Stencil of continuum eq.
Direct Method Directional Splitting Method
Copyright © Global Scientific Information and Computing Center,
Tokyo Institute of Technology
GP GPUGP GPUConservative Semi-LagrangianConservative
Semi-LagrangianNumerical fluxes are determined by integrating
interpolation function (Lagrangian Polynomials) 0
xu
t
if
if
tt
tudt
if 1f i
ix 1ixpx
xx
x
Xi
i
idxxFf )(2/1
z = ( 1.0/16.0*u[j‐2] ‐
9.0f/16.0*u[j‐1] ‐
9.0/16.0*u[j+1] + 1.0/16.0*u[j+2] )*dt/dx;zz
= z*z; zzz = zz*z;
fn[j] = f[j] – 1.0/6.0*(zzz ‐
z)*f[j+2] + (1.0/3.0*zzz + 0.5*zz
+ 5.0/6.0*z)*f[j+1]+ (1.0/6.0*zzz ‐ 0.5*zz
+ 1.0/3.0*z)*f[j] + 1.0/6.0*(zzz ‐ z)*f[j‐1];
Copyright © Global Scientific Information and Computing Center,
Tokyo Institute of Technology
GP GPUGP GPUCPU, MIC, GPU PerformancesCPU, MIC, GPU
Performances
53.6 59.5 57.1
457
866958
1218
0
200
400
600
800
1,000
1,200
1,400
Perf
orm
ance
[GFl
ops]
Xeon E5‐2600
Xeon Phi3110P
Xeon Phi5110P
TeslaM2050
TeslaK20C
TeslaK20X
GeForceGTX Titan
Shared Memory Use in the x-directional kernel. Super function
unit Loop unrolling Variable reuse in the y- and z- loops Reduction
of branch diverges
Kepler GPU Tuning
Copyright © Global Scientific Information and Computing Center,
Tokyo Institute of Technology
GP GPUGP GPU
1 1 1 1 1
1 1 132
Some leaves extinct by coarseningf
De-fragmentationDe-fragmentation
Defragmentation by re-numbering
1 1 1 1 1
new leaves are generated by refinement.
1 1 132 1 1
GPU memory pool
-
Copyright © Global Scientific Information and Computing Center,
Tokyo Institute of Technology
GP GPUGP GPU
Space-Filling CurveSpace-Filling Curve
Hilbert CurveHilbert Curve Morton Curve
Copyright © Global Scientific Information and Computing Center,
Tokyo Institute of Technology
GP GPUGP GPU
Configuration 0 Configuration 1 Configuration 2 Configuration
3
Configuration 4 Configuration 5 Configuration 6 Configuration
7
Base of Hilbert CurveBase of Hilbert Curve
Copyright © Global Scientific Information and Computing Center,
Tokyo Institute of Technology
GP GPUGP GPULeaf RefinementLeaf Refinement
Configuration 0 Configuration 2
Copyright © Global Scientific Information and Computing Center,
Tokyo Institute of Technology
GP GPUGP GPULeaf RefinementLeaf Refinement
Configuration 5 Configuration 7
-
Copyright © Global Scientific Information and Computing Center,
Tokyo Institute of Technology
GP GPUGP GPU
Hilbert CurveHilbert Curve
Copyright © Global Scientific Information and Computing Center,
Tokyo Institute of Technology
GP GPUGP GPU
Morton CurveMorton Curve
Copyright © Global Scientific Information and Computing Center,
Tokyo Institute of Technology
GP GPUGP GPU
Domain DecompositionDomain Decomposition
Copyright © Global Scientific Information and Computing Center,
Tokyo Institute of Technology
GP GPUGP GPU
Domain DecompositionDomain Decomposition
-
Level 0 Primitives Level 1 Refinements
3D Mesh Refinement
Copyright © Takayuki Aoki / Global Scientific Information and
Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
: Level-Set function(distance function)
: Heaviside function
The Level-Set methods (LSM) use the signed distancefunction to
capture the interface. The interface isrepresented by the
zero-level set (zero-contour).
Re-initialization for Level-Set function
Fig. Takehiro Himeno, et. Al., JSME,
65-635,B(1999),pp2333-2340
Level-Set method (LSM)Level-Set method (LSM)
Advantage : Curvature calculation, Interface boundaryDrawback :
Volume conservation
-
Copyright © Takayuki Aoki / Global Scientific Information and
Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
Anti-diffusive Interface CaptureAnti-diffusive Interface
Capture
・VOF(volume of fluid) type interface capturing method・Flux from
tangent of hyperbola function・Semi-Lagrangian time integration
[ Xiao, etal, Int. J. Numer. Meth. Fluid. 48(2005)1023 ]
・1D implementation can be applied to 2D & 3D → Simple
・Finite Volume like usage* THINC is the method how to compute
flux
→ 3 krenel (x, y, z) can be fused to 1 kernel. Merit in memory
R/W
THINC (tangent of hyperbola for interface capturing) Scheme
Copyright © Takayuki Aoki / Global Scientific Information and
Computing Center, Tokyo Institute of Technology
GP GPUGP GPUThinc WLICThinc WLIC
WLIC – splitting the interface intospatial directions
Weighting factors depending on the normalDirections.
: nomal vector to the interface
Calculation of the normal vector
Copyright © Takayuki Aoki / Global Scientific Information and
Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
39
Advection of VOFAdvection of VOFNumerical Diffusion of Advection
Computationsfor VOF (Volume of Fluid) depending on mesh
resolutions.
128×128 256×256 ???×???
Thinc WLIC Scheme for Anti-diffusion:
Copyright © Global Scientific Information and Computing Center,
Tokyo Institute of Technology
GP GPUGP GPUOctree-based GPU-AMROctree-based GPU-AMR
GPUleaf
Octreedata structure
Hilbert Space-filling Curve
Interface Adaptation
高解像度が必要な界面に動的に細かい格子を集め、計算領域全域を細かくした場合の数%の格子点数で効率的に計算する。
leaf memory is managed on CPU
-
Copyright © Global Scientific Information and Computing Center,
Tokyo Institute of Technology
GP GPUGP GPURayleigh-Taylor InstabilityRayleigh-Taylor
Instability
41
Density Profile Level Set Function
Copyright © Global Scientific Information and Computing Center,
Tokyo Institute of Technology
GP GPUGP GPURayleigh-Taylor InstabilityRayleigh-Taylor
Instability
42
Copyright © Global Scientific Information and Computing Center,
Tokyo Institute of Technology
GP GPUGP GPUOctree-based GPU-AMROctree-based GPU-AMR
Interface Adaptation
Copyright © Global Scientific Information and Computing Center,
Tokyo Institute of Technology
GP GPUGP GPU
Thank youThank you
44