Parallel Multigrid for Electrokinetic Simulation in Particle-Fluid Flows Dominik Bartuschat, Madrid, Spain July 4, 2012 International Conference on High Performance Computing & Simulation (HPCS) 2012 Mittwoch, 4. Juli 2012
Parallel Multigrid for Electrokinetic Simulation in Particle-Fluid Flows
Dominik Bartuschat,Madrid, SpainJuly 4, 2012
International Conference on High Performance Computing & Simulation (HPCS) 2012
Mittwoch, 4. Juli 2012
D.Bartuschat, D.Ritter, U.Rüde
Chair for System Simulation, FAU Erlangen-Nürnberg
Mittwoch, 4. Juli 2012
Madrid, 04.07.2012 - Dominik Bartuschat - System Simulation Group - Parallel Multigrid for Electrokinetic Simulation(Contact: [email protected])
Outline
● Motivation● The waLBerla Simulation Framework● Charged Particles in Fluid Flow● Multigrid Solver● Results
3Mittwoch, 4. Juli 2012
Madrid, 04.07.2012 - Dominik Bartuschat - System Simulation Group - Parallel Multigrid for Electrokinetic Simulation(Contact: [email protected])
Motivation
4
Simulating agglomeration of charged particles in (micro-)fluid flow on charged plane.
● Medical applications:● Deposition of charged aerosol particles in
respiratory tract (e.g. drug delivery).● Optimization of Lab-on-a-Chip systems:● Trapping cells and viruses.● Separation of different cells.
© Kang and Li „Electrokinetic motion of particles and cells in microchannels“ Microfluidics and Nanofluidics
● Industrial applications:● Filtering particulates from exhaust gases.● Charged particle deposition in cooling
systems of fuel cells.
Mittwoch, 4. Juli 2012
The waLBerla Simulation Framework
Mittwoch, 4. Juli 2012
Madrid, 04.07.2012 - Dominik Bartuschat - System Simulation Group - Parallel Multigrid for Electrokinetic Simulation(Contact: [email protected])
waLBerla● Widely applicable Lattice Boltzmann framework.● Suited for various flow applications.● Large-scale, MPI-based parallelization.● Dynamic application switches for heterogeneous architectures and
optimization.
6Mittwoch, 4. Juli 2012
Madrid, 04.07.2012 - Dominik Bartuschat - System Simulation Group - Parallel Multigrid for Electrokinetic Simulation(Contact: [email protected])
waLBerla Concepts
7
Block concept:● Domain partitioned into cartesian grid of blocks.● Blocks can be assigned to different processes.● Blocks contain:
● cell data, e.g. electric potential.● global information e.g. MPI rank, location.
Communication concept:● Simple communication mechanism on uniform grids, utilizing MPI.● Ghost layers to exchange cell data with neighboring blocks.
Sweep concept:● Sweeps are work steps of a time-loop, performed on block-parallel level.● Example: MG sweep, contains sub-sweeps (restriction, prolongation, smoothing).
Mittwoch, 4. Juli 2012
Charged Particles in Fluid Flow
Mittwoch, 4. Juli 2012
● Discrete lattice Boltzmann equation (single relaxation time).● Domain discretized in cubes (cells).● Discrete velocities ci and associated distribution functions fi per cell.
Madrid, 04.07.2012 - Dominik Bartuschat - System Simulation Group - Parallel Multigrid for Electrokinetic Simulation(Contact: [email protected])
Lattice Boltzmann Method
9
D3Q19 model
fi (x + �ci∆t, t +∆t)− fi (x , t) = −1
τ(fi − f eqi ).
⟶
Mittwoch, 4. Juli 2012
Madrid, 04.07.2012 - Dominik Bartuschat - System Simulation Group - Parallel Multigrid for Electrokinetic Simulation(Contact: [email protected])
Stream-Collide
● Stream step
10
The equation is solved in two steps:
● Collide step
fi (x + �ci∆t, t +∆t) = f̃i (x , t +∆t)
f̃i (x , t +∆t) = fi (x , t)−1
τ(fi − f eqi )
Mittwoch, 4. Juli 2012
Madrid, 04.07.2012 - Dominik Bartuschat - System Simulation Group - Parallel Multigrid for Electrokinetic Simulation(Contact: [email protected])
Fluid-Particle Interaction - Coupling waLBerla and pe
● Particles are mapped onto lattice Boltzmann grid.● Each lattice node with cell center inside object is treated as moving boundary.● Hydrodynamic forces of fluid on particle computed by momentum exchange method*.
11
* D.Yu, R. Mei, L.-S. Luo, W.Shyy „Viscous flow computations with the method of lattice Boltzmann equation“
Mittwoch, 4. Juli 2012
Madrid, 04.07.2012 - Dominik Bartuschat - System Simulation Group - Parallel Multigrid for Electrokinetic Simulation(Contact: [email protected])
Poisson Equation and Coulomb Force on Particles
● Electric potential described by Poisson equation with particle charge density on RHS:
● Discretized by finite differences, solved with multigrid (MG) solver.● Electrostatic force on particle:
● Force density added to pe for charged particle cells:
12
−∆Φ(x) =ρparticles (x)
�r �0
−→f (x̄b) = −ρ(x̄b)
Q�
i=1,i �=C
titC
Φ(x̄b −−→ci ) ·−→ci
−→Fq = −qparticle ·∇Φ(x)
ci: discrete lattice velocity ti: direction dependent weighting factor
Mittwoch, 4. Juli 2012
Madrid, 04.07.2012 - Dominik Bartuschat - System Simulation Group - Parallel Multigrid for Electrokinetic Simulation(Contact: [email protected])
Charged Particles Algorithm
13
foreach time step, do// solve Poisson equation with particle charge density
set RHS of Poisson equation
while residual too high doapply multigrid solver to Poisson equation
calculate and MPI Reduce residual norm
// solve lattice Boltzmann equation considering particle velocities
beginstream PDFs (stream step)
calculate macroscopic variables
relax towards equilibrium PDF (collide step)
MPI-communicate PDFs to ghost layers
end// couple potential solver and LBM with pebegin
calculate and add hydrodynamic force on particles
calculate and add electrostatic force on particles
pe moves particles depending on forces
end
Mittwoch, 4. Juli 2012
Multigrid Solver
Mittwoch, 4. Juli 2012
● Based on● Smoothing principle:
High-frequency error elimination by iterative solvers (e.g. GS).
● Coarse grid principle: Restriction to coarser grid transforms low-frequency error components to relative higher-frequency ones.
● Smoothing applied on coarse grids.● Prolongation of obtained correction
terms to finer grid.
● Iterative method for efficient solution of sparse linear systems.
Madrid, 04.07.2012 - Dominik Bartuschat - System Simulation Group - Parallel Multigrid for Electrokinetic Simulation(Contact: [email protected])
Multigrid
15
● Applied recursively, v(νpre, νpost)-cycle.
Mittwoch, 4. Juli 2012
● All operations implemented as compact stencil operations.
● Design goals:● Efficient and robust black-box solver.● Handling complex boundary conditions on coarse levels.● Naturally extensible to jumping coefficients.
➡ Method of choice: Galerkin coarsening.
● Finite difference stencils stored for each unknown.
● Averaging restriction, nearest-neighbor prolongation.● Preserves D3Q7 stencil on coarse grids.● Violates constant convergence rate.● Workaround for Poisson problem: Overrelaxing prolongation*.
Madrid, 04.07.2012 - Dominik Bartuschat - System Simulation Group - Parallel Multigrid for Electrokinetic Simulation(Contact: [email protected])
Cell-Centered Multigrid - Implementation
16
* Bramble et. al „The analysis of multigrid algorithms for cell centered finite difference methods“, Adv. Comput. Math
Mittwoch, 4. Juli 2012
Results
Mittwoch, 4. Juli 2012
Madrid, 04.07.2012 - Dominik Bartuschat - System Simulation Group - Parallel Multigrid for Electrokinetic Simulation(Contact: [email protected])
Charged Particles in Fluid Flow
18
Agglomeration of charged particles on charged plane in water flow.
Channel: 2.56 x 5.76 x 2.56 mmDx=10µm, Dt=4⋅10-5s, τ=1.7
Particle radius: 60µmParticle charge: 8000e
Inflow velocity: 1mm/sOther walls: No-slip BCs
Potential: Bottom -100V, Top 0V Other walls: homogen. Neumann BCs
•Computed on 144 cores (12 nodes)•71.600 time steps•643 unknowns per core•6 MG levels
Mittwoch, 4. Juli 2012
Madrid, 04.07.2012 - Dominik Bartuschat - System Simulation Group - Parallel Multigrid for Electrokinetic Simulation(Contact: [email protected])
19
Scaling Setup and Environment Weak scaling:
● Costant size per core:● 1283 cells.● 9.4% moving obstacle cells.
● Size doubled (y-dimension).● MG (Residal L2-Norm ≤ 10-9):
● v(2,2) with 7 levels.● 100 RBGS coarse-grid iterations.
● 3x2x2 cores per node:
Strong scaling:● Constant total size:
● 384x128x256 cells.● 9.4% moving obstacle cells.
● # nodes doubled (all dimensions).● MG (Residual L2-Norm ≤ 10-9):
● v(2,2) with 6 levels.● 180 RBGS coarse-grid iterations.
● 3x2x2 cores per node.
Executed on RRZE‘s LiMa cluster:● 500 compute nodes, each:
● 2 Xeon 5650 "Westmere-EP" chips @2.66 GHz (2 x 6 cores),
● 24 GB DDR3 RAM.● Infiniband interconnect (40 Gbit/s).
Mittwoch, 4. Juli 2012
! " # $ % & ' ( ) !* !! !"*
"
$
&
(
!*
!"
+,-./0-1230/,,4
56
785
9:2;3<0,=0>,<3?
@433A:4
Madrid, 04.07.2012 - Dominik Bartuschat - System Simulation Group - Parallel Multigrid for Electrokinetic Simulation(Contact: [email protected])
20
● LBM scales very well.● MG speedup:
● Saturation for more than 6 cores,● memory bandwidth limitation.
Single Node PerformanceWeak scaling (first time-step, size doubled here in different dimensions)
Highest performance with 12 cores➡ used for further scaling experiments.Total run-time
6.7sLBM0.67s
MG (8 v-cycles)6.4s
Mittwoch, 4. Juli 2012
Madrid, 04.07.2012 - Dominik Bartuschat - System Simulation Group - Parallel Multigrid for Electrokinetic Simulation(Contact: [email protected])
21
LBM Scaling Results
! " # $ % &! &" &# &$ &%!
&!!
"!!
'!!
#!!
(!!
$!!
)*+,-./+0123
456723-./+0123
89*+0-./+0123
:;<=*6-7>-279*.
?@ABC4
! " # $ % &! &" &# &$ &%!
&!
"!
'!
#!
(!
$!
)*+,-./+0123
456723-./+0123
89*+0-./+0123
:;<=*6-7>-279*.
?ABC4
MFLUPS: Mega fluid lattice site updates per second● Nearly ideal weak scaling (communication 2.2% ➞ 3.4%).● Very good strong scaling (communication 2.5% ➞ 6.9%).
Mittwoch, 4. Juli 2012
Madrid, 04.07.2012 - Dominik Bartuschat - System Simulation Group - Parallel Multigrid for Electrokinetic Simulation(Contact: [email protected])
22
MG Scaling Results
! " # $ % &! &" &# &$ &%!
&!!
"!!
'!!
#!!
(!!
$!!
)*+,-./+0123
456723-./+0123
89*+0-./+0123
:;<=*6-7>-279*.
?@ABC4
! " # $ % &! &" &# &$ &%!
&!
"!
'!
#!
(!
$!
)*+,-./+0123
456723-./+0123
89*+0-./+0123
:;<=*6-7>-279*.
?ABC4
! " # $ % &! &" &# &$ &%!
&!!
"!!
'!!
#!!
(!!
$!!
)*+,-./+0123
456723-./+0123
89*+0-./+0123
:;<=*6-7>-279*.
?ABC4-D72*-EF/G/0*H
8 v-cycles, in first time-step (later 3) MLUPS: Mega lattice site updates per second● Very good weak scaling (communication 4.4% ➞ 7.0%).● Good strong scaling (communication 4.6% ➞ 17.6%).➡ Limited by many small messages on coarsest grid.
Mittwoch, 4. Juli 2012
Madrid, 04.07.2012 - Dominik Bartuschat - System Simulation Group - Parallel Multigrid for Electrokinetic Simulation(Contact: [email protected])
Summary
● Parallel multiphysics algorithm for charged particles in fluid flow.● Cell-centered multigrid with variable stencils for Poisson problem.● Performance results of main components (preliminary).
● Performance evaluation on single (many-core) node● Scaling experiments on several nodes.
● Possible MG improvements:● Splitting of red and black variables (of RBGS).● Implementation of more efficient coarse-grid solver (e.g. Conjugate Gradients).● Higher order prolongation scheme, conserving seven-point stencils.
23Mittwoch, 4. Juli 2012
Thank you for your attention!
Mittwoch, 4. Juli 2012