HPC enabling of OpenFOAM for CFD applications

HPC enabling of OpenFOAM R© forCFD applications

HPC simulation of volcanic ash plumes and application ofOpenFOAM to CFD volcanological problems

06-08 April 2016, Casalecchio di Reno, BOLOGNA.

Matteo Cerminara – [email protected] Esposti Ongaro – [email protected]

Mattia de’ Michieli Vitturi – [email protected]

Istituto Nazionale di Geofisica e VulcanologiaIstituto Nazionale di Oceanografia e Geofisica Sperimentale

Effusive and explosive eruptions:

The example of Etna volcano

effusive (lava flows)

(2006)

(lava fountains)

(2014)

explosive (ash plumes)

(2015)

Explosivity is mostly controlled by multiphase processes in the magma(melt+gas+crystals):

• Gas phase transitions (gas exsolution, bubble nucleation andexpansion).

• Non-linear magma rheology (brittle transition = fragmentation).

Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 2 / 44

Challenges in

magma ascent modeling with OpenFOAM

Effusive regime

• Introduce multicomponent physics and phase transitions

• Bubble nucleation

• Manage phase separation and degassing

Explosive regime

• Wave propagation

• Fragmentation conditions

• Manage different domains (below/above fragmentation)


TwoPhaseChangeEulerFoam

A new multiphase multicomponent model with phase change has beendeveloped. Each phase is the mixture of several components andexsolution/evaporation laws can be defined for each component.

Test 1 (decompression experiment):

• validation through comparisonswith decompression experimentsperformed at LMU, Munich.

• silicon oil chosen as analogue formagmatic melt, and saturatedwith Argon at 10MPA

• slow decompression toatmospheric conditions




Test 1 (decompression experiment):

• validation through comparisonswith decompression experimentsperformed at LMU, Munich.

• silicon oil chosen as analogue formagmatic melt, and saturatedwith Argon at 10MPA

• slow decompression toatmospheric conditions




Test 2 (gas-driven effusive eruption):

• rise of hot and high-viscositymagmatic mixture:

• liquid phase: melt, dissolvedH2O, dissolved CO2;

• gas phase: exsolved H2O,exsolved CO2, air;


Explosive eruptions:

Volcanic plumes

Weak Plume

(Etna, 2011)

Strong Plume

(Chaiten, 2008)


Multiscale dynamics

Spatial and temporal scales

• Mass flow rate: Q ∼ UD2

• Reynolds number: Re = UDµ ∼ 109

• Typical Large-Eddy Scale: L ∼ (Str)D

• Kolmogorov’ length: η ∼ L−3/4 ∼ L× 10−7

• Integral length scale (plume height): H ∼ F 1/4S−3/4

(F = g ′UD2 = g ′Q; S = −g/ρ dρdz )

• Integral time scale: τ ∼√

Tv

(1+n)Γg ∼ 1− 5× 102 s

Number of degrees of freedom

• Minimum grid size: ∆x ∼ L

• Minimum number of cells: N ∼ (H/L)3 ∼ Q−1/4

• Minimum time-step ∆t ∼ ∆x/U

• Ntot(Weak) ∼ 1014; Ntot(Strong) ∼ 1011


Challenges in

volcanic plume modeling with OpenFOAM

Multiphase flow

• Describe non-equilibrium gas–particle dynamics

• Manage compressibility, turbulence, heat exchange

• Model subgrid turbulence

Numerical solution• Time-step constrained by vent conditions

• Compressible turbulence needs appropriate discretization schemes

• Number of cells is necessarily large! Effective parallelism.


Model-related uncertainty

Weak Plume Strong Plume

Does uncertainty blur accuracy?

• A ”blind” intercomparison test (4 models)

• Model-related uncertainty is still much larger than• Error associated with grid size• Error associated with SGS model

• Measurement uncertainty is within the model error


Table of contents

1 The ASHEE model

2 Verification and validation study

3 Volcanologic application


Eulerian fields

A new model ASHEE (Ash Equilibrium Eulerian) has been developedtaking advantage of the OpenFOAM infrastructure

• it models a mixture of I gas species and J solid particle species, eachwith diameter dj

• the ith gas species is characterized by the following fields in eachpoint (x, t)

• bulk density ρi• velocity ug

• temperature Tg

• the jth solid species is characterized by the following fields in eachpoint (x, t)

• bulk density ρj• velocity uj

• temperature Tj


Mass averaged Eulerian fields

The conservation equations for mass, momentum and energy are writtenfocusing on the mass averaged mixture properties:

Mass averaged field ψm

Firstly we define the mixture density as ρm =∑I

i=1 ρi +∑J

j=1 ρj , so thatthe mass fractions of each phase are defined:

• yi = ρi/ρm

• yj = ρj/ρm.

Thus, given a generic field ψ(x, t) for the ith gas species (ψi ) or for thejth solid species (ψj), we define

ψm =I∑

i=1

yiψi +J∑

j=1

yjψj


Physical configuration

The Eulerian model in “mixture” formulation

∂tρm +∇ · (ρmum) =∑j∈J

Sj ;

∂t(ρmyi ) +∇ · (ρmugyi ) = 0, i ∈ I ;

∂t(ρmyj) +∇ · (ρmujyj) = Sj , j ∈ J ;

∂t(ρmum) +∇ · (ρmum ⊗ um + ρmTr) =

= −∇p +∇ · T + ρmg +∑j∈J

Sjuj ;

∂t(ρmhm) +∇ · [ρmhm(um + vh)] +

+ ∂t(ρmKm) +−∇ · [ρmKm(um + vK )] =

= ∂tp +∇ · (T · ug − q) + ρm(g · um) +∑j∈J

Sj(hj + Kj) .


Physical configuration

• The “mixture” formulation is useful in two-way coupled multiphasesystems, where the mass of the dispersed phase has a non-negligibleeffect on the dynamics.

• The problem is formulated in the dispersed regime εs . 10−3,collisions between particles are disregarded

• It focuses the mathematical problem on the mass averaged fields:ρm , um , hm (improving stability).

• All the effects due to the kinematic decoupling are confined in theterms Tr(yj , vj) , vh(yj , vj , hj , hm) , vK (yj , vj ,Kj ,Km) , keeping intoaccount the effect of the relative velocity vj = uj − ug

• No explicit dependence on the drag functional expression isnecessary at this level


Equilibrium Eulerian model

• if particles are perfectly coupled both thermally (Tj = Tg = T ) andkinematically (uj = ug = u) we recover the dusty gas model, whereall the decoupling terms are zero

• in ASHEE we adopt the equilibrium-Eulerian model, where thesystem is thermally perfectly coupled while the kinematic decouplingis approximated via an asymptotic expansion relative to the Stokestime τj :

uj = ug + wj − τj(ag + wj · ∇ug) + O(τj)

where wj = τjg is the settling velocity and ag is the gas phaseacceleration


Equilibrium Eulerian model

• The contribution of the particle inertia can be accurately taken intoaccount by using standard Navier-Stokes numerical algorithm,without needing to implicitly solve the drag term

• Particle decoupling and preferential concentration well modeled upto St . 0.2, keeping the advantages of the dusty gas model

• Total number of equation highly reduced for a polydispersed mixture(4J PDEs less):

• Eulerian: I + 3 + 5J• Equilibrium-Eulerian: I + 3 + J.

• Allows to solve efficiently the multiphase dynamics at geophysicalscale for particles of size up to ' 1 mm.


Paper on the ASHEE model


Numerical configuration

We implemented the following subgrid-scale LES models for the subgridterms of the compressible equilibrium-Eulerian model:

• Compressible Smagorinsky (static and dynamic)

• Turbulent Kinetic Energy model (static and dynamic)

• WALE model (static and dynamic)


Numerical configuration

We modified the compressible monophase PISO-PIMPLE algorithm:

• predictor for the mixture density ρm• PIMPLE loop

• solve for the mass fractions yi,j• predictor for the mixture velocity um

• solve for the enthalpy hm, fixing the compressibility• PISO loop

• solve the decoupling• solve for pressure p• correct velocity and fluxes

• solve LES models

• correct mixture density


Table of contents

1 The ASHEE model




DHITDecaying Homogeneous and Isotropic Turbulence (2563 cells)The solver is able to simulate accurately the turbulence.

k

E(k)

100 101 10210-10

10-9

10-8

10-7

10-6

10-5

10-4

10-3

10-2

DNS Bernardini & Pirozzoli

OpenFoam

Figure: Test case validation: comparison with an eight order DNS after one large-eddy turnovertime (10000 time steps).


Scalability

64 128 256 512 10240,000,100,200,300,400,500,600,700,800,901,001,10

FermiPLX

# cores

Efficiency

64 128 256 512 10240

200

400

600

800

1000

1200

Fermi

Ideal

PLX

# coresSpeedup

Figure: Scalability test on PLX and FERMI environments (with little outputwork). Collaboration between us and Paride Dagna.

In order to fix ideas, the solver reach a velocity in the range 1÷10 Mcells/s on 1024 cores

(multiphase÷monophase) .


SGS LES models

1e-06

1e-05

0.0001

0.001

1 10

ener

gy s

pect

rum

E(k

)

wavenumber k

[noM] 2563

[noM] 323

[sma] 323

[oneEqEddy] 323

[wale] 323

Figure: DHIT using static SGS LES models in a box with 323 cells


SGS LES models

1e-06

1e-05

0.0001

0.001

1 10

ener

gy s

pect

rum

E(k

)

wavenumber k

[noM] 2563

[dynSma] 643

[moin] 323

[dynSma] 323

[dynOneEqEddy] 323

[dynWale] 323

[cubic] [dynOneEqEddy] 323

Figure: DHIT using dynamic SGS LES models in a box with 323 cells


Kinematic decoupling

Slice of the turbulent box at t/τe ' 2.2. The two panels representrespectively a logarithmic color map of yj=2 (Stmax = 0.5) and of |ag|


Kinematic decoupling

1e-05

0.0001

0.001

0.01

0.1

1

0.01 0.1 1

τ j τξ ⟨P⟩ j

(LE

S);

τ

j τη ⟨

P⟩ j

(D

NS

)

Stξ (LES); Stη (DNS)

[noM], 2563

[dynWale], 323

1.52*St2

Rani 2003

fit of Rani 2003

Figure: Evolution of the degree of preferential concentration with Stξ (LES) orStη (DNS). We obtain a good agreement between equilibriumEulerianLES/DNS and Lagrangian DNS simulations.


Advection schemes

Figure: Advection test case from Holzmann-cfd solved with ASHEE


Advection schemes

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

-0.02 -0.015 -0.01 -0.005 0 0.005 0.01 0.015 0.02

trac

er

time

upwindlinearUpwind

linearlimitedLinear

limitedLimitedLinearvanLeer

limitedVanLeercubic

limitedCubicUMISTSFCD

QUICKfilteredLinear

Figure: Tracer mass fraction along the cavity section.


Advection schemes

10-6

10-5

10-4

10-3

1 10

E(k

)

k

[noM], 2563

[dynWale], 323

[dynWale], [MUSCL], 323

[dynWale], [UMIST], 323

[dynWale], [vanLeer], 323

[dynWale], [rk4blended], 323

[dynWale], [filtlin0200], 323

Figure: Performances of OpenFOAM schemes in DHIT LES


Other benchmarks

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2

ρ

x/ct

analytic solutionrhoCentralFoam

ASHEE

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2

ρ

x/ct

Sod’s shock tube

1

10

10 100 1000

Nu

t [s]

Ra = 1e6, Nu = 8.800

Ra = 1e5, Nu = 4.519

Ra = 1e4, Nu = 2.243

Ra = 1e3, Nu = 1.118

Natural convection

Experimental plumeMixing


Table of contents

1 The ASHEE model




Plinian eruption

We have used ASHEE to simulate volcanic eruptions (scale ≈ 100 km)from the vent (mass eruption rate up to Mton/s) to the atmosphere


Volcanic plumes

We are participating to an international benchmark initiative involvingtwo numerical simulations:

weakPlume• duration: 0.2 hours

• Mass flow rate: 1.5 ∗ 106 kg/s

• Exit velocity: 135 m/s

• Exit temperature: 1273 K

• Exit gas fraction: 3 wt%

• Grain size distribution:• coarse: 1 mm• fine: 62.5 µm

strongPlume

• duration: 2.5 hours

• Mass flow rate: 1.5 ∗ 109 kg/s

• Exit velocity: 275 m/s

• Exit temperature: 1053 K

• Exit gas fraction: 5 wt%

• Grain size distribution:• coarse: 0.5 mm• fine: 15.6 µm


Effects of resolution and SGS modelTime and horizontal average of the velocity field, with 8, 16 and 32 cellsin a vent diameter; with and without decoupling model (weakPlume)

z [k

m]

U [m/s]

[dynWale], high res., [eqEu][dynWale], mid. res., [eqEu][dynWale], low res., [eqEu][dynWale], low res., [dusty]

0

2

4

6

8

10

12

14

0 20 40 60 80 100 120 140


Effects of resolutionTime and horizontal average of the velocity field, with 8, 16 and 32 cellsin a vent diameter (strongPlume)

z [k

m]

U [m/s]

[dynWale], high res., [eqEu][dynWale], mid. res., [eqEu][dynWale], low res., [eqEu]

0

5

10

15

20

25

30

35

40

45

0 50 100 150 200 250 300


Effects of SGS modelTime and horizontal average of the velocity field, with 16 and 32 cells ina vent diameter; with different SGS models (strongPlume)

z [k

m]

U [m/s]

[dynWale], high res., [eqEu][noM], high res., [eqEu]

[dynWale], mid. res., [eqEu][moin], mid. res., [eqEu]

[oneEqEddy], mid. res., [eqEu]

0

5

10

15

20

25

30

35

40

45

0 50 100 150 200 250 300


Volcanic infrasound

Compressibility and turbulence generate infrasound


Observations & simulations

Observation of 8 March 2005 MountSt. Helens eruption (Matoza et al.2009) Simulation with ASHEE of the

strongPlume eruption

0.001

0.01

0.1

1

10

100

0.01 0.1 1 10

20

30

40

50

60

70 0.01 0.1 1

Ep(

Str)

[Pa

]

Ep(

Str)

[dB

re

20 µ

Pa]

Str

f [Hz]

-11/4

Ep

-0.6-0.4-0.2

0 0.2 0.4 0.6 0.8

1 1.2

0 100 200 300 400 500 600 700 800

p [k

Pa]

t [s]


Summary of the second part

• We have developed the compressible version of theequilibrium-Eulerian model (ASHEE model)

• We have implemented it into the OpenFOAM infrastructure

• the solver has been tested up to 1024 cores. It shows a reasonableefficiency (> 60%) on the Cineca Fermi infrastructure

• The numerical scheme has been tested and chosen to maximizeaccuracy and stability of dynamic LES simulations

• A number of different benchmarks have been performed satisfactorily

• The new solver is able to accurately and efficiently captureclustering, preferential concentration and settling up to 1 mmparticles

• Applications to volcanological scale have been performed, comparingresults with other models and with observations


Volcanic hazard assessment:

an HPC/HTC challenge

Complex physics, multiscale dynamics

• Non-newtonian (non-linear) rheology

• Multiphase flows

• Broad range of spatial/temporal scales

Uncertainty on initial/boundary conditions

• Partial knowledge of the initial state

• Catastrophic dynamics

• Difficulty of measurements

• Repeatability issues


Future perspectives

Magma ascent modeling

• Implementation of fragmentation conditions

• More complex geometries

• Fluid-structure interaction

Volcanic plume modeling

• Add an arbitrary wind field

• Add volcano topography

• Add micro-physics, as water condensation, ash particles shape andaggregation


Future perspective

Numerical issues• Non-reflecting boundary conditions

• Improve the advection scheme

• Implement a shock capturing correction

• Optimize parallel linear algebra in OpenFOAM

Volcanic hazard assessment• Sensitivity and uncertainty analysis (coupled with Dakota).

• Coupling with meteorological solvers.

• ”Nowcasting” of volcanic plume scenarios.


Thank You!

Contacts• research web-page:https://sites.google.com/site/matteocerminara

• CFD movies:https://www.youtube.com/user/MatteoCerminara

Acknowledgments: we acknowledge CINECA for the availability ofhigh-performance computing resources and technical support on portingOpenFOAM on HPC architectures in the framework of ISCRA projects:IsB06 VolcFOAM, IsC26 VolcAshP and IsC07 GEOFOAM.


HPC enabling of OpenFOAM for CFD applications

Documents