Compatible finite element methods and parallel-in-time ... · Consideration of alternative spatial discretizations: ... prototyping of algorithms based on the compatible finite element

Compatible finite element methods and parallel-in-timeschemes for numerical weather prediction

J. Shipton1, C. J. Cotter2, T. M. Bendall3, T. H. Gibson4, L. Mitchell5, D. A. Ham2, B. Wingate1, D. Acreman1, M. Schreiber61University of Exeter, UK, 2Imperial College London, UK, 3Met Office, Exeter, UK, 4Naval Research Laboratory, Monterey, California, USA, 5Durham University, UK, 6TU Munich

The Gung Ho Project(UK Met Office)

The problem:

“overset grids.” There have also been recent attempts to use grids based on octahedrons (e.g., McGregor, 1996; Purser and Rancic, 1998). A “Fibonacci grid” has also been suggested (Swinbank and Purser, 2006).

Grids based on icosahedra offer an attractive framework for simulation of the global circulation of the atmosphere; their advantages include almost uniform and quasi-isotropic resolution over the sphere. Such grids are termed “geodesic,” because they resemble the geodesic domes designed by Buckminster Fuller. Williamson (1968) and Sadourny (1968) simultaneously introduced a new approach to more homogeneously discretize the sphere. They constructed grids using spherical triangles which are equilateral and nearly equal in area. Because the grid points are not regularly spaced and do not lie in orthogonal rows and columns, alternative finite-difference schemes are used to discretize the equations. Initial tests using the grid proved encouraging, and further studies were carried out. These were reported by Sadourny et al. (1968), Sadourny and Morel (1969), Sadourny (1969), Williamson (1970), and Masuda (1986).

The grids are constructed from an icosahedron (20 faces and 12 vertices), which is one of the five Platonic solids. A conceptually simple scheme for constructing a spherical geodesic grid is to divide the edges of the icosahedral faces into equal lengths, create new smaller equilateral triangles in the plane, and then project onto the sphere. See Fig. 11.9. One can construct a more homogeneous grid by partitioning the spherical equilateral triangles instead. Williamson (1968) and Sadourny (1968) use slightly different techniques to construct their grids. However, both begin by partitioning the spherical icosahedral triangle. On these geodesic grids, all but twelve of the cells are hexagons. The remaining twelve are pentagons. They are associated with the twelve vertices of the original icosahedron.

Williamson (1968) chose the nondivergent shallow water equations to test the new grid. He solved the two-dimensional nondivergent vorticity equation

!"!t

= #J $ ,%( ) ,

(52)where ! is relative vorticity, ! = " + f is absolute vorticity and ! is the stream function, such

that

Fig. 11.8: Various ways of discretizing the sphere. Figure made by Bill Skamarock of NCAR.

Mesoscale & Microscale Meteorology Division / ESSL / NCAR

MPAS

Future Weather/Climate Atmospheric Dynamic Core

Consideration of alternative spatial discretizations:

Priority Requirements:

Lat-Lon Icosahedral-triangles Icosahedral-hexagons Cubed Sphere Yin-Yang

• Efficient on existing and proposed supercomputer architectures

• Scales well on massively parallel computers

• Well suited for cloud (nonhydrostatic) to global scales

• Capability for local grid refinement and regional domains

• Conserves at least mass and scalar quantities

Problems with lat-lon coordinate for global models

• Pole singularities require special filtering

• Polar filters do not scale well on massively parallel computers

• Highly anisotropic grid cells at high latitudes

! Revised Monday, November 30, 2009! 16

An Introduction to Numerical Modeling of the Atmosphere

The clustering of points at the pole of traditionallat-lon grids (left) is a bottleneck to parallel scaling.Can we use non-orthogonal grids such those on theright, while retaining the discretisation propertiesdescribed in [1]?

The solution:‘mimetic’ or ‘compatible’ finite element spatial dis-cretisations that preserve discrete versions of thecontinuous vector calculus identities: div(curl)=0and curl(grad)=0

Example: shallow water:Prognostic variables velocity u ∈ V1 and depthD ∈V2. Vorticity ζ ∈ V0 can be used to diagnose aconsistent potential vorticity flux as in [2]

V0︸︷︷︸Continuous

∇⊥−−→ V1︸︷︷︸Continuous normals

∇·−→ V2︸︷︷︸Discontinuous

Figure 1:Top: lowest order function spaces on triangles; bot-tom: next-to-lowest-order function spaces on quadrilaterals

Gusto: the dynamical core toolkit

Gusto is dynamical core toolkit, built on top of the Firedrake finite element library, which enables rapidprototyping of algorithms based on the compatible finite element discretisations developed in the Gung Hoproject.

• compatible function spaces dictated by thelinear equations• stable, accurate advection schemes available• solves a range of GFD equations• can be run in different geometries• includes moist physics schemes

Some snapshots of simulation results can be seenon the right.

Top left: Potential vorticity from the shallow water flowover a mountain test case.Top right: Potential temperature of a falling bubble thathas spread across the bottom of the domain, developing aKelvin-Helmholtz instability.Bottom: Wet equivalent potential temperature andvertical velocity of a rising thermal in a cloudyatmosphere.

8

Figure 2. The (left) ✓e field contoured every 0.5 K and (right) vertical velocity w field contoured every 2 m s�1, with both fields plotted at t = 1000 s for a simulation ofthe moist benchmark case Bryan & Fritsch (2002) representing a thermal rising through a saturated atmosphere. The 320 K contour has been omitted for clarity in the ✓e

field. This simulation is with the k = 0 lowest-order set of spaces, with grid spacing �x = �z = 100 m and a time step of �t = 1 s. These solutions are visibly similarto those presented in Bryan & Fritsch (2002).

Figure 3. Outputted fields from the k = 1 next-to-lowest degree space simulation at t = 1000 s of the moist benchmark from Bryan & Fritsch (2002). (Left) ✓e withcontours spaced by 0.5 K and (right) vertical velocity w contoured every 2 m s�1. The simulation used grid spacing �x = �z = 100 m and a time step of �t = 1 s.The 320 K contour has been omitted for clarity in the ✓e field. A second plume can be seen forming at the top of the primary plume.

5. Test Cases

In this section we demonstrate the discretisation detailed inprevious sections through a series of test cases, with somecomparison of the k = 0 and k = 1 configurations of the model.Two new variants of test cases are presented, featuring a gravitywave in a saturated atmosphere and a three-dimensional risingthermal in a saturated atmosphere.

Throughout this section, x and y are the horizontal coordinatesand z is the vertical coordinate. For the two-dimensional tests, r

is given byr =

p(x � xc)2 + (z � zc)2, (26)

while for three-dimensional tests it is

r =p

(x � xc)2 + (y � yc)2 + (z � zc)2. (27)

5.1. Bryan and Fritsch Moist Benchmark

The first demonstration of our discretisation is through the moistbenchmark test case of Bryan & Fritsch (2002), which simulates

a rising thermal through a cloudy atmosphere. The domain is avertical slice of width L = 20 km and height H = 10 km. Periodicboundary conditions are applied at the vertical boundaries, but thetop and bottom boundaries are rigid, so that v · bn = 0 along them.As in Bryan & Fritsch (2002), we include no rain microphysicsand no Coriolis force.

The initial conditions defined in Bryan & Fritsch (2002)specify a background state with constant rt = 0.02 kg kg�1 andconstant wet-equivalent potential temperature ✓e = 320 K, whichis defined in (46). Along with these, the background state isgiven by the requirements of hydrostatic balance, rv = rsat andp = 105 Pa at the bottom boundary. The procedure described inAppendix B.2 allows us to obtain the prognostic variables ✓vd,⇢d, rv and rc that approximately satisfy these conditions.

The following perturbation is then applied to ✓vd

✓0vd =

⇢�⇥ cos2

�⇡r2rc

�, r < rc,

0, otherwise,(28)

c� 0000 Royal Meteorological Society Prepared using qjrms4.cls

Parallel timestepping schemes

Motivation:There is a limit to parallel speed up from spatial domain decomposition. Adding more processors leads toincreased communication costs.

A solution:Parallel exponential integrators The solution ofthe linear system is

U (t) = e−(t−t0)LU (t0)Compute the matrix exponential using therational approximation

eτLU (t0) ≈N∑

n=−N

βnτL + αn

U (t0)

where τ = t− t0, and αn, βn ∈ C are given in []Writing V = eτLU (t0) for each n we can solvein parallel :

(τL + αn)V n = βnU (t0)

1.03

-0.8

-0.4

0

0.4

0.8

h1r

-1.03

-1

0

1

u1r X

-1.56

1.7 0.634

-0.8

-0.4

0

0.4

u1r Y

-1.13

h, u and v for the wave benchmark for the linear f-planeshallow water equations [3]

-0.432

-1600

-1200

-800

-400

h1r

-1.82e+030

1e+4

2e+4

3e+4

4e+4

h1r

-7.79e+03

4.65e+04

left: h at t = 36000s, linear solid body rotation.right: h at t = 36000s, wave propagation with initialconditions given by a Gaussian bump in h at north pole.

References

[1] Andrew Staniforth and John Thuburn.Horizontal grids for global weather and climate prediction models: areview.Quarterly Journal of the Royal Meteorological Society,138(662):1–26, 2012.

[2] Jemma Shipton, Thomas H Gibson, and Colin J Cotter.Higher-order compatible finite element schemes for the nonlinearrotating shallow water equations on the sphere.Journal of Computational Physics, 375:1121–1137, 2018.

[3] Martin Schreiber, Pedro S Peixoto, Terry Haut, and Beth Wingate.Beyond spatial scalability limitations with a massively parallel methodfor linear oscillatory problems.The International Journal of High Performance ComputingApplications, 32(6):913–933, 2018.

[4] Colin J Cotter and Jemma Shipton.Mixed finite elements for numerical weather prediction.Journal of Computational Physics, 231(21):7076–7091, 2012.

[5] Andrea Natale, Jemma Shipton, and Colin J Cotter.Compatible finite element spaces for geophysical fluid dynamics.Dynamics and Statistics of the Climate System, 1(1), 2016.

[6] Hiroe Yamazaki, Jemma Shipton, Michael JP Cullen, LawrenceMitchell, and Colin J Cotter.Vertical slice modelling of nonlinear Eady waves using a compatiblefinite element method.Journal of Computational Physics, 343:130–149, 2017.

[7] Thomas M Bendall, Colin J Cotter, and Jemma Shipton.The ‘recovered space’advection scheme for lowest-order compatiblefinite element methods.Journal of Computational Physics, 390:342–358, 2019.

[8] Thomas M Bendall, Thomas H Gibson, Jemma Shipton, Colin JCotter, and Ben Shipway.A compatible finite element discretisation for the moist compressibleEuler equations.arXiv preprint arXiv:1910.01857, 2019.

Contact Information

•Web: https://www.firedrakeproject.org/gusto/•Email: [email protected]