Regional Earth System Model (RegESM) using … · Outline • Evolution of the RegESM modeling system • Basic Design • Common Problems and Solutions – Unaligned land-sea masks

Regional Earth System Model (RegESM) using NUOPC/ESMF���

���Ufuk Turuncoglu (1,2)���

���������

(1) Informatics Institute, ���Computational Science and Engineering, ���

ITU, Turkey���(2) ESP Section, ���

ICTP, Italy

The Third Workshop on Coupling Technologies for Earth System Models, 20-22 Apr. 2015

Outline•  Evolution of the RegESM modeling system•  Basic Design•  Common Problems and Solutions

–  Unaligned land-sea masks–  Imposing rivers (source or surface boundary conditions)

•  New component–  Wave

•  Performance Results

Why do we need new model?•  To gain experience about design and use of coupled modeling

systems–  The applications of coupled modeling systems getting increased in

last decade along with the rapid development in HPC–  Need to consider different components of the climate system

•  To design easy to use and extend modeling system–  ENEA’s (Italian Energy Agency) PROTHEUS system was based on

MITgcm+RegCM3+OASIS–  It is not easy to use, extend and upgrade–  It uses multiple executable, no online interpolation support

•  To support Med-CORDEX and other applications …–  Set of coordinated experiments to better understand

Mediterranean climate: standalone and coupled RCMs–  It would be good to have another regional coupled modeling

system that uses different model components (ensemble)

Evolution of Regional Earth System Model (RegESM)•  The first prototype version is created in 2012 @ICTP

–  No driver–  RegCM was hosting other components of the coupled model–  Single ocean component was supported (ROMS-Ice)–  Poor energy conservation (for exchange fields)–  Applied to Caspian Sea (Turuncoglu et al., 2013, GMD)–  Hard to extend (i.e. adding new components as wave, river routing)

•  Then, more generic version is designed and released in 2013-… @ITU–  Centralized driver using ESMF’s NUOPC layer (via connectors)–  All the components plugged into driver (less dependence to the model

component code)–  Support for two different ocean model (ROMS and MITgcm)–  Energy conservation is improved (custom global conservation using bilinear)–  Support for unaligned land-sea masks–  Applied to Mediterranean, Black Sea and Caspian Sea (ITU)–  More applications are on the way - Caribbean (ICTP), High Res.

Mediterranean (ICTP and ENEA), Indian Ocean, Antarctica (ITU)

Design•  Components:

ATM:ICTP’s RegCM (4.4.5.4)

OCN:Rutgers Univ. ROMS (r737)���MIT, MITgcm (c63s)

RTM:Max Planck’s HD (mod. 1.0.2) special thanks to ���Prof. Stefan Hagemann

WAV:ECMWF’s WAM (4.5.3 MPI)

DRV:RegESM (7.0.0b38)

atmosphere(RegCM)

land(BATS)

ocean(ROMS / MITgcm)

sea iceunnamed

driver

land(CLM)

river routing

(HD)

initial and boundaryconditions

anthropogenicemissions

naturalemissions

full gaschemistry

aerosol

wave(WAM)

initial conditions

initial conditions

ESMF+NUOPC

in progress ready and tested

atmospheric���chemistry

Two differentland surface ���

model

Two differentoceanmodel

representation is barrowed from ���Alexander and Easterbrook

ESMF/NUOPC in RegESM•  Model components are coupled using NUOPC “connectors”

–  AAA•  The modeling system has single executable

•  Easy to use and debug•  Library for all components (i.e. libatm.a)

•  The connector uses all PETs of the ���active model components

•  NUOPC layer help to manage:•  Execution order of the components•  Synchronization (i.e. slow vs. fast coupling���

time steps) •  Exchange fields (via field table)•  …

Run Sequence in ESMF/NUOPC•  The NUOPC Layer is capable of implementing many different

coupling schemes–  Explicit:

•  Exchange data in same time. Two ���connector (i.e. backward and ���forward)

–  Explicit with slow and fast ���clock support:•  Different interaction time ���

step among the components–  Semi-implicit (leap-frog style ���

interaction)–  Fully-implicit

•  Complex interaction among the���components.

ATM

OCNt=0h t=1h ...

ATM-OCNOCN-ATM

coupling time

Ex: 1 hour

EXPLICIT RUN SEQUENCE

Initializationof models

ATM

OCNt=0h t=1h ...

SEMI-IMPLICIT RUN SEQUENCE

ATM

OCNt=0h t=1h ...

IMPLICIT RUN SEQUENCEt=1/2 h

2

34 5

1 2

1

2

34

2

1

2

34

2

34 5

4

3

1

3

4

data exchangemodel integrate

x2

16

16

7 7

Run Sequence in RegESM•  The RegESM uses explicit coupling schemes along with the

support of fast and slow time steps.–  Fast interaction among ATM, OCN and WAV (i.e. 1 or 3 hours)–  Slow interaction between ATM and RTM, RTM-OCN (i.e. 1 day)–  Semi-implicit type coupling is also implemented.

i.e. three component case (ATM+OCN+RTM) + explicit coupling

Special thanks to ESMF Group (especially to Gerhard Theurich) for their support and help

Sequential vs. Concurrent Execution•  RegESM also supports two different approach to run the model

components.–  Sequential: model components are run in order–  Concurrent: all models are active at same time (it does not allow

overlapping of the used PETs – cores / CPUs)•  User able to assign last PET or individual PET to RTM (seq.)

i.e. three component case (ATM+OCN+RTM)

Managing Exchange Fields•  Exchange fields between model components are defined by

extra configuration file (exfield.tbl)

•  The user can chose the exchange fields from the field pool

extrapolation is activated  conservation ��� is activated  

add_offset and���scale_factor used for���

unit conversion  

interpolation ���type   stencil for source���

and destination grids���used in interpolation  

exchange fields(set of pre-defined���fields can be used)  

Unaligned Land-Sea Masks•  The unaligned land-sea mask might produce unrealistic heat

fluxes around coastlines•  It is more apparent when horizontal resolution differences are

high among model components (i.e. 50 km ATM, 8 km OCN)•  The interpolation must be performed only over sea for ATM-

OCN components or only over land for other interactions•  Possible solutions:

–  Manual editing of land sea masks•  It is not generic solution and must be repeated when horizontal ���

grids and application are changed L–  Two-step interpolation

•  It is generic and independent from application J•  It might produce high interpolation error around coastlines and

complex land-sea mask features L

Unaligned Land-Sea Masks …•  Our solution: Two-step interpolation

–  Need to find mapped and unmapped grids L–  It is performed only once J

•  The algorithm���modifies land sea���mask and defines���mapped and���unmapped grid���points

•  It is activated by���driver configuration ���file

Special thanks to ESMF Group (especially to Bob Oehmke) for their support and help

Unaligned Land-Sea Masks …•  Two-step interpolation:

•  Other extrapolation methods ???

STEP 1

STEP 2

STEP 3

1.  Interpolate from ATM to OCN���using bilinear type interpolation. Use only sea grid points

2.  Use result of previous step (step 1) and interpolate data from OCN to OCN from mapped grid points ���to unmapped ones (over sea) using���nearest-neighbor type regridding.

3.  Merge results of step 1 and 2 to���create result field

Interpolation Error•  Pseudo spherical harmonics (L=32, M=16) from SCRIP

•  The error increases in the regions that has complex land-sea mask and when the resolution difference is high

•  The error is also depend on the field itself and its behavior. ���More tests with realistic fields are needed !!!

f = 2+ sin16 (2θ )cos(16φ) ⇒ θ = lat, φ = lon

50 km ATM 1/12° (~8km) OCN

Interpolation Error …•  Two step interpolation (ATM to OCN)

Min: -0.207���Max: 0.071  

Min: -0.025���Max: 0.014  

LRES

ATM

- 50

km  

HRE

S ATM

- 10

km  

Relative Error = (Mod/Obs)-1  

•  High error around coastline���especially Azov Sea

•  The error pattern might be���different with different field  

•  Errors are reduced when it is compared with 50 km ATM case

•  Keep the resolution difference minimal !!!  

RTM Grid

OCN Grid

River

R

unaffectedgrid points

effectedgrid points

1

2

3

1 Find river location (i,j) in RTM grid

2 Find closest OCN model grid to RTMgrid

3 Find OCN grids in effective radius (R)

Algorithm

Interaction between OCN and RTM•  How we define rivers in ocean models?

–  As source / sink points–  As surface boundary conditions (BC) such as rain or E-P

•  The RegESM support both of them–  Fully automatized algorithm���

to distribute river discharge���as surface BC

–  Also, possible to change���effected grids as a function of ���river discharge !!!

river position as grid coordinates (i,j) or geographic coordinates (lat,lon)

represent as constant monthly values or RTMeffective radius  

Interaction between OCN and RTM•  How we define rivers in ocean models?

–  The Max Planck’s HD model has 0.5x0.5 fixed global grid–  The temporal resolution is also fixed (1-day)–  Only major rivers are represented –  It calculates river discharge using surface+sub-surface runoff ���

if the grid point has data–  The surface+sub-surface runoff came from ATM

salt from ATM+OCN+RTM  

Domain specific ���routing algorithms ���might be ���implemented to represent rivers more accurately?

active   monthly values  

Do we really need wave coupling?•  The RegCM model generally overestimates the wind speed over

the sea due to poor representation of air-sea interface

•  In general, the standalone atmosphere model tends to overestimate the wind speed but it is more dominant in winter season

•  It also effects the circulations in the ocean components

Observational ���data is satellite ���derived 0.25° ���resolutionBlended Sea ���Winds ���(Zhang et al., 2006)

1989-2010 ���climatology ofERA-Interim andNCEP/NCAR runs

Parameterization of Ocean Fluxes in RegCM•  The model supports three different scheme to represent ���

ocean fluxes–  BATS1e

•  It uses Monin-Obukhov Similarity Theory (MOST). –  Zeng et al., 1998

•  Widely used parameterization implemented in RegCM 4.X that allows more sophisticated representation of air-sea interface

–  COARE bulk flux algorithm (experimental)•  It is implemented for ocean-atmosphere coupling. It also includes

representation of fluxes for sea-ice condition•  The latest version of the RegCM model (4.4.X) also includes

slab ocean model–  Based on GFDL FMS Slab ocean model (or mixed layer model)

•  islab_ocean =1

Parameterization of Ocean Fluxes in RegCM•  Representation of surface roughness length of momentum (z0)

in Zeng Ocean Flux Algorithm (Zeng et al, 1998)

iocnrough = 1

iocnrough = 2

•  Roughness and drag coefficient increases with wind speed

zo = βu*2

g0.0065 ���

(Charnock’s relation, 1955)

Friction velocity

Acceleration of gravity

zo = βu*2

g+α

νa

u*0.013 0.011

Kinematic viscosity of ���dry air (Andreas, 1989)

Smooth flow contribution ���at low wind speeds

(Smith, 1988)  

CD =u*U(z)!

"#

$

%&

2

U(z) = u*κ log(z z0 )

Horizontal ���wind speed

u*2 =

τ aρa

Wave Coupling•  ECMWF’s wave model (WAM) is selected•  WAM is a 3rd generation model

–  The model is slightly modified for ESMF coupling•  Different coupling types?

–  Depends on used model components and application–  It is better to support as much as possible combination–  Need to add threshold for roughness length in ATM for stability

ATM  

WAV  

Wind  or  stress  

Charnock  

ATM  

WAV  

Wind   Roughness  Length  

ATM  

WAV  

Wind   Phase  Speed  +  Mean  Wave    

Dir.  

ATM  

WAV  

Wind   Sig.  Wave  Hgt.  +  Wave  Length  

Domain Decomposition in WAM•  2D domain is mapped onto 1D array of following increasing

latitude lines

•  It seems that it is a common convention in wave modeling (i.e. WAVEWATCH-III) but ESMF does not support yet …

latitu

de

longitude

1 2 3 4 5 6

7 8 9 10 11 12

13 14 15 16 17 18

19 20 21 22 23 24

25 26 27 28 29 30

31 32 33 34 35 36

8 9 10 11 13 14 15 16 17 18 19 20 21 22 23 24 26 27 28 29 33 34

Total number of active sea points

CPU(0) = 5 CPU(1) = 5 CPU(2) = 4 CPU(3) = 4 CPU(4) = 4

Fortran:•  PACK and UNPACK routines (using

land-sea mask) are used in WAM

To define grid in ESMF:•  2d decomposition is used•  Data is collected in root PET to

exchange (better approach needed!)

1D decomposition  

land

sea

Initial Results•  ATM-WAM coupling in Mediterranean

ATM

WAV

2008-2012WAM Resolution: 0.125

•  ATM-WAV coupling���help to reduce wind���speeds in all reasons.

•  It is more apparent in ���JJA.

Surenkok and Turuncoglu, EGU, 2015  

EMOD-Net Observations  

Performance of the Modeling System•  Test with Mediterranean Domain

–  RegCM (Ext. + Default)���12 km – 24 layer

–  ROMS: ���1/12 – 32 sigma layer

•  Test with different coupling���interval (30m / 1-hour / 3-hour)

•  Test with different execution type (sequential vs. concurrent)•  Test with different number of component (ATM-OCN, ATM-

OCN-RTM)•  Environment:

–  CURIE, France (provided by PRACE - 2010PA2442)–  Infiniband (QDR) performance network–  Fat (128 core/node) and Thin (16 core/node) Nodes (Intel Xeon)

ATM-Extended

ATM

OCN

•  Individual Model components: RegCM and ROMS

Benchmark Results

•  It also aims to find best 2d decomposition parameters���for OCN omponent

•  The scaling results are better in ATM-Extended domain

•  Feed the processors !!!

•  Effect of coupling time step

•  The effect is very limited between 1-hour ���and 3-hours cases (around %2-5).

•  The results of 30 min case shows more fluctuations. It is better to do additional tests for 30 min case.

•  Currently, there is no available computing resource L

Benchmark Results …#  core   %  diff  

30m/1hr  %  diff  

3hr/1hr  

64   -­‐0.38   1.29  128   1.06   2.15  192   1.45   2.89  256   1.86   1.58  288   9.20   0.95  320   -­‐1.27   5.55  336   17.50   3.84  384   29.66   0.18  416   13.04   -­‐0.92  528   13.50   -­‐4.83  576   9.57   -­‐0.75  640   1.93   8.68  AVG   8.09   1.72  

•  Effect of number of component: ATM-OCN / ATM-OCN-RTM

•  Also need to test concurrent case.•  RTM component could be tightly coupled with ATM model•  The current coupled model uses connector. •  It is hard to find optimum PET distribution in concurrent case.

More tests are needed …

Benchmark Results …

Last PET is shared ���between OCN and ���RTM

Having additional ���component for RTM ���reduces performance  

Repository and Project Home•  RegESM project is hosted by GitHub���

https://github.com/uturuncoglu/RegESM

•  The user documentation is ready–  Basic model design–  Installation of libraries and model ���

components–  Definition of configuration files–  Known bugs and limitations–  Bugs:

•  Handled by GitHub•  Model is tested in different���

architectures (IBM Blue Gene)

Future Plans•  The modeling system needs more accurate (+ parallel ?) river

rooting component–  RAPID (The Routing Application for Parallel computation of

Discharge), https://github.com/c-h-david/rapid/•  Ocean + wave interaction is still missing. It must be defined ���

for both ROMS and MITgcm cases•  Support for ESMF’s high order conservative regridding•  Making modeling system self-describing (via ESMF Attributes)

and integrating with scientific workflows (SWf)•  We need more application to test the modeling system•  Build a user community

–  ENEA, ICTP @ Italy + ITU @ Turkey–  Training events related with coupled model design and use …

Questions!

Contact:[email protected]

This  study  has  been  supported  by  a  research  grant  (113Y108)  provided  by  The  ScienPfic  and  Technological  Research  Council  of  Turkey  (TUBITAK)  and  partly  by  The  Abdus  Salam  InternaPonal  Center  for  TheorePcal  

Physics  (ICTP)  Associateship  Scheme.  The  compuPng  resources  used  in  this  work  were  provided  by  the  NaPonal  Center  for  High  Performance  CompuPng  of  Turkey  (UHEM)  under  grant  number  5003082013.  We  also  

acknowledge  PRACE  for  awarding  us  access  to  resource  CURIE  based  in  France  at  Très  Grand  Centre  de  Calcul  (TGCC)  under  project  number  2010PA2442.