Technical aspects of IPSL Earth System model · Technical aspects of IPSL Earth System model Arnaud Caubel (IPSL, France) ... • NEMO_v3.6 (ocean), LIM3 (sea-ice) ... or single (parallel

Technical aspects of IPSL Earth System model

Arnaud Caubel (IPSL, France)

April 20th 2015, Workshop on Coupling Technologies for Earth System Models

Context : IPSL Earth System model

LMDZ

(atmosphere)

OASIS3-MCT

(coupler)

ORCHIDEE

(land surfaces,

vegetation)

INCA/REPROBUS

(atmospheric

chemistry, aerosols)

OPA

(ocean)

LIM

(sea ice)

PISCES

(marine biogeochemistry)

NEMO

• IPSL Earth System model

– History of IPSL-CM : use of previous versions of OASIS as coupler.

– Different configurations at different resolutions are set up and maintained around the physical atmosphere-land-ocean-sea ice model : carbon cycle, stratospheric chemistry, tropospheric chemistry, aerosols,…

– From paleoclimate to decadal studies

– Web page : http://icmc.ipsl.fr/

Context : IPSL Earth System model

Atmospheric LMDZ

lon-lat grid Oceanic NEMO

curvilinear grid OASIS

• IPSLCM5 version (CMIP5 simulations), IPSLCM6 is the current version.

New developments in IPSLCM6 are :

– Update of components versions

• NEMO_v3.6 (ocean), LIM3 (sea-ice)

• new physics for LMDZ atmospheric model

• …

– OASIS3-MCT parallel coupler

– MPI-OpenMP hybrid parallelization for atmospheric /land /atmospheric chemistry components

– XIOS (XML-IO-Server) as output library for all IPSL components

• library dedicated to IO management of climate code, developped at IPSL by Y. Meurdesoif

• XML configuration file

• attached mode (library) or server mode (asynchroneous transfer), multiple (sequential writing) or single (parallel writing) output file

• NetCDF format (GRIB2 in progress, ICHEC collaboration)

IPSL Earth System model : technical developments

Atmosphere LMDZ

Land ORCHIDEE

Chem/Aerosols INCA

Ocean

Sea-Ice

Biogeochemistry

NEMO

Oasis

MCT

Oasis

MCT

MPI process MPI process

OpenMP thread

OpenMP Master thread XIOS

client XIOS

client

XIOS

server XIOS

server

Asynchroneous mode Asynchroneous mode

one output NEMO ocean file

one output NEMO sea-ice file

one output NEMO bio file

one output atmospheric file

one output land file

…

…

Temporal Series TS files

Two computing centers : Curie TGCC, Ada IDRIS

Atmosphere LMDZ

Land ORCHIDEE

Chem/Aerosols INCA

Ocean

Sea-Ice

Biogeochemistry

NEMO

Oasis

MCT

Oasis

MCT


OpenMP thread


client XIOS

client

XIOS

server XIOS

server







…

…



Atmosphere LMDZ

Land ORCHIDEE

Chem/Aerosols INCA

Ocean

Sea-Ice

Biogeochemistry

NEMO

Oasis

MCT

Oasis

MCT


OpenMP thread


client XIOS

client

XIOS

server XIOS

server







…

…



Atmosphere LMDZ

Land ORCHIDEE

Chem/Aerosols INCA

Ocean

Sea-Ice

Biogeochemistry

NEMO

Oasis

MCT

Oasis

MCT


OpenMP thread


client XIOS

client

XIOS

server XIOS

server







…

…



Atmosphere LMDZ

Land ORCHIDEE

Chem/Aerosols INCA

Ocean

Sea-Ice

Biogeochemistry

NEMO

Oasis

MCT

Oasis

MCT


OpenMP thread


client XIOS

client

XIOS

server XIOS

server







…

…



IPSLCM6 : recent technical developments : what is the gain ?

Components

– MPI-OpenMP hybrid parallelization for atmospheric /land /atm chem components

• more computing cores used (distributed memory for horizontal decomposition, shared memory for vertical decomposition)

• reduces elapsed time of a simulation

• target efficiently SMP architectures (shared memory intra-node, distributed memory inter-node)

– XIOS (XML-IO-Server) as output library

• server mode (asynchroneous mode for writing) reduces elapsed time of the simulation

• post-processing steps (i.e output files rebuild step, times series) not needed anymore= running environment more reliable.

• XML configuration file : flexibility and easy handling

Whole coupled model

– OASIS3-MCT parallel coupler

• parallel use of interpolation weights

• no significative impact when increasing resolution, coupling frequency,…

– Complex application

• MPMD model, diffferent levels of parallelization (MPI for ocean, MPI+OpenMP for atmosphere, IO servers distributed on each computing node,… )

• Not a standard application for computing centres (process assignment, tools,…)

• IPSL context : several computing centres

IPSLCM6 : performances

• Load balancing (difference between computing time of components) ?

– elapsed time of coupled model = elapsed time of slowest component (concurrent mode)

– how to measure load balancing ?

• Not easy to use standard tools to analyse : « vampir », « paraver »,…

• LUCIA : tool to measure performances within an OASIS3-MCT-based coupled system.

• Developped by E. Maisonnave (CERFACS)

• Easy to use : activation via OASIS namcouple, succint metrics (computing time and waiting time are integrated over all model processes and over all the simulation).

• http://www.cerfacs.fr/globc/publication/technicalreport/2014/lucia_documentation.pdf

Waiting time model 2

model 1

lmdz

model 2

nemo

Waiting time

model 2

Vampir trace LUCIA

Waiting time model 2

• IPSLCM5 version used for CMIP5 simulations in 2010-2011

• IPSLCM6 used for CMIP6 simulations in 2016-2017, different resolutions :

– Low Resolution : Atm: 2.5⁰ x 1.5⁰ L79 Oce: 1⁰ L75

– High Resolution : Atm: 0.9⁰ x 0.4⁰ L79 Oce: 0.25⁰ L75

IPSLCM6 : performances and plans

0

2

4

6

8

10

12

0 100 200 300 400 500 600 700

NB

of

sim

ula

ted

yea

rs p

er d

ay

NB cores

ORCA1_LIM3_PISCES

LMDZ 144x143x79 (NP)

LR Ocean component NEMO 1° L75

LR Atmospheric component LMDZ 2° L79

Curie SMP machine

Intel Sandybridge






0

2

4

6

8

10

12

0 100 200 300 400 500 600 700

NB

of

sim

ula

ted

yea

rs p

er d

ay

NB cores

ORCA1_LIM3_PISCES




4 OMP thread

Speedup 3.75

8 OMP thread

Speedup 5.6

Curie SMP machine

Intel Sandybridge






0

2

4

6

8

10

12

0 100 200 300 400 500 600 700

NB

of

sim

ula

ted

yea

rs p

er d

ay

NB cores

ORCA1_LIM3_PISCES




IPSLCM6-LR Atmospheric model LMDZ 2°L79

Ocean model NEMO 1°L75

480 CPUs

6 years per day

Curie SMP machine

Intel Sandybridge

• Recent technical developments of IPSLCM

– MPI-OpenMP, OASIS3-MCT, XIOS : faster than previous IPSLCM5 version

– XIOS : flexibility and running environment more reliable (good point for CMIP6 output requirements)

– more complex application : not standard use on architecture (process assignment, tools,…), not easy on IPSL context with many computing centers.

• CMIP6 simulations : intensive production mode

• Next developments

– LIM3 as separated executable

– XIOS with more functionalities

– parallel reading (forcing files,…)

– on line post-treatment : grid remapping, interpolations, zonal means , grid coarsening, downscaling, global/partial reduction

– Atmospheric component : new dynamical core DYNAMICO (2017 for HighRes MIP ?)

– better performances/scalability

– MPPs, MICs, GPUs architectures.

IPSLCM conclusions and perspectives

Lon-lat grid Icosahedral grid

Technical aspects of IPSL Earth System model · Technical aspects of IPSL Earth System model Arnaud Caubel (IPSL, France) ... • NEMO_v3.6 (ocean), LIM3 (sea-ice) ... or single (parallel

Documents