Welcome to the Practical NWP Exercises · Welcome to the Practical NWP Exercises Ulrich Schättler, Ulrich Blahak Deutscher Wetterdienst ... To do the practical exercises, we have

Welcome to the

Practical NWP Exercises

Ulrich Schättler, Ulrich Blahak

Deutscher Wetterdienst

BU Research and Development

Department for Numerical Modelling

Alexander Smalla

Zentrum für GeoInformationswesen

der Bundeswehr

Gruppe Meteorologie beim DWD

Welcome to the

Practical NWP Exercises

Ulrich Schättler, Ulrich Blahak

Deutscher Wetterdienst

BU Research and Development

Department for Numerical Modelling

Alexander Smalla

Zentrum für GeoInformationswesen

der Bundeswehr

Gruppe Meteorologie beim DWD

What you will learn this week

Installation of the COSMO-Model package

Preparing External, Initial and Boundary Data

Running the COSMO-Model in NWP mode

Visualizing COSMO-Model output using GrADS (NWP)

Running Idealized Test Cases

Troubleshooting for the COSMO-Model

Individual / Practical Exercises

3 COSMO/CLM/ART Training 2015

Basic Informations

To do the practical exercises, we have got 16 terminals, but we are about

28 trainees. So it would be good to share your terminal.

You need some knowledge in working with Linux / Unix and how to use a

text-editor. If you are not familiar with this, you should look for an

experienced partner.


Installation

In this lesson you will learn

how to login to and use DWD‘s supercomputers, the Cray XC30, and its

frontend, the xce

what are the necessary compontents of the COSMO-Model software

package

how these components can be installed on a computer

how jobs can be started

what to do with the reference data

And you will do your first simulations based on the reference data.


Logging in

To the PC

There is a plastic sheet on your desk with the

Username (Benutzername)

Password (Kennwort)

And the Domain (Domäne)

Start Cygwin Server: Click Icon on the Desktop

Start a Putty session (see extra sheet on your desk)

Terminal-Keyboard

Windows- Colours

Connection-SSH-X11

Set host name and save session

Log in to xce (xce00.dwd.de, xce01.dwd.de, xce02.dwd.de, xce03.dwd.de)

Username

Password


The Computers at DWD

xce[00/01/02/03]

These are special nodes on the Cray XC30 which are used as login-nodes (file handling, compiling, linking, starting jobs, visualization)

There are 3 different filesystems:

/e/uhome/trngxxx: for source code and run-scripts

/e/uwork/trngxxx and /e/uwork/uscratch: for model data

The Cray XC30

INT2LM and the COSMO-Model run on the Cray XC30 machines.

It has 364 compute nodes, each node has 2 Intel processors (IvyBridge) with 10 cores each (adds up to 7280 cores).

You should not use more than 5-10 nodes for a COSMO-Model job (how to that you will see later). INT2LM usually runs on one node.


The Computers at DWD

The Cray XC40

There is another partition of the computer which is used for ICON

It has 432 compute nodes, each node has 2 Intel processors (Haswell) with 12 cores each (adds up to 10368 cores)


Compiler Environment and Libraries

Compilers: We use the Cray Compiling Environment (cce/8.3.3)

Grib-API: Pre-installed library: grib_api 1.11.0

NetCDF: Pre-installed library: cray-netcdf/4.3.2

Type

module list for other pre-installed libraries / components

module avail for available libraries / components and their versions

To access the pre-installed libraries, you have to use the environment variables:

$(CRAY_LD_LIBRARY_PATH)


The Batch System PBS

Jobs are submitted to the Cray XC30 with the batch system PBS (Portable Batch System).

The basic commands are:

qsub jobname: to submit the job described in jobname.

qstat: to query the status of all batch jobs

qstat | grep xc_normal: to query the status of all batch jobs on XC30

qstat –u <usrname>: to query the status of your batch jobs

qdel <jobnr>: to delete the job with the number <jobnr> . <jobnr> is given by qstat.

At the end of each job you get an output file <jobname>.o<jobnr> with status information (possible errors, successful job termination) and standard ASCII output of the program

Use xtnodestat to see, how many nodes are free

Some more information can be found in Appendix A of the Tutorial


NWP Exercises

Files and Data for these exercises can be accessed via

/e/uhome/fegast3/TRAINING_2015/

auxiliary files that might be necessary for running

cosmoart for COSMO-ART

data the reference data set

docu available documentation

grads files for the visualization exercises

ideal run-scripts for the idealized test runs

source tar-files containing the necessary packages

topo external parameters for COSMO / ICON / GME

troubleshooting files for the troubleshooting exercises


Files and Data

From /e/uhome/fegast3/TRAINING_2015/source/ you should copy the following files to your home directory:

DWD-libgrib1_110128.tar.gz

int2lm_141125_2.01.tar.bz2

cosmo_141128_5.01.tar.bz2

From /e/uhome/fegast3/TRAINING_2015/data you should copy the following file to your work or scratch directory

reference_data_5.01.tar.bz2

Unzip and de-tar all files and try to install the programs according to theTutorial 1.2 and 1.3 and the reference data according to 1.4

In addition you need:

grib_api: from /e/uhome/fegast3/opt/grib_api/lib, …/include, …/bin

NetCDF library: available as system library


Installing the

GRIB1 Library


GRIB

GRIB is a WMO standard for packing and distributing meteorological fields.

GRIB1: GRIdded Binary: still used in the COSMO Software based on

DWD libgrib1.

GRIB2: General Regularly-distributed Information in Binary form:

implemented in the COSMO Software lately based on ECMWF Grib-API

freely available from https://software.ecmwf.int/wiki/display/GRIB/Releases

GRIB coded data consist of sequences of octets (1 octet = 1 byte) and is

not human readable (also not the metadata). You need some tools to examine the data (from grib_api or wgrib)


https://software.ecmwf.int/wiki/display/GRIB/Releases

The Directory Structure

tar xvf DWD-libgrib1_110128.tar(.gz) gives the directory DWD-libgrib1_110128 with subdirectories

include: directory with include files

README_grib: File with some more details

source: subdirectory with all source files and a Makefile

Go to directory DWD-libgrib1_110128/source and take a look to the Makefile (using a text-editor of your choice)


The Makefile

You have to set the following variables

LIBPATH path where the library is written to

INCDIR path for the include directory (do not change!)

AR archiving utility (usually ar)

FTN Call to the Fortran compiler (choose Cray environment)

FCFLAGS Fortran compiler flags (choose Cray Linux)

F90FLAGS Fortran 90 compiler flags

CC Call to the C-compiler

CCFLAGS C compiler flags

Save the changes, close the text-editor and type: make


Big Endian / Little Endian

Most (Unix) supercomputers today use big-endian processors. This means that the most significant (biggest) byte is stored first.

Typical PC (Intel) Processors (with Linux) do it the other way round and store the less significant (smallest) byte first. This is called little endian.

Grib, by definition, is big-endian!

Example: every Grib record starts with 4 bytes, that contain the 4 ASCII signs: G R I B

Representation in big endian: G R I B

Representation in little endian: B I R G

On little endian machines, a byte-swapping has to be done when reading Grib-files. This is implemented in the Grib library and can be switched on by defining –D__linux__


DWD Specialities for GRIB1

There are two DWD Grib1 specials when working with the DWD Grib1 library that have to be taken care of:

Controlwords between Grib Records: Every Grib record begins with „GRIB“ and ends with „7777“. DWD Grib files contain 8 extra bytes (a controlword) between Grib records. For a proper treatment of these controlwords, an environment variable has to be set: LIBDWD_FORCE_CONTROLWORDS=1

ASCII Bitmaps: Not the full data, but only a subset of the GME data is sent as driving data. The mapping of the data to the grid is done by a bitmap in ASCII format (by GRIB definition it should be binary). To indicate the format, another environment variable has to be set: LIBDWD_BITMAP_TYPE=ASCII


Installing the

INT2LM



tar xvf int2lm_yymmdd_x.y.tar(.bz2) gives the directory int2lm_yymmdd_x.y with subdirectories and files

1.11.0 / 1.12.3 definition files necessary for grib_api

DOCS change documentations (.htm) and User Guide (.pdf)

LOCAL examples for different compilers

Makefile, ObjDependencies, ObjFiles

Fopts to specify the Compiler Options

runeu2de, rungme2eu, runicon2eu

Run scripts to start the program

RUNSCRIPTS more run-scripts examples

src directory with the source code

obj directory with object files

work


Makefile

Makefile organizes the compiling and linking of the program. It includes

Fopts see next slide

ObjFiles List of all files that have to be compiled

ObjDependencies Compiler calls for all files

Makefile also defines the possible actions:

parexe build an executable (binary) for a parallel machine

seqexe build an executable for a pure sequential machine

parexe is the default


Fopts

Fopts contains the compiler and linker calls together with compiler options and the necessary libraries to load. Open the file in a text-editor and set the following variables:

F90 Fortran compiler call with basic options:

special options: -DGRIBDWD –DGRIBAPI -DNETCDF

COMFLGx Options for Fortran compiler

LDPAR Linker call for parallel programs

LDSEQ Linker call for sequential programs

LDFLAG Options for linking

LIB Directories and names of used libraries

OR: choose an appropriate Fopts-example from LOCAL and copy it to ./Fopts

Type make exe or gmake –j 8 exe to create the binary


About External Libraries: DWD GRIB1 Library

-DGRIBDWD

DWD‘s GRIB1 library is used since the beginning of the COSMO-Model

development and we still recommend to use it for writing GRIB1 files.

The pragma –DGRIBDWD therefore has to be set as compiler option.

If it is not set, all parts using the DWD Grib1 library are not compiled and

used.


About External Libraries: ECMWF grib_api

-DGRIBAPI

ECMWF‘s grib_api has been implemented into the COSMO-Model and

INT2LM in 2014, but still is not used outside DWD.

We are in the process of writing a comprehensive documentation how to

use the grib_api, but are not yet finished.

But you need grib_api to read DWD‘s GRIB2 data from ICON and GME in the INT2LM, therefore you have to set –DGRIBAPI

The installation process for grib_api takes a longer time, therefore we use

a pre-installed library this week from

/e/uhome/fegast3/opt/grib_api/include include files

/e/uhome/fegast3/opt/grib_api/lib library files

/e/uhome/fegast3/opt/grib_api/bin tools


About External Libraries: NetCDF

-DNETCDF

INT2LM also needs the NetCDF library, if ICON data have to be read. Therefore you need to compile with the pragma -DNETCDF

If –DNETCDF is not set, the corresponding parts of the code are not

compiled and cannot be used.

If compiling with NetCDF, you can also use NetCDF as I/O format for

reading and writing COSMO-Model data.

The NetCDF library is pre-installed and can be accessed on the Cray by loading the appropriate module: cray-netcdf


Installing the

COSMO-Model



tar xvf cosmo_yymmdd_x.y.tar(.bz2) gives the directory cosmo_yymmdd_x.y with subdirectories and files

DOCS change documentations (.htm) and User Guide (.pdf)

LOCAL examples for different compilers

Makefile, ObjDependencies, ObjFiles

Fopts to specify the Compiler Options

run_cosmo_eu, run_cosmo_de

Run scripts to start the program

RUNSCRIPTS more run-scripts examples

src directory with the source code

obj directory with object files

work


Makefile

Makefile organizes the compiling and linking of the program. It includes

Fopts see next slide

ObjFiles List of all files that have to be compiled

ObjDependencies Compiler calls for all files

Makefile also defines actions to build different executables

purpar pure COSMO-Model without Nudging and RTTOV

allpar including Nudging and RTTOV libraries

nudpar COSMO-Model with Nudging but without RTTOV

satpar COSMO-Model without Nudging but with RTTOV

allpar is the default. All binary variants could also be build as sequential executables without message passing (MPI)


Fopts

Fopts contains the compiler and linker calls together with compiler options and the necessary libraries to load. Open the file in a text-editor and set the following variables:

F90 Fortran compiler call with basic options:

special options: -DGRIBDWD –DGRIBAPI –DNETCDF –D__COSMO__

COMFLGx Options for Fortran compiler

LDPAR Linker call for parallel programs

LDSEQ Linker call for sequential programs

LDFLAG Options for linking

LIB Directories and names of used libraries

OR: choose an appropriate Fopts-example from LOCAL and copy it to ./Fopts

Type make exe or gmake –j 8 exe to create the binary


About External Libraries

In the COSMO-Model we still recommend to work with DWD‘s GRIB1 library and to set –DGRIBDWD

Grib_api is not really needed then, but it does not harm to also compile with -DGRIBAPI

NetCDF is necessary, if you

want to use it as I/O format (usually done by CLM group)

want to run the Nudging scheme, because observations are now taken as

NetCDF

During this week we compile with –DNETCDF, but we do not use it.

You definitely have to set –D__COSMO__ to use proper parts of the physical

parameterizations (which are shared with the ICON model)


The Reference Data Set


Installation

Copy /e/uhome/fegast3/TRAINING_2015/data/reference_data_5.01

to your $WORK directory and de-tar it

Directory structure

COSMO_2_input input data for 2.8 km application

COSMO_2_output output from 2.8 km application

COSMO_2_output_ascii ASCII output from 2.8 km application together

with a run-script to start model

COSMO_7_input input data for 7 km application

COSMO_7_output output from 7 km application

COSMO_7_output_ascii ASCII output from 7 km with run-script

GME_2013051312_sub necessary GME input data for INT2LM

int2lm_2_output_ascii ASCII output from INT2LM 7 2.8 km with run-script

int2lm_7_output_ascii ASCII output from INT2LM GME 7 km with run-script


How to Use the Reference Data Set

In the following we want to run

INT2LM: for interpolating the GME data to a 7 km COSMO-Grid

COSMO-Model: on this 7 km grid

INT2LM: for interpolating the COSMO 7 km grid output to a 2.8 km grid

COSMO-Model: on this 2.8 km grid

We want to keep the ASCII output from the reference data set to compare our

results with this example output

Therefore copy the run-scripts to your int2lm- or cosmo- directories, resp.,

to start the jobs

What is a Run-Script???


Run-Scripts


Runscripts

Runscripts take over most of the tasks necessary to start a program. For

the INT2LM and the COSMO-Model, these tasks are:

- Set number of processors and directory names

- Set batch commands for the machine to run on

- Cat together the INPUT files for Namelist input

- Start the program

- Do some clean up after the run

See Appendix A of the Tutorial for some more details


Runscript for INT2LM

In the reference data set (dir: int2lm_7_output_ascii) you will find the

runscript run_gme_2_cosmo_7. Copy it to your int2lm_141125_2.01

directory and take a look

First check the number of processors used: NPX, NPY: you can think of the

rectangular grid as a chess-board: NPX processors are used in the x- and

NPY processors are used in the y-direction: Use 4 x 5 processors

Among others you have to check / modify the directories for I/O:

LM_EXT: directory with external parameters for COSMO-Model domain:

IN_EXT: directory with external parameters for coarse grid domain:

you can set both to: /e/uhome/fegast3/TRAINING_2015/topo

IN_DIR: directory with input data from coarse grid model:

/e/uwork/trngxxx/reference_data_5.01/GME_2013051312_sub

LM_DIR: directory with output from INT2LM:

/e/uwork/trngxxx/reference_data_5.01/COSMO_7_input


Runscript for INT2LM (II)

You can also set the BASE variable to shorten things

The batch-commands should be ok

Note: #PBS –l place=pack working on 1 node

In this case you do not have to change anything for the Namelist Input

Run the program: Should also be ok, especially:

LIBDWD_BITMAP_TYPE=ASCII: we are working with a GME bitmap

aprun –n $NP –N 20 –j 1 –d 1 –m 1500m tstint2lm

Then just type: run_gme_2_cosmo7

After the run has finished successfully, you can compare the ASCII output file YUCHKDAT with the one given in the reference data set.


Runscript for COSMO

In the reference data set (dir: COSMO_7_output_ascii) you will find the

runscript run_cosmo_7. Copy it to your cosmo_141128_5.01 directory

and take a look

Number of processors used: NPX=5, NPY=8

Check and modify the directory names (with using BASE-variable)

BASE=/e/uwork/trngxxx/reference_data_5.01

INIDIR: directory with initial fields for COSMO-Model:

BD_DIR: directory with boundary fields for COSMO-Model

you have to set both to $BASE/COSMO_7_input

OUTPUT: directory with output data for COSMO-Model:

$BASE/COSMO_7_output

RESTAR: not used right now


Runscript for COSMO

PBS commands: Note the line

#PBS –l place=scatter: if you use more than 1 node

No changes necessary for the Namelist Input

Also the run-command is ok

Just type: run_cosmo_7


Additional Informations

There are also other batch systems (NQS, LOADLEVELER, SLURM, etc): If

you use those, you have to take a look to the manual to set the proper

variables

aprun is a special Cray command. Other commands to start a parallel

program are

mpirun: most common on Linux machines with GNU compiler

poe: on IBM computers

Of course, other commands have different options!


Running the 2.8 km Application

Now do the same for int2lm_2_output_ascii/run_cosmo7_2_cosmo2

LM_EXT=/e/uhome/fegast3/TRAINING_2015/topo

IN_EXT=$BASE/COSMO_7_output

IN_DIR=$BASE/COSMO_7_output

LM_DIR=$BASE/COSMO_2_input

And also for COSMO_2_output_ascii/run_cosmo_2

INIDIR= ???

BD_DIR= ???

OUTPUT= ???



Thank you

very much

for your

attention

Welcome to the Practical NWP Exercises · Welcome to the Practical NWP Exercises Ulrich Schättler, Ulrich Blahak Deutscher Wetterdienst ... To do the practical exercises, we have

Documents