Slide 1 TIGGE-LAM Workshop Bologna 19-21 Jan 2009 1 TIGGE-LAM: Archiving at ECMWF Manuel Fuentes Data and Services Section ECMWF.

TIGGE-LAM Workshop Bologna 19-21 Jan 2009 1

Slide 1

TIGGE-LAM: Archiving at ECMWF

Manuel Fuentes

Data and Services Section

ECMWF

TIGGE-LAM Workshop Bologna 19-21 Jan 2009 2

Slide 2

The TIGGE-Global database

Three archive centres: CMA, NCAR and ECMWF

Ten data providers:

- ECMWF, JMA (Japan), UK Met Office (UK), CMA (China), NCEP (USA), MSC (Canada), Météo-France (France), BOM (Australia), KMA (Korea), CPTEC (Brazil)

Exchanges using UNIDATA LDM, HTTP and FTP

Operational since 1st of October 2006

174 TB, growing by ~ 1.5 TB/week

- 1.5 million fields/day

TIGGE-LAM Workshop Bologna 19-21 Jan 2009 3

Slide 3Archive Centre

Data Providers

NCARNCEP

CMC

UKMO

ECMWF

MeteoFrance

JMAKMA

CMA

BoM

CPTEC

TIGGE Archive Centres and Data Providers

TIGGE-LAM Workshop Bologna 19-21 Jan 2009 4

Slide 4

Strong governancePrecise definition of:

- Which products: list of parameters, levels, steps, units,…

- Which format: GRIB2

- Which transport protocol: UNIDATA’s LDM

- Which naming convention: WMO file name convention

Only exception: the grid and resolution

- Choice of the data provider. Data provider to provide interpolation to regular lat/lon

- Best possible model output

Many tools and examples:

- Sample dataset available

- Various GRIB2 tools, “tigge_check” validator, …

- Scripts that implement exchange protocol

Web site with documentation, sample data set, tools, news….

TIGGE-LAM Workshop Bologna 19-21 Jan 2009 5

Slide 5

Using SMS to handle TIGGE flow

TIGGE-LAM Workshop Bologna 19-21 Jan 2009 6

Slide 6

Quality assurance: homogeneity

Homogeneity is paramount for TIGGE to succeed- The more consistent the archive the easier it will be to develop

applications

There are three aspects to homogeneity:- Common terminology (parameter names, file names,…)

- Common data format (format, units, …)

- Definition of an agreed list of products (Parameters, Steps, levels, …)

What is not homogeneous:- Resolution

- Base time (although most provider have a run at 12 UTC)

- Forecast length

- Number of ensemble

TIGGE-LAM Workshop Bologna 19-21 Jan 2009 7

Slide 7

QA: Checking for homogeneityE.g. Cloud-cover: instantaneous or six hourly?

TIGGE-LAM Workshop Bologna 19-21 Jan 2009 8

Slide 8

QA: Completeness

The objective is to have 100% complete datasets at the Archive Centres

Completeness may not be achieved for two reasons:

- The transfer of the data to the Archive Centre fails

- Operational activities at a data provider are interrupted and back filling past runs is impractical

Incomplete datasets are often very difficult to use

Most of the current tools (e.g. epsgrams) used for ensemble forecasts assume a fixed number of members from day to day

- These tools will have to be adapted

TIGGE-LAM Workshop Bologna 19-21 Jan 2009 9

Slide 9

QA: Checking completeness

TIGGE-LAM Workshop Bologna 19-21 Jan 2009 10

Slide 10

Access to TIGGE data

Data is available to Research and Education after 48 hours

- Self registration by agreeing to the terms and conditions

Portals at CMA, NCAR and ECMWF

ECMWF portal offers:- Access to offline data

- Aggregation along any axis (date, level, parameter, origin, ensemble, …)

- Provision of multi-model data on a single grid (regridding to any lat/lon grid)

- Sub-area selection

- Reduces volumes to be downloaded by many order of magnitude

- Batch access via the portal

TIGGE-LAM Workshop Bologna 19-21 Jan 2009 11

Slide 11

TIGGE portal at ECMWF

TIGGE-LAM Workshop Bologna 19-21 Jan 2009 12

Slide 12

TIGGE portal at ECMWF (using Ajax)

TIGGE-LAM Workshop Bologna 19-21 Jan 2009 13

Slide 13

TIGGE Portal: grid selection

TIGGE-LAM Workshop Bologna 19-21 Jan 2009 14

Slide 14

TIGGE Portal: area selection

TIGGE-LAM Workshop Bologna 19-21 Jan 2009 15

Slide 15

TIGGE Portal: use SMS for scheduling users’ requests

Monitoring

Per user limits

Global limits

TIGGE-LAM Workshop Bologna 19-21 Jan 2009 16

Slide 16

TIGGE Portal: direct access to MARS

Most of the data is on tape: access to long timeseries

Even finer control of resource usage

TIGGE-LAM Workshop Bologna 19-21 Jan 2009 17

Slide 17

TIGGE Portal: Batch access

TIGGE-LAM Workshop Bologna 19-21 Jan 2009 18

Slide 18

TIGGE Portal: Active users

Active users

0

5

10

15

20

25

30

35

40

45

50

Dec-06

Jan-07

Feb-07

Mar-07

Apr-07

May-07

Jun-07

Jul-07

Aug-07

Sep-07

Oct-07

Nov-07

Dec-07

Jan-08

Feb-08

Mar-08

Apr-08

May-08

Jun-08

Jul-08

Aug-08

Sep-08

TIGGE-LAM Workshop Bologna 19-21 Jan 2009 19

Slide 19

TIGGE Portal: Users by country

Cuba, 1

Finland, 1

France, 7

Germany, 16

Croatia, 1

USA, 41

Italy, 9

Japan, 13

Kenya, 1

Korea, 9

Lithuania, 1

Netherlands, 5

UK, 22

Algeria, 1

Australia, 5Brazil, 7

Bulgaria, 1

Canada, 4

China, 37

Colombia, 1

Turkey, 1

Poland, 1

New Zealand, 2

Thailand, 5

Russia, 1Slovenija, 1

South Africa, 1

Spain, 7

Switzerland, 3

Sweden, 1

Taiwan, 1

Ireland, 1

Ghana, 1Iceland, 1

India, 11Greece, 2

TIGGE-LAM Workshop Bologna 19-21 Jan 2009 20

Slide 20

TIGGE Portal: Data volumes

Data volumes (GBytes)

0

1000

2000

3000

4000

5000

6000

Dec-06

Jan-07

Feb-07

Mar-07

Apr-07

May-07

Jun-07

Jul-07

Aug-07

Sep-07

Oct-07

Nov-07

Dec-07

Jan-08

Feb-08

Mar-08

Apr-08

May-08

Jun-08

Jul-08

Aug-08

Sep-08

Retrieved

Delivered

TIGGE-LAM Workshop Bologna 19-21 Jan 2009 21

Slide 21

TIGGE: Lessons learnt

Strong commitment required:

- During development: encoding, units

- Day to day operations: failed jobs need to be re-run

Robust suites and scripts

- Problems with transmission or data corruption

- Servers may not available (retry)

- Keep data for at least 3 days

Technical contacts point

TIGGE-LAM Workshop Bologna 19-21 Jan 2009 22

Slide 22

TIGGE-LAM Archiving at ECMWF

Re-use TIGGE-Global technology

TIGGE-LAM Archiving meeting in September 2008

- Un-rotated lat/lon, 0.10, virtually on (0,0)

- HP parameters, addition of LSP, ORO and LSM

- Specific name for each LAM EPS configuration

- Data format: GRIB2

TIGGE-LAM Workshop Bologna 19-21 Jan 2009 23

Slide 23

TIGGE-LAM: Suite names for Europe

LAM EPS System Suite Name at ECMWF Institute -Consortium

MOGREPS MOGREPS-MO- EUA UK Met Office - UK

SREPS AEMet-SREPS-MM-EUAT AEMET- Spain HIRLAM

SRNWP PEPS SRNWP-PEPS

COSMO-LEPS COSMOLEPS-ARPASIMC-EU ARPA-SIM - Italy COSMO

NORLAMEPS NORLAMEPS Met.NO

ALADIN LAEF ALADIN-LAEF ZAMG / Austria

COSMO DE EPS COSMO-DE EPS DWD - Germany SRNWP

COSMO-SREPS COSMO-SREPS-BO-EU ARPA-SIM - Italy COSMO

GLAMEPS GLAMEPS DNMI/Univ Oslo –Norway HIRLAM ALADIN

PEARCE PEARCE Meteo-France / France

DMI - HIRLAM DMI- HIRLAM DMI

OMSZ ALADIN EPS OMSZ- ALADIN-EPS Hungary ,Czech Republic, Croatia

TIGGE-LAM Workshop Bologna 19-21 Jan 2009 24

Slide 24

TIGGE-LAM: Data format

Encoding in GRIB2

- Support for all HP parameters

Possibly add TIGGE-LAM GRIB2 local extension

- Differentiate between different systems

- Preserve Data Provider’s own local extensions

For Data Providers that cannot provide GRIB2, ECMWF can help convert from GRIB1 using grib_api

Examples from COSMO/LEPS and SREPS

Send sample data to: [email protected]

TIGGE-LAM Workshop Bologna 19-21 Jan 2009 25

Slide 25

TIGGE-LAM: Data Transmission & Archiving

No need for LDM

- FTP, push of pull or other

Filenames can be agreed between Data Provider and ECMWF

- Date & time required to avoid overlapping

ECMWF will archive data

- Consistency checks performed after archiving

TIGGE-LAM Workshop Bologna 19-21 Jan 2009 26

Slide 26

Conclusion

TIGGE-Global has been running for more than 2 years

- Robust infrastructure in place

Basis to ensure TIGGE-LAM success

Thank you!