ECMWF new Data Centre · Agenda • ECMWF current computing facilities • Our new Data Centre • Bologna Information System design • Procurement, delivery and installation timeline

© ECMWF January 17, 2019

ECMWF new Data CentreIndustry Day, Bologna, 16 January 2019

Matteo Dell’AcquaRelocation Programme – Technical [email protected]

Agenda• ECMWF current computing facilities

• Our new Data Centre

• Bologna Information System design

• Procurement, delivery and installation timeline

2EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS

ECMWF current Computing Facilities


ECMWF current Computing Facilities


ECMWF’s production workflow


ProductGeneration

Dissemination

Products

MemberStates

&Customers

MARS

Perpetual Archive

IFSModel

Fields

Acquisition

GlobalObservations

Obs

Data Workflow

• Post-process data for client products

• Post-process for statistics and data analysis

• Perpetual archival of generated data sets

HPC: a need for weather forecasting

• Scientists quickly realized that computing would be the key for weather prediction– Knowing the current weather

– And calculating its evolution

• From dream to reality : first computer– And the first numerical weather prediction model

• Since, computing performance kept increasing– This is one of the main reason for the progress in forecast


ENIAC 1950

CRAY System

Evolution of Archive and HPC sustained performance


Log

scal

e

The ECMWF data Archive

• Data archival and retrieval system for all ECMWF data– 310 PB primary data

• This data is stored on cartridges and disks

• If this data was printed, there would be enough books to reach the moon 8 times

• Large meteorological archive– Direct access from Member States

– Available to research community worldwide

• User access via ECMWF developed applications– MARS – Meteorological Archival and Retrieval System

• Access via “meteorological terms”, e.g. fields

– ECFS – ECMWF Common File System

• File based access for non-meteorological data


The ECMWF data Archive


Every day the archive grows more than 250TB. This is approximately 1.75 Petabytes (PB) every week

If this data was printed, the books produced could be put in a 11,372 km long row. This is about the distance travelling from Bologna to Nova Scotia, Canada and back again! 11

,372

Data acquisition and dissemination• A total of 1163 Destinations across 79 countries:

– Dissemination: 547 Destinations across 78 countries

– Acquisition: 557 Destinations across 34 countries

– Data Portal: 59 Destinations (mostly Copernicus CAMS and C3S)


Data dissemination volumes• Monthly Volume of Data Transmitted: 857TB (~28.6TB/day)

• Exponential Increase trend for Internet


0

50

100

150

200

250

300

350

400

450

500

Internet LAN RMDCN

2007

2008

2009

2010

2011

2012

2013

2014

2015

2016

2017

2018

Estimated Growth in model2015

16km, 137 levels

Time critical

• 21 TB/day written

• 22 Million fields

• 85 Million products

• 11 TB/day send to customers

Non-time critical

• 100 TB/day archived

• 400 research experiments

• 400,000 jobs / day


2020

Increase: 2 horizontal, 1 upper air

Time critical

• 128 TB/day written

• 90 Million fields

• 450 Million products

• 60 TB/day send to customers

Non-time critical

• 1 PB/day archived

• 1000 research experiments

10-Year Challenge


Data acquisition

Forecast run Product generation

DisseminationRMDCN

Internet

Web services Internet

Archive Data Handling System

10x more observational data per day

2000x more model data per time step

25x more forecast product data per day in critical path

30x more data sent to customers per day in critical path

100x more data archived per day

Agenda• ECMWF current computing facilities

• Our new Data Centre

• Bologna Information System design

• Procurement, delivery and installation timeline


ECMWF new Data Centre

• The Data Centre will be located , in the Tecnopolodi Bologna

– in the former tobacco factory, built in 1949 and closed in 1998


How it looks today

Data Centre Design Master Plan

Data Centre layout


How it will look

Decoupling front / backend: Mcloud Services


Frontend (definition) Backend (execution)

Interpolation / graphical Services / virtual desktopenvironment

dispatch

FieldsT8000(N x

670MB)

FieldsN640(N x

10MB)workstation

FDBareagridrotationformat

Bologna target functional designHPC DHSMCLOUD EUROPEAN

WEATHER CLOUD

LONG TERM ARCHIVE

ACQUISITION

PREPROCESSING

EXTERNAL DISSEMINATION

CORPORATE SERVICES

ACCESS SERVICES

IDENTITY & ACCESS MANAGEMENT

GRAPHICAL DISSEMINATION

ONLINE STORAGE

QUALITY CONTROL & VERIFICATION

IAAS

SAAS (FUTURE)

PAAS (FUTURE)WEB SERVICES

TELECOM

INTERNET

RMDCN

LEASED LINES

SATELLITES

EXTERNAL CLOUD

MONITORING

TITLEECMWF Bologna Architecture

DATE

23 July 2018Tier 1.1 = Critical PathTier 1.2 = Immediate Call OutTier 2 = Restart Next Sociable HourTier 3 = Restart Next Working Day

NETWORKING

VIRTUAL DESKTOP

GENERAL PURPOSE BATCH

INTERACTIVE

HIGH PERF. BATCH

HIGH SPEED PARALLEL STORAGE

INTERNAL DISSEMINATION

POSTPROCESSING


FORECASTING SYSTEMORCHESTRATION

SERVICE

FDB

MARS

ECFS

Bologna target functional design – tenders scopeHPC DHSMCLOUD EUROPEAN

WEATHER CLOUD

LONG TERM ARCHIVE

ACQUISITION

PREPROCESSING


CORPORATE SERVICES

ACCESS SERVICES



ONLINE STORAGE


IAAS

SAAS (FUTURE)


TELECOM

INTERNET

RMDCN

LEASED LINES

SATELLITES

EXTERNAL CLOUD

MONITORING


DATE


NETWORKING

VIRTUAL DESKTOP


INTERACTIVE

HIGH PERF. BATCH



POSTPROCESSING



SERVICE

FDB

MARS

ECFS


WEATHER CLOUD

LONG TERM ARCHIVE

ACQUISITION

PREPROCESSING


CORPORATE SERVICES

ACCESS SERVICES



ONLINE STORAGE


IAAS

SAAS (FUTURE)


TELECOM

INTERNET

RMDCN

LEASED LINES

SATELLITES

EXTERNAL CLOUD

MONITORING


DATE


NETWORKING

VIRTUAL DESKTOP


INTERACTIVE

HIGH PERF. BATCH



POSTPROCESSING



SERVICE

FDB

MARS

ECFS

Serv

ers

Serv

ers


WEATHER CLOUD

LONG TERM ARCHIVE

ACQUISITION

PREPROCESSING


CORPORATE SERVICES

ACCESS SERVICES



ONLINE STORAGE


IAAS

SAAS (FUTURE)


TELECOM

INTERNET

RMDCN

LEASED LINES

SATELLITES

EXTERNAL CLOUD

MONITORING


DATE


NETWORKING

VIRTUAL DESKTOP


INTERACTIVE

HIGH PERF. BATCH



POSTPROCESSING



SERVICE

FDB

MARS

ECFS

Routers, firewall, …

Serv

ers

Serv

ers


WEATHER CLOUD

LONG TERM ARCHIVE

ACQUISITION

PREPROCESSING


CORPORATE SERVICES

ACCESS SERVICES



ONLINE STORAGE


IAAS

SAAS (FUTURE)


TELECOM

INTERNET

RMDCN

LEASED LINES

SATELLITES

EXTERNAL CLOUD

MONITORING


DATE


NETWORKING

VIRTUAL DESKTOP


INTERACTIVE

HIGH PERF. BATCH



POSTPROCESSING



SERVICE

FDB

MARS

ECFS


Serv

ers

Serv

ers

SAN, NAS, …


WEATHER CLOUD

LONG TERM ARCHIVE

ACQUISITION

PREPROCESSING


CORPORATE SERVICES

ACCESS SERVICES



ONLINE STORAGE


IAAS

SAAS (FUTURE)


TELECOM

INTERNET

RMDCN

LEASED LINES

SATELLITES

EXTERNAL CLOUD

MONITORING


DATE


NETWORKING

VIRTUAL DESKTOP


INTERACTIVE

HIGH PERF. BATCH



POSTPROCESSING



SERVICE

FDB

MARS

ECFS


Serv

ers

Serv

ers

SAN, NAS, …

Electrical switchboard, busbar, racks, fibbers, cable, ….

Data Centre fit-out timeline

Procurements Operational servicesDelivery and installation

2019 2020Q1 Q3 Q4 Q1 2020 Q2 Q3

Start Procurements

DHS operation in Bologna

03-04-2020

Bologna DC delivered6-10-2019

HPC operation in Bologna

Q4Q2

Q4 Q1 2020

100 Gbps link Reading<-> Bologna

Mid-09-2019DHS operation in

Bologna

Equipment delivery

Early-10-2019

Data Centre fit-out timeline

Procurements Busbar, racks, fibbers installation

Q2Q3

Network, servers and storage installation and configuration

Services deployment

2019 2020Q1 Q3 Q4 Q1 2020 Q2 Q3

Start Procurements

DHS operation in Bologna

03-04-2020Bologna DC delivered6-10-2019

HPC operation in Bologna

Q4Q2

Q&A29EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS

Grazie mille!

ECMWF new Data Centre · Agenda • ECMWF current computing facilities • Our new Data Centre • Bologna Information System design • Procurement, delivery and installation timeline

Documents