Regional Production, Quarterly report on the daily ... · Copernicus Atmosphere Monitoring Service CAMS50_2015SC2 – SILAM Production Report – SON2017 Page 5 of 22 Executive summary

ECMWF COPERNICUS REPORT

Copernicus Atmosphere Monitoring Service

Regional Production, Quarterly report on the daily analyses and forecasts activities, and verification of the SILAM performances

September – October – November 2017

Issued by: METEO-FRANCE / G. Collin

Date: 13/02/2018

Ref: CAMS50_2015SC2_D50.3.1.2.SILAM_201802_Daily_Analyses_Report_v1 CAMS50_2015SC2_D50.3.2.2.SILAM_201802_Daily_Forecasts_Report_v1 CAMS50_2015SC2_D50.5.1.1.SILAM_201802_NRT_Verification_Report_v1

This document has been produced in the context of the Copernicus Atmosphere Monitoring Service (CAMS). The activities leading to these results have been contracted by the European Centre for Medium-Range Weather Forecasts, operator of CAMS on behalf of the European Union (Delegation Agreement signed on 11/11/2014). All information in this document is provided "as is" and no guarantee or warranty is given that the information is fit for any particular purpose. The user thereof uses the information at its sole risk and liability. For the avoidance of all doubts, the European Commission and the European Centre for Medium-Range Weather Forecasts has no liability in respect of this document, which is merely representing the authors view.


CAMS50_2015SC2 – SILAM Production Report – SON2017 Page 3 of 22

Contributors

FMI M. Sofiev R. Kouznetsov

METEO-FRANCE M. Pithon M. Plu V. Petiot M. Joly J. Arteta G. Collin N. Assar



Table of Contents

1. The SILAM model 6

1.1 Product portfolio 61.2 Availability statistics 61.2.1 Indicators 61.2.2 Problems encountered 71.3 Use of observations for data assimilation 71.3.1 Use of observations – September 2017 81.3.2 Use of observations – October 2017 91.3.3 Use of observations – November 2017 10

2. Verification report 11

2.1 Verification of NRT forecasts 112.1.1 SILAM forecasts: ozone skill scores against data from representative sites 122.1.2 SILAM forecasts: NO2 skill scores against data from representative sites 132.1.3 SILAM forecasts: PM10 skill scores against data from representative sites 142.1.4 SILAM forecasts: PM2.5 skill scores against data from representative sites 152.2 Verification of NRT analyses 162.2.1 SILAM analyses: ozone skill scores against data from representative sites 172.2.2 SILAM analyses: NO2 skill scores against data from representative sites 182.2.3 SILAM analyses: PM10 skill scores against data from representative sites 192.2.4 SILAM analyses: PM2.5 skill scores against data from representative sites 202.3 Analysis of the SILAM performances over the quarter 21



Executive summary The Copernicus Atmosphere Monitoring Service (CAMS, atmosphere.copernicus.eu/) is establishing the core global and regional atmospheric environmental service delivered as a component of Europe's Copernicus programme. The regional forecasting service provides daily 4-day forecasts of the main air quality species and analyses of the day before, from 7 state-of-the-art atmospheric chemistry models and from the median ensemble calculated from the 7 model forecasts. The regional service also provides posteriori reanalyses using the latest validated observation dataset available for assimilation. This report covers the D50.3.1.2, D50.3.2.2 and D50.5.1.1 deliverables related to the SILAM Near Real Time Production (NRT), for the quarterly period ending November 30th, 2017. Verification is done against in-situ surface observations; these are described in the D50.1.1.2 report covering the same period. The verification of analyses is done against non-assimilated observations. During this quarter, production reliability was excellent, with a 100% availability score for the forecasts and 99% for analyses for the whole quarter. The single problem experienced on analysis production occurred in November was due to a scripting error on upgrade of the operational suite. The SILAM scores at surface reached their target this quarter for every pollutant (forecasts and analyses). In fact, SILAM outperforms the ENSEMBLE on ozone in morning hours and in NO2 during the day time. The SILAM ozone bias oscillates around zero and is by far lower than the ENSEMBLE. Night-time NO2 requires more attention. The nature of substantial increase of RMSE for PM25 (less pronounced in PM10) on the first forecasting day is unclear and needs further investigation. Regarding analyses, the SILAM assimilation of ozone and NO2 surface observations improves greatly the fields. Improvement in the analysis fields, particular for PM10 are expected in the future quarters, since the analysis suite has been upgraded to new mixing and data assimilation schemes only in November 2017, and assimilation of PM10 was implemented. At the beginning of this quarter, an update of the list of stations used for the verification of forecasts and of analyses was done, and a one-hour time shift was applied to the hour of validity of observations. Thus, the improvement of the SILAM performance since last year, it is hardly possible to disentangle it from the effect of the new observation dataset that is used for assimilation and for evaluation. Consistently with this change of dataset, we note in particular an improvement of RMSE for all pollutants, an increase of PM bias (from negative to less negative or to positive), and a degradation of NO2 correlation. But, we note that the scores of SILAM ozone forecasts have been closer to the ENSEMBLE since last year, suggesting improvements of SILAM, that may be attributed to a model upgrade (June 2017) leading to improvements in the ABL mixing scheme and emission vertical



1. The SILAM model

1.1 Product portfolio

Item Forecast Analysis Description Forecast at surface, 50m, 250m,

500m, 1000m, 2000m, 3000m, 5000m above ground

Analysis at the surface, 50m, 250m, 500m, 1000m, 2000m, 3000m, 5000m above ground

Available for users at 3:00 UTC 09:30 UTC for the day before Species O3, NO2, CO, SO2, PM2.5, PM10,

NO, NH3, NMVOC, PANs, Birch pollen at surface during season

O3, NO2, CO, SO2, PM2.5, PM10, NO*, NH3*, NMVOC*, PANs*

Time span 0-96h, hourly 0-24h for the day before, hourly * Non-assimilated species

1.2 Availability statistics The statistics below describe the ratio of days for which the SILAM model outputs were available on time to be included in the ENSEMBLE fields (analyses and forecasts) that are computed at Meteo-France. They are based on the following schedule for the provision at Meteo-France of: • Forecasts data before: 05:30 UTC for D0-D1 (up to 48h), 07:30 UTC for D2-D3 (from 49h to 96h); • Analyses data: before 11:00 UTC. These schedules were set to meet the IT requirements for ENSEMBLE products (no later than 8 UTC for 0-48h, 10 UTC for 49-96h and 12 UTC for analyses).

1.2.1 Indicators Availability_model_Forecast Quarterly basis

D0: 100% D1: 100% D2: 100% D3:100%

Availability_model_Analysis Quarterly basis

D: 99%



1.2.2 Problems encountered The following issue was encountered by the SILAM production system:

Date Problem description Impact on production 21/11/2017 SILAM Analyses was not uploaded

due to a scripting error. SILAM analysis results (D0-D3) non available for ENS calculation.

1.3 Use of observations for data assimilation Please see the next three pages.



1.3.1 Use of observations – September 2017

Day O3 NO2 NO SO2 CO PM10 PM2.5 1 9611 9356 5224 1724

2 9464 9222 5152 1707

3 9315 9143 5088 1681

4 9559 9373 5173 1704

5 9601 9471 5233 1614

6 9493 9241 5132 1652

7 9531 9289 5220 1741

8 9427 9330 5160 1654

9 8360 8420 4657 1367

10 8474 8449 4708 1374

11 8673 8781 4856 1475

12 7976 8918 4914 1514

13 8772 8831 4898 1481

14 8817 9060 4960 1439

15 8638 9022 5036 1582

16 7723 8201 4391 1408

17 8515 9108 4884 1528

18 9165 9330 5268 1604

19 8786 8784 4922 1547

20 9388 0 5277 1687

21 8961 9034 5213 1620

22 8284 8663 4954 1549

23 0 0 5436 1611

24 8808 9031 5188 1633

25 9098 9152 5268 1655

26 9220 9370 5424 1660

27 9069 9272 5538 1700

28 9229 9286 5518 1719

29 9332 9361 5546 1749

30 8927 8645 5304 1609



1.3.2 Use of observations – October 2017


2 9464 9222 5152 1707

3 9315 9143 5088 1681

4 9559 9373 5173 1704

5 9601 9471 5233 1614

6 9493 9241 5132 1652

7 9531 9289 5220 1741

8 9427 9330 5160 1654

9 8360 8420 4657 1367

10 8474 8449 4708 1374

11 8673 8781 4856 1475

12 7976 8918 4914 1514

13 8772 8831 4898 1481

14 8817 9060 4960 1439

15 8638 9022 5036 1582

16 7723 8201 4391 1408

17 8515 9108 4884 1528

18 9165 9330 5268 1604

19 8786 8784 4922 1547

20 9388 0 5277 1687

21 8961 9034 5213 1620

22 8284 8663 4954 1549

23 0 0 5436 1611

24 8808 9031 5188 1633

25 9098 9152 5268 1655

26 9220 9370 5424 1660

27 9069 9272 5538 1700

28 9229 9286 5518 1719

29 9332 9361 5546 1749

30 8927 8645 5304 1609

31



1.3.3 Use of observations – November 2017


2 9221 9177 5470 1711

3 9426 9312 5460 1753

4 8997 9072 5305 1604

5 8959 8784 5208 1606

6 9235 9074 5300 1712

7 9128 9082 5363 1693

8 9074 9010 5274 1871

9 9158 9164 5476 1905

10 9272 9266 5353 1751

11 8895 9225 5238 1619

12 9223 8966 5277 1632

13 9236 9136 5350 1666

14 9408 9301 5434 1862

15 9308 9426 5300 1817

16 9485 9513 5424 1859

17 9506 9487 5444 1747

18 9596 9590 5277 1899

19 9448 9453 5237 1796

20 9498 9581 5222 1840

21 9510 9652 5346 1832

22 7466 7163 4206 1422

23 9464 9605 5367 3203 5259 1741

24 9920 9912 5657 3392 5538 1897

25 9316 9594 5242 3092 5266 1864

26 9222 9464 5106 3065 5154 1728

27 9727 9945 5394 3512 5475 1750

28 9594 9668 5313 3440 5203 1717

29 9373 9758 5083 3251 5526 1929

30 9502 10031 5075 3198 5652 1970



2. Verification report This verification report covers the quarterly period ending November 30th, 2017. The SILAM skill scores are successively presented for four pollutants: ozone, NO2, PM10 and PM2.5. The skill is shown for the entire forecast horizon from 0 to 96h (hourly values), allowing to evaluate the entire diurnal cycle and the evolution of performance from day 0 to day 3. The forecasts and the analyses cover a large European domain (25°W-45°E, 30°N-70°N). The statistical scores that are reported are the root-mean-square error, the modified mean bias and the correlation. The surface observations that are acquired by Meteo-France and used for verification are described in D50.1.1.2 covering the same period. At the beginning of this quarter, an update of the list of stations used for the verification of forecasts and of analyses was done. This update intends to improve the representativeness of the observations used by Regional CAMS. Besides, the hour of validity of observations was checked and corrected by a one-hour time shift. The impact of these changes on scores was documented in a previous report (D50.5.5.1, Update on the best use of EIONET observation database for assimilation and verification of the NRT Regional Production). As a consequence, the yearly evolution of scores is interpreted in regards of this update.

2.1 Verification of NRT forecasts The following figures present, for each pollutant (ozone, NO2, PM10 and PM2.5): • In the upper-left panel, the root-mean square error of daily maximum (for ozone and NO2) or of

daily mean (PM10 and PM2.5) for the first-day forecasts with regard to surface observations, for every quarter since DJF2014/2015, a target reference value is indicated as an orange line;

• In the upper-right panel, the root-mean square error of pollutant concentration forecasts with regard to surface observations as a function of forecast term;

• In the lower-left panel, the modified mean bias of pollutant concentration forecasts with regard to surface observations as a function of forecast term;

• In the lower-right panel, the correlation of pollutant concentration forecasts with regard to surface observations as a function of forecast term.

The graphics show the performance of SILAM (black curves) and of the ENSEMBLE (blue curves).



2.1.1 SILAM forecasts: ozone skill scores against data from representative sites

The RMSE of SILAM ozone daily maximum reaches just the target value and it is above the ENSEMBLE value. Interestingly, unlike SON 2016, the diurnal cycle of Silam bias is quite different from one of the Ensemble. The SILAM ozone bias oscillates around zero and is by far lower than the ENSEMBLE. Compared to past (2016) autumn, the SILAM ozone scores have improved, partly due to the use of a new observation dataset for evaluation. We note also that the scores of SILAM ozone forecasts have been closer to the ENSEMBLE since last year, suggesting improvements of SILAM, that may be attributed to a model upgrade (June 2017) leading to improvements in the ABL mixing scheme and emission vertical profiles.



2.1.2 SILAM forecasts: NO2 skill scores against data from representative sites

The RMSE of SILAM NO2 daily maximum reaches the target and it is above the ENSEMBLE score. The SILAM RMSE oscillates with a 12h-frequency like the ENSEMBLE, and it reaches its highest values around 9 UTC and 21 UTC. Compared to past (2016) autumn, the SILAM NO2 RMSE and bias have improved and the correlation has been degraded, similarly to the ENSEMBLE. These common changes may be related to the use of a new observation dataset for evaluation.



2.1.3 SILAM forecasts: PM10 skill scores against data from representative sites

The RMSE of SILAM PM10 daily mean reaches the target and is above the ENSEMBLE score. The SILAM bias is largely negative and the correlation is very close to the ENSEMBLE. On the first day of the forecast, SILAM shows higher RMSE values than on the following days, which is an unexpected behaviour that deserves further investigation.



2.1.4 SILAM forecasts: PM2.5 skill scores against data from representative sites

The RMSE of SILAM PM2.5 daily mean is close to the ENSEMBLE performance, however the difference is slightly smaller than in the previous year. The PM25 RMSE of Silam shows a clear degradation on the first day of the forecast. This degradation is completely missing from the Bias and correlation scores. The reason is unclear and needs further investigation. RMSE and bias (except for the first day) have been improved since SON2016. The negative bias has been reduced together with its diurnal variability. The performance of correlation of Silam PM25 reaches the one of Ensemble, whereas in SON2016 a gap of ~0.1 was persistent.



2.2 Verification of NRT analyses The following figures present, for each pollutant (ozone, NO2, PM10 and PM2.5): • In the upper-left panel, the root-mean square error of daily maximum (for ozone and NO2) or of

daily mean (PM10) for the analyses (solid line) and for the first-day forecasts (dashed line) with regard to surface observations, for every quarter since DJF2014/2015, a target reference value is indicated as an orange line;

• In the upper-right panel, the root-mean square error of pollutant concentration of the analyses (solid line) and of the first-day forecasts (dashed line), with regard to surface observations as a function of forecast term;

• In the lower-left panel, the modified mean bias of pollutant concentration forecasts of the analyses (solid line) and of the first-day forecasts (dashed line), with regard to surface observations as a function of forecast term;

• In the lower-right panel, the correlation of pollutant concentration of the analyses (solid line) and of the first-day forecasts (dashed line), with regard to surface observations as a function of forecast term.

The graphics show the performances of SILAM (black curves) and of the ENSEMBLE (blue curves). The superposition of the analysis scores (solid lines) and of the forecast scores (dashed lines) computed over the same observation dataset is helpful to assess the added value of data assimilation.



2.2.1 SILAM analyses: ozone skill scores against data from representative sites

The RMSE of daily-max O3 analysis has been reduced since SON2016, though not as much as one for the forecast. The reason is that different model set-ups were used for AN and FC during SON2017: the analysis has been switched to the improved ABL scheme and unified chemistry only in Nov 2017, whereas the forecast has been upgraded in June 2017. Thus the difference between analysis and forecast is not only due to the data assimilation, but also due to the general model performance change. The performance of the analysis in SON2017 got noticeably better since SON2016 due to the change of the model version from v5_4 to v5_5, while keeping the same setup as previous year.



2.2.2 SILAM analyses: NO2 skill scores against data from representative sites

The RMSE of SILAM NO2 analysis have been improved since SON2016 mostly due to the newer model version. Unlike in SON2016, daytime Silam NO2 analysis is on a par with the analysis. The assimilated NO2 fields got a noticeable positive bias in night time with no obvious reason. The correlation of NO2 analysis has been reduced since SON2016 for both Ensemble and Silam, probably due to difference in meteorology and/or set of the stations used for the evaluation.



2.2.3 SILAM analyses: PM10 skill scores against data from representative sites

PM10 was not assimilated in Silam before Nov 2017. i.e the improvement of scores due to assimilation comes primarily from cross-correlation between PM10 and assimilated PM2.5.



2.2.4 SILAM analyses: PM2.5 skill scores against data from representative sites

The difference between assimilated and non-assimilated PM25 substantially increased since SON2016. The gap in RMSE and correlation of assimilated PM2.5 between Silam and Ensemble has been reduced. The bias for Silam got more uniform over a day.



2.3 Analysis of the SILAM performances over the quarter Overall, the SILAM forecast scores for the four pollutants that are evaluated (ozone, NO2, PM10 and PM2.5) reach the target this quarter. The differences between SILAM and ENSEMBLE performances are generally lower than during past (2016) autumn, which suggests that some improvements of the SILAM performances occur, on top of the yearly change of scores that may be related to the use of a new observation dataset for evaluation. The main improvement in forecasts comes from the improvements in the ABL mixing scheme and emission vertical profiles introduced in June 2017. Silam outperforms Ensemble on Ozone in morning hours and in NO2 during the day time. Night-time NO2 requires more attention. The nature of substantial increase of RMSE for PM25 (less pronounced in PM10) on the first forecasting day is unclear and needs further investigation. The improvement in the analysis fields is less pronounced, since the analysis suite has been upgraded to new mixing and data assimilation schemes only in November 2017, i.e. in the very end of the quarter. The improvement, however is seen in practically all species due to the change of the model version from Silam v5_4 to Silam v5_5 since SON2016, while the assimilation set-up stayed unchanged.


atmosphere.copernicus.eu copernicus.eu ecmwf.int

ECMWF - Shinfield Park, Reading RG2 9AX, UK Contact: [email protected]

Regional Production, Quarterly report on the daily ... · Copernicus Atmosphere Monitoring Service CAMS50_2015SC2 – SILAM Production Report – SON2017 Page 5 of 22 Executive summary

Documents