Highlights from Highlights from Operational Verification Operational Verification in COSMO in COSMO WG5 COSMO General Meeting, Rome 2011 Authors: ALL Presented by Adriano Raspanti
Jan 03, 2016
Highlights from Operational Highlights from Operational Verification in COSMOVerification in COSMO
WG5 COSMO General Meeting, Rome 2011
Authors: ALL
Presented by Adriano Raspanti
WG5 COSMO General Meeting, Rome 2011
• Quick look to some common plots
• Intercomparison between driving model and high resolution
• Conditional verification
• Fuzzy verification
• Long term trends (mainly precipitation)
TEMPERATURE AT 2 M - SON 2010 - MAM 2011
MEAN SEA LEVEL PRESSURE - SON 2010 – MAM 2011
Grafikinhalte
BC from GME CEU,CPL,CRU !!!
BC from IFS C7,CI7,CGR !!!
BC from GME ???
BC from IFS ???
WIND SPEED AT 10 M - SON 2010 - MAM 2011
TOTAL CLOUD COVER - SON 2010 - MAM 2011
WG5 COSMO General Meeting, Rome 2011
• Quick look to some common plots
• Intercomparison between driving model and high resolution
• Conditional verification
• Fuzzy verification
• Long term trends (mainly precipitation)
COSMO General Meeting – Roma 05-09 Sept 2011
COSMOME vs ECMWF Temperature
SON
MAM
DJF
JJA
COSMO General Meeting – Roma 05-09 Sept 2011
COSMOME vs ECMWF Wind Speed
SON
JJA
MAM
DJF
COSMO General Meeting – Roma 05-09 Sept 2011
COSMOI7 vs ECMWF Temperature
SON
JJA
MAM
DJF
COSMO General Meeting – Roma 05-09 Sept 2011
COSMOI7 vs ECMWF Wind Speed
SON
JJA
MAM
DJF
COSMO General Meeting – Roma 05-09 Sept 2011
TemperatureCOSMOME vs COSMOIT
SON
JJA
MAM
DJF
COSMO General Meeting – Roma 05-09 Sept 2011
Wind SpeedCOSMOME vs COSMOIT
SON
JJA
MAM
DJF
Temp 2m - 7km vs 3kmTemp 2m - 7km vs 3km
WG5 COSMO General Meeting, Rome 2011
Fall Winter
Spring Summer
Underestimation of Temp, mainly in winter. error ~2o, worse with 7km by ~0.5o
Clear diurnal cycle
Wind Speed - 7km vs 3kmWind Speed - 7km vs 3km
Fall Winter
Spring Summer
Overestimation of wind (DJF,SON)2-2.5deg bias
similar attitude of 2 models
WG5 COSMO General Meeting, Rome 2011
17 Verification results at MeteoSwiss in 2011
COSMO GM / WG5 Parallel Session, 05.09.2011
Precipitation (12h-sums +36 to +48h):Spring 2011 over Switzerland (SYNOP‘s)frequency bias: COSMO-7 & IFS
observed frequency
V. Stauch
18 Verification results at MeteoSwiss in 2011
COSMO GM / WG5 Parallel Session, 05.09.2011
Precipitation (12h-sums +12 to +24h):Spring 2011 over Switzerland (SYNOP‘s)COSMO-7 & COSMO-2
V. Stauch
for both models mean over 9 gridpoints foreach station
T2m COSMO-RU 2.2 and 7 km, Sochi, station Krasnaya Polyana
19
2.2 km –Less overestimating
7 km
T2m in COSMO-RU 7 and 2.2 km, Krasnaya Polyana
20
Method: 1) nearest point 3D optimized !
COSMO-RU 2.2 km is better than COSMO-RU 7 km for Krasnaya Polyana
T2m in COSMO-RU 7 and 2.2 km, Moscow
21
Method: 1) nearest point 3D optimized !
COSMO-RU 2.2 km RMSE is even slightly higher than that of COSMO-RU 7 km for Moscow
COSMO General Meeting – Roma 05-09 Sept 2011
PERFORMANCE DIAGRAM
Period March 2010 - April 2011
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 1 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 5 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 10 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 20 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
1 Point (maximum) > 1 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
1 Point (maximum) > 5 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
1 Point (maximum) > 10 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
1 Point (maximum) > 20 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 1 mm/24h&
Maximum > 25 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 1 mm/24h&
Maximum > 50 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 1 mm/24h&
Maximum > 75 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 1 mm/24h&
Maximum > 100 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 5 mm/24h&
Maximum > 25 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 5 mm/24h&
Maximum > 50 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 5 mm/24h&
Maximum > 75 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 5 mm/24h&
Maximum > 100 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 10 mm/24h&
Maximum > 25 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 10 mm/24h&
Maximum > 50 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 10 mm/24h&
Maximum > 75 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 10 mm/24h&
Maximum > 100 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 20 mm/24h&
Maximum > 25 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 20 mm/24h&
Maximum > 50 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 20 mm/24h&
Maximum > 75 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 20 mm/24h&
Maximum > 75 mm/24h
WG5 COSMO General Meeting, Rome 2011
• Quick look to some common plots
• Intercomparison between driving model and high resolution
• Conditional verification
• Fuzzy verification
• Long term trends (mainly precipitation)
COSMO General Meeting – Roma 05-09 Sept 2011
Conditional Verification Temp – TCC obs <=25%
SON
MAM
DJF
JJA
Better behaviour for all the seasonsCompare to no condition model
COSMO General Meeting – Roma 05-09 Sept 2011
Conditional Verification Temp – TCC obs >=75%&Wind Speed (obs) <=2 m/s
SON
MAM
DJF
JJA
Similar. Differences in bias
COSMO General Meeting – Roma 05-09 Sept 2011 COSMO General Meeting – Roma 05-09 Sept 2011
Conditional Verification Temp – TCC obs <=25%&Wind Speed (obs) <=2
m/s
SON
MAM
DJF DJF
JJA
Similar. Differences in bias
2mT 2mT vsvs 2mT in overcast conditions2mT in overcast conditions 2mT overcast/no wind2mT overcast/no wind
Fall Winter
Spring Summer
2mT 2mT vsvs 2mT in skyclear conditions2mT in skyclear conditions 2mT skyclear/no wind2mT skyclear/no wind
Fall Winter
Spring Summer
CAPE>50
CAPE<50
Very high POD values for unstable conditions, FAR not so different
WG5 COSMO General Meeting, Rome 2011
WS - Z0>0.25
WS - Z0<0.25
Wind Speed with respect to Roughness Length
Strong underestimation of wind in positions with small roughness length and increased error
WG5 COSMO General Meeting, Rome 2011
Td, 2mT – dry soil
Td, 2mT – wet soil
W_SO Water content of first soil layer(kg/m2) 1cm.
Td: Higher error in dry soil and larger underestimation2mT: Higher error in wet soil and larger understimation
WG5 COSMO General Meeting, Rome 2011
• Quick look to some common plots
• Intercomparison between driving model and high resolution
• Conditional verification
• Fuzzy verification
• Long term trends (mainly precipitation)
57 Verification results at MeteoSwiss in 2011
COSMO GM / WG5 Parallel Session, 05.09.2011
results for 20103h accumulated precipitation sumsover the domain of the swiss radar composit
models: COSMO-2 and COSMO-7for all 8 daily forecast runs, precipitation sums from +3 to +6h
observation precipitation estimates of the swiss radar composit
in case of a missing value, the full date will not be evaluated
Neighborhood verification for precipitation
58 Verification results at MeteoSwiss in 2011
COSMO GM / WG5 Parallel Session, 05.09.2011
FSS, COSMO-2 minus COSMO-7, 2010
Winter Spring
AutumnSummer
numbers = FSS-Score of COSMO-2
colours = differences COSMO-2 minus COSMO-7
COSMO-2 better
COSMO-7 better
• COSMO-2: similar skill in all seasons
• COSMO-2 better than COSMO-7 on almost all scales
• COSMO-2 better than COSMO-7 especially in Winter and Summer
Tanja Weusthoff
59 Verification results at MeteoSwiss in 2011
COSMO GM / WG5 Parallel Session, 05.09.2011
3. „Fuzzy in Time“
• Extension of the spatial window with a window in time volume (dx * dy * dt)
• Evaluation of the forecasts in this volume• Time-window ntm = [1h,3h,5h,7h,9h]
dt
dx
dy
T. Weusthoff
60 Verification results at MeteoSwiss in 2011
COSMO GM / WG5 Parallel Session, 05.09.2011
dxdy
dx
dymodel
observation
e.g. FSS
fraction blue pxiels model = 12/75 (dt=3), 5/25 (dt=1)
fraction blue pxiels obs = 12/75 (dt=3), 3/25 (dt=1)
t0
t0-1
t0+1
t0
t0-1
t0+1
T. Weusthoff
61 Verification results at MeteoSwiss in 2011
COSMO GM / WG5 Parallel Session, 05.09.2011
00 01 02 03 04 05 06 07 08 09 10 11 12 13 1420 21 22 23
00-02
01-03
02-04
03-05
04-06
05-07
…
23-01
22-00
21-23
20-22
hourly accumulated precipitation
Evaluation of 3h accumulations, time window ntm = 5 (t0 +-2h)
00 UTC 03 UTC 06 UTC 08 UTC
22 UTC 02 UTC
01 UTC 05 UTC
3 hourly accumulated precipitation
Evaluation of 3h accumulations, time window ntm = 1
09 UTC00 UTC 03 UTC 06 UTC
23 UTC 01 UTC 02 UTC 04 UTC
T. Weusthoff
62 Verification results at MeteoSwiss in 2011
COSMO GM / WG5 Parallel Session, 05.09.2011
FSS for different time-windowsCOSMO-2, July 2010
T. Weusthoff
63 Verification results at MeteoSwiss in 2011
COSMO GM / WG5 Parallel Session, 05.09.2011 T. Weusthoff
FSS for different time-windowsCOSMO-7, July 2010
64 Verification results at MeteoSwiss in 2011
COSMO GM / WG5 Parallel Session, 05.09.2011
FSS for different time-windows COSMO-2 minus COSMO-7, July 2010
T. Weusthoff
65 Verification results at MeteoSwiss in 2011
COSMO GM / WG5 Parallel Session, 05.09.2011
Summary „fuzzy in time“
• FSS increases on all scales with increasing time-window• greatest effect for small spatical scales• lowest effect for high threshods
• Both models show a similar increase• difference COSMO-2 minus COSMO-7 stays equal, resp.
becomes littler for high time-windows
• For Upscaling the influence of a time tolerance is relatively low und restricted on low thresholds ( effect of the avergaing)
Application of time-windows on the gridscale would make sense; simultaneous application with space tolerance brings no great change
T. Weusthoff
Ulrich Damrath: Fuzzy and standard verification for COSMO-EU and COSMO-DE, COSMO GM, Rome September 2011
Configuration of precipitation verification with FUZZY-methods
Up to May 2011: Observation data: Radar data prepared by assimilation scheme
Model data: GME-, CEU- and CDE-GRIBS interpolated to CDE-grid (nearest gridpoint)
Run: 00 UTC
Forecast times: GME, CEU: 06-18, 06-30, CDE: 06-18 hours
Verification area: part of CDE that is covered by radar data
Since May 2011: Observation data as before, modell data: CEU- and CDE-GRIBS interpolated to CDE-grid
(nearest gridpoint)
Run: 00, 03, 06, 09, 12, 15, 18, 21
Forecast times: 01-04, 03-06, 06-12, 12-15, 15-18, 18-21 hours
Verification aread : CDE, Northern part of Germany, Southern part of Germany, North-Western part of Germany, North-Eastern part of Germany , South-Western part of Germany, South-Eastern part of Germany
Ulrich Damrath: Fuzzy and standard verification for COSMO-EU and COSMO-DE, COSMO GM, Rome September 2011
Application of Fuzzy-methods
Calculation of all Fuzzy-scores with the IDL-Program by Beth Ebert.
Monthly evaluation of data for Fractions Skill Score and Upscaling ETS
Generation of results for 8 (forecast runs)
* 7 (forecast intervals)
* 3 (2 models and one difference)
* 7 (regions)
* 2 (scores)
---------------------------------------------2352 Plots per time interval
Necessity to have a fast access to the data
Ulrich Damrath: Fuzzy and standard verification for COSMO-EU and COSMO-DE, COSMO GM, Rome September 2011
Some examples:ETS upscaling July 2011, Run: 00 UTC, forecast time 01-04 hours
Ulrich Damrath: Fuzzy and standard verification for COSMO-EU and COSMO-DE, COSMO GM, Rome September 2011
Some examples :FSS July 2011, Run: 00 UTC, forecast time 01-04 hours
Ulrich Damrath: Fuzzy and standard verification for COSMO-EU and COSMO-DE, COSMO GM, Rome September 2011
Some examples :ETS upscaling July 2011, Run: 00 UTC, forecast time 12-15 hours
Ulrich Damrath: Fuzzy and standard verification for COSMO-EU and COSMO-DE, COSMO GM, Rome September 2011
Some examples :FSS July 2011, Run: 00 UTC, forecast time 12-15 hours
Ulrich Damrath: Fuzzy and standard verification for COSMO-EU and COSMO-DE, COSMO GM, Rome September 2011
Some examples :ETS upscaling July 2011, Run: 00 UTC, forecast time 18-21 hours
Ulrich Damrath: Fuzzy and standard verification for COSMO-EU and COSMO-DE, COSMO GM, Rome September 2011
Some examples :FSS July 2011, Run: 00 UTC, forecast time 18-21 hours
WG5 COSMO General Meeting, Rome 2011
• Quick look to some common plots
• Intercomparison between driving model and high resolution
• Conditional verification
• Fuzzy verification
• Long term trends (mainly precipitation)
Ulrich Damrath: Fuzzy and standard verification for COSMO-EU and COSMO-DE, COSMO GM, Rome September 2011
COSI @ DWD
Ulrich Damrath: Fuzzy and standard verification for COSMO-EU and COSMO-DE, COSMO GM, Rome September 2011
COSI @ DWD
Ulrich Damrath: Fuzzy and standard verification for COSMO-EU and COSMO-DE, COSMO GM, Rome September 2011
COSI @ DWD
Ulrich Damrath: Fuzzy and standard verification for COSMO-EU and COSMO-DE, COSMO GM, Rome September 2011
COSI @ DWD
Seasonal trend 0.2mm/24h + ECMWF
All the versions present a seasonal cycle with an overestimation during summertime (except COSMO-7 and I2) COSMO-7 and I2 underestimate Overestimation error decreases in D+2 (spin-up effect vanished)Latest summer worsening EU and I2
Dataset: high resolution network of rain gauges coming from COSMO dataset and Civil Protection Department 1300 stationsMethod: 24h/6h averaged cumulated precipitation value over 90 meteo-hydrological basins
Seasonal trend 20mm/24h + ECMWF
• Slight bias reduction during latest seasons
• winter 2010: all the versions overestimate (probably due to lack of representativeness of the rain gauges over the plain during snowfall)
• Strong COSMO-7 underestimation BUT slight improvement during latest seasons
•General underestimation during latest seasons exc. I7
Seasonal trend 0.2mm/24h + ECMWF
• Very light improvement trend• Seasonal error cycle: lower ets during winter and summertime• no significant differences between D+1 and D+2 winter 2010 (very snowy particularly in Northern Italy): low ets value (D+1 and D+2) model error or lack of representativeness of the rain gauges over the plain during snowfall ?
Seasonal trend 20mm/24h + ECMWF
• Low values during summertime (in general)
•All the versions present two “big jump” at jja08 and jja09, after the values increase and become quite stationary
•Skill decreases with forecast time