Assessing added value of high resolution forecasts Emiel van der Plas Maurice Schmeits, Kees Kok

Assessing added value of high resolution forecasts

Emiel van der PlasMaurice Schmeits, Kees Kok

KNMI, The Netherlands

2/15

IntroductionQuestion: do high resolution (convection resolving) models perform better than the previous generation models?

T2M, wind, precipitation!

KNMI: Harmonie (2.5 km) > Hirlam(11, 22 km) ?Harmonie > ECMWF (deterministic run: T1279)?

Verification of high resolution NWP forecasts is challengingPrecipitation: highly localisedRadar/stationdata: double penalty!

If there is extra skill, how to demonstrate objectively?In this talk: Fuzzy methods and Model Output Statistics

3/15

Set-upHarmonie (‘ECJAN’):

2.5 km, 300x300 domain, AROME physics, 3DVARECMWF boundariesRun with 800x800 points, Hirlam boundaries: no sufficient archive available…

Hirlam (D11):22 km, 136 x 226, 3DVAR

ECMWF Operational (T1279)±16 km, global, 3DVAR

Radar: Dutch precipitation radar composite1 km

•Period: 1st February 2012 - 31st May 2012

All output resampled to Harmonie grid (nearest neighbour)

4/15

Example of Direct Model OutputE.g. frontal precipitation, 7th March 2012ECJAN, Hirlam, ECMWF, Radar

HarmonieECMWFD11

RADAR

Neo-classical verification: fuzzy methods• MET: suite of verification tools by NCAR (WRF)• Grid based scores: with respect to gridded radar observations

–Fractions Skill Score (Roberts, Lean 2008)–Hanssen-Kuiper discriminant, Gilbert Skill Score (ETS), …

• Object based scores (not in this paper)

5/15

FSS, 3x3, > 1mm/3h GSS, 25x25, > 2mm/3h

6/15

MOS: what is relevant in DMO?• How would a trained meteorologist look at direct model output?

?

7/15

Predictors• How would a trained meteorologist look at direct model output?

8/15

Predictors• How would a trained meteorologist look at direct model output?

9/15

Predictors• How would a trained meteorologist look at direct model output?

10/15

Model Output Statistics: predictive potentialConstruct a set of predictors (per model, station, starting and lead time):

For now: use precipitation onlyUse various ‘areas of influence’: 25,50,75,100 kmDMO, coverage, max(DMO) within area, distance to forecasted precipitation, … , threshold!

Apply (extended) logistic regression [Wilks 2009]Use threshold (sqrt(q)) as predictor:

complete distribution function (Wilks, 2009)Forward stepwise selection, backward deletion

using R: stepPLR (Mee Young Park and Trevor Hastie, 2008)

Verify probabilities based on coefficients of selected predictors in terms of reliability diagrams, Brier Skill Score

11/15

Results: example poor skill

Harmonie

ECMWF

D11

00UTC+003

12/15

Results: example good skill

Harmonie

ECMWF

D11

00UTC+012

Outlook• No conclusive results

• Grid-based, “fuzzy” methods suggest reasonable skill for high resolution NWP model (Harmonie)

• MOS: mixed bag Frontal systems (FMAM) well captured by hydrostatic models

• To do:Larger datasetTraining data, independent dataConvective season: more cases, higher thresholdsInclude Harmonie run on large domain…

13/15

Binary predictand yi (here: precip > q)

Probability: logistic:

Joint likelihood:

L2 penalisation (using R: stepPLR by Mee Young Park and Trevor Hastie, 2008):minimise

Use threshold (sqrt(q)) as predictor: complete distribution function (Wilks, 2009)

Few cases, many potential predictors: pool stations, max 5 terms

14/15

Extended Logistic Regression (ELR)

ECJAN

15/15

Period1st February 2012 -31st May 2012

The archive available for Harmonie was the limiting factorMostly frontal precipitation

ECMWFD11

RADAR

Period: base rate (HSS, HK, FBIAS)

HK

17/15

Verification: classical, Fraction Skill ScoreClassical or categorical verification, eg:

Hanssen-Kuiper discriminant, (aka True Skill Statistic, Peirce Skill Score)(a d – b c)/(a + c)(b + d)

Fraction Skill Score:(Roberts & Lean, 2008)

Straightforward interpretationbut: Double penalty

CTS Observed

yes no

Forecast yes| a b

no | c d

18/15

Verification: MODE (object based), waveletsMET provides access to MODE analysis:“Method for Object-based Diagnostic Evaluation”

Forecast, observation: convolution, thresholded, …

FC OBS

19/15

Verification: MODE (object based), waveletsMET provides access to MODE analysis:“Method for Object-based Diagnostic Evaluation”

… merged, matched and compared.

Center

of mass

Area,

Angle,

Convex hull,

…FC OBS