© Crown copyright Met Office Standard Verification System for Long-range Forecasts (SVSLRF) Richard Graham, Met Office Hadley Centre. With acknowledgements.

© Crown copyright Met Office

Standard Verification System for Long-range Forecasts (SVSLRF)Richard Graham, Met Office Hadley Centre. With acknowledgements to all GPCs, LC-SVSLRF and members of the WMO Expert Team on Extended and Long-range Forecasting (ET-ELRF)

WMO/THORPEX/WCRP sub-seasonal to seasonal implementation planning meeting, WMO HQ, Geneva,

2-3 December 2011

Lead Centre for Standard Verification of Long-range forecasts (LC-SVSLRF)

• repository of global SVSLRF scores for GPC forecasts

• Core of SVS: ROC maps and diagrams, reliability diagrams, MSSS

• documentation on verification scores (Attachment II-8 of the WMO manual on the GDPFS)

• observation datasets for verification, and code for calculating the SVSLRF scores

http://www.bom.gov.au/wmo/lrfvs/

Jointly operated byBureau of Met. BoM) & Met. Service of Canada (MSC)David Jones, Bertrand Denis

SVSLRF: skill measures

Probability forecasts (for tercile categories):

• Relative Operating Characteristics (ROC): diagrams and scores (area under ROC curve)

• Reliability diagrams and sharpness diagrams

Deterministic forecasts (ensemble mean):

• Mean Square Skill Score (MSSS) and decomposition terms (correlation, variance, bias) – climatology as reference

Variables:

• T2m, precipitation, SST, Nino3.4 index

Level 1: ‘bulk’ scores and diagrams for regions/indices (for admin.)

• ROC curves and scores; reliability diagrams and freq histograms; MSSS

• Variables/indices: T2m, precip, Nino3.4

• Regions: Tropics (20N to 20S), N. Extratropics (20-90N); S. Extratropics (20-90S)

• Seasons: MAM, JJA, SON, DJF (Nino3.4: calendar months)

• Lead-times: all available (but not > 4 months for T2m and precip; not > 6 months for Nino3.4)

Level 2: scores at grid points for generating skill maps (for users)

• ROC scores; MSSS (and decomposition)

• Variables/indices: T2m, precip, SST

• Regions: global, each defined grid point

• Seasons: 12 rolling 3-month seasons

• Lead-times: all available (but not > 4 months)

• Stratification: Where sufficient hindcast years, stratify into seasons with El Niño active and seasons with La Niña active

Lead Centre SVSLRF

ROC curvesROC score maps

Reliability/sharpness diagrams

Courtesy: David Jones (BoM)

Sample verification: ROC scores for SON precipitation, 1-month lead

ToulouseWashington TokyoMelbourne

above

below

0.6

Some issues

• Each GPC calculates own scores and submits to LC-SVSLRF. Centralised verification would help standardisation (e.g. in category definition, verification datasets used) and speed processing;

• Good practice = all forecast products have corresponding verification products – big task, we are still catching up to a degree;

• Only GPC minimum variable set are required to be verified (T2m, precip, SST, Nino3.4) – but forecasts for other variables (e.g. Z500) are available

• No verification yet of individual month forecasts (available on LC-LRFMME website)

• No verification yet of the LC-LRFMME multi-model combinations

Thank you! Any questions?