Verification Methods for High Resolution Model Forecasts Barbara Brown ([email protected]) NCAR, Boulder, Colorado Collaborators: Randy Bullock, John Halley Gotway, Chris Davis, David Ahijevych, Eric Gilleland, Beth Ebert Hurricane Diagnostics and Verification Workshop May 2009
29
Embed
Verification Methods for High Resolution Model Forecasts
Verification Methods for High Resolution Model Forecasts. Barbara Brown ([email protected]) NCAR, Boulder , Colorado Collaborators : Randy Bullock, John Halley Gotway, Chris Davis, David Ahijevych, Eric Gilleland, Beth Ebert Hurricane Diagnostics and Verification Workshop May 2009. Goals. - PowerPoint PPT Presentation
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Verification Methods for High Resolution Model Forecasts
Neighborhood verification methods• Give credit to "close" forecasts
Object- and feature-based methods• Evaluate attributes of
identifiable features
New spatial verification approaches
Keil and Craig, 2008
13
Intensity-scale methodCasati et al. (2004)
Evaluate forecast skill as a function of the precipitation intensity and the spatial scale of the error
14
Scale wavelet decomposition of binary error
Scale l=8 (640 km)
Scale l=1 (5 km)
mean (1280 km)
Scale l=6 (160 km)
Scale l=7 (320 km)
Scale l=5 (80 km) Scale l=4 (40 km)
Scale l=3 (20 km) Scale l=2 (10 km)
1
0
-1
L
lluu EE
1,
L
lluu MSEMSE
1,
From Ebert 2008
15
MSE skill score LMSE
MSEMSEMSEMSE
SS lu
randomlubestlu
randomlululu /12
1 ,
,,,,
,,,,
Sample climatology(base rate)
From Ebert 2008
16
Neighborhood verification Also called “fuzzy”
verification Upscaling
• Put observations and/or forecast on coarser grid
• Calculate traditional metrics
Provide information about scales where the forecasts have skill
17
Neighborhood methods
From Mittermaier 2008
Fractional Skill Score (FSS)
18
Feature-based verification Composite approach
(Nachamkin) Contiguous rain area
approach (CRA; Ebert and McBride, 2000; Gallus and others)
Error components• displacement• volume• pattern
19
Spatial verification method: MODE MODE: Method for
Object-based Diagnostic Evaluation
Goals• Mimic how a human
would identify storms and evaluate forecasts
• Measure forecast “attributes” that are of interest to users
Steps• Identify objects• Measure attributes• Match forecast
attributes• Measure differences in
attributes
Example: Precipitation; 1 Jun 2005
Forecast Observed
User inputs: Object identification; attribute selection and weighting
20
Object-based example: 1 June 2005 MODE quantitative
results indicate• Most forecast areas too
large• Forecast areas slightly
displaced• Median and extreme
intensities too large• BUT – overall – forecast
is pretty good
In contrast:• POD = 0.40• FAR = 0.56• CSI = 0.27
1
23
Forecast precipitation objects with Diagnosed objects overlaid
21
Back to the original example… MODE “Interest”
measures overall ability of forecasts to match obs
Interest values provide more intuitive estimates of performance than the traditional measure (ETS)
But – even for spatial methods, Single measures don’t tell the whole story!
22
Spatial Method Intercomparison Project (ICP)
Goal: Compare the various approaches using the same datasets (real, geometric, known errors)
Includes all of the methods described here; international participants
Collection of papers in preparation (Weather and Forecasting)
http://www.rap.ucar.edu/projects/icp/index.html
23
What do the new methods measure?
Attribute Traditional Feature-based
Neighbor-hood Scale
Field Defor-mation
Perf at different scales
Indirectly Indirectly Yes Yes No
Location errors No Yes Indirectly Indirectly Yes
Intensity errors Yes Yes Yes Yes Yes
Structure errors No Yes No No Yes
Hits, etc. Yes Yes Yes Indirectly Yes
24
Applicability to ensemble forecasts
From C. Davis
25
Statistical inference
Confidence intervals are required to provide• Meaningful evaluations
of individual model performance
• Meaningful comparisons of model performance
Threshold
24-h QPF, 24-h lead
26
Method availability Many methods available
as part of the Model Evaluation Tools (MET)• MODE• Neighborhood• Intensity-scale
MET is freely available• Strong user support
Software for some others is available on the intercomparison website or from original developers http://www.dtcenter.org/met/users/
27
Conclusion New spatial methods provide great
opportunities for more meaningful evaluation of precipitation forecasts – and other forecast fields• Feed back into forecast development• Provide information to users
Each method is useful for particular types of situations and for answering particular types of questions
28
Topics for discussion Consider how new methods may be
beneficial (and adaptable) for high-res NWP for hurricanes• Can these methods help?• How do they need to be
adapted/altered?• Would they be useful for other fields
(e.g., winds)?• Are other kinds of new methods
needed? Use of aircraft observations –
incomplete grids (reflectivity) Methods for evaluation of genesis? A
global problem… Need to consider false alarms as well as misses