Barbara Brown 1 , Tara Jensen 1 , Michelle Harrold 1 , Tressa Fowler 1 , Randy Bullock 1 , Eric Gilleland 1 , and Brian Etherton 2 1 NCAR/RAL - Joint Numerical Testbed Program 2 NOAA/ESRL - Global Systems Division And Developmental Testbed Center Verification Methods for High Resolution Ensemble Forecasts Warn-On-Forecast Workshop: 8-9 February 2012 Evaluation and Diagnostic
24
Embed
Verification Methods for High Resolution Ensemble Forecasts
Evaluation and Diagnostic. Verification Methods for High Resolution Ensemble Forecasts. Barbara Brown 1 , Tara Jensen 1 , Michelle Harrold 1 , Tressa Fowler 1 , Randy Bullock 1 , Eric Gilleland 1 , and Brian Etherton 2 - PowerPoint PPT Presentation
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Barbara Brown1, Tara Jensen1, Michelle Harrold1, Tressa Fowler1, Randy Bullock1, Eric Gilleland1, and Brian Etherton2
1NCAR/RAL - Joint Numerical Testbed Program2NOAA/ESRL - Global Systems Division
AndDevelopmental Testbed Center
Verification Methods for High Resolution Ensemble Forecasts
Warn-On-Forecast Workshop: 8-9 February 2012
Evaluation and Diagnostic
Large variability in space and time Difficult to identify meaningful
impactsExtreme, high impact, weather
eventsSmall regions of importance
Difficult to identify impacts of forecast “improvements” across whole domain
Verification scores Desire for a single score... But CSI
alone does not give a lot of information about performance or improvements
Relationships and dependencies among scores
Double penalty for displaced high-res forecasts
Challenges for objective evaluation of convective scale short-term prediction...
•ObservationsUncertainties in qualityNeed to be on time and spatial scaled that support evaluation
Traditional Verification Scores
CSI is a nonlinear function of POD and FARCSI depends on base rate (event frequency) and Bias
Ex: Relationships among scores
FAR
POD
CSI
Very different combinations of FAR and POD lead to the same CSI value
1CSI 1 1 1POD 1 FAR
PODBias1 FAR
Freq Bias
Results from HMT-West 2010-2011 seasonCourtesy of Ed Tollerud, DTC/HMT Collab
9km - Ensemble Mean – 6h Precip
6HR>0.1 in.
6HR>0.5 in.
6HR>1.0 in.
6HR>2.0 in.
Performance DiagramAll on same plotPOD1-FAR
(aka Success Ratio)CSIFreq Bias
Best
Success Ratio (1-FAR)
Freq
Bia
s
CSI0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Here we see:Decreasing skill with higher thresholds even with multiple metricsRoebber (WAF, 2009)
Wilson (presentation, 2008)
Dots representDifferent leads
Looking at variability in space and time
14 May 2009 Init: 00 UTC MODE Objects Thresh: 30dBZ
No RadarAssim.
Objects ForecastField
ObservedField
RadarAssim.
SolidFCSTOBJ
LineOBS OBJ
What if you had this: Instead of this:
Objects are 30% too big (MODE area ratio=1.3) FBIAS = 1.3 for grid
Shifted west 40 km(MODE centroid distance = 10 gs) POD = 0.35
Moving too slow(MODE-TD angle diff = 15%)
FAR = 0.72
Peak Rain 1/2” too much(MODE diff in 90th percentile of intensities = 0.5”)
CSI = 0.16
Comparing objects can give more descriptive assessment of the forecast
QPE field Probability Field
Traditional MetricsBrier Score: 0.07Area Under ROC: 0.62