NWP Verification with Shape-matching Algorithms: Hydrologic Applications and Extension to Ensembles Barbara Brown 1 , Edward Tollerud 2 , Tara Jensen 1 , and Wallace Clark 2 1 NCAR, USA 2 NOAA Earth System Research Laboratory, USA [email protected]ECAM/EMS 2011 14 September 2011
23
Embed
NWP Verification with Shape- matching Algorithms: Hydrologic Applications and Extension to Ensembles Barbara Brown 1, Edward Tollerud 2, Tara Jensen 1,
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
NWP Verification with Shape-matching Algorithms:
Hydrologic Applications and Extension to Ensembles
Barbara Brown1, Edward Tollerud2, Tara Jensen1, and Wallace Clark2
1NCAR, USA2NOAA Earth System Research Laboratory, USA
Developmental Testbed Center (DTC) Mission: Provide a bridge between the research
and operational communities to improve mesoscale NWP
Activities: Community support (e.g., access to operational models); Model testing and evaluation
Goals of interactions with other “testbeds”: Examine latest capabilities of high-resolution
models Evaluate impacts of physics options New approaches for presenting and evaluating
forecasts
Testbed collaborations
Hydrometeorological Testbed (HMT) Evaluation of regional ensemble
forecasts (including operational models) and global forecasts in western U.S. (California)
Winter precipitation Atmospheric Rivers
Hazardous Weather Testbed (HWT) Evaluation of storm scale
ensemble forecasts Late spring precipitation,
reflectivity, cloud top height Comparison of model capabilities
for high impact weather forecasts
Testbed Forecast Verification
Observations HMT: Gauges and Stage 4 gauge analysis HWT: NMQ 1-km radar and gauge analysis; radar
Traditional metrics RMSE, Bias, ME, POD, FAR, etc. Brier score, Reliability, ROC, etc.
Spatial approaches
Spatial approaches are needed for evaluation of ensemble forecasts for same reasons as for non-probabilistic forecasts (“double penalty”, impact of small errors in timing and location etc.)
Neighborhood methods Method for Object-based Diagnostic Evaluation (MODE)
New Spatial Verification Approaches
NeighborhoodSuccessive smoothing of
forecasts/obsObject- and feature-
basedEvaluate attributes of
identifiable featuresScale separationMeasure scale-dependent errorField deformationMeasure distortion and
displacement (phase error) for whole field
Web site: http://www.ral.ucar.edu/projects/icp/
HMT: Standard Scores for Ensemble Inter-model QPF Comparisons
Example: RMSE results for December 2010
Dashed – HMT (WRF) ensemble members
Solid: Deterministic members
Black: Ens Mean
HMT Application: MODE
19 December 2010, 72-h forecast, Threshold for Precip > 0.25”
OBS Ens Mean Ens Mean
MODE Application to atmospheric rivers
• QPF vs. IWV and Vapor Transport
• Capture coastal strike timing and location
• Large impacts on precipitation in the California Coast and Coastal mountains=> Major flooding
Evaluation of high-impact weather is moving toward use of spatial verification methods
Initial efforts in place to bring these methods forward for ensemble verification evaluation
MODE-based evaluations of AR objects
Spatial method motivation
Traditional approaches ignore spatial structure in many (most?) forecasts Spatial correlations
Small errors lead to poor scores (squared errors… smooth forecasts are rewarded)
Methods for evaluation are not diagnostic Same issues exist for ensemble forecasts
Forecast Observed
MODE example: 9 May 2011
Ensemble Workshop 2111 May 2011
MODE Example: combined objects
22
Consider and compare various attributes, such as:• Area• Location• Intensity distribution• Shape / Orientation• Overlap with obs• Measure of overall “fit”
to obs
Summarize distributions of attributes and differences
In some cases, conversion to probabilities may be informative
Spatial methods can be used for evaluation
Spatial attributes
Object intersection areas vs. lead time
Overall field comparison by MODE (“interest” summary) vs.