Topic 3.1: Effective Warning Process: Forecast Validation Barbara Brown 1 , Beth Ebert 2 1 NCAR, Boulder, Colorado, USA ([email protected]) 2 CAWCR / BOM. Melbourne, Australia ([email protected]) Working group: G. Chen, STI, China; E. Fukada, JTWC, USA; E. Gilleland, NCAR, USA; P. Otto, BOM, Australia; A. Tyagi, Ministry of Earth Sciences, India; Hoa Vo Van, NCHMF, Vietnam; L. Wilson, EC, Canada; Hui Yu, STI, China International Workshop on Tropical Cyclones – 8 Jejju, South Korea 4 December 2014
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Topic 3.1: Effective Warning Process:
Forecast Validation
Barbara Brown1, Beth Ebert2 1NCAR, Boulder, Colorado, USA ([email protected])
Working group: G. Chen, STI, China; E. Fukada, JTWC, USA; E. Gilleland, NCAR, USA;
P. Otto, BOM, Australia; A. Tyagi, Ministry of Earth Sciences, India;
Hoa Vo Van, NCHMF, Vietnam; L. Wilson, EC, Canada; Hui Yu, STI, China
International Workshop on Tropical Cyclones – 8
Jejju, South Korea
4 December 2014
2
Role of verification / validation
Improve forecasting models and processes (feedback into the forecast development process) Develop understanding of prediction errors
Diagnose and quantify systematic and random errors so improvements can be made to operational forecasting methodologies and NWP models
Provide uncertainty and reliability information to users of TC forecasts Users can make better decisions
Forecasters use NWP verification info to make optimal use of NWP models Optimal use of multiple sources of guidance
IWTC VIII, December 2014, Jeju, South Korea
3
Survey of methods in W. Pacific
A survey of the operational practice of TC forecast verification was carried out in 2012, covering all Members of the ESCAP/WMO Typhoon Committee, RSMC Tokyo Typhoon Center, and JTWC.
Aiming to gain an idea on: What verification techniques and products on what
forecast elements are currently available to TC forecasters?
What are the weak points in the verification capabilities which it would beneficial to consider improving across the region in the future?
Yu, H., S.T. Chan, B. Brown, and Coauthors, 2012: Operational tropical cyclone forecast verification practice in the western
North Pacific region. Tropical Cyclone Res. Rev., 1(3), 361-372.
When possible, uncertainty in verification results should be represented using
Statistical confidence intervals and hypothesis tests
Box plots or other methods to represent distributions of errors or other statistics
Insight through
stratifying data
Stratification of results to aid in understanding and provide forecasters with
additional insights
Ex: time of year, basin, storm speed, track characteristics, ensemble spread
Must maintain large enough sample size to ensure meaningful results
Relevant metrics Verification measures reported should be selected to be relevant for the particular
users of the information (i.e., answer specific questions about forecast performance
of interest)
17
Gaps
Consistency in best track analyses
Has profound impact on verification and ability to intercompare forecasting systems
Need for better observations
Gridded wind fields
Observation uncertainty information
Incorporation of observation uncertainty into verification studies/analyses
Limited evaluation of ensemble predictions
Going beyond the mean (i.e., probabilistic treatment)
IWTC VIII, December 2014, Jeju, South Korea
18
Recommendations: Observations
WMO should continue to encourage and facilitate greater sharing of relevant observational data for verification of TC forecasts
Best track datasets should be shared as soon as possible to facilitate inter-comparisons between international models and forecasts.
Improvement of observations for verification of landfall location, timing, and weather hazards.
Better documentation of observations of sustained wind and wind gusts, including metadata, assumptions, and (ideally) estimates of the uncertainties
Improvement of estimates of TC structure from remotely sensed and conventional data
IWTC VIII, December 2014, Jeju, South Korea
19
Recommendations: Improvement of
verification approaches
WMO should continue to promote good general verification practices and specific methodologies for TC verification Mechanisms: relevant publications, online resources,
workshops, and in-country training.
NMHSs and agencies should focus additional attention on verifying the extremes of weather – e.g., heavy precipitation, strong wind, storm surge, and dangerous waves Specific, relevant, measures should be used to evaluate
these forecasts
Researchers should continue to develop and improve methodologies for verifying forecast aspects of tropical cyclone formation, structure, evolution, and motion
IWTC VIII, December 2014, Jeju, South Korea
20
Recommendations: Coordination
Specify a basic set of TC verification metrics for use by NMHSs and partner agencies. Relevant groups to involve:
Working Group on Tropical Meteorology Research in coordination with the Tropical Cyclone Programme,
Joint Working Group on Forecast Verification Research
Public Weather Services Programme, and other relevant groups identified by the WMO.
Consider establishment of a Lead Center for Tropical Cyclone Verification to ensure consistent and timely verification of forecasts from NWP models for all basins in which TCs occur. Similar to existing centers for deterministic, ensemble,