1 Probabilistic Forecast Verification Allen Bradley IIHR Hydroscience & Engineering The University of Iowa RFC Verification Workshop 16 August 2007 Salt Lake City
Jan 30, 2016
1
Probabilistic Forecast Verification
Allen BradleyIIHR Hydroscience &
Engineering The University of Iowa
RFC Verification Workshop16 August 2007Salt Lake City
2
Advanced Hydrologic Prediction Service
Ensemble streamflow forecasts
3
Advanced Hydrologic Prediction Service
Ensemble streamflow forecastsMultiple forecast locations
4
Advanced Hydrologic Prediction Service
Ensemble streamflow forecastsMultiple forecast locationsThroughout the United States
5
Forecast Location
Forecast Date
Forecast Variable
How good are the
ensemble forecasts
produced by AHPS?
AHPS Verification
6
Outline
Illustrate a consistent diagnostic framework for verification of AHPS ensemble forecastsDescribe a prototype interactive web-based system for implementing this verification framework within an RFCPresent a future vision for the role of verification archives in AHPS forecasting
6
7
Forecast Verification Framework
8
Perspective: Forecast Users
Evaluate the quality of forecasts at a specific location for a particular forecast variable and date
Examine one “element” in the data cube
8
9
Elemental Problem
Use a distributions-oriented approach (DO) to evaluate probability forecasts for “events” defined by a flow thresholdForecast quality attributes quantified over a range of flow thresholds
9
10
0
2000
4000
6000
8000
10000
Jun/14 Jul/7 Jul/31 Aug /23 Se p /15
Da
ily F
low
Vo
lum
es
(cfs
-da
ys)
D a te
D es M oines R iver
E nsem ble Stream flow P red ictions
Ensemble Forecast
11
Ensemble Forecast
Probability forecast of a discrete event
C onditional D istribu tion Forecast
1 0 4
1 0 5
1 0 6
1 0 7
.01 .1 1 5 10 2030 50 7 08 0 9 0 9 5 99 9 9 .9
Pe rc e nt
D es M oines R iver
Sea
son
al F
low
Vo
lum
e (
cfs-
da
ys)
yy
ff
12
Ensemble Forecast
Probability forecast of a discrete eventProbability forecasts of multicategory events
C onditional D istribu tion Forecast
1 0 4
1 0 5
1 0 6
1 0 7
.01 .1 1 5 10 2030 50 7 08 0 9 0 9 5 99 9 9 .9
Pe rc e nt
D es M oines R iver
Sea
son
al F
low
Vo
lum
e (
cfs-
da
ys)
WetWet
Near Near AvgAvgDryDry
fdry favg fwet
12
13
Ensemble Forecast
Generalize by defining event forecasts as a continuous function of threshold
C onditional D istribu tion Forecast
1 0 4
1 0 5
1 0 6
1 0 7
.01 .1 1 5 10 2030 50 7 08 0 9 0 9 5 99 9 9 .9
Pe rc e nt
D es M oines R iver
Sea
son
al F
low
Vo
lum
e (
cfs-
da
ys)
f0.50
yy0.500.50
yy0.750.75
f0.75
yy0.250.25
f0.25
Index function by the threshold’s climatological probability
13
14
Ensemble Forecast Verification
Forecast y<y p?
Date f x
1949/09 0.805 1
1950/09 0.952 1
1951/09 0.128 0: : :
1964/09 0.804 0
1965/09 0.732 01966/09 0.962 1
: : :1999/09 0.365 02000/09 0.130 1
Compute forecast-observations pairs for specific thresholds yp
Evaluate forecast quality for a range of thresholds yp
15
Des Moines River near Stratford
Standard Errors
Skill Skill depends on the thresholdUncertainty is greater for extremes
April 1st Forecasts
15
16
Distributions-Oriented Measures
Skill Score Decomposition:
(SS)Skill
22
22
x
xf
x
ffxfxMSESS
(RES)Resolution
(CB)Conditional
Bias
(UB)Unconditional
Bias
SlopeReliability
StandardizedMean Error
PotentialSkill
17
-0 . 6
-0 . 4
-0 . 2
0
0 . 2
0 . 4
0 . 6
0 . 8
1
0 0 . 2 0 . 4 0 . 6 0 . 8 1
N o n e x c e e d a n c e p r o b a b i li ty ( )p
D e s M o i n e s R i v e r a t J a c k s o n ( J C K M 5 )
S S
P S
S R E L
S M E
April 1st
Implications for Verification
IncreaseProbability
forecast skill
Eliminatewith bias-correction
Minimum 7-Day Flow
SSRESCBUB
17
18
Perspective: NWS RFC Forecaster
Assess the overall performance of the forecasting system Diagnose attributes limiting forecast skill (e.g., biases)
Examine “slices” and “blocks” of the data cube
18
19
Multidimensional Problem
The forecaster needs summary verification measures suitable for comparing forecasts at different locations and/or forecasts issued on different datesSummary measures describe attributes of the skill functions derived from the elemental verification problem
19
20
Summary Verification Measures
Ranked Probability Skill Score (RPSS):
MSEiMSEi
i SSpSSwRPSS )(
Weighted-average skill over probability thresholds
iii
iii pp
ppw
)1(
)1(
21
Summary Verification Measures
Skill RPSS shows average skillCenter of mass shows asymmetries in the skill function
RPSS
Center ofMass
21
22
Hypothetical Skill Functions
All skill functions have same average skill
22
23
Hypothetical Skill Functions
All skill functions have same average skillSecond central moment shows shape
23
24
Hypothetical Skill Functions
All skill functions have same average skillSecond central moment shows shape
24
25
NCRFC Forecasts
7-day minimum flow forecasts for mainstem locations for three rivers
MinnesotaRiver (MIN)
Des MoinesRiver (DES)
Rock River (RCK)
25
26
Forecast Skill Attributes
Forecasts made at the 1st of the month
26
27
Forecast Skill Attributes
Average skill is highest for DESThe skill function is peaked in the middle
27
28
Summary Measure Decomposition
Skill Score Decomposition:
Skill
)()()()( iiii pUBpCBpRESpSS
Resolution ConditionalBias
UnconditionalBias
UBCBRESSSRPSS
Weighted-average measures of resolution and biases
29
AHPS Minimum 7-Day Flow
A single MIN site has large biases for low flowsThe largest biases for other sites centered on higher flows
29
30
Forecast Bias Attributes
Unconditionalbias is dominateSimple bias-correction can significantly improve forecasts
Simple bias-correction
Post-hoccalibration
30
31
Verification Framework
Forecast quality for ensemble forecasts (e.g., skill) is a continuous function of the forecast outcome (or its climatological probability)Summary measures can be interpreted as measures of the “geometric shape” of the forecast quality functionThis interpretation provides a framework for concisely summarizing the attributes of ensemble forecasts
31
32
AHPSVerification
System
33
AHPS Verification System
Web-based tools for online
access, analysis, and comparison of retrospective
AHPS forecasts for River Forecast
Centers (RFCs)http://www.iihr.uiowa.edu/ahps_ver
34
Map-Based Navigation
2
1
3
35
Verification Data Archive
35
36
0
2000
4000
6000
8000
10000
Jun/14 Jul/7 Jul/31 Aug /23 Se p /15
Da
ily F
low
Vo
lum
es
(cfs
-da
ys)
D a te
D es M oines R iver
E nsem ble Stream flow P red ictions
Verification Data Archive
Retrospective forecasts for a 50-year period
36
37
Verification Data Archive
Retrospective forecasts for a 50-year periodProcessed ensemble forecasts & observations
C onditional D istribu tion Forecast
1 0 4
1 0 5
1 0 6
1 0 7
.01 .1 1 5 10 2030 50 7 08 0 9 0 9 5 99 9 9 .9
Pe rc e nt
D es M oines R iver
Sea
son
al F
low
Vo
lum
e (
cfs-
da
ys)
37
38
Verification Data Archive
Retrospective forecasts for a 50-year periodProcessed ensemble forecasts & observationsVerification results
38
39
Verification System ConceptsRetrospective ensemble traces available in their native format (*.VS files)Processed ensemble forecasts & observations for a suite of variables
Uses *.qme files from the calb systemForecast quality measures based on the ensemble forecasts
39
40
Disk Requirements
• 6 Forecast periods per month (72 per year)• All segments have 50 years observed record
1 600Segment Segments
Elements (MB) (GB)Ensemble Traces (*.VS) 96.8 58.1Ensemble forecasts/obs 91.3 54.8Verification measures 89.3 53.6
Total Disk Usage 277.4 166.5
41
Advantages
Interactive exploration of verification results
Provides a diagnostic “report card” for sites within an RFC
Instant access to forecasts and quality measures for verification sitesSeamless integration with other components of the NWSRFS system
41
42
A Vision for theFuture
43
Vision
Generation and archival of retrospective forecasts will be a routine component of forecasting systems
Verification methods can assess qualityVerification results would form the basis for accepting (or rejecting) proposed improvements to the forecasting systemArchival information will form the basis for generating improved forecast products
44
Product Generation with ArchiveRaw ESP forecastArchive verification indicates biases and skillOptimal merging and bias correctionEnsemble Forecast
VerificationArchive
OptimizedCS
45
Conclusions
46
ConclusionsA consistent verification framework provides both users and forecasters with the means evaluating forecast products (exploring the “data cube”)AHPS-VS integrates retrospective forecast generation and forecast verification within the operational setting of an RFCRetrospective forecast archives will become a routine component of a hydrologic forecasting system, enhancing forecast evaluation and product generation
47
48
Des Moines Forecast Skill
Skill is higher (lower) downstream (upstream)Skill decline from April to June
JCKM5
DESI4
STRI4
TotalBias