Non-parametric postprocessing of ensemble forecasts for extreme and rare events: a focus on daily rainfall using weighted scoring rules for verification Maxime Taillardat 1,2,3 O. Mestre 4 , M. Zamo 4 , P. Naveau 2 and A-L. Fougères 3 1 CNRM/Météo-France 2 LSCE 3 ICJ 4 Météo-France April 25, 2016
50
Embed
Non-parametric postprocessing of ensemble forecasts for ... · Non-parametric postprocessing of ensemble forecasts for extreme ... a focus on daily rainfall using weighted scoring
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Non-parametric postprocessing of ensemble forecasts for extremeand rare events: a focus on daily rainfall using weighted scoring
rules for verification
Maxime Taillardat1,2,3
O. Mestre4,M. Zamo4,P. Naveau2 andA-L. Fougères3
1CNRM/Météo-France 2LSCE 3ICJ4Météo-France
April 25, 2016
The QRF technique Ensemble forecast verification Results Prospects
Ensemble forecast
source : P. Naveau
Maxime Taillardat 1/20
The QRF technique Ensemble forecast verification Results Prospects
Ensemble forecast
source : P. Naveau
Maxime Taillardat 2/20
The QRF technique Ensemble forecast verification Results Prospects
Motivations for statistical post-processing
◮ Ensembles are subject to model biases and underdispersion for
surface weather variables. (Hamill and Colucci 1997 ...)
◮ A simple bias correction is not sufficient.
◮ The skill added by post-processing is not reduced byimprovements in ensemble developments. (Hemri et al. 2014)
Our goal
Most of recent developments are based on parametric techniques.
◮ We want to focus on non-parametric/data-driven techniques.
◮ We want to deal with “tricky” weather variables (precipitation
accumulation).
Maxime Taillardat 3/20
The QRF technique Ensemble forecast verification Results Prospects
Main calibration techniques
Most popular techniques :
◮ Analog Method (Hamill and Whitaker 2006)
◮ Find in the model climate situations which are the closest
(according to a metric) of a given prediction◮ Substitute this prediction by the “analogs” observations
◮ Bayesian model averaging (Raftery et al. 2005)
◮ Ensemble model output statistics (Gneiting et al. 2005)
Maxime Taillardat 4/20
The QRF technique Ensemble forecast verification Results Prospects
Main calibration techniques
Most popular techniques :
◮ Analog Method (Hamill and Whitaker 2006)
◮ Bayesian model averaging (Raftery et al. 2005)
Forecasted proba =K∑
k=1
[proba from forecaster k
× posterior of forecaster k being correct]
◮ Ensemble model output statistics (Gneiting et al. 2005)
Maxime Taillardat 4/20
The QRF technique Ensemble forecast verification Results Prospects
Main calibration techniques
Most popular techniques :
◮ Analog Method (Hamill and Whitaker 2006)
◮ Bayesian model averaging (Raftery et al. 2005)
◮ Ensemble model output statistics (Gneiting et al. 2005)Under Gaussianity, the EMOS predictive mean is a bias-corrected
weighted average of the ensemble member forecasts. The EMOS
predictive variance is a linear function of the ensemble variance.
Maxime Taillardat 4/20
The QRF technique Ensemble forecast verification Results Prospects
Plan
1 The QRF technique
2 Ensemble forecast verification
3 Results
4 Prospects
Maxime Taillardat 5/20
The QRF technique Ensemble forecast verification Results Prospects
Quantile Regression Forests (QRF)
◮ Meinshausen 2006 (package R “quantregForest”)
◮ Quantile Regression : estimation of the conditional median or
any other quantile of the response variable given a set ofpredictors (Koenker and Bassett Jr 1978)
◮ Random Forests : aggregating predictions from binary decision
trees (CART) (Breiman 2001)
◮ Non-parametric : elimination of any assumption on the variable
subject to calibration
Maxime Taillardat 6/20
The QRF technique Ensemble forecast verification Results Prospects
Quantile Regression Forests (QRF)
◮ Meinshausen 2006 (package R “quantregForest”)
◮ Quantile Regression : estimation of the conditional median or
any other quantile of the response variable given a set ofpredictors (Koenker and Bassett Jr 1978)
◮ Random Forests : aggregating predictions from binary decision
trees (CART) (Breiman 2001)
◮ Non-parametric : elimination of any assumption on the variable
subject to calibration
Maxime Taillardat 6/20
The QRF technique Ensemble forecast verification Results Prospects
From CART to QRF
◮ Binary decision tree
A
B C
Maxime Taillardat 7/20
The QRF technique Ensemble forecast verification Results Prospects
From CART to QRF
◮ Binary decision tree
A
B C
◮ Let s be the threshold of a predictor Xi , s must create the mostpossible homogeneous branches in terms of variance :
∆R(s, b) = maxs∈Σ[R(b) − (R(bl ) + R(br ))]
where
R(t) =∑
X∈b
(yi − y(b))2
Maxime Taillardat 7/20
The QRF technique Ensemble forecast verification Results Prospects
From CART to QRF
◮ Binary decision tree
A
B C
◮ Unstable trees (low bias but very high variance) : One fits K trees
using K random samples with replacement of the training set
(bootstrap) : Tree Bagging◮ Strongly correlated trees : each split of each bagged tree is built
on a random subset of the predictors in Σ : Random Forests◮ For each final leaf of each tree one does not compute the mean
of the predictand’s values but instead their empirical CDF :
Quantile Regression
F̂x (y) = P̂(Y ≤ y |X = x) =
n∑
i=1
πi (x)I(Yi ≤ y)
Maxime Taillardat 7/20
The QRF technique Ensemble forecast verification Results Prospects
Comparison between EMOS and QRF
◮ Raw ensemble : EMOS
QRF
Maxime Taillardat 8/20
The QRF technique Ensemble forecast verification Results Prospects
Comparison between EMOS and QRF
◮ Raw ensemble : EMOS◮ Calibrated ensemble :
QRF
Maxime Taillardat 8/20
The QRF technique Ensemble forecast verification Results Prospects
Comparison between EMOS and QRF
◮ Raw ensemble : EMOS◮ Calibrated ensemble :
QRF◮ Calibrated ensemble :
Maxime Taillardat 8/20
The QRF technique Ensemble forecast verification Results Prospects
Comparison between EMOS and QRF
◮ Raw ensemble : EMOS◮ Calibrated ensemble :
QRF◮ Calibrated ensemble :
Maxime Taillardat 8/20
The QRF technique Ensemble forecast verification Results Prospects
Comparison between EMOS and QRF
◮ Raw ensemble : EMOS◮ Calibrated ensemble :
QRF◮ Calibrated ensemble :
Maxime Taillardat 8/20
The QRF technique Ensemble forecast verification Results Prospects
Comparison between EMOS and QRF
◮ Raw ensemble : EMOS◮ Calibrated ensemble :
QRF◮ Calibrated ensemble :
Maxime Taillardat 8/20
The QRF technique Ensemble forecast verification Results Prospects
Plan
1 The QRF technique
2 Ensemble forecast verification
3 Results
4 Prospects
Maxime Taillardat 9/20
The QRF technique Ensemble forecast verification Results Prospects
Paradigm of verification, scoring rules
“The paradigm of maximizing the sharpness of the predictivedistributions subject to calibration” (Gneiting et al. 2006)
◮ A proper score : the CRPS (Murphy 1969 ; Gneiting and Raftery2007 ; Naveau et al. 2015 ; MT 2016)
CRPS(F , y) =
∫
∞
−∞
(F (x)− 1{x ≥ y})2dx
= EF |X − y | −1
2EF |X − X ′|
= y + 2[
F (y)EF (X − y |X > y)− EF (XF (X))]
= EF |X − y |+ EF (X) − 2EF (XF (X))
◮ Test of equal predictive accuracy : the Diebold-Mariano test type
(1995)◮ The CRPSS (Skill score)
CRPSS(A,B) = 1 −CRPSA
CRPSB
Maxime Taillardat 10/20
The QRF technique Ensemble forecast verification Results Prospects
Plan
1 The QRF technique
2 Ensemble forecast verification
3 Results
4 Prospects
Maxime Taillardat 11/20
The QRF technique Ensemble forecast verification Results Prospects
Data and model fitting
◮ 4 yr of PEARP (ARPEGE 35-member Ensemble PredictionSystem) data from 2011 to 2014 on 87 French SYNOP stations
for 24h lead time, initialization 18H UTC
◮ For EMOS precipitation : GEV and Censored/Shifted Gammadistributions are selected (as in Hemri et al. 2014 ; Scheuerer,
Baran 2015) on a overall CRPS minimization criterion. (GPD isrejected)
◮ For Analog Method : Mahalanobis metric is kept.
◮ Two sets of predictors for QRF technique :
◮ Only with predictors concerning the variable of interest (QRF_O)◮ Like QRF_O with also the first, ninth and fifth decile of other
variable PEARP distributions (QRF_M)
Maxime Taillardat 12/20
The QRF technique Ensemble forecast verification Results Prospects
Daily rainfall
Maxime Taillardat 13/20
The QRF technique Ensemble forecast verification Results Prospects
Assessing performance for extreme and rare events
◮ A weighted score : the wCRPS (Gneiting and Ranjan 2012 ;
Naveau et al. 2015 ; MT 2016)
CRPS(F , y) =
∫
∞
−∞
w(x)(F (x) − 1{x ≥ y})2dx
= W (y) + 2[
F (y)EF (W (X) − W (y)|X > y)− EF (W (X)F (X))]
= EF |W (X)− W (y)|+ EF (W (X)) − 2EF (W (X)F (X))
where W =∫
w and 0 <∫
wf < ∞
◮ The weight function cannot depend on the observation : it leads
to improper scores.
Maxime Taillardat 14/20
The QRF technique Ensemble forecast verification Results Prospects
Weight functions used
0 5 10 15 20 25
05
10
15
20
25
Weight functions
rainfall
W
W1
W2
W3
Weight functions◮
w1(x) = 1 −f (x)
f (0.2)
f is the PDF of the climatology
◮
w2(x) = 1{x ≥ 20}
◮
w3(x) = 2w1(x)W1(x)
Maxime Taillardat 15/20
The QRF technique Ensemble forecast verification Results Prospects
Daily rainfall with weighted scoring rules
Maxime Taillardat 16/20
The QRF technique Ensemble forecast verification Results Prospects
Daily rainfall with weighted scoring rules
w4(x) = 1{x ≤ 15}
Maxime Taillardat 17/20
The QRF technique Ensemble forecast verification Results Prospects
Plan
1 The QRF technique
2 Ensemble forecast verification
3 Results
4 Prospects
Maxime Taillardat 18/20
The QRF technique Ensemble forecast verification Results Prospects
Prospects
◮ QRF technique gives at least the same or even betterperformance than EMOS unless for very high thresholds :
This is normal : if ie. an event occurs 5 times per year, we have to
get a 7-year training sample in order to build a sound 35-memberensemble with data-driven techniques.
◮ Reforecast work (Hamill and Whitaker 2006 : a 25-yrreforecast has been used)
◮ Combination of QRF and GPD CDF fitting (see tomorrow)
◮ Deal with other parameters (TCC : good preliminary results)
The QRF technique Ensemble forecast verification Results Prospects
Prospects
◮ QRF technique gives at least the same or even betterperformance than EMOS unless for very high thresholds :
◮ Reforecast work (Hamill and Whitaker 2006 : a 25-yr
reforecast has been used)◮ Combination of QRF and GPD CDF fitting (see tomorrow)◮ Deal with other parameters (TCC : good preliminary results)◮ Recovering spatio-temporal trajectories (eg. ECC Schefzik
2013)
Maxime Taillardat 19/20
The QRF technique Ensemble forecast verification Results Prospects
References
◮ Taillardat, M., O. Mestre, M. Zamo, and P. Naveau, 2016 :Calibrated Ensemble Forecasts using Quantile Regression
Forests and Ensemble Model Output Statistics. Mon. Wea. Rev.doi :10.1175/MWR-D-15-0260.1, in press.