Point, Interval, and Density Forecast Evaluation of Linear ...cepr.org/sites/default/files/Slides7d.pdf · Point, Interval, and Density Forecast Evaluation of Linear versus Nonlinear

Point, Interval, and Density Forecast Evaluationof Linear versus Nonlinear DSGE Models

Francis X. Diebold Frank Schorfheide Minchul Shin

University of Pennsylvania

May 4, 2014

1 / 33

Motivation

The use of DSGE models as a forecasting tool in practice calls forevaluation of its performance.

Summary of DSGE forecasting evaluation literature:

I Point forecasts are as good as other statistical models such asVAR.

I Density forecasts are too wide.

Typically, DSGE models in this literature are

I Solved with linearized method.

I Constant volatility.

Forecasting with a nonlinear DSGE model:

I Pichler (2008) considers the second-order perturbationmethod.

2 / 33

Nonlinearity and Forecasting

Not much work has been done for nonlinear DSGE forecasting.

Two types of nonlinearitiesI Time-varying volatility vs. Constant volatility

I There are many papers on DSGE models with time-varyingvolatility but there is no paper that systematically evaluatesforecasts.

I Most papers report that density forecasts generated from theDSGE models with constant volatility is too wide.

I Nonlinear approximation vs. Linear approximationI Pichler (2008)’s point forecast evaluation.I Interval and density forecasts evaluation.

3 / 33

Summary

MethodI Small-scale DSGE model.

I Linear DSGE model with constant volatility.I Linear DSGE model with time-varying volatility.I Quadratic DSGE model (based on the second-order perturbation

method).I Quadratic DSGE model with time-varying volatility.

I Bayesian estimation and forecasting. (US data, 1964-2011)

Finding

I Modelling time-varying volatility improves accuracy offorecasts.

I Second-order perturbation method does not improve.

4 / 33

Model, Estimation, and Forecasting

Small-Scale DSGE Model

Small-scale model used in Herbst and Schorfheide (2012)

I Euler equation, NK Phillips curve, Monetary policy rule

I 3 exogenous shocks: technology, government spending,monetary policy (zt , gt ,mpt)

Measurement equations:(YGRt

INFt

FFRt

)= D(θ) + Z (θ) st

Transition equations:

st = Φ(st−1, εt ; θ)

where

st = [yt , yt−1, ct , πt ,Rt ,mpt , zt , gt ]′

εt : Innovations

θ : DSGE parameters

6 / 33

Linear DSGE Model

Linear approximation methods lead to a linear Gaussian state spacerepresentation with the following transition equation

st = H(θ)st−1 + R(θ)εt , εt ∼ iidN (0,Q(θ)).

I Coefficient matrices (H(θ),R(θ),Q(θ)) are the nonlinearfunction of θ.

I We obtain posterior draws based on the Random WalkMetropolis (RWM) algorithm with the Kalman filter.

7 / 33

Linear+Stochastic Volatility

Following Justiniano and Primiceri (2008),

st = H(θ)st−1 + R(θ)εt , εt ∼ N (0,Qt(θ))

where

diag(Qt(θ)) = [e2hmp,t , e2hz,t , e2hg,t ]′

hi ,t = ρσihi ,t−1 + νi ,t , νi ,t ∼ iidN (0, s2i ),

for i = mp, g , z .

I The system is in a linear Gaussian state-space formconditional on Qt(θ).

I We use the Metropolis-within-Gibbs algorithm developed byKim, Shephard, and Chib (1998) to generate draws from theposterior distribution.

8 / 33

Quadratic DSGE Model

The equilibrium law of motion based on the second-orderperturbation method has the following form:

s1,t = G0(θ) + G1(θ)s2,t + G2(θ)(s2,t ⊗ s2,t)

s2,t = H0(θ) + H1(θ)s2,t−1 + H2(θ)(s2,t−1 ⊗ s2,t−1) + R(θ)εt

where εt ∼ iidN (0,Q(θ)), st = [s1,t , s2,t ]′ and ⊗ is a Kronecker

product.

I We have an additional quadratic term.

I We need a nonlinear filter to get posterior draws.

I Particle filter + RWM (exact).I Second-order Extended Kalman Filter + RWM (approximate).

I Current version of the paper utilizes the second-order extended Kalmanfilter and RWM.

I We also consider Quadratic DSGE model with stochastic volatility.

9 / 33

Predictive Distribution

We generate draws from the posterior predictive distribution basedon the following decomposition,

p(YT+1:T+H |Y1:T ) =

∫p(YT+1:T+H |θ,Y1:T )p(θ|Y1:T )dθ

I p(θ|Y1:T ) : Posterior sampler.

I p(YT+1:T+H |θ,Y1:T ) : Given θ, simulate the model economyforward.

Draws {Y (j)T+1:T+H}

nsimj=1 can be turned into point and interval

forecasts by the Monte Carlo average,

E[yT+h|T

]=

∫yT+h

yT+h p(yT+h|Y1:T )dyT+h ≈1

nsim

nsim∑j=1

y(j)T+h.

10 / 33

Evaluation Methods

Point Forecast

Point forecasts from two different models (Inflation rate)

We compare RMSEs for variable i = {YGR, INFL, FFR},

RMSE(i |h) =

√√√√ 1

P − h

R+P−h∑t=R

(yi,t+h − yi,t+h|t)2

where R is the index denotes the starting point of the forecasts evaluationsample and P the number of forecasting origins.

12 / 33

Interval Forecast

Interval forecasts from two different models (Inflation rate)

I We compute α% interval forecasts for a particular element yi,T+h of yT+h

by numerically searching for the shortest connected interval that containsa α% of the draws {y (j)

i,T+h}nsimj=1 (the highest-density set).

I If the interval forecast is well calibrated, actual variables are expected tobe inside of α% interval forecasts at the same frequency.

13 / 33

Interval Forecast

Interval forecasts from two different models (Inflation rate)

Christoffersen (1998)’s LR tests of the correct coverage (α):

I Define the sequence of hit indicators of a 1-step-ahead forecast interval,

Iαt = 1{realized yt falls inside the interval}

I TestIαt ∼ iidBernoulli(α).

14 / 33

Density Forecast

Density forecasts from two different models (Inflation rate)

The probability integral transform (PIT) of yi,T+h based on time T predictivedistribution is defined as the cumulative density of the random variable,

zi,h,T =

∫ yi,T+h

−∞p(yi,T+h|Y1:T )dyi,T+h.

I If the predictive distribution is well-calibrated, zi,h,T should follow theuniform distribution.

I For h = 1, zi,h,T ’s follow independent uniformly distribution.

I Diebold, Gunther, and Tay (1998).

15 / 33

Density Forecast Evaluation: Predictive Likelihood

The one-step-ahead predictive likelihood,

PL(t) = p(yt+1|Y1:t)

I Height of the predictive density at the realized value yt+1.

I Log predictive score:∑R+P−1

t=R logPL(t).

We approximate

PL(t) ≈ 1

M

M∑m=1

p(yt+1|Y1:t , θ

(m))

where {θ}(m) is a sequence from the posterior simulator using dataY1:t .

16 / 33

Results

Data

For the evaluation of forecasts, we use the real-time data setconstructed by Del Negro and Schorfheide (2012).

I Forecast horizons and data vintages are aligned with BlueChip survey publication dates.

I Output growth, Inflation, Federal Funds rate.

I Generate forecasts four times a year (January, April, July, andOctober).

To evaluate forecasts we recursively estimate DSGE models overthe 78 vintages starting from January 1992 to April 2011.

I All estimation samples start in 1964.

I Compute forecast errors based on actuals that are obtainedfrom the most recent vintage.

18 / 33

Data

Data, 1964Q2-2011Q1

19 / 33

Extracted Volatility

Stochastic Volatility, 1964Q2-2010Q3

Standard deviations of the structural shocks (Posterior mean):

I Dotted : Linear DSGE with constant volatility

I Solid : Linear DSGE with SV

20 / 33

Point Forecasts Evaluation: RMSEs

Linear vs Linear+SV, 1991Q4-2011Q1

I SV improves point forecasts for output growth.

I Difference in RMSEs for output growth is significant (Dieboldand Mariano, 1995)

21 / 33


Quadratic vs Quadratic+SV, 1991Q4-2011Q1

I SV improves point forecasts for output growth for Quadraticmodel as well.

22 / 33


Linear+SV vs Quadratic+SV, 1991Q4-2011Q1

I Quadratic term does not improve point forecasts.

I Similar results as in Pichler (2008) but with SVs.

23 / 33

Interval Forecasts: Coverage Rate

Coverage Rate of 70% Interval Forecasts, 1991Q4-2011Q1

If the interval forecast has a correct coverage rate, then it should be around 0.7line (dotted black line).

I Interval forecasts from the models with SV are closer to 0.7 line.

I Quadratic+SV has better coverage rates for inflation rate h ≥ 3.

24 / 33

Interval Forecasts

Output Growth Interval Forecasts (h = 1), 1991Q4-2011Q1

I In general, interval forecasts are shorter for the model with SV.

I pre-Great moderation sample effect.

I Rolling window estimation (with 80Q) helps but notmuch.

25 / 33

Interval Forecasts

Output Growth Interval Forecasts (h = 1), 1991Q4-2011Q1

I In general, interval forecasts are shorter for the model with SV.

I pre-Great moderation sample effect.

I Rolling window estimation (with 80Q) helps but notmuch.

26 / 33

Density Forecasts Evaluation: PITs, (h = 1)

PITs, 1-Step-Ahead Prediction, 1991Q4-2011Q1

Linear DSGE Model

Linear+SV

PITs are grouped into five equally sized bids. Under a uniform distribution,

each bin should contain 20% of the PITs, indicated by the solid horizontal lines

in the figure.

27 / 33

Density Forecasts Evaluation: PITs (h = 1)



Quadratic + SV



in the figure.

28 / 33



Linear DSGE Model

Linear+SV



in the figure.

29 / 33




Quadratic + SV



in the figure.

30 / 33

Density Forecasts Evaluation: Normalized Error

wi ,t = Φ−1(zi ,t)

LR Tests of normalized errors of 1-step ahead real-time forecasts

Std. Dev. (= 1) Mean (= 0) AR(1) coef. (= 0) LR test

(a) Output Growth

Linear 0.543 (0.000) 0.195 (0.005) 0.357 (0.021) 53.049 (0.000)Quadratic 0.547 (0.000) 0.206 (0.003) 0.364 (0.018) 52.984 (0.000)Linear+SV 0.827 (0.349) 0.093 (0.493) 0.254 (0.072) 10.610 (0.014)Quadratic+SV 0.845 (0.422) 0.146 (0.282) 0.315 (0.022) 13.547 (0.004)

(b) Inflation RateLinear 0.762 (0.316) 0.058 (0.597) -0.125 (0.474) 10.595 (0.014)Quadratic 0.738 (0.045) 0.075 (0.598) 0.119 (0.273) 12.694 (0.005)Linear+SV 0.868 (0.461) 0.130 (0.368) -0.000 (0.998) 3.915 (0.271)Quadratic+SV 0.890 (0.592) 0.050 (0.777) 0.090 (0.516) 2.569 (0.463)

(c) Fed Funds RateLinear 0.578 (0.000) -0.070 (0.658) 0.663 (0.000) 77.837 (0.000)Quadratic 0.615 (0.000) -0.067 (0.705) 0.672 (0.000) 73.064 (0.000)Linear+SV 0.866 (0.492) -0.119 (0.593) 0.744 (0.000) 66.514 (0.000)Quadratic+SV 0.859 (0.483) -0.073 (0.743) 0.712 (0.000) 58.611 (0.000)

31 / 33

Log Predictive Density

Log Predictive Score, 1991Q4-2011Q1

h = 1Q h = 2Q h = 4Q

Linear -3.99 -4.20 -4.91Quadratic -3.97 -4.49 -5.05Linear+SV -3.82 -4.66 -5.70Quadratic+SV -3.84 -4.63 -5.52

I Linear+SV performs the best for the 1-step-ahead prediction.

I Linear without SV performs better for h ≥ 2.

32 / 33

Conclusion

Summary

I Modelling the time variation improves the density forecastsespecially in the short-run.

I The second order perturbation method does not improveforecasts’ quality in general.

Future worksI More features

I Time-varying inflation target.I SV process: ARMA(p,q) as opposed to AR(1).

I Larger model

I Particle filter

33 / 33

Point, Interval, and Density Forecast Evaluation of Linear ...cepr.org/sites/default/files/Slides7d.pdf · Point, Interval, and Density Forecast Evaluation of Linear versus Nonlinear

Documents