1 Mt. Mansfield, VT. Aka….New England Days 4-7 Forecast Test Paul A. Sisson NOAA/NWS, Weather Forecast Office, Burlington, Vermont (BTV) Joseph Dellicarpini.

1Mt. Mansfield, VT

2

Aka….New England Days 4-7 Forecast Test

Paul A. SissonNOAA/NWS, Weather Forecast Office, Burlington,

Vermont (BTV)

Joseph DellicarpiniNOAA/NWS, Weather Forecast Office, Taunton,

Massachusetts (BOX)

Michael EksterNOAA/NWS, Weather Forecast Office, Gray, Maine

(GYX)

Todd FoisyNOAA/NWS, Weather Forecast Office, Caribou, Maine

(CAR)

David Radell and Jeff WaldstreicherNOAA/NWS, Eastern Region Headquarters, Bohemia,

New York (ERH)

Mt. Mansfield, VT

3

The real workers

• Matt Belk BOX• Conor Lahiff BTV• Margaret Curtis GYX• Roman Berdes CAR

Acknowledgements• The NWS Central Region Blender

team• Tim Barker, SOO, BOI - BOIVerify

4

Outline

• Background• Motivation• The Experiment• Verification Results• Summary

“The Blend is your friend”

5

Background• Many Studies on forecast/model consensus in general

agreement that the consensus forecast tends to be the best forecast– Gyakum (1986), Fritsch et al (2000), Roebber (2014) etc.

Baars and Mass (2005)

Consensus/Weighted MOS – “competitive or superior to human forecasts at nearly all locations”

“Human forecasts are most skillful compared to MOS during the first forecast day and for periods when temperatures differ greatly from climatology.”

6

Motivation

• Work efficiently• Improve Accuracy• Improve Consistency• Focus on what we do best – (sig departures from climo and short

term)

Note: Don’t reinvent the wheel“The Blend is your friend”

7

Consistency Problems

8

The Days 4-7 Forecast Experiment

• Oct 2013 - Mar 2014• All offices start with same model

initializations and blends• Forecaster Surveys and Verification• Does blend outperform Gridded MOS?• Is the forecast more consistent?• Does the method allow forecasters to

be more efficient and allow time to do important things?

9

Nomenclature

• CONS = Consensus • A consensus data set is

calculated by combining/averaging a “list” of guidance data sources.

10

BC = Bias Correction

• Bias corrections are run on individual guidance sources using BOIVerify software.

• BCCONS data set is generated by bias correcting the individual components before forming the consensus.

• Using previous 14 day period BC

11

BLENDS

• Blends are a combination of various datasets of various weights.

Name Description

CMCnh Canadian Meteorological Center Global model Raw Model output

GFS40 NWS Global Forecast System (GFS: 40km) Raw Model output

ECMWFEuropean Centre Medium Range Forecast model

Raw Model output

SREF NWS Short Range Ensemble Forecast (mean) Raw Model output

NAM12 NWS North American Model (NAM: 12km) Raw Model output

NAMDNG5

NWS NAM downscaled to 5km grid Raw Model output

ADJMEXGFS40km ADJusted with GFS extended MOS point forecasts

Raw Model background with MOS

ADJMENGFS40km ADJusted with GFS ensemble mean MOS pt forecasts

Raw Model background with MOS

ADJECEECMWF ADJusted with ECMWF MOS pt forecasts

Raw Model background with MOS

ADJECMECMWF ADJusted with ECMWF ensemble mean MOS pt forecasts

Raw Model background with MOS

MOSG25 GFS Gridded Model Output Statistics (2.5km) GFS MOS

*Note: all forecasts mapped/downscaled to 2.5km grid

Forecast Databases

Name Description Databases

CONSAllConsensus Raw Models and MOS

CMCnh, GFS40, ECMWF, SREF, NAM12, NAMDNG5, MOSG25, ADJMEX, ADJMEN, ADJECE, ADJECM (equal weights)

BCCONSAll

Consensus of Bias-corrected Raw Model and MOS databases

CMCnhBC, GFS40BC, ECMWFBC, SREFBC, NAM12BC, NAMDNG5BC, MOSG25BC, ADJMEXBC, ADJMENBC, ADJECEBC, ADJECMBC

HPCGuideHuman Forecast by Weather Prediction Center

Human adjusted Blend

OfficialPrevious Weather Forecast Office forecast

Human adjusted Blend

Forecast Databases

14

SuperBlend

• Previous Forecast + latest blends

• Official (25%), • HPCGuide (25%) • CONSALL (25%) • BCCONSALL (25%)

• 50/50 Man Machine mix

15

Verification Results

• MaxT, MinT, T, Td, Wind Speed, PoP

• Use Real-Time Mesoscale Analysis adjusted by Observation as verification and for bias-correction

RTMA MaxT

Obs to ADJ RTMA MaxT

18

RESULTS

19

Max_T Min_T T Td Wind Speed

-10

-5

0

5

10

15

20

25

30All WFOs Days 4-7 Mean % Improvement

Over GMOS

OfficialHPCGuideSuperblendCONSALLBCCONSALL

% Im

prov

emen

t ove

r GM

OS

BOX: Nov-Mar 2014.GYX: Jan-Mar 2014.CAR: Nov/Mar2014.BTV: Oct.-Mar 2014.

20

AD

JECM

Raw

Blen

dA

llBle

ndSu

perB

lend

Offi

cial

AD

JECE

CON

SAll

AD

JECM

BCA

DJE

CEBC

BCCO

NSA

llBC

CON

SRaw

ECM

WFB

CCO

NSR

awCO

NSM

OS

AD

JMEX

BCCO

NSM

OS

CMCn

hBC

AD

JMEN

MO

SG25

CMCn

hEC

MW

FA

DJM

EXBC

HPC

Gui

deM

OSG

uide

MO

SG25

BCA

DJM

ENBC

MO

SGui

deBC

GFS

40BC

NA

M12

BCU

KMET

BCN

amD

NG

5BC

GFS

40EK

DM

OSB

CD

GEX

EKD

MO

SH

PCG

uide

BCU

KMET

DG

EXBC

NA

M12

Nam

DN

G5

2.003.004.005.006.007.008.009.00

10.0011.00

4.11 4.25

4.27

4.29 4.46 5.

11

5.19

6.79

Max T Avg MAE Periods 6-14 BTV CWA vs Obs

MAE

deg

F

Day 5 MinT M

OS

G2

5M

OS

Gu

ide

CO

NS

MO

SA

DJ

ME

XC

ON

SA

llG

FS

40

All

Ble

nd

HP

CG

uid

eO

ffic

ial

Su

pe

rBle

nd

MO

SG

uid

eB

CM

OS

G2

5B

CB

CC

ON

SM

OS

GF

S4

0B

CP

rev

iou

sB

CC

ON

SA

llB

CC

ON

SR

aw

EC

MW

FB

CA

DJ

ME

XB

CD

GE

XB

CC

MC

nh

BC

Ra

wB

len

dB

CC

ON

SS

ho

rtC

ON

SS

ho

rtC

ON

SR

aw

DG

EX

CM

Cn

hE

CM

WF

-6

-5

-4

-3

-2

-1

0

1

2

3

4

Day 5 MinT Bias BTV CWA

Series1

Day 4 PoP Reliability

Under forecast

Over forecast

23

Survey Results

• Compare Before, During, & After

• Before: Gridded MOS method

• After: SuperBlend method

Impact of SuperBlend on Days 4-8 Forecast Process

MEAN – 4.69

Mid-Test Survey Pre-Test Survey (MOSGuide)

MEAN – 4.48 MEAN – 3.45

Forecaster ModificationsWhat Doesn’t Work Well

SuperBlend

MOSGuide

SuperBlend PerformanceRegime Changes / Anomalous Conditions

SuperBlend PerformanceRegime Changes / Anomalous Conditions

MEAN – 4.64

Mid-Test Survey Pre-Test Survey (MOSGuide)

MEAN – 4.31 MEAN – 3.16

Forecaster Overall Evaluation of Approach

29

Consistency Mar 2013Less Than 80 %

80 - 90 % 90 - 95 % 95 - 100 % 100 %

30

QPF

31

SnowAmt

32

Summary

• Blends provide a more accurate and consistent starting point

• Forecasters survey show confidence in the method

• Allows more time for operations vs grid preparation

33

Bill Belichick, Head CoachNew England Patriots

Dec 14, 2009

Caveat: "Stats are for losers," "The final score is for winners."

34

Or….stated another way

Caveat: Stats are helpful

Clearly Communicating Accurate and Timely Weather Information is the goal.

35

The End

Name Description Databases

CONSRawConsensus of Raw Models

CMCnh, GFS40, ECMWF, SREF, NAM12, NAMDNG5 (equal weights)

BCCONSRaw

Consensus of Bias-corrected Raw Model databases

CMCnhBC, GFS40BC, ECMWFBC, SREFBC, NAM12BC, NAMDNG5BC (equal weights)

CONSMOSConsensus of MOS databases

MOSG25, ADJMEX, ADJMEN, ADJECE, ADJECM, EKDMOS, ADJMAV, ADJLAV, ADJMET (equal weights)

BCCONSMOS

Consensus of Bias-corrected MOS databases

MOSG25BC, ADJMEXBC, ADJMENBC, ADJECEBC, ADJECMBC, EKDMOSBC, ADJMAVBC, ADJLAVBC, ADJMETBC (equal weights)

AllBlend50/50 Blend of Official and CONSAll

Official, CONSAll

BCAllBlend50/50 Blend of Official and BCCONSAll

Official, BCCONSAll

RawBlend50/50 Blend of Official and CONSRaw

Official, CONSRaw

BCAllBlend50/50 Blend of Official and BCCONSRaw

Official, BCCONSRaw

Forecast Databases

Name Description Databases

SuperBlend Previous Forecast + latest blends Official (25%), HPCGuide (25%), CONSALL (25%), and BCCONSALL (25%)

CONSAll Consensus Raw Models and MOS CMCnh, GFS40, ECMWF, SREF, NAM12, NAMDNG5, MOSG25, ADJMEX, ADJMEN, ADJECE, ADJECM,, HPCERP (equal weights)

BCCONSAll Consensus of Bias-corrected Raw Model and MOS databases

CMCnhBC, GFS40BC, ECMWFBC, SREFBC, NAM12BC, NAMDNG5BC, MOSG25BC, ADJMEXBC, ADJMENBC, ADJECEBC, ADJECMBC

CONSRaw Consensus of Raw Models CMCnh, GFS40, ECMWF, SREF, NAM12, NAMDNG5 (equal weights)

BCCONSRaw Consensus of Bias-corrected Raw Model databases

CMCnhBC, GFS40BC, ECMWFBC, SREFBC, NAM12BC, NAMDNG5BC (equal weights)

CONSShort CONSRaw databases, Local WRF's and short term MOS

CMCnh, GFS40, ECMWF, SREF, NAM12, NAMDNG5, RUC13BC, HIRESWarwBC, HIRESWnmmBC, BTV4, BTV12, BTV6, ADJMAV, ADJMET, ADJECE, ADJLAV (equal weights)

BCCONSShort Consensus of Bias-corrected CONSShort databases

CMCnhBC, GFS40BC, ECMWFBC, SREFBC, NAM12BC, NAMDNG5BC, RUC13BC, HIRESWarwBC, HIRESWnmmBC,BTV4BC, BTV12BC, BTV6BC, ADJMAVBC, ADJMETBC, ADJECEBC, ADJLAVBC (equal weights)

CONSMOS Consensus of MOS databases MOSG25, ADJMEX, ADJMEN, ADJECE, ADJECM, EKDMOS, ADJMAV, ADJLAV, ADJMET (equal weights)

BCCONSMOS Consensus of Bias-corrected MOS databases

MOSG25BC, ADJMEXBC, ADJMENBC, ADJECEBC, ADJECMBC, EKDMOSBC, ADJMAVBC, ADJLAVBC, ADJMETBC (equal weights)

AllBlend 50/50 Blend of Official and CONSAll Official, CONSAll

BCAllBlend 50/50 Blend of Official and BCCONSAll Official, BCCONSAll

RawBlend 50/50 Blend of Official and CONSRaw Official, CONSRaw

BCAllBlend 50/50 Blend of Official and BCCONSRaw Official, BCCONSRaw

38

Max_T Min_T T Td Wind Speed

-15

-10

-5

0

5

10

15

20

25

30

WFO BOX Days 4-7 % Improvement Over GMOS

OfficialHPCGuideSuperblendCONSALLBCCONSALL

% Im

prov

emen

t ove

r GM

OS

39

Max_T Min_T T Td Wind Speed

-15

-10

-5

0

5

10

15

20

25

30

WFO BTV Days 4-7 % Improvement Over GMOS

OfficialHPCGuideSuperblendCONSALLBCCONSALL

% Im

prov

emen

t ove

r GM

OS

All data: Oct.-Mar.

40

Max_T Min_T T Td Wind Speed

-15

-10

-5

0

5

10

15

20

25

30

WFO CAR Days 4-7 % Improvement Over GMOS

OfficialHPCGuideSuperblendCONSALLBCCONSALL

% Im

prov

emen

t ove

r GM

OS

All data: Mar. 2014Min/Max: Nov.-Jan.

41

Max_T Min_T T Td Wind Speed

-15

-5

5

15

25

35

45

55

WFO GYX Days 4-7 % Improvement Over GMOS

OfficialHPCGuideSuperblendCONSALLBCCONSALL

% Im

prov

emen

t ove

r GM

OS

All data: Jan-Mar 2014.