AN EVALUATION OF THE AN EVALUATION OF THE ETA ETA - - CMAQ AIR QUALITY FORECAST MODEL CMAQ AIR QUALITY FORECAST MODEL AS PART OF NOAA AS PART OF NOAA ’ ’ S NATIONAL PROGRAM S NATIONAL PROGRAM CMAQ CMAQ Brian Brian Eder Eder * * Ken Ken Schere Schere * * Robert Gilliam* Robert Gilliam* Jonathan Jonathan Pleim Pleim * * AIRNOW AIRNOW Atmospheric Sciences Modeling Division Atmospheric Sciences Modeling Division NOAA NOAA – – Air Resources Laboratory Air Resources Laboratory * On assignment to NERL, U.S. EPA * On assignment to NERL, U.S. EPA Daiwen Daiwen Kang Kang UCAR Visiting Scientist UCAR Visiting Scientist U.S. EPA, R.T.P., NC U.S. EPA, R.T.P., NC August 26,2003 August 26,2003 Although this work was reviewed by EPA and approved for publication, it may not necessarily reflect official Agency policy.
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
AN EVALUATION OF THE AN EVALUATION OF THE ETAETA--CMAQ AIR QUALITY FORECAST MODELCMAQ AIR QUALITY FORECAST MODELAS PART OF NOAAAS PART OF NOAA’’S NATIONAL PROGRAMS NATIONAL PROGRAM
CMAQ CMAQ Brian Brian EderEder**Ken Ken SchereSchere**Robert Gilliam*Robert Gilliam*Jonathan Jonathan PleimPleim**
AIRNOWAIRNOW Atmospheric Sciences Modeling DivisionAtmospheric Sciences Modeling Division
NOAA NOAA –– Air Resources LaboratoryAir Resources Laboratory* On assignment to NERL, U.S. EPA* On assignment to NERL, U.S. EPA
DaiwenDaiwen Kang Kang UCAR Visiting ScientistUCAR Visiting ScientistU.S. EPA, R.T.P., NCU.S. EPA, R.T.P., NC
August 26,2003August 26,2003 Although this work was reviewed by EPA and approved for publication, it may not necessarily reflect official Agency policy.
Forecast ConfigurationForecast Configuration
- Eta Meteorology- CBIV Mechanism- SMOKE Emissions (Offline)- 12 km grid resolution - 22 Vertical Layers- 48 Hr. Forecast (12Z Init.)
Simulation PeriodSimulation Period
- 7 July – 30 September, 2003 - 12 – 19 August (Rerun with land-use correction)
Domain
Models-3 CMAQ
This evaluation used:This evaluation used:
Hourly O3 concentrations (ppb) from EPA’s AIRNOW network
521 stations
7 July - 30 September
A suite of statistical metrics for both:
discrete forecasts and categorical forecasts
for the:
hourly, maximum 1-hr, maximum 8-hr O3 simulations
Two Forecast / Evaluation TypesTwo Forecast / Evaluation Types
-- Discrete Forecasts Discrete Forecasts
[Observed] [Observed] versus versus [Forecast][Forecast]
-- Category Forecasts Category Forecasts (Two Category) (Two Category)
Observed Exceedances, NonObserved Exceedances, Non--ExceedancesExceedancesversusversus
Forecast Exceedances, NonForecast Exceedances, Non--ExceedancesExceedances
Discrete Forecast / Evaluation Discrete Forecast / Evaluation
StatisticsStatistics- Summary- Regression
- Biases
- Errors
AIRNOW
MB Model ObsN
N= −∑1
1( )
NMBModel Obs
Obs
N
N=−
⋅∑
∑
( )
( )
1
1
100%
RMSE Model ObsN
N= −
⎛
⎝⎜⎜
⎞
⎠⎟⎟∑1 2
1
0 5
( ).
NMEModel Obs
Obs
N
N=−
⋅∑
∑1
1
100%( )
[Observed] versus [Forecast]
Category Forecast / Evaluation Category Forecast / Evaluation
-- Two Category Forecasts Two Category Forecasts
Observed Exceedances, NonObserved Exceedances, Non--ExceedancesExceedances
versusversus
Forecast Exceedances, NonForecast Exceedances, Non--ExceedancesExceedances
a b
c d
Fore
cast
Exc
eeda
nce
No
Yes
No YesObserved Exceedance
a ba b
c dc d
Category ForecastCategory ForecastAccuracyPercent of forecasts that correctly predict event or non-event.
Bias Indicates if forecasts are under-predicted (false negatives) or over-predicted (false positives)
False Alarm Rate
Percent of times a forecast of high ozone did not occur
Ab c
a b c d%=
++ + +
⎛⎝⎜
⎞⎠⎟ ⋅100
Ba bb d
=++
⎛⎝⎜
⎞⎠⎟
a ba b
c dc d
FARa
a b%=
+⎛⎝⎜
⎞⎠⎟ ⋅100
Critical Success Index
How well the high ozone events were predicted.
Probability Of Detection
Ability to predict high ozone events
CSIb
a b d=
+ +⎛⎝⎜
⎞⎠⎟ ⋅100%
Category ForecastCategory Forecasta ba b
c dc d
PODb
b d%=
+⎛⎝⎜
⎞⎠⎟ ⋅
100
a
c
a ba b
c d c d a= 154b= 1c= 36,023d= 4n= 36,182
CMAQ = 34.5 + 0.63(AIRNOW)
Max 1Max 1--hr Ohr O33
7 July – 30 September
SummaryStatistics
Discrete Evaluation
Categorical Evaluation
CMAQ = 34.5 + 0.63 (AIRNOW)
Ozone 125 ppb
A 99.6%
B
FARCSI
POD
r
N 26.0
BIASES
MB 99.4%0.6%
16.7%
NMB
ERRORS
RMSE
NME
[ppb] CMAQ AIRNOW
Mean 68.1 53.1 0.62
SD 17.3 16.7 36,814
15.028.2%
21.1
32.2%
CV 25.3 31.5Max 182.9 132
95th 99.2 8175th 78.7 65
50th 66.0 53
25th 55.6 41
5th 44.2 27
Min 0 1
Max 1Max 1-- hr Ohr O33
Spatial Evaluation Spatial Evaluation
Max 1Max 1-- hr Ohr O33CorrelationCorrelation
-92 -90 -88 -86 -84 -82 -80 -78 -76 -74 -72 -70 -68
34
36
38
40
42
44
46
48
0 to 0.25 0.25 to 0.5 0.5 to 0.75 0.75 to 1
0.00 – 0.250.25 – 0.500.50 – 0.750.75 – 1.00
Mean = 0.62
-92 -90 -88 -86 -84 -82 -80 -78 -76 -74 -72 -70 -68
34
36
38
40
42
44
46
48
-10 to 10 10 to 20 20 to 30 30 to 40 40 to 50
Max 1Max 1-- hr Ohr O33Mean BiasMean Bias
Spatial Evaluation Spatial Evaluation
-10 – 1010 - 2020 - 3030 – 4040 – 50
Mean = 15.0
-92 -90 -88 -86 -84 -82 -80 -78 -76 -74 -72 -70 -68
34
36
38
40
42
44
46
48
0 to 10 10 to 20 20 to 30 30 to 40 40 to 50
Spatial Evaluation Spatial Evaluation
Max 1Max 1-- hr Ohr O33Root Mean Square ErrorRoot Mean Square Error
0 – 1010 - 2020 - 3030 – 4040 – 50
Mean = 21.1
CMAQ = 35.1 + 0.62(AIRNOW)
a ba b
c d c d a= 3537b= 152c= 33,242d= 67n= 36,998
Max 8Max 8--hr Ohr O33
SummaryStatistics
Discrete Evaluation
Categorical Evaluation
CMAQ = 35.1 + 0.62 (AIRNOW)
Ozone 85 ppb
A 89.6%
B
FARCSI
POD
r
n 10.3
BIASES
MB 96.0%3.7%
41.0%
NMB
ERRORS
RMSE
NME
[ppb] CMAQ AIRNOW
Mean 64.0 46.7 0.59
SD 15.8 15.0 36,998
17.437.3%
22.2
39.9%
CV 24.6% 32.2%
Max 162.2 108.4
95th 92.1 71.675th 73.9 57.2
50th 62.2 46.1
25th 52.6 35.7
5th 42.1 23.3
Min 0 1
Max 8Max 8-- hr Ohr O33
Spatial Evaluation Spatial Evaluation
Max 8Max 8-- hr Ohr O33CorrelationCorrelation
-92 -90 -88 -86 -84 -82 -80 -78 -76 -74 -72 -70 -68
34
36
38
40
42
44
46
48
-0.1 to 0.25 0.25 to 0.5 0.5 to 0.75 0.75 to 1
0.00 – 0.250.25 – 0.500.50 – 0.750.75 – 1.00
Mean = 0.59
Spatial Evaluation Spatial Evaluation
Max 8Max 8-- hr Ohr O33Mean BiasMean Bias
-92 -90 -88 -86 -84 -82 -80 -78 -76 -74 -72 -70 -68
34
36
38
40
42
44
46
48
-10 to 10 10 to 20 20 to 30 30 to 40 40 to 50
-10 – 1010 - 2020 - 3030 – 4040 – 50
Mean = 17.4
-92 -90 -88 -86 -84 -82 -80 -78 -76 -74 -72 -70 -68
34
36
38
40
42
44
46
48
0 to 10 10 to 20 20 to 30 30 to 40 40 to 50
Spatial Evaluation Spatial Evaluation
MaxMax--8 hr O8 hr O33Root Mean Square ErrorRoot Mean Square Error
0 – 1010 - 2020 - 3030 – 4040 – 50
Mean = 22.2
LandLand--Use ErrorUse ErrorApproximately two months into the forecast period, AMDB discoverApproximately two months into the forecast period, AMDB discovered ed the the landthe the land--use fields associated with Eta were being postuse fields associated with Eta were being post--processed processed incorrectly by NCEP. As a resultincorrectly by NCEP. As a result::
-- Most of the domain was classified as water.Most of the domain was classified as water.-- Dry deposition was greatly under simulatedDry deposition was greatly under simulated-- Concentrations were over predictedConcentrations were over predicted
This error was corrected on Sept. 9This error was corrected on Sept. 9thth..
-- An eight day period (12An eight day period (12--19 August) was re19 August) was re--simulated.simulated.-- Positive biases were cut in half, errors reduced also.Positive biases were cut in half, errors reduced also.
Erroneous Corrected
Temporal EvaluationTemporal Evaluation
–– Max 1 hr OMax 1 hr O33
Land-use Correction
Run rMB
(ppb)NMB(%)
RMSE(ppb)
NME(%)
A(%)
B FAR(%)
- 100.0
100.0-
CSI(%)
POD(%)
Initial 0.64 16.2 27.5 23.0 31.7 99.0 0.0 -
Corrected 0.66 7.6 13.0 16.6 21.7 99.6 0.0 -
Max 1-hr O3
Max 8-hr O3
Comparison BetweenInitial and Corrected Simulations
August 12 –19 2003
Run rMB
(ppb)NMB(%)
RMSE(ppb)
NME(%)
A(%)
B FAR(%)
- 100.0
92.03.5
CSI(%)
POD(%)
Initial 0.62 19.2 37.2 24.6 39.9 76.2 0.0 -
Corrected 0.64 10.4 20.1 17.1 26.3 90.7 6.6 28.0
SummarySummaryThe Eta-CMAQ modeling system performed reasonably well, in this, its first attempt at forecasting ozone concentrations:
Correlation: 0.59 - 0.62Bias: 15.1 ppb (28.2%) - 17.4 ppb (37.3%)
Error: 21.1 ppb (32.2%) - 22.2 ppb (39.9%)Accuracy: 89.6 - 99.6%
An error was discovered in Eta’s post processed land-use designation that resulted in the:
– under-estimation of dry deposition and – hence over-simulation of concentrations
Once corrected, the positive biases and errors were greatly reduced when the model was re-run for an eight day period:
Correlation: 0.64 - 0.66Bias: 7.6 ppb (13.0%) - 10.4 ppb (20.1%)
Error: 16.6 ppb (21.7%) - 17.1 ppb (26.3%)Accuracy: 90.7 - 99.6%
Related Documents
