Validation of short and medium term operational solar radiation forecasts in the US Richard Perez a,⇑ , Sergey Kivalov a , James Schlemmer a , Karl Hemker Jr. a , David Renne ´ b , Thomas E. Hoff c a ASRC, University at Albany, Albany, New York, United States b National Renewable Energy Laboratory, Golden, Colorado, United States c Clean Power Research, Napa, California, United States Received 4 May 2010; received in revised form 23 August 2010; accepted 25 August 2010 Available online 8 October 2010 Communicated by: Associate Editor Frank Vignola Abstract This paper presents a validation of the short and medium term global irradiance forecasts that are produced as part of the US Solar- Anywhere (2010) data set. The short term forecasts that extend up to 6-h ahead are based upon cloud motion derived from consecutive geostationary satellite images. The medium term forecasts extend up to 6-days-ahead and are modeled from gridded cloud cover fore- casts from the US National Digital Forecast Database. The forecast algorithms are validated against ground measurements for seven climatically distinct locations in the United States for 1 year. An initia l analys is of regional perfo rmanc e using satellite -deriv ed irradia nces as a bench mark reference is also presented. Ó 2010 Elsevier Ltd. All rights reserved. Keywords: Solar resource assessment; Irradiance; Forecast; Prediction; Valida tion 1. Introduction There ar e two ba sic appr oa ches to solar ra diat ion forecasting. The first approach consists of numerical weather predic- tion (NWP) models. NWP models can be global – e.g., GFS Model (2003), ECMWF (2010) – regional or local –e.g., WRF (20 10) . For irra dia nce pre dictions, the NWP for e- casts are inherently probabilistic because they infer local cloud formation probability (and indire ctly transmi tted radiation) through dynamic modeling of the atmosphere. NWP models cannot, at this stage of their deve lopme nt, pr e- dict the exact position and extent of individual clouds or cloud fields affecting a given location’s solar resource. The second app roa ch con sists of pro ject ing obs erv ed solar radiation conditions based on immediate measured hi st or y: The posi tio n and impact of future clo uds is infe rr ed from their moti on de te rmined from recent observatio ns; these observatio ns can be eith er remote (from satellites) or from appropriate ground based sky- imaging instru mentation (e.g., Slater et al., 2001). This appr oach is initial ly de terminist ic becaus e the initi al pos ition of clouds affectin g a solar installation is pre - cisely known. Lorenz et al. (2007) have shown that the cloud motion- based forecasts tend to provide better results than NWP forecasts up to forecast horizons of 3–4 h, beyond which NWP models perform better. In this article, we evaluate the NWP-based and satellite- derived cloud motion forecast models producing the Solar- Any where’s hou rly global hor izonta l irradiance (GHI ) forecasts. 0038-092X/$ - see front matter Ó 2010 Elsevier Ltd. All rights reserved. doi:10.1016/j.solener.2010.08.014 ⇑ Cor resp ondi ng auth or. Address: Atmospheri c Scie nces Res ear ch Center, The University at Albany, 251 Fuller Rd., Albany, NY 12203, USA. Tel.: +1 5184378751; fax: +1 5184378711. E-mail address: [email protected](R. Perez). www.elsevier.com/locate/solener Available online at www.science direct.com Solar Energy 84 (2010) 2161–2172
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Validation of short and medium term operational solarradiation forecasts in the US
Richard Perez a,⇑, Sergey Kivalov a, James Schlemmer a, Karl Hemker Jr. a,David Renne b, Thomas E. Hoff c
a ASRC, University at Albany, Albany, New York, United Statesb National Renewable Energy Laboratory, Golden, Colorado, United States
c Clean Power Research, Napa, California, United States
Received 4 May 2010; received in revised form 23 August 2010; accepted 25 August 2010Available online 8 October 2010
Communicated by: Associate Editor Frank Vignola
Abstract
This paper presents a validation of the short and medium term global irradiance forecasts that are produced as part of the US Solar-Anywhere (2010) data set. The short term forecasts that extend up to 6-h ahead are based upon cloud motion derived from consecutivegeostationary satellite images. The medium term forecasts extend up to 6-days-ahead and are modeled from gridded cloud cover fore-casts from the US National Digital Forecast Database.
The forecast algorithms are validated against ground measurements for seven climatically distinct locations in the United States for1 year. An initial analysis of regional performance using satellite-derived irradiances as a benchmark reference is also presented.Ó 2010 Elsevier Ltd. All rights reserved.
Keywords: Solar resource assessment; Irradiance; Forecast; Prediction; Validation
1. Introduction
There are two basic approaches to solar radiationforecasting.
The first approach consists of numerical weather predic-tion (NWP) models. NWP models can be global – e.g.,GFS Model (2003), ECMWF (2010) – regional or local – e.g., WRF (2010). For irradiance predictions, the NWP fore-casts are inherently probabilistic because they infer localcloud formation probability (and indirectly transmittedradiation) through dynamic modeling of the atmosphere.NWP models cannot, at this stage of their development, pre-dict the exact position and extent of individual clouds orcloud fields affecting a given location’s solar resource.
The second approach consists of projecting observedsolar radiation conditions based on immediate measuredhistory: The position and impact of future clouds isinferred from their motion determined from recentobservations; these observations can be either remote(from satellites) or from appropriate ground based sky-imaging instrumentation (e.g., Slater et al., 2001). Thisapproach is initially deterministic because the initialposition of clouds affecting a solar installation is pre-cisely known.
Lorenz et al. (2007) have shown that the cloud motion-based forecasts tend to provide better results than NWPforecasts up to forecast horizons of 3–4 h, beyond whichNWP models perform better.
In this article, we evaluate the NWP-based and satellite-derived cloud motion forecast models producing the Solar-Anywhere’s hourly global horizontal irradiance (GHI)forecasts.
0038-092X/$ - see front matter Ó 2010 Elsevier Ltd. All rights reserved.
doi:10.1016/j.solener.2010.08.014
⇑ Corresponding author. Address: Atmospheric Sciences ResearchCenter, The University at Albany, 251 Fuller Rd., Albany, NY 12203,USA. Tel.: +1 5184378751; fax: +1 5184378711.
Forecasts are validated against single point ground-truth stations. In addition, the ability of forecast modelsto account for local microclimatology is investigated byobserving the distribution of mean predictions overextended areas.
3.1. Single point ground-truth validation
Hourly forecasts are tested against irradiance data fromeach station of the SURFRAD network (SURFRAD,
2010) including Desert Rock, Nevada; Fort Peck, Mon-tana; Boulder, Colorado; Sioux Falls, South Dakota;Bondville, Illinois; Goodwin Creek, Mississippi; and PennState, Pennsylvania.
These stations cover several distinct climatic environ-ments ranging from arid (Desert Rock) to humid continen-tal locations (Penn State) and from locations with somesubtropical influence (Goodwin Creek) to the northernGreat Plains (Fort Peck). Boulder is a challenging sitefor all types of solar radiation models, because of its highelevation ($2000 m) and of its position at the RockyMountains’ eastern edge, at the junction between two
weather regimes.
The present validation period spans a little over 1 yearfrom August 23, 2008 to August 31, 2009.
3.1.1. Validation metrics
In addition to the standard Mean Bias and Root MeanSquare Errors (respectively RMSE and MBE) resultingfrom the direct comparison between hourly forecasts andhourly measurements, we also consider two metrics thatquantify the ability of a model to reproduce observed fre-quency distributions. The first of these is the Kolmogo-rov–Smirnov test Integral (KSI) goodness of fit test(Espinar et al., 2009) recommended by the International
Energy Agency Solar Heating and Cooling ProgrammeTask 36 for data benchmarking (Task 36, report, 2010).The KSI metric is obtained by integrating the absolute dif-ference between the measured and modeled cumulative fre-quency distributions of the considered variable asillustrated in Fig. 1. The second, termed OVER, is calcu-lated by integrating the absolute difference between themeasured distribution and the measured distribution plusor minus a buffer determined by the Kolmogorov–Smirnovcritical V
capproximated here by (NIST, 2010):
V c ¼ 1:63= ffiffiffi
np
where n is the number of consider data points.
Table 1 (continued )
MBE Desert Rock Fort Peck Boulder Sioux Falls Bondville Gdwn Creek Penn State
Both the KSI and OVER metrics are quantified here asfractions of the critical value.
3.1.2. Persistence benchmarking
The single site performance of the forecast models isevaluated by comparing it to measured persistence; sameday measured persistence is obtained by time extrapolationof measured irradiances using a constant KtÃindex. Nextday and multi-day persistences are obtained by extrapolat-ing the previous day’s mean daily measured KtÃ.
3.1.3. Results
Tables 1 and 2 report respectively the MBEs and RMSEsobserved at all sites for all time horizons for the forecast mod-els andpersistencebenchmarks. Results are reportedyearly aswell as seasonally. Forecasts include 1–6 h cloud motion fore-casts and same day to 6-days-ahead NDFD forecasts. Thesatellite model’s MBEs and RMSEs are also included inTables 1 and 2 as an additional performance reference.
All forecasts are validated against the same set of exper-imental values. Hence, because 6-h cloud motion forecastscannot be generated until the sun is up, the experimental“common validation denominator pool” is limited topoints 6 h after sunrise.
Fig. 2 summarizes the results of Table 2, plotting the
yearly RMSE trend for all sites and all models as a func-
tion of the forecast time horizon. The satellite model’sRMSE is included as a reference and appears as a horizon-tal line across all forecasts horizons.
Fig. 3 provides an illustrative sample of measured vs.model scatter plots at four of the seven sites, includingBondville, Boulder, Desert Rock and Goodwin Creek. Thissample includes the reference satellite model, the 1 and 3-hcloud-motion forecasts, the next day and 3-days-aheadNDFD forecasts, as well as the same time horizons forthe measured persistence benchmarks.
Tables 3 and 4 report the annual KSI and OVER Statis-tics for all sites, forecast time horizons, satellite reference,and persistence benchmarks.
3.1.4. Discussion
The NDFD–NWP forecasts analyzed here lead toresults which are largely consistent with initial evaluations(Perez et al., 2007; Remund et al., 2008). This is an impor-tant result, because these initial evaluations only covered alimited period spanning summer months only.
When considering the RMSE metric, it is remarkable toobserve that the performance of the 1 and 2-h cloudmotion forecasts is comparable to that of the satellitemodel from which they are extracted. The 1-h forecastsactually have a lower RMSE than the satellite model at
all sites but Boulder: despite the loss of deterministic infor-
Table 2 (continued )
RMSE Desert Rock Fort Peck Boulder Sioux Falls Bondville Gdwn Creek Penn State
mation due to cloud motion, the image smoothing inherentto the forecasts – via convergence and divergence of motionvectors, and additional post-processing pixel averaging –
results in lowering the RMSE that quantify short termaccuracy. Hence, in effect, lowering the resolution of thesatellite model increases its short term accuracy. A corol-lary of this is that attempting to achieve better short termaccuracy for satellite models by increasing spatial resolu-tion might be illusory given the satellite navigation errorsand parallax uncertainties (cloud shadowing, sun/satelliteangles) if these uncertainties are not specifically addressedvia more complex models. Note that this pixel-averagingperformance improvement technique is known and hasbeen previously discussed, e.g., by Stuhlmann et al.(1990) when developing his physical satellite-to-irradiance
model.
Cloud motion forecasts are always better than persis-tence forecasts derived from actual measurements, evenafter a little as 1 h.
The break-even point between cloud motion and NDFDforecasts is between 5 and 6 h ahead. We note, however,that satellite-aided multiple output statistics (MOS) real-time feedback (e.g., see Dennstaedt, 2006), whereby inthe present case the numerical weather forecasts would becorrected from the most recent satellite-derived irradiancehistory, could improve the NDFD forecasts. Such a feed-back process has not been implemented here.
The cloud motion forecasts’ MBE is consistently small,to the exception of sites experiencing important wintersnow cover where the accuracy of the current satellitemodel, relying solely on the visible channel, is limited – a
new model proposed by the authors and utilizing the infra-
cloud cover (Western US and Great Plains). The eastern-most sites, Penn State and Goodwin Creek, where localizedcloud formation is more frequent, exhibit a tendency to neg-ative bias. The seasonal pattern shows a tendency of NDFDforecasts towards positive irradiance bias in the fall (cloudi-ness underprediction) and negative bias in the other seasons,particularly in the spring (cloudiness overprediction).Despite these shortcomings, the NDFD forecast performconsiderably better than persistence up to 6 days-ahead.
When considering KSI and OVER statistics it is not sur-prising to observe that the persistence-based forecasts tendto score better than the forecast models. For the shortterm, same day forecasts, the statistical distribution of per-sistence forecasts should be almost identical to measure-ments’ (since they are measurements themselves, albeittime-extrapolated) hence exceed the performance of thesatellite model and of its derived forecasts. One exceptionis Boulder, CO, where the very marked diurnal patternsat that site produce different statistical distributions for dif-ferent times of day and where the cloud motion provides
better results. The 1–6-days-ahead persistence forecast also
exhibit a better performance than the NDFD whenassessed via the KSI and OVER metrics. As for cloudmotion, the NDFD performance statistics deteriorate sen-sibly with the time horizon, reflecting a loss of dynamicrange for both. This is due to pixel convergence/averagingin the case of cloud motion, and likely due to the tendencyavoid extreme forecasts (clear or cloudy) as the time hori-zon increases for the NDFD models.
Fortunately, the KSI and OVER metrics are largely rel-evant to microclimatological site-characterization and arenot critical for forecast operations where the short termaccuracy (RMSE) is the key performance factor.
3.2. Extended-area validations
This validation is largely qualitative and focuses on theability of the forecast models to account for the solarresource’s microclimatic features over a given period. Thevalidation criterion is a visual evaluation of mapped solarresource computed from ongoing forecast data. Because
we do not dispose of gridded instrumentation spanning
Table 3Annual KSI metric summary.
KSI Desert Rock Fort Peck Boulder Sioux Falls Bondville Gdwn Creek Penn State All sites
the considered areas, we rely on satellite-derived irradi-ances data as a performance benchmark.
We considered 2° Â 2° regions ($30,000 sq. km) sur-rounding each ground-truth station. The results are illus-trated by presenting the case of Boulder and Desert Rock,which have the strongest (orography-driven) microclimatic
features. Fig. 4 compares the mapped irradiances for DesertRock in summer and for Boulder in the fall, spring and year-around. The maps consistof the satellite model, the 1 and 3-hcloud motion forecasts and the next day, 3 days, and 6-days-ahead forecasts. The orographic features that influence solarresource microclimates may be seen in Fig. 5.
The NDFD model does account for orography-drivenmicroclimates, but, apparently, only when cloudinessincreases with elevation. This underlying assumption holdsin Desert Rock in summer and in the spring time in Boul-der. However in the fall of 2008 clouds preferentiallyformed immediately east of the Rocky Mountains likely
linked to the presence of easterly winds“
upslope”
cloudformation. This preferential cloud formation trend wasnot taken in account by the NDFD models.
The smoothing effect of the cloud motion tends to erasesome of the terrain features (pixel convergence andaveraging).
Fig. 4. Long-term average GHI from in a 2° Â 2° region surrounding the sites of Boulder and Desert Rock, for the satellite model, cloud motion forecasts(1 and 3 h ahead) and NDFD forecasts (1, 3 and 6-days-ahead). Note that maps contain the same number of points for all models (i.e., slightly biased
towards afternoon conditions to accommodate the fact that 3-h cloud motion forecasts cannot be made for the first 3 h of each day).
2170 R. Perez et al. / Solar Energy 84 (2010) 2161–2172
Finally, Fig. 4 also shows the discontinuities inherent tothe NDFD process, whereby global forecasts are modifiedindependently by regional offices before being reassembledon the NDFD grid. The discontinuity at the top of the Boul-der maps (appearing as a horizontal discontinuity) marksthe boundary between two US National Weather Serviceoffices which produce a different assessment of local cloud-iness that becomes apparent over integrated time scales.
4. Conclusions
The numerical weather prediction-based irradiance fore-cast analyzed here lead to results which are consistent withour previous limited evaluations. The present validationsinclude a more diverse set of climatic environments andinclude winter months when models performance tends tobe poorer then in summer.
All the considered short-term and long-term forecastsperform significantly better than persistence from actualhigh accuracy measurements. Satellite-derived cloud
motion-based forecasting leads to a significant improve-ment over NDFD forecasts up to 5 h ahead. One-hourforecasts are on par or slightly better than the satellitemodel from which they are derived; the probable reasonis that the cloud motion methodology results in a smooth-ing of the predicted images which tends to mitigate satel-lite’s navigation and parallax uncertainties. A corollary of this maybe that the short term accuracy of satellite modelsmay not be improved significantly by increased image res-olution – of course this comment applies only to short termdata and does not pertain to long-term means and thedelineation of solar microclimates, where high resolution
would be beneficial.
Acknowledgements
The forecast modeling capability was developed as partof the construction of SolarAnywhereÒ under fundingfrom Clean Power Research. The present validation analy-sis performed under funding from NREL (ContractAEK98833801). The first author of this paper and his teamat the University at Albany receives funding from Clean
Power Research to develop and produce the Solar Any-where solar resource satellite and forecast data evaluatedin this paper.
References
Dennstaedt, S., 2006. Model output statistics provide essential data forsmall airports. The Front 6 (2), 1–4.
ECMWF, 2010. European Centre for Medium-Range Weather Forecasts.<http://www.ecmwf.int/>.
Espinar, B., Ramırez, L., Drews, A., Beyer, H.G., Zarzalejo, L.F., Polo,J., Martın, L., 2009. Analysis of different error parameters applied tosolar radiation data from satellite and German radiometric stations.
Solar Energy 83 (1).The GFS Model, 2003. Environmental Modeling Center, NOAA, Washing-
ton, DC. <http://www.emc.ncep.noaa.gov/gmb/moorthi/gam.html>.IEA-SHCP Task 36, 2010. Final Report. Subtask A – Standard
Qualification for Solar Resource Products.Lorenz, E., Heinemann, D., Wickramarathne, H., Beyer, H.G., Bofinger,
S., 2007. Forecast of ensemble power production by grid-connected PVsystems. In: Proc. 20th European PV Conference, Milano, Italy.
The National Forecast Database (NDFD), 2010. National WeatherService. NOAA, Washington, DC. <http://www.weather.gov/ndfd/>.
National Institute of Standards and Technology (NIST), 2010. Engineer-ing and Statistics Handbook. <www.itl.nist.gov>.
Perez, R., Ineichen, P., Moore, K., Kmiecik, M., Chain, C., George, R.,Vignola, F., 2002. A new operational satellite-to-irradiance model.Solar Energy 73 (5), 307–317.
Fig. 5. Illustration of the orographic features in the regions analyzed in Fig. 4.
R. Perez et al. / Solar Energy 84 (2010) 2161–2172 2171
Perez, R., Ineichen, P., Kmiecik, M., Moore, K., George, R., Renne, D.,2004. Producing satellite-derived irradiances in complex arid terrain.Solar Energy 77 (4), 363–370.
Perez, R., Moore, K., Wilcox, S., Renne, D., Zelenka, A., 2007.Forecasting solar radiation – preliminary evaluation of an approachbased upon the national forecast data base. Solar Energy 81 (6), 809– 812.
Perez, R., Kivalov, S., Zelenka, A., Schlemmer, J., Hemker Jr., K., 2010.Improving the performance of satellite-to-irradiance models using thesatellite’s infrared sensors. In: Proc. of American Solar EnergySociety’s Annual Conference, Phoenix, AZ.
Remund, J., Perez, R., Lorenz, E., 2008. Comparison of solar radiationforecasts for the USA. In: Proc. of 23rd European PV Conference,Valencia, Spain.
Slater, D.W., Long, C.N., Tooman, T.P., 2001: Total sky imager/wholesky imager cloud fraction comparison. In: Eleventh ARM ScienceTeam Meeting Proceedings, Atlanta, Georgia, March 19–23, 2001.
SolarAnywhere, 2010. Clean Power Research. <www.solaranywhere.com>.
Stuhlmann, R., Rieland, M., Raschke, E., 1990. An improvement of theIGMK model to derive total and diffuse solar radiation at the surfacefrom satellite data. Journal of Applied Meteorology 29, 586–603.
The SURFRAD Network, 2010. Monitoring Surface Radiation in theContinental United States. National Weather Service, NOAA, Wash-ington, DC.
WRF, 2010. Weather Research and Forecasting (WRF) Model. <http://www.wrf-model.org>.
Xu, K.M., Randall, D.A., 1996. A semiempirical cloudiness parameter-ization for use in climate models. Journal of the Atmospheric Science53, 3084–3102.
Glossary
ECMWF: European Center for Medium-Range Weather Forecasts
GFS: NOAA’S Global Forecasting SystemGHI: global horizontal irradianceGHI clear: Clear Sky Global Horizontal Irradiance
KSI: Kolmogorov–Smirnov test integralKtÃ: GHI index equal to GHI/GHIclearMBE: Mean Bias ErrorMOS: Model Output Statistics
NDFD: National Digital Forecast Data BaseNOAA: National Oceanic and Atmospheric Administration
NWP: numerical weather predictionOVER: that part of the KSI which integrates above (over) the critical
value V c
RMSE: Root Mean Square Error
V c : Kolmogorov–Smirnov critical valueWRF: Weather Research Forecasting model
2172 R. Perez et al. / Solar Energy 84 (2010) 2161–2172