Economy-Wide Modeling: Uncertainty, Verification, and Validation June 17, 2016 Prepared for the U.S. EPA Science Advisory Board Panel (SAB) on Economy-Wide Modeling of the Benefits and Costs of Environmental Regulation This paper has been developed to inform the deliberations of the SAB Panel on the technical merits and challenges of economy-wide modeling for an air regulation. It is not an official EPA report nor does it necessarily represent the official policies or views of the U.S. EPA.
35
Embed
Economy-Wide Modeling: Uncertainty, Verification, and ... · Economy-Wide Modeling: Uncertainty, Verification ... 2.2 Uncertainty ... These levels of accuracy also correspond to those
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Benefitcostandeconomicimpactanalysesofnationalregulationsdesignedtoimproveairqualityrequireinformationfromnumerousscientificdisciplinesandsourcestobelinkedandassessed.Eachcomponentoftheseanalyses(e.g.,engineeringassessment,airqualitymodeling,riskassessment,economicanalysis)maybesubjecttouncertaintyandthe linkagesbetweenthesecomponentsmayacttocompoundtheuncertainty. A recentNational Academy of Sciences study onmaking Environmental DecisionsUnderUncertaintynoted that“informed identificationanduseofuncertainties in theprocess isanessentialfeatureofenvironmentaldecisionmaking”(NAS,2013).In2002,theNASalsonotedthat:
“Evengreatuncertaintydoesnotimplythatactiontopromoteorprotectpublichealthshouldbedelayed.Decisionsaboutwhethertoact,whentoact,andhowaggressivelytoactcanonlybemadewith some understanding of the likelihood and consequences of alternative courses ofaction. The potential for improving decisions through researchmust be balanced against thepublic health costs incurred because of a delay in the implementation of controls. Completecertaintyisanunattainableideal.”
TheNASCommitteein2013classifieduncertaintiesaccordingtothreetypes:(1)statisticalvariabilityandheterogeneity,(2)deepuncertainty,and(3)modelandparameteruncertainty.Statisticalvariabilityandheterogeneityreferstoexogenoussourcesofuncertaintyinherentinthesystemorprocessunderstudy.Thesearesourcesofuncertaintythat,exante,cannotbereducedthroughadditionalresearchordatacollection,forexamplestochasticityindynamicprocesseswhosefundamentalsarewellunderstood.Deepuncertaintyreferstocaseswherethereisalackofunderstanding,ornotabledisagreement,aboutthefundamentalsofunderlyingprocessesimportanttounderstandingtheimpactofaninterventioninthesystemofinterest.Thesesourcesofuncertaintyarecharacterizedbycaseswhereadditionalresearchordatacollectionareunlikelytoreducetheuncertaintyinthetimebeforeapolicymakerneedstoselectacourse of action.Model and parameter uncertainty refers to uncertainty inmodels that are used torepresentunderlyingprocessesrelevanttounderstandingtheimpactofaninterventioninthesystemofinterest,whereamodelisdefinedasa“simplificationofrealitythatisconstructedtogaininsightsintoselectattributesofaparticularphysical,biologic,economic,orsocialsystem”(NRC,2009).Thesesourcesofuncertaintyariseduetocurrentlimitationsofscientificandeconomicprocesses,nowandinthefuture,andintheorycouldbereducedbyadditionalresearchpriortomakingadecision.
While all these sources of uncertaintymay be relevant for decisionmaking regarding environmentalregulations designed to improve air quality, this paper focuses primarily on model and parameteruncertainty related to benefit cost analysis and the potential use of CGE models for economy-wideanalysis.1Thiswhitepaperintroducessomeofthekeysourcesofuncertaintyassociatedwithbenefitcost
intheeconomy.Whileoutsidethescopeofthispaper,dynamicstochasticgeneralequilibrium(DSGE)modelscan capture some aspects of decisionmaking under uncertainty by economic agents. For examples ofDSGEmodels applied to environmental issues see the white paper on Economy-Wide Modeling: Evaluating theEconomicImpactsofAirRegulations.
5
analysesofair regulationsandadditionalsourcesofuncertainty thatmayarisewhenextending thoseanalysestoincludeaneconomy-wideassessmentusingCGEmodels.Giventhesesourcesofuncertainty,thiswhitepaperdiscussesanumberofanalyticalapproachesthatmaybeusedtoprovidedecisionmakersandstakeholderswithadditionalinformationabouttherobustnessofmodelresults.Theseapproachesandothermethodsofmodelvalidationandverificationcanhelpprovideconfidence inthequalitativeconclusions of policy analysis based onmodeling results.While the focus of the discussion is on theestimationofbenefitsandcosts,thesamesourcesofuncertaintycanalsoaffectestimatesofeconomicimpacts.
Specifically, the white paper begins with a brief description on uncertainties inherent in traditionalengineering-basedcostassessmentsfollowedbyuncertaintiesthatmaybeassociatedwithincorporatingthis information in a CGEmodel. Subsequently, the paper provides a brief discussion of some of theuncertainties associatedwith traditional benefits assessment for regulations designed to improve airquality.ThewhitepaperthenconsiderssourcesofuncertaintyassociatedwithCGEmodeling.Followingtheintroductorydiscussionsonsourcesofuncertainty,thepaperconsidersapproachesforquantitativelyassessingtheimpactofuncertaintyontheresultsandmethodsforpresentingthoseresultsinamannerthatappropriatelyconveys the informationrelevant fordecisionmakersandstakeholders.Finally, thepaper considers approaches of model validation and verification for CGE models to further increaseconfidenceinmodelingresultsandreducepotentialsourcesofuncertainty.
Entities affected by a regulationmay incur compliance costs in the process ofmitigating pollution tocomplywith the regulation. The largest components of compliance costs are typically the capital andoperating costs associated with pollution control equipment. Capital costs are often one-timeexpenditures related to the installation or retrofit of structures or equipment to reduce emissions.Operatingcostsarerecurringannualexpendituresassociatedwiththeoperationandmaintenanceofthepollutioncontrolequipmentandwilloftenincludemonitoring,reportingandrecordkeepingexpenditures.Administrativeandenforcementcostsmayalsoaccruetolocal,state,andfederalregulatoryagencies.
Analyststypicallyestimatecompliancecostsexpectedtobeincurredbyregulatedentitiesbeforetheruleis implemented. In many cases, these ex ante estimates of compliance costs are estimated by cost
6
engineersandpolicyanalyststofulfillstatutoryobligationsoutsideofbenefitcostanalysisorpurposesofevaluating the national economic impacts of the regulation. As has been discussed by the NationalAcademies of Science (NAS, 2013), these ex ante estimates of compliance costs are often based onengineeringmodels inwhich there is uncertainty over facility characteristics, the number of affectedfacilities,andthedegreeofregulatorycompliance.Thisuncertaintyoftenleadstocasesinwhichtheexanteestimatesofcompliancecostsdifferfromtheexpost(realized)compliancecosts.
SinceEPAcompliancecostestimationisprimarilyperformedbycostanalystsandengineers,itisusefultoreview the framework developed by those professions to articulate the accuracy of engineering costestimates.Forexample,theEPAAirPollutionControlCostManual(2002)isdescribedasacomprehensivesetof“proceduresanddataforsizingandcostingcontrolequipment”forVOCs,PM,SO2,NOx,andsomeacidgases.ThismanualaswellastheAACEInternationalCostEstimatingClassificationSystem2havebeenused to provide context for the uncertainty associated with pollution control cost estimates forcompliancewithNationalAmbientAirQualityStandards(NAAQS).
Inthe2015FinalOzoneNAAQSRegulatoryImpactAnalysis(RIA),forexample,EPAstatedthatthereisarangeof ± 30percent for non-electrical generatingunit point source control costs, citing the EPAAirPollutionControlCostManual(2002).ThislevelofaccuracyisdescribedintheEPAAirPollutionControlCostManualasa“studyestimate.”AccordingtotheManual,thestudyestimateiswellsuitedforuseinregulatory development because it does not require detailed site-specific information necessary forindustrylevelanalyses.Theyalsocanbepreparedatarelativelylowcostwithrelativelyminimaldata.WhileinformationthatismoredetailedmaybeavailabletotheEPA,itisoftenproprietaryandconsideredconfidentialbusinessinformation.
In addition to the studyestimate, theEPAAirPollutionControlCostManualdiscussesother typesofestimates:
These levels of accuracy also correspond to those reported in Perry’s Chemical Engineers’Handbook,anotherimportantreferenceforcostsengineersandanalysts.
ThetypeofexantecompliancecostestimatecalculatedbyEPApriortoarulemakingwillberegulationspecific,giventheheterogeneity inavailable informationandscopeofregulations.Forexample,somerulemakingsmayaffectrelativelyfewfacilities,anddetailedinformationonfacilitycharacteristicsmaybeavailablepublicallyorthroughformalInformationCollectionRequests.Inothercases,regulationmayonlyaffectnewfacilitieswhereheterogeneityinfacilitycharacteristicsthatwouldaffectretrofitcostsarenota confounding factor. Compliance cost estimates for these actions may be more precise than forregulationslikeaNAAQS,wheretheillustrativecontrolsstrategiesexaminedinNAAQSRIAsmayrequirereducingemissionsatalargenumberoffacilities(potentiallyinthethousands)acrossmultiplesectorsoverrelativelylongtimehorizon.
In its air-related RIAs, the EPA has limited experiencewith evaluating uncertainty in compliance costestimates. However, some recent examples of where the Agency has examined the implication ofuncertainfactorsoncompliancecostsinclude:
• The levelof voluntaryemissions reductionsversus regulatoryemissions reductions foroil andnaturalgasemissionssources(EPA2012);
• whether stateswould adopt a rate-based ormass-based compliance approaches formeetingrequirementsoftheCleanPowerPlan(EPA2015a);
Asnotedinthe“WhitePaperonUsingCGEModelstoEvaluateSocialCostofAirRegulations”,regulatoryanalysis may also differ due to the available information on potential pollution control options. Forexample,insomecasespollutioncontrolequipmentmayalreadybeinusewithinpartsoftheindustryorcloselyrelatedindustries.Inothercases,theremaybeuncertaintyinhowstatesorlocalgovernmentsmayimplementregulationsandthereforeinthefacilitiesaffectedandthepollutioncontrolequipmentthatmightbeadopted.Forexample,inillustrativeattainmentanalysesconductedforsomeNAAQS,onceallidentifiedcontroltechnologieshavebeenappliedsomeareasofthecountrymaystillbemodeledas
8
out of compliance with the air quality standard. In some cases, EPA has had to assume the per toncompliance costs for “unidentified controls” that reduce emissions sufficiently to bring areas intoattainmentofalternativeairqualitystandards.Whenconsideringthecostofunidentifiedcontrols,theEPAhastraditionallyofferedsensitivityanalyseswithalessexpensiveandamoreexpensivecostpertonestimate.However,thereisnotafirmanalyticalbasisforthevalueschosenforthesensitivityanalyses;rather,theyroughlycorrespondtothesame+/-30percentrangeassumedforidentifiedcontrols.
Asdiscussedin“Economy-WideModeling:SocialCostandWelfareWhitePaper”,therearefewexamplesinwhichaCGEmodelhasbeenusedtoestimatethecostsofregulationsdesignedtoimproveairquality.InthecaseswhereaCGEmodelhasbeenusedtoanalyzenon-priceregulations,suchastheemissionslimitandairqualityregulationspromulgatedbytheEPA,theyareoftenmodeledasaproductivityshock.Specifically,compliancewiththeregulationcreatesaneedforadditionalinputstoproducegoodsintheregulatedsectoralongwithpollutionabatement.Whilethetotalcostoftheseadditionalinputscanbederivedfromdetailedcompliancecostestimatesfromanengineeringorpartialequilibriummodel,itisnotalwaysclearhowtoallocatethetotalcostamongtheinputsspecifiedintheCGEmodel,becauseCGEmodelsarebytheirnatureanaggregated,parsimoniousrepresentationoftheeconomy.Onefrequentlyusedapproachistoallocatetheabatementcostsindirectproportiontotheinputs(i.e.,capital,labor,andintermediate goods) used in the regulated sector of the CGE model. In other words, regulatoryrequirementsdonot change theproportionof labor, capital, or other inputs in the firm’sproductionfunction.This“Hicks-neutral”allocationistheapproachtakenbyHazillaandKopp(1990)andJorgensonandWilcoxen(1990).BallardandMedema(1993)allocatealloftheabatementcoststocapitalandlaborinputs only. There is inherent uncertainty associated with the decision to model the production ofpollutionabatementashavingthesameproductionfunctionastheaffectedindustry.EvenincaseswheredetailedcompliancecostestimatesprovideadditionalinformationastothesharesofinputsexpectedtobeusedinpollutionabatementtheremaybeuncertaintyastohowtheinputsfromtheseengineeringcostmodelsmaptothegoodswithinaCGEmodel.Furthermore,theremaybeuncertaintyastohowthesubstitutionpossibilitiesbetweeninputstoproductionintheregulatedsectorintheCGEmodelmaptosubstitution possibilities in pollution control options that may be adopted under non-price basedregulations, or how future technology developmentmight be expected to change those substitutionpossibilities.
Itisalsoimportanttoconsiderboththespatialandtemporalallocationofcost.Theremaybeamismatchbetweenaffectedfacilitylocationswithininputandoutputmarketsandthescaleoftheeconomy-widemodel,whichistypicallyregionalornationalinscale.Thatis,itmaybedifficulttopreciselymodelchangesthat affect specific areas of the country in an aggregatedmodel. As a result, thismay lead to someuncertaintyintheresultsoftheaggregatemodel.
Likewise,costsmaybeincurredatdifferentpointsduringthetimehorizonoftheeconomy-widemodel.Withalongtimehorizon,itisnecessarytoconsidertheroleoftechnologicalchangeanditsimpactontheprices of inputs over time. Recent NAAQS RIAs include discussions of how technological change(specifically,innovationincontroltechnologies)mayreducecostsovertime,andtheempiricalliterature
9
alsohasnotedthatvariablecostsofproductionorenvironmentalabatementtendtodeclineovertimewithcumulativeexperience.WhilethestudyonthecostsandbenefitsoftheCleanAirActfrom1990to2020(EPA2011)accountedfor“learningcurve”effects(ortheextenttowhichthecostsofatechnologydeclineasexperiencewiththattechnologyincreases),regulatoryanalysesoftendonotattempttoadjustcosts to account for different assumptions about technological changeor theeffect of experienceoncontrol costs.3 As a result, when incorporating compliance costs in a CGE model, any impact oftechnologicalchangewillbedeterminedbytherepresentationoftechnologicalchangethatexistsinthemodel, rather thanassumptions that are incorporated in theengineering costestimates.AsnotedbyPindyck(2007),characterizingtheuncertaintyovertechnologicalchangeisinitselfdifficult.
3 UncertaintyinBenefitsEstimation
Aswould be the casewith entering costs into a CGEmodel, uncertainty in any inputs related to thebenefitsofaregulationwillpropagatethroughtheanalysisandleadtosomeuncertaintywithrespecttoresults.Whileacompletediscussionof thesourcesofuncertainty inbenefitsestimates isbeyondthescopeofthispaper,thissectionprovidesabriefsummaryofEPA’seffortstocharacterizeuncertaintyinthebenefitsestimatesinrecentRIAs.
3.1 UncertaintyinRegulatoryBenefitsEstimates
Inanycomplexanalysisusingestimatedparametersandinputsfromnumerousmodels,therearelikelytobemanysourcesofuncertainty.EPAbenefitsanalysesincludeinputsfrommanydatasources,includingemissions inventories, air quality data from models (with their associated parameters and inputs),populationdata,populationestimates,healtheffectestimatesfromepidemiologystudies,economicdataand parameters for valuing benefits, and assumptions regarding the future state of the world (e.g.,regulations,technology,emissions,andhumanbehavior).Theremaybeuncertaintyassociatedwitheachof these inputs. Understanding key uncertainties in each stage of the analysis, and how they mightinteract,canbe important forunderstandingthe informationcontained inthetotalquantifiedbenefitestimates.
WhiletheNationalResearchCouncil(NRC,2002,2009)reviewedEPA’smethodologyforcalculatingthebenefitsofreducingairpollutionandfoundittobereasonableandinformative,theyalsohighlightedtheneed to conduct rigorousquantitativeanalysesofuncertaintyand topresentestimates inaway thatrecognizes their inherent uncertainty. In response, EPA has continued to work to improve thecharacterizationofuncertainty inhealth incidenceandbenefitsestimates.QuantitativeapproachestouncertaintyanalysisforbenefitsassessmentinRIAshaveincluded:
• whenmonetizingclimatebenefits,theuseoffourpossiblemeasuresforthesocialcostofcarbon(SC-CO2)andsocialcostofmethane(SC-CH4),reflectingalackofconsensusontheappropriatediscount rateand toaccount for thepossibilityofhigher-than-expected impacts fromclimatechange.
Whilesuchtechniquescanprovidevaluableinformationaboutkeyuncertaintiesandhowtheyinfluencebenefitsestimates,theseapproachesmaystillfacechallengesinaccountingfortheroleofuncertaintyinother input variables, including emissions and air quality modeling, baseline incidence rates, andpopulation exposure estimates. Challenges also remain in addressing correlations between inputparametersandfullycharacterizinginputdistributions.Asaresult,reportedconfidenceintervalsandtherangeofestimatesmaypresentonlyapartialpictureoftheoveralluncertaintyinthefinalestimates.
In2009,EPAundertookaprojecttoidentifytheinputparametersandassumptionsthathavethepotentialto be significant contributors to the uncertainty in the benefits estimates produced by BenMAP, theprimarytoolEPAusestoestimatethehumanhealthimpactsandeconomicvalueofairqualitychanges.To assess the impact of uncertainty in these parameters and assumptions, sensitivity analysis wasconductedwhere a range of values for key parameterswas defined, and their effects on resultswascalculatedwhileholdingallotherparametersattheirmidvalues.Thestudyfoundthatthecomponentsof a PM2.5 benefits analysis contributingmost to uncertainty of themonetizedbenefits andmortalityincidencearetheestimateofthevalueofastatisticallife,thechoiceofconcentration-responsefunctionformortality,andthechangeinPM2.5concentration(Mansfieldetal.,2009).
As noted in the “Economy-Wide Modeling: Benefits of Air Quality Improvements White Paper”, thebenefitsofregulationscanvarysignificantlyoverspaceduetoanumberoffactors,includingthespatialheterogeneityinairqualityimpactsandthespatialvariationinpopulationandbaselineincidencerates.Anumberofstudieshaveanalyzedthesensitivityofbenefitsestimatestotheresolutionoftheairqualityinputsand/orunderlyingpopulationandincidenceratedata.Theyhavefoundthatdifferencesinspatialresolution,operationalized in thestudiesbyvaryingthesizeof thegridcellwithinwhichairquality isassumedtobehomogenous,canleadtosubstantiallydifferentbenefitsestimates(DeRidderetal.2014,Fannet al. 2011, Kheirbek et al. 2013, Li et al. 2015, Punger andWest 2013, Thompsonet al. 2014).However,thestudiesdonotfindaconsistentbiasintheresults.PungerandWest(2013)foundthatverycoarsegridresolutions(>250km)producemortalityestimatesthatarebiasedsubstantiallylowforPM2.5,
Settingasideuncertaintiesintheupstreamelementsaffectingbenefitsestimates,anybenefitsanalysiswill produce a set of estimates that may include confidence intervals for some endpoints, may notcompletelyaccount forallof thebenefits (ordisbenefits)ofanaction,andwhichmaynotbedirectlytranslatable to the inputs needed for an economy-wide model. As discussed in the “Economy-WideModeling:BenefitsofAirQualityImprovementsWhitePaper,”challengesremaininfullyimplementingtheeffectofairqualityonthebehaviorandwell-beingofeconomicagentsinaCGEmodel.Todate,therehas been some limited inclusion of air quality impacts in CGE models including changes in medicalexpendituresandeffectsonthelaborforcethroughachangeinthetimeendowment.Expandingthosecategoriesofbenefitsor introducingamorecomplete treatmentofairqualitymayrepresentnotablechallenges and be associated with uncertainty surrounding the implementing methodology. In thesecases, theremaybeuncertaintyover the structureof themodel,orat least the componentsused toincorporatetheadditionalbenefitcategories.
Aswasthecasewhenconsideringtheinclusionofcostsinaneconomicmodel,itisimportanttoconsiderboth the spatial and temporal allocation of benefits.Whilemore spatially resolved datamay lead todifferentbenefitsestimates,thespatialresolutionofthebenefitsestimateistypicallymuchfinerthanthespatialresolutionofanyCGEmodelintowhichitmayserveasaninput.Asaresult,someaggregationofbenefitsestimatesand/orairqualitymodelingwillbenecessary.ThedegreetowhichthisaggregationinfluencesCGEmodeling results is an importantquestion. Likewise, the timingof benefits affects thebudgetconstraintandtimeendowmentavailabletohouseholds,whichmayinfluencetheresultsfromaCGEmodel.
TheCGEmodelscurrentlyusedinpolicyanalysisoftencontainhundreds,ifnotthousands,ofparameters,including those that define production technologies, household preferences, benchmark economic
OnewellrecognizedareaofuncertaintyinCGEmodelsaretheestimatesofelasticitiesthathelpdefineproductiontechnologiesandagentpreferences.Uncertaintyaroundelasticityparametershasfrequentlybeen a focus of analysts as themodeling results are often highly sensitive to these parameters (e.g.,Shoven and Whalley, 1984). The sensitivity of results to elasticities has been the subject of muchdiscussiongiventhecommonapproachofselectingthevaluesthroughacalibrationprocessasopposedeconometric estimation (Hansen and Heckman, 1996). However, selecting econometrically estimatedparametervaluesfromtheliteratureisnotwithoutitsownconcernsduetoinconsistenciesbetweentheempirical analyses and the structure of the CGEmodel and a large rangeof potentially contradictorystudiesthatprovideelasticityestimates(Canova,1995).Beckmanetal.(2011)demonstratedthatuseofcurrentandwell-researched substitutionelasticities for calibratedmodels canbecrucial forachievingdefensible behavior of a CGE model, and demonstrated an approach for validating modelbehavior/parameterizationusinghistoricalobservations.Giventhatnotallparametersofamodelmaybenecessarilyverifiedinsuchaprocess,sensitivityanalysis isonemeansoftryingtoassessareasonablerangeforthemodelingresultsofinterestgivenlikelyrangesoftheinputparameters.
Previouseconomy-wideanalysesconductedbyEPAhaveprimarilyexaminedthesensitivityofresultstokey policy parameters. For example, in 2009members of the United States Congress requested EPAconductananalysisofHouseResolution2454,whichproposedamulti-sectorallowance-tradingprogramdesigned to reduce emissions of greenhouse gases.4 In that analysis, EPA examined the sensitivity ofresults from two CGE models (ADAGE and IGEM) under a number of scenarios in which key policyparameters(e.g.,availabilityofoffsets,energyefficiencyprovisions)werevaried(EPA,2010a).However,inthatanalysisandasupplementaryanalysis(EPA,2010c),EPAalsoexaminedthesensitivityofresultstokeyassumptionsregardingthefutureavailabilityofcertaintechnologies(e.g.,newnuclearpowerplants,
Ingeneral,structuraltestingismoreresourceintensivethanparametricsensitivitytesting,thoughthisdependsuponthetypeofstructuralchangeandtherangeofparametricuncertainty.Therearemanyexamplesof singlemodel structural comparison testing in the literature.An illustrative sampleof thisliterature follows. Balistreri and Rutherford (2012) examine the differences between Armington andMelitztreatmentsofinternationaltradeinastudyofbordercarbonadjustments.Theyfindhigherleakagerates and more effective border adjustments under the heterogeneous firm structure of a Melitzapproach. Jacoby and Sue-Wing (1999) and McFarland et al. (2004) show that the incorporation ofvintaged capital in aCGEmodel raises the costsof reducing greenhousegasemissions.Babiker et al.(2009)comparesforward-lookingandrecursive-dynamicspecificationsforclimatepolicyanalysis.Theynotethatthemacro-economiccostsare lower intheforward-lookingversionbecauseoftheabilitytoshift consumption optimally over time. However, the forward-looking model, due to computationallimitations, also drops the full capital vintaging specification and contains fewer explicit emissionreductiontechnologies.Thesefactorslikelyhavecountervailinginfluencesontherelativemacroeconomiccosts. EPA addressed structural uncertainty in the aforementioned economy-wide analysis of HouseResolution 2454 by using twomodels: ADAGE and IGEM (EPA, 2010a). Themodels differ on severalstructuralfronts.ADAGEisacalibratedmodelandIGEMiseconometricallyestimated.ADAGEhasamoredetailedrepresentationoftheenergysectorandenergysectortechnologies;IGEMhasmorenon-energysectordetail.
5 AnalyticalApproaches
Uncertainty associatedwith engineering cost and health and environmental benefit inputs, analyticalassumptions, andmodel parameters and structure,may in some caseswarrant additional analysis tobetterexploretheexpectedimpactsofapolicy.Insomecases,thespecificsoftheregulationinquestionmay informwhether theadditional resourcesrequiredtoperformadditionaluncertaintyanalysis isofsufficient value. For example, Circular A-4, which “provides the Office ofManagement and Budget’s(OMB's) guidance to Federal agencies on the development of regulatory analysis,” suggests a formalquantitativeanalysisof relevantuncertaintiesbeconductedformajorrulesthatareexpectedtohave“annualeconomiceffectsof$1billionormore”(OMB,2003).Thissectionconsidersdifferentquantitative
14
approaches for considering uncertainty in policy analyses conducted using CGE models, includingsensitivityanalysis,formaluncertaintyanalysis,andinter-modelcomparisons.Foreaseofexposition,wedelineate sensitivity analysis and formal uncertainty analysis, by defining the former as analyses thatconsiderscenariosinwhichoneormoreinputsarevariedoutsideofawell-definedprobabilityspaceandthelatterasanalysesthatattempttocharacterizethedistributionofmodelresults.
5.1 SensitivityAnalysis
Themostcommonapproachtotestingtherobustnessofmodelingresultsistovaryanexogenousvariableor parameter and solve the model to obtain the new set of endogenous state variables under thealternative parameterization. In this paper, such an approach is defined as sensitivity analysis todistinguishitfrommoreformaluncertaintyanalysis(Section5.2)andinter-modelcomparisons(Section5.3).Weusethetermsensitivityanalysisas inclusiveofcomparativestaticsanddynamics,alongwithsituationsinwhichthemodelsallowfortheexistenceoftemporarydisequilibrium.
Itiscommonforsensitivityanalysistovaryasingleparameterinthealternativescenariostoisolatetheimpactithasontheresultsofinterest.Thisapproachallowsanalyststogatherinformationabouthowrobustqualitativeconclusions,basedonmodelingresults,arewithrespecttospecificassumptionsorkeyparameters.EPA’sGuidelinesforPreparingEconomicAnalysis(2010b)suggeststhat“[i]ncaseswherethedataareuncertain,ornoteasilyquantified,butmayhaveasignificantinfluenceontheresults,theanalystshould describe the weaknesses in the data and assumptions, and include some type of sensitivityanalysis.”
Sensitivity analysis that varies only one, or a few, parameters at a time could potentially provide anincomplete characterization of the uncertainty surrounding the results due to important interactionsbetweenparameterswithincomplexandnon-linearCGEmodels(Ableretal.,1999).Byitsnature,basicsensitivityanalysisislimitedbythefactthatisnormallyconductedoutsideofawell-definedprobabilityspace that guideswhich parameters to vary, and themagnitude bywhich they should be perturbed.However,incaseswithincompleteinformationaboutthecompletedistributionalparameterspaceand/orlimitedresourcesandtime,basicsensitivityanalysismayprovideusefulinformationastotherobustnessofmodelingresultsandprovidevaluableinformationfordecisionmakers(Pannell,1997).Furthermore,asnotedbyOMB(2003)inCircularA-4:
“Insomecases,thelevelofscientificuncertaintymaybesolargethatyoucanonlypresentdiscretealternativescenarioswithoutassessingtherelativelikelihoodofeachscenarioquantitatively.Forinstance,inassessingthepotentialoutcomesofanenvironmentaleffect,theremaybealimitednumberofscientificstudieswithstronglydivergentresults.Insuchcases,youmightpresentresultsfroma rangeof plausible scenarios, togetherwith any available information thatmighthelp inqualitativelydeterminingwhichscenarioismostlikelytooccur.”
While this description reinforces the benefits of sensitivity analysis, it also highlights the challengespresent in conducting sensitivity analysis, as concepts such as a range of plausible scenarios are notformallydefined.However,itispossibletobringsomestructuretosensitivityanalysis.Forexample,theCongressionalBudgetOffice(CBO)approachessensitivityanalysisinitsmacroeconomicestimatesusing
15
a defined process to address the lack of structure. Specifically, in its dynamic scoring approach CBOdetermines the twoparameters in themodel towhich the results aremost sensitive. Then potentialestimatesaregeneratedbyexaminingeachcaseinwhichthosetwoparametersareattheendsoftheirrangesandotherparametersareequaltocentralestimates,whereCBOultimatelyreportsthecasesthatshowthemostandleastfavorablebudgetaryoutcomes.5ItisalsoworthnotingthatCBOdoesnotconductaquantitativeuncertaintyanalysisforeachbillconsiderediftherangederivedfromtheanalysiscouldpotentiallybemisinterpreted.Forexample,incaseswheretheunderlyingconventionalcostestimateisassociated with significant uncertainty relative to the uncertainty in the macroeconomic model,conductingsensitivityanalysisoveronlytheparametersinthemacroeconomicmodelcouldproducearange of estimates that should not be interpreted as characterizing the full range, given significantadditionaluncertaintyintheconventionalcostestimate(CBO,2015).
Sensitivityanalysisofafewselectparameters,asdescribedinSectionError!Referencesourcenotfound.,mayallowananalysttogetageneralsenseofhowthemodel’sresultsmaydependonkeyparametervalues. However, due to the lack of amethodological structure, which leads the approach to ignoreinformationcontainedinthecovarianceofuncertainparameters,thisapproachisnecessarilyimprecise(Bernheimetal.,1989).Asamorestructuredalternative,modelershaveconsideredformaluncertaintyanalysisthattakesintoconsiderationadditionalinformationabouttheuncertainparameters.6
Thecurrentclassofappliedgeneralequilibriummodelsusedforpolicyanalysishavehundredsofinputparameters.Evenforarelativelysmallmodelwitheightparametersthateachhavefivepotentialvalues,ifthemodeltookoneminutetosolveandsavetheresultsacompletefactorialexperimentdesignthatconsidered every possible combination of parameterswould require ninemonths of processing time(Pannell,1997).ItislikelythatwithmoderncomputingresourcesmodelsthatsolveinoneminutelikelyhavefarmorethaneightuncertainparametersmakingPannell’sexampleevenmoreapt.Itisalsolikelythatparametersaremoreappropriatelycharacterizedbydistributionsthanbyadiscreteset,as intheexampleset forthbyPannell (1997).Therefore,Gaussianquadrature (e.g.,Herteletal.,2007),MonteCarlo(e.g.,Selinetal.2009),andlinearization(e.g.,Jorgesonetal.,2013)methodsaremorecommonlyusedtoconductformsofprobabilisticanalysis.Thesetechniquesprovidewaysofapproximatingintegralsoveruncertainmodelparameterstoobtainexpectedvaluesandconfidenceintervalsformodelresults.
( ) [ ]( )1 1, , , , ,E H X H X Eβ θ β θ≠⎡ ⎤⎣ ⎦ , (5)
such that evaluating the change in model output under the policy at the expected values of theparameterswill not be equal to expected changewhen fully considering the parametric uncertainty.Probabilistic analysis allows analysts to providemore robust assessments of the expected change in
number of simulations conducted, with the error being of order 1 N and therefore requiring a
potentiallylargenumberofsimulations.
Incasesofhighlycomplexmodels,orrelativelyscarcecomputingresources,thenumberofsimulationsrequiredtogetreliableestimatesofthemomentsaroundthevariablesofinterestcanbeofconcern.Toaddress the computational burdens,Harrison andVinod (1992) suggest approximating theunderlyingprobabilitydistributionwithadiscretesetofpointsandprobabilities,allowingfor fasterconvergence.However, their approach of choosing the points for each parameter by dividing the support of themarginalprobabilitydistributionintoafinitenumberofequi-probableregions, isknowntounderstatethevariance(andallotherhigherorderevennumberedmoments)oftheparameterdistribution(MillerandRice,1983).ThisbehaviorisinheritedbytheapproximatejointdistributionoftheparametersandwhenusedintheMonteCarloapproachwillbiastheresults(DeVuystandPreckel,1997).
thesetsofparametersandcorrespondingweights,theanalystmay,insomecases,beabletoapproximatethemomentsaroundthemodeloutputswithfarfewermodelrunsthanundertheMonteCarloapproach,J N< .Ingeneral,thebasisforselectingtheweightsandcorrespondingparametersistoensurethatthemomentsaboutzero,uptoagivenorder,wouldbeadequatelyintegratedusingtheselectedvalues.Byusing higher order quadrature techniques greater accuracy may be achieved when integratingcomplicatedsystemsofnon-linearequations,suchasaCGEmodel,thoughtheabilitytoapproximatetheintegraldependsontheabilityofapolynomialtoapproximatethecurvatureofthemodel.
In their original application,DeVuyst andPreckel (1997) use a global, energy-explicit CGEmodelwithuncertainfossilfuelsupplyelasticitiestoevaluatetheimpactsofacarbontax.Theauthorsdemonstrate
18
theapproachwithsixuncertainparametersfromindependentdistributions,andfindthattheycanobtainareasonableapproximationof themeanandstandarddeviationof themodel’sendogenousvariableswithrelativelyfewmodelruns.Sincetheiroriginaldemonstration,GaussianQuadraturehasbeenusedtoestimate expected values and confidence intervals for key results fromCGEmodels inmore complexcontexts.Forexample,Herteletal.(2007)usethetechniquetoderivemeansandconfidenceintervalsfromaglobalCGEmodelwithdozensofuncertainparameters,whenstudyingthe impactof reducinginternationaltradebarriers.However,toapplythemethodtheyassumethatalluncertainparametersarenormallydistributedandindependent.
Forcommondistributionsandinthecaseofindependentuncertainparameters,techniquesforselectingweightsandparametricvaluesarereadilyavailable(MirandaandFackler,2002).TheoriginalapplicationusingthistechniquewithCGEmodelsfocusedoncaseswithsymmetricandindependentdistributionsforthe uncertain parameters. For the case of the non-symmetric distributions, non-diagonal covariancematrices,and/orlargeparameterspacestechniquesforGaussianQuadraturemaynotbereadilyavailableormaynotprovidethesamecomputationalimprovementoverMonteCarlosimulations.Suchlimitationsmaybe important, as Jorgensonetal. (2013)demonstrates thatassuming independenceofuncertainparameters in a CGE model can significantly bias confidence intervals around endogenous variablesdownward. However, Horridge and Pearson (2011) demonstrate an approach to applying GaussianQuadrature in the context of CGEmodels in cases with non-diagonal covariancematrices across theuncertainparameters.TheauthorsdemonstratethefeasibilityofthemethodusinganexamplefromtheGlobalTradeandAnalysisProject.
AnotheralternativetoMonteCarlosimulationsisbasedonthelinearizationoftheCGEmodel(e.g.,Paganand Shannon, 1985;Wigle, 1991). This approach is sometimes referred to as the Deltamethod. Forexample,Bernheimetal.(1989)andTuladharandWilcoxen(1998)usethisapproachtoderiveconfidenceintervalsaroundendogenousstatevariables inaCGEmodel.Themethodseekstoprovidetheanalystwith a confidence interval around the endogenous variable evaluated at the central tendency of theuncertain parameters. Themethod is based on linearizing themodel using a first order Taylor seriesexpansion around the endogenous variables with respect to the model’s parameters.7 Under thisapproximation,acovarianceformodel’spointestimateisderivedastheproductofthemodel’sJacobianwithrespecttoitsparametersandthecovariancematrixfortheparameters.Usingthenotationabove,themodelinlinearizedas
[ ]( ) ( )1 1, , , ,H X E H X J Jβ β ββ θ β θ ʹ≈ + Σ , (10)
where Jβ is the Jacobianof theCGEmodelwith respect to theuncertain parameters and βΣ is the
Given the complexity ofmost applied CGEmodels, the Jacobian is computed bymeans of numericaldifferentiationandinmostcasesmodelershaveusedabasicforwardfinitedifferencetominimizethenumberofmodelsolvesrequiredtoderivetheJacobian.Jorgensonetal.(2013)haveshownthat,eveninthecaseofcomplexdynamicCGEmodels,themethodcanprovidestandarderrorsasapercentageofthemodel’sendogenousvariablesthatarereasonablyclosetothestandarderrorsasapercentageoftheexpectedendogenousvariablesinthecaseofMonteCarlosimulations.Theirresultssuggestthatinsomecases,dependingonthemodel, theDeltamethodmaybeabletoderiverelativestandarderrorswithfewersimulationsthanaMonteCarloapproach.
With the increased availability of computing resources and simulation techniques, Monte Carlosimulationshavebeenusedinnumerousrecentstudiestoexplorethesensitivityofmodelingresults(e.g.,Websteretal.,2001;Sokolovetal.,2009;Elliotetal.2012a;Elliotetal.2012b).Forexample,Elliotetal.(2012a)takeadvantageoflarge-scaleparallelprocessingtoconductMonteCarlosimulationsthatexplorethesensitivityofCGEmodelresultstoparametricuncertaintyinthecaseofaglobalCGEmodel.8Notably,they find that thebias associatedwithmodel results basedon running theCGEmodel at the centraltendencyofallparametersversusthecasewithfulluncertaintyinparametersissmall.However,theyfindthevariancearoundtheCGEresultswasstillsignificant.Inabootstrapanalysisoftheirresults,Elliotetal.(2012a) find that the number of simulations required to obtain information about the shape of thedistribution around endogenous variables might vary considerably depending on the uncertainparametersconsideredandtheendogenousvariablesofinterest.Forexample,Elliotetal.(2012a)findthatendogenousvariableswithlowercoefficientsofvariationrequirefewersimulationstoadequatelycapturethemeanandvarianceoftheresult,comparedtovariablewithrelativelyhighercoefficientsofvariation.
BasicMonteCarloanalysisreliesonthestronglawoflargenumbersandthecentrallimittheoremforitsconvergenceproperties.Intermsofestimatingthemeanoftheendogenousvariablestheapproximationerrortendstozeroasthenumberofsimulationstendstoinfinity;however,inpracticetheapproximationerrorisrandomduetotheprobabilisticnatureoftheanalysisandcanbelargeevenwhenthenumberofsimulationsislarge.InrecentMonteCarlostudies,researchershaveusedquasi-MonteCarlotechniquestomoreefficientlysampletheparameterspaceandthereforereducethenumbersimulationsrequired.The goal of these approaches is to impose additional structureon themethodused todraw randomparametersfortheanalysis,suchthattheapproximationerrorisreducedbeyondwhatitwouldbeinabasicMonteCarloanalysis.9Forexample,Websteretal.(2001)useLatinHypercubesamplingtoreducethenumberofrunsrequiredforadesireddegreeofaccuracy.
8 Similar to other studies, Elliot et al. (2012a) assume the uncertain parameters are normally distributed and
Building uponMonte Carlo techniques, variance decompositionmethods, such as the Sobol method(Sobol,1993),producesensitivityindicesthatestimatethefractionalcontributionofaninputparameter(andparameterinteractions)tothevarianceofanoutput.Thesemethodsallowthemodelertotracethedependenciesbetweenoutputsandinputs.However,theycanbecomputationallyintensive(IoossandLemaitre, 2015). Although variance decomposition methods have been employed in environmentalmodelingandchemicalengineering, theyhavenotbeenwidelyused inconjunctionwithCGEmodels.However,inoneexampleMohora(2006)employstheSobolmethodinastudyoftheRomanianeconomyand finds that the variance decomposition identified that indirect effects and database structurecontributemost tooutputvariance. Interestingly,Mohora foundtheseresults tobe largelyconsistentwithasimplerparameterscreeningmethodusedinthestudy.
5.3 Inter-ModelComparisonExercises
Inter-modelcomparisonexercisesareameanstoexaminetheeffectofbothstructuralandparametricdifferences betweenmodels. These exercises convene severalmodeling teams to examinepolicy andmodelingissuesbycomparingmodelresultsforastandardizedsetofscenarios.Inter-modelcomparisonexercises can servemany purposes. The exercises bring togethermodelers from academia, industry,government,andotherorganizationstoexamineanissuefromdifferentperspectivesandusingdifferenttools. The section will discuss the basics of inter-model comparisons projects, past examples, andimportantconsiderationstotakeintoaccountwhendesigningsuchexercises.
Inter-modelcomparisonexercisesarestructuredaroundasetofpolicy-relevantresearchquestions.Eachofthemodelingteamsproduceresultsforacommonsetofscenarios(e.g.,withandwithoutapolicyoratechnology).Thescenariosandresultsaredevelopedandanalyzedtypicallyoveraseriesoftwotothreeworkshopsheld severalmonthsapart. Theseexercises serve several valuable functions. First, amulti-modelapproachtoatopicmayhighlightareasofrobustagreementaswellasidentifyandthekeyfactorsdrivingdisagreementacrossmodels.Second,theexercisehelpstoexplainthestrengthsandlimitationsofalternativemodelingapproaches.Third,theexercisesprovideaforumformodelerstoreceivepeerfeedbackonmodelingissuesthatrangefromdataandparameterselectiontostructuralspecificationsandmodelingtechniques.Finally,theexercisesidentifyfutureresearchandmodeldevelopmentneeds.
Although inter-model comparison exercises are valuable to both the policymaking and modelingcommunities,thereareimportantchallenges.Forexample,itcanbedifficulttodiscernthekeyfactorsthatleadtotherangeofresultsacrossmodelsbecausethemodelsmaydifferbytype(e.g.,generalvs.partialequilibrium)inadditiontostructuralandparametricdifferences.OnenotableexceptiontothisisEMF29:TheRoleofBorderCarbonAdjustmentinUnilateralClimatePolicy(Bohringeretal.,2012).EMF29wasentirelycomprisedofCGEmodelsthatusedthesameunderlyingdataset(GTAP7.1).Inaddition,cautionshouldbetakenwheninterpretingtherangeofresultsfromcomparisonexercisesbecausetherangetypicallydoesnotrepresentthefullrangeofuncertainties.Instead,therangeofresultsrepresentthe uncertainties across the participating models and possibly over a handful of parameters orassumptions in the study. Finally, a study may not necessarily reflect all models as participation isvoluntaryandmaynotbesupportedbyexternalfunds.
6 PresentationofUncertaintyAnalysis
Whileadditionalsensitivityorformaluncertaintyanalysiscanprovideuseful informationaspreviouslydescribed,italsopresentsachallengetoanalystswhomustsummarizethisadditionalinformationinaformat that is readily understood by policy makers and stakeholders. When uncertainty analysis isconducted,CircularA-4guidesanalyststotryto“providesomeestimateoftheprobabilitydistributionofregulatory benefits and costs” (OMB, 2003). However, there are different formats in which suchinformationcanbeconveyed.Informationabouttheprobabilitydistributionofbenefitsandcostscanbepresentednumerically,verballyorgraphically(NAS,2013).
Anumericalrepresentationmayincludesummarystatistics,suchthecentraltendency(e.g.,meanandmedian), variances, confidence intervals, or other potentially relevant characteristics. A numericalpresentationoftheresultscanprovideasignificantamountofinformation,however,theapproachmayrequirethattheaudiencehavetheappropriateexpertisetounderstandandinterprettheinformation.Verbalrepresentationsconveyinformationusingcuessuchaslikelyorunlikelytodifferentiatepotentialoutcomesandcharacterizetheprobabilitydistributionofbenefitsandcosts.Whileverbalrepresentationsof uncertainty may be more approachable to a broader audience, there is the potential for theinterpretationtodifferacrossindividuals.TheUnitedNationsIntergovernmentalPanelonClimateChange(IPCC)hasattemptedtoaddressthisconcernbyestablishingspecificdefinitionsanduseguidelinesfortheverbalcharacterizationintheirassessmentreports(IPCC,2010).Graphicalrepresentationsofprobabilisticinformation have the potential to convey more information than basic verbal cues while remainingaccessibletobroaderaudiences.Graphicalrepresentationscantakedifferentformssuchashistogramsorcolorwheels.Forexample,Websteretal.(2008)presentsomeresultsoftheirprobabilisticCGEanalysisasprobabilitydistributions.Figure1,whichisfromtheiranalysis,presentsthelossinglobalconsumptioninagivensimulationyearacrossdifferentpolicystringencies(i.e.,Level1through4).
Awell-known example of colorwheels to convey probabilistic information is thework byMIT’s JointProgramontheScienceandPolicyofGlobalChangeoncommunicatinguncertaintysurroundingclimatechange (Figure3).11Thisplotpresents theprobabilityofdifferentclimateoutcomeswithandwithoutpolicy as two different pie charts where the area of the slices corresponds to the probability of theoutcome conditional on the policy assumption associated with the pie chart. The graphic eases
Inadditiontopresentingtheresultsofuncertaintyanalysis,analystsmaychoosetoprovideadditionalinformationastothesourcesofuncertaintymostrelevanttotheresults.Forexample,Tornadoplotsmayalsobeuseful in communicating the resultsof sensitivityanalysis to showtheparametersofgreatestinfluenceovertheresultsofinterest.Jacobyetal.(2006)conductastudywiththeEPPACGEmodelinwhichtheyvarykeyparameters,onebyone,byplusandminusonestandarddeviation.Theythenplotthe effect of these experiments on thewelfare change of a given policy in order of greatest to leastinfluence(Figure4).
However, tornadoplotscanalsobeusedtopresenttheresultsofprobabilisticanalyses.Forexample,standardizedregressionsmaybeusedtoanalyzetheresultsofMonteCarlosimulations,asdescribedinSection 5.2, to determine the influence of different parameters on the results of the analysis. Afterstandardizing the draws for the input parameters, they may be regressed on the output variable ofinterest to obtain standardized regression coefficients that represent the impact of a one standarddeviation change in the parameters on the output variable of interest. The results of such additionalanalysisareoftenpresentedintheformoftornadoplots.
7 VerificationandValidationExercises
The methods of conducting and presenting uncertainty analysis discussed in Sections 5 and 0 areapproaches to providing decision makers and stakeholders with additional information about therobustnessofmodelresultstohelpinstillconfidenceinthequalitativeconclusionsofpolicyanalysisbasedon modeling results. Additional confidence in the model results may be provided through formalapproaches to model verification and validation. Model verification and validation represent twoimportant, interrelated activities in the CGE model building process. Carson (2002) defines modelverificationasa“processandtechniquethatthemodeldeveloperusestoassurethathisorhermodeliscorrect and matches any agreed-upon specification and assumption.” Sargent (2013) defines modelvalidationasthe“substantiationthatamodelwithinitsdomainofapplicabilitypossessesasatisfactoryrange of accuracy consistent with the intended application of the model.” This section discussestechniquesformodelverificationandvalidationwithafocusonCGEmodels.
25
7.1 ModelVerification
Verification of a CGEmodel constitutes the initial phase of themodeling project (Carson, 2002). Thisprocessinvolvesanumberofiterativeproceduresthatinclude,amongotherthings,modelspecification,dataorganization,anddebuggingofthemodelcode.Themodelerwillusuallyperformanumberoftestsandsimulationstoexaminebasicfeaturesoftheunderlyingdataandeconomicstructure.Forexample,acommoninitialexperimentattemptstotestifthemodelcodeandtheimbeddedassumptionsreplicatethebenchmarkdataset.Ifthemodelisspecifiedcorrectly,theequilibriumsolutiontothemodelwithnoshocksshouldbeidenticaltothebaselinedata.Thischeckcanbeperformedateachstageofthebaselinesettingprocess.Forexample,RauschandRutherford(2008)describeasetofprogramsthattransformIMPLANdataontheU.S.economyintoaformatthatcanbeusedinaCGEmodelimplementedintheGAMS/MPSGEmodelinglanguage.Aftereachstepintheprocess,thebenchmarkistestedinasmall-scale,highlyaggregated,open-economymodel.Thesestepsincludeaggregatingthedata,translatingparameternames,diagonalizingthedatasothateachindustryproducesonlyonecommodity,mergingstateleveldata, and balancing interregional trade flows. If the benchmark is not replicated at any point, theprogramsstopsandanerrormessagepromptstheusertocorrecttheimbalancebeforemovingon.
DixonandRimmer(2013)describeabasichomogeneitytestthatmaybeperformedinthedevelopmentofsomeCGEmodels.Ifallmodelsectorsarerepresentedwithconstantreturnstoscale(CRTS)productionfunctions, then increasing the exogenous nominal variables by a percentage should return the samepercentagechangeintheendogenousnominalvalues.For instance,doublingthecostsofall inputstoproductionshouldresultinadoublingoftheunitcostofoutput.Ifthemodelcontainsrepresentationsofnominalrigidities,suchasstickywages,adherencetohomogeneityshouldnotbeexpected.Theeffectsof these deviations can be checked by turning off the features that cause the rigidity, such as arepresentationofstickywages,totestifthefeatureiscausingthedeviationfromthehomogeneouscase.Otherdeviationsinthehomogeneitytestmayrepresenterrorsinthemodel.
CGEmodelverificationcanalsobeconductedusingGDPaccounting identities.GDPcanbecomputedfromCGEmodeling resultsbyaddingup total incomesor totalexpenditures,wherebothcalculationsshouldproducethesamevalue.Eachof thesecalculationscanalsobedone inrealornominal terms.Thesechecksareusefulbecausetheyinvolvedifferentsetsofmodelvariables,andarethereforetestingfor consistency across themodel. The expenditure calculation involves themacroeconomic variablesconsumption, investment,governmentexpenditures,andnetexports.The incomecalculation includesfactorincomesforallagents,transferpayments,andtaxrevenues.
Thesocialaccountingmatrix(SAM)usedtocalibratethemodelprovidesanadditionalcheckonthemodel.TheSAMincludesallpaymentsbetweenindustriesandagents,aswellasinternationaltradebalances,andshouldbalancebeforeandafterapolicyshock.Thismeansthatthezero-profit,market-clearance,and income balance conditions hold for each industry, good, and agent in the model. (For moreinformationonSAMssee,interalia,PyattandRound(1985).)
26
7.2 Modelvalidation
While it has been argued that a simulation model can never be completely validated (Gass, 1983),subjectingamodeltovalidationtests–andmakingappropriateadjustmentstothemodelifnecessary–canincreaseconfidenceintheusefulnessofthemodelforpolicyanalysis.Anumberofmethodsformodelvalidation have been used by CGE modelers. This section describes several of these methods and anumberofexercisesthathavebeenconductedtovalidate(orinvalidate)CGEmodels,alongwiththecostsofmodelvalidation.Validationexercisescouldbeperformedwhenamodelisfirstdeveloped,atthetimeofmajorrevisions,orsimplyovertimetocheckthemodelhascontinuedvalidity.
7.2.1 BackoftheEnvelopeModels
DixonandRimmer(2013)suggestthatbackoftheenvelope(BOTE)modelscanbeusefultoolsforCGEmodelvalidation.Theseestimatescangiveasenseofthemagnitudeoftheexpectedresultsfromthefullmodel.ABOTEmodelsimplifiesthelarge-scalemodeltogaininsightonmodelresults.Thissimplificationcanbedonebyaggregatingthenumberofindustries,factorsofproduction,and/oragentsinthemodel.Afterasimplemodeliscalibratedtotheaggregateddata,themodelercanaddpolicyvariablestotesttoseeiftheexpectedmagnitudeofchangesinthemodelresultsareproduced.Next,themodelercanaddone featureata timeto theBOTEmodel to test theperformanceof these featuresof the large-scalemodelandtogaininsightabouthoweachfeatureinfluencestheresultsofsamplescenarios.
SimpleBOTEestimatesoftheGDPimpactsofapolicyshockcanbecalculatedusingdataandparameters,suchasindustrysize,taxrates,andelasticities.EachofthecomponentsoftheGDPcalculationcanbeusedtohelpexplainmodelresultsandtheoverallimpactofapolicyonGDP.Policiesthataffecteconomicactivity can influence the componentsofGDP indifferentdirections.Wemayexpect that a rule thatrequiresinstallationofnewcapitalwouldincreasetheinvestmentcomponentofGDPanddecreasetheconsumptioncomponent.OncethemacroeconomicimpactsofthesimplifiedmodelarewellunderstoodwiththeBOTEmodel,thefullCGEmodelcanbeusedtounderstandtheeffectsonspecificindustriesandhouseholds.
7.2.2 StatisticalValidationofModelParameters
DixonandRimmer(2013)suggesttheuseofstatisticaltechniques,suchasregressionanalysis,toimprovetheunderstandingofmicroeconomicresults.Theyusetheexampleofananalysisofemploymentchangesunder a scenario in which import tariffs and quotas are removed. The USAGE CGE model forecastsemploymentchangesbystate,whicharethencomparedtoaregressionanalysisthatseekstoforecastthesamechanges.Theregressionanalysiswaspreparedforthisvalidationexerciseusinghistoricaldatatoestimatearelationshipbetweenthetradepolicyandemployment.Byrankingtherelativeimpactsbystateandcomparingtheserankingsacrossthetwomodels,theregressionexerciseallowedtheauthorstoseewherestateswererepresenteddifferentlyineachmodelandultimatelyimprovetheCGEmodel.
Althoughthepublished literature isrelativelysparse,anumberofauthorshaveundertakenvalidationexerciseswith theirmodels. Recent efforts have used increasingly sophisticated techniques thatmayallowforaprocessbywhichsignificantmodelingimprovementscanbemade.
Kehoeetal.(1995)comparedresultsfromtheirCGEanalysisoffiscalreforminSpain–replacementofacomplexsystemofindirecttaxeswithavalue-addedtax–during1985-1987withthechangesthatactuallyoccurred.Theycomparedchangesinconsumerandindustrialprices,industrialoutput,andanumberofmacroeconomic variables using two performance metrics: the weighted correlation coefficient andweightedR2.Kehoeetal.(1995)reportthattheirmodelperformedwell,particularlywhentheyincludedtwoexogenousshocksthataffectedtheSpanisheconomyduringtheperiodofanalysis(afallinthepriceofpetroleumandadeclineinagriculturalproductivityduetoadverseweatherconditions).Inaddition,they found that the results were robust to alternative specifications of the labor market andmacroeconomicclosurerules.
Kehoe (2005) evaluated the performance of threemultisector static CGEmodels used to predict theimpactoftheNorthAmericanFreeTradeAgreement(NAFTA).Kehoe(2005)reportsthatallofthemodelsdramaticallyunderestimatedtheimpactofNAFTAontradeamongsttheU.S.,Canada,andMexico.Themodelswerealsoreportedtohaveonlybeenabletocapturerelativeportionoftheimpactsindifferentsectors.Kehoe(2005)arguedthat,basedonhisanalysis,anewtheoreticalmechanismwasneededforgeneratinglargeincreasesintradeinproductcategoriesthatpreviouslyhadlittletrade.Inaddition,inorder to capture changes inmacro aggregates,models need to be better able to capture changes inproductivitybroughtaboutbytradeagreements.Whilethiseffortdidnotexplicitlyresult inmodelingimprovements, this retrospectiveanalysiswasable to identifypotentialareasof improvements in themodels.
Valenzuelaetal.(2007)developedamethodologyforvalidatingCGEmodelsonasector-by-sectorbasis,focusingonthewheatmarketintheGTAPmodel.Theyemployedastochasticsimulation,usingshocksderived from a time-seriesmodel tomeasure the randomness in annual output over the 1990-2001period. The residuals were used to create a distribution reflecting random productivity variation byproducingregion.Theseproductivityshocksgeneratedendogenousfluctuationsinproductionthatmatchthose in the data. Solving the CGEmodel repeatedly while sampling from this distribution yielded adistributionofcorrespondingmarketpricechangesforwheat,byregion.Standarddeviationsbasedonthesemodeledoutcomeswerethencomparedtoobservedoutcomesforyear-to-yearpricechanges.TheauthorsfindthatwhentheGTAPmodelfails,ittendstodosoinasystematicway,under-predictingpricevolatilityfornetexportersofwheat,andover-predictingvolatilityforimportingregions.Usingtheinsightsgainedfromthisexercise,theauthorswereabletomakemodificationstothemodelspecificationthatimprovedthefittothefluctuationsinthetime-seriesdata.
28
UsingamethodologysimilartothatofValenzuelaetal.(2007),Beckmanetal.(2011)examinedtheabilityof the GTAP-E model to reproduce historical price volatility in the petroleum market.12 WhereasValenzuelaetal.(2007)focusedonthesupply-side,Beckmanetal.(2011)includebothsupply-sideanddemand-side shocks. For the supply-side,Beckmanet al. (2011) fit a time-seriesmodel to countryoilproductionoverthe1980-2005timeperiod.Forthedemand-side,ageneralindicatorofeconomicactivity,GDP,wasused.Time-seriesresidualswerethenusedtocreateprobabilitydistributionsforrandomshockstotheunderlyingsupplyanddemandschedulesforpetroleum.Usingtheseshocks,standarddeviationsofoilpricechangesbasedonGTAP-Emodelrunsarethencomparedtothosefromthehistoricaldata.TheoriginalGTAP-Emodelsignificantlyunderestimatedpetroleumpricevolatilityincomparisonwiththedata.Inresponse,Beckmanetal.(2011)conductedanextensiveliteraturesearchandupdatedrelevantmodelparameters.There-parameterizedmodelperformedsignificantlybetter.To further test there-parameterizedmodel,Beckmanetal.(2011)alsoperformedastylizedmedium-runsimulationinwhichthey shockedpopulation, labor, capital, investment,oilprices, andTFPbyobservedchangesover the2001-2006period.Withthenowmoreinelasticdemandspecification,themodelwasabletocapturethebroadchangesinthepetroleummarketincludinganincreaseindemandthatoccurreddespitethesharpincreaseinpriceduringthisperiod.
7.2.4 ChallengesofModelValidation
Sargent(2013)pointsoutthatsimulationmodelvalidationcanbetime-consumingandcostly,andthatitisnotfeasibletoensurethatalarge-scalemodelisvalidoveritsentiredomain.SeveraloftheexamplescitedinDixonandRimmer(2013)werestatedtohavetakenyearstocomplete,duetothetimerequiredto collect the historical data, perform numerous runs, and conduct detailed analysis. Sargent (2013)further suggests that validating amodel for one specific purpose does notmean it is valid for otherapplications,andthatadditionalvalidationmaybenecessary.Furthermore,inthecontextofregulatoryanalysisbackcastingexercisesmayfaceadditionalchallengesduetothenatureofCGEmodelsastoolsto estimate changes instead of levels, and the unobserved counterfactual baselines necessary tounderstandthehistoricalchangebecauseofthepolicy.
8 ConcludingRemarks
EPAhasahistoryofconsideringuncertaintyindifferentcontextsandusingavarietyofqualitativeandquantitativeapproachestodeterminetherobustnessofpolicyoutcomes.However,asoutlined inthiswhite paper, there may be additional sources of uncertainty associated with extending benefit costanalysestoaneconomy-wideperspective.Thisleadstoquestions,suchas:
• Are certain types of uncertaintymore of a concernwhen evaluating social costs, benefits oreconomicimpactsinaneconomy-wideframework?
• Arechallengesor limitations related to theseuncertaintiesmoreofaconcern than forpartialequilibriumapproachestoestimation?
AsnotedbytheNASintheir2013report:
“Althoughsomeanalysisanddescriptionofuncertaintyisalwaysimportant,howmanyandwhattypesofuncertaintyanalysesarecarriedoutshoulddependonthespecificdecisionproblemathand the effort to analyze specific uncertainties through probabilistic risk assessment orquantitativeuncertaintyanalysisshouldbeguidedbytheabilityofthoseanalysestoaffecttheenvironmentaldecisionathand.”
When quantitative analysis of uncertainty is warranted in an economy-wide framework, there aremultipleapproachestoconductingsuchanalysis,aslaidoutinSection5.Theseapproachesrangefromlimitedsensitivityanalysistoformalprobabilisticapproachesandhavedifferentinformation,resource,andtimerequirements,whichwillvaryonacase-by-casebasis,aswillthepotentialvalueoftheadditionalanalysis. Differences in analytical approaches in terms of the information provided and inputrequirementsleadstoquestions,suchas:
Beyond addressing model and parameter uncertainty that stem from limitations in scientificunderstanding, in some cases there may be questions about the fitness of a particular model for apurpose.Inthecontextofeconomy-wideanalysisofairregulations,thismayinclude:
Abler, D., Rodriguez, A., and Shortle, J. 1999. “Parameter Uncertainty in CGE Modeling of theEnvironmentalImpactsofEconomicPolicies.”EnvironmentalandResourceEconomics14(1):75-94.
Arndt, C., and Robinson, S., and Tarp, F. 2002. “Parameter estimation for a computable generalequilibriummodel:amaximumentropyapproach.”EconomicModelling19(3):375—398.
Arndt, C. 1996. “An Introduction to Systematic Sensitivity Analysis via Gaussian Quadrature.” GTAPTechnicalPapers.7-1-1996.
Banse,M.,Shutes,L.,Dixon,P.,VanMeijl,H.,Rimmer,M.,Tabeau,A.,andRothe,A.2013.“FactorMarketsin General Computable Equilibrium Models.” Factor Markets Working Paper 47. Seehttp://www.factormarkets.eu/system/files/FM%20WP47%20Final%20Modelling%20Report.pdf.
Bernheim,B.,Scholz,J.,andShoven,J.1989.“ConsumptionTaxationinaGeneralEquilibriumModel:HowReliable are Simulation Results?” In B. Bernheim and J. Shoven (Eds.),National Saving and EconomicPerformance.UniversityofChicagoPress.
Carson, J. 2002. “Model Verification and Validation” In Proceedings of the Winter 2002 SimulationConference1:52-58.
CBO. 2015. “Budgetary and Economic Effects of Repealing the Affordable Care Act.”https://www.cbo.gov/sites/default/files/114th-congress-2015-2016/reports/50252-Effects_of_ACA_Repeal.pdf
Harisson, G., and Vinod, H. 1992. “The Sensitivity Analysis of Applied General Equilibrium Models:CompletelyRandomizedFactorialSamplingDesigns.”ReviewofEconomicsandStatistics74(2):357-362.
Hertel, T. Hummels, D., Ivanic, M., and Keeney, R. 2007. “How Confident can e be of CGE-BaseAssessmentsofFreeTradeAgreements?”EconomicModelling24:611-635.
Horridge, J., and Pearson, K. “Systematic Sensitivity Analysiswith Respect to Correlated Variations inParametersandShocks.”GTRAPTechnicalPaperNo.30.
InstituteofMedicine. 2013.EnvironmentalDecisions in the FaceofUncertainty.Washington,DC: TheNationalAcademiesPress.
Iooss, B. and Lemaitre, P. 2015. “A Review of Global Sensitivity Analysis Methods.” In UncertaintyManagementinSimulation-OptimizationofComplexSystems.Dellino,G.andMeloni,C.(eds).SpringerUS.
IPCC. 2010. “Guidance Note for Lead Authors of the IPCC Fifth Assessment Report on ConsistentTreatment of Uncertainties.” https://www.ipcc.ch/pdf/supporting-material/uncertainty-guidance-note.pdf
Mohora, M. 2006. “RoMod: A Dynamic CGE Model for Romania.” Doctoral Dissertation. ErasmusUniversityRotterdam.Availableat:http://repub.eur.nl/pub/7455/few_mohora_20060217_thesis.pdf
Pagan, A., and Shannon, J. 1985. “Sensitivity Analysis for Linearized Computable General EquilibriumModels.”InJ.PiggottandJ.Whaley(Eds.),NewDevelopmentsinAppliedGeneralEquilibriumAnalysis(pp.104—118).CambridgeUniversityPress.
34
Pannell, D. 1997. “Sensitivity Analysis of Normative Economic Models: Theoretical Framework andPracticalStrategies.”AgriculturalEconomics16(2):139-152.
Sokolov, A., Stone, P., Forest, C., Prinn, R., Sarofim, M., Webster, M., Paltsev, S., and Schlosser, C.“Probabilistic Forecast for Twenty-First-CenturyClimateBasedonUncertainties in Emissions (WithoutPolicy)andClimateParameters.JournalofClimate22:5175-5204.
Thompson, T.M., Saari, R.K., Selin, N.E.. 2014. “Air Quality Resolution for Health Impact Assessment:InfluenceofRegionalCharacteristics.”AtmosphericChemistryandPhysics14:969-978.
Valenzuela, E., Hertel, T., Keeney, R., and Reimer, J. 2007. “Assessing Global Computable GeneralEquilibrium Model Validity using Agricultural Price Volatility.” American Journal of AgriculturalEconomics89(2):383-397.
Webster,M., Paltsev, S., Parsons, J., Reilly, J., and Jacoby, H. 2008. “Uncertainty in Greenhouse GasEmissionsandCostsofAtmosphericStabilization.”MITJointProgramontheScienceandPolicyofGlobalChangeReportNo.165.