Sensitivity and Importance Analysis Risk Analysis for Water Resources Planning and Management Institute for Water Resources 2008.

Sensitivity and Importance Analysis

Risk Analysis for Water Resources Planning and Management

Institute for Water Resources

2008

Sensitivity Analysis DefinedStudy of how the variation in the output of a model can be apportioned, qualitatively or quantitatively, to different “sources of variation”

Including assumptionsInput uncertaintyScenario/model uncertainty

The PointComplex analysis may have dozens of input and output variables that are linked by a system of equationsAnalysts and decision makers must understand the relative importance of the components of an analysisSome outcomes and decisions are sensitive to minor changes in assumptions and input values

Sensitivity AnalysisIf it is not obvious which assumptions and uncertainties most affect outputs, conclusions and decisions the purpose of sensitivity analysis is to systematically find this out

Systematic Investigation of…

Future scenariosModel parametersModel inputsAssumptionsModel functional form

Why Sensitivity Analysis?Provides understanding of how output variables respond to changes in model inputs Increases confidence in analysis and its predictions

Assumptions SensitivityList the key assumptions (scenarios) of your analysisExplore what happens as you change/drop each one individually

Do your answers change?

Challenging assumptions can be effective sensitivity analysis

Sensitivity Analysis MethodsDeterministic one-at-a-time analysis of each factorDeterministic joint analysisScenario analysisSubjective estimatesParametric analysis--range of valuesProbabilistic analysis can be used for importance analysis

One-At-A-Time AnalysisHold each parameter constant

Expected valueRepresentative value

Let one input varyAssumptionInputParameter

Common, useful, dangerous

One-At-A-Time AnalysisDo not equate magnitude with influenceA=U(107,108), B=U(2,6)C = A + B; A dominatesC = AB; B dominates

One-At-A-Time AnalysisDependence and branching in model creates flaws with this logic

If A<50 thenC = B + 1Else C = B100

What value do we set A equal to?

Joint AnalysisChange combinations of variables at same timeEnables analysts to take dependencies explicitly into accountCan have same limitations as OAAT analysis

Subjective EstimatesSubjective estimates of uncertain values can be used to identify threshold values of importance to the risk assessment

Range of ValuesA specific (not subjective) range of values is used

E.g., 10th, 50th, 90th percentiles

Ceteris paribus approachAll possible combinations approach

All 10th percentiles, 10th with 90th and so on

Importance AnalysisHow much does each model input contribute to the variation in the output?Typically a few key inputs account for most output variation

These are your important inputs.

Not particularly good at identifying nonlinear or multivariate relationships

Habitat Units CreatedP

rob

ab

ility

HUs

0.000

0.005

0.010

0.015

0.020

0.025

0.030

0.035

0.040

30 60 90 120

Regression Sensitivity for Grand Total HUs/D14

Std b Coefficients

B=>15 Degrees C / Use this.../L10 .071

V5: water temperature / Me.../H9 .073

V6: Dissolved oxygen / Mea.../N22-.075

V15: Pool class / Use this.../O26-.082

V6: Dissolved oxygen / Mea.../H22-.105

B=>15 Degrees C / Use this.../O10-.106

V15: Pool class / Use this.../H26-.117

V5: water temperature / Me.../N9 .147

V6: Dissolved oxygen / Mea.../B22-.157

V5: water temperature / Me.../E9 .16

V5: water temperature / Me.../K9 .162

A=resident rainbow trout /.../X6-.163

B=>15 Degrees C / Use this.../U10-.194

B=>15 Degrees C / Use this.../E10 .22

A=resident rainbow trout /.../L6-.287

A=resident rainbow trout /.../R6-.45

-1 -0.75 -0.5 -0.25 0 0.25 0.5 0.75 1

Statistical MethodsApportion variation in output to inputs via

Regression analysisAnalysis of varianceResponse surface methodsFourier amplitude sensitivity test (FAST)Mutual information index (MII)Classification and regression trees (CART)

So What?When decision is sensitive to changes or uncertainties within realm of possibility then more precision and additional information may be required

More data (research)Better modelsConservative risk management

Take Away Points“What if” analysis is essential to good risk assessmentSystematic investigations of model parameters, model inputs, assumptions, model functional formEssential to good risk management

One-At-A-Time Analysis akaNominal range sensitivity analysis (NRSA) .Individually varying one model input across its range of plausible values holding all other inputs at nominal or base-case values Resulting difference in model output is called the sensitivity or swing weight of model to the varied input

Automatic Differentiation (AD). Systematic evaluation of partial derivative of model output with respect to a given model inputSimilar to NRSA

Only an arbitrarily small change is considered, rather than a possible range of values

Provides indication of local sensitivity.