Model Validation as an Integrated Social Process

1Rockefeller Collegeof Public Affairs and Policy

University at AlbanyState University of New York

Model Validation asModel Validation asan Integrated Social Processan Integrated Social Process

George P. RichardsonRockefeller College of Public Affairs and Policy

University at Albany - State University of New [email protected]



What do we mean by ‘validation’?What do we mean by ‘validation’?• No model has ever been or ever will be thoroughly

validated. …‘Useful,’ ‘illuminating,’ or ‘inspiring confidence’ are more apt descriptors applying to models than ‘valid’ (Greenberger et al. 1976).

• Validation is a process of establishing confidence in the soundness and usefulness of a model. (Forrester 1973, Forrester and Senge 1980).



The classic questionsThe classic questions• Not ‘Is the model valid,’ but

• Is the model suitable for its purposes and the problem it addresses?

• Is the model consistent with the slice of reality it tries to capture? (Richardson & Pugh 1981)



The system dynamics modeling processThe system dynamics modeling process

SystemConceptualization

ModelFormulationRepresentation of

Model Structure

Comparison andReconcilation

Perceptions ofSystem Structure

Empirical andInferred Time

Series

Comparison andReconciliation.

Deduction OfModel Behavior

Adapted from Saeed 1992



Processes focusing on system structureProcesses focusing on system structure

EmpiricalEvidence



Model Structure



Mental Models,Experience,Literature

Diagramming andDescription Tools



Processes focusing on system behaviorProcesses focusing on system behavior

EmpiricalEvidence


ModelFormulation

Literature,Experience


Series



ComputingAids



Two kinds of validating processesTwo kinds of validating processes

EmpiricalEvidence



Model Structure



Mental Models,Experience,Literature

Literature,Experience


Series



Diagramming andDescription Tools

ComputingAids

StructureValidatingProcesses

BehaviorValidatingProcesses



The classic testsThe classic testsFocusing on

STRUCTUREFocusing on BEHAVIOR

Testing SUITABILITY for PURPOSES

• Dimensional consistency• Extreme conditions• Boundary adequacy

• Parameter insensitivity• Structure insensitivity

Testing CONSISTENCY with REALITY

• Face validity• Parameter values

• Replication of behavior• Surprise behavior• Statistical tests

Contributing to UTILITY & EFFECTIVENESS

• Appropriateness for audience

• Counterintuitive behavior• Generation of insights

Forrester 1973, Forrester & Senge 1980, Richardson and Pugh 1981



Validation is present at every stepValidation is present at every step• Conceptualizing:

• Do we have the right people? • The right dynamic problem definition? • The right level of aggregation?

• Mapping: Developing promising dynamic hypotheses• Formulating: Clarity, logic, and extremes• Simulating: Right behavior for right reasons• Deciding: Implementable conclusions• Implementing: Requires conviction!



Do we have the right people?Do we have the right people?



Problem frame stakeholder mapProblem frame stakeholder map

Weak opponents Strong opponents

Weak supporters Strong supporters

Weak StrongStakeholder Power

High

Low

Low

High

Opp

ositi

onSu

ppor

t

Prob

lem

Fra

me

Bryson, Strategic Planning for Public and Nonprofit Organizations



Power versus Interest gridPower versus Interest grid

Subjects Players

Crowd Context setters

Weak StrongPower

High

Low

Inte

rest

Eden & Ackerman 1998



Pursuing validity in mappingPursuing validity in mapping• Think causally, not correlationally• Think stocks and flows, even if you don’t draw

them• Use units to make the causal logic plausible,

even if you don’t write them down• Be able to tell a story for every link and loop• Move progressively from less precise to more

precise -- from informal map to formal map



The standard cautionsThe standard cautions

Carbon inatmosphere

Prejudice

Discrimination

Opportunities forthe minority

Achievements ofthe minority

Understandingsof the system

Systemconceptualization

Model formulation& testing

Understandingsof the model

Carbon in algae,plants & trees

Carbon inherbivores

Carbon incarnivores

Carbon in soil



These arrows mean ‘and then’These arrows mean ‘and then’

Carbon inatmosphere

Prejudice

Discrimination








Carbon inherbivores

Carbon incarnivores

Carbon in soil

• We start with some understandings of the problem and its systemic context, and then we conceptualize (map) the system.

• Then we build the beginnings of a model, which we then test to understand it.

• Then we reformulate, or reconceptualize, or revise our understandings, or do some of all three, and then continue…



Arrows here are Arrows here are flowsflows of material of material

Carbon inatmosphere

Prejudice

Discrimination








Carbon inherbivores

Carbon incarnivores

Carbon in soil

The words here represent stocks.

This is not a causal diagram.



Only this one is a Only this one is a causalcausal loop loop

Carbon inatmosphere

Prejudice

Discrimination








Carbon inherbivores

Carbon incarnivores

Carbon in soil

No explicit stocks or flows, no clear units, but it tells a compelling story – It’s a good start.



Project modeling core structureProject modeling core structure

Work tobe done

Workreally done

Undiscoveredrework

Knownrework rework

discovery

Work in processbeginning

workcompleting

work

doing workincorrectly

startingrework



Identical structure Identical structure without explicit stocks and flowswithout explicit stocks and flows

beginningwork

completingwork

doing workincorrectly

Known rework

reworkdiscovery

startingrework

Undiscoveredrework

Work inprocess

Work reallydoneWork to be

done



Pursuing validity writing equationsPursuing validity writing equations• Recognizable parameters• Robust equation forms• Phase relations • Richardson’s Rule: Every complicated, ugly,

excessively mathematical equation and every equation flaw saps confidence in the model.



Modeling conflict within & between nationsModeling conflict within & between nations

Population

LateralpressureInternational

conflict

Consequences ofconflict

Internalstress

Technology

Populationgrowth rateTechnologygrowth rate

Domesticconflict

+

+

+

--

+

+

+

+

-

+

+

Domesticadaptation

Potential forconflict

Potential forinternational

conflict

Adaptation



Complexity & flaws destroy confidenceComplexity & flaws destroy confidence

• P of int'l conflict = DELAY FIXED ((Lateral pressure/10*Military force effect/Trade and bargaining leverage + International conflict)/Lateral conflict break point, 1 , 0)

• FlawsComplexity, discreteness, units confusion and disagreement, disembodied parameter, confusion of the effect of a concept [leverage] with the concept itself, and the wonder what keeps this probability between 0 and 1?



Robust equation formsRobust equation forms

CumulativeprogressProgress



Causal mish-mashCausal mish-mash


WorkersHours per person

per day

Normal effectiveness(tasks/hour)

Effect ofmotivation

Effect ofschedulepressure

Effect of ...

Workweek(days)



Robust equation formulationsRobust equation formulations


Effort(hours/month)

Effectiveness(tasks/hour)





Effort(hours/month)



per day

Workweek(days)





Effort(hours/month)



Effect ofmotivation


Effect of ...





Effort(hours/month)



per day


Effect ofmotivation


Effect of ...

Workweek(days)



Pursuing validity in equations: PhasingPursuing validity in equations: Phasing

Integratedinformation

Unintegratedinformation integrating info

Resourcesdevoted to info

integration adding to resourcesto integration

subtractingresources tointegration

Problemsgeneratedintegrating

info

generatingproblems

Effort tointegrating info

Ease ofintegrating

info

Willingness to cedecontrol of info

Perceived valueof integratedinformation

Problems generatedper info unit integrated

Pressure to allocateresources elsewhere

(R) Problemscompound

(R) Successenhances resources

(B) Lowhanging fruit

(B) Problemsimpede progress

Unitegratedinfo within

scopeScope of

integration effort

(B) Problemsthreaten scope

(R) Perceived valueenhances scope

(B) Problems robresources



Phase relationsPhase relations Integratedinformation

Resourcesdevoted to info

integration adding to resourcesto integration


Constant Perceived Value suggests continually rising Resources, but that doesn’t seem correct



Phase relationsPhase relations

Resourcesallocated tointegration

project adding to resourcesto integration


Resourcesplanned to info

integration

Time to allocateresources

Here, the Perceived Value of Integrated Information sets a planned level of resources



Pursuing validity fitting to dataPursuing validity fitting to data• Generally, a weak test of model validity• Whole-model procedures

• Optimization• Partial-model procedures• Reporting results

• Graphically• Numerically: Theil statistics



Example of weakness of fitting to dataExample of weakness of fitting to data• Logistic curve

• dx/dt = ax - bx2

• Gompertz curve• dx/dt = ax - bx ln(x)

CumulativeproductionPetroleum

production

Discovery / productionexperience & technology

Constraints from theresource remaining

(R)

(B)



Cum Production2 M2 M1 M1 M0018801902192419461968199020122034205620782100Time (year)

Fitting global petroleum with LogisticFitting global petroleum with LogisticProduction40,00040,00020,00020,0000018801902192419461968199020122034205620782100Time (year)



Cum Production4 M2 M2 M1 M0018801902192419461968199020122034205620782100Time (year)

Fitting global petroleum with GompertzFitting global petroleum with GompertzProduction40,00030,00020,00010,000018801902192419461968199020122034205620782100Time (year)



Presenting model fit visuallyPresenting model fit visually

Time1975 1978 1981 1983 1986

-4.00e+9

-2.00e+9

0.0

2.00e+9

4.00e+9

InsuranceAssets Hist_assets

3/21/89 9:00:46 AMTime

1975 1978 1981 1983 1986-4.00e+9

-2.00e+9

0.0

2.00e+9

4.00e+9

LossLiability Hist_liabilities

3/21/89 9:00:46 AM

Time1975 1978 1981 1983 1986

-0.800

-0.400

0.0

0.400

0.800

Prem_inc_granted Hist_prem_increases

3/21/89 9:00:46 AMTime

1975 1978 1981 1983 1986-4.00e+9

-2.00e+9

0.0

2.00e+9

4.00e+9

InsuranceSurplus Hist_surplus

3/21/89 9:00:46 AM



Presenting model fit numericallyPresenting model fit numerically• Theil statistics, for example

• Based on a breakdown of the mean squared error:

• 1 = Bias + Variation + Covariation

1/ n 2(Xi −Yi)∑ = 2(X −Y ) + 2( xs − ys ) + 2(1−r) xs ys



Presenting model fit numericallyPresenting model fit numerically



Learning from surprise model behaviorLearning from surprise model behavior• Have clear a priori expectations• Follow up all unanticipated behavior to

appropriate resolution• Confirm all behavioral hypotheses through

appropriate model tests (Mass 1991/1981)



Tests to reveal and resolve surprise behaviorTests to reveal and resolve surprise behavior

• Testing the symmetry of policy response (up and down)• Testing large amplitude versus small amplitude response• Testing policies entering at different points • Testing different patterns of behavior• Isolating uniqueness of equilibrium or steady state• Understanding forces producing equilibrium positions

(Mass 1991/1981)



SummarySummary• Modelers, stakeholders, problem experts, and others in

the modeling process pursue validity at every step along the way.

• We have rigorous traditions guiding model creation, formulation, exploration, and implications.

• We have a powerful, intimidating battery of tests of model structure and behavior.

• Model-based conclusions that make it through all this deserve the confidence of everyone in the process.



EpilogEpilog

• Reason is itself a matter of faith. It is an act of faith to assert that our thoughts have any relation to reality. (G.K. Chesterton)

• I have no exquisite reason for’t, but I have reason good enough. (Sir Andrew, Twelfth Night)

Model Validation as an Integrated Social Process

Documents