Winter is coming! – how to survive the coming critical storm and demonstrate that social simulations work

Winter is Coming!– how to survive the coming critical storm

and demonstrate that social simulations work

Bruce EdmondsCentre for Policy Modelling

Manchester Metropolitan University

The “Hype Curve”

Winter is Coming! How to survive the coming critical storm and demonstrate that social simulations workBruce Edmonds, ESSA Sum Sch. 2014, Barcelona, 2

Time

Inte

rest

/Upt

ake/

Rep

utat

ion

1. N

ovel

ty

2. The “In Thing”

3. Disillusionm

ent 4. Dem

onstr

ation

of Utili

ty

5. A

ccep

tanc

e

We are here!

What will NOT work…

• Pointing out the deficiencies of rival techniques and approaches, because people rarely change what they do

• Further hyping of social simulation • Yet more demonstrations that complexity can

emerge from simpler systems interacting• Protecting social simulation within institutional

barriers (e.g. not allowing other techniques in our journals and conferences)

• Ignoring the critics, including those who need practical, useful results


Talk Outline

• Some Symptoms• Some Underlying Problems

with Suggested Solutions• Suggestions as to what we

might deliver


Symptom 1: Lack of Connection

• Model might be attractive intellectually but does not connect to anything observed or anybody’s problems

• Potential users go away unsatisfied, not having seen the point of it

• …or worse are conned to thinking such models will be useful when there is not chance of this being realised (for years)

• Instead they are concerned with only academic issues and debates and results are judged by weird or confusing criteria

• Such fields can retreat into obscuration or irrelevance, constructing elaborate systems of self-demarcation and justification


Symptom 2: Play & Display

• Some researchers have fun making and developing their model, exploring in an informal and unsystematic manner

• They start to see the world (or parts of it) in terms of their model

• They stumble upon some interesting properties that it has

• They publish the model in their enthusiasm• The model falls apart under more rigourous

inspection and is never developed into something useful to anybody


Symptom 3: ‘Heroic” Abstraction

• In a bid to attain fame, a job, citations and even Nobel (or Swiss Bank) prizes, an abstract model is proposed with surprising explanatory power and consequences

• The model is simple enough that everyone (and their dog) can understand it, play with it and produce variations of it

• It seems to have widespread application• As a result it gets cited many times• But it is difficult to relate to data and no reliable

use ever comes of the modelWinter is Coming! How to survive the coming critical storm and demonstrate that social simulations work

Bruce Edmonds, ESSA Sum Sch. 2014, Barcelona, 7

Problem 1: Having too much fun and avoiding difficulty• Since representing observed social phenomena

is so difficult we do something easier instead, such as:– Easing the criteria under which a good model is

judged to something we can do– Pretending that any model is a stepping stone to

future useful models without any evidence this is so– Kidding ourselves as to the importance or

applicability of abstract models• Social Simulations are fun to play with, thus the

temptation is to keep to the easy fun• Social simulations are difficult to defend


Example responses

• A retreat from empirical to the purely formal with only the vaguest of motivation left

• Weakening the criteria for success (e.g. skipping model validation steps)

• Adopting defensive strategies such as vagueness or ultimate specificity

• Changing the modelling goal (e.g. to a “way of thinking about things”)

• Deliberately dealing with a wrong model because it is simple or fun


Dealing with their difficulty

• Separate exploratory and analytic stages – have fun, explore then get serious and redo the modelling properly

• Downside: this requires some acceptance of the time and effort this requires

• Do not give in to the temptation to publish before the serious stage

• Use the fact that good social simulations are easy to criticise – this is an advantage!

• Moral: Use others (the community and stakeholders) to improve and check your models


Problem 2: Illusory Generality

• Much simulation work is intended to be interpreted/used as an analogy only

• That is, there is no defined relationship with known data (although it may be motivated in an imprecise way from evidence and the model parts have meaning)

• Rather each person will interpret the model in terms of what they know in their own way

• Such models can give the illusion of generality since humans have a facility to find meaning for analogies when presented with them, so can imagine its generality

• but when a more precise relationship with data is tried this falls apart (or is not even possible)


Can occur due to abstraction

• Such “analogical models” can be created though excess abstraction or simplification

• Whilst it is possible to make a model less general by making parts of the model more specific, the opposite does not hold:– that is, abstracting parts of a model (e.g. simplifying parts) does

not make it more general, but just turns it into an analogical model that may seem general

• The reason: unless you know which parts are irrelevant, then the chances are that the resulting model has no validity anywhere – i.e. zero real generality

• For example: if you had a polynomial model and simplified it to a constant, its generality would be limited to where that constant happened to be the answer!

• Such models may only have analogical use left to themWinter is Coming! How to survive the coming critical storm and demonstrate that social simulations work


An example

• Axelrod’s “evolution of cooperation” model does not refer to anything in particular (Axelrod 19??)

• It is rather an abstract exploration of ideas• Each time someone attempts to apply it to some

observed data/evidence:1. this is difficult because the model does not really

related to data and

2. they have to invent a new way to do this each time (this is an indicator of an analogical model)

• Moral: just because you can imagine a model working in a general way does not make it so (Kuhn’s “theoretical spectacles” 1976)


Problem 3: Unclear Purpose

• The same model can be used for different purposes (or attempted to be so used)

• A common mistake is to: play with a model, (inevitably) see the world through the model (analogy again), and rush to publish it but…– not declare their purpose for the model– indeed I think often authors scrabble around to

justify their model and do not decide a purpose or conflate many different purposes

– so the model cannot be judged clearly and its ‘added value’ is unclear


Model purposes

• There are many possible purposes for a simulation model (Epstein 2008) but there are three commonly claimed:1. A theoretical model – not intended to related precisely to

data but is an exploration of a formal structure representing ideas

2. An explanatory model – relates precisely to known data. The model makes precise an explanation of the outputs in terms of the model’s set-up

3. A predictive model – this approximates unknown data (before it is seen) from other known aspects

• Other purposes include: description, illustration, and as a tool for mediation (as in companion modelling)


Notes on model purposes

• The same model can be used (and maybe justified) for more than one purpose

• Thus a version of a good explanatory model may turn out to be a good predictive one

• Indeed in mature science theoretical, explanatory and predictive models can be closely related

• But the model needs justifying against each purpose separately

• And each purpose implies different criteria to be justified against…


A theoretical model

• Can provide new insights into ideas• In particular, can be used for: counter-examples to

assumptions, to suggest hypotheses, or as a kind of pseudo-maths

• Judged on:– soundness: no bugs, inference is correct, etc. (this will

now be assumed for all model types)– meaning: parts of model have meaning (gives guidance

for its interpretation, also all types)– provides new insights

• But they do not demonstrate anything reliable about the observed world


An explanatory model

• Makes processes/structures precise and then shows that the outputs approximate some aspects of known data

• This helps establish an explanation of the outputs in terms of the model processes, structures, parameter values etc.

• The model processes etc. need to be plausible or else the explanation you establish is in terms of implausible things!

• Judged on (soundness, meaning, novelty etc.) plus:– plausibility of explanation– low chance of accidental fitting (fits outputs without too much

tuning)– generality of explanation established


A predictive model

• Given some knowledge about the system, approximates unknown (usually future) data

• Fitting out-of-sample data does not count!• If it does predict it does not have to explain• And it can be a “black-box” model• Judged on (soundness etc) plus:

– accuracy of prediction– the amount of data needed for prediction– generality of prediction– easily computable– clear conditions of application (scope)


Examples

• the Skoda-Schelling model of racial segregation (1969, 1971, 1978) is a theoretical model, used as a counter example or an exploration of possibilities

• (Axtell et al. 2002) uses a heterogeneous spatially explicit model to explain the observed dynamics of past peoples (generative archeology)

• Nate Silver (2012) used finely crafted statistical models to predict future election results (presidential and all congress seats)


Notes on Model Purposes

• The scandal of (neoclassical) economic models is that they mix criteria for judging models: they don’t succeed in producing good explanations (implausible processes) nor do they predict (they only fit known data)

• Often the reason people are unsatisfied by social simulations is that the purpose is not clearly established but rather yet another plausible model (a YAPS) is presented and confusion as to how this is progress

• Moral: decide your purpose for each paper, be very clear about it, then demonstrate it


Problem 4: Social simulation is different!Trying to fit social simulation into existing expectations, practices and norms causes problems, because (on the whole) they:

– are not probabilistic– do not predict– are not fitted to some in-sample data– are too complex to fully understand– connect the micro and macro levels formally

• If you pretend to be otherwise you will either immediately disappoint or (worse) be believed and deeply disappoint later

• Much better to tackle these head-on and deliver what social simulation can do


Probabilistic vs. Possibilistic

• Most social simulations will not produce reliable probability distributions of outcomes

• This is because the ‘unexplained variation’ comes from contingency, context-dependency, complication and complexity rather than randomness – people just do not behave randomly!

• Rather our simulations can produce real possibilities – unfolding ‘trajectories’ that might arise (not all of them but some that we could not otherwise foresee)

• Upside: we can start to deal with uncertainty and not merely with risk (Knight 1921), providing risk-analyses that go beyond the simply improbable


Prediction

• Stakeholders and others will often ask for predictions – do not give it to them!

• This places responsibility in the wrong hands• Giving a policy maker a prediction is like giving a sharp

knife to a child – someone will get hurt and you will be blamed

• Rather use the possibilistic analysis to design indicators and visualisations of the possibilities indicated which the policy maker can use to get early warning of emerging trends and so better ‘steer’ policy making

• Downside: some stakeholders will look elsewhere for those (conmen) that claim to predict

• Upside: we can do this in a way no one else can since we connect micro and macroWinter is Coming! How to survive the coming critical storm and demonstrate that social simulations work


Descriptiveness of Social Simulation• People are so used to simulations with ‘magic’ numbers in

them (free parameters) that they do not understand the straight-forwardness of (much) social simulation

• It is true that we do not know the values or exact set-up of many aspects of our simulations, but these are (usually) potentially discoverable not arbitrary

• Our creations can be seen as complex, contingent theories whose shortcomings might be fixed by future researchers rather than be right first time

• Downside: we need to look to a more collective way of working with lots of openness and sharing

• Upside: they are good at suggesting new empirical questions and are amenable to fixing


Complication and Complexity• There is no reason to suspect that the (social and cognitive)

world is made for our benefit as researchers• Thus it is unreasonable to suppose that for each modelling

target there must be a simple model that is adequate to the purpose of trying to model it

• (Unless we choose to use inadequate models) there is no avoiding making models that we will not fully understand!

• Rather we should use our models to gradually stage abstraction

• Downside: we have to get used to complicated models and the problems they cause, e.g. extended model checking and analysis of model behaviour

• Upside: complicated social simulations can form a bridge to other techniques and does so making the representation step (and hence assumptions) preciseWinter is Coming! How to survive the coming critical storm and demonstrate that social simulations work


An Example from the ‘SCID’ Project (www.scid-project.org)


Data-Integration Simulation Model

Micro-Evidence Macro-Data

Abstract Simulation Model 1

Abstract Simulation Model 2

SNA Model Analytic Model

Integration

• Due to its atheoretical and straight-forward nature, its flexibilty and its ability to connect micro and macro levels…

• …agent-based simulation is an ideal tool for integrating:– different kinds of evidence (narrative, time-series, survey,

expert/stakeholder opinion, validated theory, etc.)– different kinds of technique (statistics, data-mining, social

network analysis, qualitative approaches, participatory approaches, etc.)

• Upside: agent-based simulation reaches the parts other techniques do not reach and so can play a key role connecting them together


Summary: What can we deliver?

• Precise explanations of meso and macro social phenomena

• Counter-examples (to commonly held assumptions, showing when prediction is not possible etc.)

• Models maximally open to criticism and improvement – not only when they are not so good but where and how they are wrong

• Models that bridge between other models and techniques (including data/evidence)

• Possibilistic risk analyses• Design of early-warning indicators and visualisations of

emerging trendsWinter is Coming! How to survive the coming critical storm and demonstrate that social simulations work


Choosing the harder (but ultimately more productive) road

“It is better to be wrong clearly than vaguely right” Pat Suppes

• To accept partial failure as normal and not shamefull• To build upon other’s models/work iteratively as well as

producing novel simulations• Be as careful and as rigorous as you can with your

modelling practice• Be open with your documentation, model code, output

results and method so that it is maximally available for critique and improvement

• Moral 1: its better to try to solve the real problems and often fail than succeed at unreal ones

• Moral 2: we will only succeed as a collective enterprise over a long time, not via quick, ‘heroic’ individual effortsWinter is Coming! How to survive the coming critical storm and demonstrate that social simulations work


The End

Bruce Edmondshttp://bruce.edmonds.name

Centre for Policy Modellinghttp://cfpm.org

These slides will be at: http://slideshare.com/BruceEdmonds

Winter is coming! – how to survive the coming critical storm and demonstrate that social simulations work

Education

essa sum sch

coming critical

social simulation

policy maker

social simulations

theoretical

model processes

predictive