New Zealand as a “Social Laboratory” Cook COMPASS...New Zealand as a “Social Laboratory” [A James Cook Fellowship Proposal] COMPASS Seminar Series Monday, 3 August 2015, FalePacifika.

New Zealand as a “Social Laboratory”[A James Cook Fellowship Proposal]

COMPASS Seminar SeriesMonday, 3 August 2015, Fale Pacifika

Professor Peter Davis Department of Sociology, COMPASS Research Centre

New Zealand as a “Social Laboratory”[A James Cook Fellowship Proposal]

Preamble - Making Knowledge Claims• The Year of Evaluation – RCTs• Impact of societal inequality• Improving inference with better design• The simulation approach

The

Uni

vers

ity o

f Auc

klan

dN

ew Z

eala

nd

Pickett and Wilkinson,Soc Sci Med 2015

5

The

Uni

vers

ity o

f Auc

klan

dN

ew Z

eala

nd

Pickett and Wilkinson,Soc Sci Med 2015

6

The

Uni

vers

ity o

f Auc

klan

dN

ew Z

eala

nd

Avendano article,Soc Sci Med 2012

7

The

Uni

vers

ity o

f Auc

klan

dN

ew Z

eala

nd

Pickett and Wilkinson,Soc Sci Med 2015

8

Avendano 2012

9

New Zealand Correlation = -0.92

Source: Avendano data

France

Source: Avendano data

Correlation = +0.96

11

New Zealand as a “Social Laboratory”[A James Cook Fellowship Proposal]

Preamble - Making Knowledge Claims• The Year of Evaluation – RCTs• Impact of societal inequality• Improving inference with better design• The simulation approach

The

Uni

vers

ity o

f Auc

klan

dN

ew Z

eala

nd

Causal Inference in Observational Settings

7th Wellington ColloquiumStatistics NZ30 August 2013

Professor Peter DavisUniversity of Auckland, New Zealandand COMPASS Research Centrewww.compass.auckland.ac.nz

New Zealand as a “Social Laboratory”[A James Cook Fellowship Proposal]

The

Uni

vers

ity o

f Auc

klan

dN

ew Z

eala

nd

Assessing policy counterfactuals with a simulation-based inquiry system.

Peter Davis and ColleaguesCOMPASS Research CentreUniversity of Auckland New Zealandwww.compass.auckland.ac.nz

DISCLAIMER: Access to the data used in this study was provided by Statistics New Zealand under conditions designed to give effect to the security and confidentiality provisions of the Statistics Act 1975. The results presented in this study are the work of the author, not Statistics New Zealand.

17

The

Uni

vers

ity o

f Auc

klan

dN

ew Z

eala

nd

Assessing counterfactuals

Counterfactual paradigm of causal reasoningIf the putative causal factor had not been present, we would not have observed the recorded outcome.

• Randomised Controlled Trials (RCTs)

• Experimental and quasi-experimental methods

• Observational designs and statistical analysis

Simulation techniques

18

The

Uni

vers

ity o

f Auc

klan

dN

ew Z

eala

nd

Simulation techniques

Simulation – in our case, social simulationUse of computer models (computational techniques) to “mimic” social phenomena (e.g. social processes).

• Understand phenomena better in constructing the models

• Once understood and validated, one can alter features

• Particularly useful for sub-groups and future projections

• Ability to combine different data sources in a single model

• Overcome privacy and confidentiality issues

19

The

Uni

vers

ity o

f Auc

klan

dN

ew Z

eala

nd

Model Year Locality Type Life stage Domain Software Data Funder Collaborators End-usersMOSC 2005-8 NZ ABM/MSM Adults Marriage market,

residential segregation

NetLogoRepastJava

Census Marsden UOA

PCASO 2005-8 NZ Static discrete-time MSM

Older people Health care SAS NATMEDCANZHSANHS

HRC UOANatSem

BCASO 2009-12 NZ

Dynamic discrete-time MSM

Older people Health & social care

R NZHSNZDSCensus

HRC UOANatSem

MEL-C 2009-13 NZ Children Health, education, conduct

JavaR

CHDSDMDHSPIFSTHNRCensus2006

MBIE UOANatSemStatCan

MOEMOHMOJMSDTe Puni KokiriSUPERUChildren’s Commissioner

KNOW-LAB

2013-16 World Children & young people

Health, education, conduct, etc.

Published literature

MBIE UOAStatCan

SOCLAB 2015-17 NZ Whole society

Social Modgen/OpenM++

NZLC Royal Society

UOA Open-source modelling community

TPM 2015-20 NZ Whole society

Social Census TEC UOAMOTU

LEGENDModel Data Funder

MOSC: Modelling social changePCASO: Primary care in an ageing societyBCASO: Balance of care in an ageing societyMEL-C: Modelling the early life courseKNOW-LAB: Knowledge laboratorySOC-LAB: NZ social laboratoryTPM CORE: Te Punaha Matatini

NATMEDCA: National Primary Medical CareNZHS: NZ Health SurveyNZDS: NZ Disability SurveyANHS: Australian National Health SurveyCHDS: Christchurch Health & Development StudyDMHDS: Dunedin Multidisciplinary Health & Development StudyPIFS: Pacific Island Families StudyTHNR: Te Hoe Nuku RoaNZLC: NZ Longitudinal Census

HRC: Health Research CouncilMBIE: Ministry of Business, Innovation & EmploymentRoyal Society: James Cook FellowshipTEC: Tertiary Education Commission

CollaboratorsUOA: University of AucklandNatSem: National Centre for Social & Economic Modelling, University of CanberraStatCan: Statistics CanadaMOTU: Motu Economic and Public Policy Research

Type End-usersABM: Agent based modelMSM: Micro-simulation model

MOE: Ministry of Education, MOH: Ministry of HealthMOJ: Ministry of Justice,MSD: Ministry of Social Development

Simulation at COMPASS

20

The

Uni

vers

ity o

f Auc

klan

dN

ew Z

eala

nd

Micro-simulation approach.

We start with a sample of individualsReal (studies) / synthetic (derived from Census)

We derive statistical rules to create a “virtual cohort” that mimics the “real” one

Derive rules best able to reproduce real dataApply these rules to the base file to create a synthetic sample of typical biographies through life course

We then simulate what might happen if policywere to change, by altering parameters

Using software application to test counterfactuals21

Virtual versus real cohort: family doctor visits, reading ability, and conduct problems, by year of age

22

Year Real cohort (CHDS)n=1017

Virtual cohort (simulated) n=1017

Absolute error Absolute error / CHDS mean

Family doctor visits (mean (95% CI))1 5.82 5.82 - -2 5.34 5.28 0.06 -3 3.31 3.18 0.13 -4 3.13 3.15 0.02 -5 3.22 3.12 0.10 -6 3.35 3.32 0.03 -7 2.43 2.41 0.02 -8 2.14 2.15 0.01 -9 1.96 1.90 0.06 -

10 1.65 1.68 0.03 -All years 3.24 3.20 (3.15-3.25) 0.04 1.2%

Reading ability: BURT score (mean (95% CI))8 45.3 45.3 - -9 54.4 54.7 0.3 -

10 64.1 63.7 0.4 -11 72.8 71.9 0.9 -12 79.5 78.9 0.6 -13 85.2 84.6 0.6 -

All years 66.9 66.5 (65.7-67.4) 0.4 0.6%Conduct problems (mean (95% CI))

6 10.6 10.6 - -7 24.6 24.8 0.2 -8 24.4 25.0 0.6 -9 24.7 25.3 0.6 -

10 24.9 25.6 0.7 -All years 21.8 22.3 (22.1-22.4) 0.5 2.3%

The

Uni

vers

ity o

f Auc

klan

dN

ew Z

eala

nd

Inquiry Tool

23

New Zealand as a “Social Laboratory”[A James Cook Fellowship Proposal]

ANY QUESTIONS AT THIS POINT!?

New Zealand as a “Social Laboratory”[A James Cook Fellowship Proposal]

Assessing Counterfactuals about Society• Background and concept• Central ingredients of project• COMPASS team contribution• Building blocks• Book proposal

Outline

• Background• Central ingredients of the James Cook• COMPASS team contribution• Building blocks

– New Zealand Longitudinal Census (1981-2013)– Synthetic base file– Estimating equations– Open-source micro-simulation platform

• Book proposal

Background –assessing counterfactuals

• New Zealand, an early policy pioneer– 1890-1920 seen by observers as “social laboratory”– Social policies tried out by reforming governments– A “natural experiment” in a new, fluid society

• How to draw credible inferences about policy?– RCTs, experimental and quasi-experimental designs– Non-experimental work (e.g. case studies)– Virtual “experiments”, using simulation techniques– Any precedents? Think of climate change scenarios

Background –assessing counterfactuals

• New Zealand, an early policy pioneer– 1890-1920 seen by observers as “social laboratory”– Social policies tried out by reforming governments– A “natural experiment” in a new, fluid society

• How to draw credible inferences about policy?– RCTs, experimental and quasi-experimental designs– Non-experimental work (e.g. case studies)– Virtual “experiments”, using simulation techniques– Any precedents? Think of climate change scenarios

Central Ingredients of James Cook• Three aims

– Create model of NZ pop via synthetic cohort– Statistical model from NZLC to generate cohorts– Conduct experiments, “virtual counterfactuals”

1. Constructing smaller, synthetic cohortsNeed synthetic starting file for each cohort, 1981

2. Estimating statistical model driving cohortsMethod for reproducing biographical trajectories

3. Testing “virtual counterfactuals”Particular interest in impact of social assets

Three “Virtual Counterfactuals”• Health impact of 1980s/1990s restructuring

– Blakely et al. use a cross-comparative counterfactual (Norway)– We can try returning key exogenous parameters to the long-

term pre-disruption trend line– Assess impact on health inequalities

• Long-term impact of “Working For Families”– What would have happened had “normal” settings applied?– Did the in-work tax credit work against the poor?

• Impact of social assets on valued goals and outcomes – Determine relationship of assets (e.g. social and cultural capital)

to achievement of valued goals and outcomes– Alter distribution of these non-monetary assets to assess impact

The

Uni

vers

ity o

f Auc

klan

dN

ew Z

eala

nd

Senior RF -Barry Milne

Research-Policy Collaboration – Published 2014

34

The

Uni

vers

ity o

f Auc

klan

dN

ew Z

eala

nd

Senior RF –Roy Lay-Yee

Determinants and Disparities – Published 2015

35

The

Uni

vers

ity o

f Auc

klan

dN

ew Z

eala

nd

Statistician –Jessica McLay

Regression Estimation for Dynamic Microsimulation (McLay et al.)

• ACCEPTED WITH REVISIONS (International Journal of Microsimulation)

Abstract: Microsimulation models seek to represent real-world processes and can generate extensive amounts of synthetic data. Most often, the parameters that drive the data generation process are estimated by statistical modelling techniques, such as regression. But which techniques are best suited to this purpose? We assess the performance of five regression-style estimation techniques: ordinary least squares regression with a lagged dependent variable, random effects with and without an autoregressive order 1within-unit error structure, a hybrid model combining features from both econometric fixed effects and random effects models, and a dynamic panel model estimated with system generalised method of moments. The criterion for good performance was the proximity of fit of simulated data to empirical data on various characteristics. It was found that ordinary least squares regression with a lagged dependent variable out-performed the other techniques. Random effects with autoregressive errors of the first order was the next best, followed by standard random effects. The dynamic panel model came fourth followed by the hybrid model. This empirical assessment provides practical guidance to those contemplating dynamic microsimulation and other applications using regression-style techniques of synthetic data generation.

36

Outline

• Background• Central ingredients of the James Cook• COMPASS team contribution• Building blocks

– New Zealand Longitudinal Census (1981-2013)– Synthetic base file– Estimating equations– Open-source micro-simulation platform

• Book proposal

Cohort Estimating Equations

Open-Source Simulation Software

Book Proposal

Central Ingredients of James Cook• Three aims

– Create model of NZ pop via synthetic cohorts– Statistical model from NZLC to generate cohorts– Conduct experiments, “virtual counterfactuals”

1. Constructing smaller, synthetic cohortsNeed synthetic starting file for each cohort, 1981

2. Estimating statistical model driving cohortsMethod for reproducing biographical trajectories

3. Testing “virtual counterfactuals”Particular interest in impact of social assets

New Zealand as a “Social Laboratory”[A James Cook Fellowship Proposal]

QUESTIONS, COMMENTS!