PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:

PUBL0050: Advanced Quantitative Methods

Lecture 6: Synthetic Control Method

Jack Blumenau

1 / 50

Course outline

1. Potential Outcomes and Causal Inference2. Randomized Experiments3. Selection on Observables (I)4. Selection on Observables (II)5. Panel Data and Difference-in-Differences6. Synthetic Control7. Instrumental Variables (I)8. Instrumental Variables (II)9. Regression Discontinuity Designs10. Overview and Review

2 / 50

Lecture outline

Motivation

Synthetic Control

Inference

Additional application

Conclusion

3 / 50

Motivation

Comparative case studies have a long history in applied political science:

• Qualitative: “thick” description of the context/features of two or moreinstances of specific phenomena. Aim to describe contrasts orsimilarities across the cases and reason inductively about causality

• Quantitative: more explicitly causal, using (normally) aggregate datafrom one treated unit and a small set of control units. Often based on‘natural experiments’ where a shock affects one unit, but not others.

Unifying logic:

Systems as similar as possible with respect to as many features aspossible constitute the optimal samples for comparative enquiry.

Przeworski and Tenure, 1970, p. 32

4 / 50

Quantitative Comparative Case Studies

• Estimate effects of events or policy interventions that take place at anaggregate level

• Types of unit: cities, states, countries, etc

• Types of intervention: passage of laws, economic shocks, etc

Approach:

• Compare the evolution of an aggregate outcome for the unit affectedby the intervention to the evolution of the same outcome for somecontrol group

• e.g. Card (1990), Card and Krueger (1994), Abadie and Gardeazabal(2003)

5 / 50

Quantitative Comparative Case Studies

Advantages:

• Policy interventions often take place at an aggregate level• Aggregate/macro data are often available

Problems:

• Selection of control group is often ambiguous• Standard errors do not reflect uncertainty about the ability of thecontrol group to reproduce the counterfactual of interest

Solution:

• If you don’t have a good control group: synthesize one

6 / 50

Running example

Reunification of West and East GermanyWhat were the economic effects of reunification on the West Germaneconomy? Many economic historians argue that reunification had largenegative economic costs, but identification is difficult because there is noobvious country with which we can compare the growth trajectory of WestGermany. Abadie et al (2015) estimate the effects of reunification bycomparing the actual time series for West Germany with a syntheticcontrol group which provides the counterfactual.

• Outcome: GDP per capita (inflation adjusted)• Treatment: Reunification (1 for W. Germany after 1990, 0 otherwise)• Time: Years (1960 to 2003)

7 / 50

What should be the control group?

What is the most appropriate control group for evaluating the effects ofreunification on West Germany in 1990?

• Geographical/cultural: Austria?• Economic: USA?• Average: OECD countries?

The choice of the control group matters!

8 / 50

1960 1970 1980 1990 2000

West Germanyrest of OECD sample

reunification

9 / 50

1960 1970 1980 1990 2000

West GermanyMean of OECD sample

reunification

10 / 50

1960 1970 1980 1990 2000

West GermanyAustria

reunification

11 / 50

1960 1970 1980 1990 2000

West GermanyUSA

reunification

12 / 50

1960 1970 1980 1990 2000

West GermanyUSA and Austria

reunification

13 / 50

Synthetic control moves away from using a single control unit or a simpleaverage of control units.

Instead we use a weighted average of the set of control or “donor” units.

Rather than assuming that either the USA or Austria are similar to W.Germany, we calculate a weighted average (the synthetic control) which ismore similar to West Germany than any individual country.

IntuitionWhen we only have a few aggregate units, a ‘synthetic’ combination ofcontrol units may do a better job of reproducing the characteristics of thetreated unit than any one unit alone.

14 / 50

Synthetic Control

Notation

DefinitionFor units 𝑗 ∈ 1, ..., 𝐽 + 1:• Unit 1 is the unit of interest (which receives the treatment)

• Units 2 to 𝐽 + 1 are the ‘donor pool’ or potential comparison unitsTime periods ∈ 1, ..., 𝑇 :• Pre-treatment period: 𝑡 = 1, ..., 𝑇0

• Post-treatment period: 𝑡 = 𝑇0 + 1, ..., 𝑇Potential outcomes:

• 𝑌 𝑁𝑖𝑡 = outcome for unit 𝑖 at time 𝑡 in the absence of the intervention

• 𝑌 𝐼𝑖𝑡 = outcome for unit 𝑖 at time 𝑡 when exposed to the intervention

15 / 50

Notation

Estimand

𝛿1𝑡 = 𝑌 𝐼1𝑡 − 𝑌 𝑁

1𝑡 = 𝑌1𝑡 − 𝑌 𝑁1𝑡 for all 𝑡 > 𝑇0

i.e. the treatment effect on the treated unit in the post-treatment periods.

Problem

We cannot observe 𝑌 𝑁1𝑡

Why? → Fundamental problem of causal inference.

The critial question, as always, is how should we impute 𝑌 𝑁1𝑡 ?

15 / 50

Imputing 𝑌 𝑁1𝑡

1. Matching

• For each time period 𝑡, find the 𝑀 ‘closest’ units to unit 1 and averagethe observed outcomes:

𝑌 𝑁1,𝑡=1 = 1

𝑀𝑀

∑𝑚=1

𝑌𝑗𝑚(1),𝑡=1

2. Diff-in-diff

• Add the average change in outcome for the control group to thetreated unit’s outcome in the pre-treatment period

𝑌 𝑁1,𝑡=1 = 𝑌1,𝑡=0 + ( 𝑌0,𝑡=1 − 𝑌0,𝑡=0)

3. Synthetic control

• Take a weighted average of the outcomes of the donor units• Weights defined by closeness to the trend of the outcome for thetreated unit in the pre-treatment period

𝑌 𝑁1,𝑡=1 =

𝐽+1∑𝑗=2

𝑤∗𝑗𝑌𝑗,𝑡=1

16 / 50

Defining the synthetic control

Definition (Synthetic control)A synthetic control is a vector of weights, 𝑊 , associated with each of theavailable 𝐽 donor units.

Compare to our three examples above. 𝑊 is a vector with…

• …equal weight for each unit (OECD average)• …0 weight for all units, except Austria where 𝑤𝑗 = 1 (Austria)• …0 weight for all units, except USA where 𝑤𝑗 = 1 (USA)• …0 weight for all units, except USA where 𝑤𝑗 = .5 and Austria where

𝑤𝑗 = .5 (USA and Austria)

There are many potential synthetic controls! The goal is to select 𝑊 suchthat the characteristics of the treated unit are best resembled by thecharacteristics of the synthetic control.

17 / 50

Estimating𝑊

Basic approach: compare treatment and control units in terms of theirpre-intervention characteristics 𝑋1 and 𝑋0. Put more weight on countriesin the donor pool that are similar to the treatment unit in terms of:

• Covariates that are predictive of post-intervention outcomes• Pre-intervention outcome values

The idea is to give more weight to units in the donor pool that closelyapproximate the treated unit in the pre-intervention period.

I.e. We look for weights, ��𝑗, such that

𝑌1,𝑡 ≈𝐽+1∑𝑗=2

��𝑗𝑌𝑗,𝑡 for all 𝑡 ∈ 1, ..., 𝑇0

18 / 50

Estimating𝑊

For each donor unit, define a weight 𝑊 = {𝑤2, 𝑤3, ..., 𝑤𝐽+1}. We findthe values of 𝑊 by minimizing the following expression:

𝑘∑𝑚=1

𝑣𝑚(𝑋1𝑚 − 𝑋0𝑚𝑊)2

1. 𝑤𝑗 ≥ 0 for 𝑗 = 2, ..., 𝐽 + 1 (only positive or zero weights allowed)2. ∑𝐽+1

2 𝑤𝑗 = 1 (weights must sum to 1)3. 𝑣𝑚 is a weight that reflects the importance of the 𝑚th variable thatwe use to measure the distance between treated and control units

19 / 50

Estimating𝑊

There are two components that we need:

1. 𝑤𝑗 – donor unit weights, indicating which units are most similar to thetreated unit

2. 𝑣𝑚 – covariate weights, indicating which covariates are mostpredictive of the outcome

Estimate by minimizing the mean squared prediction error for thepre-intervention periods:

∑𝑡=1

(𝑌1𝑡 −𝐽+1∑𝑗=2

𝑤𝑗(𝑉 )𝑌𝑗𝑡)2

IntuitionMost weight (𝑤𝑗) is put on control units which are similar to the treatedunits on covariates (𝑋1, 𝑋0) that are predictive of the outcome (𝑣𝑚) inthe pre-intervention period (𝑡 ≤ 𝑇0)

20 / 50

Estimating𝑊

SC is, at heart, a sort of difference-in-differences matching estimator.

• Diff-in-diff: establish a control group which follows a parallel trend inthe absence of treatment (note we are still assuming that theparallel-ness would continue post-treatment!)

• Matching: 𝑤𝑗 calculated using observed pre-treatment observedcovariates But Abadie et al argue that SC also accounts fornon-parallel trends:

Only units that are alike in both observed and unobserved determi-nants of the outcome variable…should produce similar trajectoriesof the outcome variable over extended periods of time.

Abadie et al, 2015, p. 498

21 / 50

Estimating𝑊 (Intuition)

Goal: minimize difference in outcome trend in pre-treatment period.

1960 1965 1970 1975 1980 1985 1990

eal −

reunification

county weightUSA 0.06UK 0.06Austria 0.06Belgium 0.06Denmark 0.06France 0.06Italy 0.06Netherlands 0.06Norway 0.06Switzerland 0.06Japan 0.06Greece 0.06Portugal 0.06Spain 0.06Australia 0.06New Zealand 0.06

The Synth package in R will automate this optimization problem for us.

22 / 50

1960 1965 1970 1975 1980 1985 1990

eal −

reunification

county weightAustria 0.14Japan 0.08Netherlands 0.08USA 0.07Switzerland 0.07UK 0.06Denmark 0.06France 0.06Spain 0.06Australia 0.06Italy 0.05Norway 0.05Belgium 0.04Greece 0.04Portugal 0.04New Zealand 0.03

22 / 50

1960 1965 1970 1975 1980 1985 1990

eal −

reunification

county weightAustria 0.18Netherlands 0.13USA 0.11UK 0.07Greece 0.07New Zealand 0.07Japan 0.07Switzerland 0.05Belgium 0.04Denmark 0.04France 0.04Norway 0.04Spain 0.04Australia 0.04Portugal 0.01Italy 0.00

22 / 50

1960 1965 1970 1975 1980 1985 1990

eal −

reunification

county weightAustria 0.28USA 0.10Switzerland 0.09Netherlands 0.08Denmark 0.07Greece 0.07New Zealand 0.07Japan 0.07Belgium 0.03France 0.03Norway 0.03Portugal 0.03Spain 0.03Australia 0.03UK 0.00Italy 0.00

22 / 50

1960 1965 1970 1975 1980 1985 1990

eal −

reunification

county weightAustria 0.42USA 0.22Japan 0.16Switzerland 0.11Netherlands 0.09UK 0.00Belgium 0.00Denmark 0.00France 0.00Italy 0.00Norway 0.00Greece 0.00Portugal 0.00Spain 0.00Australia 0.00New Zealand 0.00

22 / 50

1960 1965 1970 1975 1980 1985 1990

eal −

reunification

county weightAustria 0.42USA 0.22Japan 0.16Switzerland 0.11Netherlands 0.09UK 0.00Belgium 0.00Denmark 0.00France 0.00Italy 0.00Norway 0.00Greece 0.00Portugal 0.00Spain 0.00Australia 0.00New Zealand 0.00

The Synth package in R will automate this optimization problem for us. 22 / 50

Interpreting𝑊 (country weights)

Country weights in synthetic Germany:

County Weight County WeightAustria 0.42 France 0.00USA 0.22 Italy 0.00Japan 0.16 Norway 0.00Switzerland 0.11 Greece 0.00Netherlands 0.09 Portugal 0.00UK 0.00 Spain 0.00Belgium 0.00 Australia 0.00Denmark 0.00 New Zealand 0.00

23 / 50

Interpreting𝑊 (country weights)

Country weights in synthetic Germany (SC and OLS):

Country Synth Reg Country Synth RegAustria 0.42 0.26 France 0.00 0.04USA 0.22 0.13 Italy 0.00 -0.05Japan 0.16 0.19 Norway 0.00 0.04Switzerland 0.11 0.05 Greece 0.00 -0.09Netherlands 0.09 0.14 Portugal 0.00 -0.08UK 0.00 0.06 Spain 0.00 -0.01Belgium 0.00 -0.00 Australia 0.00 0.12Denmark 0.00 0.08 New Zealand 0.00 0.12

Regression weights can be greater than 1 or less than zero → extrapolationoutside of the support of control units.

Extrapolation is not possible in the SC case because the weights are boundbetween 0 and 1.

24 / 50

Interpreting𝑊 (assessing balance)

GDP predictor means:

Treated Synthetic Rest of OECD SampleGDP per-capita 15808.9 15802.2 8021.1Trade openness 56.8 56.9 31.9Inflation rate 2.6 3.5 7.4Industry share 34.5 34.4 34.2

Schooling 55.5 55.2 44.1Investment rate 27.0 27.0 25.9

25 / 50

Interpreting 𝑣

Which variables are most important for determining the synthetic control?

Variable 𝑣GDP per-capita 0.44Investment rate 0.24Trade openness 0.13Schooling 0.11Inflation rate 0.07Industry share 0.00

The weights 𝑣1, … , 𝑣𝑘 should reflect the predictive value of the covariates.

26 / 50

27 / 50

Causal effects

Weighting donor units leads to a synthetic unit with a similar outcometrend in the pre-intervention period as the treated unit.

Given ��, an unbiased estimator of 𝛿1𝑡 is:

𝛿1𝑡 = 𝑌1,𝑡 −𝐽+1∑𝑗=2

𝑤𝑗𝑌𝑗,𝑡 for 𝑡 ∈ {𝑇0 + 1, … , 𝑇 }

• 𝑌1,𝑡 is the outcome for the treated unit in post-treatment period 𝑡• ∑𝐽+1

𝑗=2 𝑤𝑗𝑌𝑗,𝑡 is the outcome for the synthetic control unit inpost-treatment period 𝑡

• 𝛿1,𝑡 is the ATT for time period 𝑡

28 / 50

Causal effect I

1960 1970 1980 1990 2000

West Germanysynthetic West Germany

Reunification

29 / 50

Causal effect II

1960 1970 1980 1990 2000

Reunification

30 / 50

Inference

Asymptotic inference in synthetic control

Standard errors from regression/t-tests are typically used to characteriseuncertainty about aggregate data:

• i.e. use a sample of restaurants in NJ and PA to estimate employmenttrends in each state

• standard errors reflect unavailability of aggregate data onemployment

So, if we use aggregate data, is there zero uncertainty? No!

• We do not have perfect information about potential outcomes, evenwhen we use aggregate data

• We have uncertainty about the potential outcome under control forthe treated unit

But, because the number of units is small in most SC applications, largesample inferential techniques are not appropriate.

31 / 50

Permutation inference

Instead, we turn to an alternative inference technique: permutationinference.

1. Calculate the the test-statistic under the actual treatment assignment2. Calculate the distribution of the test-statistic under alternative

treatment assignments assuming treatment effects of zero3. Assess whether the ‘true’ test-statistic is unlikely under the null

distribution of treatment effects

Here, this implies constructing a synthetic control for every country in oursample, summarising the treatment effect, and comparing it to thetreatment effect in West Germany.

32 / 50

Permutation inference (example)

What is the test-statistic here? Two components:

• Treatment effect (analogous to 𝛽 in a regression)• Pre-treatment difference between unit and SC (analogous to 𝑆𝐸( 𝛽))

For each unit calculate:

RMSE𝑗,𝑇1 = ⎛⎜⎝

1𝑇1

𝑇∑

𝑡=𝑇0

(𝑌1,𝑡 −𝐽+1∑𝑗=2

𝑤𝑗𝑌𝑖,𝑡)2⎞⎟⎠

RMSE𝑗,𝑇0 = ⎛⎜⎝

1𝑇0

∑𝑡=1

(𝑌1,𝑡 −𝐽+1∑𝑗=2

𝑤𝑗𝑌𝑖,𝑡)2⎞⎟⎠

Where RMSE𝑗,𝑇0characterises the lack of fit between a unit and it’s SC in

the pre-treatment period, and RMSE𝑗,𝑇1is the equivalent for the

post-treatment period.33 / 50

Given these, the test-statistic is:

RMSE𝑗,𝑇1

RMSE𝑗,𝑇0

= Pre-intervention ‘fit’Post-intervention ‘fit’

Intuition:

• More confident that the effect is different from zero when theestimated treatment effect is larger (RMSE𝑗,𝑇1

• Less confident that the effect is different from zero when thepre-treatment fit with the SC is larger (RMSE𝑗,𝑇0

P-value: how likely would it be to observe a ratio as large as the one weactually observe if the treatment effects were zero and we picked onecountry at random?

34 / 50

Portugal

Denmark

France

Netherlands

Austria

Switzerland

Belgium

Australia

New Zealand

Greece

Norway

West Germany

5 10 15

Post−Period RMSE / Pre−Period RMSE

35 / 50

Placebos in space

1960 1970 1980 1990 2000

West GermanyDonor countries

36 / 50

Placebos in time

1960 1965 1970 1975 1980 1985 1990

per−

)West Germanysynthetic West Germany

placebo reunification

37 / 50

Additional application

California’s Proposition 99

Anti-smoking legislation and cigarette consumptionIn 1988, California passed comprehensive tobacco control legislation. Thiswas a package of measures that included a tax increase, more earmarkedspending to anti-smoking health initiatives, and anti-smoking mediacampaigns. We will investigate the effect of this legislation on cigaretteconsumption in California using synthetic control methods.

• Outcome variable (Y): Per capita cigarette sales (packs)• Treatment (D): 1 for CA after 1988, 0 for all other periods/states• Time (T): 1970 to 2000

(All states which passed similar legislation are excluded from the donor pool.)

38 / 50

1970 1975 1980 1985 1990 1995 2000

per−

)Californiarest of the U.S.

Passage of Proposition 99

39 / 50

1970 1975 1980 1985 1990 1995 2000

per−

)Californiasynthetic California

40 / 50

State Weights in Synthetic California

State WeightUtah 0.334Nevada 0.234Montana 0.199Colorado 0.164Connecticut 0.069

41 / 50

Predictor means: Real vs. Synthetic California

California Average ofVariables Real Synthetic 38 control statesLn(GDP per capita) 10.08 9.86 9.86Percent aged 15-24 17.40 17.40 17.29Retail price 89.42 89.41 87.27Beer consumption per capita 24.28 24.20 23.75Cigarette sales per capita 1988 90.10 91.62 114.20Cigarette sales per capita 1980 120.20 120.43 136.58Cigarette sales per capita 1975 127.10 126.99 132.81Note: All variables except lagged cigarette sales are averaged for the 1980-1988 period(beer consumption is averaged 1984-1988).

42 / 50

1970 1975 1980 1985 1990 1995 2000

Cigarette sales gap in CA (versus synthetic CA).

43 / 50

1970 1975 1980 1985 1990 1995 2000

) Californiacontrol states

Cigarette sales gap in all 38 states.

44 / 50

1970 1975 1980 1985 1990 1995 2000

Cigarette sales gap in states with pre-intervention MSPE < 20 ⋅ MSPECA.45 / 50

1970 1975 1980 1985 1990 1995 2000

Conclusion

Data requirements

Synthetic control has relatively low data requirements:

• Can use aggregate data (often administrative)

• e.g. economic indicators such as GDP, current-account balance, etc;political indicators such as turnout, vote share, etc

• Causal factors can be big and important

• e.g. legislation changes, macro-shocks, etc

• Units of analysis can be large

• Countries, states, regions, etc

• Does not even require full panel data for the pre-treatment period

• Can use averages of covariates rather than full panel data (useful whencovariates do not vary yearly)

48 / 50

Conclusion

The synthetic control approach…is arguably the most importantinnovation in the policy evaluation literature in the last 15 years.

Athey and Imbens, 2017

Advantages:• Builds on D&D and Matching by essentially forcing the data to exhibitparallel trends in the pre-treatment period

• Amenable to small-ish N comparisons (often easier to get data)• Clear, transparent, and easily communicable comparisons(e.g. Germany is part Austria, part USA, etc)

49 / 50

Conclusion

The synthetic control approach…is arguably the most importantinnovation in the policy evaluation literature in the last 15 years.

Athey and Imbens, 2017

Disadvantages:• Inference is not straightforward! Asymptotic inference does not workwith the SC method

• Coding is not straightforward! As you will see in the seminar :)• Pre-intervention period must be relatively large for us to trust paralleltrends holds in the post-intervention period

49 / 50

Next week

Instrumental Variables (I)

50 / 50

PUBL0050:AdvancedQuantitativeMethods Lecture6 ... · Imputing 𝑁 1𝑡 1. Matching • Foreachtimeperiod𝑡,findthe𝑀‘closest’unitstounit1andaverage theobservedoutcomes:

Documents

Imputing incomplete time-series data based on varied-window....

Imputing the Physical and Mental Summary Scores … ·...

A Guide to Imputing Missing Data with Stata Revision: 1 ·....

Analysis of Complex Survey Data Day 5, Special topics:...

Imputing Supertrees and Supernetworks from Quartets

PUBL0050:AdvancedQuantitativeMethods Lecture4 ... ·...

Labor Market Outcomes Speak Louder than Age? › PUBL0050...

The Effects of Imputing Missing Data on Ensemble Temperature...

Progetto 1000 Genomi, Imputing,...

NBER WORKING PAPER SERIES IMPUTING MISSING VALUES IN … ·...

Nearest Neighbor Methods for Imputing Missing...

A Guide to Imputing Missing Data with Stata Revision:...

Imputing Physical and Mental Summary Scores · Imputing...

Imputing a Variational Inequality Function or a Convex ...

Office of the Sheriff Fluvanna County - Frankly...

Disparaging or Imputing Mistakes or Imperfections to the...