Advanced Quantitative Research Methodology, Lecture
Notes: Research Designs for Causal Inference1
Gary King
GaryKing.org
April 10, 2015
1
c
Copyright
2015 Gary King, All Rights Reserved.
Gary King ()
Research Designs
April 10, 2015
1 / 23
Reference
Kosuke Imai, Gary King, and Elizabeth Stuart. Misunderstandings
among Experimentalists and Observationalists: Balance Test Fallacies
in Causal Inference Journal of the Royal Statistical Society, Series A
Vol. 171, Part 2 (2008): Pp. 1-22
http://gking.harvard.edu/files/abs/matchse-abs.shtml
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
2 / 23
Notation
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
3 / 23
Notation
Sample of n units from a finite population of N units (typically
N >> n)
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
3 / 23
Notation
Sample of n units from a finite population of N units (typically
N >> n)
Sample selection: Ii is 1 for units selected, 0 otherwise
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
3 / 23
Notation
Sample of n units from a finite population of N units (typically
N >> n)
Sample selection: Ii is 1 for units selected, 0 otherwise
Treatment assignment: Ti is 1 for treated group, 0 control group
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
3 / 23
Notation
Sample of n units from a finite population of N units (typically
N >> n)
Sample selection: Ii is 1 for units selected, 0 otherwise
Treatment assignment: Ti is 1 for treated group, 0 control group
(Assume: treated and control groups are each of size n/2)
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
3 / 23
Notation
Sample of n units from a finite population of N units (typically
N >> n)
Sample selection: Ii is 1 for units selected, 0 otherwise
Treatment assignment: Ti is 1 for treated group, 0 control group
(Assume: treated and control groups are each of size n/2)
Observed outcome variable: Yi
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
3 / 23
Notation
Sample of n units from a finite population of N units (typically
N >> n)
Sample selection: Ii is 1 for units selected, 0 otherwise
Treatment assignment: Ti is 1 for treated group, 0 control group
(Assume: treated and control groups are each of size n/2)
Observed outcome variable: Yi
Potential outcomes: Yi (1) and Yi (0), Yi potential values when Ti is
1 or 0 respectively.
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
3 / 23
Notation
Sample of n units from a finite population of N units (typically
N >> n)
Sample selection: Ii is 1 for units selected, 0 otherwise
Treatment assignment: Ti is 1 for treated group, 0 control group
(Assume: treated and control groups are each of size n/2)
Observed outcome variable: Yi
Potential outcomes: Yi (1) and Yi (0), Yi potential values when Ti is
1 or 0 respectively.
Fundamental problem of causal inference. Only one potential
outcome is ever observed:
If Ti = 0, Yi (0) = Yi Yi (1) = ?
If Ti = 1, Yi (0) = ? Yi (1) = Yi
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
3 / 23
Notation
Sample of n units from a finite population of N units (typically
N >> n)
Sample selection: Ii is 1 for units selected, 0 otherwise
Treatment assignment: Ti is 1 for treated group, 0 control group
(Assume: treated and control groups are each of size n/2)
Observed outcome variable: Yi
Potential outcomes: Yi (1) and Yi (0), Yi potential values when Ti is
1 or 0 respectively.
Fundamental problem of causal inference. Only one potential
outcome is ever observed:
If Ti = 0, Yi (0) = Yi Yi (1) = ?
If Ti = 1, Yi (0) = ? Yi (1) = Yi
(Ii , Ti , Yi ) are random; Yi (1) and Yi (0) are fixed.
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
3 / 23
Quantities of Interest
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
4 / 23
Quantities of Interest
Treatment Effect (for unit i):
TEi
Gary King (Harvard)
≡ Yi (1) − Yi (0)
Research Designs for Causal Inference
April 10, 2015
4 / 23
Quantities of Interest
Treatment Effect (for unit i):
TEi
≡ Yi (1) − Yi (0)
Population Average Treatment Effect:
PATE ≡
N
1 X
TEi
N
i=1
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
4 / 23
Quantities of Interest
Treatment Effect (for unit i):
TEi
≡ Yi (1) − Yi (0)
Population Average Treatment Effect:
N
1 X
TEi
N
PATE ≡
i=1
Sample Average Treatment Effect:
SATE ≡
Gary King (Harvard)
1
n
X
TEi
i∈{Ii =1}
Research Designs for Causal Inference
April 10, 2015
4 / 23
Decomposition of Causal Effect Estimation Error
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
5 / 23
Decomposition of Causal Effect Estimation Error
Difference in means estimator:

 
X
1
1
D ≡ 
Yi  − 
n/2
n/2
i ∈{Ii =1,Ti =1}
Gary King (Harvard)
Research Designs for Causal Inference

X
Yi  .
i ∈{Ii =1,Ti =0}
April 10, 2015
5 / 23
Decomposition of Causal Effect Estimation Error
Difference in means estimator:

 
X
1
1
D ≡ 
Yi  − 
n/2
n/2
i ∈{Ii =1,Ti =1}

X
Yi  .
i ∈{Ii =1,Ti =0}
Estimation Error:
∆ ≡ PATE − D
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
5 / 23
Decomposition of Causal Effect Estimation Error
Difference in means estimator:

 
X
1
1
D ≡ 
Yi  − 
n/2
n/2
i ∈{Ii =1,Ti =1}

X
Yi  .
i ∈{Ii =1,Ti =0}
Estimation Error:
∆ ≡ PATE − D
Pretreatment confounders: X are observed and U are unobserved
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
5 / 23
Decomposition of Causal Effect Estimation Error
Difference in means estimator:

 
X
1
1
D ≡ 
Yi  − 
n/2
n/2
i ∈{Ii =1,Ti =1}

X
Yi  .
i ∈{Ii =1,Ti =0}
Estimation Error:
∆ ≡ PATE − D
Pretreatment confounders: X are observed and U are unobserved
Decomposition:
∆ = ∆ S + ∆T
= (∆SX + ∆SU ) + (∆TX + ∆TU )
Error due to ∆S (sample selection), ∆T (treatment imbalance), and
each due to observed (Xi ) and unobserved (Ui ) covariates
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
5 / 23
Selection Error
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
6 / 23
Selection Error
Definition:
∆S
≡ PATE − SATE
N −n
=
(NATE − SATE),
N
where NATE is the nonsample average treatment effect.
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
6 / 23
Selection Error
Definition:
∆S
≡ PATE − SATE
N −n
=
(NATE − SATE),
N
where NATE is the nonsample average treatment effect.
∆S vanishes if:
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
6 / 23
Selection Error
Definition:
∆S
≡ PATE − SATE
N −n
=
(NATE − SATE),
N
where NATE is the nonsample average treatment effect.
∆S vanishes if:
1
The sample is a census (Ii = 1 for all observations and n = N);
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
6 / 23
Selection Error
Definition:
∆S
≡ PATE − SATE
N −n
=
(NATE − SATE),
N
where NATE is the nonsample average treatment effect.
∆S vanishes if:
1
2
The sample is a census (Ii = 1 for all observations and n = N);
SATE = NATE; or
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
6 / 23
Selection Error
Definition:
∆S
≡ PATE − SATE
N −n
=
(NATE − SATE),
N
where NATE is the nonsample average treatment effect.
∆S vanishes if:
1
2
3
The sample is a census (Ii = 1 for all observations and n = N);
SATE = NATE; or
Switch quantity of interest from PATE to SATE (recommended!)
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
6 / 23
Decomposing Selection Error
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
7 / 23
Decomposing Selection Error
Decomposition:
∆S = ∆SX + ∆SU
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
7 / 23
Decomposing Selection Error
Decomposition:
∆S = ∆SX + ∆SU
∆SX = 0 when empirical distribution of (observed) X is identical in
population and sample: Fe (X | I = 0) = Fe (X | I = 1).
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
7 / 23
Decomposing Selection Error
Decomposition:
∆S = ∆SX + ∆SU
∆SX = 0 when empirical distribution of (observed) X is identical in
population and sample: Fe (X | I = 0) = Fe (X | I = 1).
∆SU = 0 when empirical distribution of (unobserved) U is identical in
population and sample: Fe (U | I = 0) = Fe (U | I = 1).
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
7 / 23
Decomposing Selection Error
Decomposition:
∆S = ∆SX + ∆SU
∆SX = 0 when empirical distribution of (observed) X is identical in
population and sample: Fe (X | I = 0) = Fe (X | I = 1).
∆SU = 0 when empirical distribution of (unobserved) U is identical in
population and sample: Fe (U | I = 0) = Fe (U | I = 1).
conditions are unverifiable: X is observed only in sample and U is not
observed at all.
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
7 / 23
Decomposing Selection Error
Decomposition:
∆S = ∆SX + ∆SU
∆SX = 0 when empirical distribution of (observed) X is identical in
population and sample: Fe (X | I = 0) = Fe (X | I = 1).
∆SU = 0 when empirical distribution of (unobserved) U is identical in
population and sample: Fe (U | I = 0) = Fe (U | I = 1).
conditions are unverifiable: X is observed only in sample and U is not
observed at all.
∆SX vanishes if weighting on X
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
7 / 23
Decomposing Selection Error
Decomposition:
∆S = ∆SX + ∆SU
∆SX = 0 when empirical distribution of (observed) X is identical in
population and sample: Fe (X | I = 0) = Fe (X | I = 1).
∆SU = 0 when empirical distribution of (unobserved) U is identical in
population and sample: Fe (U | I = 0) = Fe (U | I = 1).
conditions are unverifiable: X is observed only in sample and U is not
observed at all.
∆SX vanishes if weighting on X
∆SU cannot be corrected after the fact
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
7 / 23
Decomposing Treatment Imbalance
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
8 / 23
Decomposing Treatment Imbalance
Decomposition:
∆T = ∆TX + ∆TU
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
8 / 23
Decomposing Treatment Imbalance
Decomposition:
∆T = ∆TX + ∆TU
∆TX = 0 when X is balanced between treateds and controls:
Fe (X | T = 1, I = 1) = Fe (X | T = 0, I = 1).
Verifiable from data; can be generated ex ante by blocking or
enforced ex post via matching or parametric adjustment
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
8 / 23
Decomposing Treatment Imbalance
Decomposition:
∆T = ∆TX + ∆TU
∆TX = 0 when X is balanced between treateds and controls:
Fe (X | T = 1, I = 1) = Fe (X | T = 0, I = 1).
Verifiable from data; can be generated ex ante by blocking or
enforced ex post via matching or parametric adjustment
∆TU = 0 when U is balanced between treateds and controls:
Fe (U | T = 1, I = 1) = Fe (U | T = 0, I = 1).
Unverifiable. Achieved only by assumption or, on average, by random
treatment assignment
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
8 / 23
Alternative Quantities of Interest
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
9 / 23
Alternative Quantities of Interest
Sample average treatment effect on the treated:
X
1
SATT ≡
TEi
n/2
i∈{Ii =1,Ti =1}
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
9 / 23
Alternative Quantities of Interest
Sample average treatment effect on the treated:
X
1
SATT ≡
TEi
n/2
i∈{Ii =1,Ti =1}
Population average treatment effect on the treated:
X
1
PATT ≡ ∗
TEi
N
i∈{Ti =1}
(where N ∗ =
population)
Gary King (Harvard)
PN
i=1 Ti
is the number of treated units in the
Research Designs for Causal Inference
April 10, 2015
9 / 23
Alternative Quantities of Interest
Sample average treatment effect on the treated:
X
1
SATT ≡
TEi
n/2
i∈{Ii =1,Ti =1}
Population average treatment effect on the treated:
X
1
PATT ≡ ∗
TEi
N
i∈{Ti =1}
PN
(where N ∗ = i=1 Ti is the number of treated units in the
population)
When are these of more interest than PATE and SATE? Why never
for randomized experiments? Why usually for matching?
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
9 / 23
Alternative Quantities of Interest
Sample average treatment effect on the treated:
X
1
SATT ≡
TEi
n/2
i∈{Ii =1,Ti =1}
Population average treatment effect on the treated:
X
1
PATT ≡ ∗
TEi
N
i∈{Ti =1}
PN
(where N ∗ = i=1 Ti is the number of treated units in the
population)
When are these of more interest than PATE and SATE? Why never
for randomized experiments? Why usually for matching?
Analogous estimation error decomposition: ∆0 = PATT − D, holds:
∆0 = (∆0SX + ∆0SU ) + (∆0TX + ∆0TU )
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
9 / 23
Effects of Design Components on Estimation Error
Design Choice
Gary King (Harvard)
∆ SX
Research Designs for Causal Inference
∆SU
∆ TX
∆ TU
April 10, 2015
10 / 23
Effects of Design Components on Estimation Error
Design Choice
Gary King (Harvard)
∆ SX
Research Designs for Causal Inference
∆SU
∆ TX
∆ TU
April 10, 2015
10 / 23
Effects of Design Components on Estimation Error
Design Choice
Random sampling
Gary King (Harvard)
∆ SX
avg
= 0
Research Designs for Causal Inference
∆SU
avg
∆ TX
∆ TU
= 0
April 10, 2015
10 / 23
Effects of Design Components on Estimation Error
Design Choice
Random sampling
Complete stratified random sampling
Gary King (Harvard)
∆ SX
avg
= 0
=0
Research Designs for Causal Inference
∆SU
avg
∆ TX
∆ TU
= 0
= 0
avg
April 10, 2015
10 / 23
Effects of Design Components on Estimation Error
Design Choice
Random sampling
Complete stratified random sampling
Focus on SATE rather than PATE
Gary King (Harvard)
∆ SX
avg
= 0
=0
=0
Research Designs for Causal Inference
∆SU
avg
∆ TX
∆ TU
= 0
= 0
=0
avg
April 10, 2015
10 / 23
Effects of Design Components on Estimation Error
Design Choice
Random sampling
Complete stratified random sampling
Focus on SATE rather than PATE
Weighting for nonrandom sampling
Gary King (Harvard)
∆ SX
avg
= 0
=0
=0
=0
Research Designs for Causal Inference
∆SU
avg
∆ TX
∆ TU
= 0
= 0
=0
=?
avg
April 10, 2015
10 / 23
Effects of Design Components on Estimation Error
Design Choice
Random sampling
Complete stratified random sampling
Focus on SATE rather than PATE
Weighting for nonrandom sampling
Large sample size
Gary King (Harvard)
∆ SX
avg
= 0
=0
=0
=0
→?
Research Designs for Causal Inference
∆SU
avg
= 0
= 0
=0
=?
→?
∆ TX
∆ TU
→?
→?
avg
April 10, 2015
10 / 23
Effects of Design Components on Estimation Error
Design Choice
Random sampling
Complete stratified random sampling
Focus on SATE rather than PATE
Weighting for nonrandom sampling
Large sample size
Random treatment assignment
Gary King (Harvard)
∆ SX
avg
= 0
=0
=0
=0
→?
Research Designs for Causal Inference
∆SU
avg
= 0
= 0
=0
=?
→?
∆ TX
∆ TU
→?
= 0
avg
avg
avg
→?
= 0
April 10, 2015
10 / 23
Effects of Design Components on Estimation Error
Design Choice
Random sampling
Complete stratified random sampling
Focus on SATE rather than PATE
Weighting for nonrandom sampling
Large sample size
Random treatment assignment
Complete blocking
Gary King (Harvard)
∆ SX
avg
= 0
=0
=0
=0
→?
Research Designs for Causal Inference
∆SU
avg
= 0
= 0
=0
=?
→?
∆ TX
∆ TU
avg
→?
= 0
=0
avg
→?
= 0
=?
avg
April 10, 2015
10 / 23
Effects of Design Components on Estimation Error
Design Choice
Random sampling
Complete stratified random sampling
Focus on SATE rather than PATE
Weighting for nonrandom sampling
Large sample size
Random treatment assignment
Complete blocking
Exact matching
Gary King (Harvard)
∆ SX
avg
= 0
=0
=0
=0
→?
Research Designs for Causal Inference
∆SU
avg
= 0
= 0
=0
=?
→?
∆ TX
∆ TU
avg
→?
= 0
=0
=0
avg
→?
= 0
=?
=?
avg
April 10, 2015
10 / 23
Effects of Design Components on Estimation Error
Design Choice
Random sampling
Complete stratified random sampling
Focus on SATE rather than PATE
Weighting for nonrandom sampling
Large sample size
Random treatment assignment
Complete blocking
Exact matching
Assumption
Gary King (Harvard)
∆ SX
avg
= 0
=0
=0
=0
→?
Research Designs for Causal Inference
∆SU
avg
= 0
= 0
=0
=?
→?
∆ TX
∆ TU
avg
→?
= 0
=0
=0
avg
→?
= 0
=?
=?
avg
April 10, 2015
10 / 23
Effects of Design Components on Estimation Error
Design Choice
Random sampling
Complete stratified random sampling
Focus on SATE rather than PATE
Weighting for nonrandom sampling
Large sample size
Random treatment assignment
Complete blocking
Exact matching
Assumption
No selection bias
Gary King (Harvard)
∆ SX
avg
= 0
=0
=0
=0
→?
∆SU
avg
= 0
= 0
=0
=?
→?
∆ TX
avg
→?
= 0
=0
=0
avg
avg
= 0
Research Designs for Causal Inference
∆ TU
→?
= 0
=?
=?
avg
avg
= 0
April 10, 2015
10 / 23
Effects of Design Components on Estimation Error
Design Choice
Random sampling
Complete stratified random sampling
Focus on SATE rather than PATE
Weighting for nonrandom sampling
Large sample size
Random treatment assignment
Complete blocking
Exact matching
Assumption
No selection bias
Ignorability
Gary King (Harvard)
∆ SX
avg
= 0
=0
=0
=0
→?
∆SU
avg
= 0
= 0
=0
=?
→?
∆ TX
avg
→?
= 0
=0
=0
avg
avg
= 0
Research Designs for Causal Inference
∆ TU
→?
= 0
=?
=?
avg
avg
= 0
avg
= 0
April 10, 2015
10 / 23
Effects of Design Components on Estimation Error
Design Choice
Random sampling
Complete stratified random sampling
Focus on SATE rather than PATE
Weighting for nonrandom sampling
Large sample size
Random treatment assignment
Complete blocking
Exact matching
Assumption
No selection bias
Ignorability
No omitted variables
Gary King (Harvard)
∆ SX
avg
= 0
=0
=0
=0
→?
∆SU
avg
= 0
= 0
=0
=?
→?
∆ TX
avg
→?
= 0
=0
=0
avg
avg
= 0
Research Designs for Causal Inference
∆ TU
→?
= 0
=?
=?
avg
avg
= 0
avg
= 0
=0
April 10, 2015
10 / 23
Comparing Blocking and Matching
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
11 / 23
Comparing Blocking and Matching
Adding blocking to random assignment is always as or more efficient,
and never biased (blocking on pretreatment variables related to the
outcome is more efficient)
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
11 / 23
Comparing Blocking and Matching
Adding blocking to random assignment is always as or more efficient,
and never biased (blocking on pretreatment variables related to the
outcome is more efficient)
Blocking is like regression adjustment, where the functional form and
the parameter values are known
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
11 / 23
Comparing Blocking and Matching
Adding blocking to random assignment is always as or more efficient,
and never biased (blocking on pretreatment variables related to the
outcome is more efficient)
Blocking is like regression adjustment, where the functional form and
the parameter values are known
Matching is like blocking, except:
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
11 / 23
Comparing Blocking and Matching
Adding blocking to random assignment is always as or more efficient,
and never biased (blocking on pretreatment variables related to the
outcome is more efficient)
Blocking is like regression adjustment, where the functional form and
the parameter values are known
Matching is like blocking, except:
1
to avoid selection error, must change quantity of interest from PATE to
PATT or SATT,
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
11 / 23
Comparing Blocking and Matching
Adding blocking to random assignment is always as or more efficient,
and never biased (blocking on pretreatment variables related to the
outcome is more efficient)
Blocking is like regression adjustment, where the functional form and
the parameter values are known
Matching is like blocking, except:
1
2
to avoid selection error, must change quantity of interest from PATE to
PATT or SATT,
random treatment assignment following matching is impossible,
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
11 / 23
Comparing Blocking and Matching
Adding blocking to random assignment is always as or more efficient,
and never biased (blocking on pretreatment variables related to the
outcome is more efficient)
Blocking is like regression adjustment, where the functional form and
the parameter values are known
Matching is like blocking, except:
1
2
3
to avoid selection error, must change quantity of interest from PATE to
PATT or SATT,
random treatment assignment following matching is impossible,
Exact matching, unlike blocking, is dependent on the already-collected
data happening to contain sufficiently good matches.
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
11 / 23
Comparing Blocking and Matching
Adding blocking to random assignment is always as or more efficient,
and never biased (blocking on pretreatment variables related to the
outcome is more efficient)
Blocking is like regression adjustment, where the functional form and
the parameter values are known
Matching is like blocking, except:
1
2
3
4
to avoid selection error, must change quantity of interest from PATE to
PATT or SATT,
random treatment assignment following matching is impossible,
Exact matching, unlike blocking, is dependent on the already-collected
data happening to contain sufficiently good matches.
In the worst case scenerio, matching (just like regression adjustment)
can increase bias (this cannot occur with blocking plus random
assignment)
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
11 / 23
Comparing Blocking and Matching
Adding blocking to random assignment is always as or more efficient,
and never biased (blocking on pretreatment variables related to the
outcome is more efficient)
Blocking is like regression adjustment, where the functional form and
the parameter values are known
Matching is like blocking, except:
1
2
3
4
to avoid selection error, must change quantity of interest from PATE to
PATT or SATT,
random treatment assignment following matching is impossible,
Exact matching, unlike blocking, is dependent on the already-collected
data happening to contain sufficiently good matches.
In the worst case scenerio, matching (just like regression adjustment)
can increase bias (this cannot occur with blocking plus random
assignment)
Adding matching to a parametric model almost always reduces model
dependence and bias, and sometimes variance too
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
11 / 23
The Benefits of Major Research Designs: Overview
Ideal experiment
Randomized clinicial trials
(Limited or no blocking)
Randomized clinicial trials
(Full blocking)
Social Science
Field Experiment
(Limited or no blocking)
Survey Experiment
(Limited or no blocking)
Observational Study
(Representative data set,
Well-matched)
Observational Study
(Unrepresentative but partially,
correctable data, well-matched)
Observational Study
(Unrepresentative data set,
Well-matched)
∆SX
→0
∆ SU
→0
∆ TX
=0
∆ TU
→0
6= 0
6= 0
avg
avg
6= 0
6= 0
=0
6= 0
6= 0
→0
→0
→0
→0
→0
→0
≈0
≈0
≈0
6= 0
≈0
6= 0
≈0
6= 0
6= 0
6= 0
≈0
6= 0
= 0
= 0
avg
= 0
For column Q, “→ 0” denotes E (Q) = 0 and limn→∞ Var(Q) = 0,
avg
whereas “ = 0” indicates zero on average, or E (Q) = 0, for a design with
a small n. ∆S can be set to zero if we switch from PATE to SATE.
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
12 / 23
The Ideal Experiment (according to the paper)
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
13 / 23
The Ideal Experiment (according to the paper)
Random selection from well-defined population
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
13 / 23
The Ideal Experiment (according to the paper)
Random selection from well-defined population
large n
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
13 / 23
The Ideal Experiment (according to the paper)
Random selection from well-defined population
large n
blocking on all known confounders
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
13 / 23
The Ideal Experiment (according to the paper)
Random selection from well-defined population
large n
blocking on all known confounders
random treatment assignment within blocks
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
13 / 23
The Ideal Experiment (according to the paper)
Random selection from well-defined population
large n
blocking on all known confounders
random treatment assignment within blocks
E (∆SX ) = 0, limn→∞ V (∆SX ) = 0
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
13 / 23
The Ideal Experiment (according to the paper)
Random selection from well-defined population
large n
blocking on all known confounders
random treatment assignment within blocks
E (∆SX ) = 0, limn→∞ V (∆SX ) = 0
E (∆SU ) = 0, limn→∞ V (∆SU ) = 0
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
13 / 23
The Ideal Experiment (according to the paper)
Random selection from well-defined population
large n
blocking on all known confounders
random treatment assignment within blocks
E (∆SX ) = 0, limn→∞ V (∆SX ) = 0
E (∆SU ) = 0, limn→∞ V (∆SU ) = 0
∆ TX = 0
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
13 / 23
The Ideal Experiment (according to the paper)
Random selection from well-defined population
large n
blocking on all known confounders
random treatment assignment within blocks
E (∆SX ) = 0, limn→∞ V (∆SX ) = 0
E (∆SU ) = 0, limn→∞ V (∆SU ) = 0
∆ TX = 0
E (∆TU ) = 0, limn→∞ V (∆TU ) = 0
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
13 / 23
An Even More Ideal Experiment
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
14 / 23
An Even More Ideal Experiment
Begin with a well-defined population
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
14 / 23
An Even More Ideal Experiment
Begin with a well-defined population
Define sampling strata based on cross-classification of all known
confounders
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
14 / 23
An Even More Ideal Experiment
Begin with a well-defined population
Define sampling strata based on cross-classification of all known
confounders
Random sampling within strata (if strata sample is proportional to
population fraction, no weights are needed)
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
14 / 23
An Even More Ideal Experiment
Begin with a well-defined population
Define sampling strata based on cross-classification of all known
confounders
Random sampling within strata (if strata sample is proportional to
population fraction, no weights are needed)
large n
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
14 / 23
An Even More Ideal Experiment
Begin with a well-defined population
Define sampling strata based on cross-classification of all known
confounders
Random sampling within strata (if strata sample is proportional to
population fraction, no weights are needed)
large n
blocking on all known confounders
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
14 / 23
An Even More Ideal Experiment
Begin with a well-defined population
Define sampling strata based on cross-classification of all known
confounders
Random sampling within strata (if strata sample is proportional to
population fraction, no weights are needed)
large n
blocking on all known confounders
random treatment assignment within blocks
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
14 / 23
An Even More Ideal Experiment
Begin with a well-defined population
Define sampling strata based on cross-classification of all known
confounders
Random sampling within strata (if strata sample is proportional to
population fraction, no weights are needed)
large n
blocking on all known confounders
random treatment assignment within blocks
∆ SX = 0
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
14 / 23
An Even More Ideal Experiment
Begin with a well-defined population
Define sampling strata based on cross-classification of all known
confounders
Random sampling within strata (if strata sample is proportional to
population fraction, no weights are needed)
large n
blocking on all known confounders
random treatment assignment within blocks
∆ SX = 0
E (∆SU ) = 0, limn→∞ V (∆SU ) = 0
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
14 / 23
An Even More Ideal Experiment
Begin with a well-defined population
Define sampling strata based on cross-classification of all known
confounders
Random sampling within strata (if strata sample is proportional to
population fraction, no weights are needed)
large n
blocking on all known confounders
random treatment assignment within blocks
∆ SX = 0
E (∆SU ) = 0, limn→∞ V (∆SU ) = 0
∆ TX = 0
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
14 / 23
An Even More Ideal Experiment
Begin with a well-defined population
Define sampling strata based on cross-classification of all known
confounders
Random sampling within strata (if strata sample is proportional to
population fraction, no weights are needed)
large n
blocking on all known confounders
random treatment assignment within blocks
∆ SX = 0
E (∆SU ) = 0, limn→∞ V (∆SU ) = 0
∆ TX = 0
E (∆TU ) = 0, limn→∞ V (∆TU ) = 0
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
14 / 23
Randomized Clinical Trials (Little or no Blocking)
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
15 / 23
Randomized Clinical Trials (Little or no Blocking)
nonrandom selection
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
15 / 23
Randomized Clinical Trials (Little or no Blocking)
nonrandom selection
small n
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
15 / 23
Randomized Clinical Trials (Little or no Blocking)
nonrandom selection
small n
little or no blocking
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
15 / 23
Randomized Clinical Trials (Little or no Blocking)
nonrandom selection
small n
little or no blocking
random treatment assignment
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
15 / 23
Randomized Clinical Trials (Little or no Blocking)
nonrandom selection
small n
little or no blocking
random treatment assignment
∆SX 6= 0
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
15 / 23
Randomized Clinical Trials (Little or no Blocking)
nonrandom selection
small n
little or no blocking
random treatment assignment
∆SX 6= 0
∆SU 6= 0
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
15 / 23
Randomized Clinical Trials (Little or no Blocking)
nonrandom selection
small n
little or no blocking
random treatment assignment
∆SX 6= 0
∆SU 6= 0
E (∆TX ) = 0
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
15 / 23
Randomized Clinical Trials (Little or no Blocking)
nonrandom selection
small n
little or no blocking
random treatment assignment
∆SX 6= 0
∆SU 6= 0
E (∆TX ) = 0
E (∆TU ) = 0
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
15 / 23
Randomized Clinical Trials (Full Blocking)
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
16 / 23
Randomized Clinical Trials (Full Blocking)
nonrandom selection
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
16 / 23
Randomized Clinical Trials (Full Blocking)
nonrandom selection
small n
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
16 / 23
Randomized Clinical Trials (Full Blocking)
nonrandom selection
small n
Full blocking
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
16 / 23
Randomized Clinical Trials (Full Blocking)
nonrandom selection
small n
Full blocking
random treatment assignment
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
16 / 23
Randomized Clinical Trials (Full Blocking)
nonrandom selection
small n
Full blocking
random treatment assignment
∆SX 6= 0
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
16 / 23
Randomized Clinical Trials (Full Blocking)
nonrandom selection
small n
Full blocking
random treatment assignment
∆SX 6= 0
∆SU 6= 0
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
16 / 23
Randomized Clinical Trials (Full Blocking)
nonrandom selection
small n
Full blocking
random treatment assignment
∆SX 6= 0
∆SU 6= 0
∆ TX = 0
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
16 / 23
Randomized Clinical Trials (Full Blocking)
nonrandom selection
small n
Full blocking
random treatment assignment
∆SX 6= 0
∆SU 6= 0
∆ TX = 0
E (∆TU ) = 0
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
16 / 23
Social Science Field Experiment
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
17 / 23
Social Science Field Experiment
nonrandom selection
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
17 / 23
Social Science Field Experiment
nonrandom selection
large n
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
17 / 23
Social Science Field Experiment
nonrandom selection
large n
limited or no blocking
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
17 / 23
Social Science Field Experiment
nonrandom selection
large n
limited or no blocking
random treatment assignment
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
17 / 23
Social Science Field Experiment
nonrandom selection
large n
limited or no blocking
random treatment assignment
∆SX 6= 0 or change PATE to SATE and ∆SX = 0
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
17 / 23
Social Science Field Experiment
nonrandom selection
large n
limited or no blocking
random treatment assignment
∆SX 6= 0 or change PATE to SATE and ∆SX = 0
∆SU 6= 0 or change PATE to SATE and ∆SU = 0
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
17 / 23
Social Science Field Experiment
nonrandom selection
large n
limited or no blocking
random treatment assignment
∆SX 6= 0 or change PATE to SATE and ∆SX = 0
∆SU 6= 0 or change PATE to SATE and ∆SU = 0
E (∆TX ) = 0, limn→∞ V (∆TX ) = 0
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
17 / 23
Social Science Field Experiment
nonrandom selection
large n
limited or no blocking
random treatment assignment
∆SX 6= 0 or change PATE to SATE and ∆SX = 0
∆SU 6= 0 or change PATE to SATE and ∆SU = 0
E (∆TX ) = 0, limn→∞ V (∆TX ) = 0
E (∆TU ) = 0, limn→∞ V (∆TU ) = 0
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
17 / 23
Survey Experiment
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
18 / 23
Survey Experiment
random selection
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
18 / 23
Survey Experiment
random selection
large n
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
18 / 23
Survey Experiment
random selection
large n
limited or no blocking
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
18 / 23
Survey Experiment
random selection
large n
limited or no blocking
random treatment assignment (only question wording treatments)
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
18 / 23
Survey Experiment
random selection
large n
limited or no blocking
random treatment assignment (only question wording treatments)
E (∆SX ) = 0, limn→∞ V (∆SX ) = 0
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
18 / 23
Survey Experiment
random selection
large n
limited or no blocking
random treatment assignment (only question wording treatments)
E (∆SX ) = 0, limn→∞ V (∆SX ) = 0
E (∆SU ) = 0, limn→∞ V (∆SU ) = 0
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
18 / 23
Survey Experiment
random selection
large n
limited or no blocking
random treatment assignment (only question wording treatments)
E (∆SX ) = 0, limn→∞ V (∆SX ) = 0
E (∆SU ) = 0, limn→∞ V (∆SU ) = 0
E (∆TX ) = 0, limn→∞ V (∆TX ) = 0
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
18 / 23
Survey Experiment
random selection
large n
limited or no blocking
random treatment assignment (only question wording treatments)
E (∆SX ) = 0, limn→∞ V (∆SX ) = 0
E (∆SU ) = 0, limn→∞ V (∆SU ) = 0
E (∆TX ) = 0, limn→∞ V (∆TX ) = 0
E (∆TU ) = 0, limn→∞ V (∆TU ) = 0
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
18 / 23
Observational Study, well-matched
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
19 / 23
Observational Study, well-matched
nonrandom selection
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
19 / 23
Observational Study, well-matched
nonrandom selection
large n
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
19 / 23
Observational Study, well-matched
nonrandom selection
large n
no blocking
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
19 / 23
Observational Study, well-matched
nonrandom selection
large n
no blocking
nonrandom treatment assignment
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
19 / 23
Observational Study, well-matched
nonrandom selection
large n
no blocking
nonrandom treatment assignment
∆SX ≈ 0 if representative, corrected by weighting, or for estimating
SATE; or 6= 0 otherwise
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
19 / 23
Observational Study, well-matched
nonrandom selection
large n
no blocking
nonrandom treatment assignment
∆SX ≈ 0 if representative, corrected by weighting, or for estimating
SATE; or 6= 0 otherwise
∆SU 6= 0
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
19 / 23
Observational Study, well-matched
nonrandom selection
large n
no blocking
nonrandom treatment assignment
∆SX ≈ 0 if representative, corrected by weighting, or for estimating
SATE; or 6= 0 otherwise
∆SU 6= 0
∆TX ≈ 0 (due to matching well)
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
19 / 23
Observational Study, well-matched
nonrandom selection
large n
no blocking
nonrandom treatment assignment
∆SX ≈ 0 if representative, corrected by weighting, or for estimating
SATE; or 6= 0 otherwise
∆SU 6= 0
∆TX ≈ 0 (due to matching well)
∆TU 6= 0 except by assumption
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
19 / 23
What is the Best Design?
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
20 / 23
What is the Best Design?
The ideal design is rarely feasible
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
20 / 23
What is the Best Design?
The ideal design is rarely feasible
Effort in experimental studies: random assignment
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
20 / 23
What is the Best Design?
The ideal design is rarely feasible
Effort in experimental studies: random assignment
Effort in observational studies: measuring and adjusting for X (via
matching or modeling)
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
20 / 23
What is the Best Design?
The ideal design is rarely feasible
Effort in experimental studies: random assignment
Effort in observational studies: measuring and adjusting for X (via
matching or modeling)
the Achilles heal of experimental studies: ∆S and small n
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
20 / 23
What is the Best Design?
The ideal design is rarely feasible
Effort in experimental studies: random assignment
Effort in observational studies: measuring and adjusting for X (via
matching or modeling)
the Achilles heal of experimental studies: ∆S and small n
the Achilles heal of observational studies: ∆U
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
20 / 23
What is the Best Design?
The ideal design is rarely feasible
Effort in experimental studies: random assignment
Effort in observational studies: measuring and adjusting for X (via
matching or modeling)
the Achilles heal of experimental studies: ∆S and small n
the Achilles heal of observational studies: ∆U
Each design accomodates best to the applications for which it was
designed
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
20 / 23
Fallacies in Experimental Research
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
21 / 23
Fallacies in Experimental Research
Failure to block on all known covariates
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
21 / 23
Fallacies in Experimental Research
Failure to block on all known covariates
seen incorrectly as requiring fewer assumptions (about what to block
on)
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
21 / 23
Fallacies in Experimental Research
Failure to block on all known covariates
seen incorrectly as requiring fewer assumptions (about what to block
on)
In fact, blocking helps (except in strange situations)
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
21 / 23
Fallacies in Experimental Research
Failure to block on all known covariates
seen incorrectly as requiring fewer assumptions (about what to block
on)
In fact, blocking helps (except in strange situations)
Blocking on relevant covariates is better, so choose carefully.
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
21 / 23
Fallacies in Experimental Research
Failure to block on all known covariates
seen incorrectly as requiring fewer assumptions (about what to block
on)
In fact, blocking helps (except in strange situations)
Blocking on relevant covariates is better, so choose carefully.
“Block what you can and randomize what you cannot” (Box et al.)
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
21 / 23
Fallacies in Experimental Research
Failure to block on all known covariates
seen incorrectly as requiring fewer assumptions (about what to block
on)
In fact, blocking helps (except in strange situations)
Blocking on relevant covariates is better, so choose carefully.
“Block what you can and randomize what you cannot” (Box et al.)
Conducting t-tests to check for balance after random treatment
assignment
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
21 / 23
Fallacies in Experimental Research
Failure to block on all known covariates
seen incorrectly as requiring fewer assumptions (about what to block
on)
In fact, blocking helps (except in strange situations)
Blocking on relevant covariates is better, so choose carefully.
“Block what you can and randomize what you cannot” (Box et al.)
Conducting t-tests to check for balance after random treatment
assignment
variables blocked on balance exactly after treatment assignment; so if
you’re checking, you missed an opportunity to increase efficiency
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
21 / 23
Fallacies in Experimental Research
Failure to block on all known covariates
seen incorrectly as requiring fewer assumptions (about what to block
on)
In fact, blocking helps (except in strange situations)
Blocking on relevant covariates is better, so choose carefully.
“Block what you can and randomize what you cannot” (Box et al.)
Conducting t-tests to check for balance after random treatment
assignment
variables blocked on balance exactly after treatment assignment; so if
you’re checking, you missed an opportunity to increase efficiency
if variables became available after treatment assignment, a t-test only
checks for whether randomization was done appropriately
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
21 / 23
Fallacies in Experimental Research
Failure to block on all known covariates
seen incorrectly as requiring fewer assumptions (about what to block
on)
In fact, blocking helps (except in strange situations)
Blocking on relevant covariates is better, so choose carefully.
“Block what you can and randomize what you cannot” (Box et al.)
Conducting t-tests to check for balance after random treatment
assignment
variables blocked on balance exactly after treatment assignment; so if
you’re checking, you missed an opportunity to increase efficiency
if variables became available after treatment assignment, a t-test only
checks for whether randomization was done appropriately
randomization balances on average; any one random assignment is not
balanced exactly (which is why its better to block)
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
21 / 23
60
20
1
Math test score
"Statistical
insignificance" region
40
2
t−statistic
3
80
4
100
The Balance Test Fallacy in Matching Research
0
0
QQ Plot Mean Deviation
0
100 200 300 400
Number of Controls Randomly Dropped
Difference in Means
0
100 200 300 400
Number of Controls Randomly Dropped
Randomly dropping observations “reduces” imbalance???
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
22 / 23
The Balance Test Fallacy: Explanation
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
23 / 23
The Balance Test Fallacy: Explanation
hypothesis tests are a function of balance AND power; we only want
to measure balance
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
23 / 23
The Balance Test Fallacy: Explanation
hypothesis tests are a function of balance AND power; we only want
to measure balance
for SATE, PATE, or SPATE, balance is a characteristic of the sample,
not some superpopulation; so no need to make an inference
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
23 / 23
The Balance Test Fallacy: Explanation
hypothesis tests are a function of balance AND power; we only want
to measure balance
for SATE, PATE, or SPATE, balance is a characteristic of the sample,
not some superpopulation; so no need to make an inference
Simple linear model (for intution):
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
23 / 23
The Balance Test Fallacy: Explanation
hypothesis tests are a function of balance AND power; we only want
to measure balance
for SATE, PATE, or SPATE, balance is a characteristic of the sample,
not some superpopulation; so no need to make an inference
Simple linear model (for intution):
Suppose E (Y | T , X ) = θ + T β + X γ
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
23 / 23
The Balance Test Fallacy: Explanation
hypothesis tests are a function of balance AND power; we only want
to measure balance
for SATE, PATE, or SPATE, balance is a characteristic of the sample,
not some superpopulation; so no need to make an inference
Simple linear model (for intution):
Suppose E (Y | T , X ) = θ + T β + X γ
Bias in coefficient on T from regressing Y on T (without X ):
E (βˆ − β | T , X ) = G γ (where G are coefficients from a regression X
on a constant and T )
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
23 / 23
The Balance Test Fallacy: Explanation
hypothesis tests are a function of balance AND power; we only want
to measure balance
for SATE, PATE, or SPATE, balance is a characteristic of the sample,
not some superpopulation; so no need to make an inference
Simple linear model (for intution):
Suppose E (Y | T , X ) = θ + T β + X γ
Bias in coefficient on T from regressing Y on T (without X ):
E (βˆ − β | T , X ) = G γ (where G are coefficients from a regression X
on a constant and T )
Imbalance is due to G only.
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
23 / 23
The Balance Test Fallacy: Explanation
hypothesis tests are a function of balance AND power; we only want
to measure balance
for SATE, PATE, or SPATE, balance is a characteristic of the sample,
not some superpopulation; so no need to make an inference
Simple linear model (for intution):
Suppose E (Y | T , X ) = θ + T β + X γ
Bias in coefficient on T from regressing Y on T (without X ):
E (βˆ − β | T , X ) = G γ (where G are coefficients from a regression X
on a constant and T )
Imbalance is due to G only.
If G = 0, bias=0
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
23 / 23
The Balance Test Fallacy: Explanation
hypothesis tests are a function of balance AND power; we only want
to measure balance
for SATE, PATE, or SPATE, balance is a characteristic of the sample,
not some superpopulation; so no need to make an inference
Simple linear model (for intution):
Suppose E (Y | T , X ) = θ + T β + X γ
Bias in coefficient on T from regressing Y on T (without X ):
E (βˆ − β | T , X ) = G γ (where G are coefficients from a regression X
on a constant and T )
Imbalance is due to G only.
If G = 0, bias=0
If G 6= 0, bias can be any size due to γ
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
23 / 23
The Balance Test Fallacy: Explanation
hypothesis tests are a function of balance AND power; we only want
to measure balance
for SATE, PATE, or SPATE, balance is a characteristic of the sample,
not some superpopulation; so no need to make an inference
Simple linear model (for intution):
Suppose E (Y | T , X ) = θ + T β + X γ
Bias in coefficient on T from regressing Y on T (without X ):
E (βˆ − β | T , X ) = G γ (where G are coefficients from a regression X
on a constant and T )
Imbalance is due to G only.
If G = 0, bias=0
If G 6= 0, bias can be any size due to γ
If you cannot control G (and we don’t usually try to minimize γ so we
don’t cook the books), then G must be reduced without limit.
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
23 / 23
The Balance Test Fallacy: Explanation
hypothesis tests are a function of balance AND power; we only want
to measure balance
for SATE, PATE, or SPATE, balance is a characteristic of the sample,
not some superpopulation; so no need to make an inference
Simple linear model (for intution):
Suppose E (Y | T , X ) = θ + T β + X γ
Bias in coefficient on T from regressing Y on T (without X ):
E (βˆ − β | T , X ) = G γ (where G are coefficients from a regression X
on a constant and T )
Imbalance is due to G only.
If G = 0, bias=0
If G 6= 0, bias can be any size due to γ
If you cannot control G (and we don’t usually try to minimize γ so we
don’t cook the books), then G must be reduced without limit.
No threshold level is safe.
Gary King (Harvard)
Research Designs for Causal Inference
April 10, 2015
23 / 23