Pushing Forward from BHR through Random Individual- Level Variation in Program Components within Sites: The HPOG Impact Study Stephen H. Bell APPAM Research Conference November 7, 2013
Jan 18, 2018
Pushing Forward from BHR through Random Individual-Level Variation in Program Components within Sites: The HPOG Impact Study
Stephen H. Bell
APPAM Research Conference
November 7, 2013
Abt Associates | pg 2
Levels and Mechanisms of Program Variation in an RCT Context
MECHANISMLEVEL Random NaturalSite Cluster randomization
Multi-level estimationMulti-level explanatory models (e.g., BHR)
Individual Three-way within-site randomizationStandard experiment-al estimation
Analysis of symmetrically predicted endogenous subgroups (ASPES; Peck, 2003; Bell & Peck, 2013)
Abt Associates | pg 3
Strengthen multi-level explanatory model using three-way random assignment of individuals in a subset of sites (e.g., Health Profession Opportunity Grants evaluation)
_______________________________________
1. Omitted variable bias in BHR
2. How third experimental arm removes bias for the randomized element
3. How third experimental arm reduces bias for BHR-style cross-site comparisons
4. Implications / next steps
Goal and Outline
Abt Associates | pg 4
HPOG and Its Impact Evaluation
Career Pathways framework-based training for TANF and low-income individuals to pursue healthcare sector careers
HPOG-Impact is part of a rich research “portfolio” at ACF
Impact Evaluation involves an experimental design, with randomization of eligibles to control and treatment groups, with randomization to enhanced treatment in some locations
Abt Associates | pg 5
Study Sample and Data Collection
Sample size– Individuals: about 10,500 overall: 7,000 T; 3,500 C– Study sites: 38 study sites programs across 20 grantees– Planned variation sample (TBD)
• Peer support • Emergency financial assistance • Non-cash incentives
Data collection– At baseline (before RA), from PRS & supplement – Quarterly wage data (NDNH)– Follow-up surveys at 15 months post-randomization– Implementation study site visits– Grantee, staff/management and other surveys
Abt Associates | pg 6
T
Estimating this model gives , . . . , as non-experimental estimates of the influence on impact magnitude of the P program features
Two-Level Model of Site-Level Determinants of Intervention Impacts
Abt Associates | pg 7
What if there is an omitted site-level factor, , that
• Influences impact magnitudes
• Is not included in nor in site level covariates
• Correlates with one or more ?
Plug formula for ) into equation:
Omission of biases wherever Cov(
Omitted-Variable Bias Concern
Abt Associates | pg 8
= + + +
where = 1 if person i in site j is assigned to the basic intervention (T1)
= 0 otherwise (T2, C)
and = 1 if person i in site j is assigned to the enhanced intervention (T2)
= 0 otherwise (T1, C)
= +
Two-Level Model with Three-Arm Random Assignment
Abt Associates | pg 9
Add and subtract to expression for :
+ ( + ) + ( - ) +
This replaces with
)
which, unlike, does not depend on F no omitted variable bias on estimate of incremental impact of the enhancement
How the Third Arm Removes Bias
Abt Associates | pg 10
bias
Bias is smaller when and – the site-comparison-based non-experimental (BHR) estimate and the within-site experimental estimate – for the randomized element are closer together
Tweak the model – especially the impact equations – to reduce - reduces all bias
All estimates are distorted by the same F factor “whittle down” this distortion for one and do so for all
Using the Experimental Evidence to REDUCE the Bias Risk in Non-Experimental Estimates
𝜋𝓂∧𝑆 𝜋𝑒∧𝑆 𝜋𝑒∧𝑋
Abt Associates | pg 11
A) Vary as many individual program features as possible through 3-arm random assignment within sites (e.g., HPOG)
B) Better yet . . .
– get many dozens of sites
– randomize program features across sites
Use A to approximate B by adding lots of natural variation sites to a small number of three-arm random assignment sites
______________________________________
Next: Determine statistical power of the design for HPOG Impact
Then: Determine whether site-focused methods can support / enhance individual-focused methods like ASPES . . . or vice versa
Implications for Design / Extensions
Abt Associates | pg 12
Levels and Mechanisms of Program Variation in an RCT Context
MECHANISMLEVEL Random NaturalSite Cluster randomization
Multi-level estimationMulti-level explanatory models (e.g., BHR)
Individual Three-way within-site randomizationStandard experiment-al estimation
Analysis of symmetrically predicted endogenous subgroups (ASPES; Peck, 2003, Bell & Peck, 2013)
Further Information
Molly IrwinFederal Project Officer, HPOG HHS/ACF/[email protected]
Stephen BellPrincipal ScientistAbt Associates [email protected]
Further Information
Molly IrwinFederal Project Officer, HPOG HHS/ACF/[email protected]
Stephen BellPrincipal ScientistAbt Associates [email protected]