Testing the Causal Effects of Social Capital: Design for a Cluster-Randomized Field Trial Adam Gamoran and Ruth N. López Turley University of Wisconsin-Madison.

Testing the Causal Effects of Social Capital: Design for a Cluster-Randomized Field Trial

Adam Gamoran and Ruth N. López Turley

University of Wisconsin-Madison

Social Capital: Conceptual and Causal Ambiguity

Social capital is one of the most popular terms in social science today Viewed as the source of many positive outcomes

Test scores, school completion, social adjustment, mental and physical health

Decline of social capital is seen as responsible for many social ills Crime, apathy

Causal role of social capital is ambiguous

Social Capital: Conceptual and Causal Ambiguity

Concept of social capital is also ambiguous Relations of trust, mutual expectations, and

shared values embedded in social networks Not possessed by individuals Resides in the relationships individuals have with

one another Individuals can draw upon social capital in their

networks Social capital facilitates the flow of information

and the development and enforcement of norms

Concepts of Social Capital Qualities of social networks that signify social

capital: How do we know if social capital is present? Intergenerational closure

Intergenerational Closure

Source: Coleman, Am. J. Soc., 1988

Concepts of Social Capital Qualities of social networks that signify social

capital: How do we know if social capital is present? Intergenerational closure Trust

Network members rely on one another Facilitates sharing of norms and information

Shared expectations Also facilitates supporting norms and distributing

information

Concepts of Social Capital Contrary to Coleman (1988), we do not define

social capital by its function Contrary to Portes (1998), we view social

capital as a collective rather than as an individual attribute

We follow Sampson et al. (1999): “…social capital for children refers to the resource potential of personal and organizational networks…”

Domains of Social Capital Parent-school relationships Parent-parent relationships Parent-child relationships

Parent-school Parent-parent Parent-child

Trust

Shared expectations

Intergenerational closure

Mechanisms of Social Capital For young children, social capital operates

through their parents Two primary mechanisms

Social support Parents who feel more connected to others have better

access to information and are better able to establish and enforce norms with their children

Social control Parents’ positive social networks offer collective

socialization of children

Social Capital and Inequality Unequal social capital contributes to unequal

child development Among U.S. Latinos, social capital within

family networks is high, but parent-school social capital is low

Building family-school social capital may enhance child outcomes particularly for Latinos – the focus of our empirical analysis

The Causal Role of Social Capital Many studies have tested the relation between

social capital and child outcomes Most rely on longitudinal data Nonetheless, causal direction is ambiguous

Does social capital foster school success, or do stronger social ties emerge in communities that have more effective schools?

Some group norms can negatively affect child outcomes!

The Causal Role of Social Capital Survey research may overestimate effects of social

capital (Mouw, 2006) Endogeneity: Group members influence one another at

the same time Unobserved selectivity: unmeasured conditions lead to

both common memberships and common outcomes Statistical efforts to resolve these causality issues

rely on questionable assumptions E.g., effects are unbiased net of control variables

The Causal Role of Social Capital An experimental design offers a more

rigorous approach to testing social capital’s causal role No unobserved selectivity: Assignment to

“treatment” is random Avoid endogeneity problem through multilevel

assessment of social capital effects

The Causal Role of Social Capital Conditions for an experimental assessment of social

capital effects:1. An intervention that manipulates social capital

2. Random assignment to treatment and control

3. Random assignment of groups of individuals (because social capital is an attribute of groups, not individuals)

4. Tools for measuring social capital and outcomes

5. Statistical methods suitable for analysis of a cluster-randomized trial

1. An Intervention that Manipulates Social Capital FAST: Families and Schools Together

A multi-family group prevention program Implemented in three stages

Outreach to parents 8 weeks of multi-family group meetings 2 years of monthly follow-up meetings led by parents

1. An Intervention that Manipulates Social Capital Elements of FAST

Led by a parent-professional partnership Culturally representative and adapted Research-based activities

Family meal Group singing Family games Parent support/ children’s time One-to-one responsive play Closing circle

1. An Intervention that Manipulates Social Capital Prior research on FAST

4 previous randomized trials have documented positive outcomes for children’s social and academic outcomes

These studies have occurred at the individual or classroom levels

School-wide, “multi-hub” FAST is likely to have even more powerful effects

1. An Intervention that Manipulates Social Capital Prior research on FAST

FAST builds social capital Parent-school: Reduces alienation from school

authorities, and increases comfort level Parent-parent: Reduces isolation of parents by

creating a parent support group Parent-child: Improve relationship through one-on-

one responsive play

Particularly valuable for immigrant communities

Conceptual Model

2. Random Assignment to Treatment and Control: Experimental Design Research Sites

San Antonio, TX: A large, long-standing Latino populations (51% of students)

Milwaukee, WI: A rapidly growing Latino population (21% of students)

Experience with FAST, community agencies available to implement

Agreed to implement FAST in treatment schools, not in control schools Subject to agreement of principals and teachers They love FAST, this won’t be a problem

3. Random Assignment of Groups of Individuals: Experimental Design 26 schools from each district (13 treatment

and 13 control), total of 52 schools All first-grade families will be invited to

participate We anticipate 75% participation rate, 20%

attrition rate = 60% long-term follow-up Three years of data collection (grades 1 to 3)

3. Random Assignment of Groups of Individuals: Experimental Design How did we decide on 52 schools?

Power analysis

Power Analysis: Assumptions Power criterion (1 – β) = .80 Probability of Type I error () = .05 Within-school sample size (n) = 60 Effect size () =.25 Intraclass correlation () = .10 Covariate correlation (r) = .40-.60

Power Analysis: Softwarehttp://sitemaker.umich.edu/group-based/optimal_design_software

Power Analysis

Power Analysis: Conclusion Under reasonable assumptions, a sample of

52 schools will provide sufficient power to detect the effects of social capital, if they exist.

4. Tools for measuring social capital and outcomes Outcomes

Parent and teacher ratings of child social skills and problem behaviors (grades 1 and 3)

Teacher ratings of child academic competence High-stakes standardized tests of reading and

mathematics

4. Tools for measuring social capital and outcomes Social capital

Parent social capital questionnaire Our only pre-intervention measure

Not really needed for experimental design, but of interest in its own right

Follow-up measures in the spring of grades 1 and 3

Key sources: Bryk and Schneider (2002); McDonald and Moberg (2002)

Parent Social Capital Questionnaire Parent-school trust, shared expectations

Parent Social Capital Questionnaire Parent-parent closure, trust, shared expectations

Parent Social Capital Questionnaire Parent-child trust, shared expectations

4. Tools for measuring social capital and outcomes Social capital

Parent Involvement in School Questionnaire Indicators of trust and shared expectations in

parent-school and parent-child relationships Separate forms with parallel questions from

parent and teacher perspectives Source: Shumow, Vandell, and Kang (1996) Completed by teachers and parents at the end of

grades 1 and 3

4. Tools to measure social capital and outcomes: Other variables as indicated

5. Statistical methods suitable for analysis of cluster-randomized trial This study relies on place-based random

assignment CRT: Cluster-randomized trial Randomization is at the aggregate level Well suited to contextual investigations Must assess the intervention at the level at

which randomization occurs

5. Statistical methods suitable for analysis of cluster-randomized trial A multilevel model is the appropriate

statistical approach to analysis of CRT Captures variability both at the level of the cluster

and within clusters In our case: students within schools Treatment is at the level of the school Theoretically, social capital is also at the level of

the school We allow for individual-level variation

5. Statistical methods suitable for analysis of cluster-randomized trial School-level control variables reduce

variation between schools, permit more precise treatment effects

Individual-level background controls also increase precision More importantly, multilevel interactions permit

estimation of differential treatment effects

Multilevel Models: Linear OutcomesLevel 1. Yij = ß0j + ß1j(SEX)ij + ß2j(LATINO)ij +

ß3j(BLACK)ij + ß4j(POVERTY)ij + rij

Level 2. ß0j = γ00 + γ01(MEAN PRIOR ACH)j + γ02(PERCENT POVERTY)j + γ03(PERCENT LATINO)j + γ04(PERCENT BLACK)j + γ05(FAST)j + γ06(CITY)j + γ07(PERCENT LATINO x FAST)j + γ08(PERCENT BLACK x FAST)j + u0j

Level 2. ß2j = γ20 + γ21(FAST)j + γ22(CITY)j + u2j

ß3j = γ20 + γ21(FAST)j + γ22(CITY)j + u3j

ß4j = γ20 + γ21(FAST)j + γ22(CITY)j + u4j

5. Statistical methods suitable for analysis of cluster-randomized trial By adding social capital to the model, we test

whether social capital accounts for the effects of FAST on child outcomes

Main focus is on school-level effects

Multilevel Models: Linear OutcomesLevel 1. Yij = ß0j + ß1j(SEX)ij + ß2j(LATINO)ij + ß3j(BLACK)ij

+ ß4j(POVERTY)ij + ß5j(SOCIAL CAPITAL)ij

+ rij

Level 2. ß0j = γ00 + γ01(MEAN PRIOR ACH)j + γ02(PERCENT POVERTY)j + γ03(PERCENT LATINO)j + γ04(FAST)j + γ05(MEAN SOCIAL CAPITAL)j + γ06(CITY)j + u0j

5. Statistical methods suitable for analysis of cluster-randomized trial Additional challenges

Uncommon measures: Different tests in Texas and Wisconsin Linking strategy, corrected for unreliability Examine probability of reaching the proficiency

threshold rather than test score

5. Statistical methods suitable for analysis of cluster-randomized trial Additional challenges

Bias in social capital effects FAST effects will be estimated without selectivity bias Social capital effects will also be estimated without selectivity

bias if they derive only from FAST This is probably not the case

If social capital occurs independently of FAST, an omitted variable may affect social capital and child outcomes Use pre-FAST measure to check Use FAST as an instrument for social capital Control for pre-FAST social capital

5. Statistical methods suitable for analysis of cluster-randomized trial Additional challenges

Bias in social capital effects Differential non-response by treatment and control

parents Consent will be obtained prior to randomization Follow up a random subsample of non-respondents with

home visits

5. Statistical methods suitable for analysis of cluster-randomized trial Additional challenges

Fidelity of implementation Implementation study

Implementation checklist Interviews, focus groups with parents and teachers Including interviews with 2 non-participating parents in

each treatment school

Qualitative data will provide more nuanced insights on the mechanisms through which FAST affects (or does not affect) child outcomes

Conclusions The term “social capital” has reflected many

different ideas in different writings Causal ambiguity has been a consistent

limitation of social capital research By manipulating social capital

experimentally, we aim to provide a more persuasive test of social capital effects

References Bryk, A. S., & Schneider, B. L. (2002). Trust in schools: A core resource for

improvement. New York: Russell Sage Foundation. Coleman, J. S. (1988). Social capital in the creation of human capital. American

Journal of Sociology, 94(Suppl.), S95–S120. McDonald, L., & Moberg, D. P. (2002). Social relationships questionnaire.

Madison, WI: FAST National Training and Evaluation Center. Mouw, T. (2006). Estimating the causal effects of social capital: A review of recent

research. Annual Review of Sociology, 32, 79–102. Portes, A. (1998). Social capital: Its origins and applications in modern sociology.

Annual Review of Sociology, 24, 1–24. Sampson, R. J., Morenoff, J. D., & Earls, F. (1999). Beyond social capital: Spatial

dynamics of collective efficacy for children. American Sociological Review, 64(5), 633–660.

Shumow, L., Vandell, D. L., & Kang, K. (1996). School choice, family characteristics, and home-school relations: Contributors to school achievement? Journal of Educational Psychology, 88, 451–460.

Further Reading on Cluster-Randomized Trials

Bloom, H. S. (2006). Learning more from social experiments: Evolving analytic approaches. New York: Russell Sage Foundation.

Bloom, H. S., Bos, J. M., & Lee, S. W. (1999). Using cluster random assignment to measure program impacts: Statistical implications for the evaluation of education programs. Evaluation Review, 23, 445–469.

Borman, G. D., Slavin, R. E., Cheung, A., Chamberlain, A., Madden, N., & Chambers, B. (2005). Success for All: First-year results from the national randomized field trial. Educational Evaluation and Policy Analysis, 27(1), 1–22.

Boruch, R., May, H., Turner, H., Lavenberg, J., Petrosino, A., & de Moya, D. (2004). Estimating the effects of interventions that are deployed in many places: Place-randomized trials. American Behavioral Scientist, 47, 608–633.

Raudenbush, S. W. (1997). Statistical analysis and optimal design for cluster randomized trials. Psychological Methods, 2, 173–185.

Further Reading on FAST Abt Associates. (2001). National evaluation of family support programs: Vol. B. Research

studies: Final report. Cambridge, MA: Author. Retrieved February 12, 2007, from http://www.abtassoc.com/reports/NEFSP-VolB.pdf

Kratochwill, T. R., McDonald, L., Levin, J. R., Young Bear-Tibbetts, H., & Demaray, M. K. (2004). Families and Schools Together: An experimental analysis of a parent-mediated multi-family group intervention program for American Indian children. Journal of School Psychology, 42, 359–383.

McDonald, L., Moberg, D. P., Brown, R., Rodriguez-Espiricueta, I., Flores, N., Burke, M. P., et al. (2006). After-school multifamily groups: A randomized controlled trial involving low-income, urban, Latino children. Children and Schools, 18, 25–34.

U.S. Office of Juvenile Justice and Delinquency Prevention. (2006). Families and Schools Together (FAST). In U.S. Office of Juvenile Justice and Delinquency Prevention, OJJDP model programs guide. Retrieved February 11, 2007, from http://www.dsgonline.com/mpg2.5/TitleV_MPG_Table_Ind_Rec.asp?ID=459

U.S. Substance Abuse and Mental Health Services Administration. (2005). Families and Schools Together (FAST). In U.S. Substance Abuse and Mental Health Services Administration, SAMHSA model programs: Effective substance abuse and mental health programs for every community. Washington, DC: Author. Retrieved February 11, 2007, from http://www.modelprograms.samhsa.gov/pdfs/Details/FAST.pdf