Mixed Methods Dissertation Design Challenges and Opportunities: A Sequential, Explanatory Approach to Studying Students’ 2/4 Transfer Robin LaSota, PhD,

Mixed Methods Dissertation Design Challenges and Opportunities: A Sequential, Explanatory Approach to Studying Students’

2/4 Transfer

Robin LaSota, PhD, University of WashingtonPost Doctoral Research Associate, University of Illinois Urbana-

Champaign (UIUC)Office of Community College Research and Leadership (OCCRL)

Higher Education Collaborative PresentationFebruary 25, 2014

College of Education at the University of Illinois Urbana-Champaign

Presentation Overview

• Evaluation Criteria – Mixed Methods Research• Dissertation Design Summary• Quantitative and Qualitative Strands

– Design Choices– Paradigm– Findings– Policy Inferences

• Analytical Reflections• Mixed Methods Analysis Revisited

Some Criteria for Evaluating Mixed Methods Research

• Clearly stated research purpose(s) and reasons for design choices about methods and mixing

• Alignment between research methods and purpose

• Conclusions offer the “value added” from mixed methods approach

• Conclusions include an analysis of study legitimacy, limitations, and paradigms used for mixed method analyses

Reasons for Mixing Methods

1. Triangulation2. Complementarity3. Development (of sampling, instrumentation,

measurement decisions, etc.)4. Initiation (of theoretical perspective, new questions,

contradictions)5. Expansion of breadth and range of inquiry

From: Greene, Caracelli, and Graham (1989) in Creswell and Clark (2011), p. 62.

My Research Purpose

• To offer “complementary findings regarding strategies, policies, and factors including improvements to students’ upward transfer, particularly for students most at risk of not transferring and attaining a baccalaureate degree” (p. 251)

• Each strand of inquiry (quantitative and qualitative) has separate research questions and purposes

A First Area of Inquiry

Q1.1 How do student behaviors, community college characteristics, and state policies and conditions influence students’ upward transfer probability?

Q 1.2 How do these factors, policies, and conditions influence upward transfer probability, particularly for low-income and first-generation community college students?

Student, College, and State Factors Influencing 2/4 Transfer

Academic and MjSocial Mj

Integration, MjCollege Major, MjCollege GPA... Mj

State Articulation and MjTransfer Policies and State Mj

Characteristics Mj

Public Two-Year Colleges' MjCharacteristics Mj

Six-Year Transfer MjProbability from MjTwo to Four-Year Mj

Institution Mj

Student MjBackground Mj

Characteristics Mj

Precollegiate MjTransfer Mj

Intentions Mj

Factors Affecting MjCollege Mj

Persistence (Work MjHrs, FT vs. PT) Mj

A Second Area of Inquiry

Q2.1 What are promising practices in colleges and states aimed at improving students’ upward transfer, and how may they constitute a system of support?

Q2.2 How do leaders engage in ongoing

innovation around these practices?

Explanatory Sequential Mixed Methods Design

Multi-

level modeling analysis of BPS

and supplemental data 2003-

09

Case studies of 6

colleges in 3

states

QUANT/QUAL

Integration:

1) Using QUANT

analysis to guide QUAL

sampling

2) Cross-reference

claims from QUANT and

QUAL3) Use QUAL

to complement and extend

QUAN.

Research Paradigm – Quantitative Strand

• To “quantitatively measure theoretically and empirically-derived factors (from prior research) found to produce a greater likelihood of students’ transfer from public two-year to four-year institutions in a recent, nationally representative sample, particularly for students who are low-income and first-generation to earn a bachelor’s degree.” (pp. 7-8)

Sampling Design Choices from BPS

• N=5010 community college students; weighted N=1,528,900

• Students not co-enrolled in two or more colleges• BPS nationally representative longitudinal population

survey of all postsecondary entrants• BPS not representative of states, colleges, or CC entrants• State and community college factors investigated build

upon existing literature

• State articulation and transfer policies? Very little, if any influence (Roksa, Kienzl, Goldhaber & Gross)

• State cooperative agreements? Maybe. (Kienzl)

• Community college practices? Depends. Perhaps not much. (MDRC)

• Community college expenditures? Slight/student services expenditures (Gross and Goldhaber). None (Stange).

• Community college smaller size, higher faculty-to-student ratio? Yes. (Bailey et al., Gross & Goldhaber)

• Degree of college mission stratification (emphasis on transfer-oriented programs vs. non-transfer oriented, e.g. health/vocational/technical). Influential. (Dougherty)

• Proximity and selectivity of nearest public four-year institution? Maybe. (Rouse)

Which State and Community College Factors May Influence Transfer?

Rationale for Multi-Level Modeling Methodology

Results of Unconditional Model or Intra-Class Correlation –• State location – explains 2% of variance in 2/4 transfer probability • Primary college attended – explains 6% of variance• Student characteristics – explains most of the variance

Therefore, used multi-level logistic regression –• Randomly varying intercepts and slopes between colleges and states for

– Low-income, first generation– First generation, not low income– Planned to transfer at time of entry– Declared major in health/vocational/technical field

Positive Predictors Associated with Upward Transfer Probability

Se-ries1

0.00 0.05 0.10 0.15 0.20 0.25

0.010.040.050.06

0.070.08

0.090.09

0.120.21

0 = 50/50 Probability of 2/4 TransferConditioned on Factors in the Model

Primarily Full-Time

Planned to Transfer at Entry

Aged 15-19 at Entry

Worked 1-19 Hrs/Wk on Average

GPA in First Year (tenths)

Sports Participation Often or Sometimes

STEM, Humanities, Education Major

Gross State Product (standardized)

Academic Advising Often or Sometimes

CC Transfer Out Rate

Negative Predictors Associated with Upward Transfer Probability

Series1

-0.25 -0.20 -0.15 -0.10 -0.05 0.00

-0.01

-0.02

-0.04

-0.04

-0.14

-0.15

-0.19

0 = 50/50 Predicted Probability of 2/4 TransferConditioned on Factors in the Random Effects Model

Primarily Part Time

First Generation, Low Income

First Generation, Not Low Income

CC Pct Health Voc Completions

Took Any Remedial Education

Unemployment in CC's County

Health, Vocational, or Prof/Tech Major

College Characteristics: Association with Upward Transfer

• Proportion of associates’ degree completions in health/vocational fields (neg., p<.10)

• College transfer-out rate (2% increased odds of transfer) in regression without analysis of random effects by slope

• County-level unemployment (neg., p<.10)

Not sig. = i.e. per-student expenditures for instruction or student services, distance to nearest public four-year institution, distance to nearest non or less-selective four-year institution, faculty-to-student ratio, community college enrollment size, percent of full-time faculty, percent of full-time students

State Policies’ Association with Students’ Upward Transfer

Policy Components- Transfer data reporting- State transfer incentives- State transfer guide- Transferable general education

curriculum

- Statewide cooperative agreements- Common course numbering- Statewide articulation/transfer

policy

• Main Effects Model, Random intercepts only, no varying slopes+ 35% higher 2/4 transfer odds: State with one standard deviation higher Gross State Product Per Capita in 2003

• None of the State Articulation and Transfer Policy Components explained variance in 2/4 transfer probability.

Regression Results: Slopes for Sub-Populations that Vary by College and/or State

Low-Income, First Generation:• Higher gross state product• Common course numbering• College transfer-out rate

First Generation, Not Low Income:• Higher Gross State Product• Common Course Numbering

Planned to Transfer (vs. Not Transfer Intending):

• College transfer-out rate

Health/Vocational Major (vs. business/undeclared):

• State articulation/transfer policies not sig.

• Transfer-out rate not sig.

Random and Fixed Effects Modelshowed that these factors moderate 2/4 transfer probability

for these populations.

Green = Sig. at p<.05; Gray=Not Sig, p>.05

Findings:Quantitative Inquiry Strand

• Affirmed prior research about ambiguous or unknown effects of state transfer and articulation policies

• Offered new evidence about the role of state common course numbering in increasing first-generation students’ transfer

• Influential college-level factor – College mission focus; college’s transfer-out rate

• Full-time attendance and transfer intention are particularly influential student factors

Some Implications: Quantitative Strand

• Promising areas for policy intervention, esp. in high schools – Help students create specific plans for obtaining a

bachelor’s degree aligned in a specific field and outline a transfer pathway

– Promote continuous full-time attendance and advising with incentives and accountability

– Widely promote available state resources and policies for improved transfer

Rationale for Case Study Design

• Goal: To explore and identify possible state policy actions and college policies or practices that enhance student 2/4 transfer probability

• Structure analysis for meaningful contrasts relative to the goal– States and Colleges with Higher Transfer vs. Average Transfer Rates

(within their state)– Policy Innovative States in Articulation and Transfer– Colleges Engaged in Data-Use and Innovation– States with significant CC sector and states & colleges with student

populations of interest

Research Paradigm – Qualitative Strand

• To offer “a more nuanced and detailed picture” of college and state practices supporting students’ upward transfer, that “built on and complemented my quantitative findings” (p. 101)

• Use framing literature to “interpret institutional cultures, data use, and innovation in practices at the six colleges”…and “to help pinpoint the context for innovation…[as well as] transfer-related practices.”

State Case Selection: Florida, Georgia, and Washington

• Used OLS regression to find states performing above average in transfer, controlling for state and student population characteristics

• Considered prior research on policy innovative states in transfer and articulation

• Chose states with a considerable proportion of postsecondary students enrolled in two-year colleges and with racial/income diversity

College Selection: Above-Average and Average Performer

• Used OLS regression to find colleges performing above average in transfer, controlling for college and student population characteristics

• Consulted State Higher Education Executive Officers (SHEEO) from each state and Aspen Prize Top 120 data

• Used SHEEO advice and college’s participation in Achieving the Dream indicators of data-use and innovation

Qualitative Methods

•Interviews with state policy officials in articulation and transfer (N=20)

•Interviews with college administrators, faculty, and student affairs staff (N=110)

•Individual interviews and focus groups with students (N=49)

•N=179 overall

Semi-structured, one-hour interviews

Analytic Strategy: Analytic Memo Writing and Data Synthesis

Findings: Qualitative Inquiry Strand

Academic and MjTransfer Advising Mj

Practices Mj

Data Use for MjDecision Support Mj

Leadership Support Mjand Structure for Mj

Innovation Mj

State Policy, Influential MjNetworks and Labor Market Mj

Trends Mj

Low-Income, First- MjGeneration, Under- Mj

Represented MjStudents Mj

Findings – Advising in Above-Average Performers

Transfer not a universal outcome or push for all students…

College-level systems of support for transfer generally constrained…

Above-average colleges generally have:

•Academic leaders who champion students’ transfer and successfully engage others in this work

•Mandatory student advising models

•Student affairs staff dedicated to coaching students’ on transfer

Findings – Advising in Above-Average Performers

Above-average colleges also tend to have:

•Faculty contracts which include student advising hrs.

•Faculty and staff engaged in planning out-of-class supports and enrichment experiences for students that aid transfer

•Campus supports for TRIO and similar STEM programs for low-income, minority, and first generation students

•Key Support for Stronger Advising: Active communication/coordination with public and private four-year institutions within major fields by administrators and faculty

Findings – State Policy as a Context for Colleges’ Innovation

Common Course Numbering: Lessons Learned from Florida

Moderating positive influence of common course numbering (CCN) for first-generation college students from quantitative inquiry…• CCN Proxy for a more robust transfer policy context?

• CCN built from communication across lower and upper division faculty and programs

• Florida: CCN in place for 30 yrs; created when 2 yrs and 4 yrs were governed in one system

Some Implications:Qualitative Inquiry

States: • Incentives and support for college-level innovation• Support for measuring innovation effectiveness• Build transfer into performance accountability

Colleges: • Collaborative problem-solving re: transfer• Broad implementation of personalized learning & transfer advising• Incentives to be transfer champions

States and colleges: • More efficient, accessible processes to using data for decision support about

students’ transfer

Analytical Reflections - Quantitative

• Data Limitations– BPS measures of academic and social integration– State policy measures binary coding– No adequate measure of policy strength for the period– Available college-level data mostly not predictive of transfer

• Not a causal inference multi-level model• Does not examine reasons for stopping out or mixed

attendance – e.g. role of financial aid

Analytical Reflections - Qualitative

• Examined broad scope of practices affecting transfer probability (from pre-college to graduation check/final term advising) rather than one or two specific innovations

• Used analytical memo writing not software-based coding methodology

• Inductive approach to claim formulation rather than deductive hypothesis-testing

• Different framing literatures inform each strand, complementary analyses, not necessarily integrated

Mixed-Methods Analysis Revisited

• “Complementary strengths, overlapping weaknesses” of the two strands

• Strands answer different questions, however, from the QUAN findings, QUAL cases consider:– Role of common course numbering– Students’ experience with academic advising and

colleges’ work to reform it– Additional policy supports for improved upward transfer

(not represented in original coding)

With Appreciation• To all the participants in my study• To Debra Bragg and OCCRL for post-doc support• To NISTS for the honor of the award and presentation with you• To my chair, Bill Zumeta• To my co-advisor, Marge Plecki• To my committee members:

– Mike Knapp– Bob Abbott– Jennie Romich

• To my fellow doctoral students• And to IES and AIR for funding

Sponsored by the US Department of Education, Institute of Education Sciences (#R305B090012)and the Association of Institutional Research Dissertation Grant

Questions/Discussion

Follow up contact information:[email protected]

Office of Community College Research and LeadershipCollege of Education at Illinois

[email protected]