William C. Hiss, Principal Investigator Valerie W. Franks, Co-Author & Lead Researcher OACAC Articulation Conference Denison University Granville, OH September.

“DEFINING PROMISE: OPTIONAL STANDARDIZED TESTING POLICIES IN

AMERICAN COLLEGE AND UNIVERSITY ADMISSIONS”

William C. Hiss, Principal Investigator Valerie W. Franks, Co-Author & Lead Researcher

OACAC Articulation ConferenceDenison University

Granville, OHSeptember 9, 2014

http://www.nacacnet.org/research/research-data/nacac-research/Documents/DefiningPromise.pdf

2

“Take Back the Conversation…” *

Ethical questions… • Can optional testing lower access barriers and expand college bound populations?• Who is exploring the breadth of human intellect and promise in imaginative ways?• Who is doing the heavy lifting, serving broad constituencies?

Practical questions… • What happens when institutions admit students without considering their

standardized test scores?• Who are the non-submitters?• How are these “non-submitter” students doing, compared to submitters?• Are college admissions decisions still reliable without testing?• Does a non-submitter policy come with any advantages or disadvantages to the

institution? (e.g., institutional geographic reach, diversity, academic achievement, tuition income and financial aid)

*The NACAC Commission on Standardized Tests included a recommendation to “take back the conversation” about testing from the various groups for whom testing was either a profession or a cause. This study is a contribution to that conversation.

3

Policy Variations

Policy Variations Not Considered in this Study: Test Flexible – Policy whereby students have the option to submit scores from other standardized tests in place of the SAT or ACT

Very Limited Options – Only specific categories, e.g., non-traditional students, specific degree programs, …

• Institutions on the Fairtest list were reviewed and categorized as follows:

1. Academic Threshold – Policy whereby students who meet certain academic criteria (e.g., rank, HSGPA) can choose whether or not to submit standardized testing scores as part of the admissions decision, or their score is still required to be submitted but waived in the admissions decision if criteria is met (typically referred to as Assured Admission or Guaranteed Admission).

2. Recommended for Placement – Policy whereby standardized testing is recommended to be submitted for placement purposes, and if submitted, is typically not used as a data point in the admissions decision.

3. Optional for All – Policy whereby almost all prospective students can choose whether or not to submit standardized testing scores (SAT or ACT) as part of their admissions application.

4. Optional Plus – Policy whereby students are required to submit something else in lieu of testing (e.g., schedule an interview or supply extra writing samples).

5. Recommended, Not Required – Policy whereby testing options are qualified for prospective students in some fashion (e.g., recommended, highly recommended, preferred, etc.).

43%

13%

23%

5%

17%

4

Policy Breakdown of our Institutions

• The institutions in our study use four of the policy category options. 60% of students in our sample entered higher education under the “Academic Threshold” type of policy, whereby meeting a HSGPA or rank threshold gained them guaranteed admission.

Optional Plus

Recommended for Placement

Optional for All

Academic Threshold

0 10000 20000 30000 40000 50000 60000 70000 80000

Number of Students Enrolled by Policy Type

Number of Students in the Study

5

Research Sample Overview

20 Private

Colleges and Universities(37,611 records)

6 Public

Universities(71,831 records)

5 Minority Serving

Institutions(12,691 records)

2Arts

Institutions(783 records)

• Total of 33 Institutions with 122,916 Student Records (Approximate 5% sample of optional testing institutions in each institutional category)

• From 23 states and US territories

• Normally submitted 4 cohort years, with data from alumni and currently enrolled students

• Pool: roughly 850 institutions listed by Fairtest, reduced to about 450 to only include 4-year, non-profit, IPEDS-submitting, and with national visibility. 120 institutions and state systems examined and contacted to choose 33.

6

Publically Announced Institutions in the Study

7

Summary by Institution Type

Private Colleges and Universities• Primarily “Optional for All” policy, the

best-known form of optional testing.• Average non-submitter population was

35% in 2010 (up from 26% in 2003).

Public Institutions• Primarily “Academic Threshold” policies based

on HSGPA or HS Rank. Often state mandated, & called “guaranteed” or “assured” admission.

• 62% average non-submitter population in 2010 (down from 66% in 2003).

• This analysis focuses on non-submitters with testing below that institution’s standardized testing threshold or average. This allowed us to better understand the students who might not have been admitted without the policy.

Minority Serving Institutions• Primarily “Recommended for Placement”

or open admission.• Average non-submitter population was

27.5% in 2010.• Data gaps limit the analysis in this dataset.

Arts Institutions• Primarily “Optional for All” • Average non-submitter population was

64% in 2010.• Portfolio ratings were included in the data

request.• Small N’s, so limited conclusions.

8

Principal Findings

1. There are no significant differences in either Cumulative GPA or graduation rates between submitters and non-submitters. Across the study, non-submitters (not including the public university non-submitters with above-average testing, to focus on the students with below-average testing who are beneficiaries of an optional testing policy) earned Cumulative GPAs that were only .05 lower than submitters, 2.83 versus 2.88. The difference in their graduation rates was .6%. By any standard, these are trivial differences.

2. College and university Cumulative GPAs closely track high school GPAs, despite wide variations in testing. Students with strong HSGPAs generally perform well in college, despite modest or low testing. In contrast, students with weak HSGPAs earn lower college Cum GPAs and graduate at lower rates, even with markedly stronger testing. A clear message: hard work and good grades in high school matter, and they matter a lot.

3. Non-submitters are more likely to be first-generation-to-college enrollees, all categories of minority students, Pell Grant recipients, and women. But across institutional types, white students also use optional testing policies at rates within low single digits of the averages, so the policies have broad appeal across ethnic groups.

4. College admissions decisions made without testing are apparently just as reliable as those made with testing. Testing may serve to artificially truncate the applicant pools of students who would succeed if they could be convinced to apply.

9

Additional Findings

5. In a surprise finding, non-submitters display a distinct two-tail or bimodal curve of family financial capacity. First-generation, minority and Pell-recipient students will often need financial aid support, but large pools of students not qualifying for or not requesting financial aid help balance institutional budgets.

6. LD students, from a modest sample of 1050 students at 8 institutions, are much more likely to apply as non-submitters, and much more likely to apply ED. They perform at levels close to the rest of their classmates. The evidence from a long-term study at Bates found that given the modest accommodations to which these students are legally entitled, their GPAs and graduation rates come up to class averages, helping to increase the institution’s overall graduation rates.

7. Non-submitters may commonly be missed in consideration for no-need merit financial awards, despite better Cum GPAs and markedly higher graduation rates than the submitters who receive merit awards. Institutions may want to examine their criteria for merit awards, especially the use of standardized testing to qualify students for no-need merit funding.

8. Non-submitters often expand applicant pools, apply Early Decision at higher rates, increase minority enrollments, expand geographic appeal, and allow for success by Learning Difference students.

10

College Cumulative GPA versus High School GPA(Aggregate Data – All students entering 2003, 2004, 2006, 2007, 2008, 2009, 2010)

1.6 2.0 2.4 2.8 3.2 3.6 4.01.6

2.0

2.4

2.8

3.2

3.6

4.0Submitter (N:41608)

Non-Submitters -- W/O Public Above-Average-Testing (N:24610)

High School GPA

Colle

ge C

umul

ative

GPA

11

College Cumulative GPA versus SAT(Aggregate Data – All students entering 2003, 2004, 2006, 2007, 2008, 2009, 2010)

600 800 1,000 1,200 1,400 1,6001.6

2.0

2.4

2.8

3.2

3.6

4.0Submitter (N:55673)

Non-Submitters -- W/O Public Above-Average-Testing (N:24824)

SAT(No Writing Score; All ACT scores converted to SAT Scores)

Colle

ge C

umul

ative

GPA

12

Without Above-Average-Testing Students

Aggregate Graduation Rate

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

15488

15488

10409

18193 students

Non-Submitter Submitter

The aggregate non-submitter graduation rate is: 65.8%.

Non-submitter rate is 1.3% higher.

The aggregate non-submitter graduation rate is: 63.9%

Non-submitter rate is 0.6% lower.

Graduation Rate

Graduation Rate Comparison(Aggregate Data – Graduated Cohorts entering 2003,2004, 2005, 2006, 2007)

13

Summary of Key Statistics - All 33 Institutions(students entering 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010)

< 0.1 = trivial difference0.1 - 0.3 = small difference0.3 - 0.5 = moderate difference> 0.5 = large difference

COLOR KEYCohen’s d

Note: Details on Cohen’s d can be found in the accompanying excel

spreadsheet.

No Significant Difference

Statistically Significant Difference p < .001

COLOR KEYChi-Square Tests

Note: Details on chi-square tests can be found in the accompanying

excel spreadsheet.

Non-Submitter

s

Submitters

n 62067 60743

High School GPA 3.45 3.28 Cohen’s d

SAT (See caveat below) 1129 1154 Cohen’s d

Cumulative GPA 2.92 2.88 Cohen’s d

Graduation Rate 65.8% 64.5% Chi-Square

Without Above-Average-Testing Students

n 36648 60743

High School GPA 3.35 3.28 Cohen’s d

SAT (See caveat below) 1041 1154 Cohen’s d

Cumulative GPA 2.83 2.88 Cohen’s d

Graduation Rate 63.9% 64.5% Chi-Square

SAT Caveat: 82.0% of Non-Submitters still submitted scores . This data only represents that 82.0%. For the second chart, the results were calculated with those students at the six public universities removed who had testing above the average of their institution. In this way, the data reflects only those students in public institutions with testing below their institutional averages who were beneficiaries of an automatic admission program based on HSGPA or HS rank,, or who chose to apply as a non-submitter in an institution that had a pure optional testing policy.

14

Academic Rating: All institutions submitted their respective scales, but for comparison purposes we converted all of them to a 10 point scale, where 10 is the highest rating.SAT Caveat: Only 41% of Non-Submitters still submitted scores . This data only represents that 41%.

Non-Submitter

s

Submitters

n 12004 24855

High School GPA 3.47 3.51 Cohen’s d

Academic Rating 6.53 6.76 Cohen’s d

SAT (See caveat below) 1096 1245 Cohen’s d

First Year GPA 2.98 3.13 Cohen’s d

Cum GPA (enrolled cohorts) 3.04 3.17 Cohen’s d

Cum GPA (graduated cohorts) 3.08 3.18 Cohen’s d

Graduation Rate** 77.7% 76.6% Chi-Square

Completion Rate** 101.4% 102.2% Cohen’s d

Underrepresented Minority 16% 9% Chi-Square

First Generation 16% 10% Chi-Square

Gender (Female) 65% 59% Chi-Square

Pell 23% 17% Chi-Square

EFC $21,790 $26,303 Cohen’s d

EFC – Adjusted for Inflation $10,570 $12,817 Cohen’s d

STEM Major 24% 32% Chi-Square

Summary of Key Statistics – Private Institutions(students entering 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010)

< 0.1 = trivial difference0.1 - 0.3 = small difference0.3 - 0.5 = moderate difference> 0.5 = large difference

COLOR KEYCohen’s d

Note: Details on Cohen’s d can be found in the accompanying excel

spreadsheet.

No Significant Difference

Statistically Significant Difference p < .001

COLOR KEYChi-Square Tests

Note: Details on chi-square tests can be found in the accompanying

excel spreadsheet.

15

SAT Caveat: Only 98.9% of Non-Submitters still submitted scores . This data only represents that 98.9%.

Non-Submitter

s

Submitters

n 19976 26330

High School GPA 3.40 3.12 Cohen’s d

SAT (See caveat below) 1037 1130 Cohen’s d

First Year GPA 2.76 2.68 Cohen’s d

Cum GPA (enrolled cohorts) 2.74 2.74 Cohen’s d

Cum GPA (graduated cohorts) 2.78 2.62 Cohen’s d

Graduation Rate** 67% 63% Chi-Square

Completion Rate** 112.2% 113.8% Cohen’s d

Underrepresented Minority 24% 15% Chi-Square

First Generation 32% 22% Chi-Square

Gender (Female) 60% 45% Chi-Square

Pell 27% 15% Chi-Square

EFC $14,825 $17,271 Cohen’s d

EFC – Adjusted for Inflation $7,409 $8,627 Cohen’s d

STEM Major 51% 53% Chi-Square

Summary of Key Statistics – Public Institutions(students without above-average-testing entering 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010)

< 0.1 = trivial difference0.1 - 0.3 = small difference0.3 - 0.5 = moderate difference> 0.5 = large difference

COLOR KEYCohen’s d

Note: Details on Cohen’s d can be found in the accompanying excel

spreadsheet.

No Significant Difference

Statistically Significant Difference p < .001

COLOR KEYChi-Square Tests

Note: Details on chi-square tests can be found in the accompanying

excel spreadsheet.

16

SAT Caveat: The average for non-submitters represents only one institution that had scores for non-submitters, so it is not an accurate comparison with submitters across the institutions. ** Graduated Cohorts Only

Non-Submitter

s

Submitters

n 3494 9197

High School GPA 2.61 3.00 Cohen’s d

SAT (See caveat below) 791 974 Cohen’s d

First Year GPA 2.52 2.76 Cohen’s d

Cum GPA (enrolled cohorts) 2.43 2.69 Cohen’s d

Cum GPA (graduated cohorts) 2.31 2.66 Cohen’s d

Graduation Rate** 24% 37% Chi-Square

Completion Rate** 114% 117% Cohen’s d

Underrepresented Minority 51% 41% Chi-Square

First Generation 42% 40% Chi-Square

Gender (Female) 55% 59% Chi-Square

Pell 49% 43% Chi-Square

EFC $8,966 $13,634 Cohen’s d

EFC – Adjusted for Inflation $4,586 $6,666 Cohen’s d

STEM Major 5% 11% Chi-Square

Summary of Key Statistics – Minority Serving Institutions(students entering 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010)

< 0.1 = trivial difference0.1 - 0.3 = small difference0.3 - 0.5 = moderate difference> 0.5 = large difference

COLOR KEYCohen’s d

Note: Details on Cohen’s d can be found in the accompanying excel

spreadsheet.

No Significant Difference

Statistically Significant Difference p < .001

COLOR KEYChi-Square Tests

Note: Details on chi-square tests can be found in the accompanying

excel spreadsheet.

Application Comparisons – Merit Award Recipients(students entering 2003, 2004, 2006, 2007, 2008, 2009, 2010)

17SAT Caveat for All Merit Recipient Chart: 90.7% of Non-Submitters still submitted scores . This data only represents that 90.7%.SAT Caveat for Merit Recipients Without Above-Average-Testers: Only 74.9% of Non-Submitters still submitted scores . This data only represents that 74.9%.** Graduated Cohorts Only

Non-Submitters

Submitters

n 13,708 13,603

High School GPA 3.69 3.44 Cohen’s d

SAT (See caveat below) 1205 1197 Cohen’s d

First Year GPA 3.21 3.07 Cohen’s d

Cum GPA (enrolled cohorts) 3.19 3.13 Cohen’s d

Cum GPA (graduated cohorts) 3.24 3.02 Cohen’s d

Graduation Rate** 81% 70% Chi-Square

Completion Rate** 109% 107% Cohen’s d

Underrepresented Minority 16% 14% Chi-Square

First Generation 20% 18% Chi-Square

Gender (Female) 57% 54% Chi-Square

Pell 16% 15% Chi-Square

EFC $21,607 $24,364 Cohen’s d

EFC – Adjusted for Inflation $10,734 $12,045 Cohen’s d

STEM Major 51% 44% Chi-Square

Non-Submitters

Submitters

5064 13,603

3.54 3.44

1054 1197

2.99 3.07

2.97 3.13

3.05 3.02

76% 70%

107.2% 106.9%

26% 14%

24% 18%

64% 54%

18% 15%

$22,702 $24,364

$11,267 $12,045

49% 44%

All Merit Recipients Merit Recipients Without Above-Average-Testers

18

Not a victory lap but a legacy lap… The next steps?• Examine the twin issues of “false negatives” of low testing with potential “false

positives” of high testing created by coaching. ₋ An ethicist’s question: if we had a medical test for a serious condition with a

30% rate of false negatives, would that be OK?₋ If 30% is the non-submitter share of enrolling students, what is the true share of

false negatives created by testing, including those who are refused, attending community colleges and for-profit colleges, or not attending at all?

• Examine college success using 4-year Cum GPAs and graduation rates rather than first-year GPAs as the principal yardstick. Add alumni and grad school outcomes in future studies. (See the Bates 25-year look-back study for some longer-term data.)

• Evaluate a broader band of research tools. Add Cohen’s d, Chi Square, bar charts and scatterplots to regression (R-square) analysis.

• Share published research on optional testing. Good models are available from Bates and Ithaca, and in Joseph Soares, ed: SAT Wars: The Case for Test-Optional Admissions. See also Bowen, Chingos & McPherson, Crossing the Finish Line: Completing College at America’s Public Universities.

• A one-day conference on optional testing?

19

Institutions Recently Adopting Test-Optional Policies

20

Questions and Discussion

William C. HissPrincipal Investigator

[email protected] Hadfield RoadMinot, ME 04258

Valerie W. FranksCo-Author and Lead Researcher

[email protected] Lindenwood Road

Cape Elizabeth, ME 04107