Early Warning Grades: Heeding the Warning

Early Warning Grades:Heeding the Warning

William Polley, Ph.D.Angela Lynn, Ph.D.

Fall 2012 IACRAO ConferenceSession 25.25

Early Warning Grades: Heeding the Warning

Institutional Profile

• Public, Four-Year University in West Central IL– Residential campus in Macomb, IL– Non-residential campus in Quad Cities/Moline, IL

• Founded as a Teacher’s College in 1899• Offer 66 baccalaureate degrees, 36 master’s

degrees and 1 doctoral degree• Total enrollment Fall 2012: 12,205 students

(Undergraduate: 10,263, Graduate: 1,942)

Early Warning Background

• Originated in Fall 1982• Initial target population

– Freshmen, sophomores, transfers, academic probation– Notices to students earning less than C grades

• Current population– Undergraduates in regularly scheduled on-campus courses

• Continued purpose– Timely communication

• Improve student performance• Increase retention

Purpose of Study

• Ad hoc committee of Faculty Senate established• Membership– Voting: 6 faculty & 2 students– Ex-officio: 1 academic advisor & registrar

• Charge of committee– Identify ways to improve faculty participation– Determine best timing of early warning grades– Consider conversion to midterm grades– Address any other appropriate issues

Plan for the Study

• Consider timing issue• Review early warning and grade data from Fall

2011 and retention from fall to spring• Survey faculty and students for attitude

toward early warning system• Consider midterm grades in light of grade data

and surveys• Any obvious ways to bring system up to date?

Timing of Early Warning Grades

• Easiest issue to address• Institutional processing vs. individual notices– Early enough to affect change– Late enough to have graded work

• Early warning notices primarily dictated by last day to withdraw– Online screen open to faculty in 6th week of semester

• Two weeks before notices sent to students– Notices sent to students at start of 8th week of semester

• Two weeks before withdrawal date

Early Warning System

• Specific grades are reported only for C- and below.• C or better is reported as “*”.• If the instructor does not submit, it is reported as

“N/A”.• Student and adviser receive e-mail alert if any C-

or below.• No e-mail alert if all “*” or “N/A”.• Student can see their report on STARS even if all

are “*” or “N/A”.

Early Warning and Retention

• The wrong question to ask:– What is the retention rate among students

receiving an early warning (C- or below)?– Too easy. You already know the answer.

Early Warning and Retention

• Better questionsMore informative answers• What is the effect of receiving an early

warning report by e-mail?– The “Hidden N/A” problem

• Controlling for academic factors (GPA, whether regularly admitted, whether they maintained a C or better in any class), does a “hidden N/A” matter?

Early Warning and Retention

• A “hidden N/A” does have a statistically significant impact on fall to spring retention.

• Our statistical model (probit regression) predicts:– Students with a 2.0 GPA and who had one “C or

better” (“*”) on their report but had a “hidden N/A” had a fall to spring retention rate of 83% compared to 87% for students with no “hidden N/As.”

Early Warning and Retention

• Statistically significant, but not extremely large– About 19 students for a school our size

• Much larger effect from one of our control variables– Did they maintain at least one C or better?

Early Warning and Retention

• Students who maintain at least one C or better had a fall to spring retention rate of 90%.

• Students who did not maintain a C from midterm to final had fall to spring retention rates much lower—around 56%.

• These are overall percentages, independent of GPA (which is the most statistically significant predictor of retention included in our model).

Faculty and Student Attitudes

• Surveyed students and faculty• Focused on faculty who taught freshmen– About 70% of responding faculty (N=190)

• Most questions on a 5 point Likert scale from 1=Strongly Disagree to 5=Strongly Agree

Faculty Attitudes

• Among faculty teaching freshmen…– 43% can determine midterm grades with a high

degree of precision.– 44% can distinguish students in serious difficulty,

but not specific letter grades at midterm.• Full professors and faculty not teaching

freshmen were more likely be in the 2nd group (can’t determine specific letter grade).

Faculty Attitudes• Question A: The nature of my course is such that midterm

grades or early warning grades are unnecessary in my opinion.• About 19% of faculty respondents either agree or strongly

agree.

Faculty Attitudes• Question B: Midterm grades or early warning grades are a good

idea in general, but my students don’t need them because they should know where they stand.

• About 39% of faculty respondents either agree or strongly agree.

Faculty Attitudes• Question C: I am aware of students of mine who have

dramatically improved their performance after receiving a low early warning grade.

• About 41% of faculty respondents either agree or strongly agree.

Faculty Attitudes• Question D: Informing students of their performance throughout

the semester, including giving early warning grades, is something that I consider to be a priority.

• About 66% of faculty respondents either agree or strongly agree.

Faculty Attitudes• Question E: If no students are earning less than a C at the time

when early warning grades are due, then I am less likely to log into the system and submit the early warning grade roster.

• About 26% of faculty respondents either agree or strongly agree.

Faculty Attitudes• Question F: The culture of my department/college

encourages the submission of early warning grades.• About 59% of faculty respondents either agree or strongly

agree.

Faculty Attitudes: Major Findings

• 63% (12 of 19) of faculty who did NOT submit early warning grades responded with agreement that they are unnecessary (question A).

• 50% (9 of 18) of faculty who did NOT submit early warning grades responded with agreement that midterm grades are a good idea in general but that their students should know where they stand (question B).

Faculty Attitudes: Major Findings

• All 11 of faculty who strongly agree with the statement that they are aware of students who have improved as a result of early warning grades did submit early warning grades.

• All 41 of faculty who strongly agree with the statement that communication is a priority submitted early warning grades. (94% combined agree and strongly agree).

Faculty Attitudes: Major Findings

• Only 54% (7 of 13) of those who strongly disagree that communication is a priority submitted early warning grades.

• Submission rates decreased monotonically with increasing agreement with question E.

Student Attitudes

• The following slides report the survey results for freshmen only.– Retention focus

• Number of responses from students by class were 79, 81, 138, and 148 (Fr., So., Jr., Sr.)

• Upperclassmen responses more reflective of their experience

• Summary of differences between freshmen and upperclassmen at the end of this section

Student Attitudes

• Question A: Receiving early warning grades is important to me.

Student Attitudes

• Question B: If I received an early warning grade of C- or below, I would contact the instructor of that course as soon as possible.

Student Attitudes

• Question C: I am frequently surprised by my early warning grades.

Student Attitudes

• Question D: If I do not receive an early warning grade in a course, I have other ways of determining my grade (e.g. Western Online).

Student Attitudes

• Question E: I would like to receive a specific letter grade in each course at midterm.

Student Attitudes

• Question F: If I received an early warning grade of C- or below, I would be likely to drop the course.

Student Attitudes

• Question G: If I received an early warning grade of C- or below, I would consider dropping the course, but am more likely to remain in the course.

Student Attitudes

• Question H: If I received an early warning grade of C- or below, I would likely be contacted by my academic adviser or other University staff member regarding my grades.

Student Attitudes

• Question I: If I received an early warning grade of C- or below, I would seek additional help from tutoring labs, the Office of Academic Services, my academic adviser, and other resources as necessary.

Student Attitudes

• Question J: If I received an early warning grade of F, I would be likely to drop the course.

Student Attitudes: Major Findings

• A. Freshmen are more likely to say that “Receiving early warning grades is important to me.”

• E. Freshmen are more likely to say, “I would like to receive a specific letter grade in each course at midterm.”

Student Attitudes: Major Findings

• H. Freshmen are more likely to say, “If I received an early warning grade of C- or below, I would likely be contacted by my academic adviser or other University staff member regarding my grades.”

Student Attitudes: Major Findings

• I. Freshmen are more likely to say, “If I received an early warning grade of C- or below, I would seek additional help from tutoring labs, the Office of Academic Services, my academic adviser, and other resources as necessary.”

• J. Freshmen are less likely to say, “If I received an early warning grade of F, I would be likely to drop the course.”

The Secret of Our Success: Participation

• First, do no harm!• Voluntary system• Participation by faculty teaching freshmen is

reasonably good• Lower participation in classes where early

warnings are probably not critically needed (physical education activities, music ensembles, and the like)

Midterm Grades?

• Changing to a midterm grade system reporting specific grades (A, B, C, D, F with +/-) would probably not improve retention.– Simply maintaining a C or better is what we find to

be critical to retention.• Participation in a voluntary system of midterm

grades would almost surely be lower than early warning system.

• Mandatory midterm grades seem redundant in an era of online gradebooks.

Low Hanging Fruit?

• Is there any easy change to the system that would have the potential to extend the reach, encourage more participation, etc.?– Yes! Online courses

• Online courses had been excluded from the population of courses for which early warnings were submitted.– Why? Possibly because when online courses first

appeared, they were often asynchronous, not to mention experimental. Not so today.

Committee Recommendations

• Maintain the existing timetable for submission of early warning grades

• Maintain the system of early warning grades, as opposed to specific midterm grades

• Expand the early warning grade population to include online courses

• Recommend to Provost that resources be allocated for the purchase of GradesFirst for the Office of Academic Services

• Study the effectiveness of the inclusion of online courses and the use of GradesFirst after their implementation

Questions???

Fall 2012 IACRAO ConferenceSession 25.25