Using Grades over 26 Years to Evaluate a Career Course: Two Studies Robert C. Reardon, Ph.D. Stephen J. Leierer, Ph.D. Donghyuck Lee, M.Ed. Florida State.

Using Grades over 26 Years to Using Grades over 26 Years to Evaluate a Career Course: Evaluate a Career Course:

Two Studies Two Studies

Robert C. Reardon, Ph.D.Robert C. Reardon, Ph.D.Stephen J. Leierer, Ph.D.Stephen J. Leierer, Ph.D.Donghyuck Lee, M.Ed.Donghyuck Lee, M.Ed.

Florida State UniversityFlorida State University

Course History

• 1973: The career course developed

• 1980: The team-teaching of 3 or 4 instructors

• 1981: From the quarter to semester system

• 1984: Plus/minus grading system

• 1984 – 1988: Catalyst materials introduced.

Course History

• 1994: Cognitive Information Processing (CIP) theory introduced.

• 1996 – 2000: Once per week sections offered.

• 1998: Friday section offered.

• 1999: CIP text and student manual introduced.

• 1999: Internet used in students’ research

Course Objectives

1. Perceive the individual as purposefully responsible and active in the life/career planning process and to develop skills for increasing such behavior in others and oneself.

2. Understand how personal characteristics influence career development.

3. Become oriented to the socioeconomic world of work as it impacts individual and family career systems.

4. Identify appropriate academic major and/or occupational alternatives in relation to personal characteristics.

Course Objectives

5. Learn about and use a variety of information resources to explore academic major or occupational options.

6. Understand career development theories and use decision-making skills for life/career planning and management.

7. Learn about and use job-hunting skills needed for employment.

8. Formulate action plans and strategies for moving oneself or other persons toward implementation of life/career goals.

Course ActivitiesActivities & Assignments Total Points Related Course Objective(s)

Participation 30 1

Attendance 58 1

Chapter 1 Quiz 10 6

Autobiography 60 2

SDS Interpretive Report 10 2,4,6

2 Occupational Interviews 40 1,3,5

Career Field Analysis 200 3,4,5

Skills Assessment Activity 10 2,4

SIGI PLUS or Discover Computer Feedback Form 10 2,4

Choices Computer Feedback Form 10 2,4

Career Thought Inventory Profile 10 1,6

Individual Action Plan 10 1,5,8

Academic/Career Plan Project 100 1,8

Resume 15 7

Cover Letter 5 7

Performance Tests 75 2,3,5,6,7

Course Structure

• Unit I: “Career Concepts and Applications”

– Focuses on self-knowledge, knowledge about options, and decision making

– Assignments: Writing an autobiography, completing the Self-Directed Search, and a skills assessment activity.

Course Structure

• Unit II: “Social Conditions Affecting Career Development”– Focuses on current social, economic, family, and

organizational changes affecting the career planning process and the need for students to develop more complex cognitive schema to solve career problems.

– Assignments: Complete autobiography, Career Field Analysis (CFA) paper, and two information interview reports

Course Structure

• Unit III: “Implementing a Strategic Career Plan”

– Focuses on employability skills and strategies for implementing academic/career plans.

– Assignments: Two information interview reports, the completion of a resume and cover letter, and a strategic/academic career plan paper

Instructional Methods

• A mixture of lecture, panel presentations, and small and large group activities.

• Each instructor works with a small group of students in breakout sessions and evaluates their work.

• Instructors meet individually with the students to discuss their assessments and progress.

Course Grading Procedures

• Grades are based on the successful execution of a performance contract (PC) by the student.

• The PC includes 16 different graded activities spread across the three units of the course.

• 28 different activities are graded in the 3-credit version of the course.

Population

• 6,176 undergraduate students who completed the course, “Introduction to Career Development.”

• 15% to 25% of the class composed of students with officially undeclared majors.

• 60% unsure, dissatisfied, or undecided with current career situation.

• 75 academic periods (semester/quarter) studied.

Gender Distribution

Male40%

Female60%

Grade Distribution

Freshmen15%

Juniors20%

Seniors20%

Sophomores45%

Ethnic Distribution

4%12%

3% 7%

74%

4%

AmericanIndian

AfricanAmerican

Asian HispanicAmerican

Caucasian Other

Satisfaction with Current Career Situations

40%

60%

Satisfied Unsure, Dissatisfied, or undecided

Data Collection Procedures

• Study #1: Archived course grade data 1978-2004 by academic term obtained from the university registrar.

• Study #2: Archived course grade data and student evaluation of teaching (SET) data 1999-2004 by course section obtained from registrar and instructors.

Working with Archival Data

• Retrieving electronic grades by historical term– Aggregate group records by quarter/semester– Precluded analysis of student characteristics

• Issues in retrieving SET from faculty– Confidentiality and security– Course section data

• Large dataset: 6,176 students over 75 terms

Research Questions Study #1

1. Did grades provide evidence of students meeting the course learning objectives?

2. How did changes in course structures and procedures affect student learning?

3. Did grades differ by semester?

4. Did the mean grade point average earned in the course change over time?

Results

• Evidence of Students Meeting the Course Learning Objectives– 74% of the students earned a B+ or better– Mean GPA was 3.44 (SD = .84)– Negative correlation between mean semester GPA

and the semester identification number (r = -.38, p = .002)

Results

44.4%

19.2%

10.0%

13.4%

3.7%1.8% 2.5%

0.9% 0.3% 0.6% 0.3%2.8%

0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

30.0%

35.0%

40.0%

45.0%

A A- B+ B B- C+ C C- D+ D D- F

Results

• Eight course objectives are connected to the 16 graded course learning objectives.

• 3% receiving ‘F’ grade provides evidence of the course demands.

Results

• Course Structures and Procedures

M SD F (1, 6174)

Quarter system

Semester system

3.47

3.44

.88

.84.28

5-point grading system

11-point grading system

3.48

3.44

.92

.841.09

Results

• Course Structures and Procedures

M SD F (1, 6174)

Semester without Catalyst materials

Semester with Catalyst materials

3.41

3.53

.88

.7425.30*

Before CIP integrated into the course

After CIP integrated into the course

3.56

3.33

.75

.92115.06*

Before textbook used

After textbook used

3.51

3.24

.80

.90127.75*

Results

• The changes in semester system and grading system did not produce a difference in course grades.

• The intensive infusion of the work-family life balance materials associated with higher grades.

• The Infusion of CIP theory and textbook made the course more challenging and lowered grades.– Using internet-based sources in researching occupations

resulted in lower grades on the career field analysis (CFA) research paper.

Results

• Grade Distribution by Semester– Significant difference in the aggregated GPA by semester

(F = 6280.86, p < .0005)

3.48

3.58

3.39

3.25

3.3

3.35

3.4

3.45

3.5

3.55

3.6

Spring Summer Fall

Aggre

gat

ed G

PA

Results

• Course grades varied by semester.• Grades in the summer were significantly higher than

in other terms.– The summer term provides a more intensive course

experience (intensive class schedule; students’ study load).

• Grades in the fall were significantly lower than in other semesters.– Four-month intermission after registration (perhaps lowed

motivation).

Results

• Grade Distribution over Time– Significant difference in the aggregated GPA by time

(F = 23.69, p < .0005)

3.633.5

3.543.47 3.36

3.23

33.13.23.33.43.53.63.73.83.9

4

1 2 3 4 5 6

Time Period

Agg

rega

ted

GPA

1: Fall 1978 - Spring 1981

2: Fall 1981 - Summer 1985

3: Fall 1985 - Summer 1990

4: Fall 1990 - Summer 1995

5: Fall 1995 - Summer 2006

6: Fall 2000 - Summer 2004

Results

• Students in the latest time period (fall 2000 through summer 2004) had significantly lower grades than those in any other time period.

• Grade inflation was not the case with this career course.

Conclusions

• The career course investigated in this study appears to be an effective intervention as evident in student grades.

• Grade inflation was not the case for this course.

• However, grades were affected by historical events, temporal conditions, and course modifications.

Limitations

• Using aggregate grades across academic terms rather than individual student grades.

• This precluded an examination of ethnicity, gender, or other learner characteristics in this research.

Research Questions Study #2

1. What was the nature of students’ evaluation of teaching (SET) in this career course?

2. Were earned or expected grades in class sections different across semesters or class meeting times?

3. Were earned or expected grades related to SET ratings?

Sample for Study #2

• Fall 1999 – Summer 2004

• 62 course sections led by 12 different instructors who taught from 1 to 10 times

• 74% of sections reported expected grades (we found no bias pattern in missing data)

• 92% of sections reported SET ratings

ResultsOverall Student Evaluation of Teaching

(SET) very positive

Data Analysis

1 Day 2 Days 3 Days 4 Days

Fall

Spring

Summer

• The ideal design to examine the research questions about SET, Earned Grade Point Average, and Expected Grade Point Average would be a Split-Plot design.

• The data were archival, there are no observations for some Semester X Days combinations. There are two possible solutions to this problem

– Split-plot design using ANOVA for unbalanced data (paper)– Create a 7-level variable of Semester-Days (NCDA presentation)

Student Evaluation of Teaching (SET)Students’ Evaluation of Teaching

Results

Student Evaluation of Teaching (SET)Earned Grades across Class Semesters and Times

Results

Student Evaluation of Teaching (SET)Expected Grades across Class Semesters and Times

Results

Student Evaluation of Teaching (SET)

Earned-Expected Grades across Class Semesters & Times

Results

Student Evaluation of Teaching (SET)Earned and Expected Grades with SET

Results

Student Evaluation of Teaching (SET)Summary

Fall-Spring 1, 3 Fall-Spring 2 Summer 4

Student Evaluation of Teaching

Less than

the Mean

-

Greater than

the Mean

+

Less than

the Mean

-Earned Grade

Less than

the Mean

-

Less than

the Mean

-

Greater than the Mean

+

Expected Grade

Less than

the Mean

-

Greater than

the Mean

+

Greater than the Mean

+

Student Evaluation of Teaching (SET)

• The aggregated earned grade in the Summer semester was significantly higher than the aggregated earned grade during the Fall or Spring semesters.

• The aggregated expected grade was significantly different across semesters. Summer > Fall-Spring-2 > Fall-Spring 1, 3

• The aggregated expected grade was significantly higher than the aggregated earned grade.

• The difference between earned and expected grades was influenced by the Semester-Days variable. Summer > Fall-Spring-2 > Fall-Spring 1, 3

Summary

Student Evaluation of Teaching (SET)

• SET ratings on the difference of aggregated earned and expected grade varies according to semester-class times

Summer > Fall-Spring-2 > Fall-Spring 1, 3

• Difference in EGPA and XGPA influenced by SET

Summary

Student Evaluation of Teaching (SET)

• When controlling for SET, there was no significant difference between earned GPA and expected GPA.

• Aggregated EGPA and XGPA were influenced by variables related to the process of teaching, e.g., number of class meetings per week (1, 2, 3) and length of the semester (6 vs. 17 wks.).

• A significant amount of variability that exists between earned and expected grades can be traced to Student Evaluation of Teaching (SET).

Conclusions

Student Evaluation of Teaching (SET)

• Archival data analysis can become complex.• Most of the variables in these studies are aggregate measures. • Using aggregated grades as individual performance

indicators can be misleading.• SET is voluntarily collected, making for poor coverage and

probably poor utility for an ongoing archival project. • It is often unclear whether SET came as a result of the

outcome (high or low expected and earned grades) or it led to the outcome.

• These data are based on archived records collected by the University in multiple classes for the purpose of grading students in a class.

Limitations

Student Evaluation of Teaching (SET)

Presentation available at:

www.career.fsu.edu/techcenter

Robert Reardon, PhD

[email protected]

Steve Leierer, PhD

[email protected]

Donghyuck Lee, MEd

[email protected]

Contact Information