Conversion of Faculty Evaluations to an On- Line Format Catherine Hackett Renner SUNY Geneseo Larry Piegza Gap Technologies, Inc. OnlineCourseEvaluations.com.

Conversion of Faculty Evaluations to an On-Line FormatCatherine Hackett Renner SUNY Geneseo

Larry PiegzaGap Technologies, Inc.OnlineCourseEvaluations.com

The Paper Process ~22,000-25,000 forms per term

were hand sorted into course sections

Distributed to faculty the last 2 weeks of classes Faculty returned them to IR

Office Forms were separated, scanned,

then hand sorted back into course sections Invariably evaluations would

be turned in late Faculty received the results

back mid-February

Complaints with Process Faculty

Time delay in getting results back Late returns were not processed

Students Anonymity Detail in comments

Classroom time did not allow for thoughtful detail in comments

Convenience Would like to see results online

Conversion to On-line Format

Internal conversion was not feasible Migration to new system Internal survey software was not secure

enough for this process Decision was made to outsource

Constraints on company Had to use our own instrument Had to be able to conform to the levels of

access that were dictated by Faculty Senate

Company Choice

Best met our needs Could take our current evaluation system and upload it Could reproduce our levels of access Very customer-oriented

Spring 2006 Pilot

Students enrolled in Psychology classes and Astronomy Laboratories provided their Student Evaluation of Faculty Instruction (SOFI) ratings in an on-line format rather than in-class paper format 51 course sections 1,199 individual students providing1,806

evaluations These 1,199 students were also able to

compare the on-line version to in-class paper format (used in their other classes)

Getting the Word Out Students were informed of the pilot by their course

instructors at the beginning of the semester One week prior to the SOFI evaluation period

informational flyers were posted within the classrooms of the 51 sections

Article on the process was published in the student newspaper

Student Government Association endorsed the process and was involved with disseminating information

On the first evaluation day faculty were given handouts to distribute to their students explaining the process

The evaluation period started on April 17th and ended on May 3rd

Pilot Procedure

Students received an e-mail with their Login ID and password E-mail contained direct link the login webpage

E-mail reminders 48 hours Reminders stopped once all evaluations were

performed

Faculty received e-mails requesting they remind the students of the evaluation period Within the e-mail faculty also received the

response rate for each of their courses

Pilot Results

The response rate for the pilot was 76%! This was consistent

with the response rate for the in-class SOFI (70-73%)

Some bumps in the road All occurred within the

first 2 days All were minor All were resolved in

less than 24 hours

Student Satisfaction Survey

Seven survey questions designed to assess the following: Anonymity Detail of Comments Convenience Choice

This final question was designed to be the “litmus” test of the pilot

This question asks students which format they would prefer to use if given the choice

Anonymity

In general, I believe in-class SOFIs are: Percent

In general, I believe on-line SOFIs are: Percent

not anonymous 3.7 not anonymous 6.2

slightly anonymous 13.6 slightly anonymous 7.4

somewhat anonymous 57.4 somewhat anonymous 34.6

completely anonymous 25.3 completely anonymous 51.9

Detail of Comments

Was the amount of detail you provided in your comments different in the on-line version

compared to the in-class version? Percent

Yes, I was able to provide more detail than in class 27.8

Yes, I provided less detail than in class 13.6

No, it was about the same 57.4

Convenience

How convenient was it to use the on-line method compared to the in-class method? Percent

Not convenient at all 4.9

somewhat inconvenient 14.8

somewhat convenient 32.1

very convenient 47.5

Results

How often have you referred to the SOFI results in any given term? Percent

never 70.4

occasionally 22.2

often 3.7

very often 3.1

Results

SOFI results will be available for students within 3 weeks of the last day of completion and will be accessible on-line. Given this, how likely will you refer to the on-line SOFI results?

Percent

never 13.6

occasionally 38.3

often 28.4

very often 19.1

Choice

If you could choose which SOFI format

would you prefer to use? Percent

on-line 53.7

in-class paper 17.3

doesn't matter 27.2

Decision

We went to a full implementation Fall 2006

80% response rate Smooth run

What SUNY Geneseo Did Well

Create a login link from their web portal Erased student confidentiality concerns

by selecting an off-site vendor  Communicated with their teachers Shared results to their students 

What SUNY Geneseo Did Well

Sent out regular emails to their students Have teachers hand out directions Ran ad in school newspaper  Mobilized their student government

Top 10 Things Not To Do When Going Online:

10) Have a one week evaluation window for your first semester

9) Decide on a vendor two weeks before your evaluation starts

8) Spam your students

7) Change your course of action based on the teacher suggesting things at the current time

Top 10 Things Not To Do When Going Online:

6) Run concurrent with paper

Actual school's response rates

2004

20062005

0%

10%

20%

30%

40%

50%

60%

70%

Top 10 Things Not To Do When Going Online:

5) Buy a system that doesn't allow your students to click a "I dropped this class" button.

4) Don't communicate to your teachers

3) Ignore your vendor’s advice / Go it alone

2) Ask 100 questions on each survey

Top 10 Things Not To Do When Going Online:

1) Send a two page email lecturing your students on their civic responsibility to fill out course evaluations

8 Must Haves

1) Patience to change your school’s culture

Response Rates For Edgewood College

0%

10%

20%

30%

40%

50%

60%

70%

2004 2005 2006

Year

Fall

Spring

Summer

8 Must Haves

2) Eliminate common questions for team taught courses

3) Prizes to give away to students

4) Share results to students

5) Allow teachers to email to their students

6) Have a contest

8 Must Haves

7) Follow-up question technology

Did the instructor speak audibly and clearly?

What could the instructor have done to speak more clearly?

Get a microphone Not face the blackboard

Speak slower Speak faster

8 Must Haves

8) Dropped class survey

What was the primary reason why you dropped this class?

Course related Instructor related Financial issue

Scheduling issue Something about me Other