August 2005UCCSC San Francisco “Challenges and Advantages of Online Class Evaluations.” Network & Academic Computing Services (NACS)

August 2005UCCSC San Francisco

“Challenges and Advantages of Online Class Evaluations.”

Electronic Educational Environment (EEE)http://eee.uci.edu/

UCI’s Course Management System

Network & Academic Computing Services (NACS)http://www.nacs.uci.edu/University of California, Irvine

Please silence cell phones.

THANK YOU!






Challenges and Advantages of Online Class Evaluations

Shohreh BozorgmehriManager, Instructional Web Technologies

&

Briandy WaldenWeb Developer, Instructional Web Technologies






3

TOPICS

1. BACKGROUND

2. CHALLENGES

3. TRANSITION

4. ADVANTAGES & RESULTS

5. FACTS & FIGURES

6. FUTURE

7. CONCLUSION

8. QUESTIONS






4

BACKGROUND

• Paper• Online

• Version 1: Instructor-initiated• Version 2: Department Administrative Tool for Evaluations (D.A.T.E.)

InspirationInspiration






5

BACKGROUND

• Paper Process• Purchase Scantron forms• Organize & distribute packets• Fill in forms• Collect & return forms• Scan forms• Analyze results• Distribute results to instructors






6

BACKGROUNDAcademic Senate Role

• EEE final evaluations are based on the Campus-wide Teaching Evaluation Form (CTEF) approved by the academic senate committee, the Council on Student Experience (CSE).

• Senate proposal:

http://www.senate.uci.edu/9_IrvineManual/3ASMAppendices/Appendix11.html

http://www.senate.uci.edu/9_IrvineManual/3ASMAppendices/Appendix11App2.html






7

BACKGROUND

• Instructor initiated• Campus-wide Teaching Evaluation Form (CTEF)• Secure and anonymous• Fixed one time email announcement• Automated analysis of results

First Online Evaluations






8

BACKGROUND

Online evaluations had initially been driven by faculty requests.

With broader interest, the needs of entire Schools or Departments became the new driving force behind

the direction of the online evaluation tool’s evolution.






9

BACKGROUNDSchool of Engineering • Engineering was the first

school to use EEE’s tool school-wide• Entire school switched to

online Final Evaluations• Simple Administrative Web

page was developed• School’s evaluations fully

online by Fall 2003• Learned from experience and

incorporated feedback into version 2






10

CHALLENGES• Significant Culture change

• Concern of response rate decrease• Students would become responsible for taking and submitting evaluations on

own time and outside of class• Duration changed from 15 minutes inside the classroom to what individual

students chose on their own• Instructors would need to diligently notify and remind students in order to

receive desirable submissions• Departments would need to communicate to faculty and/or directly enrolled

students

ENCOURAGEMENT & COMMUNICATION BECAME VITAL






11

CHALLENGES• Evaluation requirements:

• CTEF format• Anonymity• Department Customizability

• Cooperation from academic units• Willingness to take part in a potentially difficult transition• Strong history of good support and cooperation helped convince Departments

that their needs would be met, and any unexpected issues would be addressed

• Instructor & Student participation• Notification of students encouragement to participate required for best results






12

TRANSITION PROCESS OVERVIEW

• TOOL FEATURES• ACTIVATION• ANNOUNCEMENT• MONITORING• RESULTS• WEB INTERFACE• STUDENT EXPERIENCE






13

TRANSITIONTool Features

Departments steered development with requests:• Optional addition of four custom questions• Scantron-like results file• Printable report with page breaks• Response rate monitoring tools• More department control over instructor options• Email reminders for multiple evaluations combined into one• Only non-respondents receive reminders






14

TRANSITION

ActivationAdministrators

• Create and upload CSV files containing instructor IDs and course codes

• Specify the bounds of instructor control over evaluation windows

Process






15

TRANSITION

Announcement• Administrators compose, schedule,

and send email announcements and/or reminders using a simple online form through the Web interface

• No limit to the number of announcements sent

• Only non-respondents do receive reminders

• Reminder emails can have different subject-line or body each time

Process






16

TRANSITION

Monitoring• Administrators monitor rate of evaluation submissions• Constant monitoring helps department administrators decide when to send

announcements or reminder emails

Process






17

TRANSITIONProcess

ResultsSpeed: • Administrators, evaluated instructors, and

teaching assistants view and download results files in desired format(s) after the “Registrar’s Final Grade Submission Deadline”.

Flexible Formats:• Comma Separated Value (CSV)• Print-friendly report• Scantron compatible






18

TRANSITION

• Error check• Prevents user errors from

adversely affecting the process

• Instructor abilities• Preserves option to allow or

disallow instructors from changing evaluation dates/times, or canceling evaluations

• Retains midterm evaluation options for all instructors, regardless of Department-activated final evaluations

The D.A.T.E. Web Interface makes Department activation of final evaluations for

multiple instructors easy and efficient.

Web Interface






19

TRANSITION

• Online form• Requires secure authentication• Available anywhere, at any time during the

evaluation window• Quick and easy multiple choice questions• Provides space for typed open-ended responses

• Incentives• “Respondent List” makes incentives possible• Anonymity preserved

• Names cannot be associated with responses• List only available if 5 or more responses

received (minimum increasable at department discretion)

Student Experience






20

ADVANTAGES & RESULTS• Save Time: 90% savings common• Save Money: $2000/yr for Scantron (PhySci)• Add Customizability: Add 4 custom questions

• Unexpected side-effect: • With online evaluations, students have begun to submit more thorough and

detailed comments, presumably because they are under no time pressure, and are able to type, rather than handwrite, their comments.

• Because comments are already typed, they can be distributed and read more easily and quickly.

• Results are compiled and ready for distribution and review in matter of hours rather than months.

• Faculty are enabled to powerfully and immediately reflect the evaluation results to the upcoming class.

• Some departments more rapidly are able to renew or take away TAs or lecturers appointments.

DEFINITE IMPACT ON QUALITY OF TEACHING






21

ADVANTAGES & RESULTS

EEE’s Evaluations tool is the fastest growing tool in the EEE course management system toolbox






22

FACTS & FIGURES1 2 3 4 5

1. SurveyMonkey

2. QuestBack

3. Zoomerang

4. Paper Evaluations

5. D.A.T.E.

Supported

Unsupported

Comparison






23

FACTS & FIGURES

2981

8507

17934

96877

0

10000

2000030000

4000050000

6000070000

8000090000

100000

2001-2002 2002-2003 2003-2004 2004-2005

Number of Online Forms Submitted All Quarters






24

FACTS & FIGURES

7575

8897

3261

2251

4787

6544

4278

9430

5170

1193

55944619

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

247 234 255

356

127

279248

116

32

381

179

1030

50100150200250300350400

W04 S04 F04 W05 S05

Submissions

Classes

Engineering

Physical Sciences

Biological Sciences






25

FACTS & FIGURES

7 7 8

16

21 21

24

0

5

10

15

20

25

30

F03 W04 S04 F04 W05 S05 F05*

Number of Departments using Online Evaluations

* Projected Department participation for Fall, 2005






26

FACTS & FIGURES

100

1000

1

35

8

80

8

18

7

14

0%

20%

40%

60%

80%

100%

ENGR(90%)*

PHYS(97%)

CHEM(90%)

MATH(94%)

ECOEVO(50%)

Department Hours Spent Before Online EvaluationsDepartment Hours Spent After Online Evaluations

* Numbers in parentheses represent percentage of time saved per department






27

FUTURE• More customizability

• New survey engine will power tools:

• Evaluation tool

• Survey tool

• Quiz tool

• More mobility• Ability to share Evaluation forms with others• Re-use frequently used questions & answers






28

CONCLUSION

D.A.T.E.’s strength and key success come

from the solid partnership our team has

created and maintained with a wide

range of senior academic faculty,

committees, and campus units.






29

QUESTIONS?

For questions, please contact:

Shohreh Bozorgmehri

[email protected]

Briandy Walden

[email protected]

http://eee.uci.edu/iwt/uccsc2005/

or

[email protected]

August 2005UCCSC San Francisco “Challenges and Advantages of Online Class Evaluations.” Network & Academic Computing Services (NACS)

Documents

online evaluations slide

uccsc san francisco

university of california

advantages results

inspiration slide

academic senate committee

background instructor

return forms scan forms