OIRE Systems Projects Department Heads Meeting, 3/11/2013 SURESH NAIR, Ph.D. Interim Associate Vice Provost for Institutional Effectiveness Professor,

OIRE Systems Projects

Department Heads Meeting, 3/11/2013

SURESH NAIR, Ph.D.

Interim Associate Vice Provost for Institutional Effectiveness

Professor, School of Business

University of Connecticut, Storrs

1

Overview

Online Student Evaluation of Teaching (SET)

HuskyDM enhancements HuskyAL – Assessment of Learning OIRE Self Service DataMart

2

Online SET – Student Evaluation of Teaching

The new 5 point form required by the Senate is being implemented

It will be an online system Students in courses ending after March

22 will receive online SET links by e-mail.

3

4

History of the SET

We have been using a paper-based 10-point SET Average of ratings is reported

In Fall 2010, a new 5-point SET was approved by Senate after trials—strong reliability and validity http://senate.uconn.edu/evaluations/SEoT.guidelines.pdf Medians of ratings are to be reported

eXplorance Blue was chosen as the vendor for the new SET—the goal was to use paper SET’s with a quick turn-around and electronic reporting

A pilot was run in Fall 2012 on a subset of courses scored using both the 10-point and 5-point SET

5

Problems with Paper SET

Multiple Problems with Paper SETs scoring Printing of paper forms is a logistical problem

due to manual nature of process Resolution of scanning is on new form poor,

requiring constant redos. New scanning software yields 10% uncertainty of student responses requiring manual intervention and “guessing”

Possible inaccuracy when instructor changes after 10th day of class and paper SETs already printed with old name.

Only one vendor was available in RFP process, since almost every major university in the country schools has moved to completely online systems

Online SET – Student Interface6

Online SET – Instructor Report7

Online SET – Instructor Report (contd.)8

9

The path forward

Rollout of the 5-point fully online scoring and reporting system planned for Spring 2013 (opt outs available)

Student scoring Evaluation and comments to be entered online Students may use computers, tablets or smart

phones Strategies to improve student response rate

Push green/sustainability angle Use promotional videos to promote online

scoring Early access to grades (say, two days before

others)

10

The path forward (contd.)

Instructor and DH reporting Will be able to see reports online much sooner

than the paper system Instructors will be able to see comments online Research has shown that the quantity and

quality of text comments online is much better than comments on paper

Opt-out: Faculty may request paper scoring after obtaining consent from DH

HuskyDM status update

The Provost Annual Report (PAR) for last year was created by HuskyDM ~95% faculty entered their information We have noted all the feedback and are

making improvements this year – discussed next

Additionally, HuskyDM can this year be used for Creating PTR form drafts Merit forms

11

Publications upload

Demo

12

HuskyDM enhancements (contd.) Some of the most common complaints were

“data not appearing in proper PAR item cell” Reason – faculty may not choose to enter all the

fields required by the logic for creating PAR Resolution – we are redoing most screens to now

directly ask for the PAR# via a drop-down menu

13

HuskyDM enhancements (contd.)

Similar to Publications Upload, we are planning to upload the following information directly into HuskyDM this year (this work is not complete) Grants and Contracts HR data Advisee data

PTR forms via HuskyDM This has been tested with ~20 TT faculty

this year This is a convenience available to TT

faculty, not a requirement

14

PAR Redesign

The Provost Annual Report (PAR) has remained unchanged for about 20 years.

A committee of DHs and users is currently looking at making it less confusing and more useful

15

HuskyDM enhancements planned (contd.)

Merit forms via HuskyDM We have contacted each school to collect

their Merit forms BUS has been using HuskyDM for merit for

5 years now Again, a convenience, not a requirement

16

HuskyAL – Assessment of Learning

We are working on an online tool to replace OATS

DHs and Program Directors are responsible to continue to Assessing their programs All programs should have Mission and

Purpose, Goals, Objectives/Outcomes and Methods, as before

All programs should track results , and specify follow-up actions, as before

The only difference is, instead of posting into OATS manually, you will upload into HuskyLM using Excel.

17

OIRE DataMart

We are working on a self service DataMart for OIRE data

Departments and Schools should be able to access routine data using a WebFocus online interface.

Will have some functionality to manipulate data somewhat like Pivot tables in Excel

18

Contact Information

Suresh Nair, Interim Associate Vice Provost for Institutional Effectiveness, [email protected]

HuskyDM Alexander del Campo, School of Business,

IT Services, [email protected]

Liming Liu, OIR, [email protected] Online SET

Cheryl Williams, OIR, [email protected] Valorie Elwell, UITS, [email protected]

Georgianne Copley, UITS, [email protected]

19

20

Comparison between 10 and 5 point scales

In Fall 2012, 39 courses were scored on both the 10-point and 5-point scales

The 5-point median scale actually resulted in higher SETs than the old 10-point scale (9% higher)Average of

10-point scale

Twice median of 5-point scale

Count of courses

5-6 73% 5%6-7 118% 5%7-8 103% 13%8-9 112% 77%Total 109% 100%