Implementing and Launching Rubrics in a Clinical Setting

1ExamSoft Assessment Conference | #ASSESS2015

ExamSoft Assessment Conference

#ASSESS2015


Implementing & Launching Rubrics in a Clinical Setting


Introduction


Learning Objectives

• Develop a plan to implement ExamSoft rubrics on a large scale at their institution and in a clinical setting.

• Discuss the benefits of using ExamSoft rubrics in both a simulated and live clinical setting.

• Determine methods of addressing issues during pilot testing of performance-based assessments.

• Understand the importance of pulling longitudinal reports using rubric results to help improve student outcomes.


Presentation Overview

• Introduction of Presenters• Background on College of Dental Medicine at Western

University of Health Sciences• Laying out the Logistics• Faculty Buy-in and Training• Pilot Testing with Lead Faculty• Addressing Issues during Pilot Test Period• Launching ExamSoft Rubrics in a Live-Patient Setting• Live Demo Simulation of Rubrics • Our Conclusions and Plans for Improvement• Interactive Q&A with Presenters

6


College of Dental Medicine



• Clinical Exams– On The Fly– Challenged Individually– Two Faculty Graders– 3rd & 4th year students– 10 Clinical Exams


Previous Process

• Staff Time• Resources (i.e. 2 Staff Members + Paper)• Temporarily Misplacing Rubrics• Turnaround Time for Releasing Scores• Calculation Errors


Previous Process

Student challenges

clinical exam

2 faculty grade on

paper rubrics

Faculty submit

rubrics to clinic staff member

Staff member

calculates scores

Staff member delivers

rubrics to Office of

Academic Affairs (OAA)

OAA staff scans & files

rubric, releases score to

student, and updates

grade sheet.

1 clinic session

1 clinic session Varies 3 hours Varies 8 hours

11


Simulated Clinical Exams

• Beta-tested ExamSoft rubric feature (Feb. through April 2014)– Mock WREB Endodontics – Complex Restoration


Before Implementation

• Establish long-term goals• Have a clear vision of what you need• Develop an implementation process


Establish Long-Term Goals

• Determine types of reports to generate• Create a timeline for implementation• Assign roles or tasks

15


Clear Vision

• Live Patient Clinical Exams• Support from Administration• Unique clinical exam logistics requires

customized ExamSoft set-up


Implementation Process

• Set deadlines• Assign roles to staff members

– Inputting rubrics– Assigning graders– Training faculty members– Pulling reports– Managing grade score release



• ExamSoft set-up– Individual Courses for Students– Created Rubric & Rubric Templates– Assigned all 10 rubrics to each student



• SharePoint form developed– Form submitted by student notifying staff of

CE date, CE rubric, and faculty graders– Submission of form initiates SharePoint

workflow – Workflow generates email sent to college

ticketing system– OAA staff member tracks CE tickets and

manages grader assignments in ExamSoft


Overview of Implementation Process

Set Deadlines

Assign roles to staff members

Set-Up ExamSoft

Develop Info Gathering

Form


Faculty Buy-In and Training

• Support from college administration• Explain advantages of ExamSoft rubrics

– Prevents misplacement of rubrics – Minimizes use of paper – Mobile device grading– Decreases likelihood of miscalculations due to

human error– Rubric creation is under 10 minutes– Immediate release of results to students


Faculty Buy-In and Training

• Small group faculty training session – 2 sessions held– 1 hour long– 12 faculty selected for pilot test– Staff provided cheat sheet for reference

• Staff developed step-by-step instruction manual

• Faculty encouraged to utilize personal devices• Staff offered technical support

23


Pilot Testing

• Pilot test period: December 2014 through February 2015

• Clinical faculty leaders were assigned as ExamSoft graders

• 4 of the 10 clinical exam’s were tested• Provided feedback and suggestions for

improvement


Pilot Testing

• Staff support during test period– Observed faculty graders for 2 clinical

sessions– Obtained feedback from faculty while on the

clinic floor – Made available via phone and e-mail for any

technical or operator-related issues


Pilot Testing

• Modification of SharePoint form– Version 1: Form fields for 2 graders– Version 2: Form fields for up to 6 graders

• Allowed for 2 graders per dimension

– Version 3: Finalized following pilot test period


Addressing Issues During Pilot Test Period

• Major Issues:– Faculty login

• Graders had difficulty remembering passwords• Last minute changes to grader assignments

– Multiple faculty grading various dimensions within single rubric

– Lack of tracking for releasing scores



• Minor Issues:– Students requesting faculty graders not in

pilot test group– Incomplete assessments– High volume of reminder e-mails sent to

faculty



• Solutions:– Created 2 general faculty accounts per clinic

group– Updated SharePoint form for better tracking of

clinical exams– Changed deadline of clinical exam

assessments to the end of each week



Western University of Health

Sciences





Sciences





Sciences



Launching ExamSoft in a Live-Patient Setting

• Clinical Dean issued a deadline to faculty for final submission of paper rubrics

• Faculty in pilot group trained remaining clinical faculty members

• Academic Affairs staff provided students with updated protocol for challenging clinical exams


Launching ExamSoft in a Live-Patient Setting

• Academic Managers set up auto login to ExamSoft grading portal on clinic iPads– Devices were secured through passcode

• Academic Managers facilitated ExamSoft set up for off-site clinics


New Clinical Exam Process

Student challenges

clinical exam/

Initiates workflow

OAA staff assigns faculty graders

2 faculty grade clinical

exam

OAA staff releases score to

student and updates

grade sheet.

1 clinic session

5 minutes

1 clinic session 2 hours

36


Conclusions

• Streamlines process of clinical exam grading– Smoother workflow– More efficient use of staff time– Minimizes likelihood of miscalculations due to

human error– Quicker turnaround for releasing scores


Conclusions

• Generate meaningful data– Demonstrate connections between

dimensions of a rubric and tagged categories– Translate data into robust reports for

university’s outcomes assessment committee and accreditation bodies

– Short-term & long-term reports for individual students and as a cohort

– Recommend improvements to the curriculum


Conclusions

• Generate meaningful data– Identify longitudinal trends such as

competencies in which students either excel in or need more focused attention

– Tag rubrics with “soft skills” such as professionalism, communication, and humanism


Plans for Improvements

• Brainstorm methods to identify critical errors in grading

• Refine rubric content • Continuous feedback from faculty, staff and

student• Offer development suggestions to ExamSoft

account managers

41

42

Implementing and Launching Rubrics in a Clinical Setting

Education

examsoft rubrics

customized examsoft

assess2015 introduction

launching rubrics

paper rubrics faculty

assess2015 learning

live clinical setting

rubrics turnaround time