A National Online A National Online Essay Evaluation Essay Evaluation Service: Service: Valid, Reliable, & Valid, Reliable, & Cost Effective Writing Cost Effective Writing Assessment Assessment Leslie C. Perelman Leslie C. Perelman <[email protected]> <[email protected]> Program in Writing and Humanistic Program in Writing and Humanistic Studies Studies Massachusetts Institute of Massachusetts Institute of Technology Technology
116
Embed
A National Online Essay Evaluation Service: Valid, Reliable, & Cost Effective Writing Assessment Leslie C. Perelman Leslie C. Perelman Program in Writing.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
A National Online Essay A National Online Essay Evaluation Service:Evaluation Service:Valid, Reliable, & Cost Valid, Reliable, & Cost Effective Writing Effective Writing AssessmentAssessment
Student writes one Student writes one or two essays at or two essays at home within a home within a specified period specified period using a computer.using a computer.
Essays are scored on Essays are scored on either printed copy either printed copy or on screen by or on screen by readers in real or readers in real or virtual grading virtual grading sessionssessions
Student and advisor Student and advisor access both score access both score and text of essays.and text of essays.
TestWriteTestWrite
A Better Way to A Better Way to Assess WritingAssess Writing
Developed by Experts in Developed by Experts in Writing Assessment at Five Writing Assessment at Five Diverse UniversitiesDiverse Universities
With TestWriteWith TestWrite Students Write Students Write
This WayThis Way Not This WayNot This Way
Intelligent ReadingsIntelligent Readings
Give Students Time Give Students Time to Think & Writeto Think & Write
Assess the Writing Assess the Writing ProcessProcess
More Time (and More Time (and savings) May Allow for savings) May Allow for Two EssaysTwo Essays “ “if you want to give real weight to if you want to give real weight to
[student] essays, you must secure two [student] essays, you must secure two test essays from each student with test essays from each student with some separation in time as well as in some separation in time as well as in topic, and the least possible separation topic, and the least possible separation in time as well as in topic, and the in time as well as in topic, and the least possible separation in time is to least possible separation in time is to have one test essay written in the have one test essay written in the morning and the other in afternoon of morning and the other in afternoon of the same day.”the same day.”
-- Paul Diederich of ETS-- Paul Diederich of ETS
The founder of modern The founder of modern writing writing assessmentassessment
TestWriteTestWrite FeaturesFeatures Automates all components Automates all components
of best practices in writing of best practices in writing assessmentassessment Focusing on writing processFocusing on writing process Allowing students modern Allowing students modern
sessions with real time sessions with real time quality control measuresquality control measures
Digital Archives of Digital Archives of EssaysEssays For program assessmentFor program assessment
For portfoliosFor portfolios
For institutional researchFor institutional research
Two Perpetual Two Perpetual QuestionsQuestions What about cheating?What about cheating?
Will all students have Will all students have access?access?
Reponses & ResultsReponses & Results
Feedback from Feedback from StudentsStudents ““I am from a small town in I am from a small town in
Missouri. My writing essay was Missouri. My writing essay was easy for me because I enjoy easy for me because I enjoy thinking about these things, such thinking about these things, such as philosophy and the future. I as philosophy and the future. I went out to my cabin and I just went out to my cabin and I just sat around the campfire the sat around the campfire the whole time, just enjoying the whole time, just enjoying the silence and writing whatever silence and writing whatever came into my head. The next day came into my head. The next day I typed it up.”I typed it up.” (DePaul) (DePaul)
Feedback from StudentsFeedback from Students
““Taking the online writing Taking the online writing tests, I found, gave me the tests, I found, gave me the opportunity to leave the opportunity to leave the information when I needed information when I needed time to think and come back to time to think and come back to it when I was ready. The it when I was ready. The freedom also allowed me to freedom also allowed me to revise my work and not feel revise my work and not feel though my work was rushed or though my work was rushed or inhibited in any way.”inhibited in any way.” (DePaul) (DePaul)
Benefits for Writing Benefits for Writing FacultyFaculty Allows essays and questions to Allows essays and questions to
reflect the university’s and reflect the university’s and program’s specific educational program’s specific educational needs, priorities, and goalsneeds, priorities, and goals
Maintains control of all aspects of Maintains control of all aspects of evaluation process within the evaluation process within the university and writing programuniversity and writing program
Online grading sessions maintain Online grading sessions maintain “staff consensus building ” benefit “staff consensus building ” benefit while potentially reducing costswhile potentially reducing costs
Reflects and measures the writing Reflects and measures the writing process used and valued in collegeprocess used and valued in college
Encourages sharing of “best Encourages sharing of “best practices” among schools using the practices” among schools using the systemsystem
ReliabilityReliability
Reliability of Reliability of AssessmentAssessment ““Faculty involved in English Faculty involved in English
placement strongly feel that placement strongly feel that the Online represents much the Online represents much better the kind of writing better the kind of writing students will do in our courses, students will do in our courses, and, so far, it is working and, so far, it is working equally well for a wide range equally well for a wide range of students from honors to of students from honors to very low developmental.”very low developmental.” (U. (U. of Cincinnati)of Cincinnati)
Reliability of Assessment at Reliability of Assessment at MIT:MIT:Fall 2002 Survey of Instructors Fall 2002 Survey of Instructors Teaching Introductory Writing Teaching Introductory Writing ClassesClasses 1st Semester Students
in Introductory Writing Classes N = 147
1%12%
34%53%
1 2 3 4
1=Should not 1=Should not be in classbe in class
4=Definitely should4=Definitely shouldBe in classBe in class
Reliability of Assessment at Reliability of Assessment at MIT:MIT:Fall 2002 Survey of Faculty Fall 2002 Survey of Faculty TeachingTeachingWriting-Intensive ClassesWriting-Intensive Classes1st Semester Students in Writing-Intensive
Humanities Classes N=125
0%10%20%
30%40%50%60%70%
80%90%
100%
1 2 3 4Faculty Responses 1=Should not have been in class; 4 = appropriate writing
skills for class
Pass
Subject Req
Reliability of Assessment: Reliability of Assessment: Reduction in Complaints and Reduction in Complaints and Appeals at MIT 1992-2003Appeals at MIT 1992-2003
Appeals & Complaints by Students about Scores on MIT Placement Test
0
10
20
30
40
50
60
70
80
90
19
92
19
93
19
95
19
96
19
97
19
98
19
99
20
00
20
01
20
02
20
03
Year
# o
f S
tud
en
ts
SavingsSavings
Greater Efficiency in Greater Efficiency in Scheduling and Staffing Scheduling and Staffing ClassesClasses
Saving Money: Reliability & Saving Money: Reliability & Timeliness of Assessment Timeliness of Assessment Reduce Number of Sections Reduce Number of Sections NeededNeeded
Accuracy of Online Evaluation = Accuracy of Online Evaluation = Fewer SectionsFewer Sections LSU (4,000 1LSU (4,000 1stst Year Students) is Year Students) is
scheduling 4 fewer sections per scheduling 4 fewer sections per year because of fewer students year because of fewer students being misplacedbeing misplaced
Extrapolate to UC’s nine Extrapolate to UC’s nine undergraduate campusesundergraduate campuses
Save Money: Typed texts (even Save Money: Typed texts (even if they are longer) cost less to if they are longer) cost less to gradegrade At MIT At MIT (@$25 / hr.)(@$25 / hr.)
the average the average handwritten handwritten text costs text costs $7.15 to be $7.15 to be scored by two scored by two readersreaders
the average the average typed text typed text costs $5.40 to costs $5.40 to be scored by be scored by two readerstwo readers $0.00
$1.00
$2.00
$3.00
$4.00
$5.00
$6.00
$7.00
$8.00
Handwritten Typed
$7.15
$5.40
Save Money: Save Money: Automatically Automatically AccommodateAccommodateStudents with Students with DisabilitiesDisabilities
Save Money: Save Money: Eliminate Data EntryEliminate Data Entry
Save Money: Eliminate Save Money: Eliminate PaperworkPaperwork
Save Money: Avoid Save Money: Avoid Hotel Expenses for Hotel Expenses for ReadersReaders
Save Money: Eliminate Save Money: Eliminate Travel ExpensesTravel Expenses
Save Money: Reduce Save Money: Reduce Expense of Feeding Expense of Feeding ReadersReaders
OptionsOptions
Three ChoicesThree Choices
UC joins current consortiumUC joins current consortium
UC collaborates in further UC collaborates in further development and development and implementation through a implementation through a non-profit corporationnon-profit corporation
We all expand by partnering We all expand by partnering with for-profit companywith for-profit company
Staying SmallStaying Small
BenefitsBenefits Keep what we are doingKeep what we are doing Maintain controlMaintain control
CostsCosts Problem with reliabilityProblem with reliability Expansion difficultExpansion difficult Future uncertainFuture uncertain
Non-ProfitNon-Profit
BenefitsBenefits Maintain controlMaintain control Potential for growthPotential for growth Low economic costLow economic cost
CostsCosts Too small for economies of scaleToo small for economies of scale Build organization that actively Build organization that actively
functions for only a few months functions for only a few months a yeara year
Ensuring reliability still a Ensuring reliability still a problemproblem
PartneringPartnering
BenefitsBenefits Professional reliabilityProfessional reliability Economies of scaleEconomies of scale Limitless expansionLimitless expansion
CostsCosts More expensiveMore expensive Some loss of controlSome loss of control Possible culture conflictsPossible culture conflicts
Active relationship since 1997Active relationship since 1997 Multiple administrative initiatives including:Multiple administrative initiatives including:
MITMIT Primary partner for OpenCourseWare digital publishing Primary partner for OpenCourseWare digital publishing
initiativeinitiative Program managementProgram management Technology implementationTechnology implementation Business process outsourcingBusiness process outsourcing Application management servicesApplication management services
Open University, UKOpen University, UK Enterprise eLearning FusionEnterprise eLearning Fusion Enterprise Architecture Planning and ImplementationEnterprise Architecture Planning and Implementation Customer Relationship ManagementCustomer Relationship Management
Sapient | Some Support Sapient | Some Support ClientsClients
JanusReduced costs and eliminated bottlenecks through a 24x7 application that processes millions of dollars of online transactions a year
NissanReduced turnaround time for issue resolution to 30 minutes and drove record volumes of unique visits
Royal Mail Achieved cost and process efficiencies in the development and deployment of new applications and work packages
Opodo24x7 support and uptime for Europe's leading travel portal with access to 400 airlines, 30,000 hotel properties, and a fleet of over 750,000 hire cars worldwide, as well as travel insurance
MITProvide 24x7 support for the workflow application that feeds the MIT OCW web site.
Lloyd’s of London24x7 uptime, significantly reduced costs, and transitioned to new releases without impacting the business
Director of Writing Across the Director of Writing Across the Curriculum, MITCurriculum, MIT
Chris FelknorChris Felknor Former System Architect, Former System Architect,
Staples.comStaples.com Glenn P. StrehleGlenn P. Strehle
Principal, Campus Business AdvisorsPrincipal, Campus Business Advisors Former CFO, MITFormer CFO, MIT
Frank AndryauskasFrank Andryauskas Former CIO, Staples Inc.Former CIO, Staples Inc.
FinallyFinally
A more valid and reliable A more valid and reliable assessmentassessment
Greater faculty and student Greater faculty and student satisfactionsatisfaction
Significant reduction in costsSignificant reduction in costs Proven and reliable platformProven and reliable platform
UC System is Taking UC System is Taking Some Hard HitsSome Hard Hits Most of these cuts are Most of these cuts are
threatening one of the threatening one of the world’s greatest educational world’s greatest educational experiencesexperiences
TestWrite offers the rare TestWrite offers the rare opportunity for Win – Winopportunity for Win – Win Significant reduction in costsSignificant reduction in costs Better educational practiceBetter educational practice