Second Language Writers and The Machine Scoring of Essays
Deborah CrusanWright State University
Agenda
• Define • Discuss• Consider
But first, we must remember
• "For there is nothing either good or bad, but thinking makes it so."
~ William Shakespeare (1564-1616), Hamlet, II.ii
So . . . What is machine scoring
• Computer technology that evaluates and scores written prose (Shermis & Burstein, 2003)
• web-based • scores and responds in seconds.
Some Machine Scoring Software
Company Scoring Engine Technique Instructional Program
Scoring
ETS e-rater ® Natural Language Processing
Criterion Single holistic score
Pearson Intelligent Essay Assessor (IEA)
Latent Semantic Analysis
WriteToLearn Holistic and component scoring
Vantage Learning
Table adapted from Dikli (2006) Warschauer & Ware (2006)
IntelliMetric ™ Artificial Intelligence
MY Access! (and others)
Holistic and component scoring
How does it work?
Artificial Intelligence (AI)
Latent Semantic Analysis (LSA)
Natural Language Processing (NLP)
Recognized Machine Scoring Platforms
https://www.ets.org/criterion
Recognized Machine Scoring Platforms
Recognized Machine Scoring Platforms
Positive features
• Objective
• Consistent
• Broad in scope• High stakes testing environment
The trouble with humans
DESKILLING OF TEACHERS
AUDIENCE
INADEQUATE DESCRIPTION OF ELL ERRORS
BORROWING
SOCIAL ASPECTS OF WRITING
ONE DRAFT WONDERS
For a complete reference list, contact [email protected]