Dr Peter Vitartas, Prof Damminda Alahakoon, Dr Sarah Midford, Ms Nilupulee Nathawitharana, Dr Kok-Leong Ong, Prof Gillian Sullivan-Mort Toward an automated student feedback system for text-based assignments
Dr Peter Vitartas, Prof Damminda Alahakoon, Dr Sarah Midford, Ms Nilupulee Nathawitharana, Dr Kok-Leong Ong, Prof Gillian Sullivan-Mort
Toward an automated student feedback system for text-based assignments
2
Agenda
• Background
• Automated writing evaluation tools
• Next generation rubrics project
• Results
• Discussion
3
Background
4
Automated writing evaluation
e-rater®
Revision Assistant
5
Types of error comments provided by Criterion ®
6
Automated writing evaluation
e-rater®
Revision Assistant
7
Pearson’s Intelligent Essay Assessor
8
IEA’s feature list used in Write-to-learn
9
Automated writing evaluation
e-rater®
Revision Assistant
10
Vantage Learning
• My access provides:
• writing aids,
• word processing capabilities and
• teacher analytics
• IntelliMetric analyses “400 semantic-, syntactic-, and discourse-level features to form a composite sense of meaning”
11
Automated writing evaluation
e-rater®
Revision Assistant
12
LightSide and Revision Assistant
• LightSide is used as the turnitin scoring engine
• Revision Assistant – gives ‘signal checks’ as formative feedback– Needs to be trained
13
Automated writing evaluation
e-rater®
Revision Assistant
WriteLab
14
• Provides suggestions about:– Clarity– Logic– Cohesion– Grammar
15
Next Generation Rubrics
Push towards larger subject cohorts:– 30% of La Trobe subjects = 100+ students– 41 subjects = 500+ students– BA Interdisciplinary Cores: ‘Rethinking Our Humanity’ (HUS1FAS) and ‘Ideas that Shook the World’ (HUS1TEN) = 1000+ students
Consequences:– Increased pastoral care, assessment feedback and marking– Less time to interact with individual students– Slower turn around on assessment feedback– Difficult to monitor gaps in knowledge at a cohort level– Difficult to assess critical thinking and written skills frequently and meaningfully– Lower student satisfaction
Good pedagogical practice demands quick turn-around times on student assessment feedback, but staff resources are limited and ever-increasing student diversity requires more pastoral care
16
Our solution
Online feedback tool for text based assessments
Students get: immediate feedback on their written work designed to highlight areas that could be improved before submission for marking
Staff get: a dashboard summarising student performance, knowledge gaps and patterns at a cohort level
Benefits—
• Students receive more guidance on their written assessments than staff could otherwise provide
• Student feel less self conscious about the quality of their work because online submission is anonymous
• Staff can look at submission data to see where student skills and knowledge gaps lie and target their teaching more effectively to student needs
• Staff can use data to better design their assessments
17
Student interface
18
Student interface
19
Staff interface
20
Staff interface
21
A framework for evaluating assignments
Assignment statistics
Readability
Research
Critical thinking
Discipline theory
22
Pilot study subjects
Undergraduate Subject:
HUS1TEN – Ideas that Shook the World
1st year BA Interdisciplinary Core Subject
Assessment: Essay
Postgraduate Subject:
MKT5MMA – Marketing Management
MA level Business Subject
Assessment: Report
23
Assignment statistics
Calculates average and individual:– word count– paragraph count– page count– spelling and grammar error count
Assignment statistics
24
Readability
• Calculates Flesch reading ease and Flesch-Kincaid grade level to measure the readability
• Investigates the average readability of the assignments across marks categories
Marks Category Average Flesch Reading Ease
0-49 44.07000046
50-59 41.39999989
60-69 38.35882308
70-79 33.25000004
80-89 29.19999997
90-100 19.5
HUS1TEN Readability statistics
Readability
25
Research
• Counts correctly formatted in-text references (citations) written following La Trobe Harvard style
• Counts all in-text references (citations) disregarding format• Counts number of distinct authors in in-text references (citations)
• Counts number of references in reference list
Research
26
Critical thinking
• Extracts frequencies for both stemmed/non stemmed words using Porter’s stemming algorithm (Porter, 1980)
• Captures phrases with frequencies using TerMine web tool (Frantzi, Ananiadou, & Mima, 2000)
• Identifies Critical Thinking words and phrases and calculates the occurrence of relevant words and phrases across marks categories
Critical thinking
27
Discipline theory
• Extracts frequencies for both stemmed/non stemmed words using Porter’s stemming algorithm (Porter, 1980)
• Captures phrases with frequencies using TerMine web tool (Frantzi, Ananiadou, & Mima, 2000)
Discipline theory
28
Most frequent stemmed words – HUS1TEN
Word Frequency
relationship 1788
love 1558
peopl 648
idea 638
societi 590
half 564
marriag 513
find 507
individu 503
sexual 409
Essay Question:The idea that in love you are finding “your other half” has a long history. But is the idea that we are all one half of a whole still relevant in the current climate of changing gender norms, open relationships and soaring divorce rates? Please discuss the extent to which we are simply one half of a whole with reference to one or more of the following:• Gender ideals• Polyamory• Platonic love• Individualization• The ‘pure relationship’ (Giddens)
29
Most frequent phrases
Phrase Frequency
pure relationship 197
romantic love 125
soul mate 76
many people 46
confluent love 39
platonic love 35
monogamous relationship 22
current climate 21
gay marriage 21
polyamorous relationship 20
Essay Question:The idea that in love you are finding “your other half” has a long history. But is the idea that we are all one half of a whole still relevant in the current climate of changing gender norms, open relationships and soaring divorce rates? Please discuss the extent to which we are simply one half of a whole with reference to one or more of the following:• Gender ideals• Polyamory• Platonic love• Individualization• The ‘pure relationship’ (Giddens)
30
Human marked Grade 0-19(F/<50%)
20-23(D/50-59 %)
24-27(C/60-69%)
28-31(B/70-79%)
32-40(A/80+%)
All Assignments
# of Assignments 3 12 18 20 7 60Critical Thinking Average # Critical Thinking terms
16.7 17.5 15.7 17.7 18.7 16.3
Top 3 critical thinking terms (by frequency)
AnalyseWorldFact
WorldAnalyseData
AnalyseProblemWorld
AnalyseDataWorld
DataAnalyseProblem
AnalyseDataWorld
Citations Av Citations 9.7 17.7 12 13.2 30.7 15.2Av Distinct Authors 5.3 8.25 7.2 6.8 13.6 7.9Word Statistics Av Word Count 2151 2603 2697 2758 3284 2739Av Paragraph Count 45 66 66 68 110 70Av Spelling Error 8.3 26.8 27.5 26.5 35 26.9Av Grammar Error 1.7 6.8 10.0 7.7 4.9 7.6Readability Flesch Reading Ease 35.3 39.3 39.3 38.7 33.4 37.8Flesch-Kincaid Grade Level 14.5 12.8 12.7 12.9 14.1 13.0
31
Initial findings
• Greater understanding of student performance in a large class– Distribution of mistakes– Terms and concepts not being considered– Use of references– Extent of critical thinking terms being used in assignments
• Greater understanding of tutor performance– Variability in marking weightings– Distribution of grades– Consistency in marking assignments
32
Continuing work
• Refinement of dimensions for the assessment classifier– Testing of alternative word extraction methods (LDA, semantic analysis and NLP)– Development of ontologies for discipline areas
• Improved staff user interface being built– User-friendly settings for assignment parameters– Easy to use and relevant reporting outputs
• Improved student interface being built– User-friendly assignment submission– Assignment feedback that is easy to read, understand and put into action
• Further testing of assignments
33
Thank you!
Dr Peter [email protected]
Acknowledgement: La Trobe Learning and Teaching Digital Learning Strategy Innovative Research GrantSpecial Thanks: James Heath, Aleksandra Michalewicz and Tanvir Ahmed