cic.uts.edu.au Writing Analytics workshop Simon Buckingham Shum, Simon Knight, Andrew Gibson, Philippa Ryan Australian Learning Analytics Summer Institute 2015
cic.uts.edu.au
Writing Analytics workshop
Simon Buckingham Shum, Simon Knight, Andrew Gibson, Philippa Ryan
Australian Learning Analytics Summer Institute 2015
You have 1000 students who would like detailed feedback on their reports.
2
Now. Please.
(PS: It’s 1.30am, deadline 9am)
DraFs, not submiGed assignments
How could this be possible?
How do we design and validate wriKng analyKcs?
What issues arise? 3
TEXT ANALYTICS IS WELL OUT OF THE LAB. THE WRITING ANALYTICS PRODUCT SPACE IS GROWING…
4
http://turnitin.com/en_us/features/turnitin-scoring-engine
http://www.pearsonassessments.com/products/100000681/writing-coach.html
Session plan
5
1.30 Welcome and overview of session (SBS) 1.35 What are the biggest challenges you face in your students’ wriKng? (talk in pairs) 1.40 Quick feedback 1.45 What’s out there? Some products and free tools, and issues they raise (SK) 2.00 Introducing reflecKve wriKng + GoingOK demo (AG)
2.15 How AWA handles reflecKve wriKng + demo (SBS) 2.30 How AWA handles analyKcal wriKng + demo (SBS) 2.45 An academic’s experience of co-‐designing AWA (PR) 2.55 Comments? 3.00 Hands-‐on with GoingOK or AWA or other free web tools
3.25 Final comments 3.30 End
WHAT ARE THE BIGGEST CHALLENGES YOU FACE IN STUDENT
ACADEMIC WRITING?
6
WHAT’S OUT THERE ALREADY PROVIDING AUTOMATED
ASSESSMENT OF ACADEMIC WRITING?
7
REFLECTIVE ACADEMIC WRITING
ANALYTICS
8
DO STUDENTS WRITE REFLECTIVELY IN YOUR SUBJECT/DISCIPLINE?
uts.edu.au
“Wr i&ng br ings together past , present & future . When we wr i te we ca l l on the past and an&c ipate the future even as we inhabi t the present ; at the same &me, we engage in an act that i s both deeply inte l lectua l and overt ly phys ica l” Yagelski p.192 2012.
REFLECTIVE WRITING:
uts.edu.au
Provides opportuniKes for a deeper approach to learning Looks back on the moment and looks forward Encourages shiFs in perspecKve
Is personal SubjecKve An individual interpretaKon of experiences
Asks students to: Comment on what they DON’T know Express uncertainty Convey changes in understanding
A VERY DIFFERENT GENRE
uts.edu.au
REFLECTIVE WRITING POSES SOME PROBLEMS:
Challenging to teach:
Different interpretaKons of
what reflecKve wriKng is/
what it looks like
uts.edu.au
Challenging to assess:
How do you idenKfy a deep or a
superficial reflecKon? What are you
marking the students on? Can some
students be advantaged/disadvantaged?
A RUBRIC: LANGUAGE & DISCOURSE FEATURES & TYPE OF INFORMATION Context of the event that triggers the
reflecKon (why, when, where, who, how much, what): the more detail the beGer, as long as the event is non-‐trivial Specific informaKon about the event; specific examples of acKons , ideas, learning experiences Change between assumpKons of learning and/or prior knowledge and actual performance
uts.edu.au
Reference to the past: Kme markers and use of past tense (when I started; before my internship); shiF between habitual past tense (I used to) and the present or the recent past (since then I have …)
FROM INFORMAL RUBRIC TO FORMAL RHETORICAL PATTERN
14
15
FROM INFORMAL RUBRIC TO FORMAL RHETORICAL PATTERN
AWA: REFLECTIVE ACADEMIC WRITING
16
COMPARISON OF HUMAN AND MACHINE ANNOTATION
17
human machine
This counts as a “True Positive”
COMPARISON OF HUMAN AND MACHINE
18
human highlighting automated highlighting
19
AWA: REFLECTIVE ACADEMIC WRITING
ANALYTICAL ACADEMIC WRITING
ANALYTICS
20
TYPICAL ANALYTICAL/CRITICAL RHETORICAL MOVES BACKGROUND KNOWLEDGE
Recent studies indicate …
… the previously proposed …
… is universally accepted ...
NOVELTY
... new insights provide direct evidence ...
... we suggest a new ... approach ...
... results define a novel role ...
OPEN QUESTION/MISSING INFORMATION
… liGle is known …
… role … has been elusive
Current data is insufficient …
TREND
... emerging as a promising approach
Our understanding ... has grown exponenKally ...
... growing recogniKon of the importance ...
CONTRASTING IDEAS
… unorthodox view resolves … paradoxes …
In contrast with previous hypotheses ...
... inconsistent with past findings ...
SIGNIFICANCE
studies ... have provided important advances
Knowledge ... is crucial for ... understanding
valuable informaKon ... from studies
SURPRISE
We have recently observed ... surprisingly
We have idenKfied ... unusual
The recent discovery ... suggests intriguing roles
SIGNALLING AUTHOR INTENT
The goal of this study ...
Here, we show ...
Altogether, our results ... indicate
21
AWA: ACADEMIC WRITING ANALYTICS TOOL
22
23
AWA: ACADEMIC WRITING ANALYTICS TOOL
AWA: ACADEMIC WRITING ANALYTICS TOOL
24
Highlighted sentences are colour-coded according to their broad type
Sentences with Function Keys have more precise functions (e.g. Novelty)
AWA: ANALYTICAL ACADEMIC WRITING
25
Roll over sentences with Fkeys for a popup reminding you of their meaning
AWA: ANALYTICAL ACADEMIC WRITING (UTS CIVIL LAW)
26
ENGAGING THE ACADEMIC IN CO-‐DESIGNING AWA
27
1. Trigger curiosity through CIC briefings
2. Submit collaboraKon proposal to CIC
3. Agree Kmelines
4. Think aloud user interface walkthrough
5. Preliminary analyses
6. SystemaKc analyses
7. Refine parser
8. Design student pilot
9. Gather student feedback
28
STRUCTURED ANNOTATION FROM THE CIVIL LAW ACADEMIC TO TUNE THE PARSER FOR HER DISCIPLINE (classified comments in word)
29
STARTING FROM AWA OUTPUT. ..
30
STRUCTURED ANNOTATION FROM THE CIVIL LAW ACADEMIC TO TUNE THE PARSER FOR HER DISCIPLINE
ENGAGING THE ACADEMIC IN CO-‐DESIGNING AWA
31
1. Trigger curiosity through CIC briefings
2. Submit collaboraKon proposal to CIC
3. Agree Kmelines
4. Think aloud user interface walkthrough
5. Preliminary analyses
6. SystemaKc analyses
7. Refine parser
8. Design student pilot
9. Gather student feedback
32
UTS CIVIL LAW STUDENT VIEWS
“takes the emo&on out of having your work scru&nized” respondent 12; “it was not embarrassing in the way that it can be when a tutor or marker gives feedback” student 7 reflec1on notes
“I definitely found it useful. It also made me realise that I tend to use bold, certain language in making my point towards the end of each paragraph rather than up front at the beginning (when introducing my point).” Respondent 5
“I realise now what descrip&ve wri&ng is -‐ the soKware had quite a bit to say about my lack of jus&fica&on -‐ also true -‐ pressed for &me and difficult circumstances have caused this for me in this instance -‐ good to see it sampled.” Respondent 9
33
UTS CIVIL LAW STUDENT VIEWS
it is possible to make a clearly stated point in an academic way without using one of the markers …saying that if a paper does not use specified 'signposts' suggests that the wri&ng is not clear and ‘academic’ (see ’&ps’ on the results page), constricts wri&ng style. I think it is possible to be clear about your posi&on without explicitly saying 'my posi&on is…’. respondent 11
“found that the tool was limited in its ability to pick up on summary sentences. It was able to detect phrases such as ‘ul&mately, this essay will conclude,’ or ‘in conclusion,’ but the use of adverbs such as ‘thus,’ and ‘evidently,’ in conclusive statements failed to be recognized.”…
“Another limita&on is that certain sentences, which were recounts or mere descrip&ons were deemed important, whilst more substan&ve parts of the essay containing arguments and original voice failed to be detected.”
ONGOING WORK
From highligh&ng to ac&onable reports • How to bridge the gap between the current ability to highlight sentences, and
capability to generate a meaningful, ac&onable report “Does this highligh&ng mean it’s good?” • Without geYng into automated grading, students (and educators) are s&ll keen
to know if there are signature paZerns that are proxies for quality – a research ques&on we are now inves&ga&ng
“Algorithmic Accountability” • Cri&cal to forge trust between the stakeholders: Educators, Analysts, Students
34
WRITING ANALYTICS HANDS-‐ON
Wri&ng Tools tour: hGp://utscic.edu.au/events UTS — AWA: hGp://bit.ly/utsawa
password: alasi15 QUT — GoingOK: hGp://alasi.nlytx.io