Nadaclair : Cut L2Marking Time in Half (and maybe they’ll learn at the same time!)

Nadaclair.com: Cut L2Marking Time in Half (and maybe they’ll learn at the same time!)

Nadaclair.com: Cut L2Marking Time in Half (and maybe they’ll learn at the same time!)

Terry Nadasdi, Stéfan Sinclair, Denis CousineauTerry Nadasdi, Stéfan Sinclair, Denis Cousineau

Preliminary remarksPreliminary remarks

My focus is error identification This is an activity on which teachers spend

countless hours and dollars Is it helpful? Probably This study is indirectly interested in this Who is quantitatively better and by how

much?

My focus is error identification This is an activity on which teachers spend

countless hours and dollars Is it helpful? Probably This study is indirectly interested in this Who is quantitatively better and by how

much?

Overview of talkOverview of talk

a) discussion of previous “research”

b) discussion of methodology (who are the humans, what is the computer?)

c) presentation of quantitative results

d) discussion of results

e) conclusions about which is better (and what that means)

a) discussion of previous “research”

b) discussion of methodology (who are the humans, what is the computer?)

c) presentation of quantitative results

d) discussion of results

e) conclusions about which is better (and what that means)

AdvantagesAdvantages

a) some studies found checkers helpful for SLA since they find a large number of errors (e.g. Burston, 2001)

b) one study suggests that they are good because they make learners reflect on language (e.g. Durel, 2003)

a) some studies found checkers helpful for SLA since they find a large number of errors (e.g. Burston, 2001)

b) one study suggests that they are good because they make learners reflect on language (e.g. Durel, 2003)

DisadvantagesDisadvantages

- Too many false positives

- No L1 interface

- Not designed for L2 learners

- Not pedagogical in orientation

- Too many false positives

- No L1 interface

- Not designed for L2 learners

- Not pedagogical in orientation

Research problemsResearch problems

a) Small databases

b) Unrealistic representation of human correction.

c) Criticisms have usually been based on invented sentences, not on empirical data.

a) Small databases

b) Unrealistic representation of human correction.

c) Criticisms have usually been based on invented sentences, not on empirical data.

Our studyOur study

Who is better?

Humans or …. Computers?

Who is better?

Humans or …. Computers?

Our DataOur Data

- 30 random texts submitted to BonPatron.com

- 1000 - 1500 characters (roughly 250 words),

- Anglophones, residing in Canada (outside of Quebec)

- 30 random texts submitted to BonPatron.com

- 1000 - 1500 characters (roughly 250 words),

- Anglophones, residing in Canada (outside of Quebec)

The humansThe humans

- Two University Professors

- Over 20 years experience marking FSL texts

- PhD in French linguistics

- PhD in French literature

- Two University Professors

- Over 20 years experience marking FSL texts

- PhD in French linguistics

- PhD in French literature

Error LegendError Legendsp: Spelling (any word caught by spell check); only one error per word codedagdn: Determiner/Noun agreement error, e.g.: “tes ami”

agan: Adjective/Noun agreement error, e.g.: “premier chose”; past participle agreement, e.g.: “la chose que j’ai remarqué

agsv: Subject/Verb agreement error, e.g.:

mw: Missing word, e.g.: “Je veux _ il mange”.wf: Wrong word form (e.g. infinitive instead of past participle, etc.); generally any time a French word form (found in the dictionary), was used in place of another word form, e.g. il a manger instead of il a mange

sp: Spelling (any word caught by spell check); only one error per word codedagdn: Determiner/Noun agreement error, e.g.: “tes ami”

agan: Adjective/Noun agreement error, e.g.: “premier chose”; past participle agreement, e.g.: “la chose que j’ai remarqué

agsv: Subject/Verb agreement error, e.g.:

mw: Missing word, e.g.: “Je veux _ il mange”.wf: Wrong word form (e.g. infinitive instead of past participle, etc.); generally any time a French word form (found in the dictionary), was used in place of another word form, e.g. il a manger instead of il a mange

Error Legend (Con’t)Error Legend (Con’t)

la: Lexical (anglicism), e.g. “avoir un bon temps”wc: Word choice (e.g. avoir instead of être; wrong preposition)p: Punctuation (capitalization, missing space, missing comma, etc.)wo: Word order (syntax problems), e.g.: je veux la

el: Elision, e.g.: “je aime”

la: Lexical (anglicism), e.g. “avoir un bon temps”wc: Word choice (e.g. avoir instead of être; wrong preposition)p: Punctuation (capitalization, missing space, missing comma, etc.)wo: Word order (syntax problems), e.g.: je veux la

el: Elision, e.g.: “je aime”

Example of human correctionExample of human correctionLe*agdn premier*agan chose que nous avons remarqué*agan a été*wf le temps*la que tu te chouches*sp. Plutôt de*wc 11h 30, tu devrais te couches*wf à 10h. Ce changement est le plus importante*agan de tout. Si tu te chouches*sp à 10h, tu peux te léver*sp à une heure matinale. Le*agdn deuxième chose *mw nous avons remaqué*sp*agan a été*wf que tu prendsd*sp l’*p autobus pour arriver à l'école très vite. Nous pensons que tu devrais marcher*la à l'école. Puis tu auras marcher*wf pour*la un*pkilometre*sp, tu seras en bonne forme. Probablement que tu n'auras pas le temps *mw récontrer*sp tes amis avant le début des cours *pmais ce n'est pas très improtante*sp*agan …

Le*agdn premier*agan chose que nous avons remarqué*agan a été*wf le temps*la que tu te chouches*sp. Plutôt de*wc 11h 30, tu devrais te couches*wf à 10h. Ce changement est le plus importante*agan de tout. Si tu te chouches*sp à 10h, tu peux te léver*sp à une heure matinale. Le*agdn deuxième chose *mw nous avons remaqué*sp*agan a été*wf que tu prendsd*sp l’*p autobus pour arriver à l'école très vite. Nous pensons que tu devrais marcher*la à l'école. Puis tu auras marcher*wf pour*la un*pkilometre*sp, tu seras en bonne forme. Probablement que tu n'auras pas le temps *mw récontrer*sp tes amis avant le début des cours *pmais ce n'est pas très improtante*sp*agan …

The computer: BonPatron!The computer: BonPatron!

Designed to address many of the criticisms previously raised in software reviews

Designed to address many of the criticisms previously raised in software reviews

Basic StatsBasic Stats

Daily page views: 120,000

Daily users: 30, 000

Countries: 183

Who: school kids, university students, private industry, sword makers, roller-skaters, judo enthusiasts, self-mutilators, horse lovers, etc.

Daily page views: 120,000

Daily users: 30, 000

Countries: 183

Who: school kids, university students, private industry, sword makers, roller-skaters, judo enthusiasts, self-mutilators, horse lovers, etc.

Let’s have a look at this BonPatron of yours:Let’s have a look at this BonPatron of yours:

www.pro.bonpatron.comwww.pro.bonpatron.com

1) Il a reste 1) Il a reste

Process-writing approachProcess-writing approach

Table 1 (human results)Table 1 (human results)

Error type Super Human Computer1 Computer2

grammar 845

Punctuation 172

Spelling 206

Elision 15

Total 1238

Table 2 (computer1)Table 2 (computer1)

Error type Super Human Computer1 Computer2

grammar 845 564 (68%)

Punctuation 172 240 (140%)

Spelling 206 203 (99%)

Elision 15 14 (93%)

Total 1238 1021 (82%)

Table 3 (computer2)Table 3 (computer2)

Error type Super Human Computer1 Computer2

grammar 845 564 (68%) 729 (86%)

Punctuation 172 240 (140%) 251 (146%)

Spelling 206 203 (99%) 230 (112%)

Elision 15 14 (93%) 17 (113%)

Total 1238 1021 (82%) 1227 (99%)

Table4 (grammar subcategories)Table4 (grammar subcategories)

Error code Super human Computer1 Computer2

Adj./Nous agr. 102 62 (61%) 89 (87%)

Det/Noun agr. 116 96 (83%) 113 (97%)

Subj./verb agr. 91 77 (85%) 89 (98%)

Word choice 179 103 (58%) 138 (77%)

Word order 16 7 (44%) 11 (69%)

Missing word 47 25 (53%) 33 (70%)

Lexical anglicism 60 43 (72%) 57 (95%)

Word form 234 151 (65%) 199 (85%)

Total 845 564 (68%) 729 (86%)

Table 5: results excluding space before punctuation mark:Table 5: results excluding space before punctuation mark:

Error type Super Human Computer1 Computer 2

grammar 845 564 (68%) 729 (86%)

Punctuation 91 108 (119%) 119 (131%)

Spelling 206 203 (99%) 230 (112%)

Elision 15 14 (93%) 17 (113%)

totals 1157 889 (77%) 1095 (95%)

DiscussionDiscussion

a) what does human correction really look like?

b) do students learn from human correction?

a) what does human correction really look like?

b) do students learn from human correction?

2) “Enfin, exhaustif dans son repérage … [la médiation humaine] … est pratiquement infaillible même s’il peut parfois être influencé par des facteurs psychologiques (être tolérant ou agacé, se lasser ou même s’ennuyer) ou physiologiques (se fatiguer)” (Cordier-Gauther & Dion, 2003).

2) “Enfin, exhaustif dans son repérage … [la médiation humaine] … est pratiquement infaillible même s’il peut parfois être influencé par des facteurs psychologiques (être tolérant ou agacé, se lasser ou même s’ennuyer) ou physiologiques (se fatiguer)” (Cordier-Gauther & Dion, 2003).

Perfect in it’s ability to identify errors, human correction is almost infallible, even it can occasionally be influenced by psychological factors (to be accepting, or bothered, lose interest or even become bored or physiological (become tired).

Perfect in it’s ability to identify errors, human correction is almost infallible, even it can occasionally be influenced by psychological factors (to be accepting, or bothered, lose interest or even become bored or physiological (become tired).

In the context of the student – teacher marking experience, certain humans are at best slightly better at error identification, but computers may well be better at error remediation.

In the context of the student – teacher marking experience, certain humans are at best slightly better at error identification, but computers may well be better at error remediation.

Computer advantagesComputer advantages

a) Tireless (and always available)

b) Impartial

c) Real-time feedback

d) Explicit corrective feedback (with an example)

a) Tireless (and always available)

b) Impartial

c) Real-time feedback

d) Explicit corrective feedback (with an example)

Vast majority of research shows the positive effect of corrective feedback on L2 written production

Vast majority of research shows the positive effect of corrective feedback on L2 written production

- Ferris (1995, 1997, 1999, 2003, 2006)

- Chandler (2000, 2003)

- Bitchener (2006, 2007)

- Bitchener, Young & Cameron (2003)

- Ferris (1995, 1997, 1999, 2003, 2006)

- Chandler (2000, 2003)

- Bitchener (2006, 2007)

- Bitchener, Young & Cameron (2003)

I am in grade 7 and My French has improved because of BonPatron, I used to get c-- or c--- even F now i get c+ and sometimes c, and I hope to get a B some day! BonPatron you've changed my Life ! Geneviève

Mes notes de Français (je prends un cours d'histoire en immersion au niveau 10e année) étaient en train de baisser, mais avec l'aide de votre site mon prof a noté beaucoup d'amélioration. Merci! Hannah Jackson

A colleague of mine, gave the BonPatron site as a tool for her advanced translation class. She says she has never received assignments for that class with so few agreement errors! Cheers and many thanks to the both of you... Dana Paramskas, French Professor, University of Guelph

I am in grade 7 and My French has improved because of BonPatron, I used to get c-- or c--- even F now i get c+ and sometimes c, and I hope to get a B some day! BonPatron you've changed my Life ! Geneviève

Mes notes de Français (je prends un cours d'histoire en immersion au niveau 10e année) étaient en train de baisser, mais avec l'aide de votre site mon prof a noté beaucoup d'amélioration. Merci! Hannah Jackson

A colleague of mine, gave the BonPatron site as a tool for her advanced translation class. She says she has never received assignments for that class with so few agreement errors! Cheers and many thanks to the both of you... Dana Paramskas, French Professor, University of Guelph

Thank you so much BonPatron you have helped my mark go up at least 20 percent now I have a happy B+ My teacher was really smart to tell you this site! Sarah.

I use your website all the time. It is so awesome. I actually learn a lot of great things and I like seeing that my writing is improving because I get less and less little yellow and red marks. Keep it up, it's really great! Sarah

Thank you so much BonPatron you have helped my mark go up at least 20 percent now I have a happy B+ My teacher was really smart to tell you this site! Sarah.

I use your website all the time. It is so awesome. I actually learn a lot of great things and I like seeing that my writing is improving because I get less and less little yellow and red marks. Keep it up, it's really great! Sarah

Concluding remarksConcluding remarks

- Previous accounts of human marking have not been realistic

- Our study has shown that even when compared with super human correction, BonPatron.com fares very well (conservatively, it catches more than 70% of errors)

- It is likely that the feedback BonPatron provides is more beneficial for learning the rules of written French

- Previous accounts of human marking have not been realistic

- Our study has shown that even when compared with super human correction, BonPatron.com fares very well (conservatively, it catches more than 70% of errors)

- It is likely that the feedback BonPatron provides is more beneficial for learning the rules of written French

Thank you!Thank you!