Users reactions to innovative computerised feedback – the case of DIALANG Ari Huhta Centre for Applied Language Studies University of Jyväskylä Finland.

Users’ reactions to innovative computerised feedback – the case of DIALANG

Ari HuhtaCentre for Applied Language Studies

University of JyväskyläFinland

What is DIALANG? computerized diagnostic language

assessment system in 14 languages (www.dialang.org)

feedback (& activities) that help learners to diagnose their strengths and weaknesses plan further language learning raise awareness

unusually wide range of feedback for a test; partly unique / innovative, e.g. self-assessment

What is ’feedback’? Why feedback?

In educational contexts FB can be: ”… any communication or procedure to inform a

learner of the accuracy of a response, usually to an instructional question” (Knowledge of the Correct Response)

but also ”… other information such as precision, timeliness, learning guidance, motivational messages, ... critical comparisons, and learning focus”

”… any message or display that a computer presents to the learner after a response” (Mory 2004)

improve performance & learning (directly) but also to motivate, make learning more interesting,

encourage different learning, raise awareness, …

Why study (DIALANG) feedback?

feedback is an integral part of all behaviour, performance and learning social psychology, organizational theory,

education ...

feedback from tests is not much researched (cf. teachers as assessors)

feedback from computerized (language) tests is particularly little researched

DIALANG FEEDBACK

(test result)Level

VSPTfeedback

Se lf-a ssmfe e d b a c k

Immediateitem review

Post-te stite m re vie w+ sub skill

Extendedlevel descriptions

Advice

Infoabout SA

Research topics / questions

This study: What are users’ reactions to different types of

feedback in DIALANG? How do new types of feedback compare with more traditional feedback?

Are users’ reactions associated with their background characteristics (e.g. sex, age, language use, language proficiency)? How might possible associations be explained?

--------------------------------------------------------

related, more in-depth studies users’ reactions to self-assessment and Vocabulary Size Placement Test

Research methods & data

Quantitative & qualitative (cross-sectional)

Questionnaire 553 respondents in two countries mostly non-language majors at universities

(typical users)

Interviews of users and their teachers Written reports from users & teachers

Users’ ’reactions’?

Defined in terms of 1) Reading / using particular

feedback? 2) Interest in particular feedback? 3) Perceived usefulness of particular

feedback? 4) Other reactions (e.g. ease of use)

Likert scale + open-ended questions cf. Technology Acceptance Model

Who were the respondents?

0

20

40

60

80

%

female male

Gender of respondents

Age of respondents

0

10

20

30

40

50

60

70

80

Percent

under 18 18 - 25 26 - 35 36 - 45 46 - 55

Age

Age of respondents

# of languages & tests taken

0

5

10

15

20

25

30

%

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Number of tests

Number of tests taken

0102030405060708090

Percent

1 2 3 4

Number of languages

Number of language taken

Most common test languages

0

10

20

30

40

50

%

English French Swedish German Finnish

Most common test languages

Results: Overall reaction to FB- use & interest & usefulness

VSPT feedback

Advice

Extended descriptions

SA information

Self-assessm. FB

Post-test item review

Immediate item review

Level

0,00 0,50 1,00 1,50 2,00 2,50 3,00

Mean

Reactions to Level (test result)

0

10

20

30

40

50

60

%

Clearlynegative

on average

Somewhatnegative

on average

Somewhatpositive on

average

Verypositive on

average

Level (interest & usefulness)

VSPT feedback

Advice


SA information

Self-assessm. FB



Level

0,00 0,50 1,00 1,50 2,00 2,50 3,00

Mean

Reactions to Level (test result)

the most popular part of feedback the fact that the meaning of A1 etc

was defined was appreciated associations with user background:

correlated with proficiency: higher proficiency – more positive reactions

men were less interested in this than women

0

10

20

30

40

50

%

Clearlynegative on

average

Somewhatnegative on

average

Somewhatposivite on

average

Verypositive on

average

VSPT feedback

Reactions to Vocabulary Size Placement Test feedback / result

VSPT feedback

Advice


SA information

Self-assessm. FB



Level

0,00 0,50 1,00 1,50 2,00 2,50 3,00

Mean

Reactions to Vocabular Size Placement Test & its feedback

VSPT and its result was the most controversial part of DIALANG although, overall, the reactions were positive

strong affective reactions, opinions rather divided “Interesting, I had never thought of a way like this to

test language proficiency. Fun to do!” “Absolute nonsense! This can’t be my level!”

background: in general, those who got low VSPT results regarded it very negatively, whereas those who got good results regarded it very positively in particular, women at level C1/C2 liked VSPT

feedback

explanation: VSPT scoring is affected by test-taking strategy and produces very low scores too easily

0

10

20

30

40

50

%

Clearlynegative

onaverage

Somewhatnegative

onaverage

Somewhatpositive on

average

Verypositive on

average


0

10

20

30

40

50

%

Clearlynegative

onaverage

Somewhatnegative

onaverage

Somewhatpositive on

average

Verypositive on

average


VSPT feedback

Advice


SA information

Self-assessm. FB



Level

0,00 0,50 1,00 1,50 2,00 2,50 3,00

Mean

Reactions to immediate item review

almost 60% preferred immediate review, 30% post-test review, 10% preference

Why like: want to know immediately, remember better if see the result right away, want to learn (during test)

Why don’t like: disturbs test-taking, can be depressing background:

beginners liked it more than advanced learners in particular, C-level (C2) learners considered it less useful

and interesting C-level men used it particularly infrequently

possible explanation: many beginners & intermediate learners want to learn

from tests?? advanced learners make so few mistakes that they may

feel immediate feedback is waste of time??

Reactions to post-test item review

less popular than immediate review but a fair number of users absolutely want to have it

Background: more experienced users (# of languages used

& length of language studies) considered this more interesting & useful

possible explanation (very tentative): more experienced language learners may find

information about subskills interesting??

Reactions to self-assessment feedback (comparison of SA and test result)

0

10

20

30

40

50

%

Clearlynegative

onaverage

Somewhatnegative

onaverage

Somewhatpositive on

average

Verypositive on

average

Self-assessment feedback

VSPT feedback

Advice


SA information

Self-assessm. FB



Level

0,00 0,50 1,00 1,50 2,00 2,50 3,00

Mean

Reactions to self-assessment feedback

according to users, SA is important part of (a system like) DIALANG, as ‘popular’ as e.g. item review (joint second)

background: men had more often very negative opinions

compared with women the youngest group (under 18) found it less

interesting than the older groups possible explanation:

younger learners may not (yet) be so interested in the kind of reflection on one’s skills involved in self-assessment??

Reactions to self-assm. information (why SA and test result may not match)

0

10

20

30

40

50

%

Clearlynegative on

average

Somewhatnegative on

average

Somewhatpositive on

average

Verypositive on

average

Information about SA

VSPT feedback

Advice


SA information

Self-assessm. FB



Level

0,00 0,50 1,00 1,50 2,00 2,50 3,00

Mean

Reactions to SA information

the least used / interesting / useful part of feedback Background: negative reactions were associated with

higher proficiency, especially C1/C2 women more experience in lg learning (more lgs studied) male respondents (more very negative opinions) young age (under 18)

positive reactions were associated with: lower proficiency, especially A1/A2 men less experience in lg learning (fewer lgs studied)

men were divided in their opinions of SA information possible explanations:

if SA and test agreed, no need to read this beginners have less experience in what is described in SA

information and may find it new & potentially useful; advanced learners may feel they already know all that

Reactions to extended descriptions

0

10

20

30

40

50

%

Clearlynegative

on average

Somewhatnegative

on average

Somewhatpositive on

average

Verypositive on

average

Extended level descriptions

VSPT feedback

Advice


SA information

Self-assessm. FB



Level

0,00 0,50 1,00 1,50 2,00 2,50 3,00

Mean

Reactions to extended descriptions

fairly interesting, useful and used but less so than e.g. item review

background: men made more often very negative

evaluations of this feedback than women

Reactions to advice

0

10

20

30

40

50

%

Clearlynegative

onaverage

Somewhatnegative

onaverage

Somewhatpositive on

average

Verypositive on

average

Advice

VSPT feedback

Advice


SA information

Self-assessm. FB



Level

0,00 0,50 1,00 1,50 2,00 2,50 3,00

Mean

Reactions to advice

quite popular but not among all users – surprise?

Background: high proficiency (C1/C2) & more experience more

negative views of advice men had more very negative views than women A1 / A2 learners were divided: they often chose it as

one of the 3 most useful types of FB but many also chose it as one of the least useful types of FB

possible explanation: C-level learners no longer need advice, whereas A-

level learners do, but some of them may feel the particular advice in DIALANG is not useful??

Implications / questions

implications for DIALANG and/or other such computerized systems

users clearly like to have feedback about their self-assessment younger learners may need to be

motivated first good to have two types of item

review: immediate and post-test, based on choice preferences differ; level of proficiency

Implications / questions

VSPT divides opinions: love – hate scoring needs revising

advice appears useful for beginners in particular challenges”I don’t bother to read the advice because I know I’ll

do things my way anyway””... you don’t remember them any more when you

study”

longitudinal studies how best integrate systems like DIALANG into

teaching (long-term) effectiveness of feedback?

Users reactions to innovative computerised feedback – the case of DIALANG Ari Huhta Centre for Applied Language Studies University of Jyväskylä Finland.

Documents

dialang feedback slide

immediate feedback

feedback vspt

assessors feedback

feedback activities

traditional feedback

usefulness slide

level test result slide