Eurocall 2006 – Granada An empirical study on the influence of feedback on learning performance for second language learners of French using a web-based.

Eurocall 2006 – Granada

An empirical study on the influence of feedback on learning performance

for second language learners of French using a web-based CALL environment:

Experimental design & results of the preliminary study

Barbara Louwagie – Piet Desmet

Eurocall 2006 – Granada

Objectives

• The development of (automatic) feedback is a central point in CALL. =>The general purpose of our study: feedback = effective supplementary opportunity for learning?

• Aim of the preliminary study: optimalization of the design

• => This presentation is about the search for an adequate design

Eurocall 2006 – Granada

Structure

• Definition of the object of study

• Preliminary study

• New design

Eurocall 2006 – Granada

Definition of feedback

“In the context of teaching in general, feedback is information that is given to the learner about his or her performance of a learning task, usually with the objective of improving this performance.”

(Ur 2003)

Eurocall 2006 – Granada

Feedback?

feedback

quantitative

qualitative(correctiv

e)

evaluation

summative

formative

Eurocall 2006 – Granada

• Aim: to create learning opportunities

• Consists of– identification of errors committed– explanation of errors (cognitive strategies)– suggestions to correct the errors

(Janssens ea 2003)

Corrective feedback in formative Corrective feedback in formative evaluationevaluation

Eurocall 2006 – Granada

Advantages FB in CALL

• immediate (Heift 2001)

• individualized (Kornum 1989)

• objective (Kornum 1989)

• …

Eurocall 2006 – Granada

• Verify if the feedback provided in the learning environment used at our university is effective

• Definition of ‘efficacy’ or ‘effectiveness’“The power to produce effects or intended results. This suggests a strong causal relationship between an intervention, such as the use of a particular item of technology in a learning situation and a discernible change in the learning process, the learning climate or the learning achievement.”

(Felix 2005)

Eurocall 2006 – Granada

Preliminary study: structure

• Aim• Design• Subjects• Procedure• Learning Environment (Idioma-tic)• Statistical analyses• Results• Evaluation of the preliminary study

Eurocall 2006 – Granada

Preliminary study: aim

• To verify the effect of different kinds of feedback on the score obtained for the final test

• To test the design of the study

Eurocall 2006 – Granada

Design

• Feedback configurations

• Components & themes

• Other variables

Eurocall 2006 – Granada

5 feedback configurations, schematical representation

no feedback feedback

closed activity

closed activity half-open activity

general & specific FB

general, specific & error-specific FB

intermediary FB intermediary, general & specific FB

Eurocall 2006 – Granada

• 5 themes, 3 components:– Speech acts

• Telephone conversation

– Vocabulary• Business vocabulary• Quantitative vocabulary

– Grammar• Conjunctions• Grammatical orthography

• Different feedback conditions for each of these themes

Eurocall 2006 – Granada

Type of FB as ‘within-subjects’ factor

Grammar: grammatical orthography

Grammar: Conjunctions

Speech acts: Telephone conversation

Vocabulary: The enterprise

Vocabulary: Quantitative vocabulary

Group 1: Series 1

ClosedSpec & Gen FB ESF

Half-OpenIntermediary FB

Half-OpenIntermediary FBSpec & Gen FB

Closed ClosedSpec & Gen FB

Group 2: Series 2

ClosedSpec & Gen FB

ClosedSpec & Gen FB ESF

Half-OpenIntermediary FB

Half-OpenIntermediary FBSpec & Gen FB

Closed

Group 3: Series 3

Closed ClosedSpec & Gen FB

ClosedSpec & Gen FB ESF

Half-OpenIntermediary FB

Half-OpenIntermediary FBSpec & Gen FB

Group 4: Series 4

Half-OpenIntermediary FBSpec & Gen FB

Closed ClosedSpec & Gen FB

ClosedSpec & Gen FB ESF

Half-OpenIntermediary FB

Group 5: Series 5

Half-OpenIntermediary FB

Half-OpenIntermediary FBSpec & Gen FB

Closed ClosedSpec & Gen FB

ClosedSpec & Gen FB ESF

Eurocall 2006 – Granada

+ other variables

• Personal variables (sex, level of proficiency in French, learning style (Kolb-test), educational background, …)

• Situational variables (subject of the exercises, number of tries, time spent on a certain exercise, …)

Eurocall 2006 – Granada

Subjects

• n = 107• Undergraduate students in applied economic

science & commercial engineering• Learners of French as a second language

• 5 groups (n ~ 20)• Stratification solely based on current branch of

study, otherwise at random

Eurocall 2006 – Granada

Procedure

Duration: 9 weeksDuration: 9 weeksWeek 1• (short demo of Idioma-tic)• Orientation test itemset 1, 50 MC-questionsWeek 3• Practice session 1 (1 ½ hour) itemset A1, A2, A3, A4, A5 (± 30 questions)Week 5• Practice session 2 (1 ½ hour) itemset B1, B2, B3, B4, B5 (± 30 questions)Week 7• Practice session 3 (1 ½ hour) itemset C1, C2, C3, C4, C5 (± 30 questions)Week 9• Evaluation test itemset 2 (5*(3+3) items)

Eurocall 2006 – Granada

Learning environment

• Authoring tool IDIOMA-TIC

• Tracking and logging system

Tracking & logging

Overview of activities (± 2000)

Grammar: Grammatical orthography

Grammar: Conjunctions

Speech acts: Telephone

conversation

Vocabulary: The Enterprise

Vocabulary: Quantitative vocabulary

Total

Group 1: Series 1

Closed, FB, ESF Half-open Halfopen, FB Closed Closed, FB

427

M1 M2 M3 M1 M2 M3 M1 M2 M3 M1 M2 M3 M1 M2 M3

50 30 50 30 30 25 30 25 20 20 20 27 25 20 25

Group 2: Series 2

Closed, FB Closed, FB, ESF Half-open Half-open, FB Closed

430

M1 M2 M3 M1 M2 M3 M1 M2 M3 M1 M2 M3 M1 M2 M3

50 30 50 30 29 26 30 25 20 20 20 30 25 20 25

Group 3: Series 3

Closed Closed, FB Closed, FB, ESF Half-open Half-open, FB

424

M1 M2 M3 M1 M2 M3 M1 M2 M3 M1 M2 M3 M1 M2 M3

50 30 50 30 29 26 22 26 24 20 20 30 25 18 24

Group 4: Series 4

Half-open, FB Closed Closed, FB Closed, FB, ESF Half-open

399

M1 M2 M3 M1 M2 M3 M1 M2 M3 M1 M2 M3 M1 M2 M3

50 28 30 30 29 26 22 26 24 20 20 27 25 18 24

Group 5: Series 5

Half-open Half-open, FB Closed Closed, FB Closed, FB, ESF

402

M1 M2 M3 M1 M2 M3 M1 M2 M3 M1 M2 M3 M1 M2 M3

50 28 30 30 30 25 22 26 24 20 20 27 25 20 25

TOTAL208

2

Eurocall 2006 – Granada

Statistical analyses*

• Cross-tabulations (tabulate procedure)

• Correlations (corr procedure)

• Multi-level analysis (REML-algorithm**)

* Many thanks to M. Vandewaetere

** residual restricted maximum likelihood

Eurocall 2006 – Granada

Results: Descriptive statistics

Cross-tabulation (tabulate procedure)Distribution for sex, high school curriculum and current curriculum

sex

male female

High school curriculum 59 48

Maths-sciences 56 43

Sciences 3 3

other 0 2

Current curriculum 59 48

Applied Economic Science

39 35

Commercial engineering 20 13

Eurocall 2006 – Granada

• Distribution for exercise series and current curriculum

Series

Series 1

Series 2

Series 3

Series 4

Series 5

total

Commercial Engineering

6 6 7 8 6 33

Applied Economic Science

13 15 16 14 16 74

Total 19 21 23 22 22 107

Eurocall 2006 – Granada

Correlations (corr procedure)• Indication of influence of FB,

but just below statistical significance• Some obvious significant results:

– Correlation between the number of exercises made and the final score ( = 0.13, α = 0.0036)

– …

Analytic statistics

Eurocall 2006 – Granada

Multi-level analysis I

Multiple themes per subject are measured (5). Themes are clustered for any single student.

• Feedback has a significant effect (F(4,415) = 2.99, p < 0.05 )

• Theme has a significant effect (F(4,415) = 69.84, p < 0.0001)

students (107)

… … … …

themes (5)

individual and environmental variables

Eurocall 2006 – Granada

Additional remarks

• The foregoing analysis is not complete• There are repeated measuring moments (3) for

any single student.The scores obtained during these measuring moments for a single subject will correlate more strongly than the scores of different subjects on a single measuring moment.

• The following analysis takes into account the repeated mesures design.

Eurocall 2006 – Granada

Multi-level analysis IIMultiple themes per subject are measured (5); moreover, each theme is measured three times. Themes are clustered for any single moment, and the moments are clustered for any single student.

• There is a difference between the three measuring moments (F(2,202) = 68.55, p < 0.0001)

• There is a difference between the types of feedback (F(4,424) = 1456.89, p < 0.0001)

• There is a difference between the themes(F(4,424) = 114.39, p < 0.0001)

students (107)

… … … …

moments (3)

… …

themes (5)

individual and environmental variables

Eurocall 2006 – Granada

• These significant main effects (previous slide) can, however, not be interpreted unequivocally, – because the two-way interaction terms are significant

• feedback*moment, F(8,768) = 15.52, p < 0.0001 • theme*moment, F(8,720) = 9.17, p < 0.0001

– Id. for three-way interaction • thema*feedback*moment: F(48,720) = 8.97, p < 0.0001

• Nevertheless, first indication of influence of FB• => Optimize the design to obtain significant

results

Eurocall 2006 – Granada

Evaluation of the preliminary study

• Some practical problems– Over-parametrization (1000 cells per subject) =>

many interaction effects– Only one observation per cell: 2 themes can not be compared– Stratification solely on the basis of current curriculum

• Personal variables– Maintain these variables– Level of proficiency in French: Orientation test: OK– Learning style test – Kolb test: not validated psychometrically

speaking, the results can not be validated

Eurocall 2006 – Granada

• Exercise series– Re-usable – Pay attention to differences in the level of difficulty

between themes (grammatical orthography is comparatively easier than the four other themes)

• Authoring tool:– Stable learning environment– Tracking & logging: no scoring provided for half-open

assignments

Eurocall 2006 – Granada

New design

• Add qualitative study (questionnaires)• Feedback configurations: maintain 5 types• Themes: simplification: 5 => 3 themes

– Speech acts: telephone conversation– Vocabulary: business vocabulary– Grammar: conjunctions

• Other variables: maintain and extend:– Self-evaluation as to knowledge of and attitude towards French

(motivation) – Familiarity with the on-line tool– …

Eurocall 2006 – Granada

Type of FB as a ‘between-subject’ factor

Grammar: Conjunctions

Vocabulary: The enterprise

Speech acts: Telephone conversation

Group 1: Series 1

ClosedSpec & Gen FB

ESF

Group 2: Series 2

ClosedSpec & Gen FB

Group 3: Series 3

Closed

Group 4: Series 4

Half-OpenIntermediary FBSpec & Gen FB

Group 5: Series 5

Half-OpenIntermediary FB

Eurocall 2006 – Granada

Procedure

• Orientation test itemset 1, MC questions– Exercise moment 1 itemset A1, A2, A3, A4, A5

• evaluation 1 itemset 2 (20 closed assignments, 20 half-open assignments)

– Exercise moment 2 itemset B1, B2, B3, B4, B5

• evaluation 2 itemset 3 (different from 2)– Exercise moment 3 itemset C1, C2, C3, C4, C5

• evaluation 3 itemset 4 (different from 2 & 3)

Eurocall 2006 – Granada

• Extended stratification: – current curriculum,– sex, – score for orientation test, – educational background, …

• Each time trial run• Items at random, for 2 reasons:

– Concentration effect– Time lapse

Eurocall 2006 – Granada

More information

Presentation: http://www.kuleuven-kortrijk.be/ALT/

Barbara LouwagieK.U.Leuven Campus [email protected]

Piet DesmetProfessor of French and Applied Linguistics

K.U.Leuven (Campus Kortrijk)[email protected]

www.ling.arts.kuleuven.ac.be/franling_e/pdesmet