University of Michigan English Language Institute Testing and Certification Division www.lsa.umich.edu/eli Behavioural scales of language proficiency: insights from the use of the Common European Framework of Reference Spiros Papageorgiou 4 th International GALA conference, Thessaloniki, 14-16 December 2007
16
Embed
University of Michigan English Language Institute Testing and Certification Division
14 th International GALA conference, Thessaloniki, 14-16 December 2007. Behavioural scales of language proficiency: insights from the use of the Common European Framework of Reference Spiros Papageorgiou. University of Michigan English Language Institute - PowerPoint PPT Presentation
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
University of MichiganEnglish Language Institute
Testing and Certification Divisionwww.lsa.umich.edu/eli
Behavioural scales of language proficiency: insights from the use of the Common European
Framework of Reference
Spiros Papageorgiou
14th International GALA conference, Thessaloniki, 14-16 December 2007
University of MichiganEnglish Language Institute
Testing and Certification Divisionwww.lsa.umich.edu/eli
Outline • Background
• Aims
• Data collection
• Data analysis
• Results
• Implications
University of MichiganEnglish Language Institute
Testing and Certification Divisionwww.lsa.umich.edu/eli
Background • Advent of the CEFR: increased interest in
behavioural scales of language proficiency
• Using the CEFR scales: Problems
Designing test specifications (Alderson et al., 2006) Measuring progression in grammar (Keddle, 2004) Describing the construct of vocabulary (Huhta &
Figueras, 2004) Designing proficiency scales (Generalitat de Catalunya,
2006)
University of MichiganEnglish Language Institute
Testing and Certification Divisionwww.lsa.umich.edu/eli
Background (2)• Using the CEFR scales: Criticism
Equivalence of tests constructed for different purposes (Fulcher, 2004b;Weir, 2005)
Danger of viewing a test as non valid because of not claiming relevance to the CEFR (Fulcher, 2004a)
Progression in language proficiency not based on SLA research but on judgements by teachers (cf. North 2000; North & Schneider 1998)
University of MichiganEnglish Language Institute
Testing and Certification Divisionwww.lsa.umich.edu/eli
Aims of the study• Investigation of three research questions:
Can users of the CEFR rank-order the scaled descriptors in the way the appear in the 2001 volume?
If differences in scaling exist between the users of the CEFR and the 2001 volume, why does this happen?
Can training contribute to more successful scaling?
University of MichiganEnglish Language Institute
Testing and Certification Divisionwww.lsa.umich.edu/eli
Data collection• 12 users of the scales acting as judges in relating
two language examinations to the CEFR
• Data collected during Familiarisation sessions described in the Manual for relating examinations to the CEFR
• Part of a doctoral thesis at Lancaster University (Papageorgiou, 2007) and a research project at Trinity College London
• Task: sort descriptors into the six levels
University of MichiganEnglish Language Institute
Testing and Certification Divisionwww.lsa.umich.edu/eli
Data collection (2)Descriptors N Number of judges per administration Ratings
Sept 2005 1st
Sept 2005 2nd
November 2005
February 2006
July 2006
Speaking 30 12 12 10 11 - 1350
Writing 25 12 12 10 11 - 1125
Listening 19 12 12 10 11 - 855
Reading 20 12 12 10 11 11 1120
Global 30 12 12 10 11 - 1350
Total 124 5800
University of MichiganEnglish Language Institute
Testing and Certification Divisionwww.lsa.umich.edu/eli
Data analysis • Analysis: FACETS Rasch computer program• 3 facets: descriptors-raters-occasions• Rank-ordering of elements of facets on a common
scale• Fit statistics (Bond and Fox, 2001; McNamara,
1996) Overfit: too predictable pattern Misfit: more than expected variance• Acceptable range of fit statistics Descriptors: .4-1.2 (Linacre & Wright, 1994) Raters: .5-1.5 (Weigle, 1998)
University of MichiganEnglish Language Institute
Testing and Certification Divisionwww.lsa.umich.edu/eli