USEFULNESS IN ASSESSMENT Prepared by Vera Novikova and Tatyana Shkuratova.

USEFULNESS IN ASSESSMENTPrepared by Vera Novikova and Tatyana Shkuratova

Usefulness

Assessments are used to:• help people take decisions about test takers;• encourage learning;• give fair access to opportunities.If we have evidence that an assessment does

these things, we can be confident that it is useful.

‘Usefulness, consisting of several qualities is an overriding consideration for quality control throughout the process of designing, developing, and using a particular [assessment]’.

(Bachman & Palmer, 2010)

Usefulness

Usefulness: 4 essential qualities(Green 2013)

Reliability+ Construct validity+ Authenticity+ Interactiveness+ Impact+ Practicality

= Usefulness

(Bachman and Palmer 2010)

Usefulness

Three principles to rationalize usefulness

1. Overall usefulness of the assessment more important than any single quality.

2. Judgement that an assessment is useful must take account of combined effect of qualities on the usefulness of the test.

3. Usefulness depends on the situation as different balance of qualities will be important to tests used for different purposes: usefulness must be determined for each assessment situation.

(Bachman and Palmer 2010)

Practicality

• Resources sufficient to support assessment? Time, money, space, equipment etc.

• Good classroom assessment should be "teacher friendly” - developed, administered, and marked within available time with available resources.

• Classroom assessments are only useful if feedback is provided and is timely and well understood by the students (Coombe, Folse and Hubley, 2007).

Reliability:

• stability: results would be the same if the assessment was given at another time;

• consistency: different parts of an assessment give similar results;

• rater reliability: results are similar when different people score the assessment.

Validity:

• extent to which assessment provides users with the information needed for sensible decisions;

• assessment must therefore actually assess what it is intended to assess - language ability - not ability to do Maths, memorise answers, guess or cheat etc.;

• in other words, in educational settings, assess what you teach and how you teach it!

Impact:

• ways in which use of an assessment affects society, educational systems, individuals (Bachman and Palmer, 2010);

• in classroom: test washback and effects of feedback on learning.

Usefulness: 4 essential qualities

(Green 2013)

OTHER QUALITIES OF USEFUL ASSESSMENTS

Authenticity:

• extent to which assessments reflect realistic language use situations;

• language learners are motivated to perform when they are faced with tasks that reflect real-world situations and contexts (Coombe, Folse and Hubley, 2007).

Interactiveness:

• extent and type of involvement of test taker’s individual characteristics in accomplishing a test task (Coombe, Folse and Hubley, 2007);

• do test takers use and process language in realistic ways when performing assessment tasks?

15

Absence of bias:

• all test takers with similar level of ability in the skills being assessed should achieve similar results;

• no advantage for one group over another (e.g. boys over girls, urban children over rural children) because of factors other than what is being assessed.

[e.g. if all locations described in the passages in a reading test are in Kaliningrad, that might advantage learners from Kaliningrad over those from Vladivostok]

Transparency

Availability of clear, accurate information to students. Information should include:• outcomes to be evaluated;• formats used;• weighting of items and sections;• time allowed to complete the test;• grading criteria.

(Coombe, Folse and Hubley, 2007)

Security:

• Security is part of practicality, reliability, validity and impact for all tests.

• Security needs resources. Results obtained through any form of cheating are unreliable, invalid and likely to have a negative impact on society.

• If a teacher invests time and energy in developing good tests that accurately reflect the course outcomes, then it is desirable to be able to recycle the test materials: this means keeping them protected.

(Coombe, Folse and Hubley, 2007)