Writing assessment descriptors Ensuring validity and reliability in assessment Olwyn AlexanderFebruary 2015.

Writing assessment descriptors

Ensuring validity and reliability in assessment

Olwyn Alexander February 2015

Outline

Principles of test/assessment usefulness

Focus on validity: assessment construct

Focus on reliability: assessment criteria

Principles of test usefulness

Validity Reliability Authenticity Interactiveness Impact/Washback Practicality

Bachman & Palmer,1996


Validity – the test actually measures the performance it claims to measure and this is appropriate for the test purpose

Reliability – measurement is consistent between different assessors and different test takers

Authenticity – the tasks in the test are representative of tasks in the target situation

Bachman & Palmer,1996, Alexander et al., 2008


Interactiveness – the tasks in the test engage the test takers’ communicative language ability, background knowledge and strategic competence

Impact – the test will affect classroom teaching and learning (and wider institutional policy – entry levels)

Practicality – resources will be required to develop the test and train assessors to administer it

Bachman & Palmer,1996, Alexander et al., 2008

Test specification

Who individual/one class/all classes?

Why achievement/proficiency?

What construct/knowledge/skills?

How format/rating scales?


Test specification

Who students in10-week pre-sessional classesat or near CEFR B2

Why to measure achievement of targetperformance in subject-specific context

What construct/knowledge/skills

How 2,000 word literature review exploring asubject-specific research question


Test specification – construct

What is a literature review?

1. Audience/Purpose/Structure

2. Level of target discipline knowledge

3. Level of critical engagement with sources


Literature review construct

1) Involves more than linguistic knowledge and skills

2) Requires students to engage with texts in the disciplines & struggle to understand them

3) Requires evaluation and management of information

4) Raising awareness of graduate attributes

5) Target performance described at SQA level 10: exit level of UG and entry level of PG degrees

Graduate attributes

• a sense of ‘research-mindedness’ enabling a wider more analytical perspective on individual practice

• the ability to identify problems, formulate research questions & interpret complex data to seek answers

• the ability to derive meaning from complexity and make informed judgments on the basis of evidence

• an openness to learning and positive orientation to new opportunities, ideas and ways of thinking

• a tolerance for ambiguity and unfamiliarity.

SCQF level 10 knowledge

A critical understanding of the principal theories, concepts and principles in a subject/discipline.

Detailed knowledge and understanding in one or more specialisms some of which is informed by or at the forefront of a subject/discipline.

Knowledge and understanding of the ways in which the subject/discipline is developed, including a range of established techniques of enquiry or research methodologies.

SCQF level 10 application

Execute a defined project of research, development or investigation and identify and implement relevant outcomes.

Practise in a range of professional level contexts which include a degree of unpredictability and/or specialism.

SCQF level 10 cognitive skills

Critically identify, define, conceptualise, and analyse complex/professional level problems and issues.

Critically review and consolidate knowledge, skills, practices and thinking in a subject/discipline.

Make judgements where data/information is limited or comes from a range of sources.

Test specification – construct

What is a literature review?

1. Audience/Purpose

2. Structure overall & within sections/paragraphs

3. Level of target discipline knowledge

4. Level of critical engagement with sources

Task: Choose one of the areas above and brainstorm ideas for this aspect of the construct

Audience

peers, novices (non-specialists) in the community of practice

lecturers (specialists) who assess research projects, dissertations, theses

educated non-specialists, e.g. in funding councils, NGOs or government departments who award grants or use the findings

Purpose

in-depth exploration of a specific aspect of a research area to indicate the current state of the art and suggest

new research directions OR to define and limit the scope of and provide a

context and framework for a piece of research. In-depth = reference to wide range of sources Context = current state of the art Framework = theoretical basis for research design

Structure Thematic comparison and classification of sources,

based on clear criteria usually specified in advance. Relates key papers & ideas to each other, e.g. shows

which make similar/opposite/more developed claims; which ideas stimulated by which earlier contributions

Paragraphs develop from general to specific with claims supported by evidence from relevant sources.

Review (& sections/paragraphs) structured from familiar to new; structure made explicit for the reader by summarising at the end of each section what has been discussed and how this links to what follows.

Subject knowledge

Responsibility of student (or subject expert if joint marking opportunity).

Includes recognition of key figures/papers which have moved the field on and therefore should be cited.

EAP tutors do not have this knowledge Nevertheless have to be able to read a text they do

not fully understand to assess whether it achieves its purpose for the stated audience through its structure and the level of critical engagement it demonstrates.

Critical engagement

Not simply description/summarising (SCQF, 2012) identify, define, conceptualise, & analyse

complex/ professional level problems & issues. (Bruce, 2014) evaluative judgement made within any

field of human activity about some aspect, object or behaviour of that field.

(Dodd, 2014) the different usefulness of knowledge…, appreciating how knowledge in its various forms holds potentially different value for different people in different places at different times.

Critical engagement

Not simply description/summarising (Argent, 2013) showing awareness of different

perspectives/stances (Argent, 2014) arguing cogently why some work makes a more relevant, useful or powerful contribution to the field generally or in the context of specific research.

(Spencer & Alexander, 2014) relates not to finding fault (criticising) but to criteria; evaluation must be in terms of /based on specific criteria, stated explicitly.

Test specification

How 2,000 word literature review withsubject-specific research questionSubmitted to TurnitinAssessed using customised

holistic assessment descriptors


Assessment descriptors

Analytic (e.g. IELTS) or holistic (e.g. TOEFL)

Holistic – quicker/easier to give an impression mark

A response at this level largely achieves all of the following

A response at this level is marked by one or more of the following

A response at this level may reveal one or more of the following weaknesses


Task: develop a set of statements which capture the aspects outlined in the construct


1. Content & task achievement (audience/purpose)

2. Structure overall & use of sources & stance

3. Paragraph structure & use of language


Task: develop sets of statements which capture varying levels of achievement


A response at this level is marked by one or more of the following

A response at this level may reveal one or more of the following weaknesses


Example: task achievement and content

Fully addresses the specific area (concept, problem, method) to be explored, showing achievements/gaps.

Addresses the specific area in sufficient depth to cover the main points, showing some achievements/gaps

The main points are covered but with some redundant ideas. Achievements/gaps are not shown clearly.

Not all aspects of the area are covered and achievements/gaps in research are not shown.

Discussion

Any questions & comments?

Writing assessment descriptors Ensuring validity and reliability in assessment Olwyn AlexanderFebruary 2015.

Documents