1 Innovation in Assessment and Evaluation Prof. dr. Martin Valcke mvalcke/CV/CVMVA.htm Ghent University Maputo July 2011.

1

Innovation in Assessment and

Evaluation

Prof. dr. Martin Valckehttp://allserv.ugent.be/~mvalcke/CV/CVMVA.htm

Ghent University

Maputo July 2011

Structure

• Advance organizer

• Traditional approaches

• Trends in assessment & evaluation

• Assumptions in relation to these trends

• Research

• Practices

• Conclusions2

3

Advance organizer

Advance organizer

5

Vorm/ manier van overbrengenVorm/ manier van overbrengen3. oogcontact3. oogcontact

Bert probeert het anders:

Denk…

Transmission of your message3. Eye Contact

6

Kan ik het wel leren?Kan ik het wel leren?

Iedereen kan het leren

door ‘oefening’ + ‘feedback’

Luister maar eens naar de Getuigenis van Bert (student handelswetenschappen)(linkermuisklik in beeld)

1. Geloof in verandering!

2. Doe het ‘stap voor stap’ : 1 of 2 aandachtspunten tegelijk

3. Reflecteer zoals Bert: denk na over sterke punten en over veranderingspunten

Wat kunnen we leren van Bert? Denk even na en klik pas voor de antwoorden!

Can I learn this?

The importance of assessment & evaluation

8

Traditional approaches

9

Critical issues

• Validity of evaluation approach in view of assessment of skills and complex knowledge

• Fant et al., (1985)• Rating scales, daily logs, anecdotal records, behavior

coding, and self-assessment for evaluating student teachers.• Oral examinations, portfolio assessment, central assessment

centres, 360° assessment

– …

10

Recent developments

Individual learner

Group learner

External institution

Teachers Expert eacher

Assessment systemInstitutional level

Recent developments

• Stronger focus on “consequential validity”of measurement (Gielen, Dochy & Dierick, 2003)

• Stronger emphasis on feedback value of assessment

• What is the “learning potential” of the assessment approach

12

Recent developments

• Stiggins (1987): performance assessment

• Performance assessment is expected to be geared in a better way to assess complex behavior in medical, legal, engineering, … and educational contexts (Sluijsmans, et.al., 2004).

13

Concrete examples

• Self- and peer assessment

• Rubrics based assessment

14

Self- and peer-assessment

15

How to help the student to assess him/herself?

16

Self- and peer assessment

• Learn about your own learning process.

• Schmitz (1994): “assessment-as-learning”.

• ~ self corrective feedback

17

• See experiential learning cycle of Kolb.

• Boekaerts (1991) self evaluation as a competency.

• Development of metacognitive knowledge and skills (see Brown, Bull & Pendlebury, 1998, p.181).

• Freeman & Lewis (1998, p.56-59): developing pro-active learners

18

19 The Learning Cycle Model

Is it possible?

21

Group evaluations tend to fluctuate around the mean

Learning to evaluate

• Develop checklists

• Give criteria

• Ask to look for quality indicators.

• Analysis of examples good and less good practices: develop a quality “nose”

22

Learning to evaluate

• Freeman & Lewis (1998, p.127) :• Learner develops list of criteria.• Pairs of learners compare listed criteria.• Pairs develop a criterion checklist.• Individual application of checklist.• Use of checklist to evalute work of other learner.• Individual reworks his/her work.• Final result checkeed by teacher and result compared to learner

evaluation.• Pairs recheck their work on the base of teacher feedback.

23

Learning to evaluate

• Peer evaluation is not the same as Peer grading

• Final score is given by teacher

• Part of score could build on accuracy of self/peer evaluation and self-correction

24

25

Rubrics

26

Rubrics

• Rubrics focus on the relationship between competencies, criteria, and indicators and are organized along mastery levels (Morgan, 1999).

27

28http://web.njit.edu/~ronkowit/teaching/rubrics/samples/rubric_apa_research.pdf

Rubrics• Rubric: scoring tool for a qualitative assessment of the

quality level of an authentic or complex activity– A rubric builds on criteria, enriched with a scale to indicate a

mastery level.

– For each level, standards are indicated that reflect this level.

– A rubric dictates both teacher and student what is concretely expected.

– Rubrics are used for “high stake assessment” and “formative assessment” (Arter & McTighe, 2001; Busching, 1998; Perlman, 2003).

• Rubrics focus on the relationship between competencies, criteria, are organized along mastery levels (Morgan, 1999).

29

Rubrics: indicator-based assessment

• Assessment objective– Criteria

• Enriched with indicators in terms of observable behavior

• Limited number of indicators

30

Critical thinking rubric

31

http://academic.pgcc.edu/~wpeirce/MCCCTR/Designingrubricsassessingthinking.html

Assumptions about rubrics

• Larger consistency in scores (reliability).

• More valid assessment of complex behavior.

• Positive impact on related learning process.

32

Critical issues• Adoption of this assessment approach is marred by

teacher beliefs about nature of evaluation (see e.g., Chong, Wong, & Lang, 2004);

• Also student beliefs (Joram & Gabriele, 1998)• Validity of the criteria and indicators (Linn, 1990), • Reliability of performance evaluation, e.g., when

multiple evaluators assess and score performance (Flowers & Hancock, 2003).

33

Research about rubrics use• Review article 75 studies about rubrics usage• Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability,

validity and educational consequences. Educational Research Review, 2, 130–144.– (1) the reliable scoring of performance assessments can be enhanced by the

use of rubrics, especially if they are analytic, topic-specific, and complemented with exemplars and/or rater training;

– (2) rubrics do not facilitate valid judgment of performance assessments per se. However, valid assessment could be facilitated by using a more comprehensive framework of validity;

– (3) rubrics seem to have the potential of promoting learning and/or improve instruction. The main reason for this potential lies in the fact that rubrics make expectations and criteria explicit, which also facilitates feedback and self-assessment.

34

Conditions effective usage

• Check frame of reference for the rubric: tasks, objectives

• Train the users

• Use multiple assessors: interrater usage

• Developed by teacher and/or students!

35

Development of rubrics

• Choose quality criteria: 4 to 15 statements describing the nature of a criterion

• Determine bandwidth to judge differences in quality related to the criterion: e.g., 0-5 or qualitative descriptors

• Elaborate descriptors for each bandwidth level: concrete operational terms

• Start from available student work!

36

• Writing a fiction story

• Complex skill

• Criteria?

Rubrics: example

Rubrics

39

Rubrics• Rubric: scoringstool voor een kwalitatieve inschatting van het niveau

van een authentieke of complexe activiteit.– Een rubriek bouwt verder op criteria die verrijkt worden met een schaal

waarop beheersingsniveaus zijn aangegeven

– Per beheersingsniveau worden standaarden aangegeven die die niveau weerspiegelen.

– Een rubric geeft zowel voor de lesgever als de student aan wat concreet verwacht wordt.

– Rubrics worden voor “high stake assessment” gebruikt en voor “formatieve toetsing” (in functie van leren).(Arter & McTighe, 2001; Busching, 1998; Perlman, 2003).

• Rubrics focus on the relationship between competencies-criteria, and indicators and are organized along mastery levels (Morgan, 1999).

40

Performance assessment

• Rubrics focus on the relationship between competencies-criteria, and indicators and are organized along mastery levels (Morgan, 1999).

41

Aanpak ontwikkeling rubric

• Kies criteria voor verwacht gedrag– 4 tot 15 statements die criterium beschrijven

• Bepaal bandbreedte die verschil in bereiken criterium weergeven– Bijv. 0-5 of kwalitatieve termen

• Werk een beschrijving uit voor elke waarde in de bandbreedte– Concreet observeerbare/vaststelbare kwalificaties

42

43http://www.teach-nology.com/cgi-bin/presentation.cgi/

44http://teachers.teach-nology.com/cgi-bin/research.cgi

More information?• Overview of tools, examples, theory, bacdkground, research:

http://school.discoveryeducation.com/schrockguide/assess.html

• Critical thinking rubrics: http://academic.pgcc.edu/~wpeirce/MCCCTR/Designingrubricsassessingthinking.html

• Rubric generators: http://www.teach-nology.com/web_tools/rubrics/

• Interesting rubric sites: http://web.njit.edu/~ronkowit/teaching/rubrics/index.htm

• Rubric APA research paper: http://web.njit.edu/~ronkowit/teaching/rubrics/samples/rubric_apa_research.pdf

• K12 examples: http://school.discoveryeducation.com/schrockguide/assess.html

• General intro and overview:http://web.njit.edu/~ronkowit/teaching/rubrics/index.htm

45

46

http://www.teach-nology.com/web_tools/rubrics/

Statements about evaluation

• Learners should be trained to develop themselves such rubrics.

• Staff should collaborate in developing formal assessment and summative assessment rubrics

• Rubrics will help staff to be more concrete as to their teaching and learning focus

47

48

Innovation in Assessment and

Evaluation

Prof. dr. Martin Valckehttp://allserv.ugent.be/~mvalcke/CV/CVMVA.htm

Ghent University

Maputo July 2011