Aligment of assessment with learning outcomes...Use assessment tasks that also contain that verb, thus enabling you to judge with the help of rubrics if and how well they have achieved

Aligment of assessment with

learning outcomes

Jadwiga Mirecka

Department of Medical Education

Jagiellonian Unversity Medical College

Learning Outcomes

• For the level of education

• For disciplines (eg.UK

Benchmark Statements,

Tuning projects)

• For the institution (graduate

outcomes)

• For the program

• For the module

• For the teaching unitAfter M.F.Frankowicz

A good balance of learning outcomes

Typical learning outcomes in higher education

• Knowledge, comprehension, application, analysis,

synthesis, evaluation, etc. (Bloom)

• Problem solving

• Working alone and in teams; personal and interpersonal

skills

• Communications; “information literacy”.

Which of these are included in your courses? Are all of

them assessed?

Seminario Internacional SCT, Pucón, Chile 31 Aug 2011

Why should we assess students’ progress?

Passed?

Failed?

„Assessment drives learning”

„The tail wags the dog”

66

TeacherPerspectives: Learning Outcomes Teaching Activities Assessment

StudentPerspectives: Assessment Learning Activities Outcomes

The relationship between Learning Outcomes and

assessment as seen by teacher and students

Biggs (2003)

Drivers of Outcomes Assessment

• A revolution in education: the student centered learning (teachers go from being the “sage on

the stage” to the “guide on the side”)

• Higher Education’s response to theTotal Quality Management movement ( Review, assess, improve … review,assess, improve …

• Historical role of accreditation in ensuring quality

“Constructive Alignment”

Constructive

• The students construct understanding for themselves through learning activities. “Teaching is simply a catalyst for learning” (Biggs).

Alignment

• Alignment refers to what the teacher does in helping to support the learning activities to achieve the learning outcomes.

Aligning the assessment with the learning outcomesmeans that students know how their achievements will be measured.

Seminario Internacional SCT, Pucón,

Chile 31 Aug 2011

(Biggs, 2005)

Stages in the design of constructively aligned

teaching and assessment

1. Describe the intended learning outcome in the form of a

verb (activity), its object (the content) and specify a

standard the students are to attain.

2. Create a teaching-learning environment that address

that verb and are likely to bring about the intended

outcome.

3. Use assessment tasks that also contain that verb, thus

enabling you to judge with the help of rubrics if and how

well they have achieved the intended learning outcomes

4. Transform these judgments into standard grading

criteria.

(Biggs 2007)

Choosing the form of assessment

• What will the student’s work /performance tell me about

their level of competence on the targeted learning

objectives?

• How will my assessment of their work help guide

students’ practice and improve the quality of their work?

• How will the assessment outcomes for the class guide

my teaching practice?

Example of Matching the Assessment

to the Learning Outcome

Learning outcomes

1. Demonstrate good

presentation skills.

2. Formulate food product

3. Identify an area for

research

4. Identify signs of

financial crisis

Assessment?a) Multiple choice

questions

b) Prepare a 1000-word research proposal

c) Lab-based project

d) Make a presentation to peers

Seminario Internacional SCT, Pucón,

Chile 31 Aug 2011

Constructively aligned program

Course LOs

Course LOs

Course LOs

Program LOs

Courses are

coordinated to

assure steady

development of

the learning

outcomes leading

to achievement of

the intended

program-level

learning

outcomes

Alignment of teaching and assessment with LOs

Course level

Learning Outcomes

Mapping Program-level LOs to Teaching,

Learning and Assessment

Programe

Learning

Outcomes

Degree

Level

Expectations

Teaching

Learning

Activities

Assessment

and

Evidence

Looking for Evidence that each program-level Learning

Outcome has been achieved. It is better not to assess

individual outcomes but focus on complex competencies

Some assessment methods are more suitable for single LOs

in courses, others for program-level LOs (portfolios and

capstone projects)

• Program outcomes are more global but should match a

specific domain

• Course outcomes are more specific

• Assessment of learning outcomes at the course level

can be aggregated (by learning outcome) to the level of

the program

Program vs. Course Outcomes

Aligning assessments with LOs.

• increases the probability that we will provide students

with the opportunities to learn and practice the

knowledge and skills that will be required on the various

assessments we design

• “good grades” are more likely to translate into “good

learning”

Assessment of LOs achievement

- essentially in the category YES/NO

- for all (representative) LOs if licensing is

considered

- for selected LOs (but students do not know which

of them will be chosen)

Types of assessment

• Formative vs Summative

• Criterion-referenced vs Norm-referenced

• Direct vs Indirect

• Holistic vs Atomistic

• Objective vs Subjective

• Quantitative vs Qualitative

• Longitudinal vs Transversal

• Divergent vs. Convergent Assessment

• Process Assessment vs Product Assessment

Norm- referenced Assessment

“Competitive assessment”

Compares students’ achievement with those of his /her

classmates

Student X has answered exam questions better than 25% of her colleagues , which means that she was better prepared

for 100MCQs

60 mean

Norm-referenced assessment

Mean result +/- 2 standard deviations

Can be also done by looking on “breaks” in distribution

“curve” assessment

failed v.good

Norm-referenced assessment

Can be used for students’ selection, comparison

Is not suitable for assessment of learning outcomes

achievements

Its results can hide very low level of students preparation

Is de-motivating for learning

Criterion-referenced assessment

“Assessment of competence”

Is based on standards established before exam

Student X has answered correctly 70% of questions, which means that she

achieved the required level of knowledge

Criterion-referenced assessment

• Reflects achievements on the required level

• Promotes group cooperation

• Gives information regarding efficiency of training

Criteria are best expressed in a numerical way:

• % of correct answers

• Number of points to be collected etc.

Miller’s pyramid

Behavior

Performance

Structure of the Observed Learning

Outcome (SOLO)

• SOLO is a means of classifying learning outcomes in

terms of their complexity

• At first we pick up only one or few aspects of the task

(unistructural), then several aspects but they are

unrelated (multistructural), then we learn how to

integrate them into a whole (relational), and finally, we

are able to generalised that whole to as yet untaught

applications (extended abstract)

• This approach is helpful in implementing constructive

alignment

Represents learning through 5 levels from quantitative to qualitative

BIGGS

Assessment of complex achievements

• Alternative assessment applies to any and all assessments that differ from the multiple-choice, timed, one-shot approaches that characterize most standardized and many classroom assessments.

• Authentic assessment applies to assessments whichengage students in applying knowledge and skills in the same way they are used in the "real world" outside of school.

• Performance assessment is an assessment in whichstudents are given opportunities to demonstrate their understanding and to apply knowledge, skills, and habits of mind in a variety of contexts.

Authentic Assessment

Three factors determine the authenticity of an assessment:

• the task (possibly interdisciplinary.

• the context, (as realistic as possible including time available, access to resources, group work)

• the evaluation criteria (similar to those used for evaluation of professionals

Wiggins (1990),

Performance assessment

Requires students to demonstrate that they have mastered

specific skills and competencies by performing or

producing something

• designing and carrying out experiments;

• writing essays, reports, poems

• using equipment or a technique

• building models

• playing musical instruments;

Performance criteria must define the attribute(s) being

evaluated and also develop a performance continuum.

Multi source assessment (360-degree

feedback )• is a method of systematically collecting

opinions about an individual's performance from a wide range of

coworkers

• is utilized by organizations to solicit information from a variety of workplace sources on an employee's work-related behavior and/or performance

• In medicine feedback can be collected from: patients, secretary, senior doctors

Portfolios:

• A student portfolio is a systematic collection of student work and related material that depicts a student's activities, accomplishments, and achievements in one or more school subjects.

• Portfolios are very useful for summative assessment of attitides and behaviors

• Portfolio may contain various types of student work such as:

-essays;

-reports from laboratories

-research projects

-volunteering activities

-experience abroad

Self-assessment

„Some people are highly competent but are unaware of it,„

Gruppen says. "I am less concerned with those individuals than with the students who are poor self-assessors and are doing poorly. They are the ones most at risk for making dangerous mistakes out of overconfidence."

`

T. DeANGELIS

Can be performed in different forms:

• Peer ranking (each member ranks all others on one or

more factors)

• Peer nomination ( indicating a member highest on a

particular characteristics or dimension)

• Peer rating (rating each other using the scale)

• Voting with cards given to chosen classmates

Advantages of self and peer assessment

• develops learner confidence in

assessing/ marking peers (through

practice).

• develops learner ability to self-

evaluate and reflect.

• saves teachers time

• makes feedback faster

• students can see tests not as

punishments but as useful feedback

Principles of students assessment from the

perspective of Learning Outcomes

•Assess all the outcomes

•Use assessment methods adequate for outcomes

•Combine formative and summative assessment

•Use criterion based assessment

•Standardize methods of assessment

Principles of students assessment from the

perspective of Learning Outcomes cont.

• Do not compensate lack of achievement in some

LOs by excellent achievement in others

• Consider external benchmarking

• Decide about % representation of LOs in the

exam

- Assess important aspects

- Use multiple methods of assessment

- You can assess several LOs in one exam/task

Assessment of students vs.

assessment of Learning Outcomes

Assessment of students’

achievements vs assessment of

learning outcomes

Achievements

of studentsLearning

outcomes

On the level of

institution /course/

On the level of an

individual person

Without gradesWith grades (sum of achievements)

Typical Grade Book

Summary

• Analyzing the row data results in a

summary of individual student

performance.

• Analysis of the column focuses on

learning outcomes and can identify

strengths and weaknesses of the course

which are not apparent when looking at

student summary data only.

Student Learning Outcomes-Competences

Student A B C D E F Student

performance

1 1 0 1 1 1 1 84%

2 0 1 1 1 1 1 84%

3 1 1 1 1 0 1 84%

4 0 0 1 1 1 1 67%

5 1 1 1 1 1 1 100%

6 1 0 1 1 1 1 84%

7 1 1 0 1 1 1 84%

8 1 0 1 1 1 1 84%

9 1 1 1 1 1 1 100%

10 1 1 1 1 1 1 100%

Outcome 80% 60% 90% 100% 90% 100%

Grading vs. Assessment

• The goal of grading is to evaluate individual students’ learning and performance. Grades may incorporate criteria – such as attendance, participation, and effort –that are not direct measures of learning.

• Assessment goes beyond grading by systematically examining patterns of student learning across courses and programs and using this information to improve educational practices.

Grading vs. assessment of LOs

100%

%0

LO achievement level

Grades indicate achievement ABOVE the required level.

Student’s

achievement

Grading

• Evaluates the quality of a student’s work;

• Communicates with the student, as well as employers,

graduate schools, and others;

• Motivates how the students study, what

• Organizes to mark transitions,

Criteria for grading:

• A (5) - very good – student knows very well

• B (4) - good - student knows well

• C (3) - satisfactory - student has satisfactory

knowledge

• F (2) - failed - student does not know enough

Criteria for grading

1. A when student knows 95-100%

2. B when student knows 83-94%

3. C when student knows 60-82%

4. F when student knows less than 60%

Using grades for meaurement of LOs

1. Grades should distinguish between indicators of

learning outcomes and indicators of other

behaviors.

2. Grades should be based on clearly articulated

criteria that are consistently applied.

3. Separate grades should be used for the major

components of knowledge and skills so that evidence

of students’ specific areas of strength and weakness

could be identified.

Traditional grading of the course

Requirement Points

Attendance and participation 50

Homework 100

Exams (4 at 100 points) 400

Presentation 100

Final Exam 200

TOTAL 850

K. McClendon,E. Eckert

Outcomes-based Grading Scale

Points Grade

44-50 A (superior or good on all outcomes)

37-43 B (good or adequate on all outcomes)

30-36 C (adequate all outcomes)

15-29 D (developing)

-0-14 F (non-existent or minimal)

K. McClendon,E. Eckert

Grades

• Course grades are generally calculated by averaging the results of various measures.

• Specific strengths and weaknesses are masked by the process of averaging.

• One student with a grade of "C" might perform very differently from another student with a grade of "C." And the letter grade by itself fails to suggest ways for students to improve their performance. What does a "C" grade tell a teacher or, more important, a student about how that student can improve?

Grading individual LOs

LO Identify

sol.Ex1

15pt

Identify

sol.

HW2

20pt

Impl.

sol.Ex2

10pt

Impl.

sol. HW3

10pt

Total

Impl.sol

35pt

Total

ident.sol

20 pt

Total

score 55

pt

Stud.1 10 15 6 9 25 15 40=C

Stud.2 14 18 10 8 32 18 50=B

Stud.3 14 17 5 5 31 10 41=C

Class

average

12,67 16,67 7 7,33 29,34 14,33

Maintaining Grading Consistency in Multi-

sectioned Courses

• agree your grading policies, standards, and criteria

• have teaching assistants grade assignments for students

not in their section

• have each section of an exam graded by only one

teaching assistant

• have each section of an exam graded by only one

teaching assistant

•

••

•

••

Designing the Grading System

• Determine weight of each outcome.

• Determine relationship between points and performance

levels.

• Determine cutoff for a passing grade, points needed for

higher grades.

K. McClendon,E. Eckert

But remember “no grade or grading system is immutably

right by some eternal standard”

Seven Common (Mis)Perceptions About

Outcomes Assessment

• We’re doing just fine without it. (Assessment is medicine only for the sick.)

• We’re already doing it. (Assessment is just old wine in new bottles.)

• We’re far too busy to do it. (Assessment is an “administrivial” burden.)

• The most important things we do can’t/shouldn’t be measured. (Assessment is too reductive and quantitative.)

• We’d need more staff and lots more money to do assessment. (Assessment is too complex and expensive.)

• They’ll use the results against us. (Assessment is a trick or a Trojan horse.)

• No one will care about or use what we find out. (Assessment is a waste of time.)

„it is better to measure in unperfected way

something important than to measure

perfectly something which is irrelevant”

Hamdorf i Hall 2001

THANK YOU