Jean Hughes Office of the Vice President for Learning Innovation, DCU.

Jean HughesOffice of the Vice President for Learning Innovation,DCU

Overview

DCU Approach Framework Implementation Network Learning Outcomes General – Assessment of Learning

Outcomes Common Questions

DCU

Already Modularised and Semesterised (1997) Module Descriptors use ‘Learning Outcomes’ European Credit Transfer System (ECTS) well established But

Mainly administrative exercise Academic Framework for Innovation (AFI) – for Deep

Curriculum Reform (3 year project) Bologna Compliance Demonstrable Alignment with NFQ Use of Learning Outcomes – for real change in assessment

practice Flexibility – Designed in and through new Marks &

Standards

Framework Implementation Network

Created by the IUA and the NQAI Membership

7 Universities NUI Colleges

Purpose – to collectively address outstanding issues relating to Bologna and the National Framework of Qualifications

Three separate Working Groups Award Titling Disciplinary Learning Outcomes Assessment of Learning Outcomes

Key Issues

Ensuring that Assessment, Teaching and Learning are aligned

Moving from assessing content to assessing outcomes

Ensuring all outcomes are assessed and all assessment instruments actually assess outcomes

Avoiding over-assessment

Learning Outcomes

Clearly identify what a learner can demonstrate as a result of successfully completing a part of a learning programme

They describe student attainment which is demonstrable and assessable.

Benefits of Learning Outcomes

Guide students in their learning - what is expected of them - in turn helping them to succeed in their studies.

Help lecturers to focus on exactly what they want students to achieve in terms of both knowledge and skills.

Provide a useful guide to inform employers about the general knowledge and understanding that a graduate will possess.

Challenges

Traditional view inputs = quality of learning

outcomes Moving from content (Teaching)

focus to Outcome (Learning) focus

Aligning teaching, learning and assessment

Award and Module Outcomes NFQ describes generic, high level outcomes

for awards at each level under 3 strands and 8 substrands

How do we demonstrate achievement of award outcomes?

We don’t assess at award level we assess at module level

Need to ensure that module learning outcomes collectively deliver the award outcomes

PO1 PO2 PO3 PO4 PO5 PO6Mod2 Mod1 Mod6 Mod5

Mod4 Mod3 Mod4

Mod7 Mod8 Mod3 Mod5

Mod11

Mod9 Mod2 Mod10

Alignment of Programme and Module Outcomes

PO – Programme Outcome

Mod - Module

Assessing Learning Outcomes

Constructive Alignment

is the underpinning concept behind the current requirements for programme specification, declarations of intended learning

outcomes and assessment criteria, and the use of criterion based assessment.

(Biggs, 1999)

Example – Project Management

  Learning Outcome(Mismatched)

Learning Outcome(Matched)

Tasks  

Students will understand how to plan a multi-disciplinary project

Students will demonstrate ability to plan a multi-disciplinary project

Assignment 1 Students will complete an in-class test on Project Management

Students will devise an appropriate project plan from a scenario/business case study

Assignment 2 Students will create a Project Plan using MS Project during labs

Students will create a Project Plan in MS Project from the plan in Assignment 1. They will be demonstrate appropriate use of Base Lines, Serial and Parallel Tasks, Variances, Resource Allocation and Critical Path, through an in-lab demonstration.Final exam 1 Question on final exam Case Study exam question requiring analysis of a Business Problem, design of appropriate project plan and discussion of variety of project planning issues.

Student Perspective

From our students’ point of view, assessment always defines the actual curriculum (Ramsden, 1992)

Biggs (2003)Teacher Perspectives:Objectives Learning Outcomes Teaching Activities Assessment

Assessment Learning Activities OutcomesStudent Perspectives

If curriculum is reflected in the assessment, the teaching activities and the learner activities are directed towards the same goal – in preparing for the assessment students will be learning the curriculum

Alignment

Assessment – Impact(Boud, 1995)

“There is probably more bad practice and ignorance of significant issues in the area of assessment than in any other aspect of higher education”.

“The effects of bad practice are far more potent than they are for any aspect of teaching. Students can, with difficulty, escape from the effects of poor teaching, they cannot (by definition if they want to graduate) escape the effects of poor assessment”.

Assessment - Terminology

Learning Outcomes What a student should be able to demonstrate (May imply the assessment criteria)

Assessment Criteria The basis on which a judgement of the adequacy of the

student work is made

Assessment Methods The task(s) undertaken by the student – writing an essay,

answering an exam question etc- that is subject to assessment.

Many assessment methods may be appropriate to the same outcomes – e.g. Written, Oral, Visual, Project etc

Must be valid and reliable – Fit for purpose

Over-Assessment

Typical with semesterisation and modularisation High risk with learning outcomes

Tend to be quite granular and have several independent outcomes

To avoid Try combining the assessment of more than one LO LOs only need to be assessed once Use assessments which inform/follow on from each

other Check what you are assessing

Checking Assessment

Tasks LO1 LO2 LO3 LO4 LO5 LO6

Assignment 1

Assignment 2

Assignment 3

Final exam

Issues:

What does assignment 3 assess?

How is Learning Outcome 2 assessed?

Are Learning Outcomes 1 and 4 over-assessed?

Refining Assessment

Tasks LO1 LO2 LO3 LO4 LO5 LO6

Assignment 1

Assignment 2

Assignment 3

Final exam

Perhaps design a new assessment 3 to re-balance?

Common Questions

Must all outcomes be assessed? Yes, but

Not independently Look at how you write the learning outcomes

• Students are expected to be able to 1. Write a business report2. Present orally3. Design a visual presentation

versus • Students are expected to be able to

1. Communicate effectively using a combination of written reports, oral presentations and visual media

– This reduces the number of independent outcomes and also recognises the combination of communication mechanisms

– BUT For certain disciplines, for example languages, you may

need these to be independently demonstrated

Example

In a Business Module (1 outcome) Students are expected to be able to communicate

effectively in a business context using appropriate communication tools including Written, Oral and Visual

The outcome is communication, the tools are some means of demonstrating that outcome

In a languages Module (4 outcomes) Students are expected to be able to demonstrate written

proficiency in French Students are expected to be able to translate a piece of

French text Students are expected to be able to comprehend a piece of

French text Students are expected to be able to deliver an oral

presentation in French

Must all outcomes be passed? In theory, yes, but… Threshold Achievement

Implies that ALL learning outcomes must be passed in order to pass a module

• Very strict interpretation of LO approach• Leads to reductionist or ‘Tick Box’ approaches

Typical Achievement Learning outcomes are written (and

assessed) with the ‘typical’ or ‘’average’ student in mind

• Grading criteria will allocate marks to degree of achievement

But, we don’t currently assess everything….

Exams often require only 3 out of 5 questions Therefore we don’t assess everything But

Generally this refers to content Outcomes should be about demonstrating

knowledge/understanding aided by knowledge of content, not content in its own right

Assessment criteria and methods should satisfy you that students can demonstrate achievement of the outcomes

Worst-case scenario – no worse than current situation!

What if some outcomes are failed?

Depends on the criticality of the outcome If particular outcomes must be passed then

passing other outcomes should not be able to compensate

Need to look at ‘Threshold’ and ‘Typical’ Outcomes

Need to write outcomes carefully Need to ensure that supplemental/repeat

assessment is also valid

What about repeats?

Did they actually fail? “Students will present orally on 20th century politics”

• If they don’t attend they fail….. Is this what you meant? or

“Students will effectively communicate on 21th century politics”

• You may prioritise Oral assessment and allocate more marks, but this allows more choice of methods

If a student has to repeat the supplemental assessment must be valid if different from the original Look at how you write the outcomes

Is it an Outcome, An Assessment or an Efficiency? Often mix up pedagogical (or efficiency) approaches with learning

outcomes Example Group Working

Learning Outcome• ‘Students will demonstrate effective group working skills’ • Group working needs to be ‘taught’ and assessed• Can’t assume that putting people in groups will automatically lead to

effective group working!• How does an individual student repeat? • ‘Students will be able to evaluate group dynamics and effective group

working’ • Would this suffice but give an alternative assessment approach for repeat

students? Pedagogical Approach

• Group working is used to facilitate learning of a particular outcome• Not assessing the group working – assessing the actual outcome

Efficiency• Often used to reduce the volume of marking• Need to be careful that it is pedagogically sound• Not assessing group working

Making the Tacit Explicit

Some disciplines rely on expertise of the individual lecturer ‘I know a good piece of work when I see it’ ‘You can’t possibly express Creativity, Performance,

Medieval English….. Using learning outcomes NOT about subjectivity – expertise, tacit knowledge etc

How do students know what to aim for? How do novice lecturers know about standards? How can we demonstrate consistency, transparency

etc?

Example

First, rewrite the outcome:“By the end of this module students will understand the importance of post-

renaissance European art in the context of art history”Redesigned: By the end of this module students will be able to: Evaluate and criticise post-renaissance European art within its historical

context Date and identify key works of this specific period Demonstrate an understanding of the particular art-historical debates

that have taken place about this period Show ability to argue why this particular place and time raise interesting

issues about the discipline as a whole University of Sussex

Then, use a rubric

Also called grading criteria or scoring guide A set of guidelines for marking or scoring Can be a list, chart, or guidelines They state all dimensions being assessed They contain a scale They assist the marker on rating

assessments on the scale

Rubrics - 3 Types

Checklists Indicating list of criteria

Rating Scales Checklist with rating scale added Show degrees by which students meet criteria

Holistic Scoring Guide Used where assignments can vary i.e. projects Narrative descriptions of characteristics of different

grades of work Not list of criteria

Rubric Design

Achievement Levels

Criteria

Objective 1

Objective 2

Objective 3

Excellent Good Needs Work Not acceptable

Accepted Minor Major Rejected

revision revision

Expert Advanced Intermediate Novice

6-5 4-3 2-1 0

2.6.2 A Checklist Rubric for a Web Site (Suskie, 2004)

The purpose of the site is obvious. The site’s structure is clear and intuitive. Titles are meaningful. Each page loads quickly. The text is easy to read. Graphics and multimedia help convey the site’s main points. The design is clean, uncluttered and engaging. Spelling, punctuation and grammar are correct. Contact information for the author or webmaster is given. The date each page was last update is given.

Simple Checklist Rubric

Learning Outcome

Und

erst

andi

ng

Satis

fact

ory

Inad

equa

te

Insu

ffici

ent

Info

rmati

on to

Ev

alua

te

1. Write articulate, persuasive and grammatically correct business materials.

2. Use critical, flexible and creative thinking to generate sound conclusions, ideas and solutions to problems.

3. Use software and networking services to obtain, manage and share information.

4. Apply understanding of domestic and international diversity concepts and issues to business situations.

5. Recognise ethical challenges and reach ethical business decisions.

Rubric is for a written Business Assignment

Criterion: Position taken on a debateable issue

Description

Weighting

Student takes a defensible position on the issue posed and states the position clearly. Position does not merely state the obvious or parrot one of the readings, but shows a creative mind at work.

5

Student takes a defensible position on the issue posed and states the position clearly. Position may be somewhat obvious or closely parallel to one of the readings.

4

Student takes a defensible position on the issue posed and states the position clearly but the position may state the obvious or simply paraphrase one of the readings.

3

Student takes a defensible position on the issue posed, but the statement is ambiguous, carelessly stated or must be inferred.

2

Student does not clearly state a defensible position, or position is not defensible or is irrelevant to the question posed.

1

This assignment asked students to take a position on a debateable issue regarding interpretation of literature they had studied. The lecturer has identified 3 major criteria – Position, Support and Acknowledgement of Alternative Points of View and has weighted them different levels of attainment of each criterion. Opposite is the detailed rubric for the Position criterion.

In Summary

Need to write learning outcomes so that they can be demonstrated and assessed

Need to identify assessment criteria and appropriate assessment methods

Need to be sure exactly what it is students need to demonstrate

Making the tacit explicit – Clearly written outcomes and the use of rubrics to articulate achievement

Questions?