Modeling Units of Assessment for Sharing Assessment Process Information: towards an Assessment Process Specification Yongwu Miao, Peter Sloep, and Rob.

Modeling Units of Assessment for Sharing Assessment Process Information:

towards an Assessment Process Specification

Yongwu Miao, Peter Sloep, and Rob Koper

Educational Technology Expertise CentreOpen University of The Netherlands

ICWL 2008, 20-22 August 2008, Jinhua, Zhejiang, China

Overview

• Where are we from: OUNL

• What we have done: EML => IMS LD

• What is missing in IMS LD: Assessment

• What is our objective: A New Specification

• What is the current status: Conceptual Model

• How can it be used: Olympic Games

• Summary

Where are we from: OUNL

Where are we from: OUNL

The Open University of the Netherlands (OUNL)

– Started in 1984; national institute;– Open distance education provider

• 7 faculties, 24000 students• 24 study centres in Netherlands and Belgium• education delivered using a variety of

technologies (print, cd-rom, internet, face to face contact sessions, practical rooms, etc.)

– R&D: Educational Technology Expertise Center

What we have done: EML => IMS LD

• Educational Modelling Language (EML)• IMS Learning Design (LD): An e-learning technical specification

IMS LD is derived from EML

IMS Global Learning Consortium, Inc.OUNL is contributing member (voting rights) of IMS

– Also Apple, Blackboard, Microsoft, Cisco, Sun, Texas Instruments, …

1998

* Information model

* Best practice guide

* XML binding

2003

What we have done: EML => IMS LD

Basic idea:Develop a ‘language’ to describe the learning/teaching processes with associated content in a formal, semantic way, so that it can be interpreted by computers.

Compare: music notationThis description should be independent of any delivery platform => can be deployed in any platform that provides the adequate functionality

What we have done: EML => IMS LD

Learning object:- Book,- Audio & video

Whichnotationsare needed to describe a lesson plan?

What we have done: EML => IMS LD

IMS LD, an education-specific process modelling language, can be used to describe a wide variety of pedagogical strategies or approaches in terms of a play including acts and role-parts (based on metaphor of theatrical play)

•People act in different roles•working towards certain objectives•by performing learning and/or support activities•within an environment, consisting of learning objects and services used in the performance of the activities.

Moving focus from what to learn to how to learn: Lesson plan or learn-flow

play

Act 1 Act 2 Act 3 Act 4 Act 5

Role-part 1

Role-part 2

Role-part 3

Role-part …

Role Activity Environment

Learning objects

Learning services

Activity-Description

method

components

What we have done: EML => IMS LD

What we have done: EML => IMS LD

LD authoring tool

LD-compatiblerun-time environment

actual or imaged pedagogical processes

a formal pedagogical model

multiple executions of the same modelcarried out by different groups

the events that can be observed or expected

represented in LD(in the form of XML)

Run-time

Design-time

author

learner,teacher,and other stakeholder

What is missing in IMS LD: Assessment

LD can not explicitly support various types of assessment.

Assessment components within the Educational Modeling Language were excluded when LD was adopted by IMS, because of the existence of IMS Question and Test Interoperability (IMS QTI).

QTI is also an open e-learning technical standard, which describes a data model for the representation of assessment item/test and their results. It defines a set of interaction types which can be used to specify basic question types (e.g., multiple-choice, fill_in_the_blank, and slider) and complicated question types through combination.

What is our objective: A New Specification

1. facilitating representation, understanding, communication, comparison, and reuse of a variety of assessment practices at design-time;

2. guiding and scaffolding learner, tutors, and other stakeholders to perform a variety of activities at run-time by providing guidance and awareness information, by configuring a workspace for carrying out prescriptive tasks, by controlling and changing the sequence of activities based on the execution state and circumstantial information, and by orchestrating the efforts made by different roles/participants.

Develop a high-level assessment process modeling language for facilitating both experts and practitioners to share assessment process information by extending QTI and LD with an additional layer.

What is the current status: Conceptual Model

• Semantic aggregation model• Conceptual structure model• Process structure model

Developed a conceptual model, the core of the specification, which can be represented as:

The first validation study has been conducted through investigating whether the conceptual model of specification meets the requirements of completeness, flexibility, adaptability, and compatibility.

What is the current statusSemantic aggregation model:

What is the current statusConceptual structure model:

What is the current statusProcess structure model

What is the current statusRestrictions between stages and activities:

How can it be used: in Olympic Games

It is not necessary to develop a software applications for supporting each assessment

How it can be used: in Olympic Games

rate 2.1

perform1

Athlete Rater 1 Rater 2 Rater n

rate 1.1 rate n.1

rubricvalues 1.1

rubricvalues 2.1

rubricvalues n.1

rate 2.5

perform5

rate 1.5 rate n.5

rubricvalues 1.5

rubricvalues 2.5

rubricvalues n.5

collect and calculate5

score5 and total

…

…

…

calculate1

decisionrules

score1 publish and view result1

…

publish and view result5

system

assessmt.form

assessmt.form

assessmt.form

assessmt.form

assessmt.form

assessmt.form

How it can be used: in Olympic Games

Use existing Run-time Environment

Player.1 Player.2 Player.n

internet

Dispatcher/Service broker

IMS LD engine IMS QTI engine

A A

Assessment service

UoL managementand enactment service

…

an assessment model

multiple instances

server

How can it be used: in Olympic Games

An assessment model can be reused and modified to support other assessment

Summary

IMS LD can support interoperability and reusability of a variety of pedagogical models and systems.

IMS LD provides insufficient expressiveness to model integrated learning, teaching and assessment processes.

A high-level assessment process modeling language can be developed by extending IMS QTI and IMS LD with an additional layer.

The conceptual model of the specification has been developed and validated.

Thanks for your attention

Questions?