Bringing e-assessment to a higher maturity on a national scale

Bringing e-assessment to a higher maturity on a national scale

dr. A. Hartoog, SURF, Utrecht

drs. N.R. Bos, Fontys, Eindhoven

Educause october 2011

Program

1 introductionproblems, questions, all somehow related to e-assessment

10 min

2 discussion how is this situation in the US and/or Canada?

15 min

3 elaboration how are we tackling these issues in the Netherlands, on a national scale

15 min

4 questions/answers 10 min


context

3Educause october 2011

The Netherlands: - surface area 40.000 km2 (US is 270 times bigger)- population density: 491 people/km2 / (US: 34 people/km2)

- about 60 universities,- about 600.000 students- two handfuls of government funded and commercial

R&D institutes

Problems, questions…(1)

- many students drop out in their first year of study

- and if they don’t, their success rate is less than optimal

- can we improve this situation via (digital) testing?



- Assessment is a key proces in education

- Testing students properly is tedious and time-consuming

- Teachers in general are no trained specialists in the assessment proces, and are usually already overloaded

- How to improve this situation?



- Clinical reasoning is a key competence for doctors

- proper assessment in their education is crucial

- given the high number of students in medical schools,

- how on earth do you assess this competency reliably



- teaching statistics is a key component in many educations

- statistics is often a stumbling block for students

- statistics typically has slow and fast students

- how do you make statistics education, training and assessment more effective and fun for the slow and fast alike



- apprentices in nursing experience a gap between education and practise, leading to drop-outs

- there are many educations in nursing, with the associated variety in end-terms

- how to tackle this?


Discussion

- can e-assessment reduce teacher workload?

- does information technology raise the quality of testing?

- can e-assessment reduce students dropping out?

- is it possible to test complex skills with closed questions?

- how could e-assessment improve statistics education?

- could e-assessment reduce the reality gap in nursing?

- would a national infrastructure for e-assessment be useful?

- how can you stimulate collaboration between educations?

- how do you introduce e-assessment in law education?

all the above on a national scale!


So, what we did?

Context:Role of SURF in Higher Education in the Netherlands

Program:Testing and test driven learning

Impact:Things that work, things that don’t



SURF: ICT innovation by and for higher education and research

Research universities, universities of applied sciences, and research institutions work together in SURF in order to: • improve the quality of research and

education by:• creating pioneering ICT innovations.

For more than twenty years, collaboration within SURF has led to products and services that the participating institutions could not have achieved on their own.

11

between all Dutch institutions for higher education & research


42Universities of

applied sciences

Dutch Organisation for Applied Scientific

Research (TNO),Telematics Institute

14Universities

National Library

Smallerinstitutions of

higher education

Netherlands Organisation for

Scientific Research (NWO)

Royal Netherlands Academy of Arts

and Sciences (KNAW)

CommercialR&D institutes

Collaboration=

Success factor!

SURF is a partnership

12

Collaboration within SURF makes it possible to:

• carry out a joint (public/private) survey of what higher education & research requires/will receive

• cluster ICT requirements on the demand side (demand pull)

• arrive at sustainable ICT services suitable for joint use

• be innovative in contracting out services

• encourage the use of innovative services (supply push).

SURF as an innovator

Educause october 2011 13

SURF subjects

- advanced networks

- security/identity management/federation

- collaborative environments

- software procurement/licence management

- digital rights & open access

- open educational resources

- SURF academy

- information architecture

- national information brokerage


The SURF method

Implementation

SURF directs and coordinates the implementation of funded projects: • Projects are carried out close to the source

of expertise: at institutions of higher education and in enterprise.

• SURF sets up the programme, invites tenders, ensures independent quality assessment and monitoring, makes project results available.

Educause october 2011 15

Program “Testing and test driven learning”

Problems in higher education in the Netherlands:

- high drop-out rate- high workload on teachers

SURF launched a national program to tackle these problems with e-assessment


Purpose of this program

- improving transition from secondary higher education with diagnostic tests and revision material, to be jointly developed by institutions,

- improving students’ progress by means of progress tests,

- reducing the time needed to develop tests by collaboration, i.e. developing collections of digital tests,

- reducing the time taken to mark tests by deploying computer-based testing,

- providing expertise and professional development for instructors in the field of testing.


programme composition

the program focusses on three main area’s:

1. enabling institutions to jointly develop computer-based tests and assessments

2. realising an expertise network, aimed at improving quality of testing

3. realising a national technical infrastructure to combine successful national initiatives for computer based testing


What kind of projects?bachelor nursing:

- “body of knowledge”

medicine:- interuniversity progress test medicine (2)- testing of clinical reasoning- testing with 3D images (research)

math and statistics- foundation of national test database- research into effects of progresstesting- test-based learning (adaptive testing) (2)

teacher education- research into testing abilities of teacher educators- research into methods for portfolio assessments

-> and: new tender by 1 november


Math & Statistics projects

- Stumbling block- Faster and slower students- Based on earlier project “Mathematics Garden”

-> computer adaptive monitoring

- Students: can practicecan adjust their learning process- Reduction in drop out

- Teachers: can adjust their teaching process- Reduction in workload

University of Amsterdam, University of Utrecht, Twente University

Medical projects

- Low dropout rates students but high workload teachers (doctors)

- Development of questions on a national scale- Sharing these in national databases

- Different goals:- Measuring progress- Measuring Clinical Reasoning (complex skills)- Improvement gap practice/ theory

- Involvement: all nursing schools, all medical schools

Research projects

3D Testing (CT/MRI)

The project aims at improving the psychometric quality and validity of assessment of 3D images. Also gain insight into the cognitive processes required in the interpretation of 3D images

Progress testing and study success- First year economic students will get weekly

progress tests- Will this increase [retention rates]?- Does this also effect workload for teachers, costs of

assessment, quality of testing?

Network of expertise

Fysically:- a Special Interest Group of inspired specialist- seminars, schools, master classes (SURF Academy)- small scale practical research projects

Virtually:web portal that acts as a centre of expertise:- good practises- white papers- links to literature- ask and find specialists- blogs, discussions forum- agenda

free accessible and maintained by the higher education community


National infrastructure

ambition:

investigate, realize and exploit the advantages of a common infrastructure for computer based assessment on a national scale

healthy in all respects, to stand on its own when the program finishes, and external funding is discontinued


what’s to gain? (1)

B

user i/f

applications

middleware

operating systems & networks

hardware

A

Ccombi

?

user i/f

applications

middleware

operating systems & networks

hardware

user i/f

applications

middlewareoperating systems & networks

hardware

user i/f

applications

middlewareoperating systems & networks

hardware

sharing systems:


what’s to gain? (2)

A

B

Ccombi

?

helpdesk

funct.mgnt

appl.mgnt

technical mgnt

infra.mgnt

helpdesk

funct.mgnt

appl.mgnt

technical mgnt

infra.mgnt

helpdesk

funct.mgnt

appl.mgnt

technical mgnt

infra.mgnt

helpdesk

funct.mgnt

appl.mgnt

technical mgnt

infra.mgnt

sharing system management:


lessons learned up to now

- subject domains in education (math, health, law, humaniora, etc) are very different

- educational institutions are very different

- the advantages of sharing an infrastructure are not always obvious

- computer based testing is far from mature


so, what do we do

- confine within domains- investigate and explore along the functional lines- make advantages explicit- join already successful initiatives- take small steps, but think big- develop business models

and this might actually work:- example


collaboration in medical and nursing education

education in nursing and

medicine

BoKSBody of Knowledge

& Skills

iPTMinter University

Progress Test Medicine

ClinicalReasoning

Testingin 3D

AdaPT

?

?

?


Questions, answers

shoot!


Alfred [email protected] www.surf.nl

Nynke [email protected]


mailto:[email protected]

http://www.surf.nl/

http://www.amc.nl/

Impact, how to measure

- this four-year program is one year underway

- high level goals:- reduce number of students that drop-out- reduce workload on teaching staff- raise quality level of e-assessment

How do you measure this?basic problem: there are many factors (not related to the program) that may have a dominant influence on above subjects.


evaluation and impact

- decompose high-level program goals to manageable project proportions

- find ways to measure these parameters, and assess them during the project

- develop a model to arrange the findings in levels of evidence (based on the knowledge-pyramid from Kennisnet)

- inspiration (this is great: it should work)- perception (this actually works for me)- existence (others confirm that it works)- evidence (scientific proof that it works)


Evidence:measured benefits

Perception:experienced benefits

Existence:deployment of the project

Inspiration:Idea


Bringing e-assessment to a higher maturity on a national scale

Education

special interest

deze doet

low success

higher education

operating

test driven

higher education

measures students