Use of an objective assessment tool to evaluate students basic electrical engineering skills Nandini Alinier University of Hertfordshire, U.K. Engineering.

Use of an objective assessment tool to evaluate students’ basic

electrical engineering skills

Nandini Alinier

University of Hertfordshire, U.K.

Engineering Education Conference 2006

Overview

• Background & origins• Application in an Engineering context• Stages of OSTE

– Development of stations– Preparation for sessions– Pilot– Implementation

• My perspective• Students’ perspective• Reflections

Nandini Alinier, EE2006, 25 July, Liverpool, U.K.

Background• Objective Structured Clinical Examination (OSCE)

(Harden 1975)

• Originally developed in the mid-seventies in Dundee, Scotland– Used in medical education– Method to objectively assess the skills of medical trainees

• Used in a project funded by the British Heart Foundation at UH during 2001-03 (Alinier et al 2004)

– Involved as an assessor– Lead investigator’s involvement

• Perceived positively by students & lecturers (Alinier 2003)


OSCE• Comprises 15 to 20 stations• Station: short task performed in 5 minutes

– Practical exercise• Requires an assessor

– Theoretical exercise• Pen & paper

• Gap period between stations (1 minute)– Students rotate individually through the stations– Allows time for assessors to complete marking & reset any equipment

• Timing of session by an electronic clock• Students given explicit written instructions• Objective marking sheet for each station


Applied to Engineering

• Project undertaken in the School of Electronic, Communication & Electrical Engineering at the University of Hertfordshire, UK

• Funded by the Higher Education Academy Engineering Subject Centre (Learning & Teaching Support Network (LTSN) Engineering)

• Academic year 2004-05

• Objective Structured Technical Examination (OSTE)


Study about the OSTE

• Used for formative assessment

• Piloted with 2nd year BEng students

• Conducted with 1st year BEng students

• Circulated feedback questionnaires for students + assessors


Stages of the OSTE


Stage No. Stage

1 Development of OSTE stations

2 Preparation for OSTE sessions

3 Pilot of OSTE

4 Running of OSTE

Stage 1: Development of OSTE stations

• Gathering a variety of ideas about essential electrical engineering skills– Seminar within School– Call for ideas via Email

• Identified appropriate range of exercises doable in 5 minutes

• Based on learning objectives for relevant group of students• 16 stations were short-listed• Developed instruction sheets & marking criteria for each

station


Stage 1: Development of OSTE stations (continued)

• Types of stations:– Theoretical: pen & paper– Practical: assessor required– Computer-based: assessor required


Stage 2: Preparation for OSTE sessions

• Development of PC-based clock with an integrated voice prompt

• Briefing of students

• Training of staff to act as assessors

• Designing of student and assessor feedback questionnaires


Stage 3: Pilot of OSTE

• Group of 15 volunteer 2nd year BEng students

• Resulted in changes in:– Tasks to be carried out– Marking sheet– Instruction sheet


Example of a computer-based station


Stage 4: Running of OSTE

• 32 BEng students in the second semester of their 1st year

• 16-station OSTE

• 2 sessions on the same afternoon

• Formative OSTE– Feedback given to students by assessor during

the gap period


My perspective: benefits

• Identifying students’ strengths and weaknesses

• Students forced to do practical work individually

• Being assessment-driven, hence OSTE encourages students to participate

• Feedback given individually to students


My perspective: concerns

• Development of the OSTE stations is time consuming– Once the stations have been developed, running

the OSTE session is fairly straightforward

• Running the OSTE is human resource intensive– However no need for marking after the session

as most of it is done during the session

• A few students may find the OSTE stressful due to the intense assessment


Students’ perspective

• Lets students know what is expected of them and what their strengths and weaknesses are

– “The OSTE session helped clarify what I was supposed to be able to achieve”

– “I know my weaknesses and what I need to know”


Students’ perspective (continued)

• Students feel they have learnt something during the OSTE

– as a result of immediate feedback from the assessors

– OSTEs had addressed areas that would not have been picked up by other forms of lab work

– when marking coursework and lab books, the lecturer might assume competence (e.g. in setting up an oscilloscope), when in fact the student had only just coped or had relied on their partner to do that part of the experiment

– OSTE would show whether students could do the task:• Effectively – getting the best settings, not just something usable

• Efficiently – because of the tight time limit


Students’ perspective (continued)

• Unlike my concern, students did not find the exercise stressful“Some people get a bit stressed before exams, whereas

this is informal, you don’t get so worried”

• For similar reasons, students were against the idea of summative OSTEs, when they would probably attempt to cram specifically for the exam“if we’re assessed for it we’re all going to prepare for it…

rather than be tested on what we know”


Benefits

• Assesses specific key competencies that may be missed by other broader assessments

“Your lab skills are not always assessed, they just check your logbook and see your final result. This OSTE, it checks your method… how you’re connecting equipment … not everyone uses [the equipment] properly in the right way”


Benefits (continued)

• Provides an opportunity for students to learn core skills, allowing students to focus on deeper aspects of experimental work during practical sessions

• Lets students know what is expected of them and what their strengths and weaknesses are


Reflections

• Time consuming• Human resource intensive• Valuable tool:

– Assess core skills– Highlight deficiencies– Focussed one-to-one teaching– Can be tailored to assess learning outcomes

• Students think that OSTEs should be used formatively in every module they study


Thanks for your attention

Any questions?

[email protected]


Use of an objective assessment tool to evaluate students basic electrical engineering skills Nandini Alinier University of Hertfordshire, U.K. Engineering.

Documents

students nandini alinier

running of oste slide

students lecturers alinier

station oste

development of oste

assessors nandini alinier

weaknesses students

students strengths