Monitoring and Evaluation Harvey Hurree David Tye @LondonMetUni /londonmetuni londonmet.ac.uk
Jan 12, 2016
Monitoring and Evaluation
Harvey HurreeDavid Tye
@LondonMetUni
/londonmetuni
londonmet.ac.uk
Agenda
1 Overview – Monitoring & Evaluation - Harvey Hurree
2 Performance Enhancement Meetings - David Tye
3 Course Evaluation - Harvey Hurree
londonmet.ac.uk
The Quality Code (Chapter B8) sets out the following Expectation about programme monitoring and review, which higher education institutions are required to meet.
Higher education providers, in discharging their responsibilities for setting and maintaining academic standards and assuring and enhancing the quality of learning opportunities, operate effective, regular and systematic processes for monitoring and for review of programmes.
The following Expectation (Chapter A: Expectation A3.3) is also relevant to this topic:
Degree-awarding bodies ensure that processes for the monitoring and review ofprogrammes are implemented which explicitly address whether the UK thresholdacademic standards are achieved and whether the academic standards required by the individual degree-awarding body are being maintained.
londonmet.ac.uk
The Indicators
Indicator 1Higher education providers maintain strategic oversight of the processes for,and outcomes of, programme monitoring and programme review, to ensure processes are applied systematically and operated consistently.
Indicator 2Higher education providers take deliberate steps to use the outcomes of programme monitoring and review processes for enhancement purposes.
Indicator 3Higher education providers operate a process to protect the academic interests ofstudents when a programme is closed.
Indicator 4Higher education providers define processes, roles and responsibilities for programme monitoring and programme review and communicate them to those involved.
londonmet.ac.uk
Indicator 5Higher education providers evaluate their processes for programme monitoring andreview and take action to improve them where necessary.
Indicator 6Higher education providers make use of reference points and draw on expertise from those outside the programme in their processes for programme monitoring and review.
Indicator 7Higher education providers involve students in programme monitoringand review processes.
Indicator 8Higher education providers enable staff and other participants to contribute effectively to programme monitoring and programme review by putting in place appropriate arrangements for their support and development.
londonmet.ac.uk
londonmet.ac.uk
Course design in line with agreed
benchmarks, frameworks and
regulations
Course Validation
Student Feedback
Course Performance
Data / External
Examiner
Monitoring
Module and Course
Modification
Course Review
Quality Cycle at London Met
Monitoring & Evaluation Working Group (MEWG)
Set up April 2013
Chaired by Dominic Palmer BrownAll 4 faculties represented
Office of Institutional EffectivenessAcademic Registry
QEU
londonmet.ac.uk
Monitoring & Evaluation Working Group (MEWG)
Tasks• To consider the effectiveness and value of current monitoring and evaluation
procedures• To consider the UK Quality Code Expectations in respect of monitoring and
evaluation• To consider sector good practice in respect of monitoring and evaluation
Objectives• To recommend a cohesive, timely and effective set of quality monitoring and
evaluation practices to the University’s Undergraduate and Postgraduate Committee
• To identify actions (quick wins) that might be achieved quickly and easily• To identify resources that might yield early results or that could deliver impact
londonmet.ac.uk
Monitoring & Evaluation Working Group (MEWG)
What did it look at?
londonmet.ac.uk
Procedures• Module Logs• Module evaluation questionnaires• External Examiner reports• PEMS• Course Committees• Course Logs• PEMs• FAMs• Student Engagement• Employer feedback
Data• RPA data• DLHE data• NSS data
Recommendations
New Module and Course Logs with prepopulated data (course logs only).Move from consideration of module performance at an early July PEM to consideration of course performance and modules of concern, at a late July PEM.PEM Chair Report (PCR) of Course and Module logs when final data are available in November.Development of a new Faculty Annual Monitoring Statement (FAMS) that will provide the University with:
a) an evaluation of Faculty performance at undergraduate and postgraduate level, b) confirmation of the implementation of the University’s quality assurance procedures, c) information on validation/review/modification/deletion activity and d) an update on enhancement within the Faculty (QEU0022-FAMS).New Course and Module Modification Deadlines – 2nd February 2015 for modifications that impact on programme planning and two weeks post PEM for modifications that arise from PEM discussionsOnline course evaluation – pilot in May.
londonmet.ac.uk
Monitoring and evaluation process
PerformanceEnhancement
Meeting (PEM) in July, then
PEM Chair Report in November
Faculty UGPG Committee
UniversityUGPG
Committee
Module monitoring data - available from OIE Annual Monitoring Livelink site
Supplementary module performance data- Published by Academic Registry
Marksheets- Component marks produced by Module Leader
External Examiner comments - Module Confirmation Proforma (MCP) - Published by Academic Registry SSB Secretary
Results from module questionnaires - from Faculty records
External Examiner Annual Report ORExternal Examiner written commentary in advance of PEM (where the External Examiner cannot attend the PEM)-available from the EE and QEU Livelink sites-Published by Academic Registry/Assessment on EE Livelink siteCourse (& SSB) monitoring data- Available from OIE Annual Monitoring Livelink site
Course Committee minutes - from Faculty records
Results from course questionnaires - from Faculty records
NSS (previous year data only for July)Supplied by OIE via Faculty senior managers
DLHE data- Supplied by OIE in course monitoring data
PTES data- Available from Research Office
Module Log Completed by Module Leader
Course LogData provided by OIE, Commentary
sections completed by Course Leader
PEM Chair ReportData provided by OIE, Commentary sections
completed by PEM Chair
Faculty Annual Monitoring
Statement (FAMS)Completed by Associate Dean
Scrutiny Forum
Boxes on the left denote information type and the documentation into which it feeds. Information is either supplied by the Office of Institutional Effectiveness (OIE) or must be actively sourced by the person responsible for the production of the core document. All Course Logs, PEM Chairs’ Reports,
FAMS, and PEM minutes need to be provided to QEU who will hold them on Livelink.
Core documentationInformation and source
Annual Monitoring for Collaborative Partners
Module Level Student Feedback
Module Level External Examiner Comment
Course Level External Examiner Comment (aggregated from modules)
Course Level Performance Data (available from the Office of Institutional Effectiveness (OIE) the Academic Registry, and/or the collaborative partner)
Overall summary of good practice/issues to be addressed and annual update on partner activity
University evaluation of collaborative partner performance
QEU Collaborative Performance Report
Module Logs are shared with the Academic Liaison Tutor
Faculty UGPG Committee
Collaborative Course Log produced by
collaborative partner
Faculty update on partner activity
University UGPG Committee
Collaborative Module Log produced by collaborative
partner
Collaborative Annual Monitoring Statement (CAMS) produced by collaborative partner
Partnership Annual Monitoring Statement
(PAMS) produced by Faculty
Collaborative Partner PEMs
Course Level Student Feedback (aggregated from modules)
Other relevant course information e.g. employment statistics (available locally)
Module Level Performance Data
The boxes on the left denote the action and the documentation into which it feeds. Advice is supplied (variously) by the Quality Enhancement Unit (QEU), the Registry and the Office of Institutional Effectiveness (OIE). The boxes on the right denote the forum at which
the annual monitoring documents are considered
Scrutiny DocumentationInformation
londonmet.ac.uk
Core Documentation for Module, Course, Subject and Faculty Monitoring & Evaluation
PEM Chair Report (PCR)
Aggregated and individual data
provided by OIE, Commentary
sections completed by
PEM Chair
Course LogData provided
by OIE, Commentary
sections completed by Course Leader
Module Log Completed by Module Leader
Faculty Annual
Monitoring Statement
(FAMS)Completed
by Associate Dean
londonmet.ac.uk
Review Forums for Module, Course, Subject and Faculty Monitoring & Evaluation
PerformanceEnhancement Meeting (PEM)
Faculty UGPG Committee
UniversityUGPG
Committee
PEM Chair’s Report (PCR)
July PEM November PEM (Chair only)
November Faculty UGPG
December/January University UGPG
londonmet.ac.uk
Monitoring & evaluation activity July 2015
Action Responsibility Date Original date
Academic Registry produce & distribute module data to PEM Chairs and Secretaries, for distribution to Module Leaders. Based on module results from standard publication date 25 June with subsequent updates to 10 July. Module Log pro-formas included.
Academic Registry Fri 10 July Original date proposed was 1 July; extended to allow for late marks
OIE issue data for Course Logs, together with data consolidated at PEM level, plus DLHE and NSS data as available, to PEM Chairs / Secretaries for onward distribution to Course Leaders. OIE also make data available to Academic Registry and QEU. Data based on awards and progression data from standard publication date of 7 July with subsequent updates to 10 July. Course Log proformas included.
OIE Tue 14 July
Original date proposed for Course data was 10 July; extended to
allow for late awards and progression arising from late
marks. Original date for PEM data was 17 July
Module Leaders complete Module Logs & distribute to Course Leaders, PEM Chair/Secretary, and Academic Registry
Module Leaders Wed 15 JulyOriginal date proposed was Wed 8 July
Academic Registry distribute to External Examiners the relevant course data (not Course Logs) and completed Module Logs as available
Registry Fri 17 JulyOriginal date proposed was 15 July
Course Leaders complete Course Logs & distribute to PEM Chair/Secretary and QEU
Course Leaders Fri 17 JulyOriginal date proposed was 17 July
External Examiners unable to attend PEMs feedback any comments to PEM Chair & Secretary, cc Academic Registry
External Examiners with Academic Registry support for queries
Thur 23 JulyOriginal date proposed was 24 July
PEMs meetPEM Chairs, Secretaries, and other members
Mon 20 to Thur 23 JulyOriginal dates proposed 20 - 23 July
PEM minutes produced, approved and distributed to all PEM Members. It is up to PEM Chairs whether they produce also a draft PEM Chairs Report at this stage; the formal requirement for a PCR is in November.
PEM Chairs & PEM Secretaries Friday 31 JulyOriginal date proposed was 30 July
External Examiner Annual Reports (UG modules/courses only)
External Examiners to submit to Academic Registry, who publish via Livelink and circulate to Chairs and designated faculty contacts
Fri 4 SeptOriginal date proposed was 4 Sept
londonmet.ac.uk
Course Evaluation
New process
Consistent practice
Central accessibility
Online evaluation
Reaching all students of London Met
Collaborative students
Asking the question
Why core questions
What questions • National Student Survey (NSS)?
What about local level questions
londonmet.ac.uk
Course Evaluation Pilot 2015
Programme EvaSys
Options Hard Copy Online
What courses
25 of the largest from across the
Faculties
Data Where does it go Who looks at it