Understanding the work of the HKEAA: Quality Assurance

1

Understanding the work of Understanding the work of the HKEAA: Quality Assurancethe HKEAA: Quality Assurance

C. S. Tong C. S. Tong Secretary GeneralSecretary General

7 June 20137 June 2013

About HKEAA Established in 1977

(independent statutory body) 400 staff in 7 office

locations 8,000 exam personnel

(seasonal)

Mission: To provide valid, reliable and equitable examinations and assessment services in a professional, innovative, efficient and effective manner.

Quality Assurance

How are examination papers set?

Curriculum and Assessment (C&A) Guides Subject Committees to oversee the

continual development of each subject Moderation committees to set papers with

rigorous security and quality control mechanism

Moderation committee members are appointed on an annual basis subject to various security requirements

Who are Moderation Committee members?

Subject Expert/

University Academics

HKEAA Subject

Managers

Open nomination for membership from schools as well as from the relevant Subject Committee (current S5 or S6 teachers not eligible)

Secondary School

Teachers

6

How are scripts handled? All scripts are scanned and then archived for

safe-keeping: over 10,000,000 pages scanned for Onscreen Marking

How are Written Papers marked?

Marking schemeMarking scheme Appointment of Markers

Appointment of Markers

Sample scripts selection

Sample scripts selection

Standardisation meeting

Standardisation meeting

Markers’ meeting

Markers’ meeting

Allocation of scripts to markers

Allocation of scripts to markers

MarkingMarking CheckmarkingCheckmarking Statistical analysis of marks

Statistical analysis of marks

8

How are Written Papers marked?

OSM facilitates real time, continuous monitoring of the quality of the marking:

Marking by questions to enhance consistency in applying the marking criteria

Double marking for open-ended questions Use of control scripts to validate marking

consistency Scripts are selected for checkmarking by

experienced Chief/Assistant Examiners

Where are the Assessment Centres?Where are the Assessment Centres?

Sheung Shui:185Sheung

Shui:185

Tsuen Wan:163Tsuen Wan:163

Lai King:376Lai King:376

Sha Tin Wai:92Sha Tin Wai:92

San Po Kong:73San Po Kong:73

Fortress Hill:382

Fortress Hill:382

Tai Hing:100Tai Hing:100

Tseung Kwan O:21Tseung Kwan O:21

About 1,400

workstations

Facilities at the Assessment Centres

10Lounge/Pantry Conference Room

WorkstationsReception Area Control Room

OSM Procedures

1st checkmarking

2nd checkmarking

Markers’ meeting

satisfactorymarking

satisfactorymarking

Complete

Training scripts

Qualifying scriptsReference scriptsNormal scripts Control scripts

What are the advantages of OSM?

Security Data availability Efficiency Accuracy Marking quality

Markers’ meeting to clarify and finalize marking scheme

Provision of Reference Scripts to illustrate the marking requirements and standard

Training Scripts & Qualifying Scripts to ensure quality of marking before markers start marking ‘real’ scripts

How does the Authority monitor Quality of Marking?

Monitoring Quality of Marking

Selected scripts (including control scripts etc) have been discussed by a team including the Chief Examiner and awarded standardized scores

Control scripts are randomly distributed to markers to ensure consistency of marking standards

Monitoring Quality of Marking Marker’s score can deviate from the

standardized score to within a specified tolerance

When the deviation exceeds the tolerance level, the marker will be contacted for clarification of marking scheme/ further training and qualifying/ or stopped from further marking and problematic scripts can be voided and redistributed to other markers for marking as necessary

Standardized Score

Marker’s Score

Marker’s score agrees with Standardized Score

Tolerance level

Illustration

Standardized Score

Marker’s Score

Marker’s score much higher

than standardized

score

Marker’s score much lower than

standardized score

Marker’s score agrees with Standardized Score

Tolerance level

Illustration

Standardized Score

Marker’s Score

Marker’s score much higher

than standardized

score

Marker’s score much lower than

standardized score

Marker’s score agrees with Standardized Score

Tolerance level

Illustration

Monitoring Quality of Marking

Chief / Assistant Examiners will check each marker’s marking statistics

(including marking speed, mark distribution, score deviation from control scripts etc)

scrutinize markers’ marking records and select scripts for checkmarking

contact markers and further clarify the marking scheme if necessary

19

Why Double Marking? To address the issue of inter-rater

variability for open ended questions, double marking may be adopted

Feasibility study carried out to assess the cost benefit between such factors as Marking reliability Markers’ standardization Increased marking time Marking cost

Marker 3Chief Examiner

or Assistant Examiner

Discrepancy within range

The discrepancy among marks is still out of range

Candidate’s mark

Discrepancy within range

Big discrepancy between the two marks

Candidate’s mark

Candidate’s mark

Marker 1

Marker 2

Double Marking: how?Double Marking: how?

The Candidate’s final mark will be the sum of the two highest closest marks

Post examination reviewExternal vetting of examination papersPost examination reviewInternal audit & risk monitoringExamination Paper Quality Audit (EPQA)Internal post-mortem meetings

SBAMarking processMechanisms for handling irregularities, complaints and queriesStandards setting and maintenanceGrading proceduresQuality check of examination dataFeedback to schools

Setting of examination papersCurriculum and Assessment Guide & Assessment FrameworkExamination paper development mechanismConfidentiality and security measures

Conduct of examinationsExamination regulationsPublic Examinations Information Centre (PEIC)Public Examinations Communication and Support System (PECSS)Oral Recording System (ORS)On-site checking & auditsExamination logistics & security measures

HKDSE Recognition

Safeguarding of Standards

Compliance & Quality of

Delivery

Continuous Improvement

Processing & reporting of results

How to ensure overall Quality?

Vetting of Examination papers

Different subjects are selected annually for vetting and peer review

Vetting reports provide useful feedback for subject managers and Subject Committees

Reports and responses from the vetting exercise as well as reports on examination irregularities are submitted to the Public Examinations Board for discussion

24

Discrepancy CheckingScripts are sampled for cross checking at various stages of marking and grading as part of our quality assurance framework:

Randomly selected candidates and all their scripts extracted for manual checking

Scripts of candidates with large discrepancies in the levels obtained in different subjects

Scripts of candidates with large discrepancies in component levels (eg over 1000 scripts from Chinese Language checked in 2012)

25

Discrepancy CheckingPerformance at the school level are cross-checked across different core subjects: eg LS vs Chin 圖表標題 y = 0.9663x + 9.008

R2 = 0.9337

0

100

200

300

400

500

600

0 100 200 300 400 500 600

L

(L)線性

Outlier schools to be manually checked

26

Discrepancy Checking

Math vs English圖表標題 y = 0.9434x + 15.1

R2 = 0.8901

0

100

200

300

400

500

600

0 100 200 300 400 500 600

M

(M)線性

Assessment Quality-Assurance Platform

Based on Rasch Model Item-level and paper-level analysis to

ensure quality: Item difficulties Discrimination power Item reliability Paper reliability (internal consistency)

Item Level

AQP 評核素質保證平台

Paper Level

AQP 評核素質保證平台

Supporting Quality Assessment at Schools

User-friendly version of AQP is available for schools to support Assessment for Learning

Student-Level

AQP 評核素質保證平台

Quality Assessment Management

International Recognition

36

International Recognition

Over 150 overseas institutions responded to our survey proforma

70 institutions from the Mainland accept students from HK on the basis of HKDSE

149 institutions from Taiwan accept students from HK on the basis of HKDSE

37

International Recognition ComparisonInternational Recognition ComparisonInternational Recognition ComparisonInternational Recognition Comparison

UCAS Tariff HKDSE Advanced Placement

(AP)GCE Advanced Level

(GCE AL)International

Baccalaureate (IB)5**

140     A*  130 5*     7120 5 5 A  110       6100     B  90   4    80 4   C 570        60   3 D  50       440 3   E  30        20       3

145

香 港 中 學 文 憑 考 試

一紙文憑 一紙文憑 世界通行世界通行

Delivering HKDSE with PrideDelivering HKDSE with Pride香港中學文憑 眾志昂首同行 香港中學文憑 眾志昂首同行

Thank You!