Test Assessment and Implementation Rationale

RELTA in Practice

•Test administration

•Scoring processes

- section scores and weighting

- rating processes

•Generation of results

•Test maintenance: reliability

•Compromises: theory vs practice

Overview

Recap: Structure

Listening Speaking

Recap: Delivery

•On-site delivery •Separation of delivery and rating processes•Quality control: score reliability, training, data collation and storage•Minimise workload and human error•Maximise test security

Administration requirements

Administration: Roles

•Administrators - test centre

•Examiners = interlocutors - test centre

•Raters x2 (or 3) – anywhere (local or RMIT)

•RMIT Operations

Administration

RELTA Examining

RELTA Rating

Issuing of results

Test centreroles

RMITroles

Local administration Local examining

RELTA Examining

RELTA Rating

Issuing of results

RMIT delivery

RELTA Examining

RELTA Rating

Issuing of results

Test versions dispatched

RELTA administere

d

Speech files rated (x2 raters)

Speech files uploaded to server

Rater 1 and Rater 2 scores combined and

checked

Speech file dispatched for independent rating

Results issued: Test Centre Regulator

Candidates

Administration

Administration: Processes

1. Register candidates

2. Administer RELTA Listening – in groups

3. Listening tests marked – database

4. Examiner delivers RELTA Speaking

5. Speaking performance recorded

6. Uploaded to RMIT server

7. Raters (1 and 2) login to database and enter scores

8. Rater scores compared for reliability – 3rd rater

9. Results calculated

10. Certificates generated

Administration: SRMS

Scoring Record Management System (SRMS):

Developed to manage administration:• Distributes candidate speaking files to raters• Captures and rater scores• Allows for remote and independent rating• Identifies rater discrepancies • Generates results• Produces certificates• Stores candidate details and speaking files centrally

Online access for: • Administrators (registering of candidates, accessing

results)• Raters (enter scores for RELTA speaking)

ScoringListening(comprehension)

Marks

Speaking

Weighting

Section 1 10 20%

Section 2 10 35%

Section 3 10 45%

Pronunciation Structure Vocabulary Fluency Comprehension Interactions

Section 1

20% 4 4 5 5 N/A 5Section 2

35%

Section 3

45%

Weighted average score

OVERALL SCORES

ICAO Score(lowest overall score reported)

4

Scoring: Speaking

Pronunciation Structure Vocabulary Fluency Comprehension Interactions

Section 1

20% 4 4 5 5 N/A 5Section 2

35% 4 4 4 4 N/A 4

Section 3

45%

Weighted average score

OVERALL SCORES

ICAO Score(lowest overall score reported)

4

Pronunciation Structure Vocabulary Fluency Comprehension Interactions

Section 1

20% 4 4 5 5 N/A 5Section 2

35% 4 4 4 4 N/A 4

Section 3

45% 4 4 4 5 N/A 5

Weighted average score

OVERALL SCORES

ICAO Score(lowest overall score reported)

4

Pronunciation Structure Vocabulary Fluency Comprehension Interactions

Section 1

20% 4 4 5 5 N/A 5Section 2

35% 4 4 4 4 N/A 4

Section 3

45% 4 4 4 5 N/A 5

Weighted average score

4 4 4.2 4.65 22/30 4.65

OVERALL SCORES

ICAO Score(lowest overall score reported)

4

Pronunciation Structure Vocabulary Fluency Comprehension Interactions

Section 1

20%4 5 4 5 5 4 5 5 N/A 5 5

Section 2

35%4 4 4 4 4 4 4 5 N/A 4 4

Section 3

45%4 4 4 4 4 4 5 4 N/A 4 5

Weighted average score

4 4.2 4 4.2 4.2 4 4.65 4.55 22/30 4.2 4.65

Combined overall scores

OVERALL Score (Score reported for ICAO purposes)

Pronunciation Structure Vocabulary Fluency Comprehension Interactions

Section 1

20%4 5 4 5 5 4 5 5 N/A 5 5

Section 2

35%4 4 4 4 4 4 4 5 N/A 4 4

Section 3

45%4 4 4 4 4 4 5 4 N/A 4 5

Weighted average score

4 4.2

4.1

4 4.2

4.1

4.2 4

4.1

4.65 4.55

4.6

22/30 4.2 4.65

4.425

Combined overall scores

OVERALL Score (Score reported for ICAO purposes)

Pronunciation Structure Vocabulary Fluency Comprehension Interactions

Section 1

20%4 5 4 5 5 4 5 5 N/A 5 5

Section 2

35%4 4 4 4 4 4 4 5 N/A 4 4

Section 3

45%4 4 4 4 4 4 5 4 N/A 4 5

Weighted average score

4 4.2

4.1

4 4.2

4.1

4.2 4

4.1

4.65 4.55

4.6

22/30 4.2 4.65

4.425

Combined overall scores 4 4 4 4 5 4OVERALL Score (Score reported for ICAO purposes) 4

Scoring: Rater 1 interface

Enter scores for Sections 1, 2 and 3

Download rating scripts

Listen to candidate sound file

Scoring: Rater 2 interface

Enter scores for Sections 1, 2 and 3

Download rating scripts

Listen to candidate sound file

Admin interface

Calculates overall scores

Determines ICAO Level

Imports scores from R1 and R2

Imports Listening score

Identifies rater discrepancies

Data collation

______________

Rating discrepancies

SRMS: Rating discrepancy

Inter-rater reliability

Inter-rater reliability

Rater re-accreditation

Rater accreditation

Examiner accreditation

Manuals

Summary

Testing for ICAO compliance requires:

Test instruments that are well designed and can be implemented effectively