Effective Communication of Exam Results: What Should (or Shouldn't) be Included in the Candidate's Score Report Elizabeth A. Witt, Ph.D. American Board.

Effective Communication of Exam Results:

What Should (or Shouldn't) be Included in the Candidate's

Score Report

Elizabeth A. Witt, Ph.D.American Board of Emergency Medicine

2006 Annual Conference

Alexandria, Virginia

Council on Licensure, Enforcement and Regulation

Expect the Unexpected: Are We Clearly Prepared?

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Validity

• Degree to which evidence supports the interpretation and proposed use of test scores


Certified Chefs of America

Examination Results

Exam: South American Cuisine

Date: August 15, 2006

Name: Jane Smith

Result: Pass

Your certificate will be mailed within 10 days.



RESULTS

Summary

The following graph reveals how closely Sample Report's responses match up to those of top performers.

Overall Suitability

Overall, Sample Report exhibits satisfactory levels of key competencies, as compared to top performers. Sample's pattern of responses is similar to those of successful real estate agents, although there are areas of divergence. By targeting training and development to the areas of weakness, Sample could boost performance to a higher level. Please review the Detailed Results below to identify Sample's areas of strengths and weakness from the perspective of a career in real estate sales.

Detailed Assessment

The following graph illustrates Sample Report's areas of strength and weakness. The graph is laid out in terms of critical competencies that are required to successfully perform as a real estate agent. For each competency, Sample Report is evaluated in terms of Sample's current level of development. A high score indicates strength in that particular competency. A low score indicates that a competency may require further development. Please review the information below the graph for an interpretation of the results.

SALES ABILITY

Building Rapport And Acquiring Information

Listening And Comprehension Skills

Managing The Sales Process


What information do examinees want?

Pass/Fail status

Actual score 78

Comparison to passing score

Areas of strength/weakness

Comparison with others


What else do examinees want?

Timely notification

Explanations – graphs, technical terminology

Graphics, visual

presentation

Simplicity


Timely notificationTimely notification

Why do we not just give them what they want?

Instant notification possible with CBTBUT

Not with paper examsAND

Requires relinquishing controlINCLUDING

Elimination of poorly performing items


SimplicitySimplicity


We lose sight of the purpose/audience.

This is not a technical report!


ExplanationsExplanations – graphs, technical terminology


Not always necessary

Familiarity – lose sight of the audience

May require psychometric services


Graphics, visual Graphics, visual

presentationpresentation


Not always necessary

Small staff, small program


Why do we not just give them what they want?Actual scoreActual score – may need explanation

Raw score: number correct, percent correct

Logits

Scaled score

Scores may not be precise away from the cut



Comparison to passing scoreComparison to passing score

– Scores may not be precise except at the cut

– May need explanation (SEM, for example)

– May be self-explanatory(if a simple raw is reported, for

example)



Areas of strength/weaknessAreas of strength/weakness

- Need a subscore for each area

- Scores based on few items are not reliable

- Subscores may not sum to total test score- confusing to examinees



Comparison with othersComparison with others

- Scores are normally criterion-referenced.

- The group testing may differ by examadministration.


Recommendations

DO include:DO include:

Name, exam title, exam date

Pass/Fail Status

An interpretable score

Score required to pass

Same results by section, if appropriate

Simple, clean graphics


Recommendations

DO include:DO include:

Simple, clear explanations for any graphs

or technical terminology

Brief, simple explanation of scoring

Diagnostic information for those who fail

How to request score verification, appeal result


Recommendations

Do NOT include:Do NOT include:

DecimalsMean score = 275.358

Mean score = 275 Excessive detail Jargon

SEM Coefficient Alpha Logit Personal information (SSN, for example)

Teeny, tiny font sizes


RecommendationsCandidate score reports should be:Candidate score reports should be:

Timely

Simple

Clear

Brief

Easy to read and understand



Bottom LineWho is the audience and

what do they need to know?


References/ResourcesAmerican Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME). (1985). Standards for educational and psychological testing. Washington, DC: American Psychological Association

Ryan, J. M. (2006). Practices, issues, and trends in student test score reporting. In S. M. Downing & T. M. Haladyna (Eds.), Handbook of Test Development (pp. 677-710). Mahwah, NJ: Lawrence Erlbaum.

Schroeder, L. L. & Castle, R. Technology and testing: Reporting scores. (Winter 2006). CLEAR Exam Review, 27(1), 8-10.

http://www.ets.org/Media/Products/IDMS/pdf/51588_idmsRpt.pdf

http://www.realestatesimulator.com/pages/sample_reports/Sample3.htm

http://www.visualcertexam.com/screenshots.html


Speaker Contact Information

517-332-4800, ext. [email protected]

Elizabeth A. Witt, Ph.D.American Board of Emergency

Medicine3000 Coolidge Rd.East Lansing, MI 48823

Effective Communication of Exam Results: What Should (or Shouldn't) be Included in the Candidate's Score Report Elizabeth A. Witt, Ph.D. American Board.

Documents

virginia slide

clear annual conference

virginia graphics

virginia simplicity

cut slide

annual conferencealexandria

virginia council

score comparison