Experts Respond to Your Questions 2006 Annual ConferenceAlexandria, Virginia Council on Licensure, Enforcement and Regulation Expect the Unexpected: Are.

Experts Respond to Your Questions

2006 Annual Conference

Alexandria, Virginia

Council on Licensure, Enforcement and Regulation

Expect the Unexpected: Are We Clearly Prepared?

Are You Ready?

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Moderator/Coordinator

Chuck FriedmanAssistant Vice President

ACT

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Panelist

David FosterChief Executive OfficerKryterion, Inc.

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Panelist

Ron RodgersPresident

Employment Research InstituteDirector of Measurement Services

Continental Testing Services

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Panelist

Barbara ShowersDirector, Education and

ExaminationsWisconsin Department of Regulation and Licensing

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Speakers

You . . . the audience!!!

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

• Developing and Implementing ADA Accommodations and Preparing for ADA Litigation

Credentialing & Exam Issues Sessions

Early Bird SessionsThursday, September 14

• Psychometrics of Testing with Small Numbers

Friday, September 15

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

• Demystifying Testing: Guide to Managing Exam Programs

• Leveraging the Internet in Exam Development and Delivery

• When the Unexpected Happens: Logistical and Technical Challenges

Credentialing & Exam Issues Sessions

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

• Physical Ability Testing and Practical Examinations: They Fought the Law and the Law Won

• Reporting Examination Results to Stakeholders: Issues and Best Practices

• Test Compromise and Security: Be Prepared

Credentialing & Exam Issues Sessions

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Responding to YOUR Testing Questions

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Test Blueprint

Job

An

alysis

Relia

bility

Scoring

Validity

Passing Score

Item Writing

Standard

Adm

inistra

tion

Item A

nalysis

Item Review

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Test Blueprint

Job

An

alysis

Relia

bility

Scoring

Validity

Passing Score

Item Writing

Standard

Adm

inistra

tion

Item A

nalysis

Item Review

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

How often should my program conduct a job analysis?

Job analysis

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Job analysis required to support testLink content outline to test

referencesSurvey representative sample of

professionSupport test specifications with

job analysis resultsDocument each question with test

references identified and available to all candidates

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Job analysis should be updated when:

References supporting test have changed (new editions, out of print, etc.)

Emerging technology changes practice

Changes in practice redefine critical knowledge & skills needed to protect public & practitioners

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Should my Board or program sponsor allow candidates to review test results including test questions?

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Considerations:

Credibility/defensibility of test process

Content reviews/expert inputVerification measures for

scoringHistory of scoring errors

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Considerations:

Test exposure/security issuesExtended study time with

test contentLimit to questions wrong?Provide correct answers?

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Considerations:

Current law or legislative climateLaw on books?Transparency of governmentFairnessRemedies – is retake enough?

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

How do we increase security for both paper-and-pencil test delivery and computerized test delivery in the face of new technologies for cheating and test theft?

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Be informed; know the threats and risks

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Question

• Do you • know

what • this is?

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Question

• Do you • know

what • this is?

• Two-Way Radio

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Question

•What is the threat here?

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Question

•What is the threat here?

•Web Camera •to capture items

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Question

•What is the threat here?

•Display •to show answers

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Be informed; know the threats and risks

Detecting and monitoring the threats will reduce the risks: Examples include

Better training of site and program staffData forensicsWatch the Web

Protect your items, tests and test results with better access controls, authentication and identification, encryption, etc.

Take decisive action when a breach occurs

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

What is the relationship between the examination process and reciprocity between states? Between countries?

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Between states:Uniform national exams aid

reciprocityState-unique exams can create

barriersAuthority to regulate

professions is given to state government, but reciprocity is gaining importance

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Between countries:Global delivery of exams is

increasingly possibleEvaluations of foreign

credentials have been aided by supplemental exams – ECFMG, ECFVG

CGFNS – will they remain necessary?

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

When it is appropriate to translate or adopt and exam?

See you soon

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Public Health & Safety IssuesAre safety sources (MSDS, etc.) and

product labels available to practitioners in the target language?

Are comparable training materials and textbooks available in English & in target language?

How are target language candidates trained? In what language? With comparable health/safety resources?

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

If translation/adaptation is appropriate:

Are practice standards compromised in any way by training as delivered in target language?

Follow standards & guidelines for adapting test into new language

Review CLEAR Resource Brief on translating or adapting tests

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

What new technologies can we expect in the next five years that will affect the testing process?

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

New security technologies will enable us to prevent fraud and catch test thieves, cheaters and proxy test takers in the act

The Internet combined with test administration technologies will make possible truly ubiquitous high-stakes testing where people work, learn and live

Better measurement with innovations at the item and test levels (performance testing using simulations, for example)

Faster, virtually immediate, test revision and worldwide distribution

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

How should my program deal with an error: for example an incorrect score report, setting of an incorrect passing score, incorrect scoring on an item?

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Guiding PrinciplesAccept responsibility for the

problem and the resolutionApologize promptly and sincerelyRemediate with candidate and

with internal or external systems

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Steps to TakeFirst, investigate who is impacted

and how it happenedMost important in licensing is

pass/fail status and licensing errorsIdentify remedies – Rescore? Free

retest? Other? Request back the license?

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Thank you and Enjoy Lunch!!!!!!!!!