2007 Annual Conference In Pursuit of Testing Excellence The Experts Respond To Your Practical Testing Questions
Dec 17, 2015
2007 Annual Conference
In Pursuit of Testing Excellence
The Experts Respond To Your Practical Testing Questions
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Moderator and Coordinator
Chuck Friedman
Consultant
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
PanelistReid Klion
Chief Science Officer
pan-a TALX Company
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Panelist
Rose McCallin
Director of Examination Services
Colorado Department of Regulatory Agencies
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
PanelistPaul Naylor
Consultant
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
SpeakersYOU………the audience!!!!
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Credentialing and Exam Issues at CLEAR
• Challenges to Quality: Electronic and Candidate Integrity and Its Implications
• Subject Matter Experts: Friends or Foes?
• Getting the Most Out of Your Face-to-Face and Virtual Group Meetings
• Get Ready for Continuous Computer- Based Testing: Examination Quality and Item Security
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Credentialing and Exam Issues at CLEAR
• Best Practice in Job Analysis and Validation of Test Specifications
• Promoting Excellence in Managing ADA Accommodations
• Quality Assurance and Regulatory Excellence
• Vital Implementation Issues in Translation
• Innovations and Trends in Testing
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Responding to YOUR Testing Questions
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
ValidityReliability
Item Analysis
Item Review
Item Writing
Passing ScoreScoring
Job Analysis
Test Blueprint
Standard Administration
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Valid
ity
Reliability
Item
Anal
ysis
Item
Writin
g
Pass
ing
Scor
e
Scorin
g
Test Blueprint
Standard
Administ
ratio
n
Job Analysis Item
Rev
iew
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
What are some alternative ways to conduct a job analysis?
Job analysis
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
What are some alternative methods to conduct a job
analysis?• Full Survey
• Literature Search
• In-Depth Interviews
• Focus Groups
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
What are examples of some advanced, alternative item formats?
What are the benefits and drawbacks of using these formats?
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Innovative Item TypesOptions are Limitless…..
• Performance Testing • Analogue Work Tasks
- Visual Refraction
- Medical Diagnostic Processes
- Plant Operators
• Streaming Video and Audio• On-Screen Manipulables
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
…..but Reality Constrains
• Pragmatics– Usability– Logistics– Security ~ Salience of Rich Media– Development Costs Often High
What are we getting for our money?
• Psychometric Considerations Measures More Difficult to Develop than
Methods
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
If the testing process is disrupted, (lost answer sheets/data, power failure, computer
crash), what should my Board tell the candidate and what action should we take?
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Stuff Happens…• Force majeure (“a greater force”)
Tornado Strikes Brooklyn—August 2007
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
What Could Happen? • Natural Disaster
• Power/Internet Failure
• Local Issue/Emergency
• Many Events Unavoidable
Can Only Plan to Mitigate Impact
• Policy to Deal with Issues
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Mitigation of Computer-Based
Issues• Test Provider Issues- Redundant Systems- Backups
• End-User (or Test Taker) Problems- Backups of Test Responses
- Transmission to Servers After Each Screen Movement
-Ability to Resume Test Session
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
What feedback should be provided to candidates on score reports and results?
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
What feedback should be provided to candidates on score reports and results?• Scoring Methods
• Content Domain Scores– Interpretations
• Reliability Issues– Cautions
• Aggregate Scores
• How to Reapply
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Should my Board or program sponsor allow candidates to review test results including
test questions?
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Disclosure May Threaten Validity of Interpretations on Future Exams
Candidate Is Entitled to Fair & Reasonable Appeals Process
AERA, APA, NCME Standard 8.2
AERA, APA, NCME Standard 8.13
Allow Candidate Review?
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Discourage Over-Interpretation of Information Subject to Considerable Error
Candidate Interprets P/F Status on Basis of Isolated Items
AERA, APA, NCME Standard 1.10
AERA, APA, NCME Standard 1.14
Allow Candidate Review?
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Allow Candidate Review?
• Candidate Right To Fair, Meaningful Appeals Process– Some Programs Allow Limited Access to
Certain Test Items and At High Costs• Only Review M/C Items• Cost per M/C Item = $100+• Process for Review Not Realistic• If Credit Granted, “Not Accredited”
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Allow Candidate Review?
• How Minimize Validity Threats of Exposure & Provide Candidates Meaningful Appeals Process?– Independent Review by Qualified 3rd Party
• Non-Disclosure Contract• Supervised Environment• Authority to Challenge
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
What should I tell a candidate who failed the examination by one point?
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
• Regulatory Exams: Aims of Measurement & Validity of Interpretations
Maximize Maximize Accuracy of P/F Accuracy of P/F
Classification Classification DecisionsDecisions
Failed by “1 Point”?
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Failed by “1 Point”? • Validity of Interpretations of Performance
on Regulatory Exams– Maximize Consist, Accurate Classification
Decisions– Consequences of Errors of Measurement
• Candidates Whose Performance Is Far Above or Far Below P/F Decision Point
• Candidates Whose Performance Is Close to P/F Decision Point (False Positives & False Negatives)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Failed by “1 Point”? Provide Results Reports that Encourage Interpretations Warranted by Precision of
Classification Consistency
0
5
10
15
20
25
30
35
A. B. C. D. E. F. G. H.
0%
20%
40%
60%
80%
100%
# Items on Exam N Correct %Msty by Part
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Failed by “1 Point”? Exam Part
Exam Items
% of Test
N Correct
% Msty
A. 31 21% 24 77% B. 15 10% 11 73% C. 21 14% 16 76% D. 15 10% 11 73% E. 9 6% 5 56% F. 30 21% 22 73% G. 10 7% 6 60% H. 14 10% 11 79%
Totals 145 100% 106 73%
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
What are the practicalities and economics of maintaining a small licensing program?
Quality
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
PIA & Result
Post-Audits
Pre
limn
. T
ry O
uts
Cut S
core
s
ItemDevel.
Test
Specs.
Entry- Level Reqmts.
Small Programs: Practicalities/Economi
cs
JA for
Reg
Strong As Weakest Lnk
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Small Programs: Practicalities &
Economics• Measurement Models: CTT or Rasch
• Problems of Dimensionality– Carefully Consider Logic of Test
Specifications, Content Dimensions
• Best Practices– Good Practice Analysis, Test Specs. &
Appropriate, Well-Formulated Test Items & Try-Outs
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Validity As A Unified Concept
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
What are the benefits and drawbacks of computer based testing as opposed to paper
and pencil delivery?
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Computerized Assessment - Why?
Convenience- Test Wherever There is a Computer
- Logistics Vastly Simplified
- Management of Errata
- Possibility of Continuous Testing
- Immediate and Accurate Scoring “Hanging” Chads OpScan Issues
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Facilitation of Data Management
• Data in Electronic Format
• Web-Based Data = Immediate Availability
- Centralized Management of Assessment Programs- Access to Array of Psychometric Data
(e.g., response latencies, test taker activity)
- Facilitate Integration with Other IT Systems
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Candidate Perception• Candidate/Member Preference
- Familiarity/Perceived Ease of Use
- Demographic Issues
• Organizational Appearance
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Why Continue to Use Paper-Pencil?
• Programs That Must Test Massive Numbers Concurrently in One Locality
• Where Economics Dictate– Very Small Programs– Program Costs “Hidden”
• “Immovable” Impediments
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
We are starting a new licensing program which does not allow grandfathering. What issues need to be considered?
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
We are starting a new licensing program which does not allow grandfathering. What issues
need to be considered?• Security
• SMEs
• Limited Disclosure
• Portfolio Review
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
What are the issues and potential pitfalls in adapting/translating our examination?
See you soon!
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
What are the issues and potential pitfalls in
adapting/translating our examination?• Adaptation
– Process– Beyond Translation
• Back Translation– Intuitively Valid
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Adaptation/Translation• Translators
– Language Skills– Cultural Skills– Subject Matter Knowledge– Trained in Item Writing/Review
• Field Testing
• Ron Hambleton/Stephan Sireci
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
What is the relationship between the examination process and reciprocity
between states? Between countries?
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Relationship Between Exam Process & Mobility
• Reciprocity– Jurisdictions Agree to Recognize Each
Other’s Practitioners without Require to Demonstrate Competence
– License May or May Not Be Required from Other Jurisdiction
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Relationship Between Exam Process & Mobility
• Endorsement– Jurisdiction Recognizes License from Another
Jurisdiction– Licensee Relieved of Full Burden of Obtaining
License in New Jurisdiction
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Relationship Between Exam Process & Mobility
• Major Barriers – Standards for Entry to Practice Are Not Universally
Focused On Public Protection– Professional Interests, Scope-of-Practice Protection,
etc. = Barriers to Entry
• Change Agents – Mobility– Labor Supply/Demand– Communications Innovations
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
What is the impact of test industry consolidation (mergers, acquisitions) and
privatization for state licensing programs?
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Consolidation & Privatization
• Media Landscape: Who Owns What?
• Testing Industry Legacies: Then & Now
• Licensure Exam Industry: Impact of Privatization by Regulatory Authorities
• Independent Auditing Mechanisms
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Consolidation & Privatization
• Media Landscape: Who Owns What? (Columbia Journalism Review: http://www.cjr.org/resources )
– Three of Top 5 “Print” Corporations Affiliated with Testing Companies
• Reed Elsevier: Harcourt (The Psychological Corp.)
• NCS Pearson: Pearson Education (Pearson Assessments–Pearson VUE, Promissor, Riverside)
• McGraw-Hill: CTB
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Testing Industry Legacies
“Big Four” Educational Marketer (2001) http://www.pbs.org/
Harcourt
CTB/McGraw Hill
Riverside Publishing
NCS-Pearson
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Testing Industry Legacies: Then & Now
Reed Elsevier2001
007Pearson
Buying Harcourt(Pending SEC)
Harcourt 19051921: TPC
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Testing Industry Legacies: Then & Now
CTB/McGraw-Hill1926
2007CTB/McGraw-Hill
McGraw-Hill 1909
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Testing Industry Legacies: Then & Now
Houghton Mifflin1880
2006Pearson
Buys Riverside
Riverside Press1852
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Testing Industry Legacies: Then & Now
Pearson Education2000
2006Pearson Assessments
NCS1962
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Testing Industry Legacies: Then & Now
Thomson Prometric (CapStar)2000-2007
2007ETS Buys
Thomson Prometric
Drake 1980s-1997Sylvan 1980s-1999
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Testing Industry Today
CASTLEWW
Kry-terion
CTS
Caveon
ACT
PCS SMT
CTB
AMP
CollegeBoard
Comira
ETSProme-
tric
Pear-son
More!
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Privatization & Independent Auditing
MechanismsMarket Demand(Regulatory Agencies)
Who Is EnforcingCompliance with the Standards?
Suppliers(Testing Industry)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
How do we combat threats to security for both paper-and-pencil test delivery and CBT
in the face of new technology?
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Security• Any Exposure of Item is a Security Risk
• Issues Common Across Delivery Modalities– Content Management Handling– Item Presentation and Usage – Candidate Item Review Policies– Need for Good Proctoring
• Need to Analyze Risk Factors in Program Population
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Item Security in CBT• SSL Encryption
• Randomized Forms
• Locked-Down Browsers
• Candidate Education (May be Particularly Useful for Certain Certification Programs…)
• Signed Attestations
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Besides test delivery, how can technology impact licensing programs and state
Boards?
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Impact of Technology• Data Are Electronic
• Web 2.0 is Real (Internet as Platform)
- Immediate Availability of Assessment Data
- Centralized Management of Assessment Programs
- Facilitate Integration of Test Data with Other IT Systems and Processes
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
What is on the horizon in the next five year that will affect the testing process for
licensure and certification?
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
What is on the horizon in the next five years that will affect
the testing process for licensure and certification? • Increased Dependence on Accreditation
• Increased Consolidation in Test Industry
• P&P Hangin’ In There
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Speaker Contact Information
• Rose C. McCallin, Ph.D.– Director, Office of Examination Services– Colorado Department of Regulatory
Agencies– Division of Registrations
• 1560 Broadway, Suite 880• Denver, CO 80202• 303.894.7710• [email protected]
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Speaker Contact InformationReid E. Klion, PhD
Chief Science Officer
pan—A TALX Corporation
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Speaker Contact Information
• Paul D Naylor, Ph.D.
• Psychometric Consultant
• (919) 730-1593