5/19/2009 1 Advanced Techniques: Developing and Validating Pre- E l tT t BCGi Institute for Workforce Development Employment Tests © Copyright 2009 Biddle Consulting Group, Inc. All Rights Reserved Welcome to BCGi The webinar will begin momentarily
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
5/19/2009
1
Advanced Techniques:
Developing and Validating Pre-E l t T t
BCGiInstitute for Workforce Development
Employment Tests
© Copyright 2009Biddle Consulting Group, Inc.All Rights Reserved
Welcome to BCGi
The webinar will begin momentarily
5/19/2009
2
Visit BCGi Onlinewww.BCGinstitute.org
While you are waiting for the webinar to beginbegin, Don’t forget to check out our other training
opportunities through the BCGi website.
Join our community by signing up (its free) and we will notify you of our upcoming free t i i t ll th i f ti ftraining events as well as other information of value to the HR community.
HRCI Credit
BCG is an HRCI Preferred Provider
CE Credits are available for attending thisCE Credits are available for attending this webinar
Only those who remain with us for at least 80% of the webinar will be eligeable to receive the HRCI training completion form f CE b i ifor CE submission
5/19/2009
3
This Webinar Will Be Recorded
BCGi will record this webinar so that members can watch it at their convenience
After the webinar we will have approximately 10 to 15 minutes for questions To submit a question, use the GoToMeeting
control that should appear on your desktopcontrol that should appear on your desktop
Advanced Techniques:
Developing and Validating Pre-E l t T t
BCGiInstitute for Workforce Development
Employment Tests
© Copyright 2009Biddle Consulting Group, Inc.All Rights Reserved
5/19/2009
4
Contact Information
Jim Higgins Ed DJim Higgins, Ed.D.Biddle Consulting Group, Inc.
193 Blue Ravine, Ste. 270
Folsom, CA 95630
1-800-999-0438
(916) 204-1749 (Office)
www.BCGinstitute.com
Agenda
1. Legal Framework For Assessment
2. Approaches to Validation
3. Job Analysis
4. Content ValidationCo te t a dat o
5. Test Usage
6. Developing a Test
5/19/2009
5
Evolution Of The Concept of Adverse Impact
1964 Civil Rights Act (Title VII)
Prohibits employment discrimination on the basis of race, color, sex, national origin, and religion.
Prohibits retaliation for opposing discrimination, filing a complaint, or participating in a related proceeding.
5/19/2009
6
Griggs v Duke Power (1971)
Background High School Diploma
Off Th Sh lf I t lli T t Off-The-Shelf Intelligence Test Adverse Impact against African Americans The job in question did not appear to be related to the
need for a diploma or intelligence test scores The Court’s Findings:
Duke Power had to demonstrate a “business necessity” for its assessment/screening proceduresfor its assessment/screening procedures If an employment practice that operates to exclude
members of protected groups cannot be shown to be related to job performance, it is prohibited, notwithstanding the employer’s lack of discriminatory intent.
California’s Technical Advisory Committee on Testing 1971-1972
Partly in response to Griggs Partly in response to the lack of clarity on “how” to
determine whether adverse impact existsdetermine whether adverse impact exists California brought together 32 specialists from the
labor, employment and technical fields Goal: To craft a strategy for determining when
adverse impact actually exists 80% rule was born out of a compromise (there is
nothing magical or particularly statistical about it) and i t d d t b ki d f i k th h ld t t dwas intended to be a kind of quick threshold test and
a measure of “practical significance” If 80% rule violated, then statistical techniques
employed Expressed that statistical tests are the gold standard
5/19/2009
7
Uniform Guidelines on Employee Selection Procedures (1978)
Sections 4D and 16B Clarify Adverse Impactp Defines adverse impact as “a substantially
different rate of selection in hiring, promotion or other employment decision which works to the disadvantage of members of a race, sex or ethnic group.” (Source: uniformguidelines.com)
Outlines strategies for validating pre-employment tests.
Albermarle Paper Co. v Moody (1975)
U.S. Supreme Court reinforced the concept of “job relatedness”concept of job relatedness
“Job relatedness cannot be proven through vague and unsubstantiated hearsay.”
You must follow validation standards.
This is why we have a structured job analysis process.
5/19/2009
8
Connecticut v. Teal (1982)
An employer may be found guilty of discrimination if any part of the selectiondiscrimination if any part of the selection system adversely impacts protected group members, even if the “bottom line” rate itself shows no adverse impact.
Guardians Assoc. v. New York Civil Service Commission (1983)
Compliance is more than just having a job analysis The job analysis must bejob analysis. The job analysis must be solid, include clearly written task and KSAPCs, and document the linkage between tasks, KSAPCs and job performance.
5/19/2009
9
WARNING!
OFCCP continues to ramp-up auditing and enforcement
How Are PPTs Challenged?
Practice,Procedure,Test (PPT)
AdverseImpact?
YES NO
Is the PPTValid? END
“or”“or”
YES NO
Alternative Employment
Practice?
NODefendant Prevails
YESPlaintiff Prevails
Plaintiff Prevails
5/19/2009
10
Review To This Point
Measure job related KSAPCs
Legally Defensible
PPTs
Based on Solid Links to the Job
Alternative Procedures Considered
Adverse Impact
“Can” be OK
What is Validation?
Traditionally, it is making sure a selection procedure (i.e., test) measures what it is p ( )designed to measure
In a practical sense, a selection procedure is valid if it can be proven by an employer that it is “…job related and consistent with business necessity ”business necessity.
5/19/2009
11
Great Job Performance
Validity
Reliable…but NOT Valid
Consistently hiring the wrong
people
5/19/2009
12
Neither Reliable or Valid
Hit or Miss…
Better Productivity
Reduced Turnover &
Absenteeism
Reliable AND Valid
Less Time to Train
More Satisfied Clients
Less LiabilityEmployee
Satisfaction Increased
5/19/2009
13
Guides Related To Validation Techniques
Joint Standards
Principles (SIOP)
Uniform Guidelines
Court Precedence
Validity
Uniform Guidelines Style Validation…
Content Validity
Section14C Section14C
Criterion Validity
Section 14B
Construct Validity
S ti 14D Section 14D
5/19/2009
14
Content Validity
Test ItemsItems
KSAs
Duties
Criterion Validity
JobJob Requirements
Job Test ScoreJob Performance
5/19/2009
15
Construct Validity
District Manager
LeadershipA term naming an
d l ig
Position
Administrative Duties
Effective Manager
underlying “construct” that has been hypothesized as being important to serving as an effective manager. The construct itself is an artificial creation based solely of
Leadership
based solely of statistical evidence of its existence and relevance.
Content Validation: Nuts and Bolts
Identify Critical
Content of The Job
Develop Assessment Procedures
Verify Functioning
of The Assessment
5/19/2009
16
How To Conduct A Job Analysis
Review Historical Documents
Desk
Job Analysis Survey
Job DutiesAdministerDesk
Audits/Observations
Individual and/or Group Interviews
KSAPCs
Physical Abilities
Administer survey to
job experts
Important/Critical Job Duties
Analyze Data
Duties
Important/Critical KSAPCs
Required at Entry
Job Analysis: Duty Ratings
Frequency
Importance* p
Guidelines , 14.C[1,2,4]
Best Worker* (for possible ranking)
Guidelines , 14.C[9]
Fundamental
Americans with Disabilities Act
Assignable
Americans with Disabilities Act
5/19/2009
17
Job Analysis: KSAPC Ratings
Frequency
Importance*
Guidelines, 14.C[1,2,4] Best Worker
Guidelines, 14.C[9] Minimum v. Desirable Qualifications
Guidelines, 14.C[4] Level Needed Upon Entry*y
Guidelines, 5.F, 14.C[1] Mastery Level
knowledge only
Selection KSAs For Assessment
Only certain KSAs can be tested…
Only “concrete” KSAs can be tested under Only concrete KSAs can be tested under content validity (cannot test abstract KSAs)
KSAs need to be “operationally defined”
Important/Critical for the job
Needed “Day One” on the job
/ S Duty/KSA Linkages
5/19/2009
18
A Scale of Defensibility
LowHigh
Rely on job descriptions, class specifications with no/old job analyses
Full and comprehensive job analysis including the input of current job experts and no/old job analyses,
guesses about what is required on the job, no links between duties, KSAPCs, and test items.
psupervisors, quantitative data and analyses, links between critical job duties and KSAPCs required at entry, and test items.
Selecting KSAs For Assessment
WORK BEHAVIOR PPT REQUIREMENTS
KSA PPT REQUIREMENTS
PPT t b t ti l KSAs must be operationally defined PPT must be a representative sample of the behavior measured, or of work product of the job, so…
The manner of the PPT should closely approximate the work situation.
The setting of the PPT should closely approximate the work situation.
KSAs must be operationally defined. Knowledge = that body of learned
information which is 1) used in and 2) is a necessary prerequisite for observable aspects of work behavior. Skills or abilities = in terms of observable aspects of work behavior.
Show PPT measures the KSA. Show PPT is a representative sample of
KSA. Show KSA is used in the performance of a
The level of complexity of the PPT should closely approximate the work situation.
Show KSA is used in the performance of a critical or important work behavior(s).
KSA is a necessary prerequisite to performance of critical or important work behavior(s).
For skills or abilities, the skill or ability should either closely approximate an observable work behavior, or its product should closely approximate an observable work product.
5/19/2009
19
Developing A PPT
MQ
Critical and Required at Entry
Simulation
MQT&E
KSAPC
MC Test
Interview
Developing An Exam Plan
KSAPC (Critical and required at entry)
Structured Interview
Written Test Job Simulation
Abilit t d d tl X XX XXAbility to read and correctlyinterpret formal results of legal proceedings for the purpose of summarizing for management.
X XX XX
Ability to use basic math to calculate means and standard deviations in order t l t d i t
X XX XX
to evaluate adverse impact in pre-employment tests.
Ability to lead small groups of employees as facilitator of workgroups charged with solving business-related problems.
X XX
5/19/2009
20
How You Plan To Use Your Test Is Critical!
Pass/Fail Cutoffs:
“Normal Expectations of Acceptable Proficiency in the Workplace” (Guidelines, 5H)
Modified Angoff (U.S. v. South Carolina, USSC) Banding:
Substantially Equally Qualified Applicants
Statistically Driven (use Std. Error of Difference) Ranking:
Is there adequate score dispersion?
D th t t h hi h li bilit ? ( > 85) Does the test have high reliability? (e.g., >.85)
Is the KSA performance differentiating? Weighted/combined with other tests
How are the weights related to the job
Do they come from the job analysis or SME ratings?
How Tests Can Be Used
Ranking assumes one applicant is reliably more qualified than the other
Applicant Score
Tom 100Stacy 100
Banding considers the unreliability of the test battery and “ties” applicants
Bob 100Frank 100Julie 99
Rozanne 99Mark 98Luke 98Henry 97Paul 97Peter 96
Pass/fail cutoffs treat all applicants as either “qualified” or “not qualified”
Weighting/combining test scores can be done using “compensatory” or using cutoff on each test then weighting results
Rebecca 96Alyssa 95
Matthew 94John 93
Annette 93Ray 92
Thomas 91Julissa 90
5/19/2009
21
Characteristics of Pass/Fail Cut Scores
NOT TYPICALLY DEFENSIBLE WHEN: Using an arbitrary cutoff (e.g., 70%) Using applicant scores to benchmark (e.g., setting cutoff scores at Using applicant scores to benchmark (e.g., setting cutoff scores at
mean-SD of applicant scores) TYPICALLY DEFENSIBLE WHEN:
Consider “Normal expectations of acceptable proficiency in the workplace” (Guidelines, 5H)
Usually requires SME-level data or ratings Tied to job performance
FACTORS TO CONSIDER: Is the test supported by content validity information or criterion-
related information?related information? How critical are the KSAs measured? Does the test measure “baseline” or “differentiating” KSAs? How would current incumbents perform on this test?
Comparison Score Uses
Factor Ranking Banding Pass/Fail CutoffsCutoffs
Validation Requirements
High Moderate Low
Adverse Impact High Moderate Low
Defensibility Low High High
Litigation "Red Flag" High Moderate Low
Utility High Moderate Low
Cost Low Moderate HighCost Low Moderate High
Applicant Flow Restrictive/
Controllable
Moderate/
Controllable
High
Development Time Low Moderate High
Reliability Requirements
High Moderate Low
# Item Requirements High Moderate Low
5/19/2009
22
Creating Your Test
Rely on Job Experts in the creation of test content whenever possible. Review results of job analysis
Train them on techniques and item formats
Begin by documenting and discussing critical incidents
Link each item to critical KSASPCs that are required at entry to the job. Use job materials whenever possible.
Evaluate each item for sources of potential bias or problems.
Identify a “preliminary passing score”.
Preliminary Pass Point
For each item, identify “the percent of minimally qualified applicants” that would be likely to answer the item correctlythe item correctly. “Suppose you have 100 people who could do the
job—but just barely. How many of them do you think would answer this item correctly?
Average the ratings across each item. The result will be the score to be expected by a
minimally acceptable candidate (MAC) on the test—the preliminary cut score!the preliminary cut score!
This number is “modified” based on data collected during the first administration of the test to account for test unreliability and actual test-taker performance.
5/19/2009
23
Optimizing Your Test
Refinement! Identify “malfunctioning” items.
T diffi lt Too difficult Too easy Distracter Analysis Low or negative item-total correlations Review “Alpha if item deleted”
Examine Standardized Mean Group Differences and/or statistically significant differences by subgroup
Identify “DIF” Identify DIF Look for items with stereotypical, biased or loaded content Item sensitivity review panel
Refine or modify pass point based on conditional standard error of measurement
Related Documents
