Schmidt & Hunter Approach to r Artifact Corrections.

Schmidt & Hunter Approach to r

Artifact Corrections

Statistical Artifacts

• Extraneous factors that influence observed effect– Sampling error*– Reliability*– Range restriction*– Computational error– Dichotomization of variables

*Addressed in the analysis

Psychometric Meta-Analysis

Disattenuation for reliability

yyxx

xyC

rr

rr

xx

xyC

r

rr

yy

xyC

r

rr

Correction for both

Correction for IV

Correction for DV

Suppose rxy = .30, rxx = ryy = .80. Then:

375.80.

30.2

yyxx

xyC

rr

rr 33541.

8.

3.

xx

xyC

r

rr

Range Restriction/Enhancement

These are examples of direct RR.

Direct Range Restriction/enhancement

1)1( 22

xyX

xyXC

rU

rUr

edSDrestrict

ctedSDunrestriU X

Suppose rxy = .33, SD1=12, SD2 = 20. Then:

67.112

20

edSDrestrict

ctedSDunrestriU X

50.1)33)(.167.1(

)33(.67.1

1)1( 2222

xyX

xyXC

rU

rUr

Can also invert by uX = 1/UX

Indirect RR

xxir

xxar

Reliability of IV in restricted sample (job incumbents in I/O validation study).

Reliability of IV in unrestricted sample (job applicants in I/O validation study).

T

TT S

su Ratio of SD of true scores; analogous to uX.

)1(1 2xxiXxxa rur

2/12 )1(

xxa

xxaXT r

ruu

You will need uT AND rxxi for INDIRECT range restriction correction.

You will need rxxa for DIRECT range restriction correction.

)1(1 2xxaXxxi rUr

Meta-Analysis of corrected r

• If information is available can correct r for each study

• Compute M-A on the corrected values

• Can also be done with assumed distributions, but I don’t recommend it.

Steps (1)Record data (N, r, artifact values rxx, etc.)

Compute the corrected correlation for each study:

a

rr o

C

Note ro is observed and rC is corrected.

If there is only 1 kind of artifact, disattenuation is simple:

Where a is the disattenuation factor.

If there is range restriction, things are tricky. If INDIRECT range restriction, then use Ut instead of Ux and disattenuate for reliability before adjusting for range restriction. Use reliabilities from the restricted group.

If DIRECT range restriction, adjust for ryy, then range restriction, then rxx, but rxxa, the reliability in the unrestricted group.

Steps (1b)

Compute sampling variance of uncorrected r:

)1/(]1[)( 22 ioio NreVar

Note this is sampling variance for one study.

For each study, compute compound attenuation factor:

C

o

r

rA

Steps (2)

Compute sampling variance of disattenuated r:

2/)( AeVarev o

If there is range restriction, then do the following 2 steps. Compute adjustment for range restriction:

]1)1/[(1 22 oXrr rUa

Adjust sampling variance of disattenuated r:evave rr 2

Compute weights:

2iii ANw Note A is the compound attenuation

factor.

Steps (3)Compute the weighted mean:

i

CiC w

rwr i

Compute the weighted variance:

i

CCiC w

rrwrVar i

2][)(

Compute average corrected r sampling error:

i

ii

w

vewveAve )(

Compute variance of rho: )()(ˆ)( 2 veAverVarVar C

Psychometric M-A data

Study Ni r rxxi ryy Ux

1 200 .20 .90 .80 1.5

2 100 .20 .80 .82 1.5

3 150 .40 .85 .88 1.0

4 80 .40 .85 .90 1.2

Mean 132.5 .30 .85 .85 1.3

We’ve already done the bare-bones analysis of these data. Now we’ll analyze 3 ways: (1) just criterion reliability, (2) all artifacts with INDIRECT RR, (3) all artifacts DIRECT rr.

Correct ryy only (1)

Study 1 r = .20, rxx = .90, ryy = .80, Ux = 1.5

Disattenuation ryy : rC = .2/sqrt(.8) = .223607. Compound attenuation factor A = .20/.223607 = .894.

Suppose we only wish to correct for criterion unreliability.

Correct ryy only (2)Study r A rC N V1 V2

1 .2 .894 .224 200 .0042 .0053

2 .2 .906 .221 100 .0085 .0104

3 .4 .938 .426 150 .0057 .0064

4 .4 .949 .422 80 .0107 .0118

2868.or )1/(]2868.1[)( 221 io NeVarV

22 /)( AeVarV o

Correct ryy only (3)

Study rC A Niwi wirC

1 .224 .894 200 160 35.78

2 .221 .906 100 82 18.11

3 .426 .938 150 132 56.28

4 .422 .949 80 72 30.36

Sum 446 140.53

2iii ANw 315.446/53.140ˆ

i

CiC w

rwr i

Correct ryy only (4)

Study rCwi

Wi[rC-rbarC]2 V2 wV2

1 .224 160 1.339 .0053 .8465

2 .221 82 .728 .0104 .8508

3 .426 132 1.635 .0064 .8479

4 .422 72 .817 .0118 .8529

Sum 446 4.52 3.3981

0101.446/52.4][

)(2

i

CCiC w

rrwrVar i315.Cr

0076.446/3981.3)( 2

i

ii

w

VwveAve

Correct ryy only (5)

0502.)(

0025.0076.0101.)()()(

VarSD

veAverVarVar C

Bare-Bones

ryy corrected

M .2868 .315

V(r) .0098 .0101

SDrho .0585 .0502

95CRlow .17 .22

95CRup .40 .41

All corrections, Indirect RR

Study Ni r rxxi ryy Ux

1 200 .20 .90 .80 1.5

2 100 .20 .80 .82 1.5

3 150 .40 .85 .88 1.0

4 80 .40 .85 .90 1.2

Mean 132.5 .30 .85 .85 1.3

2868.or Already know bare-bones mean.

Indirect RR (2)Study ro rxxi ryy Ux ux rxxa UT

1 .20 .90 .80 1.5 .67 .96 1.55

2 .20 .80 .82 1.5 .67 .91 1.60

3 .40 .85 .88 1.0 1 .85 1

4 .40 .85 .90 1.2 .83 .90 1.23

2/12 )1(

xxa

xxaXT r

ruu)1(1 2

xxiXxxa rur

Indirect RR (3)Study ro rxxi ryy UT rc1 rc A

1 .20 .90 .80 1.55 .236 .351 .570

2 .20 .80 .82 1.60 .247 .378 .530

3 .40 .85 .88 1 .462 .462 .865

4 .40 .85 .90 1.23 .457 .535 .747

yyxxioc rrrr /1 1)1( 2

12

1

cT

cTc

rU

rUr

co rrA /

Indirect RR (4)Study Ni rc A wi wrc

1 200 .351 .570 64.93 22.79 .480

2 100 .378 .530 28.04 10.59 .099

3 150 .462 .865 112.2 51.89 .073

4 80 .535 .747 44.69 23.92 .431

Sum 249.86 109.19 1.082

2iii ANw 437.

86.249

19.109Cr

2)( CCi rrw

0043.86.249

082.1)( CrVar

Indirect RR (5)Study Ni rc A wi V1 V2 arr V3 wV3

1 200 .351 .570 64.93 .0042 .013 .95 .012 .76

2 100 .378 .530 28.04 .0085 .030 .94 .027 .75

3 150 .462 .865 112.2 .0057 .008 1 .008 .85

4 80 .535 .747 44.69 .0107 .019 .92 .016 .73

Sum 249.86 3.09

)1/(]2868.1[)( 221 io NeVarV 2

2 /)( AeVarV o

]1)1/[(1 22 oTrr rUa 22

3 VaV rr 0124.86.249

09.3)( veAve

Indirect RR (6)

0)(

008.0124.0043.)()()(

VarSD

veAverVarVar C

Bare-Bones

ryy corrected

Full indirect correction

M .2868 .315 .437

V(r) .0098 .0101 .0043

SDrho .0585 .0502 0

95CRlow .17 .22 .44

95CRup .40 .41 .44

Direct Range Restriction (1)Study ro rxxi ryy Ux rxxa rc1 rc2 rc

1 .20 .90 .80 1.5 .96 .22 .33 .33

2 .20 .80 .82 1.5 .91 .22 .32 .34

3 .40 .85 .88 1.0 .85 .43 .43 .46

4 .40 .85 .90 1.2 .90 .42 .49 .51

)1(1 2xxiXxxa rur

yyoC rrr /1 1)1( 21

2

12

CX

CXC

rU

rUr

xxaCC rrr /2

Direct RR (2)Study ro rc A Ni w wrc

1 .20 .33 .60 200 72.20 24.03

2 .20 .34 .59 100 35.23 11.87

3 .40 .46 .86 150 112.2 51.89

4 .40 .51 .78 80 48.30 24.86

Sum 267.92 112.66

2iii ANw 42.92.267/66.112ˆ

i

CiC w

rwr i

Direct RR (3)Study rc w

1 .33 72.20 .55

2 .34 35.23 .25

3 .46 112.2 .20

4 .51 48.30 .43

Sum 267.92 1.43

42.Cr

2)( CCi rrw

0053.92.267

43.1)( CrVar

Direct RR (4)Study Ni rc A wi V1 V2 arr V3 wV3

1 200 .33 .60 72.20 .0042 .012 .95 .011 .77

2 100 .34 .59 35.23 .0085 .024 .95 .022 .77

3 150 .46 .86 112.2 .0057 .008 1 .008 .85

4 80 .51 .78 48.30 .0107 .018 .93 .015 .74

Sum 267.92 3.13

)1/(]2868.1[)( 221 io NeVarV 2

2 /)( AeVarV o

]1)1/[(1 22 oXrr rUa 22

3 VaV rr 012.92.267

13.3)( veAve

Direct RR (5)

0)(

006.0117.0053.)()()(

VarSD

veAverVarVar C

Bare-Bones

ryy corrected

Full indirect correction

Full direct correction

M .2868 .315 .437 .42

V(r) .0098 .0101 .0043 .005

SDrho .0585 .0502 0 0

95CRlow .17 .22 .44 .42

95CRup .40 .41 .44 .42