Detecting DIF/DSF with PCMtrees Testing for DIF in the RM Standard tests Model-based recursive partitioning Extension to the PCM DIF/DSF in the PCM (Un)ordered threshold parameters Visualization Example: Verbal Aggression data Summary References Detecting Differential Item and Differential Step Functioning with Partial Credit Trees Basil Abou El-Komboz, Achim Zeileis and Carolin Strobl Detecting DIF/DSF with PCMtrees Testing for DIF in the RM Standard tests Model-based recursive partitioning Extension to the PCM DIF/DSF in the PCM (Un)ordered threshold parameters Visualization Example: Verbal Aggression data Summary References Outline Testing for DIF in the Rasch model Standard model tests Model-based recursive partitioning Extending the model-based recursive partitioning approach to the Partial Credit Model (PCM) Differential item and step functioning in the PCM (Un)ordered threshold parameters in the PCM Visualization in Partial Credit trees Example: Verbal Aggression data Summary Detecting DIF/DSF with PCMtrees Testing for DIF in the RM Standard tests Model-based recursive partitioning Extension to the PCM DIF/DSF in the PCM (Un)ordered threshold parameters Visualization Example: Verbal Aggression data Summary References Differential Item Functioning (DIF) is present when one or more items of a test are easier or harder to solve for certain subjects even though they have the same latent trait Detecting DIF/DSF with PCMtrees Testing for DIF in the RM Standard tests Model-based recursive partitioning Extension to the PCM DIF/DSF in the PCM (Un)ordered threshold parameters Visualization Example: Verbal Aggression data Summary References Standard model tests tests for k given groups graphical test, Andersen’s Likelihood-Ratio Test, Wald Tests + straightforward interpretation - only detect DIF in specified groups latent-class approach Rost’s “Mixed” (mixture) Rasch model + identifies previously unknown groups with DIF - groups are not directly interpretable ⇒ 2nd step: describe groups with covariates (e.g., Cohen and Bolt, 2005)
7
Embed
Detecting DIF/DSF with PCMtrees Detecting Di erential Item ...eeecon.uibk.ac.at/psychoco/2012/slides/Strobl.pdf · DIF/DSF with PCMtrees Testing for DIF in the RM Standard tests Model-based
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Detecting
DIF/DSF with
PCMtrees
Testing for DIF in
the RM
Standard tests
Model-based recursive
partitioning
Extension to the
PCM
DIF/DSF in the PCM
(Un)ordered threshold
parameters
Visualization
Example: Verbal
Aggression data
Summary
References
Detecting Differential Item and
Differential Step Functioning with
Partial Credit Trees
Basil Abou El-Komboz, Achim Zeileis and Carolin Strobl
Detecting
DIF/DSF with
PCMtrees
Testing for DIF in
the RM
Standard tests
Model-based recursive
partitioning
Extension to the
PCM
DIF/DSF in the PCM
(Un)ordered threshold
parameters
Visualization
Example: Verbal
Aggression data
Summary
References
Outline
Testing for DIF in the Rasch model
Standard model tests
Model-based recursive partitioning
Extending the model-based recursive partitioning approach
to the Partial Credit Model (PCM)
Differential item and step functioning in the PCM
(Un)ordered threshold parameters in the PCM
Visualization in Partial Credit trees
Example: Verbal Aggression data
Summary
Detecting
DIF/DSF with
PCMtrees
Testing for DIF in
the RM
Standard tests
Model-based recursive
partitioning
Extension to the
PCM
DIF/DSF in the PCM
(Un)ordered threshold
parameters
Visualization
Example: Verbal
Aggression data
Summary
References
Differential Item Functioning (DIF)
is present when one or more items of a test
I are easier or harder to solve for certain subjects
I even though they have the same latent trait
Detecting
DIF/DSF with
PCMtrees
Testing for DIF in
the RM
Standard tests
Model-based recursive
partitioning
Extension to the
PCM
DIF/DSF in the PCM
(Un)ordered threshold
parameters
Visualization
Example: Verbal
Aggression data
Summary
References
Standard model tests
I tests for k given groups
graphical test, Andersen’s Likelihood-Ratio Test,
Wald Tests
+ straightforward interpretation
− only detect DIF in specified groups
I latent-class approach
Rost’s “Mixed” (mixture) Rasch model
+ identifies previously unknown groups with DIF
− groups are not directly interpretable
⇒ 2nd step: describe groups with covariates
(e.g., Cohen and Bolt, 2005)
Detecting
DIF/DSF with
PCMtrees
Testing for DIF in
the RM
Standard tests
Model-based recursive
partitioning
Extension to the
PCM
DIF/DSF in the PCM
(Un)ordered threshold
parameters
Visualization
Example: Verbal
Aggression data
Summary
References
Standard model tests
●
●●●
●
●
●
●●
●
●
●
−3 −2 −1 0 1 2 3
−3
−2
−1
01
23
Geschlecht = Mann
Ges
chle
cht =
Fra
u 1
234
5
6
7
89
10
11
12
●
●●
●
●
●
●
●
●
●
●
●
(Mair, Hatzinger, and Maier, 2010, package eRm)
Detecting
DIF/DSF with
PCMtrees
Testing for DIF in
the RM
Standard tests
Model-based recursive
partitioning
Extension to the
PCM
DIF/DSF in the PCM
(Un)ordered threshold
parameters
Visualization
Example: Verbal
Aggression data
Summary
References
Standard model tests
I tests for k given groups
graphical test, Andersen’s Likelihood-Ratio Test,
Wald Tests
+ straightforward interpretation
− only detect DIF in specified groups
I latent-class approach
Rost’s “Mixed” (mixture) Rasch model
+ identifies previously unknown groups with DIF
− groups are not directly interpretable
⇒ 2nd step: describe groups with covariates
(e.g., Cohen and Bolt, 2005)
Detecting
DIF/DSF with
PCMtrees
Testing for DIF in
the RM
Standard tests
Model-based recursive
partitioning
Extension to the
PCM
DIF/DSF in the PCM
(Un)ordered threshold
parameters
Visualization
Example: Verbal
Aggression data
Summary
References
New: Model-based recursive partitioning
+ identifies previously unknown groups with DIF
+ straightforward interpretationgender
p = 0.006
1
male female
agep < 0.001
2
≤ 34 > 34
Node 3 (n = 35)
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●●
●
1 20
−2.68
4.66Node 4 (n = 74)
●
● ● ●
●
● ●
●
●
●
●● ●
●●
●
●
●
●
●
1 20
−2.68
4.66Node 5 (n = 91)
● ●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●●
1 20
−2.68
4.66
function raschtree in package psychotree
Detecting
DIF/DSF with
PCMtrees
Testing for DIF in
the RM
Standard tests
Model-based recursive
partitioning
Extension to the
PCM
DIF/DSF in the PCM
(Un)ordered threshold
parameters
Visualization
Example: Verbal
Aggression data
Summary
References
Approach used in psychotree takes care of...
I selecting splitting variables ⇔ parameter instability tests
scor
e co
ntrib
utio
ns
25 30 35 40 45
−0.
40
0.2
0.4
age
I selecting optimal cutpoints
I other multiple testing issues
I between variables in each split
I over successive splits
(Zeileis and Hornik, 2007; Zeileis, Hothorn, and Hornik, 2008;
Strobl, Malley, and Tutz, 2009; Strobl, Kopf, and Zeileis, 2010a,b)
Detecting
DIF/DSF with
PCMtrees
Testing for DIF in
the RM
Standard tests
Model-based recursive
partitioning
Extension to the
PCM
DIF/DSF in the PCM
(Un)ordered threshold
parameters
Visualization
Example: Verbal
Aggression data
Summary
References
Extending the model-based partitioning approach
Rasch trees genderp = 0.006
1
male female
agep < 0.001
2
≤ 34 > 34
Node 3 (n = 35)
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●●
●
1 20
−2.68
4.66Node 4 (n = 74)
●
● ● ●
●
● ●
●
●
●
●● ●
●●
●
●
●
●
●
1 20
−2.68
4.66Node 5 (n = 91)
● ●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●●
1 20
−2.68
4.66
Partial Credit trees
Detecting
DIF/DSF with
PCMtrees
Testing for DIF in
the RM
Standard tests
Model-based recursive
partitioning
Extension to the
PCM
DIF/DSF in the PCM
(Un)ordered threshold
parameters
Visualization
Example: Verbal
Aggression data
Summary
References
Extending the model-based partitioning approach
Rasch trees genderp = 0.006
1
male female
agep < 0.001
2
≤ 34 > 34
Node 3 (n = 35)
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●●
●
1 20
−2.68
4.66Node 4 (n = 74)
●
● ● ●
●
● ●
●
●
●
●● ●
●●
●
●
●
●
●
1 20
−2.68
4.66Node 5 (n = 91)
● ●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●●
1 20
−2.68
4.66
Partial Credit trees
Detecting
DIF/DSF with
PCMtrees
Testing for DIF in
the RM
Standard tests
Model-based recursive
partitioning
Extension to the
PCM
DIF/DSF in the PCM
(Un)ordered threshold
parameters
Visualization
Example: Verbal
Aggression data
Summary
References
Extending the model-based partitioning approach
Rasch model
I scores are 0 or 1
I each item has one location parameter = difficulty
I DIF means item is more/less difficult for certain group
Partial Credit model
I scores are between 0 and mj
I different parametrizations: e.g. mj thresholds
I DIF means entire item is more/less difficult
I DSF means some steps are more/less difficult
(may cancel out so there is no overall DIF)
Detecting
DIF/DSF with
PCMtrees
Testing for DIF in
the RM
Standard tests
Model-based recursive
partitioning
Extension to the
PCM
DIF/DSF in the PCM
(Un)ordered threshold
parameters
Visualization
Example: Verbal
Aggression data
Summary
References
(Un)ordered threshold parameters in the PCM
−5 0 5 10 15
0.0
0.2
0.4
0.6
0.8
1.0
P
(uij=
c|θ i
,δj1,..
.,δj3)
δj1 δj2 δj3
0 1 2 3
P(uij = c |θi , δj1, . . . , δjmj) =
e∑c
k=0(θi−δjk)
∑mj
l=0 e∑l
k=0(θi−δjk)
with∑0
k=0 (θi − δjk) = 0
Detecting
DIF/DSF with
PCMtrees
Testing for DIF in
the RM
Standard tests
Model-based recursive
partitioning
Extension to the
PCM
DIF/DSF in the PCM
(Un)ordered threshold
parameters
Visualization
Example: Verbal
Aggression data
Summary
References
(Un)ordered threshold parameters in the PCM
−5 0 5 10 15
0.0
0.2
0.4
0.6
0.8
1.0
P(u
ij=c|
θ i,δ
j1,..
.,δj3)
δj1δj2 δj3
0 1
2 3
P(uij = c |θi , δj1, . . . , δjmj) =
e∑c
k=0(θi−δjk)
∑mj
l=0 e∑l
k=0(θi−δjk)
with∑0
k=0 (θi − δjk) = 0
Detecting
DIF/DSF with
PCMtrees
Testing for DIF in
the RM
Standard tests
Model-based recursive
partitioning
Extension to the
PCM
DIF/DSF in the PCM
(Un)ordered threshold
parameters
Visualization
Example: Verbal
Aggression data
Summary
References
Visualization in Partial Credit trees
Want Curse
0
0.2
0.4
0.6
0.8
1
δ1δ1 δ2δ2
Do Curse
0
0.2
0.4
0.6
0.8
1
δ1δ1δ2δ2
Want Scold
0
0.2
0.4
0.6
0.8
1
δ1δ1δ2δ2
Do Scold
0
0.2
0.4
0.6
0.8
1
δ1δ1δ2δ2
Want Shout
0
0.2
0.4
0.6
0.8
1
δ1δ1 δ2δ2
Do Shout
0
0.2
0.4
0.6
0.8
1
δ1δ1δ2δ2
−2 −1 0 1 2
−2 −1 0 1 2
Category Characteristic Curves
Pro
babi
lity
Latent Trait
Cat. 0 Cat. 1 Cat. 2
Detecting
DIF/DSF with
PCMtrees
Testing for DIF in
the RM
Standard tests
Model-based recursive
partitioning
Extension to the
PCM
DIF/DSF in the PCM
(Un)ordered threshold
parameters
Visualization
Example: Verbal
Aggression data
Summary
References
Visualization in Partial Credit trees
Want Shout
0
0.2
0.4
0.6
0.8
1
δ1δ1 δ2δ2
Do Shout
0
0.2
0.4
0.6
0.8
1
δ1δ1δ2δ2
−2 −1 0 1 2 3
−2 −1 0 1 2 3
Category Characteristic Curves
Prob
abilit
y
Latent Trait
Cat. 0 Cat. 1 Cat. 2
Detecting
DIF/DSF with
PCMtrees
Testing for DIF in
the RM
Standard tests
Model-based recursive
partitioning
Extension to the
PCM
DIF/DSF in the PCM
(Un)ordered threshold
parameters
Visualization
Example: Verbal
Aggression data
Summary
References
Visualization in Partial Credit trees
Want Curse
0
0.2
0.4
0.6
0.8
1
δ1δ1 δ2δ2
Do Curse
0
0.2
0.4
0.6
0.8
1
δ1δ1δ2δ2
Want Scold
0
0.2
0.4
0.6
0.8
1
δ1δ1δ2δ2
Do Scold
0
0.2
0.4
0.6
0.8
1
δ1δ1δ2δ2
Want Shout
0
0.2
0.4
0.6
0.8
1
δ1δ1 δ2δ2
Do Shout
0
0.2
0.4
0.6
0.8
1
δ1δ1δ2δ2
−2 −1 0 1 2
−2 −1 0 1 2
Category Characteristic Curves
Pro
babi
lity
Latent Trait
Cat. 0 Cat. 1 Cat. 2
Detecting
DIF/DSF with
PCMtrees
Testing for DIF in
the RM
Standard tests
Model-based recursive
partitioning
Extension to the
PCM
DIF/DSF in the PCM
(Un)ordered threshold
parameters
Visualization
Example: Verbal
Aggression data
Summary
References
Visualization in Partial Credit trees
Wan
t Cur
se
0
0.2
0.4
0.6
0.81
δ 1δ 1δ 2δ 2
Do
Cur
se
00.2
0.4
0.6
0.8
1
δ 1δ 1δ 2δ 2
Wan
t Sco
ld
0
0.2
0.4
0.6
0.81
δ 1δ 1δ2δ 2
Do
Sco
ld
00.2
0.4
0.6
0.8
1
δ 1δ 1δ 2δ 2
Wan
t Sho
ut
0
0.2
0.4
0.6
0.81
δ 1δ 1δ 2δ 2
Do
Sho
ut
00.2
0.4
0.6
0.8
1
δ 1δ 1δ 2δ 2
−2
−1
01
2
−2
−1
01
2
Cat
egor
y C
hara
cter
istic
Cur
ves
Probability
Late
nt T
rait
Cat
. 0C
at. 1
Cat
. 2
Late
nt tr
ait
−2
−1
01
2
−2
−1
01
2
Want Curse Do Curse Want Scold Do Scold Want Shout Do Shout
inspired by “effect plots”
(Fox and Hong, 2009, package effects)
Detecting
DIF/DSF with
PCMtrees
Testing for DIF in
the RM
Standard tests
Model-based recursive
partitioning
Extension to the
PCM
DIF/DSF in the PCM
(Un)ordered threshold
parameters
Visualization
Example: Verbal
Aggression data
Summary
References
Visualization in Partial Credit trees
Wan
t Cur
se
0
0.2
0.4
0.6
0.81
δ 1δ 1δ 2δ 2
Do
Cur
se
00.2
0.4
0.6
0.8
1
δ 1δ 1δ 2δ 2
Wan
t Sco
ld
0
0.2
0.4
0.6
0.81
δ 1δ 1δ2δ 2
Do
Sco
ld
00.2
0.4
0.6
0.8
1
δ 1δ 1δ 2δ 2
Wan
t Sho
ut
0
0.2
0.4
0.6
0.81
δ 1δ 1δ 2δ 2
Do
Sho
ut
00.2
0.4
0.6
0.8
1
δ 1δ 1δ 2δ 2
−2
−1
01
2
−2
−1
01
2
Cat
egor
y C
hara
cter
istic
Cur
ves
Probability
Late
nt T
rait
Cat
. 0C
at. 1
Cat
. 2
Late
nt tr
ait
−2
−1
01
2
−2
−1
01
2
Want Curse Do Curse Want Scold Do Scold Want Shout Do Shout