validscale: A Stata module to validate subjective measurement scales using Classical Test Theory Bastien Perrot, Emmanuelle Bataille, Jean-Benoit Hardouin UMR INSERM U1246 - SPHERE "methodS in Patient-centered outcomes and HEalth ResEarch", University of Nantes, University of Tours, France [email protected]French Stata Users Group Meeting, July 6, 2017 1 / 22 validscale N
42
Embed
1cm validscale: A Stata module to validate subjective ... · validscale: A Stata module to validate subjective measurement scales using Classical Test Theory Bastien Perrot, Emmanuelle
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
validscale: A Stata module to validate subjectivemeasurement scales using Classical Test Theory
We use questionnaires to measure non-observable characteristics/traits
personality traitsaptitudes, intelligencequality of life...
The questionnaires are subjective measurement scales providing one orseveral scores based on the sum (or mean) of responses to items (binary orordinal variables)
2 / 22validscale
N
Hospital Anxiety and Depression Scale (Zigmondand Snaith, 1983)
3 / 22validscale
N
Validity and reliability of a questionnaireIn order to be useful, a questionnaire must be valid and reliable.
Validity refers to the degree to which a questionnaire measures theconcept(s) of interest accurately (e.g. anxiety anddepression).
Reliability refers to the degree to which a questionnaire measures theconcept(s) of interest consistently (e.g. Are there enoughitems ? Are the scores reproducible ?)
Reliability refers to the degree to which a questionnaire measures theconcept(s) of interest consistently (e.g. Are there enoughitems ? Are the scores reproducible ?)
Reliability refers to the degree to which a questionnaire measures theconcept(s) of interest consistently (e.g. Are there enoughitems ? Are the scores reproducible ?)
Reliability refers to the degree to which a questionnaire measures theconcept(s) of interest consistently (e.g. Are there enoughitems ? Are the scores reproducible ?)
varlist contains the variables (items) used to compute the scores.The first items of varlist compose the first dimension, thefollowing items define the second dimension, and so on.
partition allows defining in numlist the number of items in eachdimension. The number of elements in this list indicates thenumber of dimensions.
]varlist contains the variables (items) used to compute the scores.
The first items of varlist compose the first dimension, thefollowing items define the second dimension, and so on.
partition allows defining in numlist the number of items in eachdimension. The number of elements in this list indicates thenumber of dimensions.
. validscale ioc1-ioc37, part(4 4 7 3 3 4 7 5)
7 / 22validscale
N
Reliability (default output)Summary table providing indices for internal consistency (Cronbach’s alpha), dicrimination(Feguson’s delta), and "scalability" (Loevinger’s H coefficients, IRT related)
. validscale ioc1-ioc37, part(4 4 7 3 3 4 7 5) scorename(HA PSE W BCC AC AE LIMOC) compscore(sum) alpha(0.7) delta(0.9) h(0.3)
Reliability (default output)Summary table providing indices for internal consistency (Cronbach’s alpha), dicrimination(Feguson’s delta), and "scalability" (Loevinger’s H coefficients, IRT related)
. validscale ioc1-ioc37, part(4 4 7 3 3 4 7 5) scorename(HA PSE W BCC AC AE LIMOC) compscore(sum) alpha(0.7) delta(0.9) h(0.3)
. validscale ioc1-ioc37, part(4 4 7 3 3 4 7 5) scorename(HA PSE W BCC AC AE LIMOC) compscore(sum) graph
010
2030
Per
cent
1 2 3 4 5HA
05
1015
2025
Per
cent
1 2 3 4 5PSE
05
1015
20P
erce
nt
1 2 3 4 5W
05
1015
20P
erce
nt
1 2 3 4 5BCC
05
1015
20P
erce
nt
1 2 3 4 5AC
010
2030
Per
cent
1 2 3 4 5AE
010
2030
Per
cent
1 2 3 4 5LI
05
1015
2025
Per
cent
1 2 3 4 5MOC
Figure: Histograms of scores 10 / 22validscale
N
Descriptive graphs (graph)
. validscale ioc1-ioc37, part(4 4 7 3 3 4 7 5) scorename(HA PSE W BCC AC AE LIMOC) compscore(sum) graph
HA
PSE
W
BCCAC
AE
LI
MOC
-3-2
-10
12
-1 0 1 2 3 4
Figure: Correlations between scores 10 / 22validscale
N
Descriptive graphs (graph)
. validscale ioc1-ioc37, part(4 4 7 3 3 4 7 5) scorename(HA PSE W BCC AC AE LIMOC) compscore(sum) graph
ioc1
ioc2ioc3
ioc4
ioc5
ioc6ioc7
ioc8
ioc9
ioc10
ioc11 ioc12ioc13
ioc14ioc15
ioc16ioc17
ioc18
ioc19ioc20
ioc21
ioc22ioc23
ioc24ioc25
ioc26
ioc27
ioc28
ioc29 ioc30
ioc31
ioc32
ioc33
ioc34ioc35ioc36
ioc37
HA
PSE
W
BCC
AC
AE
LI
MOC
-2-1
01
-.5 0 .5 1 1.5 2
Figure: Correlations between items 10 / 22validscale
N
Confirmatory Factor Analysis (cfa)How well the supposed structure (number of dimensions, clustering of items) fit the data ?
→ Confirmatory Factor Analysis (based on the sem command)
Some criteria based of fit indices: Root Mean Square Error of Approximation (RMSEA) <0.06, Comparative Fit Index (CFI) > 0.95
. validscale ioc1-ioc37, part(4 4 7 3 3 4 7 5) scorename(HA PSE W BCC AC AE LI MOC) cfa cfacov(ioc1*ioc3)
Confirmatory factor analysis
Warning: some items have less than 7 response categories. If multivariatenormality assumption does not hold, maximum likelihood estimation might notbe appropriate. Consider using cfasb in order to apply Satorra-Bentleradjustment or using cfamethod(adf).
Covariances between errors added: e.ioc1*e.ioc3Number of used individuals: 292Item Dimension Factor Standard Intercept Standard Error Variance of
loading error error variance dimension
ioc1 HA 1.00 . 3.36 0.07 1.33 0.16ioc2 HA 2.05 0.46 3.95 0.06 0.45ioc3 HA 1.53 0.31 4.01 0.06 0.55ioc4 HA 1.47 0.34 3.77 0.06 0.68ioc5 PSE 1.00 . 3.42 0.07 1.32 0.32ioc6 PSE 1.56 0.24 3.27 0.07 0.69ioc7 PSE 1.15 0.20 3.70 0.06 0.66ioc8 PSE 1.37 0.22 2.91 0.07 0.80
(output omitted)
Goodness of fit:chi2 df chi2/df RMSEA [90% CI] SRMR NFI
Confirmatory Factor Analysis (cfa)How well the supposed structure (number of dimensions, clustering of items) fit the data ?
→ Confirmatory Factor Analysis (based on the sem command)
Some criteria based of fit indices: Root Mean Square Error of Approximation (RMSEA) <0.06, Comparative Fit Index (CFI) > 0.95
. validscale ioc1-ioc37, part(4 4 7 3 3 4 7 5) scorename(HA PSE W BCC AC AE LI MOC) cfa cfacov(ioc1*ioc3)
Confirmatory factor analysis
Warning: some items have less than 7 response categories. If multivariatenormality assumption does not hold, maximum likelihood estimation might notbe appropriate. Consider using cfasb in order to apply Satorra-Bentleradjustment or using cfamethod(adf).
Covariances between errors added: e.ioc1*e.ioc3Number of used individuals: 292Item Dimension Factor Standard Intercept Standard Error Variance of
loading error error variance dimension
ioc1 HA 1.00 . 3.36 0.07 1.33 0.16ioc2 HA 2.05 0.46 3.95 0.06 0.45ioc3 HA 1.53 0.31 4.01 0.06 0.55ioc4 HA 1.47 0.34 3.77 0.06 0.68ioc5 PSE 1.00 . 3.42 0.07 1.32 0.32ioc6 PSE 1.56 0.24 3.27 0.07 0.69ioc7 PSE 1.15 0.20 3.70 0.06 0.66ioc8 PSE 1.37 0.22 2.91 0.07 0.80
(output omitted)
Goodness of fit:chi2 df chi2/df RMSEA [90% CI] SRMR NFI
Convergent and Divergent validities (convdiv)Are the items correlated enough with the dimension they theoretically belong to ?
Are they more correlated with their own dimension than with other dimensions ?
→ Inspection of correlations between items and scores or rest-scores (i.e. the scorescomputed without the considered item)
. validscale ioc1-ioc37, part(4 4 7 3 3 4 7 5) scorename(HA PSE W BCC AC AE LI MOC) convdiv
Convergent validity: 33/37 items (89.2%) have a correlation coefficient with the score of their own dimensiongreater than 0.400Divergent validity: 33/37 items (89.2%) have a correlation coefficient with the score of their own dimensiongreater than those computed with other scores.
12 / 22validscale
N
Convergent and Divergent validities (convdiv)Are the items correlated enough with the dimension they theoretically belong to ?
Are they more correlated with their own dimension than with other dimensions ?
→ Inspection of correlations between items and scores or rest-scores (i.e. the scorescomputed without the considered item)
. validscale ioc1-ioc37, part(4 4 7 3 3 4 7 5) scorename(HA PSE W BCC AC AE LI MOC) convdiv
Convergent validity: 33/37 items (89.2%) have a correlation coefficient with the score of their own dimensiongreater than 0.400Divergent validity: 33/37 items (89.2%) have a correlation coefficient with the score of their own dimensiongreater than those computed with other scores.
12 / 22validscale
N
Convergent and divergent validities (convdiv)
. validscale ioc1-ioc37, part(4 4 7 3 3 4 7 5) scorename(HA PSE W BCC AC AE LIMOC) convdiv convdivboxplot tconc(0.4)
0.2
.4.6
Correlations between items of HA and scores
HA PSE W BCC AC AE LI MOC
13 / 22validscale
N
Convergent and divergent validities (convdiv)
. validscale ioc1-ioc37, part(4 4 7 3 3 4 7 5) scorename(HA PSE W BCC AC AE LIMOC) convdiv convdivboxplot tconc(0.4)
0.2
.4.6
Correlations between items of PSE and scores
HA PSE W BCC AC AE LI MOC
13 / 22validscale
N
Convergent and divergent validities (convdiv)
. validscale ioc1-ioc37, part(4 4 7 3 3 4 7 5) scorename(HA PSE W BCC AC AE LIMOC) convdiv convdivboxplot tconc(0.4)
-.20
.2.4
.6.8
Correlations between items of W and scores
HA PSE W BCC AC AE LI MOC
13 / 22validscale
N
Convergent and divergent validities (convdiv)
. validscale ioc1-ioc37, part(4 4 7 3 3 4 7 5) scorename(HA PSE W BCC AC AE LIMOC) convdiv convdivboxplot tconc(0.4)
0.2
.4.6
.8
Correlations between items of BCC and scores
HA PSE W BCC AC AE LI MOC
13 / 22validscale
N
Convergent and divergent validities (convdiv)
. validscale ioc1-ioc37, part(4 4 7 3 3 4 7 5) scorename(HA PSE W BCC AC AE LIMOC) convdiv convdivboxplot tconc(0.4)
-.20
.2.4
.6.8
Correlations between items of AC and scores
HA PSE W BCC AC AE LI MOC
13 / 22validscale
N
Convergent and divergent validities (convdiv)
. validscale ioc1-ioc37, part(4 4 7 3 3 4 7 5) scorename(HA PSE W BCC AC AE LIMOC) convdiv convdivboxplot tconc(0.4)
0.2
.4.6
Correlations between items of AE and scores
HA PSE W BCC AC AE LI MOC
13 / 22validscale
N
Convergent and divergent validities (convdiv)
. validscale ioc1-ioc37, part(4 4 7 3 3 4 7 5) scorename(HA PSE W BCC AC AE LIMOC) convdiv convdivboxplot tconc(0.4)
0.2
.4.6
Correlations between items of LI and scores
HA PSE W BCC AC AE LI MOC
13 / 22validscale
N
Convergent and divergent validities (convdiv)
. validscale ioc1-ioc37, part(4 4 7 3 3 4 7 5) scorename(HA PSE W BCC AC AE LIMOC) convdiv convdivboxplot tconc(0.4)
-.20
.2.4
.6.8
Correlations between items of MOC and scores
HA PSE W BCC AC AE LI MOC
13 / 22validscale
N
Reproducibility (repet)
Are the scores and items reproducible in time ?
→ Intraclass Correlation Coefficients (ICC) for reproducibility of scores;kappa’s coefficients for reproducibility of items
. validscale ioc1-ioc37, part(4 4 7 3 3 4 7 5) scorename(HA PSE W BCC AC AE LIMOC) repet(ioc1_2-ioc37_2) kappa ickappa(500)
Reproducibility
Dimension n Item Kappa 95% CI for Kappa ICC 95% CI for ICC(bootstrapped)
validscale performs the recommended analyses (under CTT) to assess thereliability and validity of a questionnaire
A dialog box allows using the command in a user-friendly way (type . dbvalidscale)
Warning/error messages are displayed to help the user during the analysis
ssc install validscale
20 / 22validscale
N
References I
Blanchette, D. (2010). Lstrfun: Stata module to modify long local macros. Statistical SoftwareComponents, Boston College Department of Economics.
Crespi, C. M., Ganz, P. A., Petersen, L., Castillo, A., and Caan, B. (2008). Refinement andpsychometric evaluation of the impact of cancer scale. Journal of the National CancerInstitute, 100(21):1530–1541.
Gadelrab, H. (2010). Evaluating the fit of structural equation models: Sensitivity tospecification error and descriptive goodness-of-fit indices. Lambert Academic Publishing.
Hamel, J.-F. (2014). Mi_twoway: Stata module for computing scores on questionnairescontaining missing item responses. Statistical Software Components, Boston CollegeDepartment of Economics.
Hardouin, J.-B. (2004a). DETECT: Stata module to compute the DETECT, Iss and R indexesto test a partition of items. Statistical Software Components, Boston College Department ofEconomics.
Hardouin, J.-B. (2004b). Loevh: Stata module to compute guttman errors and loevinger hcoefficients. Statistical Software Components, Boston College Department of Economics.
Hardouin, J.-B. (2007). Delta: Stata module to compute the delta index of scale discrimination.Statistical Software Components, Boston College Department of Economics.
Hardouin, J.-B. (2013). Imputeitems: Stata module to impute missing data of binary items.
21 / 22validscale
N
References II
Hardouin, J.-B., Bonnaud-Antignac, A., Sébille, V., et al. (2011). Nonparametric item responsetheory using stata. Stata Journal, 11(1):30.
P., F. and D., M. (2007). Quality of Life: The Assessment, Analysis and Interpretation ofPatient-reported Outcomes. Wiley.
Reichenheim, M. E. (2004). Confidence intervals for the kappa statistic. Stata Journal,4(4):421–428(8).
Ware Jr, J. E., Kosinski, M., and Keller, S. D. (1996). A 12-item short-form health survey:construction of scales and preliminary tests of reliability and validity. Medical care,34(3):220–233.
Zigmond, A. S. and Snaith, R. P. (1983). The hospital anxiety and depression scale. ActaPsychiatrica Scandinavica, 67(6):361–370.