EUROPEAN COMMISSION – NTTS 2019 Conference on New Techniques and Technologies for official Statistics A paradigm for rating data models Domenico Piccolo University of Naples Federico II, Naples, ITALY [email protected]D.Piccolo (NA Federico II) A paradigm for rating data models Brussels, 13 March 2019 1 / 60
60
Embed
University of Naples Federico II, Naples, ITALY domenico ... › eurostat › cros › system › files › piccolo_ntts_20… · University of Naples Federico II, Naples, ITALY...
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
EUROPEAN COMMISSION – NTTS 2019
Conference on New Techniques and Technologies for official Statistics
➤ In all of these surveys, questionnaires include several items where
respondents are asked to select an ordinal category.
D.Piccolo (NA Federico II) A paradigm for rating data models Brussels, 13 March 2019 5 / 60
How large should be the scale?
➤ An highly controversial argument: which scale is to be preferred?
➤ As many opinions as fields of interest . . . . . .
Sufficient categories to discriminate
Not so many categories to confusion and/or puzzle
➤ “The larger the scale, the larger the indecision . . . ” ?
➤ The “true” problem is:
are we creating or disclosing uncertainty ?
D.Piccolo (NA Federico II) A paradigm for rating data models Brussels, 13 March 2019 6 / 60
Rating data as result of an experience
➤ Since sensation is a human experience, when interviewees areasked to select a level (category) to express their personal evaluation,the response cannot be exclusively considered as a (possiblemeditated) reaction to a stimulus.
➤ Ordinal scores may express:
• opinions • agreement • judgements • perceptions
• worry • concern • pain • fear • anxiety . . . .
➤ A rating response involves respondent’s history, local and timecircumstances, mood, attitudes, emotions.
D.Piccolo (NA Federico II) A paradigm for rating data models Brussels, 13 March 2019 7 / 60
Several approaches
➤ Several approaches are available for the analysis of rating data, aslog-linear and marginal models, contingency tables inference, and soon.
➤ Latent variables and IRT are among the most diffuse methods todeal with this kind of data; often, specific variants have beenintroduced to face and solve new problems.
➤ The main term of comparison is the class of cumulative modelswhich have been embedded into the GLM perspective.
➤ Currently, several variants of cumulative are available to fit realproblems.
D.Piccolo (NA Federico II) A paradigm for rating data models Brussels, 13 March 2019 8 / 60
2. The classical paradigm
D.Piccolo (NA Federico II) A paradigm for rating data models Brussels, 13 March 2019 9 / 60
➤ In case of logistic random variables ǫi , the probability that the i-thsubject selects a rating r turns out to be:
Pr (Ri = r |θ, ti) =1
1 + exp(−[αr − tiβ])−
1
1 + exp(−[αr−1 − tiβ]),
for i = 1, 2, . . . ,n and r = 1, 2, . . . ,m.
D.Piccolo (NA Federico II) A paradigm for rating data models Brussels, 13 March 2019 13 / 60
A feature of cumulative models
➤ If no covariate is specified, the cumulative model is a saturated one.
➤ Thus, empirical and estimated distribution functions strictly coincide:a perfect fitting.
➤ In this situation there is no statistical model but only an arithmeticequivalence.
➤ This circumstance implies that, without the inclusion of covariates,those models cannot be used per se as statistical tools.
➤ Quite often, the interpretation of these models takes advantage ofodds and log-odds measures which are quantities easily manageableby Medicine and Biomedical researchers.
D.Piccolo (NA Federico II) A paradigm for rating data models Brussels, 13 March 2019 14 / 60
Some difficulties with the classical paradigm
Data generating process refers to a latent variable whose unobservable
distribution defines the discrete distribution for the observable ratings.
Cumulative models are not parsimonious since they require cutpoints
estimates in addition to explicit parameters for significant covariates.
It is difficult to accept that subjects’ decisions consider ratings not greater
than a fixed one, whereas it is more common to consider choices as
determined by the “stimulus” associated to a single category and its
surrounding values.
Without covariates, the classical setting leads to a saturated model,
which implies an arithmetic equivalence between observed and
assumed distributions.
Interpretation of the effect of a single covariate on the probability of a
category is neither easy nor immediate.
Graphical representation is not so immediate since the log-odds are
linear functions of covariates; in general, log-odds are not so easy to
interpret.
D.Piccolo (NA Federico II) A paradigm for rating data models Brussels, 13 March 2019 15 / 60
Cumulative models in Big Data era
➤ When a lot of ordinal data are collected for several items in repeated
occasions, times, units which are differentiated with respect to the available
information set, and the objective is to compare the behaviour/opinions of
subjects in different circumstances, statistical models for the responses which
depend on covariates are not a solution.
➤ This situation is more and more frequent in time where a huge mass of
opinions, judgements and preference are collected by mail, Internet and
social media.
➤ In such cases, it seems more effective to concentrate the modelling step on
data generating process of ordinal observation and discriminate among the
clusters on the basis of the observed distributions (data-dependent approach)
or an estimated structure (model-based approach).
D.Piccolo (NA Federico II) A paradigm for rating data models Brussels, 13 March 2019 16 / 60
3. A generating process for rating data
D.Piccolo (NA Federico II) A paradigm for rating data models Brussels, 13 March 2019 17 / 60
Mixture as a data generating mechanism for rating data
➤ In different contexts, mixture models have been introduced toappropriately fit data to probability mass functions.
➤ The novel paradigm insists on the psychological process whichtransforms a perception into a rating score.
➤ Experimental evidence supports that rating is the result of:
a primary component, generated by the sound impression of therespondent, related to awareness and full understanding of theproblem. It is called feeling (agreement) since it is usually relatedto subject’s motivation;
a secondary component, generated by the intrinsic indecisionabout the final choice. It is called uncertainty (fuzziness), and it ismostly dependent on circumstances that surround the evaluationprocess.
D.Piccolo (NA Federico II) A paradigm for rating data models Brussels, 13 March 2019 18 / 60
Mixture as a data generating mechanism for rating data
➤ Both components will be explicitly modelled by discrete randomvariables and, in first instances, they have been proposed as (shifted)Binomial and (discrete) Uniform, respectively.
➤ Literature confirms the usefulness and effectiveness of such choicesas proved by formal arguments and a vast empirical evidence.
D.Piccolo (NA Federico II) A paradigm for rating data models Brussels, 13 March 2019 19 / 60
Latent classes and mixture
➤ In the logic of latent class models, the mixture we will introduce maybe interpreted as requiring two clusters of respondents:
some people assume a responsible behaviour towards the survey(=their choice follows a shifted Binomial distribution)
some others adopt a totally random criterion (=their choicefollows a discrete Uniform distribution).
➤ This situation is well captured by the models we are introducing.
➤ However, we support a different interpretation of the subjectivebehaviour of the respondents.
D.Piccolo (NA Federico II) A paradigm for rating data models Brussels, 13 March 2019 20 / 60
The reference scheme
➤ New models interpret the probability of a rating R as a mixture of:
a personal decision, motivated by attraction/repulsion, likeness/worry,
agreeableness, agreement towards the item and measured by 1 − ξ;
an inherent indecision in the choice among the categories whose weight
is measured by 1 − π.
Feeling Uncertainty
C U BRandom variable
D.Piccolo (NA Federico II) A paradigm for rating data models Brussels, 13 March 2019 21 / 60
Definition of a CUB model
➤ A CUB model (= Combination of discrete Uniform and shifted Binomial
random variables) is defined by:
1 A stochastic component:
Pr(Ri = r | xi , wi) = πi
[(m − 1
r − 1
)
(1 − ξ i)r−1ξm−r
i
]
︸ ︷︷ ︸
feeling
+ (1 − πi)
[1
m
]
︸ ︷︷ ︸
uncertainty
for r = 1, 2, . . . ,m, where πi ∈ (0, 1] and ξ i ∈ [0, 1], i = 1, 2, . . . ,n.
2 Two systematic components:
logit(πi ) = log(
πi
1−πi
)
= xi β;
logit(ξ i ) = log(
ξ i
1−ξ i
)
= wiγ;⇐⇒
{
πi = 11+e−xi β ;
ξ i = 11+e−wi γ
;
where β and γ are the parameters to be estimated, and xi and wi are
the row vectors containing the values of the covariates of the i-th subject,
suitable to explain πi and ξ i , respectively.
D.Piccolo (NA Federico II) A paradigm for rating data models Brussels, 13 March 2019 22 / 60
Interpretation of a CUB model
➤ Each respondent acts with a propensity to adhere to a thoughtfuland to a completely uncertain choice, which is measured by (πi) and(1 − πi ), respectively.
➤ In case of a rating question/item with positive wording:
(1− ξ i) may be interpreted as a measure of preference towardsthe item.
(1− πi ) is a weight of the uncertainty included in the responses.
➤ When the item concerns a negative (reverse) wording (e.g. worry,disagreement, stress, fear, effort, pain, etc.) the interpretation of ξ i and1 − ξ i must be reversed.
D.Piccolo (NA Federico II) A paradigm for rating data models Brussels, 13 March 2019 23 / 60
Explicit link between parameters and subjects’ covariates
➤ A noticeable aspect of CUB models is the direct link between subjects’
covariates and parameters.
➤ Since 1 − ξ i is a direct measure of agreement, feeling, likeness with the item
and 1 − πi is a direct measure of the weight of the uncertainty distribution in
the mixture, it is convenient to express those links by means of:
D.Piccolo (NA Federico II) A paradigm for rating data models Brussels, 13 March 2019 35 / 60
What really is uncertainty?
➤ The measure of uncertainty conveyed by 1− πi includes at least three
points of view:
1 subjective indecision: when we examine 1 − πi , it is possible to consider it
as a measure of personal indecision of the i-th respondent as a function
of selected covariates.
2 heterogeneity: when we analyse a global CUB model for the given item, it
is possible to consider 1 − π as a measure of heterogeneity of the
respondents.
3 predictability: if we study a CUB model to predict ordinal outcomes, it is
possible to consider π as a direct measure of predictability of the model
with respect to two extremes:
minimum → responses follow a (pure) discrete Uniform distribution
maximum → responses follow a (pure) Binomial distribution
D.Piccolo (NA Federico II) A paradigm for rating data models Brussels, 13 March 2019 36 / 60
Comparison of POM and CUB models
➤ A comparison between POM and CUB models requires at least oneexplanatory variable given that POM are saturated when appliedwithout covariates.
➤ To make the comparison possible, a dichotomous covariate (Di = 0for i = 1, . . . , 500; Di = 1 for i = 501, . . . , 1000) has been defined.
➤ Here, with m = 7, the results for data with high heterogeneity arepresented.
D.Piccolo (NA Federico II) A paradigm for rating data models Brussels, 13 March 2019 37 / 60
Highly heterogeneous data
1 2 3 4 5 6 7
0.00
0.05
0.10
0.15
0.20
0.25
Ratings
Rela
tive
frequ
encie
s
D.Piccolo (NA Federico II) A paradigm for rating data models Brussels, 13 March 2019 38 / 60
BIC comparison for POM and CUB models
CUB POM
3800
3820
3840
3860
3880
3900
3920
3800 3840 3880 3920
3800
3820
3840
3860
3880
3900
3920
POM
CUB
D.Piccolo (NA Federico II) A paradigm for rating data models Brussels, 13 March 2019 39 / 60
4. The family of CUB models
D.Piccolo (NA Federico II) A paradigm for rating data models Brussels, 13 March 2019 40 / 60
Using and applying CUB models
➤ After their introduction (Piccolo, 2003), to cope with different realsituations, CUB models have been generalized in several directions.
➤ Family of CUB models.
➤ The program to perform estimation and testing of CUB models,originally coded in the GAUSS language, has been implemented in theR environment.
➤ Currently, the package CUB, latest version 1.1.2, is freely availableon the CRAN web repository.
➤ Further programs for users of STATA and GRETL are forthcoming.
D.Piccolo (NA Federico II) A paradigm for rating data models Brussels, 13 March 2019 41 / 60
Family of CUB models
� Variants of univariate distributions:
CUB models with both subjects’ and objects’ covariatesHierarchical and random effects CUB models (HCUB and RCUB)CUB models with a shelter effectGeneralized CUB models (GeCUB)CUSH modelsLatent Class CUB models (LC-CUB)CUB models with “don’t know” option (DK-CUB)CUB model with MIMIC structure (CUB-MIMIC)CUB time series model (CUB-TS)
� Variants of the probability distributions of components:
CUBE models without and with covariatesIHG models without and with covariatesCUB models with varying uncertainty (VCUB)CAUB modelsNon-linear CUB modelsCUP models
� Joint modelling of items:
Multi-objects modelling approachMultivariate CUB models via latent variablesMultivariate CUB models via copula functionsMultivariate mixtures (SCUB and CUSCUB)
D.Piccolo (NA Federico II) A paradigm for rating data models Brussels, 13 March 2019 42 / 60
CUB models with shelter effect
?
δ 1 − δ
ShelterChoice
CUB
π 1 − π
ShiftedBinomial
DiscreteUniform
D.Piccolo (NA Federico II) A paradigm for rating data models Brussels, 13 March 2019 43 / 60
Definition of a GeCUB model
➤ A GeCUB model with p covariates for uncertainty, q covariates for feeling
and s covariates for shelter effect is specified by:
Pr (R = r | θ∗) = (1 − δi )[
πi br (ξ i) + (1 − πi)Ur
]
+ δi D(c)r ,
where
πi =1
1 + e−xi β; ξ i =
1
1 + e−wi γ; δi =
1
1+ e−zi ω;
for i = 1, 2, . . . ,n, and xi , wi and zi are the subjects’ covariates for explaining πi ,
ξ i , and δi , respectively.
➤ These rows are included in T , a n × (k + 1) matrix of observed k covariates
related to n subjects.
➤ The columns of the X , W and Z matrices may be the same, partially
coincide or completely differ.
D.Piccolo (NA Federico II) A paradigm for rating data models Brussels, 13 March 2019 44 / 60
Level of satisfaction of the personal relationships . . . . . . . . . . . [1]
0 1 2 3 4 5 6 7 8 9 10
Family members
0.0
0.1
0.2
0.3
0.4
average = 8.588
0 1 2 3 4 5 6 7 8 9 10
Friends
0.0
0.1
0.2
0.3
0.4
average = 7.863
0 1 2 3 4 5 6 7 8 9 10
Neighbours
0.0
0.1
0.2
0.3
0.4
average = 5.782
0 1 2 3 4 5 6 7 8 9 10
Colleagues−Acquaintances
0.0
0.1
0.2
0.3
0.4
average = 6.542
D.Piccolo (NA Federico II) A paradigm for rating data models Brussels, 13 March 2019 45 / 60
Test for possible shelter effect at R = 0
➤ We test the significance of possible shelter effect at the first category (that
is, R = 0), by letting c = 0 in the model:
Pr (R = r , θ) = (1 − δ)[
π br (ξ) + (1 − π)1
m
]
+ δ D(c)r , r = 1, 2, . . . ,m
Relationship with log-lik(CUB ) log-lik(CUB +shelter) δ̂ p − value
D.Piccolo (NA Federico II) A paradigm for rating data models Brussels, 13 March 2019 46 / 60
Comparison of CUB and CUB +shelter models
0.0 0.1 0.2 0.3 0.4 0.5 0.6
0.65
0.70
0.75
0.80
0.85
0.90
CUB and CUB+shelter models visualizations
Uncertainty (1 − π)
Leve
l of S
atis
fact
ion
(1
−ξ)
Family members
Friends
Neighbours
Colleagues−Acquaintances
D.Piccolo (NA Federico II) A paradigm for rating data models Brussels, 13 March 2019 47 / 60
Motivations for overdispersion in rating data
➤ Overdispersion may be generated by a variability among individualfeelings. Personal characteristics and different response styles stronglysupport this claim.
➤ A Binomial random variable implies a very strong constraintbetween variance and mean value.
➤ Thus, a Beta-Binomial distribution has been introduced for thefeeling component according to the data generating process ofordinal data.
➤ The specification of the new model is oriented to save the sameparametric structure of CUB models.
D.Piccolo (NA Federico II) A paradigm for rating data models Brussels, 13 March 2019 48 / 60
➤ In fact, although some of them are useful, “all models aresubstantially wrong” and this aphorism is as much valid for a novelparadigm.
➤ The substantive problem is to establish the starting point for furtheradvances in order to achieve better models which in turn should everbe improved.
➤ As statisticians, we must be aware of the role and importance ofuncertainty in human decisions and CUB models may be consideredas building blocks of more complex statistical specifications.
➤ Above all, CUB models act as a benchmark for more refinedanalyses.
D.Piccolo (NA Federico II) A paradigm for rating data models Brussels, 13 March 2019 57 / 60