PAIRED COMPARISONS (Dittrich, Hatzinger, Katzenbeisser) – WU Wien – 6.11.2003 1 PAIRED COMPARISONS MODELS AND APPLICATIONS Regina Dittrich Reinhold Hatzinger Walter Katzenbeisser
PAIRED COMPARISONS (Dittrich, Hatzinger, Katzenbeisser) – WU Wien – 6.11.2003 1
PAIRED COMPARISONS
MODELS AND APPLICATIONS Regina Dittrich Reinhold Hatzinger Walter Katzenbeisser
PAIRED COMPARISONS (Dittrich, Hatzinger, Katzenbeisser) – WU Wien – 6.11.2003 2
PAIRED COMPARISONS (PC) a method of data collection where individuals are asked to judge a number of different pairs of objects, taken from a larger set of J objects.
for each comparison between two objects j and k , the individual can respond j preferred to k
k preferred to j aim is to rank objects into a preference order – obtain an overall ranking of the objects common in marketing (food tasting,..), social and management sciences, …
PAIRED COMPARISONS (Dittrich, Hatzinger, Katzenbeisser) – WU Wien – 6.11.2003 3
THE BASIC BRADLEY TERRY MODEL (BT MODEL) for the comparison ( )jk of object j to object k : we observe jkN , the number of times where j is preferred to k we observe kjN , the number of times where k is preferred to j ( )jkn is the number of times this comparison was performed, ie, jk kjN N+
let jkp be the probability that j is preferred to k in comparison ( )jk the BT-model is
jπ are a set of so-called worth parameters
(positive, constrained to sum to one)
jjk
j kp
ππ π=+
PAIRED COMPARISONS (Dittrich, Hatzinger, Katzenbeisser) – WU Wien – 6.11.2003 4
THE LOG-LINEAR FORM OF THE BRADLEY TERRY MODEL the BT-Model can be formulated as a log-linear model following the usual Multinomial / Poisson - equivalence. the expected value jkm of jkN is given as ( )jk jk jkm n p=
j kjjk
j k k j j kp
π πππ π π π π π= =+ +
then our basic paired comparison model (PC Model) is
jλ are the object parameters
( )jkα are nuisance parameters
this model formulation is feasible for further extensions
( )ln jk jk j km α λ λ= + −
PAIRED COMPARISONS (Dittrich, Hatzinger, Katzenbeisser) – WU Wien – 6.11.2003 5
EXTENSIONS TO THE BASIC PC MODEL - RESPONSES
ALLOW FOR TIES (TRICHOTOMOUS RESPONSES, NOMINAL)
for each comparison between two objects j and k , the response can be
j preferred to k no preference k preferred to j
ORDINAL COMPARISONS (POLYTOMOUS RESPONSES)
for each comparison between two objects j and k , the response can be
strong for j moderate for j … … strong for k
PAIRED COMPARISONS (Dittrich, Hatzinger, Katzenbeisser) – WU Wien – 6.11.2003 6
EXTENSIONS TO THE BASIC PC MODEL – STRUCTURE
POSITION EFFECTS
Is there an effect due to the ordering of the presentation of objects? Does it make a difference ?
MANCHESTER UNITED – INTER MILAN (playing in Manchester)
INTER MILAN – MANCHESTER UNITED (playing in Milan) introduce additional parameter(s) for position effects
PAIRED COMPARISONS (Dittrich, Hatzinger, Katzenbeisser) – WU Wien – 6.11.2003 7
EXTENSIONS TO THE BASIC PC MODEL – COVARIATES
OBJECT COVARIATES ( Dittrich, Hatzinger, Katzenbeisser, J. Royal Statistical Society, C, 1998 )
to model the objects by a few characteristics
jqx covariate for characteristicq of object j qβ effect of characteristic q
c
CATEGORICAL SUBJECT COVARIATES ( Dittrich, Hatzinger, Katzenbeisser, J. Royal Statistical Society, C, 1998 )
CONTINOUS SUBJECT COVARIATES
one contingency table for each subject
SMOOTHED SUBJECT COVARIATES (GAMS) (Francis, Dittrich, Hatzinger, Penn, J. Royal Statistical Society, C, 2002)
1
QOj q jq
qxλ β
==∑
PAIRED COMPARISONS (Dittrich, Hatzinger, Katzenbeisser) – WU Wien – 6.11.2003 8
CATEGORICAL SUBJECT COVARIATES ( Dittrich, Hatzinger, Katzenbeisser, J. Royal Statistical Society, C, 1998 )
Are the preference orderings different for different groups of subjects? For one categorical subject covariate we now have
| ( )ln ( ) ( )j j k kO O S O O SSjk s jk s s j js k ksm α λ λ λ λ λ= + + + − +
where
jOjλ object parameter
jO Sjsλ interaction parameter between object j and subject category s Ssλ fixing the margin for category s of covariate S (nuisance) ( )jk sα nuisance parameters
PAIRED COMPARISONS (Dittrich, Hatzinger, Katzenbeisser) – WU Wien – 6.11.2003 9
EXAMPLE: UNIVERSITY PREFERENCES CEMS – exchange programme students of the WU can study abroad visiting one of currently 17 CEMS universities aim of the study:
• preference orderings of students for different locations • identify reasons for these preferences
data:
• PC-responses allowing for ties about their choices of 6 selected CEMS universities for the semester abroad (London, Paris, Milan, Barcelona, St.Gall, Stockholm)
• several covariates (e.g., gender, working status, language abilities, …)
PAIRED COMPARISONS (Dittrich, Hatzinger, Katzenbeisser) – WU Wien – 6.11.2003 10
Main results: • in general London by far the most popular place • depending on language abilities according places move up • full-time working students prefer Latin universities
0.0
0.1
0.2
0.3
0.4
0.5
ST
MI
SGBA
PA
LO
STOCKHOLM
ST.GALL
MILANBARCELONA
LONDON
PARIS
STOCKHOLM
ST.GALL
LONDON
PARIS
MILAN
BARCELONA
ALL FRENCH FULLTIME JOB
PAIRED COMPARISONS (Dittrich, Hatzinger, Katzenbeisser) – WU Wien – 6.11.2003 11
CONTINOUS SUBJECT COVARIATES ( Francis, Dittrich, Hatzinger, Penn, J. Royal Statistical Society, C, 2002 )
we need to extend the model on individual level
FIRST STEP: we model ,i jλ by 1
,
Q
j jq qii jq
xλ βλ=
= +∑
for each object j there is a separate set of parameters β describing the effect of the subject covariates on that object SECOND STEP: add smooth non-linear subject effects
11, ( )
M
jq qi j
Q
j jq qiq M
iqf x xλλ β
= = += + +∑ ∑
, ,,( ) ,ln i jk i j i jk i kp λα λ= + −
PAIRED COMPARISONS (Dittrich, Hatzinger, Katzenbeisser) – WU Wien – 6.11.2003 12
EXAMPLE: VALUE ORIENTATION IN EUROPE
Questions come from Inglehart(1990) ’Cultural shift in Advanced Industrial Society’ to assess postmaterialist and materialist values.
4 Inglehart Items: post-materialist items:
‘more say in government’ ‘protect freedom of speech’
materialist items ’maintain order in the nation’ ‘fight rising prices’ several sociological hypotheses: e.g., “Younger Europeans have been raised in relative prosperity, and are shifting towards postmaterial values” Data from ISSP (International Social Survey Program) 1993 • Social survey carried out each year since 1984 in 21 countries.
Same questions in each country • we analysed five European countries and look at the Inglehart items
important to sociologists – collected in every modern major social survey
PAIRED COMPARISONS (Dittrich, Hatzinger, Katzenbeisser) – WU Wien – 6.11.2003 13
MATERIALISM / POSTMATERIALISM: AGE EFFECT FOR SELECTED COUNTRIES
20 30 40 50 60 70 80 900.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9All countries
age
ORDER
SAYPRICES
SPEECH
20 30 40 50 60 70 80 90
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9Poland
age
ORDER
SAY
PRICES
SPEECH
20 30 40 50 60 70 80 900.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9West Germany
age
ORDER
SAY
PRICES
SPEECH
20 30 40 50 60 70 80 90
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9Great Britain
age
ORDER
SAYPRICES
SPEECH
PAIRED COMPARISONS (Dittrich, Hatzinger, Katzenbeisser) – WU Wien – 6.11.2003 14
VARIOUS DATA STRUCTURES CAN BE ANALYSED USING PC MODELS by transforming them into paired comparison data
RANKINGS
respondent is asked to rank a set of J objects - can be converted to a set of paired comparison data
consider the ranking (3, 1, 2, 4, 5) for objects (A,B,C,D,E), where a low rank means a higher preference:
example: (A,B) ⇔ (3,1) implies that B is preferred (A,C) ⇔ (3,2) implies that C is preferred, …
this correspondence was used to analyse preferences for Austrian newspapers ( Dittrich, Katzenbeisser, Reisinger, OR Spectrum, 2000 )
PAIRED COMPARISONS (Dittrich, Hatzinger, Katzenbeisser) – WU Wien – 6.11.2003 15
PARTIAL RANKINGS ( Francis, Dittrich, Hatzinger, Penn, J. Royal Statistical Society, C, 2002 )
investigation of value orientation in Europe in ISSP respondents are asked to state highest and next highest priority - rank only a subset of 4 items (objects) e.g. (ORDER, SAY, PRICES, SPEECH) with response (1, na , 2, na ) leads to the following paired comparisons (ORDER, SAY) ⇔ (1, na) implies that ORDER is preferred to SAY (ORDER, PRICE) ⇔ (1, 2) implies that ORDER is preferred to PRICE … (SAY, SPEECH ⇔ (na,na) response treated as missing
RESPONSES TO ITEMS ON A LIKERT SCALE ( Dittrich, Hatzinger, Katzenbeisser; forthcoming)
CONTINOUS RESPONSES
( Francis, Soothill, Dittrich, Brit.J.Criminol, 2001)
PAIRED COMPARISONS (Dittrich, Hatzinger, Katzenbeisser) – WU Wien – 6.11.2003 16
MODELLING DEPENDENCIES IN PAIRED COMPARISON DATA - A LOG-LINEAR APPROACH
( Dittrich, Hatzinger, Katzenbeisser, Comp.Stat.&Data Analysis, 2002 ) main assumptions so far was the independence between comparisons we assume that dependencies between responses may arise from repeated evaluation of the same objects in paired comparisons e.g., comparing (object j with object k ) and (object j with object l )
the assessment of object j might be similar in both comparisons in general the result of paired comparisons can be represented by a
response pattern vector: , , ........, ..........,(12) (13) (1 ), ( 1, )( )J J JY Y Y Y −=Y where for dichotomous responses
if object is preferred over
if object is preferred over
( )
1
1
jkjk
j k
O OY
O O
−=
PAIRED COMPARISONS (Dittrich, Hatzinger, Katzenbeisser) – WU Wien – 6.11.2003 17
from
( )( 1)j kj j
jkj k k j j k k
P Y cπ ππ π
π π π π π π π= = = =+ +
the model of independence for a certain response pattern is
where ∆ is a normalizing constant
to model dependencies we include terms of the form ( ),( ) ( ) ( )exp{ }jk jl jk jly yθ , where pairs of comparisons have one object in common.
the parameter ( ),( )jk jlθ can be interpreted as proportional to the log-odds ratio of
( )jkY and ( )jlY given the rest (conditional odds ratio) this is similar to the approach taken by Cox (1972) in modelling multivariate binary data
( )
( )jky
j
kj kP
π∆ π<
= = ∏Y y
PAIRED COMPARISONS (Dittrich, Hatzinger, Katzenbeisser) – WU Wien – 6.11.2003 18
RECENT WORK AND OUTLOOK
MODELLING REPEATED PAIRED COMPARISONS ( Dittrich, Francis, Katzenbeisser (2003) Proceedings of 18th IWSM )
comparisons of the same objects by the same judges are made on more than one occasion British Household Panel Study: follow-up of concerns about topical issues (unemployment, ozone layer, decline in moral standards)
A PAIRED COMPARISON APPROACH FOR THE ANALYSIS OF LIKERT SCALE DATA
( Dittrich, Hatzinger, Katzenbeisser; forthcoming )
Likert scale responses are transformed into ordinal paired comparison responses - log linear version of the adjacent categories models for ordinal paired comparisons Motives of students to obtain a doctoral degree at the WU: importance of motives for various groups of students
PAIRED COMPARISONS (Dittrich, Hatzinger, Katzenbeisser) – WU Wien – 6.11.2003 19
A LOG LINEAR REPRESENTATION FOR DEPENDENT MULTIVARIATE COMPARISONS
judges are asked to compare the objects on more than one attribute • dependencies between the decisions of the judges • association structures between the attributes
location of Austrian party leaders in twodimensional preference space (social and economic competence)
RANDOM EFFECTS – LATENT CLASS PC MODELS ( Francis, Dittrich (2000) Proceedings of 5th IC Logic and Methodology )
nonparametric ML estimation of subject effects aims of social work ranked by first year social work students from Australia, Canada, UK, and USA
dichotomous
INDEPENDENCE MODEL
(productmultinomial)
paired comparisons
OBJECT COVARIATES
categoricalmetric
SUBJECT COVARIATES
FIXED EFFECTScategorical
metricsmoothed
(GAMS)
SUBJECT COVARIATES
RANDOM EFFECTScategorical
metric
UNDECIDED/CATEGORY
EFFECT
POSITION EFFECT
LOCATION OF OBJECTS ON
PREFERENCE SCALE
STRUCTURE OBJECTS SUBJECTSRESPONSE STRUCTURE
OBSERVED DATA
OBSERVATION
DESIGN
CROSS-SECTIONAL
MULTI-DIMENSIONAL
REPEATED MEASURES
PERMUTATION OF PRESENTATION
OF OBJECTS
MODEL PARAMETERSRESPONSES
paired comparisons
with ties
ordinal paired comparisons
rankings
partial rankings
responses on Likert scale
trichotomous(nominal)
polytomous(ordinal)
DEPENDENCE MODEL
(loglinear, comparison patterns,transitive-
intransitive)
PAIRED COMPARISON MODELS OVERVIEW