Testing Transitivity (and other Properties) Using a True and Error Model Michael H. Birnbaum
Testing Algebraic Models with Error-Filled Data
• Algebraic models assume or imply formal properties such as stochastic dominance, coalescing, transitivity, gain-loss separability, etc.
• But these properties will not hold if data contain “error.”
Some Proposed Solutions
• Neo-Bayesian approach (Myung, Karabatsos, & Iverson.
• Cognitive process approach (Busemeyer)
• “Error” Theory (“Error Story”) approach (Thurstone, Luce) combined with algebraic models.
Variations of Error Models
• Thurstone, Luce: errors related to separation between subjective values. Case V: SST (scalability).
• Harless & Camerer: errors assumed to be equal for certain choices.
• Today: Allow each choice to have a different rate of error.
• Advantage: we desire error theory that is both descriptive and neutral.
Basic Assumptions
• Each choice in an experiment has a true choice probability, p, and an error rate, e.
• The error rate is estimated from (and is the “reason” given for) inconsistency of response to the same choice by same person over repetitions
One Choice, Two Repetitions
A B
A
B€
pe2
+ ( 1 − p )( 1 − e )2
p ( 1 − e ) e + ( 1 − p )( 1 − e ) e
p ( 1 − e ) e + ( 1 − p )( 1 − e ) e
€
p ( 1 − e )2
+ ( 1 − p ) e2
Solution for e
• The proportion of preference reversals between repetitions allows an estimate of e.
• Both off-diagonal entries should be equal, and are equal to:
( 1 − e ) e
Estimating eProbability of Reversals in Repeated Choice
0
0.1
0.2
0.3
0.4
0.5
0 0.1 0.2 0.3 0.4 0.5
Error Rate (e)
Estimating p
Observed = P(1 - e)(1 - e)+(1 - P)ee
0.00
0.20
0.40
0.60
0.80
1.00
0.00 0.20 0.40 0.60 0.80 1.00
True Choice Probabiity, P
Error Rate = 0
Error Rate = .02
Error Rate = .04
Error Rate = .06
Error Rate = .08
Error Rate = .10
Error Rate = .12
Error Rate = .14
Error Rate = .16
Error Rate = .18
Error Rate = .20
Error Rate = .22
Error Rate = .24
Error Rate = .26
Error Rate = .28
Error Rate = .30
Error Rate = .32
Error Rate = .34
Error Rate = .36
Error Rate = .38
Error Rate = .40
Error Rate = .42
Error Rate = .44
Error Rate = .46
Error Rate = .48
Error Rate = .50
Ex: Stochastic Dominance
: 05 tickets to win $12
05 tickets to win $14
90 tickets to win $96
B: 10 tickets to win $12
05 tickets to win $90
85 tickets to win $96
122 Undergrads: 59% repeated viols (BB) 28% Preference Reversals (AB or BA) Estimates: e = 0.19; p = 0.85170 Experts: 35% repeated violations 31% Reversals Estimates: e = 0.196; p = 0.50 Chi-Squared test reject H0: p < 0.4
Testing 3-Choice Properties
• Extending this model to properties using 2, 3, or 4 choices is straightforward.
• Allow a different error rate on each choice.
• Allow a true probability for each choice pattern.
Response CombinationsNotation (A, B) (B, C) (C, A)
000 A B C *
001 A B A
010 A C C
011 A C A
100 B B C
101 B B A
110 B C C
111 B C A *
Weak Stochastic Transitivity
€
P ( A f B ) = P ( 000 ) + P ( 001 ) + P ( 010 ) + P ( 011 )
P ( B f C ) = P ( 000 ) + P ( 001 ) + P ( 100 ) + P ( 101 )
P ( C f A ) = P ( 000 ) + P ( 010 ) + P ( 100 ) + P ( 110 )
WST Can be Violated even when Everyone is Perfectly
Transitive
€
P ( 001 ) = P ( 010 ) = P ( 100 ) =1
3
€
P ( A f B ) = P ( B f C ) = P ( C f A ) =2
3
Model for Transitivity
€
P ( 000 ) = p000
( 1 − e1
)( 1 − e2
)( 1 − e3
) + p001
( 1 − e1
)( 1 − e2
) e3
+
+ p010
( 1 − e1
) e2
( 1 − e3
) + p011
( 1 − e1
) e2e
3+
+ p100
e1
( 1 − e2
)( 1 − e3
) + p101
e1
( 1 − e2
) e3
+
+ p110
e1e
2( 1 − e
3) + p
111e
1e
2e
3
A similar expression is written for the other seven probabilities. These can in turn be expanded to predict the probabilities of showing each pattern repeatedly.
Starmer (1999) data
• A = ($15, 0.2; $0, 0.8)
• B = ($8; 0.3; $0, 0.7)
• C = ($8, 0.15; $7.75; 0.15; $0, .7)
• Starmer predicted intransitivity from Prospect Theory and the dominance detection (editing) mechanism.
Starmer (Best) DataObserved Trans Intrans
Data Fitted (5) Fitted (6)
000 1 2.2 1.2
001 5 4.1 3.7
010 17 22.7 16.8
011 49 43.4 50.1
100 6 4.9 5.6
101 1 2.6 3.6
110 75 81.4 75.2
111 50 42.6 47.9
Transitive Solution to Starmer Data
€
e1
≡ 0
€
e2
= 0 . 06
€
e3
= 0 . 34
€
p010 = 0.01
€
p011 = 0.34; p110 = 0.65
Full model is underdetermined. One error Fixed to zero; but other errors not equal.Most people recognized dominance.
Expand and Simplify• There are 8 X 8 data patterns in an
experiment with 2 repetitions.• However, most of these have very small
probabilities.• Examine probabilities of each of 8
repeated patterns.• Probability of showing each of 8
patterns in one replicate OR the other, but NOT both. Mutually exclusive, exhaustive partition.
New Studies of Transitivity
• Work currently under way testing transitivity under same conditions as used in tests of other decision properties.
• Participants view choices via the WWW, click button beside the gamble they would prefer to play.
Some Recipes being Tested
• Tversky’s (1969) 5 gambles.• LS: Preds of Priority Heuristic• Starmer’s recipe• Additive Difference Model• Birnbaum, Patton, & Lott (1999)
recipe.
Tversky Gambles
• Some Sample Data, using Tversky’s 5 gambles, but formatted with tickets instead of pie charts.
• Data as of May 5, 2005, n = 123.• No pre-selection of participants.• Participants served in other
studies, prior to testing (~1 hr).
Three of the Gambles
• A = ($5.00, 0.29; $0, 0.79)• C = ($4.50, 0.38; $0, 0.62)• E = ($4.00, 0.46; $0, 0.54)
Results-ACEpattern Rep 1 Rep 2 Both
000 10 21 5
001 11 13 9
010 14 23 1
011 7 1 0
100 16 19 4
101 4 3 1
110 176 154 133
111 13 17 3
sum 251 251 156
Test of WSTA B C D E
A 0.712 0.762 0.771 0.852
B 0.339 0.696 0.798 0.786
C 0.174 0.287 0.696 0.770
D 0.101 0.194 0.244 0.593
E 0.148 0.182 0.171 0.349
Comments
• Preliminary results were surprisingly transitive.
• Difference: no pre-test, selection• Probability represented by # of
tickets (100 per urn)• Participants have practice with
variety of gambles, & choices.• Tested via Computer
Test of Gain-Loss Separability
• Same Structure as Transitivity• Property implied by CPT, RSDU• Property violated by TAX.• Loss Aversion: people do not like
fair bets to win or lose.• CPT: Loss Aversion due to utility
function for gains and losses.
Notation
€
x1
< x2
< K < xn
< 0 ≤ ym
< K y2
< y1
€
G = ( x1
, p1
; x2
, p2
; K ; xn
, pn
; ym
, qm
; K ; y2
, q2
; y1
, q1
)
€
G
+
= ( 0 , pi
i = 1
n
∑ ; ym
, pm
; K ; y2
, q2
; y1
, q1
)
€
G
−
= ( x1
, p1
; x2
, p2
; K ; xn
, pn
; 0 , q
i = 1
m
∑i
)
Birnbaum & Bahra--% FChoice % G Prior TAX Prior CPT
G F G F G F
25 black to win $100
25 white to win $0
50 white to win $0
25 blue to win $50
25 blue to win $50
50 white to win $0
0.71 14 21 25 19
50 white to lose $0
25 pink to lose $50
25 pink to lose $50
50 white to lose $0
25 white to lose $0
25 red to lose $100
0.65 -21 -14 -20 -25
25 black to win $100
25 white to win $0
25 pink to lose $50
25 pink to lose $50
25 blue to win $50
25 blue to win $50
25 white to lose $0
25 red to lose $100
0.52 -25 -25 -9 -15
25 black to win $100
25 white to win $0
50 pink to lose $50
50 blue to win $50
25 white to lose $0
25 red to lose $100
0.24 -15 -34 -9 -15
Birnbaum & BahraR 1 R 2 both OR!- “true”
000 16 17 5 11.5 0.08
001 5 6 0 5.5 0.00
010 24 29 12 14.5 0.18
011 3 5 0 4 0.00
100 36 30 10 23 0.10
101 8 8 1 7 0.00
110 63 54 29 29.5 0.51
111 23 29 9 17 0.13
178 178 66 112 1.00
Summary GLS
• Wu & Markle (2004) found evidence of violation of GLS. Modified CPT.
• Birnbaum & Bahra (2005) also find evidence of violation of GLS, violations of modified CPT as well.
• TAX: In mixed gambles, losses get greater weight. Data do not require kink in the utility function at zero.
Summary
• True & Error model with different error rates seems a reasonable “null” hypothesis for testing transitivity and other properties.
• Requires data with replications so that we can use each person’s self-agreement or reversals to estimate whether response patterns are “real” or due to “error.”