True and Error Models of Response Variation in Judgment and Decision Tasks Michael H. Birnbaum.

True and Error Models of Response Variation in

Judgment and Decision Tasks

Michael H. Birnbaum

OverviewI review three papers that are available at my Website that involve application and evaluation of models of variability in response to choice problems.•Two papers are co-authored with Jeff Bahra on with tests of transitivity, stochastic dominance, and restricted branch independence. •Our findings consistently rule out assumptions of iid that are required in certain models, such as the approach of Regenwetter, Dana, and Davis-Stober (2011) Psych Review.•Transitivity is often satisfied, but a few show evidence of intransitive preferences.•No individual satisfied CPT or the priority heuristic.

Testing Algebraic Models with Error-Filled Data

• Algebraic models assume or imply formal properties such as stochastic dominance, coalescing, transitivity, gain-loss separability, etc.

• But these properties will not hold if data contain “error.”

Some Proposed Solutions

• Neo-Bayesian approach (Myung, Karabatsos, & Iverson.

• Cognitive process approach (Busemeyer)• “Error” Theory (“Error Story”) approach

(Thurstone, Luce) combined with algebraic models.

• Random preference model: choices independent; no errors: variability due to iid sampling from mixture. Loomes & Sugden.

Variations of Error Models

• Thurstone, Luce: errors related to separation between subjective values. Case V: SST (scalability).

• Harless & Camerer: errors assumed to be equal for certain choices.

• Sopher & Gigliotti: Allow each choice to have a different rate of error, assumed transitivity.

• Birnbaum proposed using repetitions within block as estimates of error rates. Birnbaum & Gutierrez, 2007; Birnbaum & Schmidt, 2008.

Basic Assumptions of TE model (2 errors model)

• Each choice in an experiment has a true choice probability, p, and an error rate, e.

• The error rate is estimated from (and is the “reason” given for) inconsistency of response to the same choice by same person over repetitions

One Choice, Two Repetitions

A B

A

B

pe2 (1 p)(1 e)2 p(1 e)e (1 p)(1 e)e

p(1 e)e (1 p)(1 e)e

p(1 e)2 (1 p)e2

Choices are not Independent

• In this model, choices are not independent, in general.

• If there is a mixture of true preferences, there will be violations of independence.

• This contrasts with the assumption of iid used by Regenwetter and colleagues.

Solution for e

• The proportion of preference reversals between repetitions allows an estimate of e.

• Both off-diagonal entries should be equal, and are equal to:

(1 e)e

Estimating eProbability of Reversals in Repeated Choice

0

0.1

0.2

0.3

0.4

0.5

0 0.1 0.2 0.3 0.4 0.5

Error Rate (e)

Estimating p

Observed = P(1 - e)(1 - e)+(1 - P)ee

0.00

0.20

0.40

0.60

0.80

1.00

0.00 0.20 0.40 0.60 0.80 1.00

True Choice Probabiity, P

Error Rate = 0Error Rate = .02Error Rate = .04Error Rate = .06Error Rate = .08Error Rate = .10Error Rate = .12Error Rate = .14Error Rate = .16Error Rate = .18Error Rate = .20Error Rate = .22Error Rate = .24Error Rate = .26Error Rate = .28Error Rate = .30Error Rate = .32Error Rate = .34Error Rate = .36Error Rate = .38Error Rate = .40Error Rate = .42Error Rate = .44Error Rate = .46Error Rate = .48Error Rate = .50

Testing if p = 0

Test if P = 0

0

0.1

0.2

0 0.1 0.2 0.3 0.4 0.5

Probability of Reversals 2e(1 - e)

True and Error Model to Individuals

• When applied to individuals, it is assumed that each person has a “true” set of preferences within a trial block.

• True preferences might differ between blocks, if the person has a mixture. If so, violates independence.

• A mixture could arise if a person’s parameters change in response to experience.

By Testing Individuals…• we can tailor the experiment (or

devise a fish net) to “catch” violations that otherwise slip through a static study of a property.

• we can see if people “learn” from internal feedback and change their behavior.

• we can test if a model that holds for one property can predict the results of other tests.

Recent Studies with Jeffrey Bahra

• Tested participants with many replications.

• Tests of transitivity• Basic tests of critical properties,

including stochastic dominance, coalescing, LCI, UCI, and others.

• Today: RBI and SD (+ transitivity)

Stochastic Dominance

A: 10 tickets to win $10 5 tickets to win $90 85 tickets to win $98

B: 5 tickets to win $10 5 tickets to win $12 90 tickets to win $99Each person received two such problems in

each repetition block of 107 choice problems. Blocks were separated by at least 50 unrelated choice problems.

SD is a critical property

• This test of stochastic dominance lies outside the probability simplex on three branch gambles.

• CPT with ANY strictly monotonic utility function and decumulative weighting function must satisfy stochastic dominance in this choice.

• We don’t need to estimate any parameters or assume any particular functions to refute CPT.

Non-nested Models

Testing CPT

• Coalescing• Stochastic

Dominance• Lower Cum.

Independence• Upper

Cumulative Independence

• Upper Tail Independence

• Gain-Loss Separability

TAX:Violations of:

Testing TAX Model

• 4-Distribution Independence (RS’)

• 3-Lower Distribution Independence

• 3-2 Lower Distribution Independence

• 3-Upper Distribution Independence (RS’)

• Res. Branch Indep (RS’)

CPT: Violations of:

Critical Tests: LS Models

• CPT and TAX satisfy transitivity• LS Models violate transitivity (includes PH)• LS Models satisfy priority dominance,

integrative independence, and interactive independence. This properties systematically violated. See my JMP 2010 article.

• PH satisfies SD in these tests and it also violates RBI in the same way as CPT.

Restricted Branch Independence

• Weaker version of Savage’s “sure thing” ax.

• 3 equally likely events: slips in urn.• (x, y, z) := prizes x, y, or z, x < y < z

• RBI: (x, y, z) f (x', y', z)

(x, y, z') f (x', y', z')

TAX, CPT, PH Violate RBI

• 0 < z < x' < x < y < y' < z'• (x, y) is “Safe”, S• (x', y') is “Risky”, R• (z, x, y) f (z, x', y') wLu(z) + wMu(x) + wHu(y) >

wLu(z) + wMu(x') + wHu(y')

TAX: SR' Violations

wLwM

< u(y') - u(y)u(x) - u(x')

< wMwH

SR': S = (x, y, z) R = (x', y', z) AND

S' = (x, y, z') R' = (x', y', z')

S = (z, x, y) vs R = (z, 5, 95)

RBI distinguishes models of RDM

•EU and SWU (Edwards, 54) imply RBI•Original prospect theory implies RBI•Cumulative Prospect theory violates RBI•CPT with inverse-S weighting function implies RS’ pattern of violations• TAX model violates RBI with SR’ pattern

Testing iid Assumptions

Each person’s data: rows represent trial blocks and the columns represent choice problems. Smith & Batchelder (2008) technique: random permutations within columns for each person. We then calculate two statistics on original data and on 10,000 simulations of the data. Variance of preference reversals and correlation between mean preference reversals and difference between repetitions.

A Test of Independence & Stationarity

• Within each subject, calculate the number of reversals between each pair of repetitions. 107 choices.

• 20 reps, so 20*19/2 = 190 pairs of reps.• Correlate the average no. reversals with

the difference in reps. For 59 participants, mean r = 0.69. Only 6 were negative. Similar results for Regenwetter et al data.

• Second test: Variance of no. reversals.

Tests of Independence

Transitivity Findings• Results were surprisingly transitive.• True and Error Model Fit data fairly well.• Data violate independence of choices.• People differ in true preferences and

people differ in “noise” levels.• No one satisfied the priority heuristic• One person satisfied linked viols of

transitivity consistent with LS models.

Very Few Intransitive Cases

• No one showed pattern predicted by PH in all three designs.

• Of the 59*3 = 177 Matrices, perhaps 4 show credible evidence of intransitivity.

• This change of procedure did not produce the higher rates of intransitivity conjectured.

Allais CC Paradox (JMP’04)• Choose Between: A = ($40, 0.2; $2, 0.8) B = ($98, 0.1; $2, 0.9)• Choose Between: C = ($98, 0.8; $40, 0.2) D = ($98, 0.9; $2, 0.1)• Many Choose B > A and C > D

Analysis of the Paradox: ($40, .2; $2, .8) f ($98, .1; $2, .9)

(Coalescing)

($40, .1;$40, .1;$2, .8) f ($98, .1;$2,.1;$2, .8)

(RBI)

($40, .1;$40, .1;$98, .8) f ($98, .1;$2,.1;$98, .8)

(Coalescing)

($98, .8; $40, 0.2) f ($98, .9; $2, .1)

Four Theories Compared

RBI holds (**cancellation)

RBI fails

Coalescing holds (*combination)

EU, CPT**, OPT*

CPTInverse-S => RS’

Coalescing fails

OPT TAX, RAMSR’

Error Models & EU

• One error model: P(SR’) = P(RS’)• Two error model allows P(RS’)>P(SR’)• Two error model: P(R’) > 1/2 implies P(R) > 1/2• Four error model allows above• All models imply P(SR) = P(RS);

P(SS,SR’)=P(SR’,SS), etc.

Results (refute 4 error EU)

SS’ SR’ RS’ RR’

SS’ 36 7 21 3

SR’ 5 4 1 3

RS’ 18 4 69 11

RR’ 1 2 12 14

Results (split form RBI)

SS’ SR’ RS’ RR’

SS’ 42 24 4 1

SR’ 17 21 2 16

RS’ 7 4 14 7

RR’ 1 6 6 39

Summary

• Most people conform to transitivity• A very small number do not, but not

consistently in all three designs as predicted by LS models.

• Data appear to violate the assumptions of random utility model; in particular, they show evidence against the assumptions of i.i.d.

Summary (cont’d)

• No one satisfied CPT, except those consistent with EU

• No one satisfied PH• People appear to change between

blocks. We suspect that parameters change systematically during the study.

• Suspect TE model oversimplified and incomplete.

Available at my Website

• Tests of LS models (JMP 2010)• Psych Review Comment RP Model (2011)• Reanalysis of Regenwetter et al data

testing iid, in JDM.• Birnbaum & Bahra: Testing transitivity in

Linked Designs (submitted)• Birnbaum & Bahra: Testing SD and RBI

(submitted)• Soon: Birnbaum & Schmidt Allais

paradoxes with 4 errors model