An Open Problem: Characterizing Random Utility Models of Best-Worst Choice A. J. Marley

An Open Problem: Characterizing Random Utility Models of Best-Worst Choice

A. A. J. Marley

University of Victoria, Victoria

DIMACS Workshop on Polyhedral Combinatorics of Random Utility, May 2006

Research Supported by

• The Australian Research Council (ARC)

• The Natural Sciences and Engineering Research Council (Canada)

Summary

• An illustrative empirical study

• Definitions and theoretical examples

• Random utility models for best-worst choice

• Characterizing random utility models of best worst choice

• (Further theoretical results on Thurstonian models of best-worst choice)

Public Issues

• Preserving the environment• Quality medical care• Cost of living• Crime and public safety• National unity/Canada’s future• Level of taxation• Poverty in Canada• Drinking and driving• A quality food supply• The drug problem• Food safety

Question & Layout for Best-Worst Task (example shows 2 questions)

Question & Layout Used for First Best-Worst Task

Identify the public issue that causes you the most concern and the public issue that causes you the least concern in each of the following sets of issues.

Check only one issue for each of the Least and Most Columns.

LEAST ISSUE MOST LEAST ISSUE MOST

  Preserving the environment     Cost of living  

  Level of Taxation     A quality food supply  

  Quality medical care     Quality medical care  

  The drug problem     Poverty in Canada  

Fractional factorial design for choice sets (Best-Worst task)

Food safety 0 0 1 0 0 1 1 1 0 1 0 1 6

Preserving the environment 1 0 0 1 0 0 0 1 1 1 0 1 6

Cost of living 0 1 0 0 1 0 1 0 1 1 0 1 6

Level of taxation 1 0 1 0 0 0 1 0 1 1 1 0 6

National unity/Canada’s future 0 0 1 0 1 0 0 1 1 0 1 1 6

Crime & public safety 0 0 0 1 0 1 1 0 1 0 1 1 6

A quality food supply 0 1 1 1 0 0 0 0 0 1 1 1 6

Quality medical care 1 1 0 0 0 0 1 1 0 0 1 1 6

Drinking & driving 0 0 0 1 1 0 1 1 0 1 1 0 6

The drug problem 1 0 0 0 1 1 0 0 0 1 1 1 6

Poverty in Canada 0 1 0 0 0 1 0 1 1 1 1 0 6

# Issues in set 4 4 4 4 4 4 6 6 8 8 8 8  

Issue 1 2 3 4 5 6 7 8 9 10 11 12# timespresent

Comparison of Most-Least totals Vs MNL estimates

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

2.5

-1.000 -0.500 0.000 0.500 1.000 1.500

MNL Estimates

Mo

st-

Le

as

t T

ota

ls

Summary

• An illustrative empirical study

• Definitions and theoretical examples

• Random utility models for best-worst choice

• Characterizing random utility models of best worst choice

• (Further theoretical results on Thurstonian models of best-worst choice)

Consistent Random Ranking Models

And the Open Problem

Observations

• A consistent random ranking model on (all the subsets of) a set T has consistent margins

• Conversely, if a set of best-worst choice probabilities satisfy a random ranking model (next slide), then one can define a set of best and worst choice probabilities through the margins of the ranking probabilities

• Thus, the major task is to characterize a random ranking model on best-worst choice probabilities

Summary

• A major open problem is to characterize the class of (best, worst and) best-worst choice probabilities that satisfy a consistent random ranking model

• We have a characterization of the maxdiff model for best-worst choice. This model is the most frequently used in applications

• The maxdiff model (with its ‘natural’ extension to best and worst choices) is not compatible with a consistent random ranking model

Thank you for listening

Summary

• A major open problem is to characterize the class of consistent random ranking models of best, worst and best-worst choice

• We have a characterization of the maxdiff model for best-worst choice. This model is the most frequently used in applications

• The maxdiff model (with its ‘natural’ extension to best and worst choices) is not compatible with a consistent random ranking model

Thank you for listening