Dennis D. Leber National Institute of Standards and Technology Jeffrey W. Herrmann University of Maryland Science of Test Workshop – Springfield, VA April 4, 2017 Allocating Information Gathering Efforts for Selection Decisions
Dennis D. Leber
National Institute of Standards and Technology
Jeffrey W. Herrmann
University of Maryland
Science of Test Workshop – Springfield, VA
April 4, 2017
Allocating Information Gathering
Efforts for Selection Decisions
of 15
Selecting a Radiation Detection System
1
Performance Operational Impact Cost
Pd SNM (e.g., WGPu, HEU)
Pd Industrial (e.g., 137Cs, 57Co)
Pd Medical (e.g., 131I, 201Tl )
Probability of false alarm
of 15
Challenge
2
How do we allocate our limited and fixed budget for
information gathering to maximize the probability
of selecting the true best alternative?
of 15
Selection Decision Terminology
3
Alternatives
a1: “townhouse”
a2 : “farmhouse”
a3 : “country house”
Attributes (value)
1: House size (mi1)
2: Lot size (mi2)
3: Cost (mi3)
4: Distance to work (mi4)
5: Quality of school (mi5)
Preferences and Decision Model
Mike Mulligan and His Stream Shovel
Virginia Lee Burton (1939)
1 1 1 2 2 2 3 3 3 4 4 4 5 5 5i i i i i iv v v v v m m m m m
of 15
Best Alternative
4
Alternative that provides the largest decision value
Function of:
1. Decision-maker’s preferences
2. True attribute values
Knowledge of decision values is uncertain
Uncertainty in attribute values is a function of amount of information gathered
1 1 1 2 2 2i i i k k ikv v v m m m
1 1 1 2 2 2ˆ ˆ ˆ ˆi i i k k ikv v v m m m
-------------- Estimate --------------
of 15
Selecting an Alternative
5
Probability that alternative ai has largest decision value: 𝑝𝑖 = 𝑃 𝜉𝑖 > 𝜉𝑟, ∀𝑟 = 1,… ,𝑚
Select as, where 𝑠 = argmax𝑖
𝑝𝑖
Define Probability of Correct Selection: PCSDM = 𝑝𝑠
of 15
Information Gathering Layout
6
Assume equal cost for all observations
B is observation budget
Performance
Measure 1
Performance
Measure 2…
Performance
Measure k
Alternative a1 𝑛11 n12 n1k
Alternative a2 𝑛21 n22 n2k
… … … …
Alternative am 𝑛𝑚1 nm2 nmk
of 15
Uniform Allocation
7
Performance
Measure 1
Performance
Measure 2…
Performance
Measure k
Alternative a1 𝑛11 =𝐵
𝑚𝑘𝑛12 =
𝐵
𝑚𝑘𝑛1𝑘 =
𝐵
𝑚𝑘
Alternative a2 𝑛21 =𝐵
𝑚𝑘𝑛22 =
𝐵
𝑚𝑘𝑛2𝑘 =
𝐵
𝑚𝑘
… … … …
Alternative am 𝑛𝑚1 =
𝐵
𝑚𝑘𝑛𝑚2 =
𝐵
𝑚𝑘𝑛𝑚𝑘 =
𝐵
𝑚𝑘
of 15
Proportional Allocation
8
Performance
Measure 1
Performance
Measure 2…
Performance
Measure k
Alternative a1 𝑛11 = 𝜆1𝐵
𝑚𝑛12 = 𝜆2
𝐵
𝑚𝑛1𝑘 = 𝜆𝑘
𝐵
𝑚
Alternative a2 𝑛21 = 𝜆1𝐵
𝑚𝑛22 = 𝜆2
𝐵
𝑚𝑛2𝑘 = 𝜆𝑘
𝐵
𝑚
… … … …
Alternative am 𝑛𝑚1 =𝜆1
𝐵
𝑚𝑛𝑚2 =
𝜆2𝐵
𝑚𝑛𝑚𝑘 = 𝜆𝑘
𝐵
𝑚
1 1 1 2 2 2i i i k k ikv v v m m m
of 15
Sequential Information Gathering
9
of 15
Ranking and Selection
10
Statistics
Bechhofer, R. E. (1954). A Single-sample Multiple Decision Procedure for
Ranking Means of Normal Populations with Known Variances. The Annals of
Mathematical Statistics, 25, 16-39.
Bechhofer, R. E., Santer, T. J., & Goldsman, D. M. (1995). Design and Analysis of
Experiments for Statistical Selection, Screening, and Multiple Comparisons. New
York: John Wiley and Sons, Inc.
Computer Simulation Kim, S.-H., & Nelson, B. L. (2006). Selecting the Best System. In S. G.
Henderson, & B. L. Nelson (Eds.), Handbook in Operations Research and Management Science (Vol. 13, pp. 501-534). Oxford: Elsevier.
Chen, C.-H., & Lee, L. H. (2011). Stochastic Simulation Optimization: An Optimal Computing Budget Allocation. Hackensack, NJ: World Scientific Publishing Co. Pte. Ltd.
of 15
Amalgamation
11
𝑛𝑖𝑗 𝑡
Prior knowledge of
attribute values, 𝜇𝑖𝑗’s Knowledge of
Decision-maker
preferences, 𝜆𝑗Updated knowledge of
attribute values, 𝜇𝑖𝑗’s with collected data, 𝑿 𝒕
Prior knowledge of ability
to measure attribute
values, U Ƹ𝜇𝑖𝑗 = 𝜎𝑗’s
Update knowledge of ability
to measure attribute values,
𝜎𝑗’s with collected data, 𝑿 𝒕
of 15
Sequential Allocation Procedure
12
Data collected thus far for alternative ai and attribute j: 𝐱𝑖𝑗 𝑡 = 𝑥𝑖𝑗1, … , 𝑥𝑖𝑗𝑛𝑖𝑗 𝑡
Knowledge of value of attribute j for alternative ai: 𝑝 𝜇𝑖𝑗|𝐱𝑖𝑗 𝑡
Knowledge of decision value for alternative ai: 𝑝 𝜉𝑖|𝐱𝑖 𝑡
Calculate PCSDM 𝑡 = 𝑃 𝜉𝑠 > 𝜉𝑟 , ∀𝑟 = 1,… ,𝑚|𝐗 𝑡
of 15
Allocation Decision
13
Next sample
Posterior predictive distribution
Allocate sample to 𝑎𝑖, 𝑗 with
largest Expected PCS𝐷𝑀𝑖𝑗
𝑡 + 1
11
ijijDMijn t
x PCS t
1|
ijijijn t
p x t
x
1 11 1 11, , ,
1, ,
1 max , , | , |ij ij ij ij
q r
DM m m iijn t ijn t ijn tq m r q
r m
E PCS t p t x d d p x t dx
X x
of 15
Allocation Procedure Performance
14
Evaluation Experiment:
50,000 decision cases
Concave efficient frontier
𝑚 = 5 alternatives
𝑘 = 2 attributes
100 ≤ 𝜇𝑖𝑗 ≤ 200
19 decision models, 𝜆1, 𝜆2 pairs
𝑣𝑗 𝜇𝑖𝑗 = 𝜇𝑖𝑗 Gaussian measurement error (known)
Bayesian prior on attribute values
𝑝 𝜇𝑖𝑗 = 𝑁 150, 352
Experimental budget 𝐵 = 50
Observed frequency of correct selection
of 15
Summary
15
Allocating a fixed experimental budget across multiple attributes and
alternatives in a selection decision where the results of the experimental
evaluations lead to uncertain estimates of the true attribute values
Allocation approaches:
1. Uniform
2. Proportional
3. Sequential
Allocation does impact the probability of selecting the true best alternative
Importance for projects focused on a selection decision to be managed so
that the decision modeling and the experimental planning are done jointly
rather than in isolation
BACKUP
16
of 15
How should we allocate a fixed budget across multiple attributes
and alternatives to maximize the probability of correct selection?
Assumptions and Problem Statement
17
1. Finite and distinct set of alternatives
attributes; true value
Separate and independent attribute measurement processes
with known
2. Decision model is provided
Linear
3. Fixed experimental budget
B sample measurements; cost equivalent
1, , ma a
1 1, ,
k
i i ik j j ijjf v m m m
2k ijm
ijl ij ijlX m
2~ 0,ijl jN 2
j
j ij ijv m m1
1k
jj
of 15
Bayesian Estimation
18
Estimate by estimating each
Decision-maker’s prior knowledge
Data
Posterior distribution on
Posterior distribution on
ijm
2
0 0,ij ijN m i
1, ,ijij ijnx x
ijm
i
2 2 2 2
0 0 0
1 2 2 2 2
0 0
| , , ~ ,ij
j ij ij ij ij j ij
ij ij ijn
j ij ij j ij ij
n xx x N
n n
m m
2 2 2 2
0 0 02
2 2 2 210 0
| ~ ,k k
j ij ij ij ij j ij
i i j j
j i jj ij ij j ij ij
n xN
n n
m
x
of 15
Results
19