A Multi-Expert Scenario Analysis for Systematic Comparison of Expert Weighting Approaches * CEDM Annual Meeting Pittsburgh, PA May 20, 2012 Umit Guvenc, Mitchell Small, Granger Morgan Carnegie Mellon University *Work supported under a cooperative agreement between NSF and Carnegie Mellon University through the Center for Climate and Energy Decision Making (SES-0949710)
20
Embed
A Multi-Expert Scenario Analysis for Systematic Comparison of Expert Weighting Approaches *
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
A Multi-Expert Scenario Analysis for Systematic Comparison of Expert Weighting Approaches*
CEDM Annual Meeting Pittsburgh, PAMay 20, 2012
Umit Guvenc, Mitchell Small, Granger MorganCarnegie Mellon University
*Work supported under a cooperative agreement between NSF and Carnegie Mellon University through the Center for Climate and Energy Decision Making (SES-0949710)
Multi-Expert Weighting: A Common Challenge in Public Policy
• Within climate change context, many critical quantities and probability distributions elicited from multiple experts (e.g., climate sensitivity)
• No consensus on best methodology if one wanted to aggregate multiple, sometimes conflicting, expert opinions
• Critical to demonstrate advantages and disadvantages of different approaches under different circumstances
2
General Issues Regarding Multi-Expert Weighting
1. Should we aggregate expert judgments at all?2. If we do, should we use a differential weighting
scheme?3. If we do, should we use “seed questions” to assess
expert skill?4. If we do, how should we choose “appropriate” seed
questions?5. If we do, how do different weighting schemes perform
under different circumstances?• Equal weights• Likelihood weights• “Classical” (Cooke) weights
3
Presentation Outline
1. Alternative Weighting Methods– Likelihood, “Classical”, Equal Weighting Schemes
• When Bias=0 for all and imprecision introduced to multiple experts, weights change to reward precision and penalize imprecision (more prominent in likelihood method)
• When Bias=0 for all and over- and under-confidence introduced to multiple experts, weights change to penalize inappropriate confidence (more prominent in likelihood method for under-confidence)
Scenario #5a: Impact of Precision & Confidence (Bias = 0 for all)
• When Bias=0 and imprecision and over-and under-confidence introduced to multiple experts• Weights change to reward “ideal” expert (more prominent in likelihood)• For “Classical”, proper confidence can somewhat compensate for imprecision, not so for
Likelihood (imprecise experts are penalized highly, even if they know they are imprecise)
Scenario #5b: Impact of Precision & Confidence(Bias for all)
• When bias for all, and varying amounts of precision and improper relative confidence introduced to multiple experts• Likelihood weights change to reward relatively precise, but underconfident experts• Classical weights shift to reward imprecise experts.
Scenario #5c: Precision & Confidence (Bias for 3 Experts)
• When there is moderate bias in a subset of “good” experts, and both imprecision and over-and under-confidence introduced to all• Likelihood rewards “best” expert significantly • Classical spreads weights across much more
Conclusions (1)• Overall: Likelihood and “Classical” similar performance (much
better than equal weights), but with very different weights assigned to experts with different degrees of bias, precision and relative confidence
• Model Check: Both assign equal weights to experts with equal skill (equal bias, precision, and relative confidence)
• Bias: Both penalize biased experts, stronger penalty in Likelihood
• Precision: Both penalize imprecise experts, but again stronger penalty in Likelihood
• Confidence: “Classical” penalizes overconfidence and underconfidence equally. Likelihood penalizes overconfidence a similar amount, but underconfidence much more so.
18
Conclusions (2)• Precision & Confidence: For “Classical”, proper (or under-)
confidence can compensate somewhat for imprecision, not so for the Likelihood weights (and over-confidence remains better for Likelihood weighting).
• Future Direction: Consider 3-parameter distributions to be fit from expert’s 5th, 50th, and 95th percentile values to enable a more flexible Likelihood approach– Conduct an elicitation in which 2- and 3-parameter likelihood
functions are used and compared.– Consider how new information affects experts' performance on seed
questions (explore VOI for correcting experts' biases, imprecision, and under- or overconfidence).