Information Elicitation Sans Veri cation - Bo Waggonerchen--2013--slides.pdf · Information Elicitation Sans Veri cation Bo Waggoner and Yiling Chen 2013-06-16 1/33. ... Bayesian

Information Elicitation SansVerification

Bo Waggoner and Yiling Chen

2013-06-161 / 33

Motivation: human computation

2 / 33


2 / 33


2 / 33

Goal: design systems for eliciting info

Question: How to construct human computationsystems?Approach: Use mechanism design

3 / 33


Question: How to construct human computationsystems?

Approach: Use mechanism design

3 / 33


Question: How to construct human computationsystems?Approach: Use mechanism design

3 / 33

Mechanism design

Mechanism design:Construct a game to optimize an objective

Game: different actions available; set of actionsmaps to an outcome and payoffs.

4 / 33

Mechanism design


Game: different actions available; set of actionsmaps to an outcome and payoffs.

4 / 33

Mechanism design


Our objective: elicit “useful” information

Our constraints:1 players may not prefer “useful” responses2 designer cannot always verify responses

Our name for this setting:Information Elicitation Without Verification(IEWV)

5 / 33

Mechanism design





5 / 33

Mechanism design



Our constraints:

1 players may not prefer “useful” responses2 designer cannot always verify responses


5 / 33

Mechanism design



Our constraints:1 players may not prefer “useful” responses

2 designer cannot always verify responses


5 / 33

Mechanism design





5 / 33

Mechanism design





5 / 33

Agenda

Plan:1 Formally define the setting,

identify limitations of prior work.

2 Prove impossibility results on the setting;demonstrate difficulty of overcominglimitations.

3 Propose new mechanism that overcomes somelimitations, avoids some impossibilities.

6 / 33

Agenda


identify limitations of prior work.2 Prove impossibility results on the setting;

demonstrate difficulty of overcominglimitations.


6 / 33

Agenda


identify limitations of prior work.2 Prove impossibility results on the setting;

demonstrate difficulty of overcominglimitations.


6 / 33

Outline

Information elicitation without verification

Formal setting and prior work

Impossibility results for IEWV

Output agreement mechanisms

7 / 33

Outline





8 / 33

Setting

Game of information elicitation without verification:

9 / 33

Setting


9 / 33

prior

Setting


9 / 33

prior events

Setting


9 / 33

prior events posterior

Setting


9 / 33

prior events posterior report

Setting


9 / 33

prior events posterior report payoff

Prior work: themes

Prior work: various mechanisms for instances of thissetting:

Peer prediction (Miller, Resnick, Zeckhauser 2005)

Bayesian truth serum (Prelec 2004)

PP without a common prior, Robust BTS(Witkowski, Parkes 2012a,b)

Collective revelation (Goel, Reeves, Pennock 2009)

Truthful surveys (Lambert, Shoham 2008)

10 / 33

Prior work: themes







10 / 33

Prior work: themes







10 / 33

Prior work: themes







10 / 33

Prior work: themes







10 / 33

Example: peer prediction

11 / 33


11 / 33

Πi(ω∗) = A

observation


11 / 33

Πi(ω∗) = A

observation

Πi(ω∗) = A

report


11 / 33

Πi(ω∗) = A

observation

Πi(ω∗) = A

report

Pr [Πj(ω∗) | Πi(ω∗) = A]

prediction


11 / 33

Πi(ω∗) = A

observation

Πi(ω∗) = A

report

Pr [Πj(ω∗) | Πi(ω∗) = A]

prediction

Πj(ω∗) = B

payoff: h a proper scoring rule

h(Pr [Πj(ω∗) | Πi(ω

∗) = A] , B)

Prior work: discussion

Limitations of mechanisms in prior work:

Somewhat complicated to explain

Only applicable in specific settings (e.g. elicitsignals)

“Bad” equilibria exist

Not detail-free (peer prediction)

Restricted domain (all)

Goal: Overcome these limitations.Obstacle: Impossibility results!

12 / 33









12 / 33









12 / 33









12 / 33









12 / 33









12 / 33









12 / 33








Goal: Overcome these limitations.

Obstacle: Impossibility results!

12 / 33









12 / 33

Outline





13 / 33

Existence of uninformative equilibria

DefinitionA strategy is uninformative if it draws a reportfrom the same distribution in every state of theworld.

14 / 33


DefinitionA strategy is uninformative if it draws a reportfrom the same distribution in every state of theworld.

14 / 33


Proposition

The following mechanisms for IEWV always haveuninformative equilibria:

Those with compact action spaces andcontinuous reward functions;

Those that: (a) are detail-free and (b) alwayshave an equilibrium.

=⇒ All mechanisms we know of; all “reasonable”mechanisms.

15 / 33


Proposition

The following mechanisms for IEWV always haveuninformative equilibria:

Those with compact action spaces andcontinuous reward functions;

Those that: (a) are detail-free and (b) alwayshave an equilibrium.

=⇒ All mechanisms we know of; all “reasonable”mechanisms.

15 / 33

Impossibility for truthful equilibria

Q: What is “truthful”?

A: define a query T specifying the truthful responsefor a given posterior belief.

truthful strategy: si(Πi(ω∗)) = T (Πi(ω

∗)).truthful equilibrium: (Given T ) one in which eachsi is truthful.

TheoremFor all detail-free M and all queries T , there existsI such that G = (M, I) has no strict truthfulequilibrium.

16 / 33







16 / 33







16 / 33





∗)).

truthful equilibrium: (Given T ) one in which eachsi is truthful.


16 / 33







16 / 33







16 / 33

How to get around this result?

Goal: overcome limitations of prior mechanisms.

Obstacle: Impossibility result!

Proposed solution: Output agreementmechanisms.

simple to explain and implement

applicable in variety of complex domains

detail-free

unrestricted domain

... but not truthful!

17 / 33


Goal: overcome limitations of prior mechanisms.Obstacle: Impossibility result!




detail-free

unrestricted domain


17 / 33






detail-free

unrestricted domain


17 / 33






detail-free

unrestricted domain


17 / 33






detail-free

unrestricted domain


17 / 33






detail-free

unrestricted domain


17 / 33






detail-free

unrestricted domain


17 / 33

Outline





18 / 33

Output agreement

Truthful → common-knowledge truthful:

19 / 33

Common Knowledge

20 / 33

Ω: possible states of the world

Common Knowledge

20 / 33

Ω: possible states of the world

P [ω]: common prior

Common Knowledge

20 / 33

Π1: player 1’s partition

Common Knowledge

20 / 33

Π1

ω∗: true state selected by nature

Common Knowledge

20 / 33

Π1

Π1(ω∗): player 1’s signal

Pr [ω | Π1(ω∗)]: player 1’s posterior

Common Knowledge

20 / 33

Π1 Π2

Common Knowledge

20 / 33

Π1 Π2

Π: common-knowledge partition

Output agreement

Truthful → common-knowledge truthful:si(Πi(ω

∗)) = T (Π(ω∗)).Previously: = T (Πi(ω

∗)).

21 / 33

Output agreement: Origins

Output agreement: informally coined by von Ahn,Dabbish 2004.

Game-theoretic analysis of ESP Game: Jain, Parkes2008. (Specific agent model, not general output agreement

framework.)

Here: first general formalization of outputagreement.

22 / 33




framework.)


22 / 33




framework.)


22 / 33

Output agreement

An output agreement mechanism:

23 / 33

Output agreement


23 / 33

report space: A

a1 a2

Output agreement


23 / 33

d(a1, a2)

report space: (A, d)

a1 a2

Output agreement


23 / 33

d(a1, a2)

report space: (A, d)

payoff: h strictlydecreasing

h(d) h(d)

a1 a2

Output agreement

TheoremFor any query T , there is an output agreementmechanism M eliciting a strictcommon-knowledge-truthful equilibrium.

24 / 33

Proof by picture

25 / 33

Π1 Π2 Π

Are “good” equilibria played?

What is “focal” in output agreement?

26 / 33


What is “focal” in output agreement?One approach: player inference, beginning withtruthful strategy.

26 / 33



26 / 33



26 / 33



26 / 33

Output agreement

Inference: iteratively compute strategy thatmaximizes expected utility.

When does inference, starting with truthfulness,converge to common-knowledge truthfulness?

Eliciting the mean: Yes!

Eliciting the median, mode: No!(arbitrarily bad examples)

27 / 33

Output agreement





27 / 33

Output agreement





27 / 33

Output agreement




Eliciting the median, mode: No!

(arbitrarily bad examples)

27 / 33

Output agreement





27 / 33

Output agreement

Mechanisms on many players?

(Yes)

28 / 33

Output agreement

Mechanisms on many players? (Yes)

28 / 33

Outline


Setting

Impossibility results

Output agreement

29 / 33

Summary

IEWV: formalized mechanism design setting.

(Almost) all mechanisms have bad equilibria.

There are no detail-free, unrestricted-domain,truthful mechanisms.

Output agreement:simpleapplicable in complex domainsdetail-free, unrestricted-domainelicits common knowledge

Thanks!

30 / 33

Summary





Thanks!

30 / 33

Summary





Thanks!

30 / 33

Summary





Thanks!

30 / 33

Summary




Output agreement:

simpleapplicable in complex domainsdetail-free, unrestricted-domainelicits common knowledge

Thanks!

30 / 33

Summary




Output agreement:simple

applicable in complex domainsdetail-free, unrestricted-domainelicits common knowledge

Thanks!

30 / 33

Summary




Output agreement:simpleapplicable in complex domains

detail-free, unrestricted-domainelicits common knowledge

Thanks!

30 / 33

Summary




Output agreement:simpleapplicable in complex domainsdetail-free, unrestricted-domain

elicits common knowledge

Thanks!

30 / 33

Summary





Thanks!

30 / 33

Summary





Thanks!

30 / 33

Information Elicitation Sans Veri cation - Bo Waggonerchen--2013--slides.pdf · Information Elicitation Sans Veri cation Bo Waggoner and Yiling Chen 2013-06-16 1/33. ... Bayesian

Documents