Crowdsourcing Control: Moving Beyond Multiple Choice (Full paper in UAI 2012)
Post on 22-Feb-2016
31 Views
Preview:
DESCRIPTION
Transcript
1
Crowdsourcing Control: Moving Beyond Multiple Choice
(Full paper in UAI 2012)
Christopher H. Lin, Mausam, Daniel S. WeldUniversity of Washington
HCOMP-2012, Toronto, ON, Canada
2
www.crowdsourcing.com
Is this a face?
Yes No
YesNo
3
www.crowdsourcing.com
Is this a face?
Yes No
YesNo
4
www.crowdsourcing.com
Is this a face?
Yes No
YesNo
5
www.crowdsourcing.com
Is this a face?
Yes No
YesNo
6
Update Posteriors
7
How many people are there in this military base?
www.crowdsourcing.com
30,000Submit
Source:Wikipedia
8
Update Posteriors
LAZYSUSAN
9
Update Posteriors
LAZYSUSAN
10
2 + 2 = ?
www.crowdsourcing.com
11
2 + 2 = ?
www.crowdsourcing.com
EASYd = 0
12
www.crowdsourcing.com
?=
13
www.crowdsourcing.com
?=HAR
Dd = 1
14Source:Wikipedia
15
Worker Ability
Difficulty
Accuracy
16[Aldous, 1985]
17
A
C
B
??
18
A
C
B
??
19
A
C
B
??
+
20
A
C
B
??
21
A
C
B
??
+
22
Correct Answer
DifficultyAgent State
23
24
LEARN NEW PARAMETERS
25
26
LazySusan MV
AverageAccuracy (%)
99.25 95.52
AverageUtility
-5.92 -9.94
27
28
LazySusan(EM)
MV
AverageAccuracy (%)
65.5 54.1
29
Contributions
• A probabilistic generative model for worker responses on tasks with an infinite number of possible answers
• A decision-theoretic agent that dynamically infers the right answer
• Real experiments that show our agent works
30
Repeated Labeling
Romney et al, 1986; Sheng et al, 2008; Raykar et al, 2010; Whitehill et al, 2009; Dai et al, 2010, 2011; Kamar et al, 2012; Parameswaran et al, 2010; Karger et al, 2011; Snow et al, 2008;
Complicated Workflows
Little et al, 2009; Bernstein et al, 2010; Liem et al, 2011; Noronha et al, 2011; Lasecki et al, 2011; Kulkarni et al, 2012;
AI Control of Workflows
Dai et al, 2010,2011; Shahaf et al, 2010; Weld et al, 2011;Kamar et al, 2012;
31
Christopher H. Lin, Mausam, and Daniel S. Weld. “Dynamically Switching between Synergistic Workflows.”
Wednesday, AI and the Web 22:25 PM
top related