Wisdom of Crowds in Human Memory: Reconstructing Events by Aggregating Memories across Individuals Mark Steyvers Department of Cognitive Sciences University of California, Irvine Joint work with: Brent Miller, Pernille Hemmer, Mike Yi Michael Lee, Bill Batchelder, Paolo Napoletano
84
Embed
Wisdom of Crowds in Human Memory: Reconstructing Events by Aggregating Memories across Individuals Mark Steyvers Department of Cognitive Sciences University.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Wisdom of Crowds in Human Memory: Reconstructing Events by Aggregating
Memories across Individuals
Mark Steyvers
Department of Cognitive Sciences
University of California, Irvine
Joint work with:Brent Miller, Pernille Hemmer, Mike Yi
Michael Lee, Bill Batchelder, Paolo Napoletano
Wisdom of crowds phenomenon
Group estimate often performs as well as or better than best individual in the group
2
Examples of wisdom of crowds phenomenon
3
Who wants to be a millionaire?Galton’s Ox (1907): Median of individual estimates comes close to true answer
Tasks studied in our research
Ordering/ranking problems declarative memory: order of US presidents, ranking cities by size episodic memory: order of events (i.e., serial recall) predictive rankings: fantasy football
Matching problems assign N items to N responses e.g., match paintings to artists, or flags to countries
Traveling Salesman problems find shortest route between cities
Australian Football League (29 people rank 16 teams)
1 10 20 300
5
10
15
20
25
Individuals
1 10 20 300
20
40
60
80
Part IIMatching Problems
49
Study these combinations
50
2 3 4 51
B C D EA
Find all matching pairs
51
Experiment
15 subjects
8 problems 4 problems with 5 items 4 problems with 10 items
52
Mean accuracy across 8 problems
53
1 2 3 4 5 6 7 8 9 10 11 12 13 14 150
0.2
0.4
0.6
0.8
1
Individuals
Mea
n A
ccur
acy
Bayesian Matching Model
Proposed process: match “known” items guess between remaining ones
Individual differences some items easier to know some participants know more
54
Graphical Model
55
i items
jx
jy
z
ja
Latent ground truth
Observed matching
Knowledge State
jsProb. of knowing
id
j individuals
logitj i js d a
~ Bernoulliij ijx s
1 1( )
1 / ! 0ij
ij ij ij
xp y z
n x
person abilityitem easiness
Modeling results across 8 problems
56
1 5 10 150
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Individuals
Mea
n A
ccur
acy
Bayesian MatchingHungarian AlgorithmIndividuals
Calibration at level of items and people
57
ITEMS INDIVIDUALS
0 0.5 10
0.2
0.4
0.6
0.8
1
D (inferred)
D (
act
ual)
Clothing and faces (5)
R=0.318
0 0.5 10
0.2
0.4
0.6
0.8
1
D (inferred)
D (
act
ual)
Clothing and faces (10)
R=0.722
0 0.5 10
0.2
0.4
0.6
0.8
1
D (inferred)
D (
act
ual)
Animals and houses (5)
R=0.433
0 0.5 10
0.2
0.4
0.6
0.8
1
D (inferred)
D (
act
ual)
Animals and houses (10)
R=0.854
0 0.5 10
0.2
0.4
0.6
0.8
1
D (inferred)
D (
act
ual)
Weapons and faces (5)
R=0.969
0 0.5 10
0.2
0.4
0.6
0.8
1
D (inferred)
D (
act
ual)
Weapons and faces (10)
R=0.893
0 0.5 10
0.2
0.4
0.6
0.8
1
D (inferred)
D (
act
ual)
Sport and faces (5)
R=0.223
0 0.5 10
0.2
0.4
0.6
0.8
1
D (inferred)
D (
act
ual)
Sport and faces (10)
R=0.898
(for weapons and faces 10 items problem)
0 0.5 10
0.2
0.4
0.6
0.8
1
A (inferred)A
(a
ctua
l)
Clothing and faces (5)
R=0.955
0 0.5 10
0.2
0.4
0.6
0.8
1
A (inferred)
A (
act
ual)
Clothing and faces (10)
R=0.994
0 0.5 10
0.2
0.4
0.6
0.8
1
A (inferred)
A (
act
ual)
Animals and houses (5)
R=0.962
0 0.5 10
0.2
0.4
0.6
0.8
1
A (inferred)
A (
act
ual)
Animals and houses (10)
R=0.971
0 0.5 10
0.2
0.4
0.6
0.8
1
A (inferred)
A (
act
ual)
Weapons and faces (5)
R=0.943
0 0.5 10
0.2
0.4
0.6
0.8
1
A (inferred)
A (
act
ual)
Weapons and faces (10)
R=0.957
0 0.5 10
0.2
0.4
0.6
0.8
1
A (inferred)
A (
act
ual)
Sport and faces (5)
R=0.953
0 0.5 10
0.2
0.4
0.6
0.8
1
A (inferred)
A (
act
ual)
Sport and faces (10)
R=0.984
Varying number of individuals
58
0 5 10 1550
55
60
65
70
75
80
85
90
95
100
Number of Individuals
Mea
n A
ccur
acy
Bayesian MatchingHungarian Algorithm
0 1-2 3-4 5+0
0.2
0.4
0.6
0.8
1
0 1-2 3-4 5+0
0.2
0.4
0.6
0.8
1
How predictive are subject provided confidence ratings?
59
# guesses estimatedby individual
Acc
urac
y
# guesses estimatedby model
(based on variable A)
r=-.50 r=-.81
Another matching problem
60
Dutch
Danish
Yiddish
Thai
Vietnamese
Chinese
Georgian
Russian
Japanese
A
B
C
D
E
F
G
H
I
godt nytår
gelukkig nieuwjaar
a gut yohr
С Новым Годом
สวั�สดี�ปี�ใหม่�
Chúc Mừng Nǎm Mới
გილოცავთ ახალწელს
Modeling Results – Declarative Tasks
62
1 10 200
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Individuals
Mea
n A
ccur
acy
Bayesian MatchingHungarian AlgorithmIndividuals
Part IIITraveling Salesman Problems
65
1
2
3
45
6
7
8
9
10
11
12
13
14
15
16
17
18
19
2021
22
23
24
25
26
27
28
29
30
B30-21
Find the shortest route between cities
66
1
2
3
45
6
7
8
9
10
11
12
13
14
15
16
17
18
19
2021
22
23
24
25
26
27
28
29
30
B30-21 - subj 5
1
2
3
45
6
7
8
9
10
11
12
13
14
15
16
17
18
19
2021
22
23
24
25
26
27
28
29
30
B30-21 - subj 83
1
2
3
45
6
7
8
9
10
11
12
13
14
15
16
17
18
19
2021
22
23
24
25
26
27
28
29
30
B30-21 - subj 60
1
2
3
45
6
7
8
9
10
11
12
13
14
15
16
17
18
19
2021
22
23
24
25
26
27
28
29
30
B30-21
B30-21
Individual 5 Individual 83 Individual 60Optimal
Dataset Vickers, Bovet, Lee, & Hughes (2003)
83 participants 7 problems of 30 cities
TSP Aggregation Problem
Propose a good solution based on all individual solutions
Task constraints Data consists of city order only No access to city locations
68
Approach
Find tours with edges for which many individuals agree
Calculate agreement matrix A A = n × n matrix, where n is the number of cities aij indicates the number of participants that connect cities i and j.
Find tour that maximizes
69
tourji
cija
),(
(this itself is a non-Euclidian TSP problem)
Line thickness = agreement
70
1
2
3
45
6
7
8
9
10
11
12
13
14
15
16
17
18
19
2021
22
23
24
25
26
27
28
29
30
B30-21
Blue = Aggregate Tour
71
Results averaged across 7 problems
0
2
4
6
8
10
12
14
16
18
Per
cent
ove
r O
ptim
al
aggregate
Part IVSummary & Conclusions
74
When do we get wisdom of crowds effect?
Independent errors different people knowing different things Some minimal number of individuals
10-20 individuals often sufficient
75
What are methods for finding experts?
1) Self-reported expertise: unreliable has led to claims of “myth of expertise”
2) Based on explicit scores by comparing to ground truth but ground truth might not be immediately available
3) Endogenously discover experts Use the crowd to discover experts Small groups of experts can be effective
76
What to do about systematic biases?
In some tasks, individuals systematically distort the ground truth spatial and temporal distortions memory distortions (e.g. false memory) decision-making distortions
Does this diminish the wisdom of crowds effect? maybe… but a model that predicts these systematic distortions might be
able to “undo” them
77
Conclusion
Effective aggregation of human judgments requires cognitive models
Psychology and cognitive science can inform aggregation models
Generative models ordering information cognitively plausible individual differences
Group response = probability distribution over all permutations of N items With N=44 items, we have 44! > 1053 combinations Approximate inference methods: MCMC
84
Model incorporating overall person ability
85
j individuals
jmx
jmy
mμ
jm
| , ~ N ,ijm m jm m jmx
( )jm jmranky x
~ Gamma ,1 /jm j j
Overall ability
Task specific ability
m tasks
j ~ Gamma ,1 /j j individuals
1 10 20 30 40 50 60 70 800
5
10
15
20
25
Individuals
Mea
n
Thurstonian Model v1Thurstonian Model v2Borda countModeIndividuals
Combinatorial optimization problem maximizes agreement in assigning N items to N responses
Hungarian algorithm construct a count matrix M Mij = number of people that paired item i with response j find row and column permutations to maximize diagonal sum O( n3 )