Diversity and Team Science Scott E Page Santa Fe Institute University of Michigan
Diversity and Team Science
Scott E Page Santa Fe Institute
University of Michigan
Outline
A Great Big Complex World
Diversity and Prediction Diversity Prediction Theorem
Model Diversity Theorem
Categorical Diversity Theorem
Big Science
A Great Big (Complex) World
http://library.sc.edu
http://crissp.eu
Authors Per Paper: Computer Science
Identity and Cognitive Diversity
Authors Per Paper: Computer Science
Courtesy: Jacob Foster
``Few if any funding programs support research on the effectiveness of science teams and larger groups.’’
Conclusions 1&2
#1: Team composition matters.
Diversity is critical
#2: Team composition systematic
Prediction
“All Models Are Wrong”
- George Box
“Hence, our truth is the intersection of independent lies.”
- Richard Levins
1197
1197
Average = 1,197
Diversity Prediction Theorem
Diversity Prediction Theorem
Crowd Error = Average Error - Diversity
Crowd Error = Average Error - Diversity
Crowd Error = Average Error – Diversity
0.6 = 2,956.0 - 2955.4
Actual Number 103 Class Prediction 70.04
Diversity Prediction Theorem Crowd = Average – Diversity
1086 = 3456 – 2370
Number Finnish Athletes
Actual Number 103 Class Prediction 65.6
Diversity Prediction Theorem Crowd = Average – Diversity
1401 = 7039 – 5638
Number Finnish Athletes
Actual Number 319
Class Prediction 318.58
Diversity Prediction Theorem Crowd = Average – Diversity
0.1 = 202,323.6 – 202,323.5
Latvian Cars Per 1000
Actual Number 319
Class Prediction 369
Diversity Prediction Theorem Crowd = Average – Diversity
2542 = 89,743 – 87,201
Latvian Cars Per 1000
Christina Romer
Model Diversity Theorem
N Models
Distribution across models:
(P1, P2, P3 … Pn)
Pj = probability someone uses model j
F = probability two individuals have the same model
Probability two people use the same model.
Match(P) = (P1)2 + (P2)2 +… (Pn)2
Diversity Index: Δ = 1/Match(P)
Δ = Effective number of parties (political science)
firms (economics)
species (ecology)
Diversity Index
Δ = (p12 + p22 + p32)-1
(1/3)2 + (1/3)2 + (1/3)2 = 3(1/9) = 1/3
Δ = 3
(1/2)2 + (1/4)2 + (1/4)2 = 6/16
Δ = 2.66
Claim: N independent predictive models with a diversity index of Δ and an average variance of V, have an expected squared error equal to:
V/Δ
Economo, Hong, and Page (2015)
Model Diversity Theorem
Size DIVERSITY (Δ) Matters
Goncola Abecasis
Categorical Diversity Theorem
Partition the set of possible instances into categories
Make a prediction for each category
Computer Science
PAC Learning: Valient Robust Classification: Provost and Foster
Ensemble Learning: Desarthy and Sheela
Categorical Predictive Models
Partition the set of possible instances into categories
Make a prediction for each category
300 200 100 200
A B C D
Chloride in Water: mg/L
300 200 100 200
A B C D
Chloride in Water: mg/L
240 170
Variation in Data: 20,000 20,000 = (200-200)2 +(300-200)2 +(100-200)2 +(200-200)2
Variation in Data: 20,000 20,000 = (200-200)2 +(300-200)2 +(100-200)2 +(200-200)2
Residual Variation: 12,000 20,000 = (200-240)2 +(300-240)2 +(100-170)2 +(200-170)2
Variation in Data: 20,000 20,000 = (200-200)2 +(300-200)2 +(100-200)2 +(200-200)2
Residual Variation: 11,000 20,000 = (200-240)2 +(300-240)2 +(100-170)2 +(200-170)2
R2: 0.45
Total Variation (20,000)
300 200 100 200
A B C D
Chloride in Water: mg/L
Categorization Loss: 10,000 Mean of Category 1: 250 Categorization Loss 1 : 5000 5000 = (200-250)2 +(300-250)2 Mean of Category 2: 150 Categorization Loss 2 : 5000 5000 = (200-150)2 +(100-150)2
Categorization Error
Possible Variation Explained 10,000
Categorization Loss
10,000
300 200 100 200
A B C D
Chloride in Water: mg/L
240 170
Prediction Error: 1000= 200 + 800 Prediction Category 1: 240 Actual Value: 250 Prediction Error: 200 = (250-240)2 +(250-240)2 Prediction Category 2: 170 Actual Value: 150 Prediction Error: 800 = (170-150)2 +(170-150)2
Categorization Error
Variation Explained 9,000
Categorization Loss
10,000 Prediction Error
1000
Categorical Diversity Theorem
Variation = Explained + Category Loss + Predictive Error
Value of Distinct Categories
Distinct Categories result in diverse categorization losses and by the Diversity Prediction Theorem lower error.
Six years of data Half million users 17,700 movies Data divided into (training, testing) Testing Data dived into (probe, quiz, test)
Singular Value Decomposition
Each movie represented by a vector: (p1,p2,p3,p4…pn)
Each person represented by a vector: (q1,q2,q3,q4…qn)
Christina and David Rom
Robert Bell
BellKor
50 dimensions
107 models
Best Model: 6.8%
BellKor
50 dimensions
107 models
Best Model: 6.8%
Combination of Models: 8.4%
BellKor’s Pragmatic Chaos
Best Model 8.4% Ensemble: 10.1%
Enter ``The Ensemble’’
23 Teams
30 Countries
And the Winner is…
Ensemble 10.06%
Bellkor 10.06%
But, the Real Winner is…
Ensemble 10.06%
Bellkor 10.06%
50-50 Combination 10.19%
Combine accurate and diverse models to make good predictions.
Big Science
Leaving Our Silos
Medicine Sociology
Chemistry Economics
Freeman and Huang
Citations Impact Factor
# Addresses + + # References + + # Past papers + + Homophily - -
Science & Engineering Social Science
Papers > 100 Cites
0.00%
0.02%
0.04%
0.06%
0.08%
0.10%
0.12%
0.14%
Team
Team
Solo Solo
Jones B, Wuchty S, Uzzi B (2008) Multi-University Research Teams: Shifting Impact, Geography, and Stratification in Science. Science 322: 1259
Inter-University Collaboration Increases Impact
Cummings, J. N., Kiesler, S., Zadeh, R., & Balakrishnan, A. (2013). Group heterogeneity increases the risks of large group size: A longitudinal study of productivity in research groups. Psychological Science, 24(6), 880-890.
Cummings, J. N., Kiesler, S., Zadeh, R., & Balakrishnan, A. (2013). Group heterogeneity increases the risks of large group size: A longitudinal study of productivity in research groups. Psychological Science, 24(6), 880-890.
Cummings, J. N., Kiesler, S., Zadeh, R., & Balakrishnan, A. (2013). Group heterogeneity increases the risks of large group size: A longitudinal study of productivity in research groups. Psychological Science, 24(6), 880-890.
Best papers have low proximity
Best patents have low proximity
Atypical Connections Variable Odds Ratio
Years since PHD 1.14
Prior Cites 2.25
Author Count 0.8
Depth (HHI) 3.29
Atypical Connect 15.17
``Recombinant search and breakthrough idea generation: An analysis of high impact papers in the social sciences’’ Melissa A. Schilling and Elad Green Research Policy, 2011, vol. 40, issue 10, pages 1321-1331
Melissa Schilling
Q?
Diversity and Team ScienceOutlineSlide Number 3Slide Number 4Slide Number 5Slide Number 6Slide Number 7Slide Number 8Identity and Cognitive Diversity�Slide Number 10Slide Number 11Conclusions 1&2Prediction�Slide Number 14“All Models Are Wrong”“Hence, our truth is the intersection of independent lies.” Slide Number 17Slide Number 18Slide Number 1911971197Diversity Prediction Theorem�Diversity Prediction TheoremSlide Number 24Slide Number 25Slide Number 26Slide Number 27Slide Number 28Slide Number 29Slide Number 30Slide Number 31Model Diversity TheoremSlide Number 33Slide Number 34Slide Number 35Diversity IndexModel Diversity TheoremSize DIVERSITY (Δ) MattersSlide Number 39Slide Number 40Categorical Diversity TheoremSlide Number 42Computer ScienceCategorical Predictive ModelsSlide Number 45Slide Number 46Slide Number 47Slide Number 48Slide Number 49Slide Number 50Slide Number 51Slide Number 52Slide Number 53Slide Number 54Slide Number 55Slide Number 56Categorical Diversity TheoremValue of Distinct CategoriesSlide Number 59Slide Number 60Slide Number 61Singular Value DecompositionSlide Number 63BellKorBellKorBellKor’s Pragmatic ChaosEnter ``The Ensemble’’And the Winner is…But, the Real Winner is… Big ScienceSlide Number 72Slide Number 73Freeman and HuangSlide Number 75Inter-University Collaboration Increases ImpactSlide Number 77Slide Number 78Slide Number 79Slide Number 80Slide Number 81Atypical ConnectionsQ? ��