Outline Collective Intelligence Judgment Analysis Citizen Science References Crowdsourcing Collective Intelligence, Judgment Analysis, Citizen Science Malay Bhattacharyya Assistant Professor Machine Intelligence Unit Indian Statistical Institute, Kolkata Guest Lecture at IIT, Kharagpur (October 04, 2018)
53
Embed
Crowdsourcing - Collective Intelligence, Judgment Analysis ...cse.iitkgp.ac.in/~saptarshi/courses/socomp2018aut/... · Collective Intelligence Collective intelligence is a form of
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Collective intelligence is a form of intelligence that emerges fromthe collaboration, collective efforts, and competition of manyindividuals [Surowiecki, 2004].
Types of wisdom of the crowd:
• Cognition: How faster and more reliable judgments can bemade through consensus decision making from crowdsourcedopinions.
• Coordination: How the crowd can be organized so as toenable them to effectively work together.
• Cooperation: How the crowd can form networks of trust.
Collective intelligence is a form of intelligence that emerges fromthe collaboration, collective efforts, and competition of manyindividuals [Surowiecki, 2004].
Types of wisdom of the crowd:
• Cognition: How faster and more reliable judgments can bemade through consensus decision making from crowdsourcedopinions.
• Coordination: How the crowd can be organized so as toenable them to effectively work together.
• Cooperation: How the crowd can form networks of trust.
People who avoid looking at the costs of good acts can be trustedto cooperate in important situations, whereas those who lookcannot. We find that evolutionary dynamics can lead tocooperation without looking at costs [Hoffman, 2015].
• Opinion: The label (annotation) marked by an annotator(basically a crowd worker) for a question.
• Judgment: The predicted label from a list of such opinions.
• Gold judgment: The actual opinion (solution) for thequestion.
An annotation process can be formally represented by a quadruplet〈Q,A,O, τ〉, which consists of the following:
• A finite set of annotators A = {A1,A2, . . . ,An},• A finite set of questions Q = {Q1,Q2, . . . ,Qm},• A finite set of opinions O = {O1,O2, . . . ,Ok},• A mapping function τ : Q × A→ O.
Problem statement: Given a set of opinions obtained from thecrowd for a given question, predict (after combining together) the‘gold’ judgment.
• Opinion: The label (annotation) marked by an annotator(basically a crowd worker) for a question.
• Judgment: The predicted label from a list of such opinions.
• Gold judgment: The actual opinion (solution) for thequestion.
An annotation process can be formally represented by a quadruplet〈Q,A,O, τ〉, which consists of the following:
• A finite set of annotators A = {A1,A2, . . . ,An},• A finite set of questions Q = {Q1,Q2, . . . ,Qm},• A finite set of opinions O = {O1,O2, . . . ,Ok},• A mapping function τ : Q × A→ O.
Problem statement: Given a set of opinions obtained from thecrowd for a given question, predict (after combining together) the‘gold’ judgment.
• Opinion: The label (annotation) marked by an annotator(basically a crowd worker) for a question.
• Judgment: The predicted label from a list of such opinions.
• Gold judgment: The actual opinion (solution) for thequestion.
An annotation process can be formally represented by a quadruplet〈Q,A,O, τ〉, which consists of the following:
• A finite set of annotators A = {A1,A2, . . . ,An},• A finite set of questions Q = {Q1,Q2, . . . ,Qm},• A finite set of opinions O = {O1,O2, . . . ,Ok},• A mapping function τ : Q × A→ O.
Problem statement: Given a set of opinions obtained from thecrowd for a given question, predict (after combining together) the‘gold’ judgment.
• Prior independent score: The label (annotation) marked byan annotator (basically a crowd worker) for a questionindependently.
• Posterior dependent score: The label (annotation) markedby an annotator (basically a crowd worker) for a question afterrevealing the prior independent scores.
An annotation process can be formally represented as a 6-tuple〈Q,A, I ,D, τ, τ ′〉, which consists of the following:
• A finite set of questions Q = {Q1,Q2, . . . ,Qm},• A finite set of annotators A = {A1,A2, . . . ,An},• A finite set of prior independent scores I = {i1j , i2j , . . . , inj},• A finite set of posterior dependent scoresD = {d1j , d2j , . . . , dnj},
• A mapping function τ : Q × A→ I ,
• Another mapping function τ ′ : (Q × A)→ D.
Problem statement: Given a set of prior independent scores andposterior dependent scores obtained from the crowd for a givenquestion, predict (after combining together) the ‘gold’ judgment.
An annotation process can be formally represented as a 6-tuple〈Q,A, I ,D, τ, τ ′〉, which consists of the following:
• A finite set of questions Q = {Q1,Q2, . . . ,Qm},• A finite set of annotators A = {A1,A2, . . . ,An},• A finite set of prior independent scores I = {i1j , i2j , . . . , inj},• A finite set of posterior dependent scoresD = {d1j , d2j , . . . , dnj},
• A mapping function τ : Q × A→ I ,
• Another mapping function τ ′ : (Q × A)→ D.
Problem statement: Given a set of prior independent scores andposterior dependent scores obtained from the crowd for a givenquestion, predict (after combining together) the ‘gold’ judgment.
Let ij and dj denote the mean of prior independent scores {i1j , i2j ,. . . , inj} and posterior dependent scores {d1j , d2j , . . . , dnj},respectively, given by the 1st , 2nd , . . . , nth annotator on question j .
• Confidence gap: The confidence gap of annotator k for aparticular question j is defined as follows
|ikj − dkj |.
• Reliability: The reliability of annotator k for question j isdefined as follows
Let ij and dj denote the mean of prior independent scores {i1j , i2j ,. . . , inj} and posterior dependent scores {d1j , d2j , . . . , dnj},respectively, given by the 1st , 2nd , . . . , nth annotator on question j .
• Confidence gap: The confidence gap of annotator k for aparticular question j is defined as follows
|ikj − dkj |.
• Reliability: The reliability of annotator k for question j isdefined as follows
Let ij and dj denote the mean of prior independent scores {i1j , i2j ,. . . , inj} and posterior dependent scores {d1j , d2j , . . . , dnj},respectively, given by the 1st , 2nd , . . . , nth annotator on question j .
• Confidence gap: The confidence gap of annotator k for aparticular question j is defined as follows
|ikj − dkj |.
• Reliability: The reliability of annotator k for question j isdefined as follows
As the opinions in the first phase are independent opinions andopinions in second phase are dependent, therefore the transition ofthe annotators has been modeled in the transition matrix. At eachtime step, the final judgment of a question can be computeddepending upon the stationary distribution and transition matrix.
A Markov model can be proposed for deriving the final judgment.
As the opinions in the first phase are independent opinions andopinions in second phase are dependent, therefore the transition ofthe annotators has been modeled in the transition matrix. At eachtime step, the final judgment of a question can be computeddepending upon the stationary distribution and transition matrix.
A Markov model can be proposed for deriving the final judgment.
Snapshot of a sample question in Image Recognition Task when allthe independent opinions are revealed (as percentage) publicly forcollecting dependent opinions.
An annotation process can be formally represented by a 4-tuple〈Q,A,O, τ〉, which consists of the following:
• A finite set of questions Q = {Q1,Q2, . . . ,Qm},• A finite set of annotators A = {A1,A2, . . . ,An},• A finite set of opinion vectorsO = {{(i111j , i121j , . . . , i1m1j ), (i211j , i
An annotation process can be formally represented by a 4-tuple〈Q,A,O, τ〉, which consists of the following:
• A finite set of questions Q = {Q1,Q2, . . . ,Qm},• A finite set of annotators A = {A1,A2, . . . ,An},• A finite set of opinion vectorsO = {{(i111j , i121j , . . . , i1m1j ), (i211j , i
Suppose, λij denotes the opinion given by a particular annotator ifor a given component j of a question. Then the probability thatthe given opinion matches with the true label is given by
Suppose, λij denotes the opinion given by a particular annotator ifor a given component j of a question. Then the probability thatthe given opinion matches with the true label is given by