A Capability Requirements Approach for Predicting Worker Performance in Crowdsourcing

Copyright 2010 Digital Enterprise Research Institute. All rights reserved.

Digital Enterprise Research Institute www.deri.ie

A CAPABILITY REQUIREMENTS APPROACH FOR PREDICTING WORKER PERFORMANCE IN CROWDSOURCING

Umair ul Hassan, Edward CurryDigital Enterprise Research Institute

National University of Ireland, Galway

9th IEEE International Conference on Collaborative Computing: Networking, Applications and WorksharingAustin, Texas, United StatesOctober 20–23, 2013


Agenda

Motivation Background Task Modelling

Capability Requirements Capabilities Taxonomy

Capability Tracing Experiment Summary

2


Motivation: Heterogeneity

3


WORKER PROFILINGTASK MODELLING

Motivation: Task Routing

Assigning heterogeneous tasks to heterogeneous workers

4

ProfilesModels

TASK ROUTING

MatchingTask↔Worker

ModelsModels

ProfilesProfiles


WORKER PROFILINGTASK MODELLING

Proposal: Performance Prediction

Predict performance of workers on new tasks based on the capabilities required for tasks and assign tasks accordingly

5

ProfilesModels

TASK ROUTING

MatchingTask↔Worker

ModelsModels

ProfilesProfiles

Capability Requirements

Approach

Capability Tracing Model

Performance Prediction


Background: Micro tasks

When micro tasks are crowd sourced Computers cannot do the task Single person cannot do the task Work can be split into smaller tasks

Some online microtask platforms

6



Most common tasks in Amazon Mechanical Turk (AMT) and CrowdFlower (CFL)

7



Example of information extraction task in AMT

8



Example of video transcription task in AMT

9


Task Modelling

Appropriate models are needed to compare and contrast micro tasks.

Capability Requirements approach Capability is defined as the ability of humans to do

things in terms of both the capacity and the opportunity. Four types of capabilities

– Knowledge, – Skill, – Ability, – Other characteristics (e.g. motivation, price, etc)

10


Capability Requirements

Taxonomies have be used to study human task performance, e.g. Fleishman’s taxonomy of human abilities Bloom’s taxonomy of classification of learning

objectives O*NET-SOC taxonomy of occupational classification

We are interested in taxonomy that Describes tasks in terms of human capabilities Helps in comparing tasks in terms of differences and

similarities of capabilities

11


Capabilities Taxonomy

Based on Fleishman’s abilities taxonomy Selected abilities relevant to micro tasks

Comprehension (C): The ability to understand the meaning or importance of something

Bilingualism (B): The ability to speak and understand two languages Writing (W): The ability or capacity to write text in a given language Comparison (M): The ability or capacity to compare things based on

some criteria Judgment (J): The act or process of judging; the formation of an

opinion after consideration Perception (P): The ability or capacity to perceive items visually or

phonetically Identification (I): The process of recognizing something Reasoning (R): The ability to draw conclusions from facts, evidence,

relationships, etc.

12


Requirements of Micro Tasks

13


Capability Tracing

How to model worker’s capabilities? Capability tracing

Inspired by Knowledge Tracing* Estimates probability of a worker knowing a capability

given worker’s responses to test tasks

Worker Profile constrains Set of binary variables representing capabilities Probability estimates of each variable being in a state

14

* A. T. Corbett and J. R. Anderson, “Knowledge tracing: Modeling the acquisition of procedural knowledge,” User Modeling and User-Adapted Interaction, vol. 4, no. 4, pp. 253–278, 1994.


Capability Tracing

Probabilistic network of a capability and four parameters of capability tracing model

15

Not Learned

Learned

Correct Incorrect

States of Capability Variable

Values of Response Variable

p(T)

p(L)

p(T): Probability of transition between states

p(L): Probability of a worker learning to employ the capability

p(G): Probability of guess

p(S): Probability of slip

p(G) p(S)


Experiment

Objective Solicit capability requirements of tasks from crowds Evaluation of capability tracing for performance

prediction Three types of micro tasks with manually created

ground truth data Fact verification Image comparison Information Extraction

37 crowd workers including University students Workers from Shorttask.com

16


Crowdsourcing

Custom web application for gathering data Example of fact verification task

17


Capability Requirements of Tasks

Objective 1: Solicit capability requirements of tasks from crowds

18

(a) fact verification (b) image comparison (c) information extraction


Crowd Performance

How the crowd performed on each type of task?

19

Fact Verification task• 37 workers• Best workers perform with

both precision and recall above 0.8

• More variation in recall means some workers were could not spot the incorrect facts

• Ideally tasks should be assigned to workers that lie in the top-right quadrant of the plot


Crowd Performance

Image Comparison (20 workers) and Information Extraction (17 workers)

20


Performance Prediction

Objective 2: Evaluation of capability tracing for performance prediction

Two phases Build model with observation tasks Predict performance on new tasks

21

AC: Consider previous Accuracy as prediction of future performance

CT: Capability Tracing


Summary

Capabilities taxonomy is first steps towards modelling of micro tasks based on human factors

Capability tracing is effective in predicting future performance Even across tasks if there are similar capabilities

Predicted performance can be used to make right task routing decisions

Future Work Evaluate on more types of tasks Evaluate capabilities such as domain knowledge and

skills Define standard tests for measuring worker capabilities

22


Further Reading

U. Ul Hassan and E. Curry, “A Capability Requirements Approach for Predicting Worker Performance in Crowdsourcing,” in 9th IEEE International Conference on Collaborative Computing: Networking, Applications and Worksharing, 2013.

http://deri.ie/users/umair-ul-hassan

23

9th IEEE International Conference on Collaborative Computing: Networking, Applications and Worksharing

Austin, Texas, United StatesOctober 20–23, 2013


Capability Tracing

Conditional probability of worker learning to employ capability p(Ln|On) is calculated

When evidence On is positive

When evidence On is negative

24


Capability Tracing

Probability of worker learning to employ capability

Performance of worker on next task

25

A Capability Requirements Approach for Predicting Worker Performance in Crowdsourcing

Technology

iewhen micro tasks

contrast micro tasks

new tasks

micro tasks comprehension

knowledge tracing

iemost common tasks

human task performance

task work