Crowdsourcing ontology engineering Elena Simperl Web and Internet Science, University of Southampton 11 April 2013
Dec 15, 2015
Crowdsourcing ontology engineeringElena SimperlWeb and Internet Science, University of Southampton 11 April 2013
2
Overview• "online, distributed problem-
solving and production model“ [Brabham, 2008]
• Varieties: wisdom of the crowds/collective intelligence, open innovation, human computation...
• Why is it a good idea?
– Cost and efficiency savings – Wider acceptance, closer to
user needs, diversity
• Approaches
– Collaborative ontology engineering
– Challenges/competitions
– Games with a purpose – Microtask/paid
crowdsourcing
• In combination with automatic techniques
3
Crowdsourcing ontology alignment• Experiments using MTurk, CrowdFlower and established benchmarks
• Enhancing the results of automatic techniques
• Fast, accurate, cost-effective
[Sarasua, Simperl, Noy, ISWC2012]
CartP301-304
100R50PEdas-Iasted
100R50PEkaw-Iasted
100R50PCmt-Ekaw
100R50PConfOf-Ekaw
Imp301-304
PRECISION 0.53 0.8 1.0 1.0 0.93 0.73
RECALL 1.0 0.42 0.7 0.75 0.65 1.0
Open questions• Quality assurance and evaluation
• Incentives and motivators
• Choice of crowdsourcing approach and combinations of different approaches
• Reusable collection of algorithms for quality assurance, task assignment, workflow management, results consolidation etc
• Schemas recording provenance of crowdsourced data
• Descriptive framework for classification of human computation systems
– Types of tasks and their mode of execution– Participants and their roles – Interaction with system and among participants– Validation of results– Consolidation and aggregation of inputs into complete solution