On the effectiveness of a Mobile Puzzle Game UI to Crowdsource Linked Data Management tasks

Emanuele Della Valle - http://emanueledellavalle.org 1

Emanuele Della Valle DEIB - Politecnico di Milanohttp://emanueledellavalle.org

Joint work with• Irene Celino

CEFRIEL• Roberto Gualandris

Politecnico di Milano

ON THE EFFECTIVENESS OF A MOBILE PUZZLE GAME UI TO CROWDSOURCE LINKED DATA MANAGEMENT TASKS

CrowdUI Workshop, AVI 2014, May 27, 2014

http://emanueledellavalle.org/

http://emanueledellavalle.org/


Agenda• Context: linked data• Problem space: assess trustworthiness and relevance of linked data• Solution space• Automatic solutions• Crowdsourcing alternative solutions (Focus on Game with a

Purpose)• Contribution: WikiFinder• Analysis of results obtained with WikiFinder• Distinguishing true from false facts• Identifying relevant facts• Other tasks• Discussion • Conclusions and future worksCrowdUI Workshop, AVI 2014, May 27, 2014


Context Linked Data publishing: a growing phenomenon



Problem spaceAssess trustworthiness and relevance of data• Are the following two statements true?• The airport of Milano Malpensa is in Milan• http://dbpedia.org/page/Malpensa_Airport

dbpedia-owl:city dbpedia:Milan

• Leornardo da Vinci was born in Vinci• http://dbpedia.org/page/Leonardo_da_Vinci

dbpedia-owl:birthPlace dbpedia:Vinci,_Tuscany

•Which is more relevant?


No, It is the airport of Milan, but it is

faraway

Yes, Vinci is a small town in

Tuscany

:-O

http://dbpedia.org/page/Malpensa_Airport

http://dbpedia.org/ontology/city



http://dbpedia.org/resource/Milan

http://dbpedia.org/page/Leonardo_da_Vinci

http://dbpedia.org/ontology/birthPlace

http://dbpedia.org/ontology/birthPlace

http://dbpedia.org/resource/Vinci,_Tuscany



Emanuele Della Valle - http://emanueledellavalle.org 5CrowdUI Workshop, AVI 2014, May 27,

2014

State of the art solution Automatic techniques can fail• Are the following two statements true?• The airport of Milano Malpensa is in Milano• A map based system can provide the answer• Leornardo da Vinci was born in Vinci• Data quality tools and instance matching may give the right answer

•Which is more relevant?• No heuristic for or dataset of human association strengths

between Linked Data entities is available


Innovative solutionCrowdsourcing

[source: http://blog.optimyth.com/wp-content/uploads/2013/01/microtask_crowdsourcing_forums1.jpg ]

Trustworthy & relevant


http://blog.optimyth.com/wp-content/uploads/2013/01/microtask_crowdsourcing_forums1.jpg




Innovative solutionGame With a Purpose and Linked Data 1/3

Jörn Hees, Thomas Roth-Berghofer, Ralf Biedert, Benjamin Adrian, Andreas Dengel: BetterRelations: Collecting Association Strengths for Linked Data Triples with a Game. SeCO Book 2012: 223-239CrowdUI Workshop, AVI 2014, May 27,

2014

Subject

Property

Object

Property

Object

BetterRelations


Innovative solutionGame With a Purpose and Linked Data 2/3

Jörn Hees, Mohamed Khamis, Ralf Biedert, Slim Abdennadher, Andreas Dengel: Collecting Links between Entities Ranked by Human Association Strengths. ESWC 2013: 517-531


Subjectrelated to

Object

Object

Knowledge Test Game


2014

Innovative solutionGame With a Purpose and Linked Data 3/3• Other similar approaches• Siorpaes, K., Hepp, M.: OntoGame: Towards Overcoming the

Incentive Bottle- neck in Ontology Building. In: Proc. of the 3rd International IFIP Workshop On Semantic Web & Web Semantics (SWWS), Vilamoura, Portugal, Springer, LNCS (2007) • Kny, E., Kölle, S., Töpper, G., Wittmers, E.: WhoKnows?

(October 2010) • Wolf, L., Knuth, M., Osterhoff, J., Sack, H.: RISQ! Renowned

Individuals Semantic Quiz – A Jeopardy like Quiz Game for Ranking Facts. In: Proc. of the I-SEMANTICS, Graz, Austria, pp. 71–78. ACM (2011)

• Broader approach• E. Simperl, B. Norton, and D. Vrandecic. Crowdsourcing Tasks within Linked Data Management. In Proceedings of COLD2011, volume 782. CEUR-WS.org, 2011.


2014

ContributionMobile puzzle games with a purpose 1/3

Subject Property

Object

Alternative 1


2014


Alternative 2Property

Object

Subject


2014


Property

Object

SubjectAlternative 3


ContributionWikiFinder: a mobile puzzle game with a purpose



ContributionWikiFinder: building the puzzle

True triples

False triples

Negation as failure



Analysis Statistics about WikiFinder • Players: 44• Game played: 440• Triples proposed: 1127• True triples: 246• False triples: 881• Total hours played: 7,3 hours• Average life play per user: 10 minutes



Analysis Distinguishing between true and false triples



Analysis Triples “popularity” – cf. with manual rank



Analysis Accuracy on property domain



Analysis Accuracy on property range



Analysis Wrapping up•WikiFinder is effective in • distinguishing true triples from false ones• identifying relevant triples• identifying the domain of

the predicate

•WikiFinder is not effective in • Identifying the range of

the predicates• A bias introduced by the

game?• Shall the player be allowed to

play also from right to left?



Conclusions and future work•WikiFinder approach • is effective for some Linked Data Management tasks• is worth further investigation• Next steps• Improve the game play• Implement the other alternatives• Focus the puzzle content• Broaden the evaluation• Run comparative evaluations against alternative mobile puzzle

games• Run comparative evaluations against other approaches



Irene Celino CEFRIELEmanuele Della Valle DEIB - Politecnico di MilanoRoberto Gualandris Politecnico di Milano

THANK YOU FOR PAYING ATTENTION

ANY QUESTION?


On the effectiveness of a Mobile Puzzle Game UI to Crowdsource Linked Data Management tasks

Internet

human association

crowdui workshop

statements

ralf biedert

andreas dengel

milano malpensa

game

relevant