Human Computation Play a Game to Develop an Ontology

11

Human ComputationHuman Computation

Play a Game to Develop an OntologyPlay a Game to Develop an Ontology

Peyman NasirifardPeyman Nasirifard

p+e+y+m+a+b-b+n p+e+y+m+a+b-b+n dotdot sin(arcsin(lastname)) sin(arcsin(lastname)) @@ deri.org deri.org

22

AgendaAgenda

IntroductionIntroduction CAPTCHACAPTCHA

Games with a purposeGames with a purpose ESP gameESP game PeekaboomPeekaboom VerbosityVerbosity

Possible game for developing simple ontologiesPossible game for developing simple ontologiesPlay a gamePlay a gameConclusionConclusion

33

IntroductionIntroductionHuman-based computationHuman-based computation is a technique when a computational is a technique when a computational process performs its function via outsourcing certain steps to process performs its function via outsourcing certain steps to humans.humans.

44

Back to HistoryBack to History

Yahoo! and Gmail are not interested to Yahoo! and Gmail are not interested to enable a bot to create thousands accounts enable a bot to create thousands accounts per day for sending spamper day for sending spam They use CAPTCHA to prevent itThey use CAPTCHA to prevent it

plus

55

CAPTCHACAPTCHA

Stands for “Completely Automated Public Stands for “Completely Automated Public Turing test to tell Computers and Humans Turing test to tell Computers and Humans Apart”Apart”Luis von AhnLuis von Ahn et al. coined the term in et al. coined the term in 20002000A Program that can tell A Program that can tell whether a user is a human whether a user is a human or a computeror a computerMany different techniquesMany different techniques

66

Some ExamplesSome Examples

77

Dog or Cat?Dog or Cat?

Human: mmm… dog

Computer: mmmmmmmmmmmmmmm…

88

Human ComputationHuman Computation

If we use people to break CAPTCHA, we If we use people to break CAPTCHA, we are doing human computationare doing human computation In some countries, some companies hire In some countries, some companies hire

people to break CAPTCHA and send spampeople to break CAPTCHA and send spam Some companies Some companies cleverlycleverly use humans to use humans to

break CAPTCHA and send spambreak CAPTCHA and send spam How?How?

99

Clever spammersClever spammers

1010

Clever SpammersClever Spammers

Type the word in the box if you want to see the next picture

Free Nude Photos

1111

Really?!Really?!Jan 2004: world without spam by 2006! Jan 2004: world without spam by 2006! Huge amount of investmentHuge amount of investment Bill Gates receives 4 million spams per dayBill Gates receives 4 million spams per day

1212

Nice QuoteNice QuoteLuis von Ahn: Luis von Ahn: Instead of hiring people and Instead of hiring people and pay them to solve our problems, we can pay them to solve our problems, we can design games and people will pay us to design games and people will pay us to play our games and solve our large-scale play our games and solve our large-scale problems!problems!

1313

The ESP gameThe ESP gameObject of the game: type the same wordObject of the game: type the same wordOnly thing in common is: an imageOnly thing in common is: an imagePlayersPlayers

Do not know each other (randomly paired)Do not know each other (randomly paired) Can not communicateCan not communicate

Advantages:Advantages: Two different sources labels the imageTwo different sources labels the image enjoyableenjoyable labels all images on labels all images on Google imageGoogle image in a short time in a short time Help to improve English!Help to improve English! There are many people that play over 20 hours a weekThere are many people that play over 20 hours a week

1414

The ESP gameThe ESP gamePlayer 1 Player 2

• CAR

• GIRL

• TREE

• WOMAN

• CAR

Agree: CAR

Get points

1515

Taboo wordsTaboo words

Taboo wordsTaboo words More difficult, but more funMore difficult, but more fun

• CAR

• WOMAN

1616

Single version of ESP gameSingle version of ESP game

• CAR

• GIRL

• TREE

• WOMAN

• CAR

• The engine records everything from previous players

• A single player will actually play with another player, but not at the same time

1717

Cheating and RepetitionCheating and Repetition

Problem: Agreement on cheatingProblem: Agreement on cheating Let’s label all images with “dog”Let’s label all images with “dog”Solution: At random, system gets players Solution: At random, system gets players test images to check whether they play test images to check whether they play honestly or nothonestly or not If they do not play honestly, the system will let If they do not play honestly, the system will let

them play, but nothing will be recordedthem play, but nothing will be recordedFor certainty, only labels which at least N For certainty, only labels which at least N pairs agreed upon will be storedpairs agreed upon will be stored

1818

The Limitations of ESPThe Limitations of ESP

The ESP Game can label images (and The ESP Game can label images (and consequently tell you what’s in them), but consequently tell you what’s in them), but it cannot:it cannot: Find the objects being labelledFind the objects being labelled Determine the way in which the object Determine the way in which the object

appears – does the label “car”appears – does the label “car”refer to the text “car” or an actual refer to the text “car” or an actual car in the image?car in the image?

1919

The place of objects in an imageThe place of objects in an image Such information would be extremely useful Such information would be extremely useful

for computer vision researchfor computer vision research

dog

man

2020

The Revealer clicks on parts of the image and shows them to the Guesser.

The Guesser guesses:•Flower•Petal•Butterfly

Server: Correct, Butterfly

2121

HintsHintsThe label “car” is ambiguous --

this is “car”

this is also “car”

The hints help distinguish the manner in which the label “car” appears:

this is the object “car”

this is the text “car”

2222

VerbosityVerbosityCollect common-sense factsCollect common-sense facts Water quenches thirstWater quenches thirst Sky is blueSky is blue Lions eat meatLions eat meat

We as human know hundreds of millions We as human know hundreds of millions common sense factscommon sense facts Computers do not knowComputers do not know

If know, potentially make them more intelligent If know, potentially make them more intelligent (e.g. search better)(e.g. search better)

2323

Common sense fact samplesCommon sense fact samples

MilkMilk

• It is liquid

• It is white

• it has lactose

• cereal is eaten with it

2424

VerbosityVerbosity

Narrator

MILK

is typically near cereal

is a liquid

Guesser

2525

VerbosityVerbosity

Narrator

MILK

Guesser

is typically near cereal

is a liquid

MILK

2626

VerbosityVerbosity

Narrator

Object

Common sense facts about the object

Guesser

2727

VerbosityVerbosity

Narrator

ObjectGuesser

Common sense facts about the object

Object

2828

TemplatesTemplates• ___ is a kind of ___. Allows for hierarchical categorization.• ___ is used for ___. Provides information about the purpose of a word.• ___ is typically near/in/on ___ (three templates). Provide spatial data.• ___ is the opposite of ___ / ___ is related to ___ (two templates). Provide data about basic relations between words.• ___. In the game, this is a “wildcard” that collects related words.

2929

Symmetric vs. AsymmetricSymmetric vs. AsymmetricVerbosity is a asymmetric game, whereas ESP game is a symmetric Verbosity is a asymmetric game, whereas ESP game is a symmetric game.game.

Symmetric games: constraint is number of outputs per inputSymmetric games: constraint is number of outputs per input

Asymmetric games: constraint is number of inputs that produces the Asymmetric games: constraint is number of inputs that produces the same outputsame output

3030

Possible game to build an ontologyPossible game to build an ontology

Several game should work togetherSeveral game should work together Images come from ESP gameImages come from ESP game Not always: only those images are selected Not always: only those images are selected

which have one object in itwhich have one object in it i.e. car, bike, monitor, mouse, house i.e. car, bike, monitor, mouse, house

These images are input to next game which These images are input to next game which tries to catch the properties of objectstries to catch the properties of objects

car has colour, car has wheels, car has car has colour, car has wheels, car has manufacture, car has owner, car has building year, manufacture, car has owner, car has building year, etc.etc.

3131

Possible game to build an ontologyPossible game to build an ontology

Cardinality will be caught by templates, as Cardinality will be caught by templates, as soon as we have properties.soon as we have properties. Car has four wheelsCar has four wheels Car has one plaqueCar has one plaque

These sentences will be transferred to These sentences will be transferred to OWL representation using a mediator. OWL representation using a mediator. The more pairs play the game, the more The more pairs play the game, the more complex the ontology will becomplex the ontology will be

3232

Contact me if you are interested to work on itContact me if you are interested to work on it

3333

This page intentionally left blankThis page intentionally left blank

3434

Guess what!Guess what!

It has usually It has usually fourfour wheels wheelsIt has usually It has usually oneone seat seatIt is kind of vehicleIt is kind of vehicleIt operates with human powerIt operates with human powerIt operates with batteriesIt operates with batteriesIt has a break systemIt has a break systemIt is a kind of chairIt is a kind of chair

3535

AnswerAnswer

3636

ConclusionConclusion

Games are enjoyable!Games are enjoyable!More than 9 billion Human-hours of More than 9 billion Human-hours of solitaire are played each yearsolitaire are played each yearWe may cleverly using humans to solve We may cleverly using humans to solve large-scale problems by designing large-scale problems by designing interesting gamesinteresting gamesMany people play word-guessing games Many people play word-guessing games to improve their Englishto improve their EnglishGo and play to promote science!Go and play to promote science!

3737

ReferencesReferences[1] Verbosity: A Game for Collecting Common-Sense Facts, http://www.cs.cmu.edu/~biglou/Verbosity.pdf [1] Verbosity: A Game for Collecting Common-Sense Facts, http://www.cs.cmu.edu/~biglou/Verbosity.pdf [2] Peekaboom: A Game for Locating Objects in Images, http://www.cs.cmu.edu/~biglou/Peekaboom.pdf[2] Peekaboom: A Game for Locating Objects in Images, http://www.cs.cmu.edu/~biglou/Peekaboom.pdf[3] Labeling Images with a Computer Game, http://www.cs.cmu.edu/~biglou/ESP.pdf [3] Labeling Images with a Computer Game, http://www.cs.cmu.edu/~biglou/ESP.pdf [4] Games with a Purpose, http://www.cs.cmu.edu/~biglou/ieee-gwap.pdf [4] Games with a Purpose, http://www.cs.cmu.edu/~biglou/ieee-gwap.pdf [5] Wikipedia, http://en.wikipedia.org/wiki/Human-based_computation [5] Wikipedia, http://en.wikipedia.org/wiki/Human-based_computation [6] We'll End Spam Within 2 Years, [6] We'll End Spam Within 2 Years,

http://www.connectedhomemag.com/Networking/Articles/Index.cfm?ArticleID=41587http://www.connectedhomemag.com/Networking/Articles/Index.cfm?ArticleID=41587http://news.bbc.co.uk/2/hi/business/3426367.stmhttp://news.bbc.co.uk/2/hi/business/3426367.stm

[7] CAPTCHA, [7] CAPTCHA, http://en.wikipedia.org/wiki/Captchahttp://en.wikipedia.org/wiki/Captcha, , http://www.captcha.nethttp://www.captcha.net

[8] ESP game, [8] ESP game, www.espgame.orgwww.espgame.org[9] Peekaboom game, [9] Peekaboom game, http://www.peekaboom.org/http://www.peekaboom.org/[10] Verbosity game, [10] Verbosity game, www.peekaboom.org/verbosity/www.peekaboom.org/verbosity/[11] Presentation, [11] Presentation, http://isandtcolloq.gsfc.nasa.gov/fall2006/presentations/Ahn.ppthttp://isandtcolloq.gsfc.nasa.gov/fall2006/presentations/Ahn.ppt[12] Presentation, [12] Presentation, www.aladdin.cs.cmu.edu/workshops/lamps05/Slides/Peekaboom.pptwww.aladdin.cs.cmu.edu/workshops/lamps05/Slides/Peekaboom.ppt

3838

Game OverGame Over

p+e+y+m+a+b-b+n p+e+y+m+a+b-b+n dotdot sin(arcsin(lastname)) sin(arcsin(lastname)) @@ deri.org deri.org