Global Budgets for Local Recommendations

1

Thomas Sandholm, Hang Ung, Christina Aperjis, Bernardo HubermanHewlett-Packard, HP Labs, Social Computing LabRecSys, Barcelona September 27, 2010

GLOBAL BUDGETS FOR LOCAL RECOMMENDATIONS

2

WHY VOTE?

How do we get more people to contribute their opinions?

3

4

500USERS

30EXPERIMENTS

6LOCATIONS

Palo AltoChicago

Paris

Mumbai

Bangalore

Athens

5

MANY PEOPLE CONSUME CONTENT – FEW LEAVE OPINIONS

May 2008300M+

April 20108M

April 201027M

Sample takenUsers

6

RATING BUDGETS AND RANKING

Bob:$10

Alice: $11

John

http://dominos.com

Pizza

Restaurants

$4

http://burgerking.com

Burgers

Restaurants

$6

http://roundtable.com

Pizza

Restaurants

$3

http://applebees.com

Diners

Restaurants

$8

Top Channels

$8 Diners$7 Pizza$6 Burgers

Friend Channels

$6 Burgers$4 Pizza

$8 Applebee’s$6 Burger King$4 Dominos$3 RoundTable

7

RATING REWARDS

Bob

Alice

John

http://dominos.com

$4

Top Rewards

(reward factor 10)$40 Bob Dominos @ A$30 Alice RoundTable @ B

http://dominos.com

$5

$3 http://roundtable.com

http://roundtable.com

$3$3 http://roundtable.com

A

B

8

CLICK TO RATING RATIO

May 2008300M+

April 20108M

April 201027M

July 20104K

Sample takenUsers

9

RECOMMENDATION SUCCESS

Overa

ll

Cities

Hotel

s

Mov

ies

Mus

ic

Notes

Phot

os

Plac

esPo

lls

Resta

uran

ts

Shop

ping

Trav

el

Wikip

edia

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Clicks as Success

Ratings as Success

Success = proportion of query sessions ending with clicks/ratings

10

SYSTEM COVERAGE

11

MECHANICAL TURK EXPERIMENTS

Setup-5 Surveys-10 URLs-6 Locations-3 Continents-500 Users

Surveys-1-5 Star-Budget-Star Bonus-Budget Bonus-Gloe

12

MECHANISM RESULTS

1-5 Star Budget 1-5 Star Bonus

Budget Bonus

Gloe0

0.5

1

1.5

2

2.5

ParisPalo AltoMumbaiChicagoBangaloreAthens

Kendall Tau Rank Correlation (higher better)

1-5 Star Budget 1-5 Star Bonus

Budget Bonus

Gloe0

10

20

30

40

50

60

ParisPalo AltoMumbaiChicagoBangaloreAthens

RMSE (lower better)

13

BONUS EFFECT ON PARTICIPATION

2 3 4 5 60

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Positive SignalNegative Signal

Probability of signals

Number of surveys taken

14

LESSONS LEARNED

– Amazon Mechanical Turk•Workers not random geographic sample•Sensitive to task complexity•Respond well to small incentives

– Budget Mechanism•Higher quality recommendations with incentives•Social/Economic/Status value extract more opinions•Tuned based on usage, e.g. reward factor

15

FUTURE WORK

– Projects•AfricaMap: map annotation in remote parts for disaster relief UNOSAT/Uni. Geneva•Mobile print provider recommendations via HP ePrint

– Research•GSP Auction for commercial bidding•LMSR Market for recommendation arbitrage•Enhance reward mechanism to both encourage and identify high quality contributions

16

Papers at www.hpl.hp.com/research/scl

Live system at www.hpgloe.com

SOCIAL COMPUTING LAB

THANK YOU

1717

BACKUP

18

SYSTEM ARCHITECTURE

Clients

Java

JavaScriptjQuery

Android OS

BlackBerry OS

iUi/iPhone Safari

HTML5

Web Server

ApacheHTTP REST

Web Application Service

DjangoJSONP

GeoRSSHTML

Caching

Memcached

Integration

Services

Google MapsGeocoding

FacebookSocialAuth

OAuth

Database Shards

MySQLGeohashing

19

SCALABILITY

20

HP GLOE: STATUS

~7m* recommendations at http://hpgloe.com

~4k* users on Android, iPhone, BlackBerry, WebOS, Web…

*Sept 2010

21

LESSONS LEARNED CONTINUED

– Gloe system•Geohash location partitioning simple and efficient•HTTP(S) GET/JSON(P) has served us well on all platforms•MySQL & Sharded architecture flexible and fast