Recommendation as Search: Reflections on Symmetry

1©MapR Technologies - Confidential

Recommendation as Search

Reflections on Symmetry


Company Background

MapR provides the industry’s best Hadoop Distribution– Combines the best of the Hadoop community

contributions with significant internally financed infrastructure development

Background of Team– Deep management bench with extensive analytic,

storage, virtualization, and open source experience– Google, EMC, Cisco, VMWare, Network Appliance, IBM,

Microsoft, Apache Foundation, Aster Data, Brio, ParAccel Proven – MapR used across industries (Financial Services, Media,

Telcom, Health Care, Internet Services, Government) – Strategic OEM relationship with EMC and Cisco– Over 1,000 installs


What is Hadoop?

A new style of computation

A new style of combining computation and storage

Allows very large computations

Used by all large internet companies, many other industries

Fundamentally changes the economics of large-scale computation


Why Big Data?

Because we can

Because we can learn new things

Because new economics of computation favors large scale

Because big data can be simpler than small data


Recommendations

Often known as collaborative filtering“People who bought x also bought y”

Actors (people) interact (bought) with items (x and y)– observe successful interaction

We want to suggest additional successful interactions

Observations are inherently very sparse


Examples

Customers buying books (Linden et al)

Web visitors rating music (Shardanand and Maes) or movies (Riedl, et al), (Netflix)

Internet radio listeners not skipping songs (Musicmatch)

Internet video watchers watching >30 s (Veoh)

iTunes song purchases or plays (Apple)


Fundamental Algorithm

History matrix A has the shape of actors x items

Cooccurrence matrix K has the shape of items x itemsan actor interacted with both x and ysum over all actors

A is also a linear operator

K tells us “users who interacted with x also interacted with y”


… Warning …


… Warning …

Mathematics ahead


Fundamental Algorithmic Structure

Cooccurrence

For very large data-sets


But Wait ...

Does it have to be that way?


But why not ...


But why not ...

Why just dyadic learning?


But why not ...


Why not triadic learning?


But why not ...


Why not p-adic learning?


For example

Users enter queries (A)– (actor = user, item=query)

Users view videos (B)– (actor = user, item=video)

A’A gives query recommendation– “did you mean to ask for”

B’B gives video recommendation– “you might like these videos”


The punch-line

B’A recommends videos in response to a query– (isn’t that a search engine?)– (not quite, it doesn’t look at content or meta-data)


Real-life example

Query: “Paco de Lucia” Conventional meta-data search results:– “hombres del paco” times 400– not much else

Recommendation based search:– Flamenco guitar and dancers– Spanish and classical guitar– Van Halen doing a classical/flamenco riff


Real-life example


Real-life example


Hypothetical Example

Want a navigational ontology? Just put labels on a web page with traffic– This gives A = users x label clicks

Remember viewing history– This gives B = users x items

Cross recommend– B’A = label to item mapping

After several users click, results are whatever users think they should be


Resources

[email protected]@ted_dunning

Slides and such:– http://info.mapr.com/ted-paris-05-2012

The original paper– Accurate Methods for the Statistics of Surprise and Coincidence– (check on citeseer)

Source code– Mahout project– contact me

mailto:[email protected]

http://info.mapr.com/ted-uk-2012-05

http://info.mapr.com/ted-uk-2012-05

Recommendation as Search: Reflections on Symmetry

Technology