Recommender Systems with Ruby (adding machine learning, statistics, etc)

Ruby in the world of recommendations

(also machine learning, statistics and visualizations..)

Marcel Caraciolo@marcelcaracioloDeveloper, Cientist, contributor to the Crab recsys project,works with Python for 6 years, interested at mobile,education, machine learning and dataaaaa!Recife, Brazil - http://aimotion.blogspot.com

Saturday, September 14, 2013

FAÇA BACKUP! NUNCA: find . -‐type f -‐not -‐name '*pyc' | xargs rm

Scientific Environment

Presentation & VisualizationExperimentation

(Re-Design)

Data AcquisitionData Analysis

Where is Ruby?

(Re-Design)

Where is Ruby?

(Re-Design)

Where is Ruby?

(Re-Design)

Where is Ruby?

(Re-Design)

Where is Ruby?

Python launched at 1991; Ruby launched at 1995

Python was highly addopted and promoted by most of the research and

development team of Google

Where is Ruby?Python lançado em 1991; Ruby lançado em 1995

Python foi altamente popularizado com a adoção oficial de boa parte do time de pesquisa do Google

Python has been an important key of Google since its beginning, and still continues as our infra-structure grows, we are always looking for more people with skills in this language.

Peter Norvig, Google, Inc.Saturday, September 14, 2013

Where is Ruby?

Python was famous even at some old scientific articles

Where is Ruby?Ruby’s popularity exploded at 2004.

Focus on web

Django - 2005; Numpy - 2005; BioPython - 2001; SAGE - 2005;

Matplotlib- 2000;

Python

Where is Ruby?Programming comes second to researchers, not

first like us. - “Ruby developer answer”

Python [(x, x*x) for x in [1,2,3,4] if x != 3]

vs Ruby`[1,2,3,4].map { |x| [x, x*x] if x != 3 }`

vs Result [(1,1), (2,4), (4,16)]

Where is Ruby?

Python

Hey, Ruby has options!

gem install nmatrix

git clone https://github.com/SciRuby/nmatrix.git

cd nmatrix/

bundle install

rake compile

rake repackage

gem install pkg/nmatrix-*.gem

>> NMatrix.new([2, 3], [0, 1, 2, 3, 4, 5], :int64).pp [0, 1, 2] [3, 4, 5]=> nil

>> m = N[ [2, 3, 4], [7, 8, 9] ]=> #<NMatrix:0x007f8e121b6cf8shape:[2,3] dtype:int32 stype:dense> >> m.pp [2, 3, 4] [7, 8, 9]

Depends on ATLAS/CBLAST and written mostly in C and C++

https://github.com/SciRuby/nmatrix/wiki/Getting-started

Data Visualization

•R•Gnuplot•Google Charts API•JFreeChart•Scruffy•Timetric•Tioga•RChart

Data Visualizationrequire 'rsruby'

cmd = %Q

pdf(file = "r_directly.pdf"))

boxplot(c(1,2,3,4),c(5,6,7,8))

dev.off()

def gnuplot(commands)

IO.popen("gnuplot", "w") { |io| io.puts commands }

commands = %Q(

set terminal svg

set output "curves.svg"

plot [-10:10] sin(x), atan(x), cos(atan(x))

gnuplot(commands)

http://effectif.com/ruby/manor/data-visualisation-with-rubyhttps://github.com/glejeune/Ruby-Graphviz/

Other tools•BioRuby

#!/usr/bin/env ruby require 'bio' # create a DNA sequence object from a Stringdna = Bio::Sequence::NA.new("atcggtcggctta") # create a RNA sequence object from a Stringrna = Bio::Sequence::NA.new("auugccuacauaggc") # create a Protein sequence from a Stringaa = Bio::Sequence::AA.new("AGFAVENDSA") # you can check if the sequence contains illegal characters# that is not an accepted IUB character for that symbol# (should prepare a Bio::Sequence::AA#illegal_symbols method also)puts dna.illegal_bases # translate and concatenate a DNA sequence to Protein sequencenewseq = aa + dna.translateputs newseq # => "AGFAVENDSAIGRL"

http://bioruby.org/Saturday, September 14, 2013

Other tools•RubyDoop (uses JRuby)

module WordCount

class Reducer

def reduce(key, values, context)

sum = 0

values.each { |value| sum += value.get }

context.write(key, Hadoop::Io::IntWritable.new(sum))

https://github.com/iconara/rubydoop

module WordCount

class Mapper

def map(key, value, context)

value.to_s.split.each do |word|

word.gsub!(/\W/, '')

word.downcase!

unless word.empty?

context.write(Hadoop::Io::Text.new(word), Hadoop::Io::IntWritable.new(1))

Coming back to the world of recommenders

The world is an over-crowded place

Coming back to the world of recommenders

!"#$%&'()$*+$,-$&.#'/0'&%)#)$1(,0#

Recommendation Systems

Systems designed to recommend to me something I may like

Recommendation Systems!"#$%&"'$"'(')*#*+,)

-+*#)+. -#/') 0#)1#

2' 23&4"+')1 5,6 7),*%'"&863

Graph Representation

And how does it work ?

What the recommenders realy do ?

1. Predict how much you may like a certain product o service

2. It suggests a list of N items ordered by the level of your interests.

3. It suggests a N list o f users to a product/service

4. It explains to you why those items were recommended.

5. It adjusts the prediction and recommendations based on your feedback and from anothers.

Content Based Filtering

Gone with the Wind

Die Hard

Similar

Armagedon ToyStore

Marcel

likesrecommends

Problems with Content Recommenders

1. Restrict Data Analysis

3. Portfolio Effect

- Items and users mal-formed. Even worst in audio and images

- An person that does not have experience with Sushi does not get the recommendation of the best sushi in town.

- Just because I saw 1 movie of Xuxa when I was child, it must have to recommend all movies of her (só para baixinhos!)

2. Specialized Data

Collaborative Filtering

Gone with the wind

Similar

Armagedon ToyStore

Marcel

like recommend

Rafael Amanda Users

Problems with Collaborative Filtering

1. Scalability

2. Sparse Data

3. Cold Start

4. Popularity

- Amazon with 5M users, 50K items, 1.4B ratings

- New users and items with no records

- I only rated one book at Amazon!

- The person who reads ‘Harry Potter’ also reads ‘Kama Sutra’

5. Hacking

- Everyone reads Harry Potter!

How does it show ?Highlights More about this artist...

Listen to the similar songs

Someone similar to you also liked this...

Since you listened this, you may like this one...

Those items come together...

The most popular of your group...

New Releases

Recommendable

Quickly add a recommender engine for Likes and Dislikes to your Ruby app

http://davidcel.is/recommendable/

Recommendable

gem 'recommendable'

Add to your GemFile:

Recommendablerequire 'redis'

Recommendable.configure do |config| # Recommendable's connection to Redis config.redis = Redis.new(:host => 'localhost', :port => 6379, :db => 0)

# A prefix for all keys Recommendable uses config.redis_namespace = :recommendable

# Whether or not to automatically enqueue users to have their recommendations # refreshed after they like/dislike an item config.auto_enqueue = true

# The name of the queue that background jobs will be placed in config.queue_name = :recommendable

# The number of nearest neighbors (k-NN) to check when updating # recommendations for a user. Set to `nil` if you want to check all # other users as opposed to a subset of the nearest ones. config.nearest_neighbors = nilend

Create a configuration initializer:

RecommendableIn your ONE model that will be receiving the

recommendations:

class User recommends :movies, :books, :minerals, :other_things

# ...end

Recommendable

>> current_user.liked_movies.limit(10)>> current_user.bookmarked_books.where(:author => "Cormac McCarthy")>> current_user.disliked_movies.joins(:cast_members).where('cast_members.name = Kim Kardashian')

You can chain your queries

Recommendable

>> current_user.hidden_minerals.order('density DESC')>> current_user.recommended_movies.where('year < 2010')>> book.liked_by.order('age DESC').limit(20)>> movie.disliked_by.where('age > 18')

You can chain your queries

RecommendableYou can also like your recommendable objects

>> user.like(movie)=> true>> user.likes?(movie)=> true>> user.rated?(movie)=> true # also true if user.dislikes?(movie)>> user.liked_movies=> [#<Movie id: 23, name: "2001: A Space Odyssey">]>> user.liked_movie_ids=> ["23"]>> user.like(book)=> true>> user.likes=> [#<Movie id: 23, name: "2001: A Space Odyssey">, #<Book id: 42, title: "100 Years of Solitude">]>> user.likes_count=> 2>> user.liked_movies_count=> 1>> user.likes_in_common_with(friend)=> [#<Movie id: 23, name: "2001: A Space Odyssey">, #<Book id: 42, title: "100 Years of Solitude">]>> user.liked_movies_in_common_with(friend)=> [#<Movie id: 23, name: "2001: A Space Odyssey">]>> movie.liked_by_count=> 2>> movie.liked_by=> [#<User username: 'davidbowman'>, #<User username: 'frankpoole'>]

RecommendableObviously, You can also DISLIKE your recommendable

objects>> user.dislike(movie)>> user.dislikes?(movie)>> user.disliked_movies>> user.disliked_movie_ids>> user.dislikes>> user.dislikes_count>> user.disliked_movies_count>> user.dislikes_in_common_with(friend)>> user.disliked_movies_in_common_with(friend)>> movie.disliked_by_count>> movie.disliked_by

RecommendableRecommendations

>> friend.like(Movie.where(:name => "2001: A Space Odyssey").first)>> friend.like(Book.where(:title => "A Clockwork Orange").first)>> friend.like(Book.where(:title => "Brave New World").first)>> friend.like(Book.where(:title => "One Flew Over the Cuckoo's Next").first)>> user.like(Book.where(:title => "A Clockwork Orange").first)=> [#<User username: "frankpoole">, #<User username: "davidbowman">, ...]>> user.recommended_books # Defaults to 10 recommendations=> [#<Book title: "Brave New World">, #<Book title: "One Flew Over the Cuckoo's Nest">]>> user.similar_raters # Defaults to 10 similar users=> [#<>> user.recommended_movies(10, 30) # 10 Recommendations, offset by 30 (i.e. page 4)=> [#<Movie name: "A Clockwork Orange">, #<Movie name: "Chinatown">, ...]>> user.similar_raters(25, 50) # 25 similar users, offset by 50 (i.e. page 3)=> [#<User username: "frankpoole">, #<User username: "davidbowman">, ...]

RecommendableJaccard Similarity

Marcel likes A, B, C and dislikes DAmanda likes A, B and dislikes CGuilherme likes C, D and dislikes AFlavio likes B, C, E and dislikes D

J(Marcel, Amanda) =([A,B].size + [].size - [C].size - [].size) / [A,B,C,D].size

J(Marcel, Amanda) =2 + 0 - 1 - 0 / 4 = 1/4 = 0.25

J(Marcel, Guilherme) =([C].size + [].size - [A].size - [D].size) / [A,B,C,D].size

J(Marcel, Guilherme) =1 + 0 - 1 - 1 / 4 = 1/4 = - 0.25

J(Marcel, Flavio) =([B,C].size + [D].size - [].size - [].size) / [A,B,C,D, E].size

J(Marcel, Flavio) =2 + 0 - 0 - 0 = 2/5 = 0.4

MostSimilar(Marcel) = [ (Flavio, 0.4) , (Amanda, 0.25) , (Guilherme, -0.25)]

RecommendableRecommendations

>> Movie.top=> #<Movie name: "2001: A Space Odyssey">>> Movie.top(3)=> [#<Movie name: "2001: A Space Odyssey">, #<Movie name: "A Clockwork Orange">, #<Movie name: "The Shining">]

The best of your recommendable models

Wilson score confidence - Reddit Algorithm

RecommendableCallbacks

class User < ActiveRecord::Base has_one :feed

recommends :movies after_like :update_feed

def update_feed(obj) feed.update "liked #{obj.name}" endend

apotonick/hooks to implement callbacks for liking, disliking, etc

Recommendable

Recommendable::Helpers::Calculations.update_similarities_for(user.id)Recommendable::Helpers::Calculations.update_recommendations_for(user.id)

Manual recommendations

redis makes the magic!

Recommendable

module Recommendable module Workers class Resque include ::Resque::Plugins::UniqueJob if defined?(::Resque::Plugins::UniqueJob) @queue = :recommendable

def self.perform(user_id) Recommendable::Helpers::Calculations.update_similarities_for(user_id) Recommendable::Helpers::Calculations.update_recommendations_for(user_id) end end endend

Recommendations over Queueing SystemPut the workers to do the job! (SideKiq, Resque, DelayedJob)

Recommended Books

SatnamAlag, Collective Intelligence in Action, Manning Publications, 2009

Toby Segaran, Programming Collective Intelligence, O'Reilly, 2007

Recommended Books

Exploring everyday thingswith R and Ruby, Sau Chang,O’Reilly, 2012

Recommended Course

https://www.coursera.org/course/recsys

Ruby developers, It does exist

Ruby in the world of recommendations

(also machine learning, statistics and visualizations..)

Marcel Caraciolo@marcelcaracioloDeveloper, Cientist, contributor to the Crab recsys project,works with Python for 6 years, interested at mobile,education, machine learning and dataaaaa!Recife, Brazil - http://aimotion.blogspot.com

Recommender Systems with Ruby (adding machine learning, statistics, etc)

ruby python

vs ruby

ruby developer answer

ruby lanado em

google python

python lanado em

machine learning

important key of google

Technology

Ruby on Rails [ Ruby On Rails.ppt ] - [Ruby - [Ruby-Doc.org....

Adding where to your ruby apps

Recommender Systems. Outline Limitations of Recommender...

Movie genome: alleviating new item cold start in movie...

Recommender...

Recommender Systems Handbook - Home -...

Recommender system

Recommender Introduction to Recommender Systems and ...

Open Recommender

TFR: A Tourist Food Recommender System based on...

Adding where to your ruby apps lt - (q con)

Personalized Recommender by Exploiting Domain based Expert.....

Affective recommender systems: the role of emotions in...

Recommender Systems - Universidade NOVA de...

Tutorial: Recommender Systemswelling/teaching/CS77B... ·.....

A Fuzzy Recommender System for eElections - unifr.ch Fuzzy.....