An architecture for evaluating recommender systems in real world scenarios

An architecture for evaluating recommender systems in real world scenarios

Master Thesis Manuel Blechschmidt 2011

SupervisorProf. Dr. Christoph MeinelM.Sc. Rehab Alnemr

Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

2

Christmas 2009 ...


3

Agenda

■ Motivation and Current Research■ Solution□ Use Cases & Requirements□ Wireframes□ Implementation

■ Related Work■ Conclusion


4

Motivation and Current Research


5

Experiment


6

Choice


7

Motivation

■ The choice overload problem is well known in psychology

□ It is necessary to do a preselection for the customer

■ Recommender systems are already very successful to decrease the choice overload problem in some domains

□ Product-to-Product Recommendation Amazon.com→□ Movie Recommendation NetFlix→

■ Algorithms already produce great results

■ Already research in soft factores like: Diversity, Serendepity, Trust, Explanations

not a lot of emprical studies how these influences customers → → no cross domain data sets not a lot of business intereset integration→


8

Current Algorithms and Developments

■ Matrix Factorization (best RMSE 0.855 for NetFlix Dataset)

□ SVD

□ SVD++ R.M.Bell, Y. Koren, and C. Volinsky

□ TimeSVD++ R.M.Bell, Y. Koren, and C. Volinsky

■ Collaborative Filtering

□ Item based

□ User based

■ Performance gains

□ ALS1 István Pilászy, Dávid Zibriczky, Domonkos Tikk

■ Some of the algorithms already implemented in a distributed manner Mahout, MyMedia


9

Empirical Studies

■ Current empirical studies (RecSys 2010)□ Understanding Choice Overload in Recommender Systems

174 participants□ Eye-Tracking Product Recommendersʼ Usage

18 participants□ Recommender Algorithms in Activity Motivating Games

180 participants□ Group-Based Recipe Recommendations: Analysis of Data Aggregation

Strategies170 participants

□ A User-Centric Evaluation Framework of Recommender Systems807 participants

□ Information Overload and Usage of Recommendations466 participants

□ ...


10

Current Problems

■ Not a lot of big empirical studies how recommender quality influence consumer behavior especially

□ Acurarcy

□ Familiarity

□ Serendipity

□ Attractiveness

□ Enjoyability

□ Novelty

□ Diversity

□ Context Compatibility

■ Taken from A User-Centric Evaluation Framework of Recommender Systems


11

Evaluating in real world

■ Most of the academia persons do not know enough persons which are willing to test the algorithms. Therefore the following things are difficult:

□ Evaluating User Interfaces

□ Evaluating Maintenance

□ Evaluating Scalibility

□ Evaluating Performance


12

Solution


13

Master Thesis

■ Building and maintaining an evaluation platform for recommender systems in real world scenarios

■ Maintenance challenges in running a recommender system

■ Empirical study about user behavior

□ Brand loyalty

□ Pricing

□ Timing


14

Solution: Use Cases


15

Roles

■ 5 Roles with different point of views and different interests and goals

■ The roles are describeded with description and goals

■ Example:

□ Provider

□ A provider is a legal personality which has as primary goal to optimize a particular objective. In an economic context this is most of the time a business goal like raise profit or optimize conversion rates. …

□ Goals:– optimizing an objective– get forecasts– ensure privacy of his data


16

Use Cases and Requirements

■ Use Cases and Requirements are described based on IEEE 830

■ A use case is defined by:

□ Id

□ Name

□ Summary

□ Roles

□ Preconditions

□ Postconditions

□ Wireframes

□ More optional attributes


17

Use Case Example C1 Design User Interaction

■ Id: C1 Name: Design User Interaction

■ Summary: When a user interaction should be run like a newsletter or an item-to-item recommendation the consultant has to do the following steps: …

■ Roles: Consultant

■ Preconditions□ User is logged in

□ User has the Consultant role

□ At least one user interaction is implemented

□ At least one provider is associated with the consultant

□ The provider has the necessary data which is needed for the user interaction

■ Postconditions□ Provider received an email for approving the user interaction

□ User interaction is created in the system


18

C1 Design User Interaction


19



20



21



22

Implemented Architecture


23

Logical Modularization


24

Survey Module Entities


25

Survey Module Services


26

Demo


27

Implemented User Interaction chocStore


28

Related Work


29

Related Work: Competition

■ NetFlix Grand Prize 2006 – 2009

□ 1.000.000 $ to make CineMatch 10% better

□ Lots research of papers

■ KDD Cup 2011 Recommending Music Itemsbased on the Yahoo! Music Dataset

■ ECML/PKDD’2007 DISCOVERY CHALLENGE

□ User 1 User’s behaviour prediction


30

Related Work: Platforms

■ GroupLens Research of University of Minnesota

□ MovieLens 1997 http://movielens.umn.edu/

■ RichRelevance RecLab 2011

□ RecLab: A System For eCommerce Recommender Research with Real Data, Context and Feedback

■ Knowledge and Data Engineering Group of Uni Kassel

□ 2006 BibSonomy is a system for sharing bookmarks and lists of literature.


31

Further Research

■ Implement more user interactions

□ Item-to-Item recommender

■ Prove that the platform is scalable

■ Run the platform for a long time and evaluate usage

■ Integrate more companies

■ Promote plattform in science and economics

■ Take part at research projects together with companies


32

Conclusion


33

Conclusion

■ An enterprise ready platform was defined and implemented

■ Companies already applied for using

■ One example user interaction was implemented

□ chocStore

■ Statistical test can be applied to the data to give scientific results


34

Questions

Questions?


35

Backup: What is a recommender?


36


37


38


39

An architecture for evaluating recommender systems in real world scenarios

Technology