Setting Goals and Choosing Metrics for Recommender System Evaluations Gunnar Schröder, Maik Thiele, Wolfgang Lehner Gunnar Schröder T-Systems Multimedia Solutions Dresden University of Technology UCERSTI 2 Workshop at the 5th ACM Conference on Recommender Systems Chicago, October 23th, 2011
13
Embed
Setting Goals and Choosing Metrics for Recommender System Evaluations
Recommender systems have become an important personalization technique on the web and are widely used especially in e-commerce applications. However, operators of web shops and other platforms are challenged by the large variety of available algorithms and the multitude of their possible parameterizations. Since the quality of the recommendations that are given can have a significant business impact, the selection of a recommender system should be made based on well-founded evaluation data. The literature on recommender system evaluation offers a large variety of evaluation metrics but provides little guidance on how to choose among them. The paper which is presented in this presentation focuses on the often neglected aspect of clearly defining the goal of an evaluation and how this goal relates to the selection of an appropriate metric. We discuss several well-known accuracy metrics and analyze how these reflect different evaluation goals. Furthermore we present some less well-known metrics as well as a variation of the area under the curve measure that are particularly suitable for the evaluation of recommender systems in e-commerce applications.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Setting Goals and Choosing Metrics for Recommender
System EvaluationsGunnar Schröder, Maik Thiele, Wolfgang Lehner
Gunnar SchröderT-Systems Multimedia SolutionsDresden University of Technology
UCERSTI 2 Workshopat the 5th ACM Conference on
Recommender SystemsChicago, October 23th, 2011
Setting Goals and Choosing Metrics for Recommender System Evaluation - Gunnar Schröder
How Do You Evaluate Recommender Systems?
Qualitative TechniquesQuantitative Techniques
RMSE
MAE
Precision
Recall
Area under the Curve
ROC Curves
Mean Average Precision
F1-Measure
Accuracy Metrics Non-Accuracy Metrics
User-Centric Evaluation
But why do you do it exactly this way?
Setting Goals and Choosing Metrics for Recommender System Evaluation - Gunnar Schröder
Some of the Issues This Paper Tries to Touch
A large variety of metrics have been published Some metrics are highly correlated [Herlocker 2004] Little guidance for evaluating recommenders and choosing
metrics
Which aspects of the usage scenario and the data influence the choice?
Which metrics are applicable? What do these metrics express? What are differences among them? Which metric represents our use-case best? How much do the metrics suffer from biases?
Setting Goals and Choosing Metrics for Recommender System Evaluation - Gunnar Schröder
Factors That Influence the Choice of Evaluation Metrics