IRG IR Group @ UAM Recommender Systems Evaluation Beyond Accuracy ACM Latin American School on Recommender Systems (LARS 2019) Fortaleza, Brazil, October 10, 2019 Recommender Systems Evaluation Beyond Accuracy ACM Latin American School on Recommender Systems Pablo Castells Universidad Autónoma de Madrid http://ir.ii.uam.es/castells Fortaleza, Brazil, October 10, 2019
85
Embed
Recommender systems evaluation beyond accuracyir.ii.uam.es/castells/lars2019.pdf · ACM Latin American School on Recommender Systems (LARS 2019) Fortaleza, Brazil, October 10, 2019
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Recommender Systems EvaluationBeyond Accuracy
ACM Latin American School on Recommender Systems
Pablo CastellsUniversidad Autónoma de Madrid
http://ir.ii.uam.es/castells
Fortaleza, Brazil, October 10, 2019
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Outline
1. Motivation: beyond relevance
2. Measuring novelty and diversity
3. Enhancing novelty and diversity
4. Biases in recommendation
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Outline
1. Motivation: beyond relevance
2. Measuring novelty and diversity
3. Enhancing novelty and diversity
4. Biases in recommendation
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Motivation
What is the purpose of recommendation?
Satisfying users …by making suggestions they like
If we recommend things that a user likes,
then the user will be satisfied
Ergo…
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Motivation
Do you like this?
Book Tourist attraction Music albumMovie
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Motivation
Would you find it useful to recommend it?
Probably notEverybody knows those already
Movie Book Tourist attraction Music album
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Motivation
Would you find it useful to recommend this?
Maybe, provided they are liked…
Movie Book Tourist attraction Music album
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Motivation
Would you find it useful to recommend this?
Not obvious or widely known
…but too much of the same genre?
Sci-fi Sci-fi Sci-fi Sci-fi Sci-fi
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Motivation
Would you find it useful to recommend this?
Sci-fi AnimationComedy Adventure Documentary
Seems better?
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Beyond accuracy…
How to improve?
Define
Understand
Measure
…then try to improve
NoveltyDiversity
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Definition
How different recommendations are
from “something else”
E.g. user knowledge or experience
Novelty
(Vargas & Castells RecSys 2011, Castells et al. Handbook 2015)
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Definition
How different recommendations are
to each other
How novel each item is to the other
recommended items
Diversity
(Vargas & Castells RecSys 2011, Castells et al. Handbook 2015)
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Why diverse and novel recommendations
For the sake of it: direct user satisfaction
Natural variety-seeking drive in human behavior
– Within a recommendation and over time
– Desire for the unfamiliar, alternation among the familiar
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Aspect-based diversity
With respect to a space of user “subtastes”:
genres, categories, etc.
Inspired on intent-oriented search diversity
(Vargas et al. SIGIR 2011, Wasilewski & Hurley UMAP 2018, Kaya & Bridge UMUAI 2019, etc.)
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Diversity in search
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Diversity in search
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
“Avoid redundancy of possible user intents (aspects)
as a means to cope with the uncertainty in the query”
Diversity in search
(Carbonell & Goldstein SIGIR 1998, Clarke et al. SIGIR 2008, Agrawal et al. WSDM 2009, Santos et al. WWW 2010, Santos et al. Found. & Trends in IR 2015)
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Search result diversity
Added utility Added utility
Rel
evan
t d
ocu
men
t ra
nk
Relevan
t do
cum
ent ran
k
Query senses / aspects
. . .
. . .
(query ambiguity / incompleteness)
Uniformresults Diverse
results
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Metrics
Aspect recall
Intent-aware metrics
ERR−IA =
𝑑𝑘∈𝑅
1
𝑘
𝑎∈𝒜𝑞
𝑟𝑒𝑙 𝑑𝑘 𝑎
𝑗<𝑘
1 − 𝛼 𝑟𝑒𝑙 𝑑𝑗 𝑎
Search diversity evaluation
=1
𝒜𝑞# 𝑎 ∈ 𝒜𝑞 ∃𝑑 ∈ 𝑅 that covers 𝑎
Novelty
Diversity
RelevanceRanking
(Clarke et al. SIGIR 2008, Agrawal et al. WSDM 2009, Chapelle et al. Inf. Ret. 2011)
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Metrics
Aspect recall
Intent-aware metrics
ERR−IA =
𝑑𝑘∈𝑅
1
𝑘
𝑎∈𝒜𝑞
𝑟𝑒𝑙 𝑑𝑘 𝑎
𝑗<𝑘
1 − 𝛼 𝑟𝑒𝑙 𝑑𝑗 𝑎
Aspects?
Query aspects: manually defined (e.g. TREC), Wikipedia
disambiguation, suggested query reformulations…
Document aspects: categories, clusters…
Search diversity evaluation
=1
𝒜𝑞# 𝑎 ∈ 𝒜𝑞 ∃𝑑 ∈ 𝑅 that covers 𝑎
(Clarke et al. SIGIR 2008, Agrawal et al. WSDM 2009, Chapelle et al. Inf. Ret. 2011, Santos et al. WWW 2010)
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
“Avoid redundancy of possible user intents (aspects)
as a means to cope with the uncertainty in the query”
in the observed evidence of user interests”
Aspect-based diversity in recommendation
(Vargas et al. SIGIR 2011, Wasilewski & Hurley UMAP 2018, Kaya & Bridge UMUAI 2019, etc.)
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
4 4 2 2 2
1 4 4 4
4 3 2 5 2
4 3 3 2 2
1 1 5 1 5 5
Use
rs
Items
𝑢
𝑖
Aspect-based diversity in recommendation
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
4 4 2 2 2
1 4 4 4
4 3 2 5 2
4 3 3 2 2
1 1 5 1 5 5
Use
rs
Items
User profile𝑢
𝑖
Aspect-based diversity in recommendation
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
4 3 2 5 2 User profile
Items
𝑢
𝑖
Aspect-based diversity in recommendation
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Items
Item
feat
ure
s Aspects from item features, using a “meaningful” item feature space
4 3 2 5 2𝑢
𝑖
Aspect-based diversity in recommendation
User profile
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Items
Item
feat
ure
s
4 3 2 5 2
“User aspects”
𝑢
𝑖
Aspect-based diversity in recommendation
Aspects from item features, using a “meaningful” item feature space
Derive user aspect distributions
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Items
Item
feat
ure
s Aspects from item features, using a “meaningful” item feature space
Derive user aspect distributions
4 3 2 5 2
“User aspects”
𝑢
𝑖
Aspect-based diversity in recommendation
IR diversity metrics and algorithms can now be applied
Other approaches to user interest subdivision have been considered
(Vargas et al. SIGIR 2011, Wasilewski & Hurley UMAP 2018, Kaya & Bridge UMUAI 2019, Vargas et al. OAIR 2013)
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Sales diversity
Seller perspective
How spread are recommendations over the item inventory
Catalog exposure to sales
Items
Nr
use
rs t
o w
ho
mit
em is
rec
om
end
ed
Items
Recommender BRecommender A
(Adomavicius & Kwon TKDE 2012, Li & Murata WI 2012, Vargas & Castells RecSys 2014, Jannach et al. UMUAI 2015, etc.)
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Sales diversity
. . .
. . .
. . .
. . .
“Ecosystem”One “species”
Set of all recommendations
Set of all items
Metrics: function over set of recommendations
Metrics adapted from ecology and other fields
Recommendation“slots”
One “individualof some species”
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Sales diversity
“Ecosystem”One “species”
Set of all recommendations
Set of all items
Metrics: function over set of recommendations
Metrics adapted from ecology and other fields
Recommendation“slots”
One “individualof some species”
. . .
. . .
. . .
. . .
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Sales diversity
“Ecosystem”One “species”
Set of all recommendations
Set of all items
Recommendation“slots”
One “individualof some species”
. . .
. . .
. . .
. . .
Aggregate diversity
Total number of different items recommended in top 𝑛
Equivalent to “species richness”
Aggdiv = ∪𝑢 𝑅𝑢
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Aggregate diversity
Total number of different items recommended in top 𝑛
Equivalent to “species richness”
Gini-Simpson index
GSI = 1 −
𝑖
𝑝𝑖2
𝑝𝑖 = ratio of users to whom 𝑖 is recommended
Gini coefficient
Entropy
H = −
𝑖
𝑝𝑖 log2 𝑝𝑖
Sales diversity
G =1
ℐ − 1
𝑘=1
ℐ
2𝑘 − ℐ − 1 𝑝𝑖𝑘
Aggdiv = ∪𝑢 𝑅𝑢
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Sales diversity
Aggregate diversity: A as good as B
Gini, Gini-Simpson, Entropy: B better than A
Items
Nr
use
rs t
o w
ho
mit
em is
rec
om
end
ed
Items
Recommender A Recommender B
𝑛 𝑛
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Common underlying principle to different diversity notions
Context
Recommended item
Target user’sexperience
Everyone else’sexperience
Everyone else’srecommendations
Other items in thesame recommendation
UnexpectednessIntra-listdiversity
Long-tailnovelty Sales diversity
Distance or identity
Item novelty model
(Vargas & Castells RecSys 2011, Castells et al. Handbook 2015)
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Breaking the feedback loop: multi-armed bandits
Banditpolicy 1. Select
arm2. Get
reward
Estimated(models)
𝜇
𝜇
𝜇
3. Update estimated reward model of arm
True (unob-served)
𝜇
𝜇
𝜇
Reward distributions
Arms
Multi-armed bandit problem:
Choose an arm iteratively and maximize total payoff
without knowing reward distributions in advance
(Sutton & Barto RL book 2018, Chapelle & Li NIPS 2011, etc.)
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Breaking the feedback loop: multi-armed bandits
Banditpolicy
Estimated(models)
3. Update estimated reward model of arm𝜇
𝜇
𝜇
True (unob-served)
𝜇
𝜇
𝜇
Recommendation keeps an ingredient of randomness (exploration) in its actions– Aware (explicit model) of uncertainty in present knowledge about the user
– Gives apparently suboptimal options a chance to be reconsidered
Actions can be items, latent factors, clusters, neighbors, algorithms…
Do much better in the mid/long run!!
Reward distributions
Arms
1. Selectarm
2. Getreward
(Li et al. SIGIR 2016, Lacerda Neurocomputing 2017, McInerney et al. RecSys 2018, etc.)
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Conclusions
Novelty, bias and reinforcement learning are related problems
Novelty & diversity are now state of the art
– Different notions and metrics for different angles
Bias: popular items score high in accuracy in offline experiments
– Progress made in understanding and seeking to avoid
Reinforcement loop bias: multi-armed bandits and
reinforcement learning can greatly help
– And improve sales diversity
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Open directions
Large room for research, matters to industry
Better understand the role of novelty and diversity in user needs
Unbiased evaluation
– How to estimate propensity
– Model complex biases e.g. involving user pairs
– Build unbiased datasets
Multi-armed bandits and reinforcement learning
– How to map the task, algorithmic research
– How to evaluate methods and represent different scenarios
(Nguyen et al. WWW 2014, Kapoor et al. RecSys 2015, Karumur et al. CSCW 2016, etc.)
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Thank you for your attention!
Questions?
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
References
Adamopoulos, P., Tuzhilin, A. On Unexpectedness in Recommender Systems: Or How to Expect the Unexpected. ACM TIST 5(4), Special Issue on Novelty and Diversity in Recommender Systems, January 2015.
Adomavicius, G., Kwon, Y. Improving Aggregate Recommendation Diversity Using Ranking-Based Techniques. IEEE Trans. on Knowl. and Data Eng. 24(5), May 2012.
Agrawal, R., Gollapudi, S., Halverson, A., Ieong, S. Diversifying search results. WSDM 2009, Barcelona, Spain, pp. 5-14.
Anderson, C. The Long Tail: Why the Future of Business is Selling Less of More. Hyperion, New York, NY, USA, 2006.
Bellogín, A. , Castells, P. and Cantador, I. Statistical Biases in Information Retrieval Metrics for Recommender Systems. Information Retrieval 20(6), July 2017, 606-634.
Brickman, P., D’Amato, B. Exposure Effects in a Free Choice Situation. Journal of Personality and Social Psychology 32(3), 1975, pp. 415-420.
Cañamares, R. and Castells, P. Should I follow the Crowd? A Probabilistic Analysis of the Effectiveness of Popularity in Recommender Systems. SIGIR 2017, Ann Arbor, MI, USA, pp. 415-424.
Cañamares, R. and Castells, P. A Probabilistic Reformulation of Memory-Based Collaborative Filtering – Implications on Popularity Biases. SIGIR 2017, Ann Arbor, MI, USA, pp. 215-224.
Cañamares, R., Redondo, M., Castells, P. Multi-Armed Recommender System Bandit Ensembles. RecSys 2019, Copenhagen, Denmark, pp. 432-436.
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Carbonell, J. G. and Goldstein, J. The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries. SIGIR 1998, Melbourne, Australia, 335-336.
Castells, P, Hurley, N. J., Vargas, S. Novelty and Diversity in Recommender Systems. In: Recommender Systems Handbook, 2nd edition, F. Ricci, L. Rokach, B. Shapira (Eds.). Springer, New York, NY, USA, pp. 881-918.
Castells, P., Wang. J., Lara, R., Zhang. D. Workshop on novelty and diversity in recommender systems – DiveRS 2011. RecSys 2011, Chicago, Illinois, USA, pp. 393-394.
Celma, O. and Herrera, P. A New Approach to Evaluating Novel Recommendations. RecSys 2008, Lausanne, Switzerland, pp. 179-186.
Chaney, A. J. B., Stewart, B. M., Engelhardt, B. E. How algorithmic confounding in recommendation systems increases homogeneity and decreases utility. RecSys 2018, Vancouver, Canada, pp. 224-232.
Chapelle, O., Ji, S., Liao, C., Velipasaoglu, E., Lai, L., Wu, S-L. Intent-based diversification of web search results: metrics and algorithms. Information Retrieval 14(6), December 2011, pp. 572-592.
Chapelle, O. and Li, L. An empirical evaluation of Thompson Sampling. NIPS 2011, Granada, Spain, pp. 2249-2257.
Chen, H. and Karger, D. R. Less is More. 29th Annual International ACM Conference on Research and Development in Information Retrieval (SIGIR 2006). Seattle, WA, USA, pp. 429-436.
Clarke, C. L. A., Kolla, M., Cormack, G. V., Vechtomova, O., Ashkan, A., Büttcher, S., MacKinnon, I. Novelty and diversity in information retrieval evaluation. SIGIR 2008, Singapore, pp. 659-666.
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Clarke, C. L. A., Craswell, N., Soboroff, I, Cormack, G. V. Overview of the TREC 2010 Web Track. TREC 2010, Gaithersburg, MD, USA.
Clarke, C. L. A., Craswell, N., Soboroff, I., Ashkan, A. A Comparative Analysis of Cascade Measures for Novelty and Diversity. WSDM 2011, Hong-Kong, China, pp. 75-84.
Cremonesi, P., Koren, Y. and Turrin, R. Performance of recommender algorithms on top-n recommendation tasks. RecSys 2010, Barcelona, Spain, pp. 39-46.
Fleder, D. M. and Hosanagar, K. Blockbuster Culture’s Next Rise or Fall: The Impact of Recommender Systems on Sales Diversity. Management Science 35(5), May 2009, pp. 697-712.
Ge. M., Delgado-Battenfeld, C., Jannach,D. Beyond accuracy: evaluating recommender systems by coverage and serendipity. RecSys 2010, Barcelona, Spain, pp. 257-260.
Hurley, N., Zhang, M. Novelty and Diversity in Top-N Recommendation – Analysis and Evaluation. ACM TIIT 10(4), March 2011.
Iaquinta, L., de Gemmis, M., Lops, P., Semeraro, G., Filannino, M., Molino, P. Introducing Serendipity in a Content-based Recommender System. HIS 2008, Barcelona, Spain, September 2008.
Jalili, M., Javari, A. Accurate and novel recommendations: An algorithm based on popularity forecasting. ACM TIST 5(4), Special Issue on Novelty and Diversity in Recommender Systems, Jan. 2015.
Jannach, D., Lerche, L., Kamehkhosh, I. Jugovac, M. What recommenders recommend: an analysis of recommendation biases and possible countermeasures. UMUAI 25(5), Dec. 2015, pp. 427-491.
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Kahn, B. E. Consumer variety-seeking among goods and services: An integrative review. Journal of Retailing and Consumer Services 2(3), July 1995, pp.139-148.
Kaminskas, M., Bridge, D. Diversity, Serendipity, Novelty, and Coverage: A Survey and Empirical Analysis of Beyond-Accuracy Objectives in Recommender Systems. ACM TIIS 7(1), March 2017.
Kapoor, K., Kumar, V., Terveen, L. G., Konstan, J. A., Schrater, P. R. “I like to explore sometimes”: Adapting to Dynamic User Novelty Preferences. RecSys 2015, Vienna, Austria, pp. 19-26.
Karumur, R. P., Nguyen, T. T., Konstan, J. A. Early Activity Diversity: Assessing Newcomer Retention from First-Session Activity. CSCW 2016, San Francisco, CA, USA, pp. 594-607.
Lacerda, A. Multi-Objective Ranked Bandits for Recommender Systems. Neurocomputing 246, July 2017, 12-24.
Li, S., Karatzoglou, A. and Gentile, C. Collaborative Filtering Bandits. SIGIR 2016, Pisa, pp. 539-548.
Maddi, S. R. The Pursuit of Consistency and Variety. In Abelson, R. P. et al. (Eds.), Theories of Cognitive Consistency: A Sourcebook, Rand McNally, Chicago, 1968, pp. 61-85.
Marlin, B. M., Zemel, R. S. Collaborative prediction and ranking with non-random missing data. RecSys 2009, New York, NY, USA, pp. 5-12.
McAlister, L. Choosing Multiple Items from a Product Class. Journal of Consumer Research 6, December 1979, pp. 213-224.
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
McAlister, L., Pessemier, E. A. Variety seeking behavior: an interdisciplinary review. Journal of Consumer Research 9, December 1982.
McNee, S. M., Riedl, J., Konstan, J. A. Being Accurate is Not Enough: How Accuracy Metrics have hurt Recommender Systems. CHI 2006, Montréal, Canada, pp. 1097-1101.
McInerney, J., Lacker, B., Hansen, S., Higley, K., Bouchard, H., Gruson, A. and Mehrotra, R. Explore, exploit, and explain: personalizing explainable recommendations with bandits. RecSys 2018, Vancouver, Canada, pp. 31-39.
Mourão, F., Fonseca, C., Araújo, C., Meira Jr., W. The Oblivion Problem: Exploiting Forgotten Items to Improve Recommendation Diversity. Workshop on Novelty and Diversity in Recommender Systems (DiveRS 2011) at RecSys 2011, Chicago, Illinois, October 2011, pp. 27-34.
Murakami, T., Mori, K., Orihara, R. Metrics for Evaluating the Serendipity of Recommendation Lists. JSAI 2007. Mizayaki, Japan, June 2007. Also in Springer Verlag LNCS Vol. 4914, 2008, pp 40-46.
Nguyen T. T., Hui, P-M., Harper, F. M., Terveen, L. G., Konstan, J. A. Exploring the filter bubble: the effect of using recommender systems on content diversity. WWW 2014, Seoul, Korea, pp. 677-686.
Onuma, K., Tong, H., Faloutsos, C. TANGENT: a novel, ‘Surprise me’, recommendation algorithm. KDD 2009, pp. 657-666.
Park, Y-J., Tuzhilin, A. The long tail of recommender systems and how to leverage it. RecSys 2008, Lausanne, Switzerland, pp. 11-18.
Patil, G. P., Taillie, C. Diversity as a Concept and its Measurement. Journal of the American Statistical Association 77(379), September 1982, pp. 548-561.
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Raju, P. S. Optimum Stimulation Level: Its Relationship to Personality, Demographics and Exploratory Behavior. Journal of Consumer Research 7(3), December 1980, pp. 272-282.
Ribeiro, M. T., Lacerda, A., Veloso, A. and Ziviani, N. Pareto-efficient hybridization for multi-objective recommender systems. RecSys 2012, Dublin, Ireland, September 2012, pp. 19-26.
Salganik, M. J., Dodds, P. S. and Watts, D. J. Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market. Science 311(5762), February 2006, pp. 854-856.
Santos, R. L. T., Macdonald, C., Ounis, I. Exploiting query reformulations for web search result diversification. WWW 2010, Raleigh, NC, USA, April 2010, pp. 881-890.
Santos, R. L. T., Macdonald, C., Ounis, I. Search Result Diversification. Foundations and Trends in Information Retrieval 9(1), 2015.
Sanz-Cruzado, J. Castells, P. Enhancing Structural Diversity in Social Networks by Recommending Weak Ties. RecSys 2018, Vancouver, Canada, pp. 233-241.
Sanz-Cruzado, J., Castells, P., López, E. A Simple Multi-Armed Nearest-Neighbor Bandit for Interactive Recommendation. RecSys 2019, Copenhagen, Denmark, pp. 358-362.
Schnabel, T., Swaminathan, A., Singh, A., Chandak, N. and Joachims, T. Recommendations as Treatments: Debiasing Learning and Evaluation. ICML 2016, New York, NY, USA, pp. 1670-1679.
Shi, Y., Zhao, X., Wang, J., Larson, M., Hanjalic, A. Adaptive diversification of recommendation results via latent factor portfolio. SIGIR 2012, Portland, OR, USA, pp. 175-184.
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Sinha, A., Gleich, D. F. Ramani, K. Deconvolving Feedback Loops in Recommender Systems. NIPS 2016, Barcelona, Spain, December 2016, pp. 3243-3251.
Smyth, B. McClave, P. Similarity vs. diversity. ICCBR 2001. London, UK, pp. 347-361.
Steck, H. Training and Testing of Recommender Systems on Data Missing not at Random. KDD 2010, Washington D. C., USA, pp. 713-722.
Steck, H. Item popularity and recommendation accuracy. RecSys 2011, Chicago, IL, pp. 125-132.
Sutton R. and Barto, A. Reinforcement Learning: An Introduction (2nd ed.). MIT Press, Cambridge, MA, USA, 2018.
Swaminathan, A., Krishnamurthy, A., Agarwal, A., Dudik, M., Langford, J., Jose, D. and Zitouni, I. Off-policy Evaluation for Slate Recommendation. NIPS 2017, Long Beach, CA, USA, pp. 3635-3645.
Vallet, D. and Castells, P. Personalized Diversification of Search Results. SIGIR 2012, Portland, OR, USA, pp. 841-850.
Varadarajan, P. Product Diversity and Firm Performance: An Empirical Investigation. Journal of Marketing 50(3), July 1986, pp. 43-57.
Vargas, S., Castells, P. and Vallet, D. Intent-Oriented Diversity in Recommender Systems. SIGIR 2011, Beijing, China, pp. 1211-1212.
Vargas, S. and Castells, P. Rank and Relevance in Novelty and Diversity Metrics for Recommender Systems. RecSys 2011. Chicago, Illinois, pp. 109-116.
IRGIR Group @ UAM
Recommender Systems Evaluation Beyond AccuracyACM Latin American School on Recommender Systems (LARS 2019)
Fortaleza, Brazil, October 10, 2019
Vargas, S. and Castells, P. Exploiting the Diversity of User Preferences for Recommendation. OAIR 2013, Lisbon, Portugal, May 2013.
Veloso, A., Ribeiro, M., Lacerda, A., Moura, E., Hata, I. and Ziviani, N. Multi-Objective Pareto-Efficient Approaches for Recommender Systems. ACM TIST 5(4), Special Issue on Novelty and Diversity in Recommender Systems, January 2015.
Yang, L., Cui, Y., Xuan, Y. , Wang, C. , Belongie, S. and Estrin, D. Unbiased Offline Recommender Evaluation for Missing-Not-At-Random Implicit Feedback. RecSys 2018, Vancouver, Canada, pp. 279-287.
Zhang, M. and Hurley, N. Avoiding Monotony: Improving the Diversity of Recommendation Lists. RecSys 2008, Lausanne, Switzerland, 123-130.
Zhang, M., Hurley, N. Novel Item Recommendation by User Profile Partitioning. Web Intelligence 2009, pp. 508-515.
Zhang, Y. C., Ó Séaghdha, D., Quercia, D., Jambor, T. Auralist: introducing serendipity into music recommendation. WSDM 2012, Seattle, WA, USA, pp. 13-22.
Zhou, T., Kuscsik, Z., Liu, J-G., Medo, M., Wakeling, J. R., Zhang, Y-C. Solving the apparent diversity-accuracy dilemma of recommender systems. PNAS 107(10), March 2010, pp. 4511-4515.
Ziegler, C-N., McNee, S. M., Konstan, J. A., Lausen, G. Improving recommendation lists through topic diversification. WWW 2005, Chiba, Japan, pp. 22-32.