| 0 Daniel Kershaw (@danjamker) Building Recommenders 20 th September 2017
| 0
Daniel Kershaw (@danjamker)
Building Recommenders
20th September 2017
| 1
Mendeley
• Reference Manager
• Social Network
• Publication Catalogue
| 2
Science Direct
• Scientific publication database
• Used by the majority of
university and research
institutions
• Contains 12 million articles of
content from 3,500 academic
journals and 34,000 e-books
| 3
Why Recommendations
Pull
Allow users to discover more content
Make it easier to navigate catalogue
| 4
Why Recommendations
Pull
Allow users to discover more content
Make it easier to navigate catalogue
Push
Highlight new content to users
Bring users back to service
| 5
The five core components
Data Collection
Recommender Model
Recommendation Post Processing
Online Modules
User Interface
| 6
Outline
Developed Algorithms – keeping it simple
Practical Considerations – don’t look stupid
Implementation – how to scale a system
Evaluation – what is good enough
Evolution – what’s changed over time
Future Direction – the future’s bright the future’s is deep
| 7
Developed Algorithms
| 8
Available Data
Implicit
User libraries (Mendeley)
User article interactions (Science Direct)
Content
Abstracts
Titles
References
| 9
Content Based
Similarity between what users
have read
Similarity in references
Collaborative Collaborative
Matrix Factorization
KNN
LDA
Potential Methods
| 10
User item interaction matrix
User base CF – (kNN)
https://buildingrecommenders.wordpress.com/2015/11/18/overview-of-recommender-algorithms-part-2/
| 11
Similarity between query users and other readers
User base CF – (kNN)
https://buildingrecommenders.wordpress.com/2015/11/18/overview-of-recommender-algorithms-part-2/
| 12
Similarity between all users
User base CF – (kNN)
https://buildingrecommenders.wordpress.com/2015/11/18/overview-of-recommender-algorithms-part-2/
| 13
Generating recommendations for user
User base CF – (kNN)
https://buildingrecommenders.wordpress.com/2015/11/18/overview-of-recommender-algorithms-part-2/
| 14
• Ability to scale
• Matrix incredibly sparse
Why not Matrix Factorization
| 15
Practical Considerations
| 16
Explore/Exploit (Dithering)
Recommendations generated in batch
Users want an interactive experience
Slight shuffles give the impression of
freshness
Allow for the exploration of the list if only
a proportion shown
𝑠𝑐𝑜𝑟𝑒𝑑𝑖𝑡ℎ𝑒𝑟𝑒𝑑 = log 𝑟𝑎𝑛𝑘 + 𝑁 0, log 𝜖
where 𝜀 =∆ 𝑟𝑎𝑛𝑘
𝑟𝑎𝑛𝑘and tipically 𝜀 ∈ [1.5,2]
| 17
Impression Discounting
• Experience deteriorates if exposed to the same information
• Push recommendations seen before down the list
Rank
Impressions
| 18
Impression Discounting
• Experience deteriorates if exposed to the same information
• Push recommendations seen before down the list
𝑠𝑐𝑜𝑟𝑒𝑛𝑒𝑤 = scoreoriginal ∗ (w1 ∗ g impCount + w2 ∗ g lastSeen )
See Lee, P. et. al
| 19
Business Logic (Pre and Post Filtering)
Don’t show items they already have (bought, added, consumed)
Don’t feed the recommender positive feedback from recommender
Don’t recommend out of stock items
• A bad recommender has a cost
- Can be greater than not receiving a recommendation
| 20
Implementation
| 21
Systems Architecture
Impression
Discounting
API
Front End
AWS
Dithering
Candidate Selection
Conte
nt
Based
Item
2Ite
m
CF
Online
Offline
Logs
| 22
The unbundled mess
| 23
System
• Which run generated the
recommendation
• What was served to the user
• How was the score modified
• What was removed from the
recommendations
User (Feedback loop)
• What was displayed
• What was clicked
• When were they served
• Where the recommendations
displayed
Logging
Used for both debugging and feeding information to recommender
| 24
Evolutions
| 25
• User to Item CF
• Impression Discounting
Mendeley – Desktop Application
| 26
Mendeley – Online
• Implicit – serves recommendations based on user libraries
• Recent Activity – based off recent additions to a users library
• Research Interests - based on user generated tags
• Discipline – based on their self identified discipline
Most Personalized
Least Personalized
See Hristakeva, M et. Al (2017)
| 27
• Remove carousels
• Focus on implicit
recommendations
• Fall back to content based
solution
Mendeley – Online
| 28
• Recommendation based of the
complete library of the user
• Don’t send the same
recommendations twice
Mendeley - Email
| 29
• Item to Item
• Take user reading history
• Get recommendations for each
item
• Interleave recommendations
• Don’t send same
recommendations twice
Science Direct - Email
| 30
Science Direct – Article Page
Item to Item
Dither
recommendations
every 30 minutes
| 31
Evaluation
| 32
Off-line Methodology
Train model QueryGround
truth
Time, user interactions
Test
| 33
Off-line evaluation - Mendeley
From Hristakeva, M et. al
| 34
Science Direct – Item-to-item
| 35
• Infrastructure takes a long time to build
• Need feedback from users to learn
1. Generate recommendations off-line
2. Send to users via email (A/A)
3. Modify method based on feedback
4. Send second set of users split into A/B buckets
Static Recommendations for quick learnings
Email to users
Modify Recommender
Email to users
| 36
Future Direction
| 37
Learning to rank (LtR)
Currently only using implicit feedback
No content used
Use CF as candidate selection
Re-rank results based on learnt model
optimised for CtR
Use item and user features
| 38
Deep Learning
Use to learn more complex features
Use as features in LtR
Build on the existing framework developed
Use pre-trained models before developing own
| 39
Conclusion (Take Homes)
• Log EVERYTHING
• Start Simple
• Iterate quickly
• Get recommendations out quickly to learn
• Don’t look stupid
• CTR ≇ Off-line Evaluation
| 40
www.elsevier.com/rd-solutions
Thank you,
Book chapter being written based on the content in this presentation
| 41
References
Hristakeva, M., Kershaw, D., Rossetti, M., Knoth, P., Pettit, B., Vargas, S., & Jack, K. (2017). Building recommender systems for scholarly information. the 1st Workshop (pp. 25–32). New York, New York, USA: ACM. http://doi.org/10.1145/3057148.3057152
Rossetti, M., Stella, F., & Zanker, M. (2016). Contrasting Offline and Online Results when Evaluating Recommendation Algorithms (pp. 31–34). Presented at the Proceedings of the 10th ACM Conference on Recommender Systems, New York, NY, USA: ACM. http://doi.org/10.1145/2959100.2959176
Lee, P., Lakshmanan, L. V. S., Tiwari, M., & Shah, S. (2014). Modeling impression discounting in large-scale recommender systems (pp. 1837–1846). Presented at the Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, New York, New York, USA: ACM Press. http://doi.org/10.1145/2623330.2623356
Koren, Y. (2010). Collaborative filtering with temporal dynamics. Communications of the ACM, 53(4), 89–97. http://doi.org/10.1145/1721654.1721677