Learned Embeddings for Search and Discovery at Instacart
Sharath RaoData Scientist / ManagerSearch and Discovery
Collaborators: Angadh Singh
v
Talk Outline
• Quick overview of Search and Discovery at Instacart
• Word2vec for product recommendations
• Extending Word2vec embeddings to improve search ranking
v
About me
• Worked on several ML products in catalog, search and personalization at Instacart since 2015
• Currently leading full stack search and discovery team
@sharathrao
v
Search and Discovery@ Instacart
Help customers find what they are looking for and discover what they might like
v
Grocery Shopping in “Low Dimensional Space”
Search
Restock
Discover
v“Here is what I want”
Query has easily accessible information content
“Here I am, what next?”
Context is the query, and recommendations vary
as contexts do
Search Discovery
v
Entities we try to model
items
products
aisles
departments
retailers
queries
customers
brands
Most of our data products are about modeling relationships
between them
(part of which is learning embeddings)
v
Common paradigm for search/discovery problems
1st phase: Candidate Generation
2st phase: Reranking
v
• Select top 100s from among potentially millions
• Must be fast and simple• often not even a learned model
• Recall oriented
Candidate Generation
v
• Ranks fewer products, but users richer models/features
• Tuned towards high precision
• Often happens in real-time
Reranking
vWord2vec for Product Recommendations
v
“Frequently bought with” recommendations
Not necessarily consumed together
Help customers shop for the next item
Probablyconsumed together
vQuick Tour of Word2vec
(simplified, with a view to develop some intuition)
v• We need to represent words as features in a model/task
• Naive representation => one hot encoding
• Problem: Every word is dissimilar to every other word => not intuitive
One-hot encodings are good, but not good enough
v
• stadium, field and restaurant are in some sense similar
• stadium and field are more similar than stadium and restaurant
• game has another meaning/sense that is similar to stadium/field/restaurant
What one-hot encodings fail to capture
Observation 1: “game at the stadium”
Observation 2: “game at the field”
Observation 3: “ate at the restaurant”
Observation 4: “met at the game”
Data What models must learn
None of these are easy to learn with one-hot encoded representations without supplementing with hand-engineered features
vYou shall know a word by the company it keeps - Firth, J.R., 1957
Core motivation behind semantic word representations
v
• Learns a feature vector per word
• for 1M vocabulary and 100 dimensional vector, we learn 1M vectors => 100M numbers
• Vectors must be such that words appearing in similar contexts are closer than any random pair of words
• Must work with unlabeled training data
• Must scale to large (unlabeled) datasets
• Embedding space itself must be general enough that word representations are broadly applicable
Word2vec is a scalable way to learn semantic word representations
vWord2vec beyond text and words
v
On graphs where random walks are the sequences
v
On songs where plays are the sequences
v
Even the emojis weren’t spared!
vYou shall know a
word product
by the company it keeps by what its purchased with
v
We applied word2vec to purchase sequences
Typical purchasing session contains tens of cart adds
Sequences of products added to cart are the ‘sentences’
v
“Frequently Bought With” Recommendations
Extract Training
Sequences
Learn word2vec
representations
Eliminate substitute products
Event Data
Approximate Nearest Neighbors
Cache recommendations
v
Word2vec for product recommendations
Surfaces complementary
products
v
Next step is to make recommendations contextual
Not ideal if already shopped for sugar recently
inappropriate if allergic to walnuts
Not if favorite brand of butter isn’t
otherwise popular
vWe see word2vec recommendations as a candidate generation step
vWord2vec for Search Ranking
v
The Search Ranking ProblemIn response to a query, rank products in the order that
maximizes the probability of purchase
v
The Search Ranking Problem
Learning to Rank Model
Candidate Generation
Boolean matching
BM25 ranking
Synonym/Semantic Expansion
(Online) Reranking
Matching features
Historical aggregates
Personalization features
v
Search Ranking - Training
Generate training features
Label Generation with Implicit Feedback
Event Data
Learning to Rank Training
Model Repository
v
Search Ranking - Scoring
Online reranker
query
Model Repository
Reranked products
Top N products
Top N
productsprocessed query
Final ranking
v
Features for Learning to Rank
Historical aggregates => normalized purchase counts
High coverage Low precision
Low coverage More precision
[query, product]
[query, product, user]
[query, brand]
[query, aisle][product]
But historical aggregates alone are not enough. Because sparsity and cold start.
[user, brand]
v
Learning word2vec embeddings from search logs
• Representations learnt from wikipedia or petabyte text aren’t ideal for Instacart
• No constraint that word2vec models must be learnt on temporal or spatial sequences
• We constructed training data for word2vec from search logs
v
Training Contexts from Search Logs
• Create contexts that are observed and desirable
<query1> <purchased description of converted product1> <query2> <purchased description of converted product2> .. ..
• Learn embeddings for each word in the unified vocabulary of query tokens and catalog product descriptions
v
Examples of nearest neighbors in embeddings space
OK, so now we have word representations
Matching features require product features in the same space
bread
orowheat bonaparte zissel ryebread
raisins
sunmade monukka raisinettes fruitsource
mushrooms
cremini portabello shitake
v
Deriving other features from word embeddings
• One option is to create contexts with product identifiers in training sequences
• Promising but we will have a product cold start problem
Use learnt word embeddings to derive features for other entities such as products, brand names, queries and users etc.
v
Simple averaging of word representations works well
Product
Average embeddings of words in product description
Brands:
Average embeddings of products sold by the brand
User
Average embeddings of products purchased by the user
Aisles/Departments
Average embeddings of products in the aisle/department
Word Representation
Learnt from converted search logs
Query
Average embeddings of words in query
v
Wait, so we have 2 different representations for products?
Embeddings learnt from purchase sequences
Embeddings derived from words from search logs
Products are similar if they bought together
Products are similar if their descriptions are semantically similar
v
Product representations in different embedding spaces
Embeddings learnt from purchase sequences
Products are similar if they bought together
v
Product representations in different embedding spaces
Embeddings derived from words from search logs
Products are similar if their descriptions are semantically similar
v
Examples of nearest neighbors for products
Cinnamon Toast Crunch CerealGolden Grahams Cereal
Not much common between the product names
v
Examples of nearest neighbors for brands
v
Using word2vec features in search ranking
[query, product]
[query, brand]
[query, aisle]
[query, department]
[user, product]
[user, brand]
[user, aisle]
[user, department]
• Construct similarity scores between different entities as features
Matching Features Personalization Features
v
We saw significant improvement with word2vec features
98.0%
99.0%
100.0%
101.0%
102.0%
103.0%
104.0%
AUC Recall@10
Relative(Improvement(with(word2vec(features
Baseline With:word2vec: features
v
word2vec features rank high among top features
[query,(aisle](word2vec
[query,(product](historical(aggregate
[query,(department](word2vec
[query,(product](historical(aggregate
[user,(product](word2vec
lda(model(2
lda(model(1(
query(length
[user,(brand](word2vec
[query,(product](word2vec
bm25
position
XGBoost logistic loss
with early stopping
vOther contextual recommendation problems
v
Broad based discovery oriented recommendations
Including from stores customers may have never shopped from
v
Run out of X?
Rank products by repurchase probability
v
Introduce customers to products new in the catalog
Also mitigates product cold start problems
v
Replacement Product Recommendations
Mitigate adverse impact of last-minute out of stocks
v
Data Products in Search and Discovery
• query autocorrection• query spell correction• query expansion• deep matching/document expansion• search ranking• search advertising
• Substitute/replacements products• Frequently bought with products• Next basket recommendations• Guided discovery• Interpretable recommendations
Search Discovery
v
Thank you!
We are hiring!
Senior Machine Learning Engineer http://bit.ly/2kzHpcg