Making search better by tracking & utilizing user search behavior

Making search better by tracking & utilizing user search behavior

Sameer Maggon

Agenda

• About Me !

• Measuring Search Quality !

• Quality Metrics based on users interactions !

• Quality Metrics based on User Tagging (labelers) !

• Improving Search Results !

• Using user behavior to improve results ranking - Example

2

3

Sameer Maggon

Built relevance based search platform for AT&T Interactive & properties including yp.com, buzz.com, yellowpages.com

Founder of Cloud based Zero Management Solution for Search

Engineering Alumni

@maggonhttp://linkedin.com/in/maggon

Consulted for numerous Startups to Fortune 500 companies around Search & Discovery.

Search seems Easy

4

Index all content Put a search box Show google like results

1 2 3

5

• How do you know that your users are finding what they are looking for? !

• How do you know what impact your one-off fix has on an aggregate? !

• Seemingly good result list to one might be irrelevant to another (e.g. mosaic)

Is It?

6

How do we measure search?

Usage Data Editorial Labeling

Collect, Analyze & Report on interactions users are having with your search functionality.

Get a set of users to mark top x results with “Relevant vs. Not” for a pre-determined sample set of searches. !Then compute specific metrics based on those.

7

Usage Data: Key Metrics to look at

• No Result Search % • Search Exits % • CTR %

!• Average Click Position • MRR (Mean Reciprocal Rank) • Clicks per Search • Paging (how deep do people have

to dig?) !

• Latency (Average, tp90 and tp95)

Aggregate & Trends

Trends

Aggregate & Trends

8

Search Analytics: Examples

9

Editorial Labeling

• Precision Recall !• DCG (Discounted

Cumulative Gain) !• nDCG (Normalized DCG)

http://en.wikipedia.org/wiki/Discounted_cumulative_gain

Relevant

Not Relevant

Relevant

Not Relevant

Not Relevant

Relevant

Not Relevantweight-age decreases as as you go down on an ordered list

10

Improving Search Results

• Popular No Result Searches - Can we use keyword stuffing? • Popular Search Exits - Eyeball outliers • Popular Searches with low CTRs • Generally improve Average Click Position / MRR via

identifying patterns

Attack Low Hanging fruit first

• Topic for some other time :)

Advanced: Learning Models

Utilizing Search Behavior to improve ranking

• Utilize Popularity (click stream) to inform search ranking (impacts CTR)

• Utilizing past search history to offer assistive features (search suggestions, related searches)

11

Using Popularity to affect search ranking

DEMO

[email protected]

www.measuredsearch.com