Laurel Poertner Head of Training Intermediate Analytics
Laurel Poertner
Head of Training
Intermediate Analytics
Introductions
Laurel Poertner
Over 15 years experience Managing technical support and education teams
Delivered technical and business process training
Supported, used and managed Knowledge Management tool, Knova
Built virtual classroom offerings for an Enterprise Resource Planning software package
Implemented several CRM and KM systems internally and as a Professional Services consultant
Certified Trainer of Knowledge Centered Service v6SM Practices
1. Review why Analytics
2. Basic Analytics - Key concepts
3. Set up your Lab Environment
4. Understand - Dashboards
5. Understand – Solution Adoption
6. Act – Content Gap and Query Performance
7. Act – Common Boosting Scenarios
8. Understand – Workflow and Self-Service
9. Automate - Machine Learning
Plan
Goal: Gain practical experience using all the Usage Analytics Tools to help
▶ Understand your users to gain additional insight into search behaviors and
patterns
▶ Analyze Analytics data using reports and the Visit Browser to identify areas
to optimize the search experience
▶ Understand the importance of Relevance and how to improve it using
Query Pipeline rules
▶ Knowing the most appropriate action to take to improve the search
experience based on performance and utilization metrics
▶ Knowing when and how to set up Machine Learning and manual search
tuning to increase relevancy
About This Course
A Review
5 © 2017 Coveo Solutions Inc - Proprietary and Confidential
Why Analytics
Change is constant
And so is the need for Relevance
Providing Relevance is a Journey in which you continuously
Understand the Behavior of your users, gain insight
Act on this insight, to provide to improve Relevance
Leverage Machine Learning to Automate the Relevance Improvement
Relevance is a Journey, not a destination
The Triumvirate of Analytics
Understand Act
Automate
What your user are searching for
What they are clicking on
Are they finding what they are looking for
What is relevant to them
Optimize the search experience
based on what your user are
doing
Tune the experience to provide
ever-improving relevance
Leverage Machine learning to automate the
Relevance tuning process as you go
Free time to do further research
1 2
3
Co
pyr
igh
t ©
20
16 C
ove
o S
olu
tion
s In
c.
All
rig
hts
re
serv
ed
. Pro
pri
eta
ry a
nd
CO
NF
IDE
NT
IAL
The Tools of Analytics
User Interface
Query Pipelines
Machine LearningUsage Analytics
Connectors
Salesforce Sitecore Web Exchange
Index
Source
Items
Source
Items
Source
Items
Source
Items
Understand 1
Act 2
Automate3
Coveo Relevance Maturity Model
manual/hard automated/easy
content intelligence
Positive
business
impact &
ROI
Negative
economic
ImpactBasic search captures user’s intent, and saves time retrieving information.
Entry-level search is resource intensive, relies heavily on taxonomies for relevance.
Federated search fails at ranking combined result sets relevantly.
Poor relevance affects conversions, clickthroughs, satisfaction, deflection, escalations, productivity, innovation, work redundancy,
...
Net net, entry level search [cheaper] is the most expensive strategy.
Siloed
search
Federated
searches
0Efficiency gains Proficiency gains
LEADERSLAGGARDS
6 Discovers user’s likely intent,
by analyzing behavioral data.
Machine learning auto-tunes recommendations and
ranking to maximize business outcome, enabling true
one-to-one user engagement and upskilling.
Self learning predictive
recommendations
PREDICTIVE[relevance is predictable]
5
…related content, experts,
products or services pushed to
expand user’s knowledge and
abilities to do more, buy more,
learn more, engage more, etc.
Contextual
suggestions
3
4
…adapt relevance through
weighting of ranking factors,
query ranking expressions,
based on known content
characteristics.
…in-product, in-task, and other
contextual signals factor the
uniqueness of the user and
query context and ranks results
with higher relevance.
Tunable
relevance
Contextual
relevance
PROACTIVE[relevance is contextual]
1
2
Unifies and ranks
information from
multiple sources.
Configurable rich facets, search
tabs, folding and security
trimming provide the very first
step of personalization for users.
Content
navigationSecured
unified
ranking
RESPONSIVE[relevance is personal]
CRMM™
stages
Reactive
Low search success
Visits w/ no results
Searches w/ no results
Low click-through/rank
Responsive
Visits w/ searches
Searches w/ clicks
Click-through increases
Avg click rank decreases
Proactive
>50% customers visit
Self-service portal
Cases submitted
decreases
Predictive
>85% customers visit
first
Click-through >50%
Avg Click rank <3
10
Self-Service maturity model
Search Clicks Search Success Relevance
11
Top Queries
Top Keywords
Visit Click-Through %
Top searches w/clicks
Top documents
clicked
Search Click-Through
%
Visits w/Searches and
a click
Visits w/results
Click-through %
Avg Click rank
Queries w/ high
relevance
Top documents by
average click rank
Identify what is successful for your users (ie. Did they find what they were looking for?)
Understand
Search Clicks Search Success Relevance
12
Top Queries w/o
results
# Queries w/o results
Queries w/o clicks
Lowest clicked
documents
Top keyword for case
creation subject
Top documents
clicked with case
creation
Queries w/ low
relevance scores
Lowest average click
rank queries
Identify what is NOT successful for your users
Understand
Key Concepts
13 © 2017 Coveo Solutions Inc - Proprietary and Confidential
Basic Analytics
1. Review why Analytics
2. Basic Analytics - Key concepts
3. Set up your Lab Environment
4. Understand - Dashboards
5. Understand – Solution Adoption
6. Act – Content Gap and Query Performance
7. Act – Common Boosting Scenarios
8. Understand – Workflow and Self-Service
9. Automate - Machine Learning
Plan
Review the important key concepts of basic analytics to delve further into more technical
areas.
Learning Objectives
1. Understand the big picture of where Usage Analytics data fits in with the Coveo
Organization
2. Review the concepts of Visits, Events, Dimensions and Filters
3. Understand the different dashboard templates and metrics
In This Module
Where are We
User Interface
Query Pipelines
Machine LearningUsage Analytics
Connectors
Salesforce Sitecore Web Exchange
Index
Source
Items
Source
Items
Source
Items
Source
Items
6Analyzing
behavioral data
Big Pictures
[Facet Component]
Usage Analytics
[Recommendations Component]
[Omnibox Component]query suggestionssuggestion engine
Index
[Search Result List]
Query Pipeline
Coveo Search Interface
Coveo Cloud Platform
queyr suggestions
Search Result(Coveo items)
Query1
2
Usage AnalyticsDatabase
Event
Coveo Analytics Admin
Explorer Dashboard
Dimensions
Metrics
Dimensions
Metrics
Send a Query
Type a Query
Click on a Query Suggest
Click on a Facet
Click on a Recommendation
Click on a Search Result
Key Concepts: Events, Dimensions and Metrics
1 An Event is composed of Dimensions, each populated with their unique Value
2 3Something happens on the page and triggersan Analytic Event
Coveo Platform compiles Metricsabout each Dimensions and their Values (Numerical Value)
Dimensions Value
User Name Laurel Poertner
User Query Adventure
Event Type Search
Date 5:13:53PM 03/21/2017
Event Cause searchboxSubmit
Origin 1 (Page/Hub) Full Learning Site
Origin 2 (Tab/Interface) All
Comprised of Dashboards to quickly monitor the search usage status and
Explorers which are subsets of dashboards to drill down into the usage analytics data
Permission Filters
Filter data in:
▶ Reports
▶ Exports
▶ Visit Browser
Exports
Access from:
▶ Visit Browser
▶ Explore Data/ Explorer
▶ Export menu
Report Access
Define who can access:
▶ Only Me
▶ Analytics Privileges
▶ Custom
Reporting
App Store Site
Lab Environment
1. Review why Analytics
2. Basic Analytics - Key concepts
3. Set up your Lab Environment
4. Understand - Dashboards
5. Understand – Solution Adoption
6. Act – Content Gap and Query Performance
7. Act – Common Boosting Scenarios
8. Understand – Workflow and Self-Service
9. Automate - Machine Learning
Plan
Understand the Coveo Learning site and environment to use throughout the training.
Learning Objectives
1. Become familiar with the App Store search page
2. Review Query Pipelines
3. Understand how the App Store context will be used as our learning environment
4. Use the App Store site to create your own test environment and pipeline
In This Module
Site Dimensions
23
https://platform.cloud.coveo.com/pages/coveolearningintermediateanalytics/Demo#t=All&sort=relevancy
Where are We
User Interface
Query Pipelines
Machine LearningUsage Analytics
Connectors
Salesforce Sitecore Web Exchange
Index
Source
Items
Source
Items
Source
Items
Source
Items
3 4Tunable
relevance
Contextual
relevance
Vocabulary Refresher
Integration
▶ System that hosts a Coveo Search Experience
Hub
▶ Name given to a hosted Search Page or Search Panel
Query Pipeline
▶ Conduit by which an interface transmits a Query to the Index for processing
▶ Alternate set of rules that can be defined to modify queries
▶ Take advantage of query pipelines when you have more than one search interfaces / audiences
▶ Distinct users
▶ Purpose
▶ Different rulesAgents
Customers
Partners
Some Vocabulary
Hub – Full Learning Site Hub – My pipeline
default
Pipeline
What we Want
Customer
Pipeline
Trainee
Pipeline
Hub – Full Learning Site Hub – My pipeline
Exercise – Create your Own Pipeline
28
▶ Add a new Query pipeline (call it your name or something unique and easy)
▶ Open the search page and add your pipeline in the link
▶ Before the # you add this: ?pipeline=MyOwnPipelineIjustCreated
▶ Example:
https://platform.cloud.coveo.com/pages/coveolearningintermediateanalytics/Demo?pip
eline=Laurel#t=All&sort=relevancy
▶ Test and Validate your pipeline
▶ Search an expression
▶ Press Alt + Double Click beside title of first result
FAQ
Q: Will I affect my production environment if I add a pipeline?
No. Traffic will first look at condition on the pipeline. If no conditions match my traffic, it will
always use the default pipeline.
Q: What happen if I have two pipelines with the same condition?
Traffic will go through the first pipeline from the top of the list that match my traffic. The other one
won’t be use. You can use AB testing to split traffic with different rules.
Dashboards
Understand
1. Review why Analytics
2. Basic Analytics - Key concepts
3. Set up your Lab Environment
4. Understand - Dashboards
5. Understand – Solution Adoption
6. Act – Content Gap and Query Performance
7. Act – Common Boosting Scenarios
8. Understand – Workflow and Self-Service
9. Automate - Machine Learning
Plan
Get practical experience in creating analytics dashboards and explorers to analyze the data.
Learning Objectives
1. Create dashboards in the following ways
a) From template
b) From blank dashboard and explorer
c) From configuration file
2. Share dashboard with Students group
In This Module
Self-Service
▶ Visits
▶ Case Creation
▶ Documents
▶ Content Gap
Customer Success
Monthly Overview
▶ Hub summary
▶ Performance
▶ QPM
▶ Custom events
▶ Agents
Community Search
▶ Utilization
▶ Performance
▶ Documents
▶ Content gap
▶ Visitors
▶ Facets
▶ Search cause
Agent
▶ Participation
▶ Attach to case
▶ Performance
▶ Content gap
Recommended Dashboards
Create a dashboard from template
Duplicate a report
Create a blank dashboard
34
Report creation
Exercise – Create/Edit Reports
35
Add a new Dashboard from template
▶ Select Summary template
▶ Add name filter: App Store Site
Add a new blank Dashboard
▶ Edit Configuration
▶ Rename Dashboard with your initial or name
▶ Add name filter: App Store Site
▶ Set the permission to custom and add the instructor member
▶ Save your dashboard
Duplicate the Activity Explorer
▶ Add name filter: App Store Site
FAQ
Q: Where can I find Custom template?
You can find custom template in our knowledge base. Search for : Dashboard template
Q: Can I share my dashboard with others?
You can copy the URL of your dashboard including a specific horizon or filter that is not necessarily
saved and send it to someone else. It will open with the same settings. (The person needs to have
access to the Cloud analytics with permissions)
Solution Adoption
37
Understand
1. Review why Analytics
2. Basic Analytics - Key concepts
3. Set up your Lab Environment
4. Understand - Dashboards
5. Understand – Solution Adoption
6. Act – Content Gap and Query Performance
7. Act – Common Boosting Scenarios
8. Understand – Workflow and Self-Service
9. Automate - Machine Learning
Plan
Analyze the data in the dashboards and explorers you created to determine what actions
to make.
Learning Objectives
1. Find out if visitors are utilizing the solution as expected
a) Tab performance
b) Facet performance
c) Custom event performance
2. Add metrics to a dashboard to analyze tab and facet performance
In This Module
Event Value
• submitButton• cancelButton• unloadPage
Event Type
• caseCreation• caseAttach/caseDetach• expandToFullUI• preferencesChange
Event Cause (Salesforce)
• searchBoxSubmit• documentOpen• documentQuickView• inputChange• interfaceChange• facetSelect
40 © 2017 Coveo Solutions Inc - Proprietary and Confidential
Custom Event Dimensions
Performance indicators
Solution Adoption
Total users using the solution
components (facets, tab, quick
view, custom objects etc..)
Facet Performance
Which facets is popular or not
used. Which facets gave
content gap or no click
Drive changes to the UI
Tabs
If your tabs performance is
low, review their location,
size, color and name
Facets
If your facets performance
is low, review their
location, color and name
▶ BEFORE: Few tabs where only
visible and available in a drop list.
Traffic was very low and content
from those tabs hardly used
▶ AFTER: Changing the color and the
display of the tabs brought an
important increase of traffic
▶ IMPACT: More traffic on hidden
content brought more training
registration and more deflection
with the know issues section
Tab PerformanceBEFORE
AFTER
IMPACT
Exercise – Create/Edit Reports
44
▶ Tabs performance in Self-Service TAB
▶ Open Self-Service tab in your dashboard
▶ In solution adoption, create a metrics box to show total unique users using tabs (search cause =
interface change)
▶ Tabs performance in a new TAB
▶ Add a new TAB called: Tabs
▶ Add time series card with total query, click-through and average click-rank
▶ Filter by tab = All
▶ Add a title: All
▶ Resize the card half page
▶ Duplicate the new section and do same exercise for the other tabs
*** Each tab is affecting the overall performance metrics. Looking at each of them gives you an idea of
which one is the most or less performant
FAQ
Q: Where can I remove or change my facets
Facet are managed by your site editor. It can also be done in the interface editor depending of
your deployment. Training is available as well.
Q: Where can I remove or change my tabs
Facet are managed by your site editor. It can also be done in the interface editor depending of
your deployment. Training is available as well.
Content Gap and Query Performance
46
Act
1. Review why Analytics
2. Basic Analytics - Key concepts
3. Set up your Lab Environment
4. Understand - Dashboards
5. Understand – Solution Adoption
6. Act – Content Gap and Query Performance
7. Act – Common Boosting Scenarios
8. Understand – Workflow and Self-Service
9. Automate - Machine Learning
Plan
Review Content Gap and Query Performance reports to determine what query pipeline
rules to add to improve relevance.
Learning Objectives
1. Review query pipeline rules
2. Analyze Content Gap and Query Performance reports
3. Evaluate which actions should be taken to solve the query performance issue
a) Enable ML
b) Add Synonyms
c) Add content
d) Boost results
e) Add Top Features
In This Module
The Tools of Analytics
User Interface
Query Pipelines
Machine LearningUsage Analytics
Connectors
Salesforce Sitecore Web Exchange
Index
Source
Items
Source
Items
Source
Items
Source
Items
Act 2
Rule
Tool that allows you to modify a Query sent from a given Hub or the Ranking of the items that would be returned to the Hub.
Query Pipeline Rules Review
Modifies the Query- Thesaurus- Stop Words
Modifies the Ranking of Results- Machine Learning (ML)- Featured Results- Ranking Expression(QRE)
Modifies the Search Page- Triggers
Co
pyr
igh
t ©
20
16 C
ove
o S
olu
tion
s In
c.
All
rig
hts
re
serv
ed
. Pro
pri
eta
ry a
nd
CO
NF
IDE
NT
IAL
Query Pipeline Rules
Thesaurus Featured Results
Ranking Expressions
Manage equivalent words used to transparently add expressions to the query before it is sent to the index
Promote specific results to appear at the top of the results list when a query matches or contains a specific expression
Influence the document score by adding query ranking expressions (QRE) to boost or lower the score within search results matching an expression
Triggers
Notify
Redirect
Execute
Query
51
Exercise – Query analysis
52
Exercise:
▶ Open Content Gap Tab explorer to analyze content gap
▶ Add synonym / content
▶ Open Query Performance tab in Portal Search dashboard to analyze poor
relevancy query
▶ Use Visit browser for analysis
▶ Add synonyms
FAQ
Q: I see content gap with long sentence
Use the partial match which will allow you to match fewer words instead of the whole sentence.
Partial match can be setup in your administration tool
Q: I see content gap with special characters (* & ? % \ / ! @ & # @ {} [] )
Use the disable query syntax option which will be soon available from pipeline. It can be setup in
the administration tool
Q: I see a lot of content gap using the same pattern typo
You can use regex to resolve pattern issue
i.e: Query = “KB 123456” expected query = “KB123456” (no space)
Common boosting Scenarios
54
Act
1. Review why Analytics
2. Basic Analytics - Key concepts
3. Set up your Lab Environment
4. Understand - Dashboards
5. Understand – Solution Adoption
6. Act – Content Gap and Query Performance
7. Act – Common Boosting Scenarios
8. Understand – Workflow and Self-Service
9. Automate - Machine Learning
Plan
Learn scenarios to add boosting rules (Query Ranking Expressions) in your environment.
Learning Objectives
1. Create a Query Ranking Expressions (QRE)
2. Add a Featured Result rule
3. Understand when and how to use conditions with your pipeline rules
4. Follow best practices to add and test new pipeline rules in a production
environment
In This Module
A condition is a rule that sets predefined requirements on something before it can be executed
Can be used on • Entire pipeline• Specific pipeline rule (thesaurus, ML
etc.)
57 © 2017 Coveo Solutions Inc - Proprietary and Confidential
Conditions
Exercise - Boosting Rules and Tops Features
Scenario
This month, it is the FIFA Word Cup and people from all over the word will watch the games. You
would like to boost a sport channel and some soccer content during that month.
▶ Add a feature: “Fox Soccer 2Go” when query contains soccer in Sport Tabs
▶ Boost the game categories in Game Tabs
58
Exercise - Boosting Rules (QRE)
Add Conditions:
▶ Steps to do in condition menu:
▶ Add condition 1: Tab is Sports OR Tab is “Myname” (cannot have duplicate condition)
▶ Add condition 2: Tab is Games OR Tab is “Myname” (cannot have duplicate condition)
Add Feature Request:
▶ Steps to add Feature requests
▶ Add Feature request using select document option and search for Fox Soccer
▶ Select the app and Save (do not add a condition yet)
▶ Select add a condition in More menu and choose the Sports Tab condition
Add Ranking Expression:
▶ Steps to do in ranking expression rule
▶ Add a QRE using the meta data @sysconcept = fifa or @sysconcept = soccer
▶ Boost to 300 points (little higher than ML boost)
▶ Select add condition in More menu and choose the Games Tab condition
59
Workflow and Self-Service
60
Understand
1. Review why Analytics
2. Basic Analytics - Key concepts
3. Set up your Lab Environment
4. Understand - Dashboards
5. Understand – Solution Adoption
6. Act – Content Gap and Query Performance
7. Act – Common Boosting Scenarios
8. Understand – Workflow and Self-Service
9. Automate - Machine Learning
Plan
Learn the difference between Self-Service Success and Case Deflection measurements.
Learning Objectives
1. Understand the metrics of Self-Service and Case Deflection
2. Understand the difference between Assumed case deflection and confirmed case
deflection
3. Gain more experience with the visit car metrics and why we use it in case deflection
In This Module
• Query Click-Through Rate (CTR) –Best in class 50%
• Average Click Rank (ACR) – Best in class <= 3
• Visit Click-Through Rate – Best in class 50%
[Search] Click-Through (CTR): percentage of searches with at least one opened document from the Coveo search results
Average Click Rank (ACR): measures the average position of opened documents in the Coveo search results list.
Visit Click-Through: percentage of visits with at least one search expression and one opened document from the Coveo search results
Benchmarks
Metrics
• Unique Visitor ID
• Unique Visit ID
Unique VisitID: a random and unique value generated every time a user visits the search site. A visit ends after 30 min of inactivity.All user interactions of the same visit are recorded using the same Visit ID.
Unique Visitor ID: similar to a Visit ID but never expires. Allows you to know the number of distinct users who submitted a query or clicked a document, counting them only once even if they performed the same events in multiple visits.
Visit Count: this is the old metric that can be seen in reports that were created before the above dimensions were created. Remove this and add one of the Unique dimensions to get a more accurate count.
Used to track how many distinct users did a query or click
Visit Dimensions
Self-Service Success Case Deflection
How do we measure success?
The rate that self-service resources eliminate a customer’s need for live assistance
The rate at which customers find the information they need on your self-service portal, which may or may not have required live assistance.
Self-Service
• Visit Click-Through
Case Deflection
• Confirmed Case Deflection• Assumed Case Deflection
Traffic Workflow
• Portal traffic• Case Deflection traffic
66 © 2017 Coveo Solutions Inc - Proprietary and Confidential
Case Creation
Highlights
• Visit Click-through is 70.8% which is
well over the best in class benchmark of
50%
67 © 2017 Coveo Solutions Inc - Proprietary and Confidential
Self-Service MetricsMonth of March
Highlights
• Traffic Workflow 39% of customers with
the intention of submitting a case will
search or navigate prior to submit the
case.
68 © 2017 Coveo Solutions Inc - Proprietary and Confidential
Site Workflow MetricsMonth of March
Highlights
• Case Deflection 29 cases were deflected
combining assumed and confirmed case
deflection which is 12% deflection.
69 © 2017 Coveo Solutions Inc - Proprietary and Confidential
Case Deflection MetricsMonth of March
Exercise – Visit Card
Open Self-Service Tab in your dashboard:
▶ Self-Service Section
▶ Edit dashboard
▶ Add Visit Card in Self-Service Section
▶ Show Visit where customer select a facet (event cause = facet select)
▶ Title of the card: Total Visits with Facet Selection
▶ Unsuccessful Visits
▶ Click the crayon on the Visit without results and analyze how the condition is built
▶ Click the crayon on the Visit Without Clicks and analyze how the condition is built
▶ Save your dashboard and click on the title: Total visit without Results. You can see which visits that meet this
condition.
70
Machine Learning
71
Automate
1. Review why Analytics
2. Basic Analytics - Key concepts
3. Set up your Lab Environment
4. Understand - Dashboards
5. Understand – Solution Adoption
6. Act – Content Gap and Query Performance
7. Act – Common Boosting Scenarios
8. Understand – Workflow and Self-Service
9. Automate - Machine Learning
Plan
Learn the how machine learning can help automate the manual actions we just learned
about.
Learning Objectives
1. Understand how machine learning works and the different models
2. Understand the different business cases to use machine learning
3. Know the requirements to set up and activate the different machine learning
models
4. See the difference in search results with machine learning activated and without
In This Module
The Tools of Analytics
User Interface
Query Pipelines
Machine LearningUsage Analytics
Connectors
Salesforce Sitecore Web Exchange
Index
Source
Items
Source
Items
Source
Items
Source
Items
Automate3
How do I fix my printer?
Thanks, that last one worked!
How do we fix our printers?
Thanks, the first one worked!
IntelligenceEngine
Use machine learning to learn from your visitors, continually improvesearch results & predict relevant content
Make self-service smart
Coveo ML Information Flow
Usage Analytics
UA
Database
ML Service
Previous Users
UA Events
Type: Page View, Query, Clicks, Custom Events
Build a model
Query Pipeline
Index
Current User
User Request
User context (e.g. user role)
User actions (e.g. page view, query) Request
Query with
ML boosts
Machine Learning Models
Tune Relevance ModelAutomatically optimize the Ranking of Coveo Items returned by a Query
1
2
Query: IPA
3
Sleeman
4
Alexander Keith’s
Goose Island
Inukshuk Island
1
2
3
Goose Island
4
Alexander Keith’s
Sleeman
Inukshuk Island
Query Suggest ModelProvides users with Query Suggestions as they are typing in the Coveo Search Box
RecommendationRecommends Coveo Items that the user might be interested on viewing next.
ipa
Goose Island IPAMolson Canadian IPA
Sleeman IPAMuskoka IPA
Recommended Topic
Stone Delicious IPA
Mill st west coast ipa
Hops & Robbers ipa
Model TypeThe type of Machine Learning relevance optimization
Training SetThe Usage Analytic data that a Model uses to self-optimize.
Update FrequencyThe frequency at which a Training set is updated with new data.
Data PeriodThe period of Usage Analytic data to be included in the Training Set with each update
Condition [optional]
The Condition(s) to trigger the Machine Learning rule
Some Vocabulary
ML Requirements
1 - Machine Learning Models are applied on a Query Pipeline
2 - Models learns from Users past Behaviors to optimize the search experience ( using data
such as Context, Query, Success rate, Click, etc.)
3 – You need approximately 10 000 Query Events in order to compile a Relevant model
4 – Depending the number of Events, compiling or updating a Model takes between 30 and
60 minutes.
Requirements
[Facet Component]
[Recommendations Component]
[Omnibox Component]query suggestionssuggestion engine
[Search Result List]
Coveo Search Interface
queyr suggestions
Tune Relevance Model
Query SuggestModel
Recommendations Model
To Remember
Model Requirements What it Does
Tune Relevance None Automatically optimize the Ranking of CoveoItems returned by a Query
Suggest Query Search Interface has an Omnibox component
Provides users with Query Suggestions as they are typing in the Coveo Search Box
Recommendation Search Interface has a Recommendations components
Recommends Coveo Items that the user might be interested on viewing next
Steps:
1. Search using an acronym or term
2. Click an appropriate result
3. Enter the words behind the acronym or a synonym of the term entered above and click the same result
4. Repeat
5. Once model is trained, in the Content Browser, ensure the model returns the expected results
Help Train Machine Learning with Thesaurus Rules
Accelerate the Machine Learning training process by pointing to synonyms as well as words behind acronyms
Highlights
• 42.8% of the clicks have been suggested
by Machine Learning
• Top Queries are about analytics and
machine learning
83 © 2017 Coveo Solutions Inc - Proprietary and Confidential
Search Activities driven by MLTop Queries with Clicks
Highlights
Based on a real scenario,
this is the difference of
results without ML and
With ML.
Without ML we show
topics around list, listed
and with ML we show
topics around thesaurus
84 © 2017 Coveo Solutions Inc - Proprietary and Confidential
ML Impact
DEMO – Adding a Tune Relevance Model to a Pipeline
Exercise – A/B Testing
Scenario
After reviewing the new Machine Learning model, we need to understand if it is having a positive
impact on query performance.
Exercise:
Use the A/B Test “FLS + ML” to create an A/B Test report dashboard using the A/B Test template
Compare Pipeline A = Full Learning Site to Pipeline B = MachineLearning
86
• Machine Learning Models are compiled from Usage Analytics Events and requires
around 10 000 Query Events to be Optimized
• Machine Learning Models are added to Query Pipelines
• There are 3 Machine Learning Models: Tune Relevance, Query Suggest and
Recommendations
• Query Suggest Model requires your Search Interface to have an Omnibox Component
• Recommendations Model requires your Search Interface to have a Recommendation
Component
In Summary
In Summary
Why Analytics
The need for relevance is a constant journey in which you
Understand
Analyze the data to gain insight into the behavior of your users
Act
Know the most appropriate action to take from this insight to provide quality
improvements to relevance and improve search success
Automate
Leverage Machine Learning to automate the relevance improvement