Usability of Grouping of Retrieval Results

Usability of Grouping of Retrieval Results

Marti Hearst School of Information, UC Berkeley

September 1, 2006

Marti Hearst, Google Visit, 9/1/06

The Need to Group

Interviews with lay users often reveal a desire for better organization of retrieval results

Useful for suggesting where to look next People prefer links over generating search

terms* But only when the links are for what they

want

*Ojakaar and Spool, Users Continue After Category Links, UIETips Newsletter, http://world.std.com/~uieweb/Articles/, 2001






Conundrum

Everyone complains about disorganized search results.

There are lots of ideas about how to organize them.

Why don’t the major search engines do so?

What works; what doesn’t?


Different Types of Grouping

Clusters (Document similarity based)

(polythetic)

Scatter/GatherGrouper

Keyword Sharing (any doc with keyword in group)

(monothetic)

FindexDisCover

Single Category

SwishDynacat

Multiple (Faceted) Categories

FlamencoPhlat/Stuff I’ve seen

Monothetic vs Polythetic After Kummamuru et al, 2004


Clusters

Fully automated Potential benefits:

Find the main themes in a set of documents Potentially useful if the user wants a summary of the main

themes in the subcollection Potentially harmful if the user is interested in less dominant

themes More flexible than pre-defined categories

There may be important themes that have not been anticipated

Disambiguate ambiguous terms ACL

Clustering retrieved documents tends to group those relevant to a complex query together

Hearst, Pedersen, Revisiting the Cluster Hypothesis, SIGIR’96


Categories

Human-created But often automatically assigned to items

Arranged in hierarchy, network, or facets Can assign multiple categories to items Or place items within categories

Usually restricted to a fixed set So help reduce the space of concepts

Intended to be readily understandable To those who know the underlying domain Provide a novice with a conceptual structure

There are many already made up!

Cluster-based Grouping

Document Self-similarity(Polythetic)


Scatter/Gather Clustering

Developed at PARC in the late 80’s/early 90’s Top-down approach

Start with k seeds (documents) to represent k clusters Each document assigned to the cluster with the most

similar seeds To choose the seeds:

Cluster in a bottom-up manner Hierarchical agglomerative clustering

Can recluster a cluster to produce a hierarchy of clusters

Pedersen, Cutting, Karger, Tukey, Scatter/Gather: A Cluster-based Approach to Browsing Large Document Collections, SIGIR 1992


The Scatter/Gather Interface


Two Queries: Two Clusterings

AUTO, CAR, ELECTRIC AUTO, CAR, SAFETY

The main differences are the clusters that are central to the query

8 control drive accident …

25 battery california technology …

48 import j. rate honda toyota …

16 export international unit japan

3 service employee automatic …

6 control inventory integrate …

10 investigation washington …

12 study fuel death bag air …

61 sale domestic truck import …

11 japan export defect unite …


Scatter/Gather Evaluations

Can be slower to find answers than linear search!

Difficult to understand the clusters. There is no consistence in results. However, the clusters do group relevant

documents together. Participants noted that useful for eliminating

irrelevant groups.


Visualizing Clustering Results

Use clustering to map the entire huge multidimensional document space into a huge number of small clusters.

User dimension reduction and then project these onto a 2D/3D graphical representation


Clustering Visualizations

image from Wise et al 95


Clustering Visualizations

(image from Wise et al 95)


Are visual clusters useful?

Four Clustering Visualization Usability Studies


Clustering for Search Study 1

This study compared a system with 2D graphical clusters a system with 3D graphical clusters a system that shows textual clusters

Novice users Only textual clusters were helpful (and

they were difficult to use well)

Kleiboemer, Lazear, and Pedersen. Tailoring a retrieval system for naive users. SDAIR’96


Clustering Study 2: Kohonen Feature Maps, Chen et al.

Comparison: Kohonen Map and Yahoo Task:

“Window shop” for interesting home page Repeat with other interface

Results: Starting with map could repeat in Yahoo (8/11) Starting with Yahoo unable to repeat in map

(2/14)

Chen, Houston, Sewell, Schatz, Internet Browsing and Searching: User Evaluations of Category Map and Concept Space Techniques. JASIS 49(7): 582-603 (1998)


Koh

on

en F

eatu

re M

aps

(Lin

92

, C

hen e

t al. 9

7)


Study 2 (cont.), Chen et al.

Participants liked: Correspondence of region size to # documents Overview (but also wanted zoom) Ease of jumping from one topic to another Multiple routes to topics Use of category and subcategory labels



Study 2 (cont.), Chen et al.

Participants wanted: hierarchical organization other ordering of concepts (alphabetical) integration of browsing and search correspondence of color to meaning more meaningful labels labels at same level of abstraction fit more labels in the given space combined keyword and category search multiple category assignment (sports+entertain)

(These can all be addressed with faceted categories)



Clustering Study 3: Sebrechts et al.

Each rectangle is a cluster. Larger clusters closer to the “pole”. Similar clusters near one another. Opening a cluster causes a projection that shows the titles.


Study 3, Sebrechts et al.

This study compared:

3D graphical clusters 2D graphical clusters textual clusters

15 participants, between-subject design Tasks

Locate a particular document Locate and mark a particular document Locate a previously marked document Locate all clusters that discuss some topic List more frequently represented topics

Visualization of search results: a comparative evaluation of text, 2D, and 3D interfaces Sebrechts, Cugini, Laskowski, Vasilakis and Miller, SIGIR ‘99.


Study 3, Sebrechts et al.

Results (time to locate targets) Text clusters fastest 2D next 3D last With practice (6 sessions) 2D neared text results; 3D still slower Computer experts were just as fast with 3D

Certain tasks equally fast with 2D & text Find particular cluster Find an already-marked document

But anything involving text (e.g., find title) much faster with text. Spatial location rotated, so users lost context

Helpful viz features Color coding (helped text too) Relative vertical locations


Clustering Study 4 Compared several

factors

Findings: Topic effects dominate

(this is a common finding)

Strong difference in results based on spatial ability

No difference between librarians and other people

No evidence of usefulness for the cluster visualization

Aspect windows, 3-D visualizations, and indirect comparisons of information retrieval systems, Swan, &Allan, SIGIR 1998.


Summary:Visualizing for Search Using Clusters

Huge 2D maps may be inappropriate focus for information retrieval cannot see what the documents are about space is difficult to browse for IR purposes (tough to visualize abstract concepts)

Perhaps more suited for pattern discovery and gist-like overviews.


Clustering Algorithm Problems

Doesn’t work well if data is too homogenous or too heterogeneous

Often is difficult to interpret quickly Automatically generated labels are unintuitive

and occur at different levels of description

Often the top-level can be ok, but the subsequent levels are very poor

Need a better way to handle items that fall into more than one cluster

Term-based Grouping

Single Term from Document Characterizes the Group

(Monothetic)


Findex, Kaki & Aula

Two innovations: Used very simple method to create the

groupings, so that it is not opaque to users Based on frequent keywords Doc is in category if it contains the keyword Allows docs to appear in multiple categories

Did a naturalistic, longitudinal study of use Analyzed the results in interesting ways

Kaki and Aula: “Findex: Search Result Categories Help Users when Document Ranking Fails”, CHI ‘05



Study Design

16 academics 8F, 8M No CS Frequent searchers

2 months of use Special Log

3099 queries issued 3232 results accessed

Two questionnaires (at start and end) Google as search engine; rank order retained


After 1 Week After 2 Months


Kaki & Aula Key Findings (all significant)

Category use takes almost 2 times longer than linear First doc selected in 24.4 sec vs 13.7 sec

No difference in average number of docs opened per search (1.05 vs. 1.04)

However, when categories used, users select >1 doc in 28.6% of the queries (vs 13.6%)

Num of searches without 0 result selections is lower when the categories are used

Median position of selected doc when: Using categories: 22 (sd=38) Just ranking: 2 (sd=8.6)


Kaki & Aula Key Findings

Category Selections 1915 categories selections in 817 searches Used in 26.4% of the searches During the last 4 weeks of use, the proportion of searches

using categories stayed above the average (27-39%) When categories used, selected 2.3 cats on average Labels of selected cats used 1.9 words on average (average

in general was 1.4 words) Out of 15 cats (default):

First quartile at 2nd cat Median at 5th

Third quartile at 9th


Kaki & Aula Survey Results

Subjective opinions improved over time Realization that categories useful only some of the

time Freeform responses indicate that categories useful

when queries vague, broad or ambiguous Second survey indicated that people felt that their

search habits began to change Consider query formulation less than before (27%) Use less precise search terms (45%) Use less time to evaluate results (36%) Use categories for evaluating results (82%)


Conclusions from Kaki Study

Simplicity of category assignment made groupings understandable (my view, not stated by them)

Keyword-based Categories: Are beneficial when result ranking fails Find results lower in the ranking Reduce empty results May make it easier to access multiple results Availability changed user querying behavior


Highlight, Wu et al.

Select terms from document summaries, organize into a subsumption hierarchy.

Highlight the terms in the retrieved documents.

Wu, Shankar, Chen, Finding More Useful Information Faster from Web Search ResultsCICM ‘03





Highlight, Wu et al.

First study: 19 undergraduates Used the system for their own queries Significant preference for the grouping interface

Second study: 6 participants Their own queries Accesses were sequential in linear interface Accesses went deeper in grouping interface Participants saved more documents per query

Category-based Grouping

General CategoriesDomain-Specific Categories



SWISH, Chen & Dumais

18 participants, 30 tasks, within subjects Significant (and large, 50%) timing

differences in favor of categories For queries where the results are in the first

page, the differences are much smaller. Strong subjective preferences. BUT: the baseline was quite poor and the

queries were very cooked. Very small category set (13 categories) Subhierarchy wasn’t used.

Chen, Dumais, Bringing Order to the Web: Automatically Categorizing Search Results CHI 2000


Test queries, Chen & Dumais

Information Need Pre-specified Querygiants ridge ski resort “giants”

book about "numerical recipes" for computer software

“recipes”

information about Indian motorcycles

“Indian”

"the home page for the band, "They Might be Giants""

“giants”

"the home page for the basketball team, the Washington Wizards"

“washington”

Chen, Dumais, Bringing Order to the Web, Automatically Categorizing Search Results. CHI 2000

Marti Hearst, Google Visit, 9/1/06Dumais, Cutrell, Chen, Bringing Order to the Web, Optimizing Search by Showing Results in Context, CHI 2001


Revisiting the Study, Dumais, Cutrell, Chen






This followup study reveals that the baseline had been unfairly weakened.

The speedup isn’t so much from the category labels as the grouping of similar documents.

For queries where the answer is in the first page, the category effects are not very strong.



DynaCat, Pratt et al.

Medical Domain Decide on important question types in an

advance What are the adverse effects of drug D? What is the prognosis for treatment T?

Make use of MeSH categories Retain only those types of categories known

to be useful for this type of query.

Pratt, W., Hearst, M, and Fagan, L. A Knowledge-Based Approach to Organizing Retrieved Documents. AAAI-99


DynaCat, Pratt et al.



DynaCat Study, Pratt et al.

Design Three queries 24 cancer patients Compared three interfaces

ranked list, clusters, categories

Results Participants strongly preferred categories Participants found more answers using categories Participants took same amount of time with all

three interfaces



DynaCat study, Pratt et al.

Faceted Category Grouping

Multiple Categories per Document


Search Usability Design Goals

1. Strive for Consistency2. Provide Shortcuts3. Offer Informative Feedback4. Design for Closure5. Provide Simple Error Handling6. Permit Easy Reversal of Actions7. Support User Control8. Reduce Short-term Memory Load

From Shneiderman, Byrd, & Croft, Clarifying Search, DLIB Magazine, Jan 1997. www.dlib.org


How to Structure Information for Search and Browsing?

Hierarchy is too rigid

KL-One is too complex

Hierarchical faceted metadata: A useful middle ground


Inflexible Force the user to start with a particular category What if I don’t know the animal’s diet, but the

interface makes me start with that category?

Wasteful Have to repeat combinations of categories Makes for extra clicking and extra coding

Difficult to modify To add a new category type, must duplicate it

everywhere or change things everywhere

The Problem with Hierarchy


The Idea of Facets

Facets are a way of labeling data A kind of Metadata (data about data) Can be thought of as properties of items

Facets vs. Categories Items are placed INTO a category system Multiple facet labels are ASSIGNED TO items


The Idea of Facets

Create INDEPENDENT categories (facets) Each facet has labels (sometimes arranged in a

hierarchy)

Assign labels from the facets to every item Example: recipe collection

Course

Main Course

CookingMethod

Stir-fry

Cuisine

Thai

Ingredient

Bell Pepper

Curry

Chicken


The Idea of Facets

Break out all the important concepts into their own facets

Sometimes the facets are hierarchical Assign labels to items from any level of the

hierarchy

Preparation Method Fry Saute Boil Bake Broil Freeze

Desserts Cakes Cookies Dairy Ice Cream Sorbet Flan

Fruits Cherries Berries Blueberries Strawberries Bananas Pineapple


Using Facets

Now there are multiple ways to get to each item

Preparation Method Fry Saute Boil Bake Broil Freeze

Desserts Cakes Cookies Dairy Ice Cream Sherbet Flan

Fruits Cherries Berries Blueberries Strawberries Bananas Pineapple

Fruit > PineappleDessert > Cake

Preparation > Bake

Dessert > Dairy > SherbetFruit > Berries > Strawberries

Preparation > Freeze


Using Facets

The system only shows the labels that correspond to the current set of items Start with all items and all facets The user then selects a label within a facet This reduces the set of items (only those that

have been assigned to the subcategory label are displayed)

This also eliminates some subcategories from the view.


The Advantage of Facets

Lets the user decide how to start, and how to explore and group.

After refinement, categories that are not relevant to the current results disappear.

Seamlessly integrates keyword search with the organizational structure.

Very easy to expand out (loosen constraints) Very easy to build up complex queries.


Advantages of Facets

Can’t end up with empty results sets (except with keyword search)

Helps avoid feelings of being lost. Easier to explore the collection.

Helps users infer what kinds of things are in the collection.

Evokes a feeling of “browsing the shelves” Is preferred over standard search for

collection browsing in usability studies. (Interface must be designed properly)


Advantages of Facets

Seamless to add new facets and subcategories

Seamless to add new items. Helps with “categorization wars”

Don’t have to agree exactly where to place something

Interaction can be implemented using a standard relational database.

May be easier for automatic categorization















Facets vs. Hierarchy

Early Flamenco studies compared allowing multiple hierarchical facets vs. just one facet.

Multiple facets was preferred and more successful.


Limitation of Facets

Do not naturally capture MAIN THEMES Facets do not show RELATIONS explicitly

AquamarineRed

Orange

DoorDoorway

Wall

Which color associated with which object?Photo by J. Hearst, jhearst.typepad.com


Usability Studies

Usability studies done on 3 collections: Recipes: 13,000 items Architecture Images: 40,000 items Fine Arts Images: 35,000 items

Conclusions: Users like and are successful with the

dynamic faceted hierarchical metadata, especially for browsing tasks

Very positive results, in contrast with studies on earlier iterations.


Post-Interface Assessments

All significant at p<.05 except “simple” and “overwhelming”


Post-Test Comparison

15 16

2 30

1 29

4 28

8 23

6 24

28 3

1 31

2 29

FacetedBaseline

Overall Assessment

More useful for your tasksEasiest to useMost flexible

More likely to result in dead endsHelped you learn more

Overall preference

Find images of rosesFind all works from a given period

Find pictures by 2 artists in same media

Which Interface Preferable For:


Summary: Evaluation Good Ideas

Longitudinal studies of real use Match the participants to the content of the

collection and the tasks Test against a strong baseline


Summary: Evaluation Problems

Bias participants towards a system “Try our interface” versus linear view

Tailor tasks unrealistically to benefit the target interface

Impoverish the baseline relative to the test condition

Conflate test conditions


Conclusions

Grouping search results seems beneficial in two circumstances:1. General web search, using transparent labeling

(monothetic terms) or category labels rather than cluster centroids.Effects: Works primarily on ambiguous queries,

(so used a fraction of the time) Promotes relevant results up from below the first page of hits

So important to group the related items together visually Users tend to select more documents than with linear search May work even better with meta-search Positive subjective responses (small studies) Visualization does not work.


Conclusions

Grouping search results seems beneficial in two circumstances:2. Collection navigation with faceted categories

Multiple angles better than single categories “searchers” turn into “browsers” Becoming commonplace in e-commerce, digital

libraries, and other kinds of collections Extends naturally to tags. Positive subjective responses (small studies)


Discussion

So … why aren’t the major web search engines doing it?

Usability of Grouping of Retrieval Results

Documents

google visit

sigir96marti hearst

cluster hypothesis

clusterbased approach

main themes

category links

multiple categories

clusterseach document