Solr Facets in Alfresco 4 Dan Tuffery. Who Am I? About Me Senior Software Engineer at Ixxus Working with Alfresco & Solr for 2.5 years Alfresco Certified.

Solr Facets in Alfresco 4

Dan Tuffery

Who Am I?

About Me

• Senior Software Engineer at Ixxus

• Working with Alfresco & Solr for 2.5 years

• Alfresco Certified Engineer and Solr Certified

• Winner of Alfresco’s 2012 Dashlet Challenge

Agenda

Talk Outline

• Alfresco Solr facets demonstration

• Solr facets overview

• Walkthrough of implementation

• Plans for the module

Alfresco Solr Facets Demonstration

Solr Facets Overview

3 Different flavours of facets

• Field facets

• Date facets

• Number range facets

Indexing Facet Fields

•Indexed

•Not Tokenised

•Not Stored

Facet Field

Parameters• facet = enables faceting

• facet.field = the field that you want to facet on

• facet.mincount = constraint count >= facet.mincount

• facet.limit = limit the number of constraints that are displayed

Facet Field

Parameters• facet = enables faceting

• facet.field = the field that you want to facet on

• facet.mincount = constraint count >= facet.mincount

• facet.limit = limit the number of constraints that are displayed

q=myfield:solr&facet=true&facet.field=myfield&facet.mincount=1&facet.limit=20

Date Facet

Parameters• facet.date = name of date field

• facet.date.start = start of the date range

• facet.date.end = End of the date range

• facet.date.gap = period of time in between each date range

Date Facet

Parameters• facet.date = name of date field

• facet.date.start = start of the date range

• facet.date.end = End of the date range

• facet.date.gap = period of time in between each date range

q=myfield:solr&facet=true&facet.date=created&facet.limit=20&facet.date.start=2012-0101T00:00:00.0Z&facet.date.gap

=1MONTH&facet.date.end=2012-10-22T00:00:00.0Z

Solr Facets In Alfresco 4

What Needed To Be Done?

Alfresco’s Facet Implementation

Out Of The Box

• Facet field POJO

• Logic to add facet field params to Solr request URL

• Facet results are mapped to JSON object

My Implementation

Five Main Tasks

1. Add facets to Alfresco (Facet Manager)

2. Solution to add facets to search request

3. Solution to return facets in the search response

4. Modify search results page to display facets

5. Add support for filter queries

1)Add Facets to Alfresco

Facet Manger

• Tool in Admin Console

• Based on current Tag Manager

• Create, Read, Update and Delete facets

• REST API in Alfresco (POST, GET, PUT, DELETE)

• Custom content model – ix:facetField

1)Add Facets to Alfresco

What facets to display• Content Type, Mime type, Property, Date

How the facet should be displayed• Display Name, Minimum Count, Limit

When the facet should be displayed• Individual Site, All Sites, Repository Search

Indexing Facet Fields in Alfresco

Best Practices

• Indexed

• Not Tokenised

• Not Stored

Indexing Facet Fields in Alfresco

Best Practices

• Indexed

• Not Tokenised

• Not Stored

Indexing Facet Fields in Alfresco

Best Practices

• Indexed

• Not Tokenised

• Not Stored

Indexed Field Name Indexed Value(s)

@{http://www.ixxus.com/model/1.0}location {en}new{en}york

@{http://www.ixxus.com/model/1.0}location.__ newyork

@{http://www.ixxus.com/model/1.0}location.u {en}New York

@{http://www.ixxus.com/model/1.0}location.__.u

New York

@{http://www.ixxus.com/model/1.0}location.sort

enNew York

Search Flow of Execution OOTB

Search Flow of Execution

2) Add Facets to Search Request

• Search request sent from Share• search.lib.js

var queryDef = { query: ftsQuery, language: "fts-alfresco", page: {maxItems: params.maxResults * 2}, templates: getQueryTemplate(), defaultField: "keywords", onerror: "no-results", sort: sortColumns, searchScope: searchScope, siteId: params.siteId, filterQueries: filterQueries };

nodes = search.query(queryDef);

2) Add Facets to Search Request

• Extended Search.java

String searchScope = (String)def.get("searchScope"); String siteId = (String)def.get("siteId"); .. .. if(searchScope != null) { List<FieldFacet> fieldFacets = facetService.getFieldFacets(searchScope, siteId); searchParameters.setFieldFacets(fieldFacets); } .. results = this.services.getSearchService().query(searchParameters);

2) Add Facets to Search Request

• Extended SolrQueryHttpClient.java• Generates Solr Request URL• Add parameters (including facets) to URL

List<FieldFacet> fieldFacets = searchParameters.getFieldFacets();

for(FieldFacet facet : fieldFacets) { url.append("&facet.field=" + facet.getField()); url.append("&facet.mincount=" + facet.getMinCount());, url.append("&facet.limit=" + facet.getLimit()); }

2) Add Facets to Search Request

• Extended SolrQueryHttpClient.java• Generates Solr Request URL• Add parameters (including facets) to URL

List<FieldFacet> fieldFacets = searchParameters.getFieldFacets();

for(FieldFacet facet : fieldFacets) { url.append("&facet.field=" + facet.getField()); url.append("&facet.mincount=" + facet.getMinCount());, url.append("&facet.limit=" + facet.getLimit()); }

facet.field=@{http://www.ixxus.com/model/1.0}location.__.u

3) Return Facets in Search Response

• Results mapped to SolrJSONResultSet.java• Response returns set of ScriptNode objects

Collection<Object> set = new LinkedHashSet<Object>(results.length(), 1.0f); .. //OOTB code to add scriptNodes to the set ..

if(searchParameters.getFieldFacets().size() > 0) { List<FieldFacet> fieldFacets = sp.getFieldFacets();

for(FieldFacet facetField : fieldFacets) { List<Pair<String, Integer>> facets = ((SolrJSONResultSet)results).getFieldFacet(facetField.getField()); set.add(new ScriptFacet(facetField, facets, getScope(), this.services)); } } return set;

3) Return Facets in Search Response

• ScriptFacet Implements Scopable• Represents a facet field and its properties• Constraint filter query is constructed here

public ScriptFacet(IxxusFieldFacet facetField, List<Pair<String, Integer>> facetList, Scriptable scope, ServiceRegistry services) {

this.services = services; this.facetField = facetField; this.scope = scope; this.constraints = getFacetResults(facetList); }

public Boolean isFacet() { return isFacet; }

3) Return Facets in Search Response

• Back up to search.lib.js to process results

var results, i, j, facetResults;

for (i = 0, j = nodes.length; i < j; i++) { if(nodes[i].isFacet) { var facet = nodes[i]; if(facet.facetHasAnyHits) { facetResults.push(facet); } } else {//it is a ScriptNode object, so process it in the usual way .. //OOTB code to add search results to results list

} }

return ({ items: results, facetItems: facetResults });

3) Return Facets in Search Response

• search.get.json.ftl "items": [ ... ], "facets": [ {

"facetDisplayName":"Location", “constraints" : [ {

"constraint": "San Jose (7)", "fq":"@{http://www.alfresco.org/model/content/1.0}title.__.u:San Jose"

},{ "constraint":"Berlin (4)",

"fq":"@{http://www.alfresco.org/model/content/1.0}title.__.u:Berlin"},...

]}

]

4) Modify Search Results Page

• Search Results page layout changed dynamically• Nested divs to display facet column

4) Modify Search Results Page

• search.js (YUI)var facets = oFullResponse.facets, resultsContainer, facetsDiv, results, body; if(facets && facets.length > 0) { resultsContainer = new Element(document.createElement("div"), { id: “results-container" }).addClass("results-container");

facetsDiv = new Element(document.createElement("div"), { id: "facetsDiv" }).addClass("search-facets"); Ixxus.search.solr.generateFacets(me, facets, facetsDiv);

body = new Element(Dom.get("body"));

results = new Element(Dom.get("results")); results.addClass("results-right");

resultsContainer.appendChild(facetsDiv); resultsContainer.appendChild(results); body.appendChild(resultsContainer);}

5) Add Support For Filter Queries

• Each facet constraint is a link

• Original search is executed + filter query/queries

• Same flow of execution as facets

• Filter queries added to the Solr request URL

5) Add Support For Filter Queries

• SolrQueryHttpClient.java

List<String> filterQueries = searchParameters.getFilterQueries();

for(String filterQuery : filterQueries) {

filterQuery = filterQuery.replace("{", "\\{");

filterQuery = filterQuery.replace("http", "http\\");

filterQuery = filterQuery.replace("}", "\\}");

filterQuery = filterQuery.replace(" ", "*");

filterQuery = filterQuery.replace("TO", " TO ");

url.append("&fq=").append(encoder.encode("{!lucene}" + filterQuery, "UTF8"));

}

Module Plans

Ixxus Solr

• Code has been contributed back to Alfresco

• Alfresco Enterprise 4.2 (Spring 2013)

• Maybe other community release before then

Module Plans

TODO

• Number range facets

• Multi field faceting

• Hierarchical faceting

Additional Information

• Search Terms Dashlet

• Uses faceting in Alfresco 4

• https://github.com/ixxus/ix-search-terms-dashlet

• Ixxus blog• http://www.ixxus.com/blog/2012/10/solr-facets-alfresco-walkthrough/

Summary

• Easy and configurable way to add facet fields

• Solr facets

• Field Facets

• Date Facets

• Support for filter queries

Questions?