Enterprise search in plone using solr

Enterprise Search in Plone using Solr

Calvin Hendryx-ParkerPlone Conference 2010

Wednesday, October 27, 2010

PLONE CONFERENCE 2010

• Java Based

• Full-Text Search

• Web Services API

• Standards Based Interfaces

• Scalable

• XML Configuration

• Extensible

What is Solr?



• Indexing

• Query

Playing with Solr







• Data Schema

• Faceted Search

• Administrative Interface

• Incremental Updates

• Supports Sharding

• Index Databases, Local Files and Web Pages

• Supports Multiple Indexes

Solr Features



• Stopwords

• Synonyms

• Highlighted Context Snippets

• Spelling Suggestions

• More Like This Suggestions

• Supports Rich Documents

Solr Features








PLONE CONFERENCE 2010Solr Performance

• Wiktionary Dataset

• 49.5 Millions lines of XML

• 1.3 GB of data

• 1.7 Million Pages Indexed in 5.5 hours

• ZODB Size after import 1.1GB



collective.solr

Integration Options with Plone



• Monkey Patching

• Relies on collective.indexing

• Duplicates all indexes

• Sub-Optimal Integration with Zope Transactions

• Relies on Thread Locals

collective.solr Issues



What to do?



Reevaluate



• No Monkey Patching

• Simpler Code

Solr Integration as a Catalog Index



• ZCatalog Index

• Doesn't depend on Plone

• Utilizes new foreign_connections Connection Method

• Pass through Solr Queries

• Direct access to the Solr Response

Enter alm.solrindex







• Still handled by the ZCatalog

• Could change in the future

Sorting



• Handle Parsing Attributes for Indexing

• Translate field-specific queries to Solr

• Registered as Zope Utilities

alm.solrindex Field Handlers



<html><body><h3>Code Sample</h3><p>Replace this text!</p></body></html>

Example Handlerclass TextFieldHandler(DefaultFieldHandler):

def parse_query(self, field, field_query): name = field.name request = {name: field_query} record = parseIndexRequest(request, name, ('query',)) if not record.keys: return None

query_str = ' '.join(record.keys) if not query_str: return None

return {'q': u'+%s:%s' % (name, quote_query(query_str))}



• GenericSetup Profile

• Tests

• Uses solrpy instead of the unsupported solr.py

Other alm.solrindex Features



• Can replace several ZCatalog indexes

• Remove any indexes you have replaced

• Use it for all Text Indexes

• Still Utilize the ZCatalog Indexes for Everything Else

Tips



DemoProject Gutenburg Data



Questions?


Check out

sixfeetup.com/demos


Enterprise search in plone using solr

Technology