Scalable Search Analytics

Scalable Search and AnalyticsRavi Krishnamurthy, VP Technical Services, [email protected]

Yann Yu, Systems Engineer, [email protected]

• Motivation: Why Search AND Analytics?

• Apache Solr and Lucidworks SILK

• Solution Architectures

• Demo(s)

• Q & A

• Resources

Agenda

Why Search AND Analytics?

AnalysisData Insight Action Value

Search is more than just a box.

personal. contextual. actionable.

Search makes data

Search is everywhere.

ecommerce

log analysis

site search

compliance

enterprise apps

Secure access to all your data through one interface, empowering everyone in your organization to access the data they need.

Search is the key to unlocking big data.

vSearch anything.

query

Traditional enterprise search was all about the query.

Search can be smarter.

location search history query permissions context

Personal, contextual, relevant results: consumer-like simplicity and power in the enterprise.

Solr in a nutshell

8M+ total downloads

Solr is both established & growing

250,000+monthly downloads

Largest community of developers.

2500+open Solr jobs.

Solr most widely used search solution on the planet.

LucidworksUnmatched Solr expertise.

1/3of the active committers

70%of the open source code is committed

Lucene/Solr Revolutionworld’s largest open source user

conference dedicated to Lucene/Solr.

Solr has tens of thousands of applications in production.

You use Solr everyday.

• Search-first NoSQL store

• Distributed, Horizontally Scalable

• Stable and Robust

• Deep Paging

• Accurate Facets and Stats

• Stats on Pivots (5.0)

• Easier to start-up; run as a service on Linux (5.0)

• Your Content, Your Way (5.0)

Solr and Analytics

• Solr - Logstash - Kibana

• http://lucidworks.com/product/integrations/silk/

• Open source at:

• https://github.com/LucidWorks/banana

• https://github.com/LucidWorks/solrlogmanager

SiLK

data enrichment

your business

your app

your datamachine learning

recommendations landing pages relevancy tuningsecurity

connector framework signal processing

api reporting admin

Lucidworks FusionEverything your team needs to rapidly design and deploy next-generation search apps to your entire organization.

Enterprise Search

Lucidworks connectors processes documents and

sends to SolrCloud

Standard document storage and search

Log record search

Machine generated log records are sent to Flume.

Flume forwards raw log record to Hadoop for archiving.

Flume simultaneously parses out data in record into a Solr document,

forwarding resulting document to Solr

Lucidworks SiLK exposes real-time statistics and analytics to end-users,

as well as full-text search

High volume indexing of many small records

Co-existence with other NoSQL solutions

eCommerce: Search is Recommendation

Catalog

Signals

Pipeline

Your App

Fusion

http://github.com/lucidworks/solr-for-datascience

• Solr: http://lucene.apache.org/solr

• Company: http://www.lucidworks.com

• Our blog: http://www.lucidworks.com/blog

• Blog on stats and facets: http://lucidworks.com/blog/you-got-stats-in-my-facets/

• Fusion: http://www.lucidworks.com/products/fusion

• Solr for Data Science code: http://github.com/lucidworks/solr-for-datascience

• Email: [email protected]; [email protected]

Resources