elasticsearch

Satish Mohan

Your Data,Your Search

Tuesday, 12 March 13

Enterprises today are collecting and have access to more data points in their ecosystem then ever.


File Store Example

File / Folder Navigation

Integration - Mount Points

Limited Metadata

Hierarchical Structure

Regular File Store


• Find a document from December 2011 about transfer containing proposal and David

• Find the document received from John containing David and transfer

• Find the revisions of transfer document

File Store Example



Limited Metadata

Hierarchical Structure


• Find a document from December 2011 about transfer containing proposal and David

• Find the document received from John containing David and transfer

• Find the revisions of transfer document

File Store Example



Limited Metadata

Hierarchical Structure Collections / Documents

Local / Distributed Integrations

Semantic Metadata

Declarative Queries

Automatic Indexing

Provenance

Automatic Organization

Virtual Collections

Regular File Store

Intelligent File Store


ElasticSearch is an open source, scalable, distributed, cloud-ready, highly-available full-text search engine and database with powerful aggregation features, communicating by JSON over RESTful HTTP, based on Apache Lucene.


Structured Data

Unstructured Data Data Refinery

Message Queues

Inverted index

Transaction Log Versioning

Source Document

Capture & Curate

Index

Streams

Analyse

Search

MemoryShared FS FS + MemoryLocal FS

Document Store

Playing with ElasticSearch


Playing with ElasticSearchRivers

• Data flows from sources using Rivers

• Continues to add data as it flows

• Can be added, removed, configured dynamically






ES NodeData Source

Data Source

Data Source

River

River

River

ES Index






ES NodeData Source

Data Source

Data Source

River

River

River

ES Index


Playing with ElasticSearchRiver Modules

• CouchDB • JDBC

• MongoDB • Solr

• Wikipedia • Jira

• Twitter • CSV

• ActiveMQ • FileSystem

• RabbitMQ • SysInfo

• NSQ • Logs

• RSS • LDAP


Playing with ElasticSearchIndex

• Describes document structure to the search engine

• Automatically created with sensible defaults

• Explicit mapping can be provided (generally, a good idea)

• Simple:

• string, integer/long, float/double, boolean, and null

• Complex:

• array, object


Structured Data


Message Queues

Inverted index


Source Document Shards

Replication Load Balancing Nodes

Distributed

Capture & Curate

Index

Streams

Analyse

Search


Document Store



Playing with ElasticSearchDistributed Model

• Number of shards is the scaling unit [ #shards > #nodes ]

• each one is a separate Lucene index thus, many per-index settings are available

• Moving shards around is faster than splitting them (no reindex)

• Replicas also serves reads, allowing to scale search

• # of replicas can be updated dynamically after index creation

Node 1

user (0)

user (1)

Node 2

user1 (0)

user (1)

Node 3

user (0)

user2 (0)

Automatic Discovery Protocol

Replica

Shard


Playing with ElasticSearchIndex Aliases

curl -X POST 'http://localhost:9200/_aliases' -d '{

"actions" : [{

"add" : { “index” : “users”,

“alias” : “user_1”,

“filter” : { “term” : { “user” : “1” } }, “routing” : “1”

}} ]

}'

Indexing and search happens on the alias, with automatic use of routing and filtering


http://localhost:9200/_aliases'


Playing with ElasticSearchIndex Aliases

curl -X POST 'http://localhost:9200/_aliases' -d

' {"actions" : [ { "add" : {

"index" : "user_1", "alias" : "users"

}

},{ "add" : {

"index" : "user_2", "alias" : "users"

}

} ]}'

users

user_1

user_2

curl -X GET "http://localhost:9200/users/_search?q=..."




http://localhost:9200/articles,users/_search?q=body:nice

http://localhost:9200/articles,users/_search?q=body:nice

Structured Data


Message Queues

Inverted index


Source Document Shards


Distributed

Capture & Curate

Index

Streams

Analyse

Search

Transport

HTTP WebSockets

Thrift

ZeroMQ

memcached

TCP


Document Store



Structured Data


Message Queues

Inverted index


Source Document

Data Sources

Tokenisers

Retrieval Models

Structured Results

Language Bindings Transport

Shards


Distributed

Capture & Curate

Index

Streams

Analyse

Search

Transport

HTTP WebSockets

Thrift

ZeroMQ

memcached

TCP

Modules

Extend


Document Store



Structured Data


Message Queues

Inverted index


Source Document

Data Sources

Tokenisers

Retrieval Models

Structured Results


Shards


Distributed

Zen EC2

Capture & Curate

Index

Streams

Analyse

Search

Transport

HTTP WebSockets

Thrift

ZeroMQ

memcached

TCP

Modules

Extend

Discovery


Document Store



Structured Data


Message Queues

Inverted index


Source Document

Data Sources

Tokenisers

Retrieval Models

Structured Results


Shards


Distributed

Zen EC2

mvel Python

Groovy

Javascript

Capture & Curate

Index

Streams

Analyse

Search

Transport

HTTP WebSockets

Thrift

ZeroMQ

memcached

TCP

Modules

Extend

Script

Discovery


Document Store



Structured Data


Message Queues

Inverted index


Source Document

Data Sources

Tokenisers

Retrieval Models

Structured Results


Shards


Distributed

Zen EC2

mvel Python

Groovy

Javascript

Capture & Curate

Index

Streams

Analyse

Search

Transport

HTTP WebSockets

Thrift

ZeroMQ

memcached

TCP

Modules

Extend

Script

Monitor

Discovery


Document Store



Structured Data


Message Queues

Inverted index


Source Document

Data Sources

Tokenisers

Retrieval Models

Structured Results


Shards


Distributed

Zen EC2

mvel Python

Groovy

Javascript

Capture & Curate

Index

Streams

Analyse

Search

Transport

HTTP WebSockets

Thrift

ZeroMQ

memcached

TCP

Modules

Extend

Script

Monitor

Discovery

RESTful


Document Store



REST API : http://host:port/[ index]/[type]/[_action/id]HTTP Methods : GET, POST, PUT, DELETE



http://host

http://host

REST API : http://host:port/[ index]/[type]/[_action/id]HTTP Methods : GET, POST, PUT, DELETE


Some Definitions.....

• index -> Like a database

• type -> Like a table

• id -> Like a row in a table


http://host

http://host

Playing with ElasticSearchREST API : http://host:port/[index]/[type]/_action/id]HTTP Methods : GET, POST, PUT, DELETE

curl -X POST "http:// localhost:9200/art ic les/art ic le/1 " -d '{

" t i t le" : "E last icSearch Understands JSON!" , "body" : "E last icSearch not only “works” with JSON, i t understands i t ! Let ’s f i rst . . . " , "publ ished_on" : "2013/02/06 10:00:00" , " tags" : [ "search" , " json"] , "author" : {

" f i rst_name" : "Bruce" , " last_name" : "Croft" , "emai l" : "bruce@croft .org"

}} '

requ

est


http://host

http://host

http://localhost:9200/articles/article


mailto:[email protected]



curl -X POST "http:// localhost:9200/art ic les/art ic le/1 " -d '{

" t i t le" : "E last icSearch Understands JSON!" , "body" : "E last icSearch not only “works” with JSON, i t understands i t ! Let ’s f i rst . . . " , "publ ished_on" : "2013/02/06 10:00:00" , " tags" : [ "search" , " json"] , "author" : {


}} '

{"ok" : true,"_ index":"art ic les" ,"_type":"art ic le" ,"_ id" :"1" ,"_version":1

}

requ

est

resp

onse


http://host

http://host






requ

est

curl -X GET "http://localhost:9200/articles/_search?q=author.first_name:BRUCE"


http://host

http://host

http://localhost:9200/articles/_search?q=author.first_name:BRUCE



{"took":1,"t imed_out" : fa lse,"_shards" :{"total" :5 ,"successful" :5 ," fa i led" :0} ,"hits" : {

" total" :1 , "max_score":0.30685282, "hits" : [{ "_ index":"art ic les" , "_type":"art ic le" , "_ id" :"1" , "_score":0.30685282, "_source" :

{ " t i t le" : "E last icSearch Understands JSON!" , "body" : "E last icSearch not only “works” with JSON, i t understands i t ! Let ’s f i rst . . . " , "publ ished_on" : "2013/02/06 10:00:00" , " tags" : [ "search" , " json"] , "author" : {


}} } ] } }

requ

est


resp

onse


http://host

http://host






{"took":1,"t imed_out" : fa lse,"_shards" :{"total" :5 ,"successful" :5 ," fa i led" :0} ,"hits" : {

"total" :1 , "max_score":0.30685282, "hits" : [{ "_index":"art ic les", "_type":"art ic le", "_ id":"1", "_score":0.30685282, "_source" :

{ "t i t le" : "Elast icSearch Understands JSON!", "body" : "Elast icSearch not only “works” with JSON, i t understands it ! Let ’s f irst . . ." , "published_on" : "2013/02/06 10:00:00", "tags" : ["search", " json"] , "author" : {

"f irst_name" : "Bruce", " last_name" : "Croft" , "email" : "[email protected]"

}}

} ] } }

requ

est


resp

onse

Location & ID

Document Source

Total number of documents


http://host

http://host



Structured Data


Message Queues

Inverted index


Source Document

Data Sources

Tokenisers

Retrieval Models

Structured Results


Shards


Distributed

Zen EC2

mvel Python

Groovy

Javascript

Capture & Curate

Index

Streams

Analyse

Search

Transport

HTTP WebSockets

Thrift

ZeroMQ

memcached

TCP

Modules

Extend

Script

Monitor

Discovery

RESTful Micro Apps


Document Store



Structured Data


Message Queues

Inverted index


Source Document

Data Sources

Tokenisers

Retrieval Models

Structured Results


Shards


Distributed

Zen EC2

mvel Python

Groovy

Javascript

HTML5/CSS3 Javascript

Capture & Curate

Index

Streams

Analyse

Search

Transport

HTTP WebSockets

Thrift

ZeroMQ

memcached

TCP

Modules

Extend

Script

Monitor

Discovery

RESTful Micro Apps


Document Store



Micro Applications

Rich, interactive single-page web applications powered by JavaScript, HTML and CSS.


Micro Applications


Micro Applications


Micro Applications


Micro Applications


Micro ApplicationsRich, interactive single-page web applications powered by JavaScript, HTML and CSS.

• A self-described framework for ambitious applications

• Rails-inspired “convention over configuration” approach

• High level abstractions, two-way binding and auto-updating templates

Data Model

ControllerRouter

View

Model Model

Controller

View View

View


Micro ApplicationsRich, interactive single-page web applications powered by JavaScript, HTML and CSS.

• A self-described framework for ambitious applications

• Rails-inspired “convention over configuration” approach

• High level abstractions, two-way binding and auto-updating templates

• Ember Data

• Client side storage adapter

• Provides a common interface to persist application data

• RESTful HTTP service - primary endpoint

• Browser’s localStorage

• Emerging web databases such as IndexedDB

Data Model

ControllerRouter

View

Model Model

Controller

View View

View


Playing with ElasticSearchMore Features.....

• document oriented • load balancing

• versioning • plugins

• parent/child docs • more_like_this

• scripting • multi_field mapping

• dynamic mapping templates • percolation

• bulk indexing • facets

• geo location • index aliases

• auto-complete • ngrams & edge-ngrams

• histograms • rivers


Structured Data


Message Queues

Inverted index


Source Document

Data Sources

Tokenisers

Retrieval Models

Structured Results


Shards


Distributed

Zen EC2

mvel Python

Groovy

Javascript

HTML5/CSS3 Javascript

Capture & Curate

Index

Streams

Analyse

Search

Transport

HTTP WebSockets

Thrift

ZeroMQ

memcached

TCP

Modules

Extend

Script

Monitor

Discovery

RESTful Micro Apps


Document Store

An alternative that would allow scientists or even casual users to perform analysis of distributed data regardless of where the data resides.


Search is the primary interface for getting information today. Let’s build on it.

Search

DiscoverAnalyse



Data Management Tools - Challenges

• Interactive queries, data exploration or iterative query refinement poses significant challenges for current methods

• Building and running jobs and queries requires deep understanding of cluster size and structure, job performance, etc.

• Time-consuming to set up, deploy and use


elasticsearch

Technology

data points

elasticsearch rivers

index settings

memorylocal fs document

search memoryshared

document structure

index creation node

separate lucene index