Elasticsearch quick Intro (English)

Elasticsearch

federico.panini@fazland.com - CTO

Federico Panini

CTO @ fazland.comemail : federico.panini@fazland.com

LikedIn : https://uk.linkedin.com/in/federicopaninislides : http://www.slideshare.net/FedericoPanini

What is Elasticsearch

full-text search engine

“A search engine is an automated system which, upon request, uses a set of data and return an index of its content classifying them based on math/stats algorithm used to set the relevance, based in a search key.”

What’s Elasticsearch ?

full-text search engine

“It’s a distributed, scalable, and highly available Real-time search and analytics software.”

What’s Elasticsearch ?full-text search engine

Real-time data Realtime data analysis Distributed system High Availability Full-text searches Document oriented DB Schemaless DB

RESTFul Api Persistence per-operation Open Source Based on Apache Lucene Optimistic version control

What’s Elasticsearch ?features

Apache Lucene #1

It’s the heart of Elasticsearch

Lucene is the search engine of Elasticsearch

Apache Lucene #1

It’s in Java

It’s an Apache Software Foundation, so Open Source!

What has more than Lucene

full-text searches

horizontal scaling high availability Easy to use near real time

Architecture

requirements - CPU

Elasticsearch doesn’t need a lot of CPU.

The advice is to use the last CPU model available.

In general is a good practice to use machines with 2 to 8 cores.

Architecture

requirements - Disco

The I/O disk need is really important for all clusters.

Please use SSD disks.

Architecture

requirements - HD - bonus slide …

One very important thing to know is you have to pay attention where data is stored and mostly how. The word you have to remember is scheduler. The scheduler on *nix system is responsible to decide when data should be “written” to disc and on which priority. Usually common unix OS setup cfq as scheduler, which for instance is a scheduler for rotating disks and optimised for them. The advice is to use SSD disks and to setup the SO to use “noop” or “deadline” which are scheduler optimised for SSD’s.

If you use the right scheduler you can reach improvements of 500x !!!

Operating Systems

Elasticsearch is written in Java, so it’s a multiplatform solution. Use the last JDK available.

Architecture

requirements - RAM

Elasticsearch is eager of RAM!!!

https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html

Architecture

memory !?!?

Use solutions with 64GB is fine not more give to the Java heap size not more than 32GB of RAM use more than one machine for elasticsearch in order

setup correctly the cluster.

Architecture

Installation

curl -L -O http://download.elasticsearch.org/PATH/TO/VERSION.zip unzip elasticsearch-$VERSION.zip cd elasticsearch-$VERSION

There are availbes packages for many distribution as Debian or RPM, and Puppet or Chef modules

Architecture

Java based

elasticsearch

Elasticsearch has been developed in JAVA

Robust Scalable Multiplatform

Talking to Elasticsearch

clients Java #1

There are 2 clients available in JAVA:

Node client : the client join the cluster as non-data node, this mean that the client knows perfectly where data are and on which node of the cluster.

clients Java #2

Transport client : is a lightweight client and is the tool used to comunicate with the cluster remotely.

clients Java #2

Both Java clients talk to the cluster on port 9300, which is the same port use by the cluster itself.

client API RESTful

All programming languages other than Java can talk to the Elasticsearch cluster through its API Rest available on port 9200.

There are many official clients available in different programming languages.: Groovy, JavaScript, .NET, PHP, Perl, Python, e Ruby

Elastic

Document oriented

NoSqlElasticsearch is a document oriented database. This mean Elasticsearch is a schema-less database.

After inserting documents inside Elasticsearch, the documents will be immediately indexed.

Elastic

Document oriented

JSONElasticseach uses JSON as interchange language between the server and the API clients.

Elastic

glossary

cluster nodes indexes shards replica segments in-memory buffers translog

Elastic

cluster

The cluster is a set which belong one or more nodes, which shares the same property cluster.name. The cluster is used to balance the load of the server itself. A node could be deleted or inserted to the cluster, the cluster itself will re-organise itself.

Elastic

cluster

Inside a cluster a node is elected as Master. The Master node is responsible to manage operations as creation or removal indexes, join or deletion of a node. Every node could be elected as Master.

Elastic

A node is a minimum element of Elasticsearch that ensures the proper working of the cluster.

Elastic

Database RDBMS Elasticsearch

DATABASE INDEX

Elastic

Table TYPE

Elastic

Document

ROW DOCUMENT

Elastic

Fields

COLUMNS FIELDS

Elastic

shards

If we want to start indexing data on Elasticsearch we need to create an index. Index is the term used only to identify a logical definition, which represent a pointer to one or more elements called SHARDS.

Elastic

shards

The shard is the low level element of Elasticsearch, and contains a subset of all the data inside and index.

The shard is in fact a single instance of Apache Lucene.

Elastic

Replica shards

Replica shards are mirrors of shards used to protect our data from hardware failures. As the shards they are used exactly as the shards.

Elastic

shards immutability

The number of shards for an index is defined at Index creation time and is IMMUTABLE.

Elastic

shards immutability

curl -X http://localhost:9200/blogs -d ‘{ "settings" : { "number_of_shards" : 3, "number_of_replicas" : 1 } }’

Elastic

shards immutability

curl http://localhost:9200/_cluster/health“{ "cluster_name": "elasticsearch", "status": "yellow", "timed_out": false, "number_of_nodes": 1, "number_of_data_nodes": 1, "active_primary_shards": 3, "active_shards": 3, "relocating_shards": 0, "initializing_shards": 0, "unassigned_shards": 3 }”

Elastic

shards immutability

Replica shards on a single node instance are useless, the meaning for cluster is nothing in this case. To make replica shard useful we need at least 2 nodes to have data redundancy.

Elastic

BONUS : manage data conflicts #1

Elastic

BONUS : manage conflicts #2 : Pessimistic Concurrency ControlUsed in standard RDBMS

This approach is based on the concept that conflict could happened frequently and so to avoid them the RDBMS lock the resource.

The process lock the access to the row before reading it, this way we the RDBMS is sure that only one process will access to this thread and can subsequently modify it and nobody else. At the end of its process (update/delete) the thread will release the LOCK.

Elastic

BONUS : manage conflicts #3 : Optimistic Concurrency Control

Elasticsearch uses OCC

This approach will consider conflicts as infrequent. The database won’t lock the resource when access to it.

The responsibility is given to the application : when data is amended between a read and write then the update fails. In this case you need to re-get the fresh new data and trying to update it.

Elastic

BONUS : manage conflicts#4 : Optimistic Concurrency Control

Elasticsearch is a distributed solution, concurrent and asynchronous. When a document is created / updated / deleted is absolutely necessary to replicate this information across the whole cluster.

Every command sent to the nodes is sent in parallel and could happen that some data will reach its destination (node) already expired.

Elastic

We need a way to understand that the entry we’re trying to update as been already updated by another process.

Elastic

VERSIONING

Elastic

In Elasticsearch every document has a field named:

_version

This system field is incremented every time an operation (update / delete) occurs over a document. In this way an update to _version:3 won’t be never applied to a document whose _version field value is at 4.

Elastic

This approach move all the responsibility from the database to the application! so WE are responsible to not create conflicts over a document or and index. If we want to be sure to not have loss of data we nee to implement writes with the use of versioning!

Elastic

http://www.jillesvangurp.com/2014/12/03/optimistic-locking-for-updates-in-elasticsearch/ h t tps : / / aphy r.com/pos ts /317 -ca l l -me-maybe-elasticsearch https://www.elastic.co/guide/en/elasticsearch/resiliency/current/index.html

Elastic

Simple searches #1

Create IndexAPI RestGETDELETEPOSTSEARCH

Elastic

Simple searches - CREATE AN INDEX

curl -XPUT http://fazlab.fazland.com:9200/fazlab-d "{ "settings" :

{ "number_of_shards" : 3, "number_of_replicas" : 1

Elastic

Simple searches - INDEX A DOCUMENT

curl -XPUThttp://fazlab.fazland.com:9200/fazlab/categories/1?pretty -d '{

nome: "Federico"}'

Elastic

Simple searches - GET A DOCUMENT

curl http://fazlab.fazland.com:9200/fazlab/categories/1?pretty

Elastic

Simple searches - DELETE A DOCUMENT

curl -XDELETE http://fazlab.fazland.com:9200/fazlab/categories/2?pretty

Elastic

Simple searches #1

DEMO SEARCHES!

Elastic

mapping and analysis

EXACT MATCH vs FULL TEXT

Elastic

Exact match Full Text

where name = ‘Federico’

and user_id = 2

and date > “2014-09-15”

“Frank has been to

South beach”

Frank / FRANK / frank

Elastic

Exact match

Full Text

binary : the document contains these values ?

How much is relevant the document compared to the

term used inside the query ?

Elastic

Elasticsearch to help a full-text search analyse the text and uses this result to build an inverted index.

Inverted Index Analyzer

Elastic

Inverted Index

1. The quick brown fox jumped over the lazy dog

2. Quick brown foxes leap over lazy dogs in summer

Elastic

Inverted Index

If we want to search the word “quick” and “brown” we will pick

only the documents where these 2 words are.

Elastic

Inverted Index

Elastic

ANALYZERS

An analyzer has 3 functions:

Character filters

Tokenizer

Token Filters

Elastic

ANALYZERS - Character Filters

The first part of an analyser is to parse every string with character filer which will clean / reorganize the strings before tokenization.

During this phase special HTML chars will be removed or & will be converted in AND.

Elastic

ANALYZERS - Tokenizer

The second phase of an analyser is tokenisation which will divide a sentence in small terms.

Elastic

ANALYZERS - Token FiltersSuccessivamente alla fase di Tokenizzazione delle stringhe in singoli termini (terms), i filtri (selezionati) sono applicati in sequenza. After tokenisation filters will be applied in sequence. For example :

- put lower case the whole text - remove stop words - add synonyms

Elastic

Standard Analyzer

“Set the shape to semi-transparent by calling set_trans(5)”

The standard analyzer is the default analyzer of Elasticsearch. Divide text in single words and remove most of punctuation.

“set, the, shape, to, semi, transparent, by, calling, set_trans, 5”

Elastic

Simple Analyzer

The simple analyser removes all characters which are not letters and put the whole text lowercase

“set, the, shape, to, semi, transparent, by, calling, set, trans”

Elastic

Whitespace Analyzer

The whitespace analyser will create token by white space and put text in lowercase

“Set, the, shape, to, semi, transparent, by, calling, set_trans(5)”

Elastic

Language Analyzer

This analyser uses a language specific feature to remove stop words or to do stemming.

“set, shape, semi, transpar, call, set_tran, 5”

Elastic

Language Analyzer

arabic, armenian, basque, brazilian, bulgarian, catalan, chinese, cjk, czech, danish, dutch, english, finnish, french, galician, german, greek, hindi, hungarian, indonesian, irish, italian, latvian, norwegian, persian, portuguese, romanian, russian, sorani, spanish, swedish, turkish, thai.

Elastic

Pre-built Analyzers

Standard Analyzer Simple Analyzer

Whitespace Analyzer Stop Analyzer

Keyword Analyzer Pattern Analyzer

Language Analyzers Snowball Analyzer Custom Analyzer

Elastic

Tokenizer

Standard Tokenizer Edge NGram Tokenizer

Keyword Tokenizer Letter Tokenizer

Lowercase Tokenizer NGram Tokenizer

Whitespace Tokenizer Pattern Tokenizer

UAX Email URL Tokenizer Path Hierarchy Tokenizer

Elastic

Token Filters

Standard Token Filter ASCII Folding Token Filter

Length Token Filter Lowercase Token Filter

NGram Token Filter Edge NGram Token Filter Porter Stem Token Filter

Shingle Token Filter Stop Token Filter

… more than 32 Filters

Elastic

Token Filters

THE END.

References• Elasticsearch : The Definitive Guide• https://en.wikipedia.org/wiki/Full_text_search• https://www.elastic.co/guide/en/elasticsearch/guide/current/

hardware.html• https://www.elastic.co/guide/en/elasticsearch/guide/current/

heap-sizing.html• https://mtalavera.wordpress.com/2015/02/16/monitoring-with-

collectd-and-kibana/• Fuzzy search : https://www.found.no/foundation/fuzzy-search/• Phonetic-plugin : https://github.com/elastic/elasticsearch-

analysis-phonetic

Elasticsearch quick Intro (English)

cto federico panini

cto memory

search engine of elasticsearch

cto requirements hd

unzip elasticsearch

systems elasticsearch

heart of elasticsearch

cto installation curl

Internet

Intro & Quick Overview

Tmux quick intro

ElasticSearch and...

Planetbiz quick intro

A Quick Intro to ReactiveX

SnapComms Quick Intro

Elasticsearch Quick Introduction

Elasticsearch intro output

Intro to Elaticsearch - Elasticsearch Bucharest Group @...

Git Quick Intro

Word 2007 – Quick Intro

SMAC Stack - A Quick Intro

Quick Intro:

Intro to search engines with Lucene and ElasticSearch

Quick Intro Guide - eatingenlightenment.com

Omnego Quick Intro