Scaling massive elastic search clusters - Rafał Kuć - Sematext

Scaling Massive ElasticSearch

Clusters

Rafał Kuć – Sematext International

@kucrafal @sematext sematext.com

Who Am I

• „Solr 3.1 Cookbook” author

• Sematext software engineer

• Solr.pl co-founder

• Father and husband :-)

Copyright 2012 Sematext Int’l. All rights reserved

What Will I Talk About ?

• ElasticSearch scaling

• Indexing thousands of documents per second

• Performing queries in tens of milliseconds

• Controling shard and replica placement

• Handling multilingual content

• Performance testing

• Cluster monitoring


The Challenge

• More than 50 millions of documents a day

• Real time search

• Less than 200ms average query latency

• Throughput of at least 1000 QPS

• Multilingual indexing

• Multilingual querying


Why ElasticSearch ?

• Written with NRT and cloud support in mind

• Uses Lucene and all its goodness

• Distributed indexing with document

distribution control out of the box

• Easy index, shard and replicas creation on live

cluster


Index Design

• Several indices (at least one index for each day

of data)

• Indices divided into multiple shards

• Multiple replicas of a single shard

• Real-time, synchronous replication

• Near-real-time index refresh (1 to 30 seconds)


Shard Deployment Problems

• Multiple shards per node

• Replicas on the same nodes as shards

• Not evenly distributed shards and replicas

• Some nodes being hot, while others are cold


Default Shard Deployment

ElasticSearch Cluster

Node 1 Node 2

Node 3

Shard 1 Shard 2 Shard 3 Replica 1

Replica 2

Replica 3


What Can We Do With Shards Then ?

• Contol shard placement with node tags:

– index.routing.allocation.include.tag

– index.routing.allocation.exclude.tag

• Control shard placement with nodes IP addresses:

– cluster.routing.allocation.include._ip

– cluster.routing.allocation.exclude._ip

• Specified on index or cluster level

• Can be changed on live cluster !


Shard Allocation Examples

• Cluster level:

curl -XPUT localhost:9200/_cluster/settings -d '{

"persistent" : {

"cluster.routing.allocation.exclude._ip" : "192.168.2.1"

}

}'

• Index level:

curl -XPUT localhost:9200/sematext/ -d '{

"index.routing.allocation.include.tag" : "nodeOne,nodeTwo"

}'


Number of Shards Per Node

• Allows one to specify number of shards per

node

• Specified on index level

• Can be changed on live indices

• Example:

curl -XPUT localhost:9200/sematext -d '{

"index.routing.allocation.total_shards_per_node" : 2

}'


Controlled Shard Deployment


Node 1 Node 2

Node 3

Shard 1

Shard 2

Shard 3 Replica 1Replica 2

Replica 3


Does Routing Matters ?

• Controls target shard for each document

• Defaults to hash of a document identifier

• Can be specified explicitly (routing parameter) oras a field value (a bit less performant)

• Can take any value

• Example:

curl -XPUT localhost:9200/sematext/test/1?routing=1234 -d '{

"title" : "Test routing document"

}'


Indexing the Data


Node 1 Node 2

Node 3

Shard

1

Shard

2

Shard

3

Replica

1

Replica

2

Replica

3

Indexing application


How We Indexed Data


Node 1 Node 2

Shard 1 Shard 2

Node 3

Indexing application


Nodes Without Data

• Nodes used only to route data and queries to

other nodes in the cluster

• Such nodes don’t suffer from I/O waits (of

course Data Nodes don’t suffer from I/O waits

all the time)

• Not default ElasticSearch behavior

• Setup by setting node.data to false


Multilingual Indexing

• Detection of document's language before

sending it for indexing

• With, e.g. Sematext LangID or Apache Tika

• Set known language analyzers in configuration

or mappings

• Set analyzer during indexing (_analyzer field)


Multilingual Indexing Example

curl -XPUT localhost:9200/sematext/test/10 -d '{

"title" : "Test document",

"langId" : "english"

}'

{

"test" : {

"_analyzer" : { "path" : "langId" },

"properties" : {

"id" : { "type" : "long", "store" : "yes", "precision_step" : "0" },

"title" : { "type" : "string", "store" : "yes", "index" : "analyzed" },

"langId" : { "type" : "string", "store" : "yes", "index" : "not_analyzed" }

}

}

}


Multilingual Queries

• Identify language of query before its execution

(can be problematic)

• Query analyzer can be specified per query

(analyzer parameter):curl -XGET

localhost:9200/sematext/_search?q=let+AND+me&analyzer=english


Query Performance Factors – Lucene

level

• Refresh interval

– Defaults to 1 second

– Can be specified on cluster or index level

– curl -XPUT localhost:9200/_settings -d '{ "index" : { "refresh_interval" : "600s" } }'

• Merge factor

– Defaults to 10

– Can be specified on cluster or index level

– curl -XPUT localhost:9200/_settings -d '{ "index" : { "merge.policy.merge_factor" : 30 } }'


Let’s Talk About Routing Once Again

• Routes a query to a particular shard

• Speeds up queries depending on number of shards for a given index

• Have to be specified manualy with routingparameter during query

• routing parameter can take any value:

curl -XGET 'localhost:9200/sematext/_search?q=test&routing=2012-02-16'


Querying ElasticSearch – No Routing

Shard 1 Shard 2 Shard 3 Shard 4


ElasticSearch Index

Application




ElasticSearch Index

Application

Querying ElasticSearch – With Routing


Performance Numbers

Queries without routing (200 shards, 1 replica)

#threads Avg response time Throughput 90% line Median CPU Utilization

1 3169ms 19,0/min 5214ms 2692ms 95 – 99%

Queries with routing (200 shards, 1 replica)

#threads Avg response time Throughput 90% line Median CPU Utilization

10 196ms 50,6/sec 642ms 29ms 25 – 40%

20 218ms 91,2/sec 718ms 11ms 10 – 15%


Scaling Query Throughput – What Else ?

• Increasing the number of shards for data

distribution

• Increasing the number of replicas

• Using routing

• Avoid always hitting the same node and

hotspotting it


FieldCache and OutOfMemory

• ElasticSearch default setup doesn’t limit field

data cache size


FieldCache – What We Can do With It ?

• Keep its default type and set:– Maximum size (index.cache.field.max_size)

– Expiration time (index.cache.field.expire)

• Change its type:– soft (index.cache.field.type)

• Change your data:– Make your fields less precise (ie: dates)

– If you sort or facet on fields think if you can reducefields granularity

• Buy more servers :-)


FieldCache After Changes


Additional Problems We Encountered

• Rebalancing after full cluster restarts

– cluster.routing.allocation.disable_allocation

– cluster.routing.allocation.disable_replica_allocation

• Long startup and initialization

• Faceting with strings vs faceting on numbers on

high cardinality fields


JVM Optimization

• Remember to leave enough memory to OS for

cache

• Make GC frequent ans short vs. rare and long

– -XX:+UseParNewGC

– -XX:+UseConcMarkSweepGC

– -XX:+CMSParallelRemarkEnabled

• -XX:+AlwaysPreTouch (for short performance

tests)


Performance Testing

• Data

– How much data do I need ?

– Choosing the right queries

• Make changes

– One change at a time

– Understand the impact of the change

• Monitor your cluster (jstat, dstat/vmstat, SPM)

• Analyze your results


ElasticSearch Cluster Monitoring

• Cluster health

• Indexing statistics

• Query rate

• JVM memory and garbage collector work

• Cache usage

• Node memory and CPU usage


Cluster Health


Node restart

Indexing Statistics


Query Rate


JVM Memory and GC


Cache Usage


CPU and Memory


Summary

• Controlling shard and replica placement

• Indexing and querying multilingual data

• How to use sharding and routing and not to

tear your hair out

• How to test your cluster performance to find

bottle-necks

• How to monitor your cluster and find

problems right away


We Are Hiring !

• Dig Search ?

• Dig Analytics ?

• Dig Big Data ?

• Dig Performance ?

• Dig working with and in open – source ?

• We’re hiring world – wide !

http://sematext.com/about/jobs.html


How to Reach Us

• Rafał Kuć

– Twitter: @kucrafal

– E-mail: [email protected]

• Sematext

– Twitter: @sematext

– Website: http://sematext.com

• Graphs used in the presentation are from:

– SPM for ElasticSearch (http://sematext.com/spm)


Thank You For Your Attention

Scaling massive elastic search clusters - Rafał Kuć - Sematext

Technology

rights reserved

index level

rights reserved

xput localhost

indexing

nodes

data

test