Autoplacer: Scalable Self-Tuning Data Placement in ......Autoplacer: Scalable Self-Tuning Data Placement in Distributed Key-value Stores ICAC’13 Jo~ao Paiva, Pedro Ruivo, Paolo Romano,

Autoplacer: Scalable Self-Tuning DataPlacement in Distributed Key-value Stores

ICAC’13

Joao Paiva, Pedro Ruivo, Paolo Romano, Luıs Rodrigues

Instituto Superior Tecnico / Inesc-ID, Lisboa, Portugal

June 27, 2013

Outline

Introduction

Our approach

Evaluation

Conclusions

Motivation

Collocating processing with storage can improve performance.

I Using random placement, nodes waste resources due tonode-intercommunication.

I Optimize data placement to improve locality and to reduceremote requests.

Motivation




Motivation




Approaches Using Offline Optimization

Algorithm:

1. Gather access trace for all items

2. Run offline optimization algorithms on traces

3. Store solution in directory

4. Locate data items by querying directory

I Fine-grained placement

I Costly to log all accesses

I Complex optimization

I Directory creates additional network usage

Approaches Using Offline Optimization

Algorithm:

1. Gather access trace for all items

2. Run offline optimization algorithms on traces

3. Store solution in directory

4. Locate data items by querying directory

I Fine-grained placement

I Costly to log all accesses

I Complex optimization

I Directory creates additional network usage

Main challenges

Cause: Key-Value stores may handle large amounts of data

Challenges:

1. Collecting Statistics: Obtaining usage statistics in anefficient manner.

2. Optimization: Deriving fine-grained placement for dataobjects that exploits data locality.

3. Fast lookup: Preserving fast lookup for data items.

Approaches to Data Access Locality

1. Consistent Hashing (CH):The “don’t care” approach

2. Distributed Directories:The “care too much” approach

Consistent Hashing

Don’t care for locality: items placed deterministically according tohash functions and full membership information.

I Simple to implement

I Solves lookup challenge by using local lookups

I No control on data placement → bad locality

I Does not address optimization challenge

Consistent Hashing

Don’t care for locality: items placed deterministically according tohash functions and full membership information.

I Simple to implement

I Solves lookup challenge by using local lookups

I No control on data placement → bad locality

I Does not address optimization challenge

Distributed Directories

Care too much for locality: nodes report usage statistics tocentralized optimizer, placement defined in a distributed directory(may be cached locally)

I Can solve statistics challenge using coarse statistics

I Solves optimization challenge with precise data placementcontrol

Hindered by lookup challenge:

I Additional network hop

I Hard to update

Distributed Directories

Care too much for locality: nodes report usage statistics tocentralized optimizer, placement defined in a distributed directory(may be cached locally)

I Can solve statistics challenge using coarse statistics

I Solves optimization challenge with precise data placementcontrol

Hindered by lookup challenge:

I Additional network hop

I Hard to update

Outline

Introduction

Our approach

Evaluation

Conclusions

Our approach: beating the challenges

Best of both worlds

I Statistics Challenge: Gather statistics only for hotspot items

I Optimization Challenge: Fine-grained optimization forhotspots

I Lookup Challenge: Consistent Hashing for remaining items

Algorithm overview

Online, round-based approach:

1. Statistics: Monitor data access to collect hotspots

2. Optimization: Decide placement for hotspots

3. Lookup: Encode / broadcast data placement

4. Move data

Algorithm overview





4. Move data

Statistics: Data access monitoring

Key concept: Top-K stream analysis algorithm

I Lightweight

I Sub-linear space usage

I Inaccurate result... But with bounded error



I Lightweight





I Lightweight



Algorithm overview





4. Move data

Optimization

Integer Linear Programming problem formulation:

min∑j∈N

∑i∈O

X ij(crr rij + crwwij) + Xij(cl

r rij + clwwij) (1)

subject to:

∀i ∈ O :∑j∈N

Xij = d ∧ ∀j ∈ N :∑i∈O

Xij ≤ Sj

Inaccurate input:

I Does not provide optimal placement

I Upper-bound on error

Accelerating optimization

1. ILP Relaxed to Linear Programming problem

2. Distributed optimization

LP relaxation

I Allow data item ownership to be in [0− 1] interval

Distributed Optimization

I Partition by the N nodes

I Each node optimizes hotspots mapped to it by CH

I Strengthen capacity constraint




LP relaxation









LP relaxation






Algorithm overview





4. Move data

Lookup: Encoding placement

Probabilistic Associative Array (PAA)

I Associative array interface (keys→values)

I Probabilistic and space-efficient

I Trade-off space usage for accuracy

Probabilistic Associative Array: Usage

Building

1. Build PAA from hotspot mappings

2. Broadcast PAA

Looking up objects

I If item not in PAA, use Consistent Hashing

I If item is hotspot, return PAA mapping

Probabilistic Associative Array: Usage

Building

1. Build PAA from hotspot mappings

2. Broadcast PAA

Looking up objects

I If item not in PAA, use Consistent Hashing

I If item is hotspot, return PAA mapping

PAA: Building blocks

I Bloom FilterSpace-efficient membership test (is item in PAA?)

I Decision tree classifierSpace-efficient mapping (where is hotspot mapped to?)

PAA: Building blocks

I Bloom FilterSpace-efficient membership test (is item in PAA?)

I Decision tree classifierSpace-efficient mapping (where is hotspot mapped to?)

PAA: Properties

Bloom Filter:

I False Positives: match items that it was not supposed to.

I No False Negatives: never return ⊥ for items in PAA.

Decision tree classifier:

I Inaccurate values (bounded error).

I Deterministic response: deterministic (item→node)mapping.

PAA: Properties

Bloom Filter:






PAA: Properties

Bloom Filter:






Algorithm Review


1. Statistics: Monitor data access to collect hotspotsTop-k stream analysis

2. Optimization: Decide placement for hotspotsLightweight distributed optimization

3. Lookup: Encode / broadcast data placementProbabilistic Associative Array

4. Move data

Algorithm Review





4. Move data

Algorithm Review





4. Move data

Algorithm Review





4. Move data

Outline

Introduction

Our approach

Evaluation

Conclusions

Experimental settings

I Integrated in Distributed Key-Value store (JBoss Infinispan)

I 40 Virtual Machines (10 physical machines)

I Gigabit network

Modified TPC-C benchmark

Induce controllable locality:

I Probability p: Nodes access data associated with a givenwarehouse.

I Probability 1− p: Nodes access data associated a randomwarehouse.

Remote operations

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 5 10 15 20 25 30

Perc

enta

ge o

f rem

ote

oper

atio

ns (%

)

Time (minutes)

100% locality90% locality50% locality

0% localitybaseline

Throughput

10

100

1000

0 5 10 15 20 25 30

Tran

sact

ions

per

sec

ond

(TX/

s)

Time (minutes)

100% locality90% locality50% locality

0% localitybaseline

Directory effects

10

100

1000

100% Locality 90% Locality 0% Locality

Tra

nsaction p

er

second (

tx/s

)Autoplacer

DirectoryBaseline

Outline

Introduction

Our approach

Evaluation

Conclusions

Conclusions

I Gather statistics only for hotspots

I Fine-grained hotspot placement

I Retain Local lookups using PAA

I Effective locality improvement

I Good network usage

I Considerable performance improvements

Conclusions

I Gather statistics only for hotspots

I Fine-grained hotspot placement

I Retain Local lookups using PAA

I Effective locality improvement

I Good network usage

I Considerable performance improvements

Thank you

Autoplacer: Scalable Self-Tuning Data Placement in ......Autoplacer: Scalable Self-Tuning Data Placement in Distributed Key-value Stores ICAC’13 Jo~ao Paiva, Pedro Ruivo, Paolo Romano,

Documents