Exponential Growth Without Sacrificing Platform Speed · 2018-05-25 · AEROSPIKE USER SUMMIT 2018 Exponential Growth Without Sacrificing Platform Speed Gil Allouche Founder and CEO
Post on 08-Jul-2020
0 Views
Preview:
Transcript
AERO SPIKE USER SUM M IT 2018
Exponential Growth Without Sacrificing Platform Speed
Gil Allouche
Founder and CEO
Metadata.io
Prithvi Sathiya
Data Engineer
Metadata.io
2 AEROSPIKE USER SUMMIT | Proprietary & Confidential | All rights reserved. © 2018 Aerospike Inc
3 AEROSPIKE USER SUMMIT | Proprietary & Confidential | All rights reserved. © 2018 Aerospike Inc
4 AEROSPIKE USER SUMMIT | Proprietary & Confidential | All rights reserved. © 2018 Aerospike Inc
The Problem
Context
● Relied on traditional SQL database
● Needed for Lookalikes and ICP’s
● Sufficient for our needs at that time
Problem Statement
● More customers needed more custom
audiences
● Data not quickly searchable and retrievable
in large quantity
● Experienced inconsistencies and downtime
5 AEROSPIKE USER SUMMIT | Proprietary & Confidential | All rights reserved. © 2018 Aerospike Inc
Speed
High latencies in read/write
throughput, including full text
search
Challenges Deep-Dive
Predictability & Stability
Downtimes and inconsistencies,
leading to unexpected results
Scale
Limited and restricted to only
few audiences to be generated
Challenge 1 Challenge 2 Challenge 3
6 AEROSPIKE USER SUMMIT | Proprietary & Confidential | All rights reserved. © 2018 Aerospike Inc
Solution
BMW i8 Is Best Selling Hybrid Sports Car In The World
Toyota Camry best selling car in US 2010
SQL
7 AEROSPIKE USER SUMMIT | Proprietary & Confidential | All rights reserved. © 2018 Aerospike Inc
1. Broadened capabilities and helped support our company’s exponential growth.
2. Boosted Profiling/Audience Lookalikes algorithms to over 5X the speed
3. Enabled us to serve more customers in a shorter amount of time with even leaner hardware resources
Solution
Aerospike database clusters
8 AEROSPIKE USER SUMMIT | Proprietary & Confidential | All rights reserved. © 2018 Aerospike Inc
Implementation & Architecture
Datasource 1
Datasource 1
Datasource 1
ETL PipelineMD Documents
Aerospike
Replicate
SlaveMaster
Search EngineSlave
Replicate
Search EngineMaster
IndexIndex
Index
IndexIndex
Index
Platform
API
MD Documents
Upload
Interval Indexing
Search with ICP criterias
Query with docID’s
Retrieve docID’s
3rd Party
9 AEROSPIKE USER SUMMIT | Proprietary & Confidential | All rights reserved. © 2018 Aerospike Inc
1) Production ready - 1 main cluster of 3 nodes + 1 replica
2) 3 different environments (dev, stage, production)
3) Auto-backup
4) Auto-logging
5) Aerospike Management Console
DevOps Cost: $0
Configuration
10 AEROSPIKE USER SUMMIT | Proprietary & Confidential | All rights reserved. © 2018 Aerospike Inc
● Handle significantly higher QPS: at least 5 times faster
● Current storage up to 500GB of data with monthly fluctuation
● Uptime for Aerospike at 99.99%
Performance
11 AEROSPIKE USER SUMMIT | Proprietary & Confidential | All rights reserved. © 2018 Aerospike Inc
Platform
Spring Data works seamlessly
with our current platform
Integration - Making the Switch
Encryption
The flexibility of encryption
(entire disk or singular field of a
document)
ETL
Python library enables ETL layer
12 AEROSPIKE USER SUMMIT | Proprietary & Confidential | All rights reserved. © 2018 Aerospike Inc
● Integrate with more 3rd party datasources
● Generate even more Lookalikes and ICP’s for our growing customer base
● Move future campaign experiments and optimizations to Aerospike
Future Approach
13 AEROSPIKE USER SUMMIT | Proprietary & Confidential | All rights reserved. © 2018 Aerospike Inc
Higher Speed, Greater Load, More Reliable
5X
Speed
top related