High Performance NoSQL Database From Development to Deployment with Docker & Aerospike Powering New Opportunities at Scale
High Performance NoSQL Database
From Development to Deployment with Docker & Aerospike Powering New Opportunities at Scale
10x LOWER TCO10X Faster and 10X Fewer REAL WORLD CUSTOMER EXAMPLE
The Bottom Line
3 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Existing Architectures Are Broken
Challenges • Complex
• Maintainability
• Durability
• Consistency
• Scalability
• Cost ($)
• Data Lag
Caching Layer
Operational Database
Real-time Consumer Facing
Pricing / Inventory/Billing
Real-time Decisioning
Streaming Data
Legacy Database (Mainframe)
RDBMS Database
Transactional Systems
Enterprise Environment
Legacy RDBMS HDFS BASED
4 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Flash Optimized High Performance OTHER DATABASE
OS FILE SYSTEM
PAGE CACHE
BLOCK INTERFACE
SSD HDD
OTHER DATABASE
AEROSPIKE FLASH OPTIMIZED IN-MEMORY DATABASE
BLOCK INTERFACE
SSD SSD
OPEN NVM
SSD
AEROSPIKE
HYBRID MEMORY SYSTEM™
• Direct device access • Large Block Writes • Indexes in DRAM • Highly Parallelized • Log-structured FS “copy-on-write” • Fast restart with shared memory
6 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Use Cases
8 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
AdTech – Real-Time Bidding
Challenge • Low read latency (milliseconds) • 100K to 5M operations/second • Ensure 100% uptime • Provide global data replication
Performance achieved • 1 to 6 billion cookies tracked • 5.0M auctions per second • 100ms ad rendering, 50ms real-time bidding,
1ms database access • 1.5KB median object size
Selected Aerospike NoSQL over competition • 10X fewer nodes • 10X better TCO • 20X better read latency • High throughput at low latency
Ads is Displayed
Publishers
Ad Networks & SSPs
Ad Exchanges
Demand Side Platform
Data Management Platforms
Brands Agencies Buyers
0 ms 100 ms
10 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Challenge • Overall SLA 750 ms • Loss of business due to latency • Every credit card transaction requires hundreds of DB reads/writes
Need to Scale Reliably • 10 à 100 TB • 10B à 100 B objects • 200k à I Million+ TPS
Selected Aerospike In-Memory NoSQL • Built for Flash • Predictable low latency at high throughput • Immediate consistency, no data loss • Cross data center (XDR) support • 20 server cluster • Dell 730xd w/ 4NVMe SSDs
Credit Card Processing System
Fraud Detection & Protection App
Rules Rule 1 Rule 2 Rule 3
Historical Data
Rule 1-Passed Rule 2-Passed Rule 3-Failed
Account Behavior
Static Data
Account Statistics
Real-time Fraud Prevention
Containers
13 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Container Mission – Reduce Complexity
Build Ship Run
OpenStandards
Plumbing
Pla3ormClustering Distribution
Image spec Container run-time spec
RunC
Notary
14 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Coalition of industry leaders join forces to eliminate fragmentation • Form a vendor-neutral, open source governance model under the
Linux Foundation • Establish common standards for container format and runtime • Docker donated its container format, runtime and associated
specifications • Appoint maintainers for the libcontainer project
Open Container Initiative (OCI)
http://www.opencontainers.org/
Aerospike and Containers
16 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
■ Shared nothing architecture ■ Automatic hashing of keys across the cluster ■ RIPEMD-160 collision free algorithm with Smart Partitions™
■ Automatic healing & rebalancing of the cluster ■ Automated cluster discovery ■ Smart Client™ ■ Java, C/C++, C#, Python, Node.js…
■ Automatic replication of data across nodes
Why Aerospike and Containers?
17 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
■ Scale (up and out) the persistence layer ■ Eliminate reconfiguration of the application and database tier ■ Utilize Containers on your dedicated infrastructure or cloud
Aerospike enables
18 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
■ Build & Run an App in Development ■ Python + Aerospike
■ Deploy to a Swarm cluster in Production ■ Scale Web services
■ Scale Aerospike Cluster in production
Demo: Development through to Production
19 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Lets build an App!
web
Aerospike
Development
python / flask
20 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Scale in Production
web2 web3 web4 webN web web1
Aerospike
Development
…
Production
HA Proxy
asd1 asdN asd2 …
21 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Demo 1 : Build an App
22 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Roll the App to Production behind HA Proxy
web web1
Development Production
Aerospike
HA Proxy
Aerospike
23 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Scale the web tier
web2 web3 web4 webN web web1
Aerospike
Development
…
Production
Aerospike
HA Proxy
24 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Docker Networking
25 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Demo 2 : Scale the Web Tier
26 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Scale the Aerospike cluster
26
web2 web3 web4 webN web web1
mongodb
Development
…
Production
HA Proxy
asd1 asdN asd2 …
27 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Demo 3 : Scale the Cluster
28 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Inside ■ Encapsulation of Concerns
Storage: Inside or outside the container?
Host
daemon
container
Host
daemon
container e.g. SSD
e.g. EBS
Outside • Separation of concerns • Storage Features (e.g. snapshots)
/data/db /mnt/xx:/data/db
/dev/xvdb
29 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
Summary
• Define Container, their contents and how they work together once • Deploy the same images in Dev, Pre-Prod and Production across
Platforms
One solution from Dev -> Production
• Ops define the whitelisted images, security policies etc. • Dev use approved images to build upon • Eliminate the complexity (and cost) of deployment
Running Docker & Aerospike in Production
30 Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
■ Code ■ http://github.com/alvinr/docker-demo/tree/master/aerospike
■ Docker Images ■ http://hub.docker.com/r/aerospike/
■ Contact me! ■ [email protected] ■ @jonnyeight
Thanks and Q&A