eBay Object Storage at Scale - SNIA · 2019. 12. 21. · 22. Challenge 5: Solutions Swift/Keystone Handshake New Proxy Middleware. Keystone. Token (scoped with zone and IP) Swift

1

eBay Object Storage at Scale

Gilbert BouzeidLead Product Manager Data Storage

2

ebayToday

$22.6Bof GMV

14KGlobalEmployees

61% InternationalRevenue

182M Active Buyers

Q2 2019 numbers

3

Object Storage

4

What is Object Storage?

Object Storage is a data storage architecture that manages data as objects, as opposed to other storage architectures like file systems and block storage.

5

The Object Storage Journey at eBay

1. A need was identified

Seller HubeBay Site Objects Analytics / ElasticSearch Index Backup

6

The Object Storage Journey at eBay

2. Object Storage Solutions:

● 2007: EPS, eBay Picture Service● 2008: EMC Storage ATMOS● 2012: zStore● 2014: Swift

7

Why Swift Object Storage?

● Scalable and Reliable● Pure software based solution with REST API● Elastic● Replication● Multiple policies● Stateless Service● Runs on heterogeneous commodity hardware

8

Swift Object Storage High-Level Deployment Diagram

9

Swift Object Stats

2MContainers

5 PBStorage

2BObjects

30Gbps

25KRequests/

sec

900Accounts

10

Swift Object StorageUse Cases

Analytics DataDeployment PackageGeneric Images/Files storageLoggingPCI / PIIBackup / ArchivalData Movement (between Zones)TransactionLogging

SellerHub

ES Index

Backup

PCI / PII

Image Search

Zone Transfer

Deployment

ReturnShipping

Label

11

Swift Object Storage and Kubernetes

● ECR (eBay Container Registry) backed by Swift

● Leveraging the existing Swift Cluster from Kubernetes

● Swift Running on Kubernetes (Bring your own Swift)

● Creating Swift Accounts Seamless to Kubernetes end users

12

Object Storage:Challenges and Solutions

13

Challenge 1

● One Global Swift Object Storage (distributed) handling a Huge number of Use Cases!

● Large number of concurrent access● Infrastructure Limitations

Swift ProxySwift Nodes

LB

😐😐

14

Challenge 1: Solutions

● Customer Education/Best Practices● Split Swift Services● Efficient Use of Hardware● SLB

Swift Object

Swift Proxy

Swift Object

Swift Proxy

HTTPS TLS 1.2 Request

ProxyAccountContainer

SLB

Object role

SLB

CPU, Memory, SSD

Dense storage disks

15

Challenge 2

● Free offering● Burst number of Requests● Uncontrolled Usage● No Quota

16

Challenge 2: Solutions

● QoS is essential● Requests rate limit● Account quota

17

Challenge 3

● 100’s of Parallel Downloads from Same Account!

● Same Object too!● Limited Disk IOPS on the Object Nodes

18

Challenge 3: Solutions

● Object Chunking● Caching Solution

Swift Object

Swift Proxy

Swift Object

LB

VarnishCache

Swift Proxy

L7

SLB

19

Challenge 4

● Geo Availability● High Availability● Disaster Recovery (DR) Capability

20

Challenge 4: Solutions

● Multi-Regional Keystone

Swift ProxySwift Proxy

Swift ProxySwift Object

SLB

GTM / DNS

Swift ProxySwift Proxy

Swift ProxySwift Object

Swift ProxySwift Proxy

Swift ProxySwift Object

CacheCache CacheCacheCacheCache

DC1 DC2 DC3

SLB SLB

21

Challenge 5

● Multi-Tenancy● Zone Isolation

22

Challenge 5: Solutions

● Swift/Keystone Handshake● New Proxy Middleware

KeystoneToken (scoped with zone and IP)

Swift ProxySwift Proxy

Zone Isolation

IP Restriction

Modified token

ACL restriction

Middlewarepipeline

Swift ProxySwift Object

Project scoped with zone

LBIP Forwarding

23

Challenge 6

● Data Archival / Backup● Longer Retention● Low Cost● High Durability● Large Amount of Data

24

Challenge 6: Solutions

● Erasure Coding● Storage Policies● 175% less storage overhead● High PUTs, Low GETs

25

Object Storage:Alerting and Monitoring

26

Alerting and Monitoring

● Elasticsearch Kibana (ELK) & Prometheus● Synthetic Tests● Zabbix● Metrics

27

Alerting and Monitoring

● Stats

28

Alerting and Monitoring

● Metrics

29

Alerting and Monitoring

● Performance

30

Thank You

Q & A Session

eBay Object Storage at ScaleSlide Number 2Object StorageWhat is Object Storage?The Object Storage Journey at eBayThe Object Storage Journey at eBayWhy Swift Object Storage?Swift Object Storage High-Level Deployment DiagramSwift Object StatsSwift Object StorageUse CasesSwift Object Storage and KubernetesObject Storage:Challenges and SolutionsChallenge 1Challenge 1: SolutionsChallenge 2Challenge 2: SolutionsChallenge 3Challenge 3: SolutionsChallenge 4Challenge 4: SolutionsChallenge 5Challenge 5: SolutionsChallenge 6Challenge 6: SolutionsObject Storage:Alerting and MonitoringAlerting and MonitoringAlerting and MonitoringAlerting and MonitoringAlerting and MonitoringThank You

Q & A Session