OpenStack Private Cloud at eBay 1 OpenStack Design Summit – Nov 2013
OpenStack Private Cloud at eBay
1
OpenStack Design Summit – Nov 2013
Marc Carges - CTO
John Donahoe- CEO
Devin Wenig – President eBay NA
Bob Swan - CFO
Analyst Conference 2013
QA DEV
Prod
PCI
QA DEV
DEV QA
Secure
Prod
4
5/1/2013 6/1/2013 7/1/2013 8/1/2013 9/1/2013 10/1/2013
Users Projects
5/1/13 6/1/13 7/1/13 8/1/13 9/1/13 10/1/13
VMs Created VMs Deleted
5/1/2013 6/1/2013 7/1/2013 8/1/2013 9/1/2013 10/1/2013
Cores
Frictionless on-demand access
Multi-tenant, isolated
One cloud on shared infra
billmelater.com
brands4friends.de
corp.ebay.com
ebay-kleinanzeigen.de
ebay.ca
ebay.com
ebay.de
gumtree.com
hunch.com
marketplats.ne
milo.com
5
Globally Shared infra (Compute, Network, Storage)
VPC VPC
project project … project project …
Availability Zone AZ
VPC
project …
Availability Zone
VPC
project …
VPC
project …
Region Region
6
Virtual Networks Block, Object Storage Flavors classes
Large scale networks Multi tiers storage Commodity H/W
VPC
Class of Service (PCI, DEV, PROD, …)
OpenStack APIs
Project -> VPC Network -> VPC Dashboard VPC selection
Capacity based Network selection
Managed images
Win/Linux
Images -> VPC
Dashboard AZ selection
7
OpenStack is not cloud
hDp://www.subbu.org/blog/2013/07/openstack-‐is-‐not-‐cloud
OpenStack APIs
Network design
Net/info sec
Builds
CI
Infra onboarding
Config management
High availability
Log processing
Metrics
Cloud monitoring
AlerRng
Incident resoluRon User
experience
Customer support
SLA
Upgrades
Capacity planning
Scale out
RemediaRon
Autoscaling
Metering and
chargeback Monitoring and alerRng
Cloud is a service!
9
Monitoring
10
nova boot -‐-‐image a4a3d552-‐4f92-‐4426-‐82d9-‐ff2226d72d08 -‐-‐flavor medium myvm
Is the cloud broken?
User
You ?
11
RabbitMQ Capacity issues
Glance down
Didn’t get DHCP in Ame for cloud init
Metadata Aming out
Quantum plugin failures
vm_state=ACTIVE does not really mean success
12
Log source logstash ØMQ
ØMQ
ØMQ
ØMQ
ElasRcSearch
ElasRcSearch
ElasRcSearch
Log source logstash ØMQ
Log source logstash ØMQ
Log source logstash ØMQ
1 Grok paDerns 3 A cluster that can
scale horizontally
kibana
logstash
logstash
logstash
2 Low latency
13
StatsD
Log source logstash ØMQ
ØMQ
ØMQ
ØMQ
ElasRcSearch
ElasRcSearch
ElasRcSearch
Log source logstash ØMQ
Log source logstash ØMQ
Log source logstash ØMQ
kibana
logstash
logstash
logstash
Graphite
Zabbix
Metrics Monitoring Alerts
14
StatsD
Log source logstash ØMQ
ØMQ
ØMQ
ØMQ
ElasRcSearch
ElasRcSearch
ElasRcSearch
Log source logstash ØMQ
Log source logstash ØMQ
Log source logstash ØMQ
kibana
logstash
logstash
logstash
Graphite
Zabbix OpenStack nodes, processes, network,
storage, …
Is the cloud broken? May be?
15
StatsD
Log source logstash ØMQ
ØMQ
ØMQ
ØMQ
ElasRcSearch
ElasRcSearch
ElasRcSearch
Log source logstash ØMQ
Log source logstash ØMQ
Log source logstash ØMQ
kibana
logstash
logstash
logstash
Graphite
Zabbix OpenStack nodes, processes, network,
storage, …
StackMetrics
StackWatch
Simulate KPIs
• Create, delete • Volumes • Ping success • Ping latency • …
hDp://www.subbu.org/blog/2013/10/monitoring-‐and-‐alerRng-‐for-‐openstack
16
Scale out
17
Capacity
Availability
One Cloud User Experience Availability for Business
1. Nova “availability zones”
2. Keystone “regions”
3. Nova “cells”
18
Region A Availability
Zone
Availability Zone
Availability Zone
Region B Availability
Zone
Availability Zone
AWS
nova API
cinder API
neutron API
swift API
Block storage backend
network driver + agents + infra
Hypervisors
storage
keystone
glance Availability Zone
19
Dashboard Keystone
Nova
Glance
Quantum Cinder Swift
region services
az services
infra
Region BDashboard Keystone
Nova
Glance
Quantum Cinder Swift
region services
az services
infra
Region C
GlanceDashboard Keystone
Nova Quantum Cinder Swift
region services
az services
infra
Region A
Dashboard behind a global load balancer VIP
user chosen replication of selected resources
Region ≈ A fully decoupled OpenStack deployment
Availability Zone ≈ Keystone “region”
Nova availability zone? Nova cells – an implementaRon detail
20
Build and Deployment
21
Ubuntu, apt, fabric
Ubuntu, apt, puppet and foreman
1. Bare-‐metal provisioning for hypervisors, controllers, NVP, …
2. Virt env tarballs, puppet and foreman 3. A full working deployment!
CaDle test clouds Patches ParRal upgrades Rollbacks
Pet test clouds
1. Change orchestraRon 2. Foreman UI “ClickOps”
22
• Puppet automaRon • Full topology with virtual environments • ProducRon ready
• Zabbix templates • StackWatch and StackMetrics • Contribute on VPCs • Contribute on scale-‐out paDerns
Our Commitment to the Community
JC MarRn
@jchmarRn
23
Subbu Allamaraju @sallamar