OpenStack Trove in Production at eBay - Trove Day 2014

Session Name-click to change

August 19, 2014

Auston McReynolds, Software Engineer, eBayMat Lowery, Software Engineer, eBay

Trove in Production at eBay

OpenStack Trove Day

The eBay Inc. Portfolio

PLATFORM

8/19/14 tesora.com 2

Private Cloud at eBay

4

eBay Private Cloud: Like a Public Cloud

8/19/14 tesora.com

• One cloud on shared infra• Like a public cloud for all eBay Inc properties

• Multi-tenant, isolated• Different tenants with different needs

• Frictionless, on-demand access• Reduce time to market• Enable innovation

tesora.com 5

Architecture: Region vs. AZ

8/19/14

Dashboard behind global load balancer VIP

Region A

Swift

Regi

on

Svcs

AZ

Serv

icesKeystone Nova CinderNeutron

Glance Dashboard MySQL Misc

Region B

Swift

Regi

on

Svcs

AZ

Serv

icesKeystone Nova CinderNeutron

Glance Dashboard MySQL Misc

Region C

Swift

Regi

on

Svcs

AZ

Serv

icesKeystone Nova CinderNeutron

Glance Dashboard MySQL Misc

tesora.com 6

Customizations

8/19/14

• VPCs• Enforce obligations, restrictions, and capabilities• Associated with Keystone projects• OpenStack resources mapped to VPC (via metadata)• Image metadata• Host aggregate metadata

• Enforcement via custom middleware or scheduler

• Keystone• LDAP• Auto-tenancy

tesora.com 7

Customizations

8/19/14

• Horizon• Most use the custom Horizon dashboard vs. API• Connect To (uses image metadata)• Copy datastore root password• Disable UI functionality based on AZ capabilities or

Datastore• Multi-AZ tenant switcher glues the clouds together

8

Packaging and Deployment

8/19/14 tesora.com

• Deployment unit: virtualenv tarball• virtualenv --no-site-packages –relocatable• Simplifies rollback• Not perfect: doesn’t handle OS libs

• Rollout automation by Puppet, Foreman• Auditing of changes• Host overrides for testing in isolation

tesora.com 9

Monitoring

8/19/14

• Metrics• Periodic gathering based on service DBs• Share with end users• Alerts to engineers

• SLAs• Ensure being met via sampling; simulates cloud user• Nova boot• Swift upload and download

• Capture logs on failures• Alerts to engineers

• Graphite for graphing and Zabbix for alerting• Logstash/Elasticsearch/Kibana stack

Trove at eBay

tesora.com 11

Trove at eBay – The Beginning

8/19/14

• Mid-2013• 1 developer• Stack on Folsom• Not integrated, still Red Dwarf• MySQL only• Single region• No backups

tesora.com 12

Trove at eBay – Now

8/19/14

• Team (all ATCs)• Icehouse• Multiple Datastores• MySQL 5.5, 5.6• MongoDB 2.4• Cassandra 2.0• Couchbase 2.2• Redis 2.8

tesora.com 13

Trove at eBay – Now - Continued

8/19/14

• Multi-region + HA, with limited VPC availabilty

• Backups for MySQL, MongoDB, & Couchbase• Contributions to Trove:• Couchbase Impl + Backups• Multi-Datastore Horizon• Clusters (in-flight)• + many more (incl. 400+ code-reviews)

tesora.com 14

Deployment Details

8/19/14

• HA• Stable* release cuts• Puppet + Foreman for Control Plane• Fabric for Guests

• Shared infra, sans RabbitMQ• RabbitMQ HA Mirrored

tesora.com 15

Deployment Details

8/19/14

• KVM• Ubuntu• Network: Bridged or Overlays• Cinder Volumes• Venv’d Vanilla Guest Agent• Diskimage-Builder• w/ reference & custom elements• Image per Datastore

tesora.com 16

Deployment Details

8/19/14

• Designate (soon)• Zabbix• w/o Heat

tesora.com 17

Challenges

8/19/14

• With shared infra:• Nova quotas vs. Trove quotas• Horizon (compute duplicity, how to connect, etc.)• True ownership of spawned assets• Failure; who’s fault?• Scheduling

tesora.com 18

Challenges

8/19/14

• With Trove:• Upgrades/Rollouts• RabbitMQ• Feature support across Datastores• Multi-region Datastore deployments

tesora.com 19

What’s Next

8/19/14

• Clusters!• Advanced scheduling• Additional monitoring• Performance tuning• More management extensions (UI + API)• API-driven Datastore differentiation

eBay Trove Team is Hiring

http://linkd.in/1sSVbHv or [email protected]

Questions