Accelerating the adoption of Cloud Computing Are We Done Yet ? Testing Your OpenStack Deployment OpenStack Summit, Paris, France November 4, 2014
Jul 15, 2015
Accelerating the adoption of Cloud Computing
Are We Done Yet ? Testing Your OpenStack Deployment
OpenStack Summit, Paris, France November 4, 2014
© Solinea, Inc. 2
Who Am I ?
Ken Pepple is the Chief Technology Officer of Solinea Prior to founding Solinea, he led the introduction of Internap's OpenStack-based public cloud services while serving as their Director of Cloud Development Author of O'Reilly "Deploying OpenStack” book OpenStack code contributor since Bexar release
© Solinea, Inc.
!
Who we are
3
Solinea is the leading software and services company that delivers integrated OpenStack™ solutions, enabling enterprise adoption of Cloud Computing
Cloud is the only domain we focus on, with vertical industry and horizontal solutions specialization
Purpose-built for cloud
Track record of success architecting, building and operating production clouds – private and public – world-wide
Proven Delivery Success
We understand cloud adoption challenges of global enterprise companies
Enterprise IT Experience
Pre-configured management and monitoring platform for OpenStack™ and other Open Infrastructure components
Differentiating IP
Built the first OpenStack production clouds and contributors to the platform since its inception
OpenStack™ Experience
© Solinea, Inc.
Agenda ! Problem statement ! Types of testing ! Testing tools ! Future improvements
4
© Solinea, Inc.
Problem ! Many components, even more connections ! No two clouds are identical
– Hypervisor choices, network choices, storage choices
! Many rarely used code paths (backups ?) ! Operational and integration concerns ! Tools for operators ! Requires skills across all the services
5
© Solinea, Inc.
In a nutshell … distributed systems
OpenStack Object Store
OpenStack Image Service OpenStack Compute
OpenStackDashboard
OpenStack Identity Service
OpenStack Compute API /
Admin API
keystone(service & admin APIs)
nova-api(OS, EC2, Metadata, Admin)
nova-consoleauth
nova-cert/objectstore
nova-consolenova-*proxy
VNC/Spice/RDP
OpenStack Object API
http://www.solinea.comOpenStack Data Processing
sahara-api
sahara-engine saharadatabase
Queue sahara-conductor
OpenStack Data Processing API
(8386)
OpenStack Identity
API
OpenStack Data Processing API
Queue
nova-compute
nova-scheduler
novadatabase
OpenStackCompute API
(8774, 8775, 8776)
OpenStack Image API
Horizon
OpenStack Image API
(9292)
identity backend
swift-proxy
objectcontaineraccount
objectstore
accountDB
containerDB
OpenStack Object API
HTTP(S)
OpenStackObject API
OpenStack Identity API
OpenStack Identity
API
OpenStack Identity
API
OpenStackImage API
OpenStack Identity
API
OpenStack Image API
catalog backendtoken backend
OpenStack Identity
API(5000)
hypervisor
libvirt, XenAPI, etc.
HTTP(S)
Amazon Web Services
EC2 API
Internet / Enterprise Network
OpenStack Network Service
glance-api
glancedatabase
OpenStack Block Storage
OpenStack Block Storage API
cinder-api
cinder-volume
neutron-server
neutron plugin(s)
OpenStack Identity
API
cinder-scheduler
cinderdatabase
OpenStack Network API
(9696)
networkprovider
OpenStack Block
StorageAPI
OpenStack Network API
policy backend
Queue
OpenStack Network API
neutrondatabase
neutron agent(s)
nova-conductor
memcached
OpenStack Identity
API
⁃ OpenStack Command Line Tools (nova-client, swift-client, etc.)⁃ Cloud Management Tools (CloudForms, Enstratius, etc.)⁃ GUI tools (Web Browsers, Cyberduck, iPhone client, etc.)
volume provider
cinder-backup
OpenStack Object API
OpenStack Identity
API
Queue
OpenStack Block Storage API
OpenStack Orchestration
heat-api
heat-engine
heatdatabase
Queue
cloudwatch-api
OpenStack Orchestration API
OpenStack Identity
API
OpenStack Compute API
OpenStack Bock Storage API
OpenStack Network API
glance-registry
OpenStack Metering Service
ceilometer-api
ceilometerdatabase
ceilometer-collector
OpenStack Database
trove-api
trove-taskmgr
trovedatabase
trove-conductor
OpenStack Database API
(8779)
OpenStack Block Storage APIOpenStack Orchestration API
OpenStack Compute API
Database(mysql, redis, mongo,
cassandra, etc.)
Queue
trove-guestagent
OpenStack Identity
API
6
© Solinea, Inc.
What Should We Be Testing? ! Functionality
– User scenarios – Not code quality … unless running trunk
! Non-functional requirements – Scalability – Performance – Security
! Operations – High availability and failover
7
© Solinea, Inc.
How Should We Test ? ! Automated
– Repeatability – Multiple checks
! Continual ! Ops not specialists
– Should need coders or QA people – Testing should be an easy command
! Integrated into operational procedures – “Let me send a canary in …”
8
© Solinea, Inc.
Types of Testing ! Unit testing ! Integration ! User Acceptance Tests ! Operational
9
© Solinea, Inc.
Functional Areas ! Networking ! Storage ! Compute ! User functional process ! OUTSIDE applications …
– Billing/metering, user signup, audit/compliance …
10
© Solinea, Inc.
Tempest ! OpenStack testing project ! Runs hundreds of tests
– Unit – Scenario – Stress – API – CLI
! Complex to configure ! Not portable to every cloud deployment configuration ! Requires significant investment in time
12
© Solinea, Inc.
An OpenStack project to create a framework for validating, performance testing and benchmarking OpenStack at scale with Tempest https://wiki.openstack.org/wiki/Rally
13
Rally
© Solinea, Inc.
Rally Scenario $ rally task start summit/create-and-list-users.json================================================================================Task 57bc6d22-76c2-4450-b834-daa48c41d6e0 is started--------------------------------------------------------------------------------
================================================================================Task 57bc6d22-76c2-4450-b834-daa48c41d6e0 is finished.--------------------------------------------------------------------------------
test scenario KeystoneBasic.create_and_list_usersargs position 0args values:{u'args': {u'name_length': 10}, u'runner': {u'concurrency': 10, u'times': 100, u'type': u'constant'}}+----------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+| action | min (sec) | avg (sec) | max (sec) | 90 percentile | 95 percentile | success | count |+----------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+| keystone.create_user | 1.767 | 3.478 | 5.944 | 4.565 | 4.694 | 100.0% | 100 || keystone.list_users | 0.077 | 0.464 | 1.516 | 1.01 | 1.143 | 100.0% | 100 || total | 1.982 | 3.942 | 6.218 | 4.956 | 5.214 | 100.0% | 100 |+----------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+Whole scenario time without context preparation: 40.6584420204
14
© Solinea, Inc.
Rally Database ! Rally stores all scenario/task runs ! Can be used to trend and benchmark results $ rally task list+--------------------------------------+----------------------------+----------+--------+-----+| uuid | created_at | status | failed | tag |+--------------------------------------+----------------------------+----------+--------+-----+| 57bc6d22-76c2-4450-b834-daa48c41d6e0 | 2014-11-05 12:10:38.235211 | finished | False | || 12d34393-0520-4830-9516-e7404935c0e9 | 2014-11-05 12:32:39.632810 | finished | False | || 3b160652-ed3b-4e2b-b09c-14b6d900e18e | 2014-11-05 12:40:21.544483 | running | False | |+--------------------------------------+----------------------------+----------+--------+-----+
15
© Solinea, Inc.
Rally Scenarios { "KeystoneBasic.create_delete_user": [ { "args": { "name_length": 10 }, "runner": { "type": "constant", "times": 100, "concurrency": 10 } } ]}
16
Method
Customization
Concurrency
© Solinea, Inc.
Rally Scenarios ! Authenticate ! Ceilometer ! Cinder ! Designate ! Glance ! Heat ! Keystone
! Neutron ! Nova ! quotas ! Requests ! Sahara ! Tempest ! VM ! Zaqar
17
© Solinea, Inc.
Operational HA Tests ! Kill process / hung process ! Lost interface ! Lost machine ! Lost rack
20
© Solinea, Inc.
Continual Testing ! Testing should be part of monitoring and
management ! “Order Canary”
– Chao monkey may not be appropriate but “happy path” test should be
– Define rally scenarios or custom scripts to feed monitoring tools
! Subset of tests that affect business critical functions
21
© Solinea, Inc. 22
Putting it all together
OpenStack Cloud Rally [os-admin]
OS API commands Monitoring [goldstone]
© Solinea, Inc.
Best Practices ! ABT (Always be testing) ! Manual testing " operational procedure
– Failover regularly as part of normal operations – HA is an event, it is SOP
! Testing is part of the deployment, monitoring and upgrade process
23
© Solinea, Inc.
Future ! /info APIs for all services
– Describe deployment beyond services
! Simplified DSL to configure tempest ! Monitoring product incorporate testing
25