Created by: Kevin Carter & Curtis Collicutt OS-Ansible- Deployment Deploying OpenStack with Ansible presentation > osad <<EOP 1
Created by: Kevin Carter & Curtis Collicutt
OS-Ansible-Deployment
Deploying OpenStack with Ansible
presentation > osad <<EOP
1
www.rackspace.com 2
Who am I?
Kevin Carter● Developer at the Rackspace Private Cloud● Open source activist● Cloud operations junkie● Python aficionado● Recovering rubyist● Beer lover● Soccer fanatic
www.rackspace.com 3
Who am I?
Curtis Collicutt● Lead OpenStack Engineer @ AURO● Information Security● Storage systems● How do computers even?● Films
4www.rackspace.com
@robertcathey
Cloud infra is hard. OpenStack is cloud infra. OpenStack is hard. This is news, apparently.
“”
www.rackspace.com 5
OSAD and what we’re about
● Deployer Experience● Vanilla OpenStack● Scalability● Stability
www.rackspace.com 6
Why are we here?
In late 2013, the Rackspace Private Cloud team set out to solve our common deployment, maintenance, scalability, and stability problems.
www.rackspace.com 7
Distribution packaging of OpenStack
● Out of date packages ● Out of band configuration● Packages include proprietary patches● Time to bug resolution is longer than it
should● Broken dependencies
www.rackspace.com 8
Available deployment tooling
● Maybe, sometimes, sorta, eventually “consistent”, kinda?● Upgrades difficult or impossible● Steep learning curve
www.rackspace.com 9
Legacy architecture does not scale
● Almost all deployment systems reference an architecture that suffers from the “controller 1 controller 2” model
● VIP failover for OpenStack supporting services bound to break and when it does it’ll break spectacularly!
www.rackspace.com 10
What we devised
A source-based installation of OpenStack, built within LXC containers, using a multi-master architecture orchestrated and deployed via Ansible.
www.rackspace.com 11
Why Ansible?
● Community engagement
● Orchestration
● Almost no code
● Low barrier to entry
● Crazy powerful, stupid simple
www.rackspace.com 12
Why containers?
● LXC ≈ More bare metal
● Compatible with many networking architectures
● Supports an LVM backend
● Stable
www.rackspace.com 13
What is OSAD?
OSAD == OpenStack Ansible Deployment
● Uses LXC containers to isolate components and services
● Deploys OpenStack from upstream sources
● Runs on Ubuntu 14.04
● Built for production
● No proprietary secrete sauce
○ But you could bolt on as much as you want
● Created following the KISS principle
www.rackspace.com 14
● All Ansible tasks and roles target multiple nodes, even if that number is
a multiple of one (1)
○ EVERYTHING is tagged!
● Process separation on infrastructure components (controller nodes)
○ Microservice-like, where it makes sense
OSAD architecture
www.rackspace.com 15
● Galera multi-master cluster
● RabbitMQ with mirrored queues and deterministic sorting of the master
queues
● “Cheese shop” index build for your environment stored within your
environment
OSAD infrastructure components
www.rackspace.com 16
● OSAD does not know about the “all in one” deployment
○ LXC enables the base system to deploy a multi-node cloud even
with only one physical node
○ An AIO in our gate job emulates a 32 node cloud
● Neutron with the Linux Bridge agent offer stability and supportability
○ Open vSwitch is feature-full but Linux Bridge “just works”™
OSAD scale
www.rackspace.com 17
Community project
● We support Juno and Icehouse but the code contains
Rackspace-isms
● Kilo is our first “community” release of OSAD
● 41 contributors presently in the project
○ Not all Rackers
www.rackspace.com 18
Community project
We take our role within the community seriously!
# Lines of change between Juno and Kilogit diff --stat juno kilo 1158 files changed, 39061 insertions(+), 81368 deletions(-)
www.rackspace.com 19
● Deployer experience: Ansible● Vanilla OpenStack: Source-based installation● Scalability: Built within LXC containers● Stability: Obviously!
OSAD and what we’re about
www.rackspace.com 20
OSAD configuration
● OSAD configuration is your window into inventory
○ lives in /etc/openstack_deploy
● Dynamic inventory generated via config
● Compatible with Ansible static inventory
● Execution made simple using the openstack-ansible wrapper.
www.rackspace.com 21
OSAD deployment
# Change to the playbooks directory
cd /opt/os-ansible-deployment/playbooks
# Open your favorite terminal multiplexer
tmux new -s osad-deployment
# Do all the things!
openstack-ansible setup-everything.yml
Go get coffee|food|beer, this will take a minute.
www.rackspace.com 22
What an OpenStack deployment looks like with OSAD
Diagram not built to scale.Derived from an All in One Installation.
www.rackspace.com 23
OSAD adding a compute node
# Execute run limited to the nova_compute group
openstack-ansible setup-everything.yml \
--limit nova_compute
compute_hosts: compute1: ip: 172.29.236.201 compute2: ip: 172.29.236.202 compute3: ip: 172.29.236.203 compute4: ip: 172.29.236.204 compute5: ip: 172.29.236.205
EDIT: /etc/openstack_deploy/openstack_user_config.yml
www.rackspace.com 24
OSAD adding an infrastructure node
# Execute the setup with a limit on the infra groups we’re adding
openstack-ansible setup-everything.yml \
--limit os-infra_all,\
shared-infra_all,\
identity_all
shared-infra_hosts: infra1: ip: 172.29.236.101
os-infra_hosts: infra1: ip: 172.29.236.101
identity_hosts: infra1: ip: 172.29.236.101
EDIT: /etc/openstack_deploy/openstack_user_config.yml
www.rackspace.com 25
OSAD reconfiguring all of neutron
# Execute a run limited to neutron_all
openstack-ansible setup-everything.yml \
--limit neutron_all
global_overrides: provider_networks:
- network:
container_bridge: "br-
vxlan"
container_type: "veth"
container_interface:
"eth10"
ip_from_q: "tunnel"
type: "vxlan"
range: "1:1000"
net_name: "vxlan"
group_binds:
-
neutron_linuxbridge_agent
EDIT: /etc/openstack_deploy/conf.d/neutron_networks.yml
www.rackspace.com 26
● AURO - Public OpenStack Cloud● Compute, Volume, Swift, Heat, Neutron● Canadian data residency, ownership● Vancouver region, Toronto up next
AURO - OpenStack
www.rackspace.com 27
● Not using as much as we’d like● Mostly the infrastructure components
○ Rabbit, Galera, Memcached, etc● Absolutely invaluable as an example● Will continue to bring in more OSAD components as we operate over
time● Team somewhat new to config mgmt
AURO & OSAD - What we are using
www.rackspace.com 28
● Great example of:o Using Ansibleo Deploying OpenStacko Testing - All in one, use of OpenStack infra
● Already supports Kilo● Packaging and deploying OpenStack (ie. not using OS packages -
Python Wheels very cool)● Segregation of services
AURO & OSAD - What we like
www.rackspace.com 29
● Public cloud● Midonet● Different HA Model● Billing● Support Model
○ Multiple tiers of internal support
AURO - Differences from OSAD
www.rackspace.com 30
● Not to restart services in same run as changeso Need to control restarts in HA manner, rolling
● Every task tagged● Continuously run (from Ansible Tower and/or Jenkins)● Installing once is easy, operating forever is hard● Ansible to help manage many small changes faster● People don’t ssh into servers, only Ansible
AURO - Ansible Guiding Principles
www.rackspace.com 31
● Easy to use mostly idempotent modules then run a command or shell
task and make a mess of it● changed_when: False is too easy to stumble with● Multiple environments● Being able to run one-time commands across all systems is as powerful
as it is dangerous
$ ansible -a reboot all
AURO - Ansible Struggles
www.rackspace.com 32
● Deploy OpenStack from source
● Segregation of services● More monitoring● Ansible callback plugins are useful● Learn more from OpenStack testing infra● Need a couple modules
o Midoneto Swift
AURO - Near term improvements
www.rackspace.com 33
● Be “Pluggable?” (What does that even mean?)o Neutron network - eg. Midoneto HA model - eg. ECMP/BGP load balancing
● Balancing community roles and playbooks with custom
requirements● Learn how to consume OSAD properly
AURO - OSAD Comments/Ideas/Questions
www.rackspace.com
● Secrets (eg. Hashicorp Vault, KeyWhiz)● Continuous integration...err integration● Caching (Ansible has Redis, other ideas?)● What is the “future” of config mgmt? Must be more than just
pkg/config/start/bootstrap● Change request workflow
34
AURO - Configuration Management Future
www.rackspace.com 35
● Increase community participation in OSAD
○ Community members wanted!
○ Pull requests welcome :)
● Build out the operational modules found within the upstream
● Modular Dynamic inventory
● etc . . .
Where does Ansible and OpenStack go from here?
www.rackspace.com 36
Q & A
Twitter: @cloudnullIRC: cloudnull
Email: [email protected]
Twitter: @ccollicuttIRC: serverascode
Email: [email protected]
37www.rackspace.com
OSAD makes OpenStack™ NOSAD“”
www.rackspace.com
Thank You!
EOP
38