Deploy OpenStack with SaltStackqiniuppt.qiniudn.com/Shi Shanshi.pdf · Backend (Spice, OpenStack) Ctrip OpenStack ... • HP's OpenStack Helion: Debian all the way down - YouTube.

Deploy OpenStack with SaltStackSHI, SHANSHI

SYSTEMS ENGINEER, CTRIP

Ctrip Cloud Platform Engineering

Started in July, 2012

Infrastructure as a Service to Site Ops & Corp IT

Virtual Machines ( VMware, KVM)

Bare-Metal Nodes (Razor)

Virtual Desktop Infrastructure to Call Centers

Thin clients (GUI, SaltStack)

Backend (Spice, OpenStack)

Ctrip OpenStack

Fleets of windows servers: pets, cattle?

Forked VMware driver: local storage, virtual switch

Naive Neutron without L3: just give me an IP

• OpenStack中的扩展--以Nova为例| Kong's Blog

• Odd Bits — Integrating custom code with Nova using hooks

• PLY Git-Based Patch Management

Ctrip OpenStack

Ubuntu 12.04 with Icehouse kernel/qemu/libvirt/ovs

https://code.launchpad.net/~ubuntu-server-dev

man deb-version[epoch:]upstream-version[-debian-revision]2:2014.1.3.2-0ctrip1

Docker for debian packaging instead of schrootaptly, packer

apt-cacher, flask-pypi-proxy

• HP's OpenStack Helion: Debian all the way down - YouTube

Decisions..

sources.list, keystone endpoints, nova.conf, …

Puppet, Chef, Salt, Ansible

Salt: remote execution { + configuration management }

Salt worked.

• Learning to Scale OpenStack_Juno Update from the Rackspace Public Cloud

• SaltConf14 - Anita Kuno, HP - Using SaltStack for event-driven orchestration of OpenStack infra - YouTube

How Salt Works

salt-master

1. salt ‘kevin’ cmd.run ‘apt-get update’

2. socket.recv{‘tgt’: ‘kevin’,‘fun’: ‘cmd.run’,‘arg’: ‘apt-get update’}

3. Encrypt the payload

4. Publish it via zmq

salt/master.py

salt-minion

1. socket.recv

2. Decrypt the payload

3. self.matcher: Am I the ‘tgt’?

4. self.functions: Loaded from salt/modules/*

5. self.functions[data[‘fun’]](data[‘arg’], …)

6. Return the result via zmq

salt/minion.py

Minion did not return

gru@jerry$ ps aux | grep apt

gru@jerry$ tail –f /var/log/salt/minion

When I run test.ping, why don't the Minions that aren't responding return anything? Returning False would be helpful.

/etc/salt/mastershow_timeout: True

Ubuntu 12.04: upgrade to the zmq from salt PPA

Job Management

topics/jobs/index.html

List active/historic jobs, query job result

{signal,term,kill}_job

• Overview of Puppet's Architecture

• The Architecture of Open Source Applications (Volume 2): Puppet

Salt States

SLS stands for SaLt State

SLS Formulas have historically been called "SLS files"

SLS files are therefore, in reality, just dictionaries

apt-key add

curl/wget/gpg

slow

firewall

bash here strings

salt ‘lary’ state.sls salt.repo test=true

apt-key add

1. salt ‘dave' cp.cache_file 'salt://salt/files/salt.key'/var/cache/salt/minion/files/base/salt/files/salt.key

2.

Renders

ref/renderers/index.html

salt/template.py:compile_template

render_pipe

old: yaml_jinja

new: jinja|yaml

jinja: templateyaml: format

Highstate

YAML renderer

salt --out=pprint '*' state.show_highstate

Salt Dafa

ref/modules/all/index.html

ref/states/all/index.html

http://jinja.pocoo.org/docs/dev/templates/

ref/states/index.html

ref/states/layers.html

ref/states/highstate.html

Loader

load_modules

os.listdir(mod_dir)

salt/{grains,pillar,modules,renderers,states,…}

Loader.gen_functions

mod.__grains__ = self.grains

mod.__pillar__ = self.pillar

mod.__salt__ = funcs

ref/states/vars.html

salt/loader.py

Develop Environment

mkvirtualenv salt

pip install –e .

~/.venvs/salt/etc/salt/{master,minion}

tmuxp load salt.yaml

openstack-formula

Static Grains

roles

openstack-controller (keystone, nova-conductor, neutron-dhcp-agent, …)

openstack-compute (nova-compute, neutron-ovs-agent)

openstack-data (mysql, rabbitmq)

openstack-slb (keepalived, haproxy)

vlan

204

500

CMDB?

OpenStack Configuration

Configuring OpenStack is nontrivial

topics/development/conventions/formulas.html

apache-formula, rabbitmq-formula

parameterization

repository structure

pillar -> map.jinja -> conf templates

Model -> Controller -> View

All config options must be “declared” in map.jinja

Pillar data can override map.jinja default values

Pillar Data

Jinja Magic

If the above macro doesn’t work for you,your pillar structure might be too complex

Conf Templates

OpenStack Releases

Upstream: grizzly-eol, stable/icehouse

In-house: cloudmgr, orca

openstack/neutron/dhcp-agent.slsopenstack/neutron/icehouse/dhcp-agent.slsopenstack/neutron/grizzly/dhcp-agent.sls

Salt: include, extend

Jinja: import, include, extends

git branch can always be created as a fallback

map.jinja override

openstack/neutron/map.jinjaopenstack/neutron/icehouse/map.jinja

1. Two Jinja contexts are serialized by the jinja renderer

2. Make sure you tell salt to override the parent one

Maintaining OpenStack configuration is also tedious

Reuse as much as possible

Task Decomposition

nova.conf should be managed by which sls?

Because salt states are dictionaries, dictionaries cannot have duplicate keys.

dpkg –S /etc/nova/nova.confapt-cache rdepends nova-common

Other nova formulas all include openstack/nova/common.sls

States Relationship

Jinja and YAML output dictionaries

Salt handles the relationship of the dictionary items

salt/modules/state.pysalt/state.py

A feature (bug): you can require an entire sls file, but the included file has to contain at least one state declaration.

salt tries to find states which are declared in the required sls file

Overstate

Databases are not necessarily installed on the same machine

nova-conductor should require database connection string, but not the database itself

Configure the data nodes first, then the controller nodes

ref/states/layers.html#overstate

States Tutorial, Part 5 - Orchestration with Salt

salt/runners/state.py:orchestrate

Stacker

Opinionated devstack replacement

First-time users should still start with devstack

Replace apt-get install with git clone

Replace service restart with tmux

Include/Extend conf templates

All Kinds of Salt

salt-ssh, fabric

Install salt-minion with salt-ssh

Changing minion environment requires a restart

Oops, minions connect to a wrong master

salt-call

topics/tutorials/standalone_minion.html

salt-call state.sls stacker.icehouse

Salt Environments

Configuring pillar is still painful

An army of reference pillar examples is needed

shanghai, nantong-prod, ssshi-dev2, …

Directory Overlay

/srv/salt/pillar/ssshi-dev2

/srv/salt/pillar/ssshi-base

Pillar: Get all controller nodes in the same environment via salt-mine, and generate haproxy pillar entries automatically/dynamically

States: Give me my bashrc/gitconfig/tmux.conf/vimrc

Bigger than Bigger

0-10, 10-100, 100-1000

gitfs backend

Pillar encryption for code review

Orchestration

Upstream openstack-formula

Testing formulas with docker

• LinkedIn - SaltStack for Web Scale – YouTube

• Google - Management at Google Scale – YouTube

• HP Cloud - Automating operations and support with SaltStack - YouTube

Summary

DSL is powerful, you just have to learn it

Salt SLS is a description language

Know the structure of the documentation

Dev + Ops + QA + Support = Problem Solvers

Learn from the best

Effective Operations, Effective Cloud

Thank U

Join us! [email protected]

DevOps Engineer, VDI Architect, Cloud Manager