How to Develop for OpenStack APIs

Developing for Openstack APIs

Yoram Weinreb, CTO OfficeRan Ziv, Cloudify R&DGigaspaces

Download the deck

Agenda- Introduction to Openstack API- Exposing the APIs in an Orchestrator- Openstack API Quirks and Pitfalls- Testing with Openstack API

Introduction to OpenStack API

How do we interact with OpenStack?- Send Actions/Commands- Collect information

Interact with OpenStack

Openstack Endpoints RESTful API

Horizon Openstack CLI Tools

Openstack Official Python Client Libraries

SDKs for other Languages

RESTful endpoints

- API Versioning- For Example Identity ver. 1, 2, 3…- Interface

- Public- Internal- Admin

- Format (JSON, XML)

The northbound interfaces

Some Endpoint examples

- Identity API v3 - Compute API v2.1- Image service API v2- Block Storage API v2- Networking API v2.0- Object Storage API v1

- Bare Metal API v1- Orchestration API v1- Telemetry API v2- Clustering API v1- Data Processing v1.1- Database Service API- Shared File Systems API v2

Debugging OpenStack API

- CLI --debug- Python SDK debug params- VM Logs on horizon & CLI- Access to Openstack logs /

DevStack

Debugging OpenStack API

Demo time...

Debugging using Python SDK

Another option to turn on logging is with environment variables:

os.environ['NEUTRONCLIENT_DEBUG'] = 'true'

Viewing VM logs

- VM Logs on horizon- VM logs on CLI nova console-log [--length <length>] <server>

- VM logs in Python client: server.get_console_output()

Viewing OpenStack logs- Most logs are located at

/var/log/<service name>

- Devstack:[[local|localrc]]

DEST=/opt/stack/

LOGFILE=$DEST/stack.sh.log

SCREEN_LOGDIR=$DEST/logs/screen

Exposing OpenStack APIs in Cloudify


- Cloudify is an open-source, pure-play orchestrator

- Based on the TOSCA open standard- Describe your application components, their

lifecycle, and their relationships to one another- Use either YAML format or the Cloudify Composer

- Supports both hybrid and multi cloud environments- Open architecture allows for extending Cloudify to

work on any environment through plugins


- Cloudify supports Openstack via the Openstack plugin

- Strives to be unopinionated, support any use-case

- Offers simplicity and ease-of-use, yet not compromise on flexibility

- Robust at dealing with cloud errors

- Makes abstractions common to other environments where possible

- Resources are exposed via types defined by the plugin

- Prominent resource parameters are exposed explicitly- Provides sensible default values, syntactic

sugaring, etc.

- Other parameters are configurable via direct override

- Parameters for Openstack clients used are also configurable



- An example:

my_subnet_node:

type: cloudify.openstack.nodes.Subnet

properties:

resource_id: my-subnet

cidr: 1.2.3.4/24

dns_nameservers: [8.8.8.8]

Subnet: # direct override

enable_dhcp: false


- A Server example- Keypair, SGs, networks/ports may be configured

directly or, preferably, via relationships

my_server_node:

type: cloudify.openstack.nodes.KeyPair

properties:

resource_id: my-server

image: Ubuntu-14.04 # translated to ID

flavor: x3.medium # translated to ID

- Some server operations take a while to complete

- VMs take long to boot (reach “ACTIVE”)- Sometimes boot in errored state, or have

connectivity issues- Cloudinit startup- retrieve a password from the metadata service- sshd service to start up

- An orchestrator needs to be able to handle asynchronous operations



Volumes considerations:

- Volumes may also require asynchronous handling

- Volumes often require a specific set of operations to become usable

- format- mkfs- mount


Security-groups considerations:

- In Openstack, a default SG always exists and will be attached automatically to a new VM which doesn’t declare otherwise

- Orchestrator must create SGs prior to the VM and declare them at boot time

- When creating a new SG, it’s instantiated with default, permissive egress rules- disable_default_egress_rules: false


- Can override Openstack clients configuration- Configurable on multiple scopes (deployment, node,

operation) to achieve both ease of use and flexibility

my_floating_ip_node:

type: cloudify.openstack.nodes.FloatingIP

properties:

openstack_config:

nova:

http_log_debug: true

neutron:

endpoint_url: https://1.2.3.4:9696

OpenStack APIQuirks and Pitfalls

OpenStack API Quirks and Pitfalls

- Some of the APIs aren’t necessarily intuitive

- Some due to historic and legacy reasons; some have been or will be declared as bugs

- The orchestrator can help with handling these quirks and making things more intuitive where possible

- A network may contain more than a single subnet

- When creating a server (or a port), required parameter is “network_id” - subnet is chosen arbitrarily

- To place a server on a specific subnet, must first create a port and supply it with the “fixed_ips” param

{'name': 'my-port',

'network_id': '4c6d….aba9',

'fixed_ips': [{'subnet_id': '4e70...f2a6'}]}


- Unlike the rest of the resource types, Keypairs are managed on a per-user basis, not per-tenant

- Can lead to funny behavior:- Breaks isolation between tenants - resource

cleanup on one tenant may affect another

- Heat stack created by one user may not be destroyable by another user on same tenant

- Keypair quotas are set per tenant but apply per user


- In Openstack, floating IPs are first allocated and then attached to a server or a port

- There’s no validation when attempting to attach a floating IP which is already attached to a resource

- The floating IP will simply detach from the previous resource

- race-condition scenarios- checking an IP is allocated doesn’t necessarily

mean it’s safe to attach it to a resource


- The Nova API for adding a SG to a server is not thread-safe (launchpad bug)

- An orchestrator that runs concurrent operations should verify the addition after making the API call

- Neutron API for adding a SG to a port is thread-safe- But not concurrency-friendly as it requires stating

all SGs rather than only the newly added one


https://bugs.launchpad.net/nova/+bug/1417975

- SG rules are defined over a port or a range of ports- ICMP rules have a “type” and a “code” but no port

association- Neutron’s security-groups API translates:

“port_range_max” → “code”

“port_range_min” → “type”

- E.g. - Allowing Ping from anywhere:

{'protocol': 'icmp',

'port_range_min': 0, # Ping’s ICMP Type

'port_range_max': 0, # Ping’s ICMP Code

'remote_ip_prefix': '0.0.0.0/24',

…}


- Nova’s API for adding a security-group to a server may receive either a security-group’s name or ID

- If the security-group was created using Nova (i.e. it’s a Nova-net security-group), passing the ID will fail

- Poses a problem when there are multiple security-groups with the same name

- Poses a problem for orchestrator - the code for a separate resource (Server) needs to be aware of the SG’s type


- Ports can be created either explicitly or implicitly (e.g. by connecting a server to a network

- Explicit ports should have to be deleted explicitly

- Up until Kilo release, deleting a server which was connected to a port that had been created explicitly would’ve deleted the port as well

- From an orchestrator’s POV, this is an abstraction breach, as one resource affects the lifecycle of another


https://bugs.launchpad.net/nova/+bug/1158684

- Keystone roles are assigned per tenant

- However, a user who’s assigned an admin role on one tenant becomes an admin across all tenants

- Should stick to setting admin users only within an “Admin” project

- Admin users can manage resources on any tenant- May actually see resources of tenant A even

while using tenant B


Testing with OpenStack API

- Unit tests- mock library- custom mocks- Mimic project

- Integration tests- Test the plugin’s operations against real

Openstack deployments, of different versions

- System tests- End-to-end tests for the plugin- The plugin is used to run many other

of Cloudify’s end-to-end tests


https://github.com/rackerlabs/mimic

https://github.com/rackerlabs/mimic


- Testing environments:- Integration tests and system tests run in parallel

over multiple Openstack tenants- Pretty good isolation- Easy clean-up

- Os-purge

- The tenants can be pre-existing- Tenant deletion requires clean-up anyway- Allows setting up an environment for

multiple tests to run serially- Take resources snapshot before and

after each test - clean-up delta


https://github.com/openstack/ospurge

https://github.com/openstack/ospurge

- Testing environments:- Tenants for tests are set across multiple

Openstack deployments- Clouds have problems:

- VMs starting up in erroneous states- Connectivity issues- Maintenance and downtime- ...

Relying on a single environment is risky

- Possibly tests multiple Openstackversions / distributions


- Testing environments:- When cleaning up a tenant’s resources after a

test, keypairs make an exception- since they’re per-user rather than per-tenant

- Each test cleans up any Keypairs it created on its own, whether it succeeds or fails

- An independent process is run when tests are inactive to clean up any Keypair leftovers from tests that’ve been stopped abruptly


- Testing multiple Openstack versions:- Devstack

git clone git://git.openstack.org/openstack-dev/devstack -b <release branch name>

- Ravello- A double virtualized environment in the

cloud.- Define Ravello blueprints that will start and

stop isolated Openstack clouds in minutes


Questions?

Thanks!Come visit us at booth #C20!


- A more advanced example:

my_subnet_node:

type: cloudify.openstack.nodes.Subnet

properties:

resource_id: my-subnet

cidr: 1.2.3.4/24

relationships:

- target: my_network_node

type: cloudify.relationships.contained_in

How to Develop for OpenStack APIs

Technology