OpenStack Resources and Capacity Management - Shimon Benattar, Mark Rasin - OpenStack Day Israel 2017

©2017 AT&T Intellectual Property. All rights reserved. AT&T and Globe logo are registered trademarks of AT&T Intellectual Property. All other marks are the property of their respective owners.

OpenStack Resources and Capacity ManagementUse case of AT&T’s Challenges Of Scaling OpenStack

5/6/2017

Shimon Benattar

Mark Rasin, AT&T



Agenda

• AT&T’s approach to OpenStack scaling

• Managing Resources across multiple OpenStacks

• OpenStack Capacity Management


Classic Multi-Region OpenStack Deployment

Keystone Authentication Flow

• DCP – Distributed control plain

• LCP – Local control plain


AIC - Share Nothing Architecture

• Every OpenStack resides in a single region.

• Cross site reference is not allowed

• Externally distribute and maintain every entity

• Scalable

• Easy Rolling upgrades

• Time to market by doing selective upgrades for certain sites where new features need to go immediately.

• Self contained, an issue with a particular site doesn’t affect other sites

CONCEPT

ADVANTAGES


The Operational Challenge

CentralizedManagement

Operational Nightmare

ORM


ORM Characteristics

• ORM is very light weight

→ Centralized management

→ Light local OpenStack agent – ORD

• ORM is the single endpoint to the outside world

• ORM provides a Region Discovery Service

• Loosely coupled, distributed and scalable model

→ The unit of instruction is a HEAT template

→ ORD simply runs these templates on each region

→ ORD then reports – asynchronously – back to ORM of the status of each such operation

• Manage and maintain Openstack resources

→ Tenant Quotas/Users/Roles

→ Nova Flavors

→ Glance public Images


Site 1Resource Distribution Flow

Site N

ORM

CMS Customer Management

Service

IMS Image Management

Service

FMS Flavor Management

Service

ACMS MuranoApp Catalog

Management Service

RDS - Resource Distribution Service (Customer,Flavor,Images,Catalog)

CLI

GUI

Region Management Service

ORM clients Central ORM

ORD Agent

Heat

Nova Murano

TemplateDocument Store

Keystone Glance

ORD Agent

Heat

Site 2ORD Agent

Heat

ORM

Database

HEAT


Site 1

Site N

CMS Customer Management

Service

IMS Image Management

Service

FMS Flavor Management

Service

ACMS MuranoApp Catalog

Management Service

RDS - Resource Distribution Service (Customer,Flavor,Images,Catalog)Region Management Service

ORD Agent

Heat

Nova Murano

Keystone Glance

Planned for the future

ORD Agent

Heat

Site 2ORD Agent

Heat

Database

HEAT

Resource Distribution Flow

ORM

CLI

GUI

ORM clients Central ORM

TemplateDocument Store


ORM Usage

• For systems: CLI and API

• For humans: CLI and GUI


OpenStack Capacity Management

IaaS systems like OpenStack provide an illusion of infinite capacity

In fact:

Cloud Provider needs to manage capacity

User needs for additional resources require capacity availability validation

Often users know their future usage pattern - information useful to the provider.

Need a Capacity Reservation Tool

Private cloud providers can impose processes to get higher predictability of the usage

Capacity management mechanisms are currently not supported by OpenStack.


Capacity Planning Tool

1. Presents current deployment, allocation, usage and reservation

2. Provide means to compare and analyze the information

3. Provide or integrate with existing reservation flows


Our Capacity

Amount of resources for various OpenStack components:

• Compute (Nova) CPU and RAM (with over-subscription)

• Block (Cinder) and Object (Swift) storage

• Networking (Neutron) capacity - most likely requires special handling

Dep

loye

d

Futu

re D

eplo

ymen

t p

lan


Allocated Quota and Actual Usage

• Quota can be over-allocated

• Actually used - a subset of quota and the deployed resources (over-subscribed)

Dep

loye

dActuallyUsed

Futu

re D

eplo

ymen

t p

lan


Resource Reservation

Policy: when is it allowed to allocate quota for future deployment?

Dep

loye

dActuallyUsed

Futu

re D

eplo

ymen

t p

lan


Dep

loye

d

Resource Reservation

• Resource reservation according to enterprise policy and approval process

• Once approved, the requested resource becomes reserved

• At some point this will be translated to increased quota (e.g. new tenant)

ActuallyUsed

Futu

re D

eplo

ymen

t p

lan

Pen

din

g re

serv

atio

n

req

ue

sts


Information Needed

• Current deployment status

• Future deployment plans

• Current allocation (quota)

• Actual use

• Reservations

• Future demand estimations

OpenStack

1. Current allocation (quota)

2. Actual use

3. Current deployment status

Enterprise data

1. Future deployment plans

2. Reservations

3. Future demand estimations

Data sources


The Alternatives

1. Commercially available systems

2. OpenStack Blazar project

3. OPNFV Promise project

4. Build your own


Use Commercially Available Systems


OpenStack Blazar

A Blazar user can request (“leased”) resources for his project for a defined duration

Virtual (Instances, Volumes, Networks) and Hardware(hosts with specific configuration) resources can be allocated via “lease”.

Blazar was created to:

• Manage cloud resources not only right now, but also in the future;

• Have dedicated recourses for a certain amount of time;

• Prepare for the peak loads and perform capacity planning;

• Optimize energy consumption.


OPNFV


Promise

A resource reservation and management project

• identifies NFV related requirements

• Realize resource reservation for future usage by capacity management

OPNFV - Promise


Build Your Own

• Requires time and resources

• Will best fit your process

• Will need to be integrated with other OSS/BSS and OpenStack

• Not an OpenSource - No gain from future development of other teams


Make Your Choice

Commerciallyavailable

OpenStackBlazar

OPNFVPromise

Build yourown

Cost $$$ $$ $$ $$$

TTM Short Medium Medium Long

GUI V X X V

Business needs specifics Partial X X V

Multiple OpenStacks support X X X V


Thank You

OpenStack Resources and Capacity Management - Shimon Benattar, Mark Rasin - OpenStack Day Israel 2017

Technology