Exploring Magnum and Senlin integration for autoscaling containers

Exploring Magnum and Senlin integration 1

Exploring Magnum and Senlin integration for autoscaling containers

Hongbin Lu (Huawei) Jay Lau, Ton Ngo (IBM)Qiming Teng, Julio Ruano (IBM)


ContributorsZilian Ji, Linjuan Xia, Shijia E, Peiyu Lin, Tianyuan Zhuang, Ji Jiang, Yongkui Wang


Talk Outline

1. Use cases2. Magnum3. Senlin4. How it works5. Demo


Use cases


Use case

• Shared COE cluster– Autoscale on some policies (utilization, etc.) – Scale number of nodes

• Multiple apps on cluster– Autoscale on some policies (request rate, etc.) – Scale number of containers

• Example: – 2, 3-tiered web apps: handle requests– SLA determines policy


Scaling scenarios (1)

node

c c c

node

c cc

node

c scale app

scale cluster

Single app on cluster• scale out containers as needed• scale cluster by utilization• coordination = utilization



node

c

node

c c

node

c c c c scale 2 apps

Two apps on cluster• scale out containers as needed• scale cluster by utilization• coordination = utilization

scale clusterunlimitedscale clusterlimited



node

c c

node

scale 2 apps

scale clusterlimited

c

node

c c c c c c

Two apps on cluster, limited cluster size• blue scales first, retains containers• red scales later, fails• maybe acceptable

• what if red is critical and blue is not• inverted priority: blue < red



node

c c scale 2 apps

Limit on apps• Place max on blue • blue scales, red does not scale• cannot use full resources


c

nodenode

c

limited



node

c c

node

scale 2 apps


c

node

c c c c c cc c c c

Priority on apps• blue scales first, red scales later • blue scales back to free up resource for red • 2 ways linkage: container cluster host cluster


Magnum


Instance

Magnum Overview

• Provision• Kubernetes• Docker Swarm• Mesos

• Scale• Add instances• Remove instances

• Manage• Pod• Service• Replication Controller• Container

Nova Instance

containercontainer

Magnum CLI

Magnum

Heat


Magnum Architecture


Magnum: Autoscale Bay

1. Raw data: Magnum pulls raw

data from Bays.

2. Metrics: Metrics, like memory

utilization, are computed and sent

them to Ceilometer.

3. Alarm: Ceilometer triggers alarms

based on user-specified scenario

and metrics.

4. Scale: Heat scales the bays.

Ceilometer

Magnum Heat

Bay1) Raw data

2) Metrics 3) Alarm

4) Scale


Kubernetes: Autoscale Pod

1. Metrics: The autoscaler

periodically queries pods’ metrics.

2. Scale: Based on collected

metrics, the autoscaler uses a

built-in algorithm to trigger the

scaling.

3. Resize: Replication Controller

resize the pods.

HorizontalPod

Autoscaler

Pods

1) Metrics 3) Resize

Replication Controller

2) Scale

16

In discussion• Magnum AutoScaler

– A new component in Magnum• Scaling Bays• Scaling Applications

– Interacts with AutoScaler from COEs• Kubernetes AutoScaler as a driver for container-level scaling• Magnum responsible for scaling RCs without Kubernetes autoscaler

– Customizability• data sources• metrics and thresholds• standby bay nodes (i.e. resource pool)

Exploring Magnum and Senlin integration


Two-Levels of AutoScaling

Working

K8S Bay

NodeNode

K8S Master

Heapster

Kublet

cAdvisor

Kublet

cAdvisor

Storage Backend

RC AutoScaler

Pod

Magnum AutoScaler Service

K8S Driver

Collector

Analyzer

Operator

Senlin

Magnum Conductor

Standby

Swarm Driver

Mesos Driver


Senlin

19

Clustering - A Missing Service in OpenStack


Scalable

Load-Balanced

Highly-Available

Manageable

......

of any (OpenStack) objects


Senlin Architecture

Senlin API

Senlin Client

REST RPC

Senlin EngineSenlin

Database

Policies

Profiles


Clustering/AutoScaling/LoadBalancing

Nova

Kubernetes

Heat

Ironic BareMetal

VMs

Stacks

Containers

Senlin

placement

deletion

scaling

health

load-balance

batching


Scaling A Cluster using Senlin

type: os.nova.serverversion: 1.0properties: image: fedora-21 flavor: 3 networks: - network: private userdata: ...

Profile

Cluster

type: senlin.policy.scalingversion: 1.0properties: event: CLUSTER_SCALE_IN adjustment: type: CHANGE_IN_CAPACITY number: 1 min_step: 1 best_effort: True

attach

Policy

type: senlin.policy.scalingversion: 1.0properties: event: CLUSTER_SCALE_OUT adjustment: . . .

attach

Policy

webhook

RESIZE


Policies

type: senlin.policy.scalingversion: 1.0properties: adjustment: min_step: 1 best_effort: true number: 1 type: CHANGE_IN_CAPACITY event: CLUSTER_SCALE_OUT

• Declarative• Can be mapped to from a standard• Example: SUR_Cluster(Add k8s Minion / Add Spark worker node)


Triggertype: CeilometerThresholdAlarmversion: 1.0rule: meter_name: memory_util comparison_operator: gt threshold: 70 period: 60 evaluation_periods: 1 statistic: avg query: - field: resource_metadata.cluster op: eq value: SUR_Cluster

• Generic abstraction• alarms in OpenStack

monitoring services• e.g. Ceilometer Alarm


How it works


Design goals

1. Coordinate scaling at apps and cluster level• Control flows in both direction• Apps -> Cluster: utilization, other methods• Cluster -> Apps: priority, other methods

2. Integrate scaling support from COE if available• Kubernetes autoscaler

3. Policy driven• Easy to express, manage complex cases


template

Outer Stack

Autoscaling: OpenStack-Only Solution (Today)

27

AutoScalingGroup Stack

Heat

Ceilometer

AH

PUp PDn

AL

ASGS1 S2 S3update

ASG


Senlin

profile

Autoscaling: OpenStack-Only Solution (New)

28

Cluster

Ceilometer

AHPUp

PDnAL

SCALE_IN

SCALE_OUT

Node1 Node2 Node3

webhookspolicies triggers


Demo


Autoscaling with Kubernetes

30

VMs

K8S Bay

NodeNode

Heapster

Kublet

cAdvisor

Kublet

cAdvisor

Pod

Senlin

Magnum

profile

CeilometerWorkload

policy


Questions


Thank You

Exploring Magnum and Senlin integration for autoscaling containers

Software