Scenario-based Distributed Virtualization Management Architecture

Scenario-based Distributed Virtualization Management Architecture for Multi-Host Environments

F. Galán (Telefónica I+D), D. Fernández (UPM), M. Ferrer (UPM), Fco. J. Martín (UPM)

DMTF 2nd Workshop on Systems and Virtualization Management (SVM 2008)Munich, 21-22 October 2008

© 2008 Telefónica Investigación y Desarrollo, S.A. Unipersonal

2TELEFÓNICA I+D


01 Introduction

02 EDIV Design

- General architecture

- Deployment controller

- Operation

03 Practical Results

04 Alignment with DMTF’s Standards

- CIM-based virtual management

- OVF alignment

05 Summary

Index

3TELEFÓNICA I+D


Introduction (1/3)Scenario-based virtualization management principles

Scenario-based virtualization management tools

— Scenario specification can be defined as a format representation (e.g., XML) of a set of virtual machines along with their interconnections in a given topology.

— Scenario-based management consist in processing scenario specifications in order to be deployed in physical infrastructure (hosts) and provide further management (e.g. monitoring, execute command sequences, undeploy, etc.).

Some tools nowadays are scenario-based, focused on experimentation environments, such as research and educational testbeds

— VNUML, NetKit, MLN

Conventional virtualization management tools (e.g. VMware VirtualCenter) for production environments use not to be scenario-based (out of our scope)

01

4TELEFÓNICA I+D


Introduction (2/3)Scenario-based management example: VNUML

01

Scenario-based tool(e.g. VNUML)

Desired scenario (designed by the

user)

Physical host

VMs and interconnection virtual networks

Scenario specificati

on(in XML)

<vnuml>

</vnuml>

5TELEFÓNICA I+D


Introduction (3/3)Problem Statement

Scenario-based virtualization tools use to consider just one physical host deployment

— Scalability problems (e.g. a 1000 VM scenario)

We have designed and implemented a distributed virtualization management tool named EDIV (Distributed Scenarios with VNUML, in Spanish) which purpose is to deploy scenarios in a cluster of N physical hosts.

— Based on VNUML (in other words, VNUML scenarios can be used in EDIV without modifications)

— Modular and extensible partition algorithm

— A partnership project between Telefónica I+D and Universidad Politécnica de Madrid (Spain) which prototype results are publicly available (GPL).

01

6TELEFÓNICA I+D


01 Introduction

02 EDIV Design



- Operation




- OVF alignment

05 Summary

Index

7TELEFÓNICA I+D


EDIV General Architecture02

Switch-based interconnection backplane(supporting 802.1q VLAN)

…

Deployment hosts (N)

VLAN802.1qtrunk

Deployment controller

managementnetwork

VNUML VNUMLVNUML

Deployment hosts

— GNU/Linux systems with VNUML installed along with other utilities (vconfig, sshd, etc.)

— VNUML provide three operations: deploy, execution and undeploy

Switch-based interconnection backplane

— One or serveral Ethernet switches providing end-to-end 802.1q VLAN trunks

— A management network (physical of VLAN) must be implemented

8TELEFÓNICA I+D


EDIV Deployment Controller02

VNUMLVNUML

VNUML

Coordinator

Switches

Deployment hosts

DB


U

DS

H

W

SegmentatorSegmentatorSegmentator

Scenario specification (VNUML language) and

operation (deploy,execution or undeploy)

(ssh/scp)

(SQL)(vendor CLI)

(Perl module)

<vnuml></

vnuml>

managementnetwork

9TELEFÓNICA I+D


EDIV OperationDeployment

02

Coor Seg H1 H2 SW

SPLIT_REQ(SC, param)segmentator algorithm maps VMs in SC tohosts (generating a mapping table)

split SC XML in two fragment (SC1, SC2) as specified in

mappingCREATE_VLAN_SUBIF (…)RUN_VNUML(‘deploy’,SC1)

CREATE_VLAN_SUBIF (…)RUN_VNUML(‘deploy’,SC2)

ENABLE_VLAN (…)

get pre-deployment information

RESULT(‘OK’)

wait all responses

store SC1 y SC2 associated with the distributed scenario name (eg., ‘tutorial’)store VLAN IDs allocated to the scenario

SPLIT_RSP(mapping)

OPER(‘deploy’,SC)

10TELEFÓNICA I+D


01 Introduction

02 EDIV Design



- Operation




- OVF alignment

05 Summary

Index

11TELEFÓNICA I+D


Practical ResultsExperiment Setup

03

H1 H2 H3

Switch

eth1

eth1eth1

eth0 eth0 eth0


Large scenario (51 VMs)Management network

Small scenario (6 VMs)

EDIV tool implemented in Perl Physical management network Partition algorithms

— Round robin— Weighted round robin (based on CPU load)— Deployment constraints (“VM A and B in the same

physical host”)

12TELEFÓNICA I+D


Practical ResultsVirtual machine distribution

03

Deployment host loads

Round-RobinWeighted

Round Robin

L1 L2 L3 H1 H2 H3 H1 H2 H3

Small scenari

o (6 VMs)

L L L 2 2 2 2 2 2

M M M 2 2 2 2 2 2

H L L 2 2 2 1 3 2

H M M 2 2 2 2 2 2

Large scenario (51 VMs)

L L L 17 17 17 17 17 17

M M M 17 17 17 17 17 17

H L L 17 17 17 7 22 22

H M M 17 17 17 13 19 19L (low) 0% load M (medium) 50% load H (high) 100% load

13TELEFÓNICA I+D


Practical ResultsTime comparison

How much does it take to deploy each scenario?

— With VNUML (reference time)

– Small scenario: 42 s

– Large scenario: 282 s

With EDIV (L/L/L, low mean load)

– Small scenario: 17 s

– Large scenario: 130 s

03

EDIV can achieve a 50-60% saving regarding

the mono-host case with VNUML

14TELEFÓNICA I+D


01 Introduction

02 EDIV Design



- Operation




- OVF alignment

05 Summary

Index

15TELEFÓNICA I+D


DMTF’s Standards AlignmentCIM-based virtual machine management

04

Deploymentcontroller

VNUML VNUML-based deployment host

(UML virtual machines)

U

H

CIMVirt. Model

WBEMServer

Provider(e.g libvirt-cim)

CIM-baseddeployment host

(CIM-manageable VMs: Xen, KVM, OpenVZ, etc.)

H’

(CIM-XML orCIM-SOAP)

CMPI, Java, …

(ssh/scp)

Scenario specification (VNUML language) and

operation (deploy, execution or undeploy)

<vnuml></

vnuml>

16TELEFÓNICA I+D


DMTF’s Standards Alignment OVF-based virtual machines in EDIV scenarios

Open Virtualization Format (OVF)— Recently published DSP0243 1.0.0d (September 2008)

Why introduce OVF in EDIV?— To use virtual appliances highly optimized for specific purposes in EDIV

scenarios (e.g., firewalls, dynamic routing stacks, etc.)

— To smooth the evolution of virtual machines from development/pre-production environments to production

How to do it?

04

Deploymentcontroller

U H/H’

(ssh/scp or CIM-XML orCIM-SOAP)

<vnuml></

vnuml>

OVF-complaintdeployer

physical hostsOVF descriptors in

scenario specifications

OVF-complaintdeployerOVF-complaint

deployerOVF-compliantdeployer

17TELEFÓNICA I+D


Summary

Main conclusions— EDIV architecture solves the problem of scenario-based

virtualization management in distributed multi-host environment (as checked by ours experiments)– Scalability

– Dramatic reduction (50-60% saving) in deployment time

— Simple and open API to develop new partition algorithms

— Transparency to VNUML users (the specification language is the same)

Work in progress— Wide-area deployment environment (instead of a local cluster)

– This is being working out in PASITO (a distributed experimentation platform within RedIris, the Spanish National Research and Education Network)

Future working lines— Alignment with DMTF’s standards

— Virtual machine images management

05


EDIV tool and documentation isfreely available at

http://www.dit.upm.es/vnuml

19TELEFÓNICA I+D


OperationCommand sequences execution

02

Coor Seg H1 H2 SW

OPER(‘exec’, cmd, ’tutorial’)

get XML1 and XML2 (which were associated to ‘tutorial’ at deployment time)

RUN_VNUML(‘exec’,cmd,XML1) RUN_VNUML(‘exec’,cmd,XML2)

RESULT(‘OK’)

wait all responses

20TELEFÓNICA I+D


OperationUndeployment

02

Coor H1 H2 SW

OPER(‘udeploy’, ‘tutorial’) RUN_VNUML(‘undeploy’,SC1)

DESTROY_VLAN_SUBIF (…)RUN_VNUML(‘udeploy’,SC2)DESTROY_VLAN_SUBIF (…)

DISABLE_VLAN (…)

remove SC1 y SC2 release VLAN IDs used by the scenario

RESULT(‘OK’)

get SC1 and SC2 and VLAN IDs (which were associated to ‘tutorial’ at deployment time)

wait all responses

21TELEFÓNICA I+D


Interconnection backplane

brA brA Deployment controller

host1 host2Net0 Net1 Net2

vm1vm2

vm3

vm5

vm4

.2

.1

.3

10.0.0.0/24 10.0.1.0/24 10.0.2.0/24

.1 .1.2

.2

<vnuml>

</vnuml>

Net1 VLAN (e.g, ID=120)

Segmentation— {vm1,vm2,vm3} → host1

— {vm4,vm5} → host2

— Inter-host networks: Net1 (vm3-vm4)

Net2vm5

vm4

vm3

Net0

vm1

vm2

vm5

vm4Net2

vm2

vm3

vm1

Net0

Environment configuration— VLAN creation (e.g., ID 120)

Environment configuration— Inter-host network bridge creation

Per-host VNUML execution

OperationExample

02

22TELEFÓNICA I+D


Segmentation: {vm1,vm2,vm3} → host1, {vm4,vm5} → host2

<vnuml> <net name="Net0" /> <net name="Net1" external="brA" /> <vm name="vm1"> <if id="1" net="Net0"><ipv4>10.0.0.1</ipv4></if> </vm> <vm name="vm2"> <if id="1" net="Net0"><ipv4>10.0.0.2</ipv4></if> </vm> <vm name="vm3"> <if id="1" net="Net0"><ipv4>10.0.0.3</ipv4></if> <if id="2" net="Net1"><ipv4>10.0.1.1</ipv4></if> </vm></vnuml>

Host1Sub-Spec

<vnuml> <net name="Net1" external="brA" /> <net name="Net2" /> <vm name="vm4"> <if id="1" net="Net1"><ipv4>10.0.1.2</ipv4></if> <if id="2" net="Net2"><ipv4>10.0.2.1</ipv4></if> </vm> <vm name="vm5"> <if id="1" net="Net2"><ipv4>10.0.2.2</ipv4></if> </vm></vnuml>

Host2Sub-Spec

<vnuml> <net name="Net0" /> <net name="Net1" /> <net name="Net2" /> <vm name="vm1"> <if id="1" net="Net0"><ipv4>10.0.0.1</ipv4></if> </vm> <vm name="vm2"> <if id="1" net="Net0"><ipv4>10.0.0.2</ipv4></if> </vm> <vm name="vm3"> <if id="1" net="Net0"><ipv4>10.0.0.3</ipv4></if> <if id="2" net="Net1"><ipv4>10.0.1.1</ipv4></if> </vm> <vm name="vm4"> <if id="1" net="Net1"><ipv4>10.0.1.2</ipv4></if> <if id="2" net="Net2"><ipv4>10.0.2.1</ipv4></if> </vm> <vm name="vm5"> <if id="1" net="Net2"><ipv4>10.0.2.2</ipv4></if> </vm></vnuml>

ScenarioSpec

Net1 is inter-host, becauseit connect vms in host1 (vm3)and host 2 (vm4)

OperationVNUML Specification Segmentation

02

Scenario-based Distributed Virtualization Management Architecture

Technology

scenariobased tool

vnuml desired scenario

scenario split

vm scenario

ediv deployment controller

vms small scenario

ediv operation deployment

s large scenario