Optimizing the Migration of Virtual Computers

7/30/2019 Optimizing the Migration of Virtual Computers

1/36

1

Optimizing the Migrationof Virtual Computers

C.P. Sapuntzakis, R. Chandra, B. Pfaff,J. Chow, M. Lam, and M. Rosenblum

OSDI 2002


2/36

2

Main Idea

To quickly move the state (state in disks,memory, CPU registers, and I/O devices) of arunning computer across a network.

They employ techniques to reduce the data overthe network.

20 minutes over 384 Kbps network withoutinstalling the application or even the underlyingoperating system


3/36

3

Benefits

Ease System Administration: Distribute acapsule with new changes

Ease User Mobility: Instantaneous access toactive environments everywhere

Backups: Save snapshots of capsules as backups

Other possible benefits: Recovery, Fail-over,Balance loads


4/36

4

Platform used

x86 computers as they are common, cheap, runmost of the commonly used software and havetools for migration.

VMware GSX server 2.0.1 running on Red HatLinux 7.3 (kernel 2.4.18-10).


5/36

5

Related Work

Disco, Denali, VMware GSX server

Rauch et al. Distribute disk images from centralrepository

Emulab, Cluster-on-demand also use disk images Work by Schmidt introduces the term capsule

Process level migration work - Condor, libckpt,CoCheck

Object-level systems - Legion, Emerald, Rover

LBFS (helped in HCP and hash cache design)


6/36

6

Virtual Machine Monitors

Layer of software on top of hardware thatexports a virtual machine abstraction

Encapsulates all of the necessary machine state

Requires no cooperation from the softwarerunning on the monitor

Provides the same device interface to the virtualmachine regardless of the underlying hardware


7/36

7

Collective

Capsule-based system architecture - Collective

Capsule: Is the hardware state including entireoperating system and applications, and runningprocesses and data.

Optimization techniques are invisible to the user

and require no modifications to the operatingsystem.


8/36

8

Technical Mechanisms (brief)

Ballooning :- Zeroes unused memory

Copy-on-write (COW) disks: Track just theupdates to capsule disks.

Demand Paging : fetches only needed blocks.

Hash-Based Compression: Hashed Copy (HCP)

avoids resending of existing data


9/36

9

Ballooning

Similar to VMWare ESX Server (Waldspurger)

run a balloon program

asks the OS for a large number of pages then zeroes those pages

more compression smaller capsule, fastercapsule start-up

but may release pages that will be usedmore demand paging, slower initial execution


10/36

10

The Collective: COW

A capsule hierarchy

root - root disk (a complete capsule disk)

nodes - COW disks

Capsule disk = a path from root to COW disk

Latest disk = COW disk at end of the path

cant write to disks in interior of hierarchy can write only to disks at leaf nodes


11/36

11

Capsule Hierarchy


12/36

12

Capsule Disks

COW Implementation bitmap file

sequence of extent files

Disk Writewrite to latest disk

update bitmap

Disk Read search bitmaps of COW

disks from the bottom up


13/36

13

COW Disk Chains

Every time an active capsule is suspended andresumed, a new COW disk is created

This can create long COW disk chains

Solution: Promotion

Add to the disk all of its parents blocks that are not

present in its own

Then can move the disk up one level in the hierarchy(the child becomes a sibling of its parent)


14/36

14

shim library

A layer of virtualization that sits between theVM and the C library

Intercepts the VMs I/O requests to plain disk

Redirects them to a local disk server

Translates them to COW disk requests

Executes I/O operations on COW disks COW disks can be locally or remotely stored


15/36

15

Demand Paging

Fetch disk data onlywhen needed

Keep a local shadowCOW disk for eachremote COW disk

contains all the locally

cached blocks of theremote COW disk


16/36

16

Capsule Execution

Transfer the memory image and COW diskbitmaps (if do not already exist locally)

Create a new latest COW disk locally

For each remote COW disk, create thecorresponding shadow COW disk (if does notalready exist locally)

Invoke the VM on the capsule


17/36

17

HCP (Hashed Copy)

Inspired by LBFS and rsync

Dont send any data blocks that the receiver

already has

Hashed Copy algorithm:

Send a hash of the data blocks

Receiver has data block that hashes to same valuecopy local version of block

Else, get the data block from the sender


18/36

18

HCP Hash Cache

Persistent cache

Primed cache:

The cache reads ahead in the remote file

This is used to avoid multiple disk accesses whentwo files contain runs of similar blocks


19/36

19

Evaluation Methods

Migration to propagate SW updates

Effectiveness and Interaction- Ballooning- Demand Paging- Hash Compression

Superbowl XXXIX: Active vs. Inactive Capsules

Simulation of user migration


20/36

20

Migration to propagate SW updates

Operating Systems Upgrade

Software Installation

Software Upgrade

Measuring: HCP


21/36

21

Operating Systems Upgrade

Installation of new Red Hat 7.3 capsule:= 1.6 GB

Savings incurred from Hashing:= 30% redundancy

Savings with gzip:

= 25% of the 1.6 GB capsule


22/36

22

Operating Systems Upgrade 2

Installation of new Red Hat 7.2 capsule:= 1.6 GB

Savings incurred from Hashing:

= 30% redundancy = 70% remaining= 1.15 GB remaining

Savings with gzip:= 25% of the 1.6 GB capsule

= 400 MB remaining = good job gzip!!!- (gzip + HCP)???


23/36

SW installation

Difference of HCP transfer of COW disk and size(installed packages)


24/36

600 KB/10 KB = 60 times larger

2500 KB/100 KB = 25x

1 MB/ .9 MB 1x


25/36

SW updates

HCP transfer of COW disksize(upgraded packages)


26/36

SW installation

HCP transfer of COW disksize(upgraded packages)

Not much of a difference


27/36

27

Migration of Active Capsules:

User Mobility

Measuring effectiveness and interaction ofballooning, demand paging and HCP.

Systems:- Windows: measure interactive processes- Linux: measure CPU + I/O bound processes

Speeds:- DSL: 384 KB/sec- LAN: 100 MB/sec


28/36

28

Results

Windows: interactive processes Local

- interactive: user 5 minutes (but cant compare)- CPU (compilation): 45 seconds

LAN- interactive part: 1.3 minutes vs. ???- CPU: 1.8 minutes

DSL- interactive part: 4.4 minutes vs. ???- CPU: 7 minutes


29/36

29

Results

Linux: CPU and I/O bound

VMware, ran locally- smttls, processor simulation: 3 minutes- gcc,: kernel compile: 4 minutes- Apache: 1 minute, but what task did they do?

vs. DSL and LAN

- no slower than local execution


30/36

Execution: disk data:

Demand paging

Migration: memory

image:Ballooning + HCP


31/36

31

Active vs. Inactive Capsules

Inactive

Mount over NFS

COW: Empty Cache

COW: Primed Cache

COW: Locally Cached

Active

Unballooned

Ballooned

Unballooned, Primed

Ballooned, Primed

Measure: best use of capsules


32/36

manual


33/36

33

Simulation of user migration

Measure: Ballooning, HCP, and gzip

Winstone user interaction12 snapshots during replay on Active capsule

|data in COW| amount of disk traffic


34/36


35/36

35

Conclusions

Over DSL, capsules can be migrated in 20minutes or less

The Collective facilitates administration

Amount of data transferred is reduced

use gzip


36/36

36

Remaining Questions

Promotion = User Data Loss?

Upgrades to immutable capsules = User DataLoss?

an abstraction that provided a private

namespace to a group of processes that can bemigrated as a unit

Optimizing the Migration of Virtual Computers

Documents