Zero-copy Migration for Lightweight Software Rejuvenation of Virtualized Systems Kenichi Kourai Hiroki Ooba Kyushu Institute of Technology
Zero-copy Migrationfor Lightweight Software
Rejuvenationof Virtualized Systems
Kenichi KouraiHiroki Ooba
Kyushu Institute of Technology
Software Aging [Huang+ FTC'95]
Virtualized systems tend to suffer from software agingThe state of running software is degraded with
timeE.g., memory leakage
Hypervisors (and management VMs) are long-running software
Source: F. Machida et al., Combined Server Rejuvenation in a Virtualized Data Center, Proc. IEEE ATC 2012.
free memory free disk space
Software Rejuvenation [Huang+ FTC'95]
Restore systems to the normal stateProactive technique for counteracting software
agingSimplest method: system reboot
Cause a long downtime in virtualized systemsNeed to stop all VMs during the rebootViolate service level agreement (SLA)
aged hypervisor
VM VM...
Rejuvenation with VM MigrationReduce downtime during rejuvenation
Migrate all VMs to another hostThe downtime due to VM migration is usually
negligibleReboot only the aged hypervisor
No VMs on it
aged hypervisor
VM...
clean hypervisor
migration
source host destination host
VM VM...VM
Performance DegradationVM migration stresses hosts and network
largelyTransfer the memory images of VMs via network
Several hundreds of GB in totalEncrypted to prevent eavesdropping/tampering
Occupy CPUs and memory/network bandwidthsDegrade the performance of virtualized systems
Source: K. Kourai et al., Fast Software Rejuvenation of Virtual Machine Monitors, TDSC, 2011.
web throughputstart end
VMBeamEnable lightweight software rejuvenation
Start a new virtualized system at the same hostUsing nested virtualization
Migrate all VMs from an aged system onto a clean oneUsing zero-copy migration
Stop the aged system
aged hypervisor
VM... zero-copymigration
source virtualized system destination virtualized system
clean hypervisor
VM VM...VM
Nested VirtualizationEnable a virtualized system to run in a VM
Guest hypervisor/VMs inside a virtualized system
Host hypervisor/VMs in the outsideThe overhead is 6-8% [Ben-Yehuda+ OSDI'10]
1% in a special-purpose host hypervisor [Tan+ DCDV'12]
guest hypervisor
guestVM
guestVM
...
guest hypervisor
guestVM
guestVM
...
host hypervisor
host VM host VM
Zero-copy MigrationRelocate the memory of guest VMs
between virtualized systems at the same hostStep 1: Share the memory between src/dst
guest VMsThe src guest VM can continue to run
Step 2: Release the memory of the src guest VMAfter the entire memory is shared
clean guest hypervisor
host hypervisor
inter-guestmemory sharing
clonedguest VM
runningguest VM
aged guest hypervisor
destinationhost VM
sourcehost VM
No Memory Re-transferZero-copy migration is completed in one
iterationNot repeat to re-transfer modified memory
areasTraditional live migration needs multiple iterations
Modifications are directly reflected to a destination guest VM by memory sharing
Reduce the migration time for memory-intensive VMs
clean guest hypervisor
host hypervisor
aged guest hypervisor
nore-transfer
clonedguest VM
runningguest VM
destinationhost VM
sourcehost VM
Reducing System LoadsNo use of the virtual network
Shared memory is used
No copy of large memory images of VMsThe memory is simply relocated
No encryption of the memory imagesAny data is not exposed to the outside of guest
VMs
No need to detect memory write in guest VMsModifications are directly reflected
CPU
Net
MemCPU
CPU
CPU
Devirtualization [Lowell+ ASPLOS'04]
Remove the overhead of nested virtualizationDisable the host hypervisor during a normal runRe-virtualize the system only during
rejuvenationCons: the guest hypervisor could directly
corrupt the hardware state
guest hypervisor
guestVM
...
host hypervisor
host VM host VM
guest hypervisor
guestVM
...
host hypervisor
host VM
devirtualize
revirtualize
<normal run> <software rejuvenation>
Isn't the Host Hypervisor Aged?Yes, but the aging speed is slower
Much smaller than the guest hypervisor6K LOC (CloudVisor) vs. 300K LOC (Xen 4.2)
Execute no complex VM operationsDevirtualization can suppress aging
The host hypervisor is disabled
minimal host hypervisor
feature-richguest hypervisor
host VMguestVM
migration
ExperimentsWe confirmed the effectiveness of zero-
copy migration in VMBeamSystem loads, migration time, and downtime
ComparisonXen-Phys
Traditional system with two physical hostsXen-Blanket [Williams+ EuroSys'12]
System with nested virtualization and fast virtual network[2 hosts]CPU: Intel Xeon E5-2665Memory: 32 GBNIC: Gigabit Ethernet
host hypervisor: Xen 4.2host Dom0 OS: Linux 3.2.0guest Dom0 OS: Linux 3.5.0
System LoadsWe measured system loads during VM
migrationVMBeam did not transfer data via virtual
networkIt used only 30% of CPU time in Xen-PhysIt did not access the VM memory (estimated)
CPU0
5
10
15
20
25
30
Xen-Phys (src) Xen-Phys (dst) VMBeam Xen-Blanket
tim
e (
10
3 s
ec)
Network0
1
2
3
4
5
tran
sfe
r (G
B)
Memory0
10
20
30
40
50
60
access (
GB
)
Migration PerformanceWe measured the migration time and
downtimeThe migration time in VMBeam was up to 5.8x
fasterThe downtime in VMBeam was 0.2s longer
Due to the overhead of nested virtualization
0 1 2 3 40
30
60
90
120
150
Xen-Phys VMBeam Xen-Blanket
memory size (GB)
mig
rati
on
tim
e
(sec)
0 1 2 3 40
0.5
1
1.5
memory size (GB)
dow
nti
me (
sec)
16s
Related WorkMicrovisor [Lowell+ ASPLOS'04]
Maintain the system in a new VM and migrate applications to it
Focus on devirtualizationRDMA-based migration [Huang+ Cluster'07]
Only one copy by InfiniBandNeed 3 copies when encrypting the memory
imageWarm-VM Reboot [Kourai+ DSN'07]
Maintain VMs in memory during rejuvenationStill cause downtime during the hypervisor
reboot
ConclusionVMBeam for lightweight software
rejuvenation of virtualized systemsNested virtualization: Run aged and clean
systems at the same hostZero-copy migration: Migrate guest VMs
efficientlySuppress system loadsMake VM migration up to 5.8x faster
Future workDevelop a minimal host hypervisorEnable devirtualization in the host hypervisor