A Case for Virtualizing Nodes on Network Experimentation Testbeds

A Case for Virtualizing Nodes on A Case for Virtualizing Nodes on Network Experimentation Network Experimentation

TestbedsTestbeds

Konrad LorinczKonrad Lorincz

Harvard UniversityHarvard UniversityApril 22, 2023April 22, 2023

April 22, 2023 2

MotivationMotivationIn the beginning …. 1960s, 1970sIn the beginning …. 1960s, 1970s

hardware was expensive => commodity computer systems shared by multiple users

TodayToday hardware is relatively cheap connectivity and location are a commodity Network experimentation testbeds are expensive => shared

» ex: PlanetLab, Globus, Centurion

PlanetLabPlanetLab

April 22, 2023 3

Sharing ScenarioSharing Scenario Current mode of sharing Current mode of sharing

1.1. Each user has an accountEach user has an account on the testbed2. To perform experiment – user logs on one node, runs script

which starts jobs on a subset of the nodes

Unfortunate ScenariosUnfortunate Scenarios1. Inconsiderate user: Node N is performing critical

experimental computation for user A» User B runs his/her script that starts computationally intensive

job on node N2. Considerate user: Node N is running A’s experiment

» User B checks load on node N, determines it’s very low, starts a moderately computation intensive job on node N

April 22, 2023 4

Current Way to Solve ThisCurrent Way to Solve This In a Small LabIn a Small Lab

John posts a message on the bulletin board …

AnalysisAnalysis May work OK in a small lab => clearly does not scale Inefficient use of resources

Please don’t use machines X, Y, and Z between5/2/02 – 5/5/02. I am running my experiments.

Thanks,-John

PS. If you absolutely must log on one of these machines during this time, please don’t run any computationally intensive jobs.

April 22, 2023 5

Desired Sharing PropertiesDesired Sharing PropertiesNeed a good mechanism for sharing so that users don’t get in each other’s way

Ideal Scenario: Researcher A specifies to the scheduler that it needs N number of machines/nodes, for H number of hours, each with W amount of RAM, X amount of disk size, CPU equivalent to Y MHz, and bandwidth of Z Mbps.

Desired Properties of Mechanism for SharingDesired Properties of Mechanism for Sharing Efficient multiplexing of resources amongst users Control over resources

» Reservation of resources» Enforcement

Isolation» Security» Performance

Customization

April 22, 2023 6

SolutionSolutionClaimsClaims

1.1. Sharing problem can be solved by Sharing problem can be solved by => => virtualizing the testbed nodesvirtualizing the testbed nodes

2.2. Good way to virtualize nodes Good way to virtualize nodes => => use a VMMuse a VMM

April 22, 2023 7

Advantages of VirtualizationAdvantages of Virtualization Efficient Multiplexing of ResourcesEfficient Multiplexing of Resources

Takes advantage of Takes advantage of statistical multiplexingstatistical multiplexing (demand-based) (demand-based) Resources requested, if granted, are guaranteed for the Resources requested, if granted, are guaranteed for the

duration of the experimentduration of the experiment» Only need to request resources needed => may be very fewOnly need to request resources needed => may be very few

April 22, 2023 8

Advantages of VirtualizationAdvantages of Virtualization Control Over ResourcesControl Over Resources

Can be fine-grained and preciseCan be fine-grained and precise Allows for reservation/allocation of resourcesAllows for reservation/allocation of resources Enforcement of reservations and policiesEnforcement of reservations and policies Commercialization? – if it can be controlled and accounted Commercialization? – if it can be controlled and accounted

for => service could be soldfor => service could be sold

April 22, 2023 9

Advantages of VirtualizationAdvantages of Virtualization IsolationIsolation

SecuritySecurity» Almost perfect security from other VMs (i.e. users)Almost perfect security from other VMs (i.e. users)» Namespaces of each VM is private => cannot even name a Namespaces of each VM is private => cannot even name a

resource on another VMresource on another VM» Even if a VM is compromised, the rest are notEven if a VM is compromised, the rest are not» DownsideDownside::

Resources don’t span multiple protection domains Resources don’t span multiple protection domains => can’t share a common service across VMs=> can’t share a common service across VMs

Performance – aim for soft guaranteesPerformance – aim for soft guarantees» Actions of other users on the node don’t affect my availability to Actions of other users on the node don’t affect my availability to

resources (guarantees)resources (guarantees) If Bob overloads his VM, it won’t affect my VMIf Bob overloads his VM, it won’t affect my VM

April 22, 2023 10

Advantages of VirtualizationAdvantages of Virtualization CustomizationCustomization

User may ask for exact machine configuration he/she requires (RAM, disk size, CPU, network bandwidth)

May load different OSes or modified/customized ones» User may require a particular OS (FreeBSD)» Denali kernel can be loaded and run on » User has superuser privileges

Easy management» user configures once a customized VM image

=> transfers it to required nodes or other testbeds

April 22, 2023 11

Arguments Against VirtualizationArguments Against Virtualization Overhead of Running a VMMOverhead of Running a VMM

Relatively small! VMware server, Denali + others report about 10%10%

Hardware is getting faster => overhead will be even less of a Hardware is getting faster => overhead will be even less of a factorfactor

April 22, 2023 12

Arguments Against VirtualizationArguments Against Virtualization Use Dedicated Physical MachinesUse Dedicated Physical Machines

Reservation too-coarse grained» must dedicate an entire physical machine to a user => may

require the equivalent of 1/8 of the machine More expensive

» More expensive to buy 6-12 machines then one more powerful one + VMM

"We're running 20 virtual machines on one four-way system, and it's handling everything from CRM applications and security to application development and testing, all of which has saved us huge amounts of time and money in hardware costs."

-- Alan ThomasSenior Technical Consultant, National Gypsum Company

April 22, 2023 13

Arguments Against VirtualizationArguments Against Virtualization Do/Implement the sharing at the OS levelDo/Implement the sharing at the OS level

VMMs are much simpler than conventional OSesVMMs are much simpler than conventional OSes Other mechanisms at the OS level impose particular software Other mechanisms at the OS level impose particular software

interfaces or models to the user => with a VMM it’s clear, you get a interfaces or models to the user => with a VMM it’s clear, you get a “raw” machine“raw” machine

Cannot provide strong performance isolation => the need to support Cannot provide strong performance isolation => the need to support high-level abstractions prevents OSes from providing strong high-level abstractions prevents OSes from providing strong performance guaranteesperformance guarantees» Precise resource accounting is difficult because resources Precise resource accounting is difficult because resources

intertwined in the implementation of abstractions. ex: file buffer intertwined in the implementation of abstractions. ex: file buffer cache and TCP\IP socket buffers consume memory resources cache and TCP\IP socket buffers consume memory resources that aren’t charged to any particular app.that aren’t charged to any particular app. [Denali paper] [Denali paper]

April 22, 2023 14

Can It Be Implemented?Can It Be Implemented? Not only feasible, but practicalNot only feasible, but practical

Several off the shelf products exist:» VMware ESX server» IBM z/VM» Disco (for NUMA)

For us, VMMs only needs to support half-doze to dozen VMs,can easily handle (VMware ESX supports up to 64 VMs)

If a particular VMM doesn’t provide guarantees for a resource, it can be easily extended with well known scheduling algs.» Fair queuing => network bandwidth» Stride scheduling => CPU» Cello framework => disk bandwidth allocation

April 22, 2023 15

ConclusionConclusion Network experimentation testbeds are a commodityNetwork experimentation testbeds are a commodity Users have different needs + require guarantees. Users have different needs + require guarantees.

Currently no formal way to handle thisCurrently no formal way to handle this

Virtualizing testbed nodes by running a VMM is a Virtualizing testbed nodes by running a VMM is a good solution.good solution.1. Efficient multiplexing of resources2. Fine-grained control over physical resources3. Isolation4. Customization

April 22, 2023 16

Hardware (HW)

VMM

OS 1 OS 2 OS 5

App

1

App

2

App

1

…

App

1

…

HW HW HW