Top Banner
PlanetLab and OneLab Presentation at the GRID 5000 School 9 March 2006 Timur Friedman Université Pierre et Marie Curie Laboratoire LIP6-CNRS PlanetLab slides based on slides provided courtesy of Larry Peterson
60

PlanetLab and OneLab Presentation at the GRID 5000 School

Jan 21, 2015

Download

Technology

Cameroon45

 
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: PlanetLab and OneLab Presentation at the GRID 5000 School

PlanetLab and OneLabPresentation at the GRID 5000

School9 March 2006

Timur Friedman

Université Pierre et Marie CurieLaboratoire LIP6-CNRS

PlanetLab slides based on slidesprovided courtesy of Larry Peterson

Page 2: PlanetLab and OneLab Presentation at the GRID 5000 School

PlanetLab• An open platform for:

– testing overlays,– deploying experimental services,– deploying commercial services,– developing the next generation of internet technologies.

• A set of virtual machines– distributed virtualization– each of 350+ network services runs in its own slice

Page 3: PlanetLab and OneLab Presentation at the GRID 5000 School

PlanetLab nodes

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

• 637 machines spanning 302 sites and 35 countries nodes within a LAN-hop of > 2M users

Page 4: PlanetLab and OneLab Presentation at the GRID 5000 School

Slices

Page 5: PlanetLab and OneLab Presentation at the GRID 5000 School

Slices

Page 6: PlanetLab and OneLab Presentation at the GRID 5000 School

Slices

Page 7: PlanetLab and OneLab Presentation at the GRID 5000 School

User Opt-in

ServerNAT

Client

Page 8: PlanetLab and OneLab Presentation at the GRID 5000 School

Per-Node View

Virtual Machine Monitor (VMM)

NodeMgr

LocalAdmin

VM1 VM2 VMn…

Page 9: PlanetLab and OneLab Presentation at the GRID 5000 School

Architecture (1)• Node Operating System

– isolate slices– audit behavior

• PlanetLab Central (PLC)– remotely manage nodes– bootstrap service to instantiate and control slices

• Third-party Infrastructure Services– monitor slice/node health– discover available resources– create and configure a slice– resource allocation

Page 10: PlanetLab and OneLab Presentation at the GRID 5000 School

Owner 1

Owner 2

Owner 3

Owner N

. . .

SliceAuthority

ManagementAuthority

Software updates

Auditing data

Create slices

. .

.

U S

E R

S

PlanetLabNodes

ServiceDevelopers

Request a slice

New slice ID

Access slice

Identifyslice users(resolve abuse)

Learn about nodes

Architecture (2)

Page 11: PlanetLab and OneLab Presentation at the GRID 5000 School

Architecture (3)

Node

MA

NM +VMM

nodedatabase

NodeOwner

OwnerVM

SCS

SAslicedatabase

VM ServiceDeveloper

Page 12: PlanetLab and OneLab Presentation at the GRID 5000 School

Long-Running Services• Content Distribution

– CoDeeN: Princeton– Coral: NYU– Cobweb: Cornell

• Internet Measurement– ScriptRoute: Washington, Maryland

• Anomaly Detection & Fault Diagnosis– PIER: Berkeley, Intel– PlanetSeer: Princeton

• DHT– Bamboo (OpenDHT): Berkeley, Intel– Chord (DHash): MIT

Page 13: PlanetLab and OneLab Presentation at the GRID 5000 School

Services (cont)• Routing

– i3: Berkeley– Virtual ISP: Princeton

• DNS– CoDNS: Princeton– CoDoNs: Cornell

• Storage & Large File Transfer– LOCI: Tennessee– CoBlitz: Princeton– Shark: NYU

• Multicast– End System Multicast: CMU– Tmesh: Michigan

Page 14: PlanetLab and OneLab Presentation at the GRID 5000 School

Usage Stats• Slices: 350 - 425• AS peers: 6000• Users: 1028• Bytes-per-day: 2 - 4 TB

– Coral CDN represents about half of this

• IP-flows-per-day: 190M• Unique IP-addrs-per-day: 1M

Page 15: PlanetLab and OneLab Presentation at the GRID 5000 School

OneLab• A potential project, currently under negotiation with the

European Commission– Project leader: UPMC/LIP6-CNRS– Technical direction: INRIA Sophia-Antipolis– Other partners:

• Intel Research Cambridge, Universidad Carlos III de Madrid, Université Catholique de Louvain, Università di Napoli, France Telecom (Lannion), Università di Pisa, Alcatel Italia, Telekomunikacja Polska

• Goals:– Extend PlanetLab into new environments, beyond the traditional

wired internet.– Deepen PlanetLab’s monitoring capabilities.– Provide a European administration for PlanetLab nodes in Europe.

Page 16: PlanetLab and OneLab Presentation at the GRID 5000 School

Goal: New Environments• Problem: PlanetLab nodes are connected to the

traditional wired internet.– They are mostly connected to high-performance

networks such as Abilene, DANTE, NRENs.– These are not representative of the internet as a whole.– PlanetLab does not provide access to emerging

environments.• OneLab will place nodes in new environments:

– Wireless: WiMAX, UMTS, and wireless ad hoc networks.

– Wired: multihomed nodes.– Emulated: for new and experimental technologies.

Page 17: PlanetLab and OneLab Presentation at the GRID 5000 School

Goal: Deepen Monitoring• Problem: PlanetLab provides limited facilities to

make applications aware of the underlying network

• OneLab’s monitoring components– Passive monitoring: Track packets at the routers– Topology monitoring: Provide a view of the route

structure

Page 18: PlanetLab and OneLab Presentation at the GRID 5000 School

PlanetLab Before OneLab

Page 19: PlanetLab and OneLab Presentation at the GRID 5000 School

PlanetLab After OneLab

Page 20: PlanetLab and OneLab Presentation at the GRID 5000 School

New Environments

Page 21: PlanetLab and OneLab Presentation at the GRID 5000 School

Monitoring Capabilities

Page 22: PlanetLab and OneLab Presentation at the GRID 5000 School

Goal: European Administration

• Problem: Changes to PlanetLab must come through the administration at Princeton.– PlanetLab in the US is necessarily less responsive to

European research priorities.• OneLab will create a PlanetLab Europe.

– It will federate with PlanetLab in the US, Japan, and elsewhere.

– The federated structure will allow:• PlanetLab Europe to set policy in accordance with European

research priorities,• PlanetLab Europe to customize the platform, so long as a

common interface is preserved.

Page 23: PlanetLab and OneLab Presentation at the GRID 5000 School

PlanetLab and GRID 5000• Some goals in common• Some differences in architecture• Possibilities for cooperation between OneLab and

GRID 5000

Page 24: PlanetLab and OneLab Presentation at the GRID 5000 School

Common goals• Test at the scale of the internet, with internet

conditions• Test new architectures and services

– Even radical departures from the current internet• PlanetLab a precursor to the GENI initiative

Page 25: PlanetLab and OneLab Presentation at the GRID 5000 School

Internet• GRID 5000 is at the scale of the internet

– Reserved fibre to interconnect clusters– Cross-traffic can be injected, if wished– Ability to control and replay experiments

• PlanetLab works over the internet– Connections between nodes pass via the public internet– Cross-traffic comes from the internet itself

• Test services in a real setting• A challenged environment is interesting

– PlanetLab provides services to internet users

Page 26: PlanetLab and OneLab Presentation at the GRID 5000 School

Clusters• GRID 5000 consists of clusters of many machines

– To participate in GRID 5000, one signs up with one of the cluster administrators

– Access is to university and state-sponsored research labs

• There are typically two PlanetLab nodes to a site– To participate in PlanetLab, one provides two nodes– University and state-sponsored research labs pay no

fees– For-profit organizations pay fees of $25K/yr.+– Available for use by industry

Page 27: PlanetLab and OneLab Presentation at the GRID 5000 School

Virtualization• GRID 5000 designed for the installation of any

OS on top of the hardware of any node– One OS per machine

• PlanetLab designed for the installation of multiple virtual machines on top of a VMM (virtual machine manager) on any node– VMM currently linux-vserver

• Could be xen-domain, or other

– Virtual machines currently only Linux• Eventually could be any suitably adapted OS

• GRID 5000’s OS could be a VMM, if desired

Page 28: PlanetLab and OneLab Presentation at the GRID 5000 School

Reservations• Users reserve GRID 5000 nodes

– No two users have the same node at the same time– Users have access to completely unloaded machines

• Users share PlanetLab nodes– The load affects the performance– Problems arise close to major conference deadlines– Services allow one to select a subset of nodes based on

their load characteristics– Services allow one to make a reservation for higher

priority on certain machines

Page 29: PlanetLab and OneLab Presentation at the GRID 5000 School

Cooperation• Test PlanetLab architectures on GRID 5000

– OneLab topology monitoring component will be tested on GRID 5000

• Joint work: Pierre Sens and Timur Friedman

• Test GRID 5000 architectures on PlanetLab?• The European Commission invites such forms of

cooperation

Page 30: PlanetLab and OneLab Presentation at the GRID 5000 School

Fin

Page 31: PlanetLab and OneLab Presentation at the GRID 5000 School

More About PlanetLab

Page 32: PlanetLab and OneLab Presentation at the GRID 5000 School

PlanetLab Architecture• What is the PlanetLab architecture?

– more a question of synthesis than cleverness• Why is this the right architecture?

– non-technical requirements– technical decisions that influenced adoption

• What is a system architecture anyway?– how does it accommodate change (evolution)

Page 33: PlanetLab and OneLab Presentation at the GRID 5000 School

Requirements

1) Global platform that supports both short-term experiments and long-running services.

– services must be isolated from each other• performance isolation• name space isolation

– multiple services must run concurrently

Distributed Virtualization– each service runs in its own slice: a set of VMs

Page 34: PlanetLab and OneLab Presentation at the GRID 5000 School

Requirements

2) It must be available now, even though no one knows for sure what “it” is.

– deploy what we have today, and evolve over time– make the system as familiar as possible (e.g., Linux)

Unbundled Management– independent mgmt services run in their own slice– evolve independently; best services survive– no single service gets to be “root” but some services

require additional privilege

Page 35: PlanetLab and OneLab Presentation at the GRID 5000 School

Requirements

3) Must convince sites to host nodes running code written by unknown researchers.

– protect the Internet from PlanetLab

Chain of Responsibility– explicit notion of responsibility– trace network activity to responsible party

Page 36: PlanetLab and OneLab Presentation at the GRID 5000 School

Requirements

4) Sustaining growth depends on support for autonomy and decentralized control.

– sites have the final say about the nodes they host– sites want to provide “private PlanetLabs”– regional autonomy is important

Federation– universal agreement on minimal core (narrow waist)– allow independent pieces to evolve independently– identify principals and trust relationships among them

Page 37: PlanetLab and OneLab Presentation at the GRID 5000 School

Requirements5) Must scale to support many users with minimal

resources available.– expect under-provisioned state to be the norm– shortage of logical resources too (e.g., IP addresses)

Decouple slice creation from resource allocationOverbook with recovery

– support both guarantees and best effort– recover from wedged states under heavy load

Page 38: PlanetLab and OneLab Presentation at the GRID 5000 School

Tension Among Requirements• Distributed Virtualization / Unbundled Management

– isolation vs one slice managing another

• Federation / Chain of Responsibility– autonomy vs trusted authority

• Under-provisioned / Distributed Virtualization– efficient sharing vs isolation

• Other tensions– support users vs evolve the architecture– evolution vs clean slate

Page 39: PlanetLab and OneLab Presentation at the GRID 5000 School

Synergy Among Requirements• Unbundled Management

– third party management software

• Federation– independent evolution of components– support for autonomous control of resources

Page 40: PlanetLab and OneLab Presentation at the GRID 5000 School

Architecture (1)• Node Operating System

– isolate slices– audit behavior

• PlanetLab Central (PLC)– remotely manage nodes– bootstrap service to instantiate and control slices

• Third-party Infrastructure Services– monitor slice/node health– discover available resources– create and configure a slice– resource allocation

Page 41: PlanetLab and OneLab Presentation at the GRID 5000 School

Trust RelationshipsPrincetonBerkeleyWashingtonMITBrownCMUNYUETHHarvardHP LabsIntelNEC LabsPurdueUCSDSICSCambridgeCornell…

princeton_codeennyu_dcornell_beehiveatt_mcashcmu_esmharvard_icehplabs_donutlabidsl_pseprirb_phiparis6_landmarksmit_dhtmcgill_cardhuji_enderarizona_storkucb_bambooucsd_shareumd_scriptroute…

N x NTrusted

Intermediary(PLC)

Page 42: PlanetLab and OneLab Presentation at the GRID 5000 School

Trust Relationships (cont)

NodeOwner

PLCService

Developer(User)1

2

3

4

1) PLC expresses trust in a user by issuing it credentials to access a slice

2) Users trust to create slices on their behalf and inspect credentials

3) Owner trusts PLC to vet users and map network activity to right user

4) PLC trusts owner to keep nodes physically secure

Page 43: PlanetLab and OneLab Presentation at the GRID 5000 School

Trust Relationships (cont)

NodeOwner

MgmtAuthority

ServiceDeveloper

(User)1

2

3

4

1) PLC expresses trust in a user by issuing credentials to access a slice

2) Users trust to create slices on their behalf and inspect credentials

3) Owner trusts PLC to vet users and map network activity to right user

4) PLC trusts owner to keep nodes physically secure

SliceAuthority

5

6

5) MA trusts SA to reliably map slices to users

6) SA trusts MA to provide working VMs

Page 44: PlanetLab and OneLab Presentation at the GRID 5000 School

Owner 1

Owner 2

Owner 3

Owner N

. . .

SliceAuthority

ManagementAuthority

Software updates

Auditing data

Create slices

. .

.

U S

E R

S

PlanetLabNodes

ServiceDevelopers

Request a slice

New slice ID

Access slice

Identifyslice users(resolve abuse)

Learn about nodes

Architecture (2)

Page 45: PlanetLab and OneLab Presentation at the GRID 5000 School

Architecture (3)

Node

MA

NM +VMM

nodedatabase

NodeOwner

OwnerVM

SCS

SAslicedatabase

VM ServiceDeveloper

Page 46: PlanetLab and OneLab Presentation at the GRID 5000 School

Per-Node Mechanisms

Virtual Machine Monitor (VMM)

NodeMgr

OwnerVM

VM1 VM2 VMn…

Linux kernel (Fedora Core)+ Vservers (namespace isolation)+ Schedulers (performance isolation)+ VNET (network virtualization)

SliverMgrProper

PlanetFlowSliceStatpl_scspl_mom

Page 47: PlanetLab and OneLab Presentation at the GRID 5000 School

VMM• Linux

– significant mind-share• Vserver

– scales to hundreds of VMs per node (12 MB each)• Scheduling

– CPU• fair share per slice (guarantees possible)

– link bandwidth• fair share per slice• average rate limit: 1.5Mbps (24-hour bucket size)• peak rate limit: set by each site (100Mbps default)

– disk• 5 GB quota per slice (limit run-away log files)

– memory• no limit• pl_mom resets biggest user at 90% utilization

Page 48: PlanetLab and OneLab Presentation at the GRID 5000 School

VMM (cont)• VNET

– socket programs “just work”• including raw sockets

– slices should be able to send only…• well-formed IP packets• to non-blacklisted hosts

– slices should be able to receive only…• packets related to connections that they initiated (e.g., replies)• packets destined for bound ports (e.g., server requests)

– essentially a switching firewall for sockets• leverages Linux’s built-in connection tracking modules

Page 49: PlanetLab and OneLab Presentation at the GRID 5000 School

Node Manager• SliverMgr

– creates VM and sets resource allocations– interacts with…

• bootstrap slice creation service (pl_scs)• third-party slice creation & brokerage services (using tickets)

• Proper: PRivileged OPERations– grants unprivileged slices access to privileged info– effectively “pokes holes” in the namespace isolation– examples

• files: open, get/set flags• directories: mount/unmount• sockets: create/bind• processes: fork/wait/kill

Page 50: PlanetLab and OneLab Presentation at the GRID 5000 School

Auditing & Monitoring• PlanetFlow

– logs every outbound IP flow on every node• accesses ulogd via Proper• retrieves packet headers, timestamps, context ids (batched)

– used to audit traffic– aggregated and archived at PLC

• SliceStat– has access to kernel-level / system-wide information

• accesses /proc via Proper

– used by global monitoring services– used to performance debug services

Page 51: PlanetLab and OneLab Presentation at the GRID 5000 School

Infrastructure Services• Brokerage Services

– Sirius: Georgia– Bellagio: UCSD, Harvard, Intel– Tycoon: HP

• Environment Services– Stork: Arizona– Application Manager: Berkeley (UC + Intel)

• Monitoring/Discovery Services– CoMon: Princeton– PsEPR: Intel

Page 52: PlanetLab and OneLab Presentation at the GRID 5000 School

Evolution vs Intelligent Design• Favor evolution over clean slate• Favor design principles over a fixed architecture• Specifically…

– leverage existing software and interfaces– keep VMM and control plane orthogonal– exploit virtualization

• vertical: management services run in slices• horizontal: stacks of VMs

– give no one root (least privilege + level playing field)– support federation (decentralized control)

Page 53: PlanetLab and OneLab Presentation at the GRID 5000 School

Other Lessons

• Inferior tracks lead to superior locomotives• Empower the user: yum• Build it and they (research papers) will come• Overlays are not networks• PlanetLab: We debug your network• From universal connectivity to gated communities• If you don’t talk to your university’s general

counsel, you aren’t doing network research• Work fast, before anyone cares

Page 54: PlanetLab and OneLab Presentation at the GRID 5000 School

Fin

Page 55: PlanetLab and OneLab Presentation at the GRID 5000 School

Available CPU Capacity

0

20

40

60

80

100

120

10 20 30 40 50 60 70 80

Pct of CPU Available

Pct of 360 Nodes

Feb 1-8, 2005 (Week before SIGCOMM deadline)

Page 56: PlanetLab and OneLab Presentation at the GRID 5000 School

Node Boot/InstallNode PLC (MA) Boot Server

1. Boots from BootCD (Linux loaded)

2. Hardware initialized

3. Read network config . from floppy

7. Node key read into memory from floppy

4. Contact PLC (MA)

6. Execute boot mgr

Boot Manager

8. Invoke Boot API

10. State = “install”, run installer

11. Update node state via Boot API

13. Chain-boot node (no restart)

14. Node booted

9. Verify node key, send current node state

12. Verify node key, change state to “boot”

5. Send boot manager

Page 57: PlanetLab and OneLab Presentation at the GRID 5000 School

Chain of ResponsibilityJoin Request PI submits Consortium paperwork and requests to join

PI Activated PLC verifies PI, activates account, enables site (logged)

User Activated Users create accounts with keys, PI activates accounts (logged)

Nodes Added to Slices

Users add nodes to their slice (logged)

Slice Traffic Logged

Experiments generate traffic (logged by PlanetFlow)

Traffic Logs Centrally Stored

PLC periodically pulls traffic logs from nodes

Slice Created PI creates slice and assigns users to it (logged)

Network Activity Slice Responsible Users & PI

Page 58: PlanetLab and OneLab Presentation at the GRID 5000 School

Slice Creation

PLC(SA)

VMM

NM VM

PI SliceCreate( ) SliceUsersAdd( )

User SliceAttributeSet( ) SliceGetTicket( )

VM VM…

.

.

.

.

.

.

(distribute ticket to slice creation service: pl_scs)

SliverCreate(rspec)

Page 59: PlanetLab and OneLab Presentation at the GRID 5000 School

Brokerage Service

PLC(SA)

VMM

NM VMSliceAttributeSet( ) SliceGetTicket( ) VM VM…

.

.

.

.

.

.

(distribute ticket to brokerage service)

rcap = PoolCreate(rspec)

Broker

Page 60: PlanetLab and OneLab Presentation at the GRID 5000 School

Brokerage Service (cont)

PLC(SA)

VMM

NM VM VM VM…

.

.

.

.

.

.

(broker contacts relevant nodes)

PoolSplit(rcap, slice, rspec)

VM

User BuyResources( ) Broker