An Experimentation Workbench for Replayable Networking Research Eric Eide Eric Eide , Leigh , Leigh Stoller, Stoller, and Jay Lepreau and Jay Lepreau University of Utah, University of Utah, School of Computing School of Computing NSDI 2007 / April 12, 2007 NSDI 2007 / April 12, 2007
32
Embed
An Experimentation Workbench for Replayable Networking Research Eric Eide, Leigh Stoller, and Jay Lepreau University of Utah, School of Computing NSDI.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
An Experimentation Workbench for Replayable
Networking Research
Eric EideEric Eide, Leigh Stoller,, Leigh Stoller,and Jay Lepreauand Jay Lepreau
University of Utah,University of Utah,School of ComputingSchool of Computing
NSDI 2007 / April 12, 2007NSDI 2007 / April 12, 2007
2
Repeated Research
“A scientific community advances when its experiments are repeated…”“A scientific community advances when its experiments are repeated…”
Translation: “I have troublemanaging my own experiments.”Translation: “I have troublemanaging my own experiments.”
current network testbedscurrent network testbeds ……manage the “laboratory”manage the “laboratory” ……not the experimentation processnot the experimentation process
a big problem for large-scale activitiesa big problem for large-scale activities
6
Need
my experiment needs…my experiment needs… encapsulationencapsulation automationautomation instrumentationinstrumentation preservationpreservation
benefitsbenefits verify previous resultsverify previous results establish base for new researchestablish base for new research my own, or someone else’smy own, or someone else’s
get the lab manager to help us out!get the lab manager to help us out! integratedintegrated support for experimental procedures support for experimental procedures resources + encapsulation + automationresources + encapsulation + automation framework:framework: rapid start & common basis rapid start & common basis
manage scientific workflow, but also manage labmanage scientific workflow, but also manage lab
8
Experimentation Workbench
an environment for “replayable research”an environment for “replayable research” experiment management + experiment executionexperiment management + experiment execution (but really: help me manage my work)(but really: help me manage my work) all Emulab-managed devices, incl. PlanetLab slivers, …all Emulab-managed devices, incl. PlanetLab slivers, …
initial design, implementation, and evaluationinitial design, implementation, and evaluation new model of testbed-based experimentsnew model of testbed-based experiments prototype implementationprototype implementation case studiescase studies lessons learnedlessons learned
definition versus instancedefinition versus instance related experimentsrelated experiments multiple trials per sessionmultiple trials per session data managementdata management
template instances can also template instances can also be created programmaticallybe created programmatically
17
Template Instance
templatetemplate
instanceinstance
runrun
activityactivity
recordrecord
a container of testbed a container of testbed resourcesresources
resource-owner role of resource-owner role of classic “experiment” notionclassic “experiment” notion
a transitory objecta transitory object created and destroyed by created and destroyed by
users and activitiesusers and activities nodes & networknodes & network
files from the datastorefiles from the datastore database for userdatabase for user
18
Run & Activity
templatetemplate
instanceinstance
runrun
activityactivity
recordrecord
run: a container of a user-run: a container of a user-defined “unit of work”defined “unit of work” defines a contextdefines a context a “trial”a “trial” one / serial / parallelone / serial / parallel
activity: a process, script, activity: a process, script, workflow, … within a runworkflow, … within a run events & data come from events & data come from
activities in a runactivities in a run
runs and activities can be runs and activities can be scripted or interactivescripted or interactive
prototype: implemented via prototype: implemented via agents & eventsagents & events
19
Record
templatetemplate
instanceinstance
runrun
activityactivity
recordrecord
the “flight recorder” of a runthe “flight recorder” of a run parameter valuesparameter values input & output files, DBsinput & output files, DBs raw data & derived dataraw data & derived data template’s by-reference template’s by-reference
resourcesresources dynamically recorded eventsdynamically recorded events
20
Record Repository
view, export, and replayview, export, and replay
Evaluation andLessons Learned
22
Evaluation
how to evaluate?how to evaluate? new capabilities new capabilities user studies user studies
goal:goal: early feedback about design & impl. early feedback about design & impl. approach:approach: three case studies three case studies outcome:outcome: specific & general lessons learned specific & general lessons learned
23
Study 1: Flexlab Development
replace ad hoc replace ad hoc experiment experiment managementmanagement
originally:originally: a configurable ns filea configurable ns file start/stop trial scriptsstart/stop trial scripts ““scaffold” in CVSscaffold” in CVS manual archivingmanual archiving destructive modificationdestructive modification
now:now: templates & paramstemplates & params runs, start/stop hooksruns, start/stop hooks scaffold & results in WBscaffold & results in WB automatic archivingautomatic archiving preserved historypreserved history
Conclusion:Conclusion: the new model “fits” developers’ model the new model “fits” developers’ modelConclusion:Conclusion: the new model “fits” developers’ model the new model “fits” developers’ model
24
Study 2: Flexlab Use
study BitTorrent on study BitTorrent on Flexlab and PlanetLabFlexlab and PlanetLab
outcome:outcome: parameterizationparameterization utilized per-run databaseutilized per-run database team communicationteam communication results for publicationresults for publication
solution:solution: pipeline record-making with user activities pipeline record-making with user activities new problem:new problem: isolation isolation new approach:new approach: branching file systems branching file systems
Record Record size (MB)size (MB)
Stored in Stored in repo. (MB)repo. (MB)
Elapsed Elapsed time (m)time (m)
BitTorrentBitTorrent 31.031.0 18.918.9 7.07.0
GHETEGHETE 70.670.6 19.819.8 14.414.4
27
What Users Want
deletion!deletion! not a space concernnot a space concern cognitive clutter — “junk”cognitive clutter — “junk” privacy — “mistakes”privacy — “mistakes”
a range of options is a range of options is requiredrequired
““true deletion” is a new true deletion” is a new requirementrequirement
28
Lessons: The Model
initial philosophy: “divide and conquer”initial philosophy: “divide and conquer” more kinds of entitiesmore kinds of entities describe notions and relationshipsdescribe notions and relationships
experience:experience: new model does map to users' abstractionsnew model does map to users' abstractions captures separations and connections...captures separations and connections... ...but not “life cycle” concerns...but not “life cycle” concerns
29
“Life Cycle” Concerns
multiple levels of abstractionmultiple levels of abstraction instance: “the lab”instance: “the lab” run & activity: “the work”run & activity: “the work”
intertwined & concurrentintertwined & concurrent workbench must manage experiments and the labworkbench must manage experiments and the lab a key difference with “ordinary” scientific workflow systemsa key difference with “ordinary” scientific workflow systems
approach: further refine and enhance our modelapproach: further refine and enhance our model e.g., adopt features of Plush [Albrecht et al., OSR 40(1)] or e.g., adopt features of Plush [Albrecht et al., OSR 40(1)] or
prototypeprototype builds on existing testbed infrastructure builds on existing testbed infrastructure model maps pretty well to user notionsmodel maps pretty well to user notions
experience:experience: strong integration is required strong integration is required ……for overlapping activities safelyfor overlapping activities safely ……for lab management + experiment managementfor lab management + experiment management ……for making it user-friendly in practicefor making it user-friendly in practice