Mininet: Moving Forward

Mininet:    Moving  Forward    

Nikhil  Handigol  

Brandon  Heller  

Vimal  Jeyakumar  

Bob    Lantz  

[Team  Mininet]  

ONRC  Annual  Day  2013  

Intro  to  Mininet  

3  

TesKng  an  SDN  Idea  

•  Physical  switches/hosts?  •  Testbed?  •  Simulator?  

•  Emulator?    Mininet  

Mininet  

Your  SDN  

Command-­‐Line  Interface  

ScripKng  API  

VM,  naKve  laptop,  or  naKve  server  

To  start  up  Mininet:  

> sudo mn

Open-­‐Source  System  w/AcKve  User  Community  

6  

•  1000+  users  •  AcKve  mailing  list    •  700+  users,  200+  domains  

•  20  Google  Summer  of  Code  apps  •  GitHub  pull  requests  

mininet.github.com  

Talk  Outline  

•  Mininet  1.0:  funcKonal  fidelity  

•  Mininet  2.0  (HiFi)  – Performance  Fidelity  

– “Network  Invariants”  – Reproducible  Research  

•  Prototypes:  greater  scale  

Mininet  2.0  (HiFi)  

8  

Verifying  Network  ProperKes  

•  “Does  my  SDN  work?”  – E.g.,  func&onal  correctness  – Same  control  program  +  OF    funcKonal  fidelity  

•  Does  not  try  to  solve  a  harder  problem:  – “How  does  my  SDN/network  perform?”  – That  is,  performance  proper&es.  

– No  guarantee  or  expectaKon  of  perf.  fidelity.    

!"#$%$

&"'($

)**'(*+,"-$

.#*($

!"#$/$ !"#$0$ !"#$1$

Example:  ConnecKvity  in  a  Fat  Tree  

servers  

switches  

10  

20x  4-­‐port  switches  

16x  servers  

sudo mn –-custom ft.py -–topo ft,4 –test pingall \ –-controller remote

Verifying  Network  ProperKes  

•  “Does  my  SDN  work?”  – E.g.,  func&onal  correctness  – Same  control  program  +  OF    funcKonal  fidelity  

•  “How  does  my  SDN  perform?”  – E.g.,  performance  proper&es  

– No  guarantee  or  even  expectaKon  here  

hosts  

switches  

12  

A   B   Y   Z  

full  throughput  

Two  1  Gb/s  flows.  

Disjoint  paths.  

Example:  Performance  in  a  Fat  Tree  

hosts  

switches  

13  

A   B   Y   Z  

X  collision  half  throughput  

Two  1  Gb/s  flows.  

Overlapping  paths.  

Example:  Performance  in  a  Fat  Tree  

Throughput  might  reduce.    But  by  how  much?    

How  do  you  trust  the  results?  

14  

overlapping  events  real  Kme  

…  …  

…   A BLink

Time

Events

A: send requestB: initxmit 1

xmit 2

B: send reponse

A B B HiFiA x1 x2B A B

LoFiA B x1 B

Packet xmit2

// A: Client

while(1) { send_request(socket); wait_for_reply(socket);}

// B: Serverinit();while(1) { wait_for_request(socket); send_response(socket);}

1

B

RealSetup

B: wait

busy

idle

x2

A BLink

Time

Events

A: send requestB: initxmit 1

xmit 2

B: send reponse

A B B HiFiA x1 x2B A B

LoFiA B x1 B

Packet xmit2

// A: Client

while(1) { send_request(socket); wait_for_reply(socket);}

// B: Serverinit();while(1) { wait_for_request(socket); send_response(socket);}

1

B

RealSetup

B: wait

busy

idle

x2

S  

Sources  of  Emulator  Infidelity  Event  Overlap  

15  

real  Kme  

…  …  

…   A BLink

Time

Events

A: send requestB: initxmit 1

xmit 2

B: send reponse

A B B HiFiA x1 x2B A B

LoFiA B x1 B

Packet xmit2

// A: Client

while(1) { send_request(socket); wait_for_reply(socket);}

// B: Serverinit();while(1) { wait_for_request(socket); send_response(socket);}

1

B

RealSetup

B: wait

busy

idle

x2

A BLink

Time

Events

A: send requestB: initxmit 1

xmit 2

B: send reponse

A B B HiFiA x1 x2B A B

LoFiA B x1 B

Packet xmit2

// A: Client

while(1) { send_request(socket); wait_for_reply(socket);}

// B: Serverinit();while(1) { wait_for_request(socket); send_response(socket);}

1

B

RealSetup

B: wait

busy

idle

x2

S  

Sources  of  Emulator  Infidelity  So8ware  Forwarding  

variable  delays  

How  can  we  trust  emulator  results?  

real  Kme  

…  …  

…   A BLink

Time

Events

A: send requestB: initxmit 1

xmit 2

B: send reponse

A B B HiFiA x1 x2B A B

LoFiA B x1 B

Packet xmit2

// A: Client

while(1) { send_request(socket); wait_for_reply(socket);}

// B: Serverinit();while(1) { wait_for_request(socket); send_response(socket);}

1

B

RealSetup

B: wait

busy

idle

x2

A BLink

Time

Events

A: send requestB: initxmit 1

xmit 2

B: send reponse

A B B HiFiA x1 x2B A B

LoFiA B x1 B

Packet xmit2

// A: Client

while(1) { send_request(socket); wait_for_reply(socket);}

// B: Serverinit();while(1) { wait_for_request(socket); send_response(socket);}

1

B

RealSetup

B: wait

busy

idle

x2

S  

CPU  <=  50%,  so  not  overloaded,  right?  Wrong.  

17  

The  Mininet-­‐HiFi  Approach  

Resource  IsolaKon  

HIGH!

Fidelity!METER!

MEDIUM!

LOW!

Fidelity    Monitor  

+  500  Mhz  

20  pkt  bufs/port  

10  Mb/s,  1ms  

Network  Invariants  

18  

A  Workflow  for  High  Fidelity  EmulaKon  

19  

Create  experiment  

Run  the  experiment  on  a  PC,    

with  logging  

Analyze  experiment  fidelity  using  “network  invariants”  

Invariants  hold:  High  Fidelity  EmulaKon!  

Instance(s)  of  behavior  that  differ  from  hardware  

Run  again:  increase  resources  or  

reduce  experiment  scale  

1:  what  to  log?  2:  which  invariants?  

3:  how  close?  

20  Packet  Gap  Invariants  

queue   link   switch   queue  

packet  spacing  (when  queue  occupied)  

Rmeasured  ≤  Rconfigured  ?          link  capacity  

Example  Workflow  for  One  Invariant  

21  

Create  experiment  

Run  the  experiment  on  a  PC,    

with  logging  

Analyze  experiment  fidelity  using  “network  invariants”  

Invariants  hold:  High  Fidelity  EmulaKon!  

Instance(s)  of  behavior  that  differ  from  hardware  

Run  again:  increase  resources  or  

reduce  experiment  scale  

2:  Measure  packet    spacing  

3:  Is  any  packet  delayed  by  more  than  one  packet  Kme?  

1:  Log  Dequeue  Events  

If  this  workflow  is  valid,  “pass”    same  result  as  hardware.  

DCTCP  

Data  Center  TCP  (DCTCP)  

[Sigcomm10,  Alizadeh  et  al.]  

Kme  

packets  in  queue  

TCP  

DCTCP  

22  

marking  threshold  

Queue  occupied  100%  throughput  

Queue  occupied  100%  throughput  

Packet  spacing  we  should  see:  

0 30 60 90 120Seconds

0510152025303540Packetsq-dctcp-plot.txt

Hardware  Results,  100  Mb/s  

23  

packets  in  

queue  

100%  throughput  

6  packets  variaKon  

queue  occupied  

Emulator  Results  

24  

Does  checking  an  invariant  (packet  spacing)  idenKfy  wrong  results?  

same  result   wrong;    limits  

exceeded  

80  Mb/s  

100%  tput  

6  pkts  var  same  result  

160  Mb/s  

100%  tput  

6  pkts  var  

320  Mb/s  

Packet  Spacing  Invariant  w/DCTCP  

25  

1  pkt  

med.   low  high  

CCDF  Percent  (log)  

25  pkts  Error:  (log)  

10%  of  the  Kme,  error  exceeds  one  packet  

x  

Percentage deviation from expected

0

1

10

100Pe

rcen

t

26  

1  pkt  error  

10  20  40  

numbers  are  in  Mb/s  

80  

CCDF  Percent  

Packet  Spacing  Invariant  w/DCTCP  

Percentage deviation from expected

0

1

10

100Pe

rcen

t

27  

10  20  40  

numbers  are  in  Mb/s  

80  

1  pkt  error  

CCDF  Percent  

Packet  Spacing  Invariant  w/DCTCP  

Percentage deviation from expected

0

1

10

100Pe

rcen

t

28  

10  20  40  

numbers  are  in  Mb/s  

80  

CCDF  Percent  

Packet  Spacing  Invariant  w/DCTCP  160  Mb/s:  failed  emulaKon?  Beauty  of  networks  invariants  is  that  it  catches  and  quanKfies  the  error  in  this  run.  

1  pkt  error  

DemonstraKng  Fidelity  

•  Microbenchmarks  

•  ValidaKon  Tests  •  Reproducing  Published  Research  – Do  complex  results  match  published  ones  that  used  custom  hardware  topologies?  •  DCTCP  [Alizadeh,  SIGCOMM  2010]  

•  Router  Buffer  Sizing  [Appenzeller,  SIGCOMM  2004]  

•  Hedera  ECMP  [Al-­‐Fares,  NSDI  2010]  

Reproducing  Research  

30  

31  

Stanford  CS244  Spring  ’12:  Advanced  Topics  in  Networking  

→  Pick  a  paper.    →  Reproduce  a  key  result,  

or  challenge  it  (with  data).  

→  You  have:  $100  EC2  credit,  3  weeks,  and  must  use  Mininet-­‐HiFi.  

32  

CoDel  HULL  MPTCP  Outcast  Jellyfish  DCTCP  Incast  Flow  CompleKon  Time  Hedera  DCell  TCP  IniKal  CongesKon  Window  Misbehaving  TCP  Receivers  RED  

Project  Topics:  Transport,    Data  Center,  Queuing  

33  

CoDel  HULL  MPTCP  Outcast  Jellyfish  DCTCP  Incast  Flow  CompleKon  Time  Hedera  DCell  TCP  IniKal  CongesKon  Window  Misbehaving  TCP  Receivers  RED  

34  

37  students  18  projects  16  replicated  

CoDel  HULL  MPTCP  Outcast  Jellyfish  DCTCP  Incast  Flow  CompleKon  Time  Hedera  DCell  TCP  IniKal  CongesKon  Window  Misbehaving  TCP  Receivers  RED  

37  students  18  projects  16  replicated  4  beyond  

35  

CoDel  HULL  MPTCP  Outcast  Jellyfish  DCTCP  Incast  Flow  CompleKon  Time  Hedera  DCell  TCP  IniKal  CongesKon  Window  Misbehaving  TCP  Receivers  RED  

37  students  18  projects  16  replicated  4  beyond  2  not  replicated  

36  

Reproduced  Research  Examples  

reproducingnetworkresearch.wordpress.com  

(or  Google  “reproducing  network  research”)  

37  

Why  might  results  be  different?  

•  Student  error  /  out  of  Kme:  Incast  

•  Original  result  fragile:  RED  •  Insufficient  emulator  capacity  to  match  hardware  of  original  experiment  – OpKon  1:  Scale  up  – OpKon  2:  Slow  down:  Time  DilaKon  

– OpKon  3:  Scale  out:  Cluster  EdiKon  

QuesKons?  

•  Check  out  Bob’s  Cluster  EdiKon  demo  

Nikhil  Handigol  

Brandon  Heller  

Vimal  Jeyakumar  

Bob    Lantz  

[Team  Mininet]