Dr. Multicast for Data Center Communication Scalability Ymir Vigfusson Hussam Abu-Libdeh Mahesh Balakrishnan Ken Birman Cornell University Yoav Tock IBM.

Dr. Multicast    for Data Center Communication Scalability

Ymir Vigfusson   Hussam Abu-Libdeh   Mahesh Balakrishnan   Ken BirmanCornell University

Yoav TockIBM Research Haifa

HotNets, October 5, 2008

IP Multicast in Data Centers

• IPMC is not used in data centers

IP Multicast in Data Centers

• IPMC is not used in data centers• Would speed up products that use multicast

IP Multicast in Data Centers

• Why is IP multicast rarely used?

IP Multicast in Data Centers

• Why is IP multicast rarely used?o Limited IPMC scalability on switches/routers and

NICs

IP Multicast in Data Centers

• Why is IP multicast rarely used?o Limited IPMC scalability on switches/routers and

NICso Broadcast storms: Loss triggers a horde of

NACKs, which triggers more loss, etc. o Disruptive even to non-IPMC applications.

IP Multicast in Data Centers

• IP multicast has a bad reputation

IP Multicast in Data Centers

• IP multicast has a bad reputationo Works great up to a point,                                

after which it breaks                                         catastrophically

IP Multicast in Data Centers

• Bottom line:o Administrators have no control over multicast

use ...o Without control, they opt for never.

Dr. Multicast  

Dr. Multicast (MCMD)

• Policy: Permits data center operators to selectively enable and control IPMC

 • Transparency: Standard IPMC interface, system

calls are overloaded. • Performance: Uses IPMC when possible,

otherwise point-to-point unicast • Robustness: Distributed, fault-tolerant service

Terminology

• Process: Application that joins logical IPMC groups

• Logical IPMC group: A virtualized abstraction• Physical IPMC group: As usual• UDP multi-send: New kernel-level system-call 

  • Collection: Set of logical IPMC groups with

identical membership

Acceptable Use Policy

• Assume a higher-level network management tool compiles policy into primitives

• Explicitly allow a process to use IPMC groupso allow-join(process,logical IPMC)o allow-send(process,logical IPMC)

• UDP multi-send always permitted • Additional restraints

o max-groups(process,limit)o force-udp(process,logical IPMC)

 Overview

• Library module• Mapping module• Gossip layer

 • Optimization

questions • Results

• Transparent. Overloads the IPMC functions o setsockopt(), send(), etc.

 • Translation. Logical IPMC map to a

set of P-IPMC/unicast addresses.o Two extremes

MCMD Library Module

• MCMD Agent runs on each machineo Contacted by the library modules  o Provides a mapping

  • One agent elected to be a leader:

o Allocates IPMC resources according to the current policy    

MCMD Mapping Role

 • Allocating IPMC resources: An optimization problem

Procs 

L-IPMC

MCMD Mapping Role

This box intentionally left  

BLACK

Procs 

Collections

L-IPMC

• Runs system-wide as part of the agent • Automatic failure detection 

 • Group membership fully replicated via gossip

o Node reports its own stateo Future: Replicate more selectively o Leader runs optimization algorithm on data and

reports the mapping    

MCMD Gossip Layer

• But gossip is slow... • Implications:

o Slow propagation of group membershipo Slow propagation of new mapso We assume a low rate of membership churn

 • Remedy: Broadcast module

o Leader broadcasts urgent messages o Bounded bandwidth of urgent channelo Trade-off between latency and scalability

MCMD Gossip Layer

Overview

• Library module• Mapping module• Gossip layer

 • Optimization

questions • Results

Optimization Questions

Procs   L-IPMC

BLACK

Collections

Procs    L-IPMC

• First step: compress logical IPMC groups

klk;l    Optimization Questions

• How compressible are subscriptions?o Multi-objective optimization: 

Minimize number of collectionsMinimize bandwidth overhead on network

 o Thm: The general problem is NP-completeo Thm: In uniform random allocation, "little"

compression opportunity. o Social preferences o Lots of duplicates due to replication (e.g. for

load balancing)   

klk;l    Optimization Questions

• Which collections get an IPMC address?o Thm: Ordered by decreasing traffic*size, 

assign P-IPMC addresses greedily, we minimize bandwidth.

• Tiling heuristic:o Sort L-IPMC by traffic*sizeo Greedily collapse identical groupso Assign IPMC to collections in reverse order of

traffic*size, UDP-multisend to the rest• Building tilings incrementally

klk;l    Experimental Results

• Insignificant overhead when mapping L-IPMC to P-IPMC.

klk;l    Overhead (max. throughput)

• Insignificant overhead when mapping L-IPMC to P-IPMC.

klk;l    Overhead (CPU utilization)

klk;l    Network Overhead

• Gossip Layer uses constant background bandwidth, urgent channel behaves well

Latency

• Latency of propagation of joins/leaves and new maps

• A malfunctioning node bombards an existing IPMC group.• MCMD policy prevents ill-effects

klk;l    Policy control

<Traffic starts<New policy

Conclusion

• IPMC has been a bad citizen...

Conclusion

• IPMC has been a bad citizen...

 • Dr. Multicast has the cure!

• Opportunity for big performance enhancements and policy control.

Thank you!

Thank you!  

• Insignificant overhead when mapping L-IPMC to P-IPMC.            

klk;l    Overhead

• A malfunctioning node bombards an existing IPMC group.• MCMD policy prevents ill-effects

klk;l    Policy control

• A malfunctioning node bombards an existing IPMC group.• MCMD policy prevents ill-effects

klk;l    Policy control

• Linux kernel module increases UDP-multisend throughput by 17% (compared to user-space UDP-multisend)

klk;l    Overhead

klk;l    Latency of events

• Gossip: 99% of nodes aware of change within 9 epochs (now 1 sec)

Conclusions

• Policy: Allows data center operators to        enable and control IPMC

 • Transparency: Standard IPMC interface, system

calls are overloaded. • Performance: Uses IPMC when possible,

otherwise point-to-point UDP • Robustness: Distributed, fault-tolerant service

klk;l    Results

• Library Moduleo Insignificant slowdown

o Linux Kernel module provides 17% speed-up for UDP multi-send

klk;l    Optimization questions

Users

Topics

This box intentionally left  

BLACKUsers

 Groups

Topics

• Multi-objective: o Minimize number of groupso Minimize bandwidth overhead on network

• Thm: This problem is NP-completeo Reduction to Minimum Normal Set Basis

MCMD Library Layer

• Overloads the IPMC functions o setsockopt(), send(), etc.

• Translates logical IPMC addresses to physical IPMC, or point-to-point UDP packets depending on policy

• Notifies MCMD immediately about joins/leaves

• Learns about new mappings from MCMD

• Keeps statistics about group traffic rates

MCMD Library Layer

• Overloads the IPMC functions o setsockopt(), send(), etc.

• Translates logical IPMC addresses to physical IPMC, or point-to-point UDP packets depending on policy

 • Caches translation maps• Maintains a connection to MCMD for

updates

Overview

• Library module• Mapping module• Gossip layer

 • Optimization

questions • Results