A case for Passive Application Layer Multicast

A Case for Passive Application Layer

Multicast ⋆

Xiaolong Li and Aaron D. Striegel ∗

Dept. of Computer Science and Engineering

University of Notre Dame

Notre Dame, IN 46556 USA

Abstract

Over the last few years, application-layer multicast (ALM) has emerged as a plausi-ble solution for supporting group-oriented applications. However, ALM suffers frominefficiency due to its reliance on end hosts and potentially significant changes atboth the server and client applications. In this paper, we propose an approach,PALM (Passive Application Layer Multicast), that removes the need for changes tothe server or client applications while allowing for dynamic discovery of supportingnetwork devices and client-side OS modules. The passive nature of the approachcomes from the fact that bandwidth savings occur transparently with zero requiredmodifications to the networking environment, similar to approaches such as webcaching. We demonstrate the performance improvements of PALM relative to ALMand compare the benefits versus network-level multicast and separate unicast ap-proaches through both simulation and experimental studies.

Key words: Application Layer Multicast, QoS

1 Introduction

Multicast is an efficient mechanism for group-oriented applications. However,multicast has suffered tremendous deployment issues due to its global scale [1].Therefore, over the last few years, alternative group communication services

⋆ This research was supported in part by the National Science Foundation throughthe grant CNS03-47392.∗ Corresponding author

Email address: xli5, [email protected] (Xiaolong Li and Aaron D. Striegel ).

Preprint submitted to Elsevier Science 8 November 2006

such as Application Level Multicast (ALM) were developed to address this is-sue [2]. ALM protocols implement all multicast related functionality includinggroup management and packet replication at the end host. ALM can pro-vide near network-level multicast bandwidth savings without needing specialsupport from network routers.

Since replication operations under ALM are typically restricted to only the endhosts (client), the distribution tree will lengthen, resulting in a longer delayand an additional potential for loss. Furthermore, the efficiency of an ALMapproach utilizing only client-side replication is directly dependent upon therichness of the downstream clients. For most broadband solutions (cable, DSL)offered to end users, the asymmetric nature of the connection (low upload,high download) severely limits the overall efficiency of the ALM tree due tothe relatively low fan-out potential. Thus, an ALM protocol must balance thetradeoff between efficiency and QoS.

Most important, ALM requires complete support to be directly incorporatedinto the applications themselves. In contrast, approaches such as web cachingoffer efficiency with both application and the network infrastructure indepen-dence. The benefit of such dual-phased independence is the ability to offerimmediate efficiency improvements to all applications (new and legacy) aswell as the ability for optional new services that further improve efficiencyand offer compelling incentives for end users, service providers, and develop-ers. The notion of independence from both the network and the applicationprovides the motivation for this paper.

In this paper, we propose a novel approach to ALM, Passive ApplicationLayer Multicast (PALM), that removes the active involvement of the applica-tion in multicast transport. In PALM, a device in the initial source network(the Virtual Group Detection Manager) intercepts redundant data packetsand aggregates the packets into virtual groups. The virtual groups are thenmulticast using ALM as the transport mechanism between dynamically de-tected ALM-friendly helper devices (AHDs) and ALM-friendly clients (ACs).However, unlike previous ALM approaches [3, 4] that utilized helper devicesthat are known a priori by the ALM protocol, PALM will dynamically detectfriendly helper devices. Furthermore, PALM provides a seamless transitiontowards ALM in that all end clients need not support the ALM transportmechanism nor participate in the ALM process.

Most notably, our paper makes several key contributions with regards toapplication-layer multicast. First, we propose a scheme that offers benefitsnot only in the first mile/initial domain but also in the last mile and ourscheme avoids dependence on the ISP for deployment. Second, we propose ascheme for dynamically detecting helper devices without a priori knowledge.The notion of a dynamic helper device allows for new service opportunities

2

and resource competition. Third, we propose protocols for dynamically de-tecting ALM-friendly clients outside of the application data stream as well asschemes for evaluating potential ALM-friendly client to non-ALM client paths.Finally, it is our belief that an approach such as PALM can improve the appealof ALM by offering an immediate (no need for application or network conver-sion) and tangible benefit that can significantly improve scalability concernsof group-oriented applications.

The rest of the paper is organized as follows. In Section 2, we discuss relatedwork and give a brief overview about VGDM operation. In Section 3, wepresent the fundamentals of the PALM model. In Section 4 and section 5,we conduct both simulation studies and Internet experiments on PlanetLabtest bed regarding the performance of the PALM model. Finally, we presentseveral conclusions and discuss future work in Section 6.

2 Related Work

While PALM employs ALM as a transport, PALM seeks to coexist with mini-mal modification to the ALM mechanism itself. Thus, PALM can leverage therich body of existing work in ALM ranging from Narada [5] to NICE [6] as wellas many others [2]. In fact, PALM could be used with explicit multicast proto-cols such as SGM (Small Group Multicast) [7] where the end-to-end multicastinformation is embedded in the packet or approaches such as REUNITE [8]that employ recursive unicasts on a hop by hop basis.

The closest bodies of work in ALM to PALM include OMNI [4] and Scattercast[3], both of which employ helper devices to assist with ALM distribution.While OMNI partitions the network into two distribution trees, a known listof MSNs (Multicast Service Nodes) and clusters of end clients, Scattercastincludes the end host helper nodes SCXs (ScatterCast proXy) in the overalldistribution tree. In contrast, PALM does not rely on a priori knowledge ofthe helper nodes but rather introduces a discovery protocol for dynamicallydetecting and including helper devices.

Furthermore, PALM further differentiates itself from the existing body of workin ALM by virtue of its passive nature. Similar to web caching [9], PALM doesnot require any modifications to the actual applications (server or clients)themselves. In [10], the authors took caching one step further by caching thedata payload itself. The work demonstrated the correct detection of redundantpayloads through the use of Rabin fingerprints on COTS hardware. However,the use of caching works best when data shows redundancy over a long timescale but can perform poorly in non-lossless links when data is continuallychanging such as seen with streaming media.

3

As a result, our previous work in [11], stealth multicast, dynamically convertsredundant packets in close proximity to/from multicast at the edge of the do-main. Although stealth multicast removes the requirement of global deploy-ability for network-level multicast and offers benefits in the first mile/initialdomain, the benefits of stealth multicast in [11] are largely restricted to theISP. In contrast, PALM will offer the potential for bandwidth savings in theglobal Internet beyond the initial domain and in both the first and last miles.Furthermore, the use of ALM, the dynamic discovery of helper devices, andsupport for heterogeneous clients (ALM and non-ALM) introduces uniquechallenges that define our contribution in this paper.

Specific contributions of this paper include:

• Inter-domain discovery: Unlike traditional ALM protocols whereby all mem-bers directly participate, PALM takes a passive approach whereby the pres-ence of ALM itself may be hidden from certain clients to allow for partialdeployment. This transparency provides unique challenges with regards todiscovery of both willing devices/end-hosts and inter-node communications(source IP filtering, etc.) that are addressed in this paper.

• Enhanced tree construction: In a similar vein, the passive/transparent na-ture of PALM necessitates enhanced tree construction mechanisms to avoiddisruptions of service due to changes in the underlying ALM tree. We intro-duce mechanisms such as the PALM Join Completed Notification (PJCN)message and efficient encoding of tree identifications to provide robust trans-port to offer seamless bandwidth conservation. Furthermore, we address treerestrictions unique to PALM with regards to active and passive participantsin the ALM process.

• Experimental results: Through experiments conducted on PlanetLab, wedemonstrate the feasibility of PALM. Code for the various PALM compo-nents is freely available for download.

3 PALM: Passive Application Layer Multicast

The PALM model is focused on providing a readily deployable ALM infras-tructure with or without support of the application itself. Rather than forcinga complete ALM infrastructure, PALM gradually discovers ALM-capable de-vices existing in either the network (ALM helper devices (AHDs)) or at theend host itself (ALM clients (ACs)). When such devices are detected, an ALMdistribution tree is constructed and the packets are transmitted using the se-lected ALM protocol. For those end hosts that do not support ALM, thehosts will appear to receive packets as if the packets had come from the orig-inal source. In practice, either the PALM distribution tree (via an AHD orAC) or the actual source may be the one providing the unicast packet which

4

V G D M C A CA CA H D A H DA C A CCCU n i c a s t

S o u r c eA p p l i c a t i o nA L M + U n i c a s tC o n v e r s i o nt o P A L M A CCA H D A L M C l i e n tN o n � A L M C l i e n tA L M H e l p e r D e v i c eU n i c a s t A L M

E R A CA CFig. 1. The PALM framework

will be nearly indistinguishable.

Our target environment for PALM is medium scale group-oriented applica-tions that do not wish to make modifications to the servers themselves. Forapplications that can take advantage of ALM, the concept of the dynamicallydiscovered helper devices from PALM can still offer a significant benefit withregards to both delay and client bandwidth asymmetry. Specifically, the keyconcepts of the PALM framework can be subdivided into the following:

• Detection/conversion of redundant unicast: The conversion of redundantpackets occurs at the Virtual Group Detection Manager (VGDM). TheVGDM assesses the appropriate candidate packets for multicast transportand is responsible for ALM tree construction and tree management.

• Discovery of helper devices: Once a client has successfully been identifiedas a candidate for participating in the multicast, the path to the client isprobed for possible helper devices as well as client-side support for PALM.Furthermore, the paths between clients themselves (AC to AC or AC tonon-AC) as well as paths between helper devices and clients to other helperdevices may also be probed. We describe our approach for the discoveryof helper devices, client-side support, and distinguishing levels of availableclient-side support.

• Tree construction/data distribution: Unlike traditional ALM approacheswhereby all clients must join the distribution tree, PALM possesses twonew types of nodes, helper devices (AHDs) and non-ALM clients (non-ACs). We discuss the impacts of these two new types and our solution fortree construction under End System Multicast (ESM) [5].

5

3.1 PALM Overview

In the PALM model, packet transport consists of three main stages: redundantpacket payload identification and consolidation into a virtual group at theVGDM, distribution via the ALM tree to AHDs and ACs, and dispatch tothe end clients. Figure 1 shows the typical application environment for PALMwherein a server dispatches information to nine separate clients using separateunicasts. Among those clients, six of the clients support PALM (denoted byAC) with the remainder of the clients unaware of PALM. In addition, the figureshows two helper devices (AHDs) located on the shortest paths to several ofthe clients.

The conversion process itself takes place at the VGDM (Virtual Group De-tection Manager). The VGDM in PALM is located immediately following theservers themselves to minimize the effect of background traffic aggregation.

The order of events for a packet being transported via PALM is discussed inmore detail below:

(1) The application transmits packets to multiple clients with the same datapayload (L7).

(2) The unicast packets immediately arrive at the VGDM. If the source ap-plication, identified by the source IP and source port, has been identifiedas an application amenable for multicast, the packets are queued inside avirtual group. If not, the packets are sent onwards immediately. Packetswithin a virtual group share the same source IP, source port, packet size,and signature (checksum to assess uniqueness [10]).

(3) After a maximum hold time (MHT) or idle time on the input to theVGDM, the virtual group is released for transmission via the ALM trans-port mechanism. Non-ALM clients significantly separated from the ALMdistribution tree may also have their packets sent onwards using unicast.

(4) ALM packets are sent onwards towards either AHDs or ACs in the ALMdistribution tree. Packets are appropriately replicated to downstreammembers of the ALM distribution tree. If necessary, packets are repli-cated to appear as the original source for non-ALM clients.

The actual ALM protocol selected is not critical to the operation of PALM.Rather than defining how a tree should be constructed, PALM provides thelist of members (AHDs, ACs) that should receive information from the ALMdistribution tree. In this sense, PALM is able to operate with any ALM pro-tocol.

The list of potential members for the multicast group is derived from clientsthat appear consistently from the same source application (identified by sourceIP, source port). From our work in [11], a history of k bits is maintained

6

N oS t a t e P r o b eN o n 'A L MC l i e n t H a sA H D A C +A H D U p d a t eT r e eO nT r e eN o n 'A L M+A H D U p d a t eT r e eH i d d e nC h i l dU ' V G D M A L M

A CU ' V G D M U ' V G D M U ' V G D M U ' V G D MU ' V G D MU ' V G D MU ' V G D M U ' A H DU ' V G D M U ' A H D A L MT r a n s p o r t M e c h a n i s mU n i c a s t f r o mV G D M U n i c a s t f r o mA H D A L MU ' A C U n i c a s t f r o mA C

P r o b ef r o mA C H i d d e nC h i l dU ' V G D M U ' A CU p d a t eT r e eU ' V G D M

T i m e rV G D MN e wC l i e n t A C A c kA H D A c k A C A c k T i m e r T i m e r A L MA c kA L MA c kT i m e r T i m e rT i m e r

A L MA c kT i m e rV G D MF i n a l S t a t e

Fig. 2. States of clients observed from a source application

for each candidate client in the context of the source application. Once theapplication passes a minimum threshold of m out of k successful inclusions inthe virtual group, the client will officially be probed using the PALM devicediscovery protocol outlined in the next subsection. Figure 2 shows the availablestates that an observed client will undergo in the process of observation. Notethat the state for removal from the tree is not shown as all of the states cantransition to that state.

The notion of states for the observed clients introduces an important pointfor PALM and a key differentiation for this work in general. While the vir-tual group represents the immediately detected redundant packets, the actualphysical group (the distribution tree) may be different from the virtual group.Packets that are not currently on the distribution tree will be sent using uni-cast (as if the VGDM did not exist) while clients in the physical group but notin the virtual group will have the delivery of the specific packet squelched. Themechanisms for data delivery are discussed in more detail in a later subsection.Finally, it is also important to note source applications must meet a minimumthreshold of consistent clients, in order to be even considered for PALM inthe first place. The analysis of this threshold is done via a background anal-ysis rather than actively delaying all packets. Thus, normal unicast packetswill not be temporarily queued for potential virtual group transmission un-til the source application exhibits an appropriate level of data redundancy.Additional details on the queuing and QoS impact of VGDM may be foundin [11].

7

3.2 PALM Device Discovery Protocol (PALM-DDP)

Unlike traditional ALM protocols or proxy-based solutions whereby the ap-plication is an active participant, the VGDM operates passively (applicationunaware) and can only observe potential clients by virtue of redundancy inthe outgoing traffic. Furthermore, since the conversion point back from PALMcan only occur at either a helper device (AHD) or PALM-friendly client (AC),PALM must discover the presence of such devices in the network.

To that end, we propose a discovery protocol for PALM to detect the presenceof AHDs or ACs. Since the VGDM is central point for the initial conversion toPALM, all initial probe messages and probe results will be passed back to theVGDM to provide a consistent picture of the AHDs and ACs available for useon the clients of each source application. We define an AHD and an AC as adevice that supports all types of PALM messages. The AHD will exist in thenetwork on the path to the clients and has the ability to replicate packets viaboth the ALM distribution tree (to ACs) and via a spoofed unicast to makethe data packet appear nearly identical (minus TTL and other appropriatechanges) to the packet originating from the server (for non-PALM clients).An AC is a PALM-friendly client that has the ability to replicate packetsto other downstream ACs and potentially has the ability to send to non-ACclients.

The probing operation begins once a client has been identified at the VGDMas a new client. For the purpose of simplicity, we view the probing operationsbetween application groups (the clients identified under a specific source ap-plication) as being independent. The probe packet (PROBE) is sent from theVGDM to the potential client as a UDP packet to a pre-defined PALM des-tination port. Both AHDs and ACs are responsible for listening for PROBE

packet from PALM and responding with a PROBEACK to the VGDM. Forthe AC, the packet will simply arrive via the network stack. In contrast, anAHD will need to promiscuously snoop for PROBE packets since it will notbe an explicit recipient of the PROBE packet. The result of a probe mayresult in one or more of the following outcomes (see Figure 3):

• AHD response: An AHD notices the probe and responds to the VGDM.The AHD is noted as being on the path to this potential client.

• AC response: The potential client is a PALM-friendly client.• No response: The lack of a response may denote either the lack of a PALM-

friendly client or the probe packet being lost. The probe packet will beresent periodically up to R times.

• ICMP response: A PALM-friendly client does not exist at the client and noprobe packets need to be resent as an ICMP error message was received.

8

V G D M n o n F A CP R O B ET i m e o u tC a s e I : N o n ] A C ] N o R e s p o n s e V G D M n o n F A CP R O B E

I C M P E r r o r M e s s a g eC a s e I I : N o n ] A C ] E r r o r M e s s a g eV G D M A CP R O B EP R O B E F A C K

C a s e I I I : A C o n l y V G D M A CA H DP R O B E F A C KP R O B E F A C KP R O B E P R O B EC a s e I V : A C + A H D

Fig. 3. Probe packet example

An important observation on the use of a unicast probe packet is that the pathshould be the same path as taken from the source application to the client.Thus, one can reasonably assume that any AHDs detected on the path to thepotential client should contribute to the overall health of the ALM distributiontree. At worst case, the potential downstream client is a non-ALM client andwill be the only client receiving packets from the AHD. At best case, the newlydetected AHD offers a significant efficiency advantage to the ALM group andcan serve multiple downstream ALM or non-ALM clients.

AHD Deployment Incentives: We view an AHD as being deployed inseveral high probability locations. First, a local broadband service providermight place an AHD just before the downstream clients. Although the locationAHD would not improve the last mile (the final hop before the end user), itwould offer a considerable incentive to the local provider. Similar to how localweb caching reduces redundant queries for the same material, a local AHDwould allow the source VGDM to stream only a single packet regardless ofthe number of downstream clients. Furthermore, the content under PALM isalways live and thus tradeoff of storage size versus cost does not exist.

Second, a company might place the combination of a VGDM and AHD ateach office. Thus, without modifications to the applications themselves, thecompany could realize immediate bandwidth savings for group-oriented appli-cations. This savings could be further augmented by the third case whereby acore ISP offers AHD services to select customers. Rather than responding toall PROBE packets, an AHD could be tuned to respond to PROBE packetsfrom select sources with an appropriate digital signature (to avoid improperresource allocation). Most notably, PALM offers deployment options rangingfrom only the end hosts to optional value-added services that can be offeredby the ISPs with appropriate economic incentives.

AC to non-AC Discovery: While it is not required for an AC to be able to

9

transmit to a non-ALM client, the ability to do so could offer significant savingsin the absence of AHDs in the network. The primary challenge associated withAC to non-ALM client transmissions lies in the fact that the non-ALM client isexpecting packets originating only from the source application. Since the non-ALM client has no knowledge of PALM, an AC to non-ALM client must beable to effectively spoof the packet as originating from the source application.

While the use of spoofing has a noble goal of increasing the overall systemefficiency, it creates a dilemma with regards to security. Given the increasingpresence of ingress/egress filtering in the Internet, one cannot assume that theability to send a spoofed packet from the AC as the source application willeven exist. For the AHD, we assume that such a capability will exist as it willbe on the natural path and hence have no difficulty with the ingress/egressfiltering.

In order to detect the ability to send AC to non-ALM clients, we propose thefollowing protocol (see Figure 4):

(1) The upstream AC sends an ICMP ping message with the VGDM IP asthe source IP. The upstream AC embeds a digital signature within thepayload of the ping packet.

(2) The downstream non-ALM client receives the ping which appears to haveoriginated from the VGDM and sends a response to the VGDM.

(3) The VGDM receives the ping message and notes the capability of theupstream AC to successfully spoof downstream packets to the non-ALMclient.

In the event that ingress filtering is being employed, the ping message will notreach the non-ALM client itself. The ability to spoof AC to non-ALM clientsis handled by the inclusion of a digital signature for the upstream AC thatwill be reflected to the VGDM upon the successful ICMP Ping Response. AnICMP message is used since it is the easiest protocol that can be successfullyredirected without knowledge of PALM. While the use of ICMP is not optimaldue to many sites squelching ICMP messages, it represents a relatively easyway to test to for AC to non-ALM client connectivity. Since the use of thisoptional probing may also be complicated by the presence of NAT devices orother firewalls, the ability to test for AC to non-ALM clients is an optionalfeature of PALM.

3.3 Tree construction and data distribution

For this paper, we present tree construction in the context of ESM (EndSystem Multicast) [5]. Once the discovery of AHDs and ACs are complete fora minimal set of clients, the next step is to construct a distribution tree using

10

V G D M A Cn o n zA C

I C M P P i n gS : 1 2 9 . 7 4 . 2 5 . 1 0 8D : 2 4 . 7 . 5 . 4P a y l o a d : D i g i t a l S i g n a t u r ef o r 5 4 . 2 7 . 8 . 2 9I C M P R e s p o n s eS : 2 4 . 7 . 5 . 4D : 1 2 9 . 7 4 . 2 5 . 1 0 8P a y l o a d : D i g i t a l S i g n a t u r ef o r 5 4 . 2 7 . 8 . 2 9 234P A L M T r e e U p d a t eS : 1 2 9 . 7 4 . 2 5 . 1 0 8D : 5 4 . 2 7 . 8 . 2 9A d d h i d d e n c l i e n t2 4 . 7 . 5 . 4D i g i t a l S i g n a t u r e f o r1 2 9 . 7 4 . 2 5 . 1 0 8 5 4 . 2 7 . 8 . 2 9

2 4 . 7 . 5 . 4

P A L M R e q u e s t P r o b eS : 1 2 9 . 7 4 . 2 5 . 1 0 8D : 5 4 . 2 7 . 8 . 2 9P r o b e 2 4 . 7 . 5 . 4D i g i t a l S i g n a t u r e f o r1 2 9 . 7 4 . 2 5 . 1 0 811 2 9 . 7 4 . 2 5 . 1 0 8Fig. 4. AC to non-ALM client capability discovery

V G D M A CA CA H DI P R o u t i n g P a t h A L M T r e eFig. 5. Avoiding AHD Leafs

the list of AHDs and ACs. In essence, the dissemination by the VGDM of thelist of initial members (AHDs/ACs) is equivalent to the bootstrap mechanismfor ESM undefined in [5]. Beyond the ESM tree construction mechanisms, wemake the following restrictions:

• All joins/leaves are assumed to originate from the VGDM and cannot orig-inate from a client.

• All data transmissions have a sequence number and duplicate packets willnot be forwarded by a member of the tree.

• An AHD should not be a leaf on the distribution tree without downstreamnon-ALM clients.

• Non-ALM clients may only receive packets from an AHD or the VGDMitself.

While the initial tree construction relies entirely on ESM, the issue of treeconstruction after the initial represents a critical aspect of PALM. Unlike ESMwhereby all end hosts are part of the tree and are aware of ESM, PALM maypossess AHDs and non-ALM clients. The issue of non-ALM clients is madesignificantly easier by restricting non-ALM clients to exist only as childrenof the VGDM or AHDs. The detail of inclusion of AC to non-ALM clients isbeyond the scope of this paper and is an open topic for research due to theintricacies associated with path-wise amenability (delay measurements withoutALM and reachability from AC to non-ALM clients).

11

Avoiding AHD Leafs: Since an AHD is only a helper device and is notactually a consumer of the data itself (except when sending to a non-ALMclient), it should not be a leaf on the distribution tree (see Figure 5). Pastapproaches such as OMNI and Scattercast solved this problem by sub-dividingthe tree construction process, a tree of the helper devices is constructed firstfollowed by clusters of devices downstream from the helper device. Thus, ineither approach, a helper device could never become a leaf with no downstreamreceivers.

Although it is possible that an AHD could become a leaf with PALM, it isextremely unlikely due to how AHDs become included in the tree in the firstplace. Since an AHD must lie on the path from the source application/ VGDMto the client in the first place, it is reasonable to assume in the current Internetthat it is highly unlikely that a shorter path will exist elsewhere to the VGDM.While optimizing the tree for overall efficiency may result in an AHD as a leaf,we believe the scenario outlined earlier would be highly unlikely to introducesuch a case.

In the event that the AHD does appear as a leaf without a downstream non-ALM client, the data distribution mechanism can be used to squelch packetsbefore they are transmitted to the leaf AHD. Alternatively, a leave may be sentto the AHD to cause it to leave the group although group dynamics (futurejoins/leaves) may make the AHD a contributor again to the efficiency of thetree. In practice, this would happen only if there are multiple AHDs existing ona single path from the VGDM to the client. In this case, the following steps areused to identify AHD leaf: Once ALM protocol underneath PALM converged,all the leafs send a PALM Notification Message to VGDM. Since VGDMknows which AHD has downstream non-ALM client or not, it is VGDM thatdetermines which leaf is AHD leaf.

Finally, it is noteworthy to comment on why AHDs have only been includedon the path between the source application/VGDM and the client. While theprobability of an AHD acting as a leaf if it appears on the shortest path fromthe source application is relatively low, the same cannot be said if AHDs arediscovered between entities not on the shortest path. For instance, supposethat AHDs probe the paths between one another for other AHDs or probethe paths to the various ACs or non-ALM clients. In such a case, it would beextremely difficult to assess which AHDs would be contributing to the overallefficiency of the distribution tree in a relatively quick manner. The issues ofassessing non-shortest path AHDs are left open as a topic for future research.

Data Distribution: Unlike ESM whereby all end hosts will always receivepackets originating from the source, there is no guarantee that a client that iscurrently a member of the distribution tree should receive a packet within aspecific group. For instance, suppose that A, B, C, and D have been consis-

12

V G D M A H D A Cn o n Ì A CP A L M1 2 9 . 7 4 . 5 . 1 0

1 2 9 . 7 4 . 3 . 5 0I P P A L MA L M D a t aT N U M B i t m a p5 1 0 1 1 1 T r e e N u m( T N U M ) B i t m a p K e y45 0 : 1 2 9 . 7 4 . 3 . 5 0 n o n Ì A C1 : 1 2 9 . 7 4 . 5 . 1 0 A C2 : 1 2 9 . 7 4 . 6 . 1 0 A C 3 : T h i s N o d e A H D4 : 1 4 0 . 6 5 . 1 0 . 4 A C. . .

Fig. 6. Data distribution with PALM

tently detected in the virtual group and are on the physical tree. A exits theapplication (unknown by the VGDM) and stops receiving packets. The nexttransmission from source will send packets to only B, C, and D. The dilemmaat the VGDM is to determine if the client has truly stopped receiving informa-tion from the source or if the lack of a packet is temporary (server overload,etc.). Since the VGDM is located very close to the source, it is relatively safeto assume that the client has left the group.

However, until A officially is purged from the physical group, the packets mustbe appropriately marked to denote this fact. Thus, a bitmap of N bits whereN is the number of total members in the physical group (AHDs, ACs, andnon-ALM clients) is included in the packet. The bitmap is accompanied by aTNUM that is used to reference a brief history of past client lists for clients onthe physical tree (see Figure 6). Clients are ordered by an assigned namingconvention included in the original client list (bootstrap mechanism). Once A

has been purged from the physical tree, a new list of clients and inherentlya new key for the bitmap is reliably distributed to each of the clients on thephysical tree.

Since PALM acts passively and does ALM distribution transparently, an im-portant issue is how to keep data distribution tree consistent. Namely, packetloss should not occur when switching from unicast to ALM or from changesin the ALM tree (join/leave). We guarantee this by using “PALM Join Com-pleted Notification” (PJCN) message. The PJCN message is used to indicatethat the ESM join has completed successfully. The PALM will not change thecurrent distribution tree until the PJCN message from the target AHD/AC isreceived. The PALM join procedure is discussed in more detail below:

• The VGDM sends a join command to the AHD/AC to have it join the ESMtree. Before a PJCN message arrives, the VGDM keeps sending unicastpackets to the associated end clients.

• Once the AHD/AC receives a join command, an ESM join is immediatelyinitialized. After the ESM join completes, the AHD/AC is ready to dis-

13

tribute PALM packets. The AHD/AC then sends a PJCN message to theVGDM. Packets received by the new joined AHD/AC before the VGDMprocesses the PJCN are silently discarded by virtue of the PALM bitmap.

• After the VGDM receives PJCN message, PALM updates the bitmap andsends a new bitmap to all members of the group. The previous bitmap willbe used until VGDM receives ack messages from all members. PALM willthen convert the unicast packet for transmission by the new distributiontree.

A final critical aspect of data distribution is that non-ALM clients must notreceive multiple copies of the same packet. While ACs will have the sequencenumber mechanism to parse duplicates, a non-ALM client will have no methodto distinguish the arrival of a duplicate packet. Our proposed data distributionmechanism ensures this will not occur.

AHD Deployment: Although AHDs are on the path from the VGDM to theclient, they are not necessarily deployed as a router. In PALM, AHDs functionpassively to sniff PALM probe packets. Neither router functionality nor beingplaced in front of the router is necessary. From the network administrator’sperspective, an AHD is just like an additional device, addition or removal ofwhich does not affect the network management since it is completely trans-parent. The VGDM is only concerned if there is an AHD on the path, ratherthan the location of the AHD.

4 Simulation Studies

In this section, we present simulation studies of the PALM model. The simu-lations were conducted using the ns-2 simulator [12]. The goal of our studieswas to examine the performance implications of PALM with regards to otherapproaches (ESM, unicast) as well as the impact on end user QoS.

The rationale behind our simulations was the following. In the network, Com-pany X hosts an on-line gaming service with applications serving up to 64clients. The applications are hosted on a set of servers with the clients exist-ing throughout the global Internet. The PALM VGDM is placed immediatelyafter the servers, thus converting traffic before the uplink to the global Inter-net. The parameters of the simulation were as follows:

• The simulations were conducted on a random network with a topology con-sisting of 32 core nodes and 16 edge nodes and an average degree of 3. Asufficient bandwidth (1 Gb/s) is used to ensure that packet losses do notoccur in the network.

• The server farm consisted of 10 separate server sources.

14

• The average number of clients listening to a server application was 32 clientsrandomly distributed amongst the edge nodes.

• Each join or leave (client joining or leaving) was exponentially distributedwith an average inter-arrival event time of 500 ms for all clients in thesimulation. The join/leave of the client was not conveyed to VGDM andobservations on joins and leaves of the clients were derived using trafficpatterns.

• The server applications sent data using UDP packets with an exponentiallydistributed packet rate of 50 ms and a packet size exponentially distributedwith a mean of 700 bytes. For simplicity, the packets were streamed usinga constant size unique to each application.

• The settings used for the VGDM are available in [11].

The primary purpose of the simulations was to evaluate the key performancemetrics for PALM (end-to-end delay and bandwidth consumption):

• Bandwidth utilization: The bandwidth consumption of the traffic across theentire network is examined. All traffic is assumed to be of equal cost on thenetwork.

• End-user QoS: The effects on end-user QoS were examined to determine theimpact that the AHDs and the initial queuing at the VGDM were havingon the user flow.

In our simulation studies, we compare the following approaches:

• Unicast: Packets are sent out using a standard IP unicast. No effort is madeto increase the efficiency of the overall distribution.

• ESM: The application itself participates in the ESM process. No helperdevices are present in the network and a pure end host approach is taken. Ageneric ESM protocol is used with each client having the capacity to streamto 5 other downstream clients.

• PALM-ESM: The PALM protocol is employed with ESM as the ALM trans-port mechanism. A generic ESM protocol is used with each client having thecapacity to stream to 5 other downstream clients. Unless otherwise speci-fied, it is assumed that 20% of the domain nodes are AHDs and 100% ofthe clients are ACs to support a fair comparison with ESM.

The settings for PALM were as follows:

• AHDs were allowed to replicate to any ACs and non-ACs, regardless ofwhether or not the AHD lies on the path to the device.

• AHDs were assumed to have an unlimited capacity for replication.• ACs were allowed to replicate to non-ACs.

15

0

5

10

15

20

25

30

10 20 30 40 50 60

Avera

ge lin

k ban

dwidt

h (MB

/s)

Average number of clients per app

ESMUnicast

PALM-ESM

Fig. 7. Effect of the number of clientsper application on bandwidth

30

35

40

45

50

55

60

10 20 30 40 50 60

End-t

o-end

delay

(ms)

Average number of clients per app

ESMUnicast

PALM-ESM

Fig. 8. Effect of the number of clients perapplication on average end-to-end delay

4.1 Effect of Number of Clients

Figure 7 shows the effect on the total bandwidth consumption as the averagenumber of clients per application is varied from 8 to 64. Figure 8 shows theeffect on delay under the same conditions. First, it is important to note that theend-to-end delay in PALM is not significantly impacted by the initial queuingapplied by the VGDM. In fact, the inclusion of AHDs allows PALM to reducethe average end-to-end client delay beyond that of ESM when sufficient clientsexist in the application. Furthermore, the actual delay impact versus unicastis fairly insignificant even with only 20% of the nodes in the core networkpotentially offering AHD services. The inclusion of the AHD nodes effectivelyoffsets queuing penalty of the VGDM.

As would be expected, the bandwidth follows an appropriate trend wherebyunicast performs significantly worse than both ESM and PALM. With PALM’sability to use AHDs, PALM is able to achieve a cost savings versus ESM,despite the fact that PALM incurs a slight overhead versus ESM. Furthermore,it is important to note that this performance improvement comes with thebenefit of avoiding modifications to the initial server applications. Thus, we feelthat with a sufficient client base, the benefit of PALM can be quite compellingdespite its relatively minor delay penalties versus separate unicasts.

4.2 Effect of AHDs

Figure 9 shows the performance of PALM as the percentage of AHDs in thenetwork is varied. At 0%, no AHDs are present while at 100%, the networkwill have an AHD on every single hop to the client. The interesting point to

16

0

5

10

15

20

25

30

0 20 40 60 80 100

Avera

ge lin

k ban

dwidt

h (MB

/s)

Percentage of AHDs in ISP domain

ESMUnicast

PALM-ESM

Fig. 9. Effect of the percentage of AHDsrelative to the number of ISP nodes onbandwidth

30

35

40

45

50

55

60

0 20 40 60 80 100

End-t

o-end

delay

(ms)

Percentage of AHDs in ISP domain

ESMUnicast

PALM-ESM

Fig. 10. Effect of the percentage ofAHDs relative to the number of ISPnodes on average end-to-end delay

make is that once a minimum threshold has been passed (causing the AHD tobe detected), additional AHDs offer little benefit. The reason for this is two-fold. First, each additional AHD will consume an additional bit in the bitmapheader. Thus, as the number of AHDs increase, the overhead for PALM willalso increase, albeit fairly slowly. Second, the AHD achieves its primary goalwhen the VGDM is able to unicast a packet a significant distance to the AHD.In the event that multiple AHDs appear consecutively on a path to a client, noadditional benefit is realized. In fact, a small penalty will be assessed due tothe additional processing required at each node. The graph in Figure 10 showsa similar trend as the delay does not show a significant improvement once aminimum threshold has been passed for the same reasons that the bandwidthdid not improve.

4.3 Effect of ACs

The effect of additional ACs in the client base is much more significant thanthe effect of AHDs. In both Figures 11 and 12, the number of AHDs is setto zero percent to demonstrate the full effect of changing the percentage ofACs. Unlike AHDs where the effect of consecutive locations removes theirbenefit, the addition of new ACs represents potential clients that can serve asreplication points from the source. Thus, while the overall bandwidth decreaseswith additional ACs from the downstream replication savings, the average end-to-end delay increases to reflect the lengthening of the distribution tree. Mostimportant, it is interesting to note that even at a fairly low percentage of ACs(30%), PALM can still realize most of its bandwidth savings.

17

0

5

10

15

20

25

30

0 20 40 60 80 100

Avera

ge lin

k ban

dwidt

h (MB

/s)

Percentage of clients that are ACs

ESMUnicast

PALM-ESM

Fig. 11. Effect of the percentage ofPALM-friendly ACs on bandwidth

30

35

40

45

50

55

60

0 20 40 60 80 100

End-t

o-end

delay

(ms)

Percentage of clients that are ACs

ESMUnicast

PALM-ESM

Fig. 12. Effect of the percentage ofPALM-friendly ACs on average end–to-end delay

5 Internet Evaluation

For validating our simulation results in a real network environment, we con-ducted experiments on a PlanetLab testbed with 21 hosts. Our experimentswere conducted based on our implementation of PALM and an implementationof ESM.

We setup an overlay topology for our experiments with 21 hosts (see Figure13). 15 of the hosts are clients in which 70% are specified as ACs and 30%are non-ALM clients. The rest of the hosts functioned as routers or AHDs. Inorder to better mimic Internet behavior, we choose hosts located at same siteas routers which generally offer the low latency/delay links to each other. Thebasic scenario of the experiments is as follows, a server offers a distributionstream at a rate of 12kB/s with the first 8 clients joining one by one every 20seconds. Following a delay of 100 seconds, the rest of clients (total of 15 clients)join following the same pattern. Figure 13 shows our experiment environment.

5.1 Experiment Methodology

Similar to our simulation studies, we compared the performance of PALM,ESM and unicast in terms of bandwidth consumption, end-to-end delay. Herewe mainly focus on the bandwidth usage of the bottleneck link which is the linkfrom server to the first egress router. The experiments were conducted multipletimes with the resulting averages presented in the figures. Each experimentlasts for 10 minutes and was run 3 times.

18

VGDM Server

Router/AHD

Server

VGDM

AC / Non-ALM Client

Bottleneck Link

S V R R

R

R C

C C

C

C

C

C

C

C

C

C

C

C C C

R

S

V

C

Fig. 13. Experiment Topology

5.2 Experiments Results

0 100 200 300 400 500 600Time(seconds)

0

50

100

150

200

250

Av

erag

e b

and

wid

th (

kB

/s)

PALMESMunicast

Fig. 14. Bottleneck Link: Average band-width usage over time

0 100 200 300 400 500 600Time (seconds)

1000

1500

2000

2500

Av

erag

e en

d t

o e

nd

del

ay (

ms)

PALMESMunicast

Fig. 15. Single Client: Average end toend delay over time

Figure 14 shows the bandwidth usage at the bottleneck link as a functionof time. Similar to our simulation results, the bandwidth usage for unicastshows a linear increase as more clients join the group, while the PALM andESM almost keep a relatively steady bandwidth usage. The reason why PALMconsumes more bandwidth than ESM is that PALM handles heterogeneousclients, namely, ALM-enabled nodes (ACs, AHDs) and non-ALM clients. Inour experiments for PALM, there are 4 non-ALM clients. Although ESM con-sumes less bandwidth, we should be aware of that ESM cannot offer savingswhen clients do not directly participate in the ALM process. As long as thenew client is an AC or could benefit from AHDs, the bandwidth usage overbottleneck link will not increase significantly. Hence, with additional PALM-amenable clients, the more the bandwidth that could be saved. For ISPs withlimited egress bandwidth, PALM would be an immediately attractive solutionwith its unique nature of zero changes to the applications themselves.

19

Finally, Figure 15 shows the same trend as our simulation results. AlthoughPALM introduces extra delay for identifying redundant packets, it achievesalmost the same end to end delay as ESM does, and not significantly morethan naive unicast does 1 . Compared with high bandwidth stress and packetloss rates for unicast, we believe additional end to end delay is more desirablefor most applications.

6 Summary

In this paper, we proposed the Passive Application Layer Multicast (PALM)model as a new approach to Application Layer Multicast (ALM). Unlike tra-ditional ALM, we demonstrated how PALM does not require modifications tothe applications themselves or a priori knowledge of in-path helper devices.Furthermore, PALM introduces interesting new service scenarios whereby endusers can either utilize PALM independently or leverage the ISP infrastructuredynamically for improved efficiency.

In this paper, we proposed dynamic discovery protocols for both helper de-vices and non-ALM client conversion capabilities at the client. We also demon-strated how PALM can be combined with an existing approach such as ESMto yield support for heterogeneous clients and dynamically changing data dis-tribution needs (partial group transmissions). Finally, we presented simulationand experimental studies demonstrating a negligible delay penalty with im-provements in almost all cases versus ALM and separate unicasts. Thus, webelieve PALM offers a compelling and complementary new approach to ALMthat merits future attention.

References

[1] C. Diot, B. Levine, B. Lyles, H. Kassem, D. Balensiefen, Deployment issues forIP multicast service and architecture, IEEE Network (2000) 78–89.

[2] A. El-Sayed, V. Roca, L. Mathy, A survey of alternative group communicationservices, IEEE Network 17 (1) (2003) 46–51.

[3] Y. Chawathe, Scattercast: an adaptable broadcast distribution framework,Multimedia Systems 9 (2003) 104–118.

1 The high delay in the experiments results from the usage of the overlay on Plan-etLab.

20

[4] S. Banerjee, C. Kommareddy, K. Kar, B. Bhattacharjee, S. Khuller,Construction of an efficient overlay multicast infrastructure for real-timeapplications, in: Proc. of IEEE INFOCOM’03, 2003.

[5] Y. Chu, S. G. Rao, S. Seshan, H. Zhang, A case for end system multicast,IEEE Journal on Selected Areas in Communication (JSAC), Special Issue onNetworking Support for Multicast.

[6] S. Banerjee, B. Bhattacharje, C. Kommareddy, Scalable application layermulticast, in: Proc. of ACM SIGCOMM’02, 2002.

[7] R. Boivie, A new multicast scheme for small groups, IBM Research ReportRC21512.

[8] I. Stoica, T. Ng, H. Zhang, REUNITE: A Recursive Unicast Approach forMulticast, in: Proc. of IEEE INFOCOM, 1999.

[9] B. Krishnamurthy, C. E. Wills, Proxy cache coherency and replacement -towards a more complete picture, in: International Conference on DistributedComputing Systems, 1999, pp. 332–339.

[10] N. T. Spring, D. Wetherall, A protocol independent technique for eliminatingredundant network traffic, in: Proc. of the 2000 ACM SIGCOMM Conference,Stockholm, Sweden, 2000.

[11] D. Salyers, A. Striegel, A novel approach to transparent bandwidthconservation, in: Proc. of IFIP Networking, Waterloo, Canada, 2005.

[12] UCB/LBNL/VINT Network Simulator - ns (version 2)Available atwww.mash.cs.berkeley.edu/ns/.

21