Applying Patterns to Develop a Pluggable Protocols ... · speed protocols and networks, real-time embedded system in-terconnects, and standard TCP/IP protocols over the Internet.

Applying Patterns to Develop a PluggableProtocols Framework for ORB Middleware

Douglas C. Schmidt, Carlos O’Ryan, and Ossama Othman Fred Kuhns and Jeff Parsonsfschmidt,coryan,[email protected] ffredk,[email protected]

Electrical & Computer Department of Computer ScienceEngineering Department Washington University

University of California, Irvine, USA St. Louis, MO, USA�

This paper appeared as a chapter in the bookDesign Pat-terns in Communications, (Linda Rising, ed.), CambridgeUniversity Press, 2000. An abridged version appeard in theIFIP/ACM Middleware 2000 Conference, Pallisades, NewYork, April 3-7, 2000.

Abstract

To be an effective platform for performance-sensitive ap-plications, off-the-shelf CORBA middleware must preservethe communication-layer quality of service (QoS) propertiesof applications end-to-end. However, the standard CORBAGIOP/IIOP interoperability protocols are not well-suited forapplications with stringent message footprint size, latency,and jitter requirements. It is essential, therefore, to developstandard pluggable protocols frameworks that allow custommessaging and transport protocols to be configured flexiblyand used transparently by applications.

This paper provides three contributions to the study ofpluggable protocols frameworks for performance-sensitiveCORBA middleware. First, we outline the key design chal-lenges faced by pluggable protocols developers. Second, wedescribe how we resolved these challenges by developing apluggable protocols framework for TAO, which is our high-performance, real-time CORBA-compliant ORB. Third, wepresent the results of benchmarks that pinpoint the impact ofTAO’s pluggable protocols framework on its end-to-end effi-ciency and predictability.

Our results demonstrate how the application of optimiza-tions and patterns to CORBA middleware can yield both highlyflexible/reusable designs and highly efficient/predictable im-plementations. These results illustrate that (1) CORBA mid-dleware performance is largely an implementation detail and(2) the next-generation of optimized, standards-based CORBAmiddleware can replace many ad hoc and proprietary solu-tions.

�This work was supported in part by ATD, BBN, Boeing, Cisco, DARPAcontract 9701516, Motorola Commercial Government and Industrial Solu-tions Sector, Motorola Laboratories, Siemens, and Sprint.

1 Introduction

Standard CORBA middleware now available off-the-shelf al-lows clients to invoke operations on distributed componentswithout concern for component location, programming lan-guage, OS platform, communication protocols and intercon-nects, or hardware [1]. However, conventional off-the-shelfCORBA middleware generally lacks (1) support for QoS spec-ification and enforcement, (2) integration with high-speed net-working technology, and (3) efficiency, predictability, andscalability optimizations [2]. These omissions have limitedthe rate at which performance-sensitive applications, suchas video-on-demand, teleconferencing, and avionics missioncomputing, have been developed to leverage advances inCORBA middleware.

To address the shortcomings of CORBA middleware men-tioned above, we have developedThe ACE ORB(TAO) [2],which is an open-source,1 standards-based, high-performance,real-time ORB endsystem CORBA middleware that supportsapplications with deterministic and statistical QoS require-ments, as well as “best-effort” requirements. This paper fo-cuses on the design and implementation of apluggable proto-cols frameworkthat can efficiently and flexibly support high-speed protocols and networks, real-time embedded system in-terconnects, and standard TCP/IP protocols over the Internet.

At the heart of TAO’s pluggable protocols framework is itspatterns-oriented OO design [3, 4], which decouples TAO’sORB messaging and transport interfaces from its transport-specific protocol components. This design allows customORB messaging and transport protocols to be configured flex-ibly and used transparently by CORBA applications. For ex-ample, if ORBs communicate over a high-speed networkinginfrastructure, such as ATM AAL5 or specialized protocolslike HPPI, then simpler ORB messaging and transport proto-cols can be configured to optimize unnecessary features andoverhead of the standard CORBA General Inter-ORB Proto-

1TAO is available atwww.cs.wustl.edu/ �schmidt/TAO.html .

1

col (GIOP) and Internet Inter-ORB Protocol (IIOP). Likewise,TAO’s pluggable protocols framework makes it straightfor-ward to support customized embedded system interconnects,such as CompactPCI or VME, under the standard CORBAGeneral Inter-ORB Protocol (GIOP).

For OO researchers and practitioners, the results in this pa-per provide two important contributions:

1. We demonstrate empirically that the ability ofstandards-based CORBA middleware to support high-performance, real-time systems is largely animplementationdetail, rather than an inherent liability,e.g.:

� TAO’s end-to-end one-way latency overhead is only�110 �secs using commercial off-the-self 200 MHzPowerPCs, a 320 Mbps VMEbus, and VxWorks.

� The overall roundtrip latency of a TAO two-way methodinvocation using the standard inter-ORB protocol and us-ing a commercial, off-the-self Pentium II Xeon 400 MHzworkstation running in loopback mode is�189 �secs.The ORB middleware accounts for approximately 48%or�90�secs of the total roundtrip latency.

� Using the specialized POSIX local IPC protocol reducesroundtrip latency to�125�secs.

These results are as fast, or faster, than manyad hoc, pro-prietary solutions, thereby motivating the use of well-tuned,standards-based CORBA middleware, even for real-time em-bedded applications with stringent QoS requirements.

2. We explore how patterns can be applied to resolve keydesign challenges. TAO’s pattern-oriented OO design can beextended to other pluggable protocols frameworks, either instandard middleware or in distributed applications using pro-prietary middleware.

The remainder of this paper is organized as follows: Sec-tion 2 outlines the CORBA protocol interoperability archi-tecture; Section 3 motivates the requirements for standardCORBA pluggable protocols and outlines TAO’s pluggableprotocols framework; Section 4 describes the patterns thatguide the architecture of TAO’s pluggable protocols frame-work and resolve key design challenges. Section 5 illustratesthe performance characteristics of TAO’s pluggable protocolsframework; Section 6 compares TAO with related work; andSection 7 presents concluding remarks.

2 Overview of the CORBA Protocol In-teroperability Architecture

The CORBA specification [5] defines an architecture for ORBinteroperability. Although a complete description of the model

is beyond the scope of this paper, this section outlines theportions of the CORBA specification that are relevant to ourpresent topic,i.e., object addressing and inter-ORB protocols.

CORBA Object addressing: To identify objects, CORBAdefines a generic format called the Interoperable Object Ref-erence (IOR). An object reference identifies one instance of anobject and associates one or more paths by which that objectcan be accessed. The same object may be located by differentobject references,e.g., if a server is re-started on a new port ormigrated to another host. Likewise, multiple server locationscan be referenced by one IOR,e.g., if a server has multiple net-work interfaces connecting it to distinct networks, there maybe multiple network addresses.

References to server locations are calledprofiles. A pro-file provides an opaque, protocol-specific representation of anobject location. Profiles can be used to annotate the server lo-cation with QoS information, such as the priority of the threadserving each endpoint or redundant addresses to increase fault-tolerance.

CORBA protocol model: CORBA Inter-ORB Protocols(IOP)s support the interoperability between ORB endsystems.IOPs define data representation formats and ORB messagingprotocol specifications that can be mapped onto standard orcustomized transport protocols. Regardless of the choice ofORB messaging or transport protocol, however, the same stan-dard CORBA programming model is exposed to the appli-cation developers. Figure 1 shows the relationships betweenthese various components and layers.

ORB MESSAGING

COMPONENT

ORB TRANSPORT

ADAPTER COMPONENT

TRANSPORT LAYER

NETWORK LAYER

GIOP

IIOP

TCP

IP

VME

DRIVER

AAL 5

ATM

GIOPLITE

VME-IOP

ESIOP

ATM -IOPRELIABLE

SEQUENCED

PROTOCOL CONFIGURATIONS

STANDARD CORBA PROGRAMMING API

Figure 1: Relationship Between CORBA Inter-ORB Protocolsand Transport-specific Mappings

In the CORBA protocol interoperability architecture, thestandardGeneral Inter-ORB Protocol(GIOP) is defined bythe CORBA specification [5]. In addition, CORBA definesa TCP/IP mapping of GIOP, which is called theInternet Inter-ORB Protocol(IIOP). ORBs must support IIOP to be “interop-

2

erability compliant.” Other mappings of GIOP onto differenttransport protocols are allowed by the specification, as are dif-ferent inter-ORB protocols, which are known asEnvironmentSpecific Inter-ORB Protocols(ESIOP)s.

Regardless of whether GIOP or an ESIOP is used, aCORBA IOP must define a data representation, an ORB mes-sage format, an ORB transport protocol or transport protocoladapter, and an object addressing format. For example, theGIOP specification consists of the following:

� A Common Data Representation (CDR) definition:CDR is a transfer syntax that maps IDL types from their nativehost format to a low-levelbi-canonicalrepresentation, whichsupports both little-endian and big-endian formats. CDR-encoded messages are used to transmit CORBA requests andserver responses across a network. All IDL data types are mar-shaled using the CDR syntax into anencapsulation, which isan octet stream that holds marshaled data.

� GIOP message formats: The GIOP specification de-fines seven types of messages that send requests, receivereplies, locate objects, and manage communication channels.The following table lists the seven types of messages in GIOP1.02 and the permissible originators of each type:

Message Type Originator ValueRequest Client 0Reply Server 1CancelRequest Client 2LocateRequest Client 3LocateReply Server 4CloseConnection Server 5MessageError Both 6

� GIOP transport adapter: The GIOP specification de-scribes the features of an ORB transport protocol that cancarry GIOP messages. Such protocols must be reliable andconnection-oriented. In addition, GIOP defines a connectionmanagement protocol and a set of constraints for GIOP mes-sage ordering.

�Object addressing: An Interoperable Object Reference(IOR) is a sequence of opaqueprofiles, each representing aprotocol-specific representation of an object’s location. Forexample, an IIOP profile includes the IP address and port num-ber where the server accepts connections, as well as the objectkey that identifies an object within a particular server. An IORmay contain multiple profiles because there may be multipleways to access a server,e.g., through different physical net-work connections or alternate protocols.

CORBA also defines other attributes that can be associatedwith a specific profile, group of profiles, or an entire IOR.

2Version 1.1 of GIOP added aFragment message and version 1.2 relaxesrestrictions on message originators.

These attributes are calledtagged components. Tagged com-ponents can contain various types of QoS information dealingwith security, server thread priorities, network connections,CORBA policies, or other domain-specific information.

An IIOP Version 1.0 profile contains the protocol version,hostname, and port number, as well as an object key that isused to demultiplex an object within a server’s Object Adapter.In IIOP version 1.1, a new field was added to the GIOPheader that defines a sequence of tagged components, whichare name/value pairs that can be used for security, QoS, orother purposes. Tagged components may contain more infor-mation than just the object location. For example, IIOP 1.1defines a flexible mechanism to include QoS parameters, secu-rity and authentication tokens, per-object policies for bridgingwith non-CORBA middleware, character set representations,and alternative addresses for a server.

ESIOP synopsis: In addition to the standard GIOP and IIOPprotocols, the CORBA specification allows ORB implemen-tors to define Environment Specific Inter-ORB Protocols (ES-IOP)s. ESIOPs can define unique data representation for-mats, ORB messaging protocols, ORB transport protocols ortransport protocol adapters, and object addressing formats.These protocols can exploit the QoS features and guaran-tees provided in certain domains, such as telecommunicationsor avionics, to satisfy performance-sensitive applications thathave stringent bandwidth, latency, and jitter requirements.

Only one ESIOP protocol is defined in the CORBA 2.xfamily of specifications: the DCE Common Inter-ORB Pro-tocol (DCE-CIOP) [5]. The OMG is attempting to standardizeother protocols for domains, such as wireless and mobile sys-tems [6], which have unique performance characteristics andoptimization points.

3 The Design of a CORBA PluggableProtocols Framework

The CORBA specification provides a standard for general-purpose CORBA middleware. Within the scope of this specifi-cation, however, ORB implementors are free to optimize inter-nal data structures and algorithms [7]. Moreover, ORBs mayuse specialized inter-ORB protocols and ORB services andstill comply with the CORBA specification.3 For example,ORB providers can develop additional ESIOPs for protocolssuch as ATM or VME, as shown in Figure 1.

This section first identifies the limitations of, and require-ments for, protocol support in conventional CORBA ORBs. Itthen describes how TAO’s pluggable protocols framework isdesigned to overcome these limitations.

3An ORB must implement GIOP/IIOP, however, to be interoperability-compliant.

3

3.1 Protocol Limitations of Conventional ORBs

CORBA’s standard GIOP/IIOP protocols are well-suited forconventional request/response applications with best-effortQoS requirements [8]. They are not well-suited, however, forhigh-performance real-time and/or embedded applications thatcannot tolerate the message footprint size of GIOP or the la-tency, overhead, and jitter of the TCP/IP-based IIOP transportprotocol. For instance, TCP functionality, such as adaptive re-transmissions, deferred transmissions, and delayed acknowl-edgments, can cause excessive overhead and latency for real-time applications [9]. Likewise, network protocols, such asIPv4, lack packet admission policies and rate control capabili-ties, which can lead to excessive congestion and missed dead-lines in networks and endsystems.

Therefore, applications with more stringent QoS require-ments need optimized protocol implementations, QoS-awareinterfaces, custom presentations layers, specialized memorymanagement (e.g., shared memory between ORB and I/O sub-system), and alternative transport programming APIs (e.g.,sockets vs. VIA [10]). Domains where highly optimizedORB messaging and transport protocols are particularly im-portant include (1) multimedia applications running over high-speed networks, such as Gigabit Ethernet or ATM, and (2)real-time applications running over embedded system inter-connects, such as VME or CompactPCI.

Conventional CORBA implementations have the followinglimitations that make it hard for them to support performance-sensitive applications effectively:

1. Static protocol configurations: Conventional ORBs sup-port a limited number of statically configured protocols, oftenjust GIOP/IIOP over TCP/IP.

2. Lack of protocol control interfaces: ConventionalORBs do not allow applications to configure key protocol poli-cies and properties, such as peak virtual circuit bandwidth orcell pacing rate.

3. Single protocol support: Conventional ORBs do notsupport simultaneous use of multiple inter-ORB messaging ortransport protocols.

4. Lack of real-time protocol support: ConventionalORBs have limited or no support for specifying and enforcingreal-time protocol requirements across a backplane, network,or Internet end-to-end.

3.2 Pluggable Protocols Framework Require-ments

The limitations of conventional ORBs described in Section 3.1make it hard for developers to leverage existing implementa-tions, expertise, and ORB optimizations across projects or ap-plication domains. Defining a standardpluggable protocols

frameworkfor CORBA ORBs is an effective way to addressthis problem. The requirements for such a pluggable protocolsframework for CORBA include the following:

1. Define standard, unobtrusive protocol configuration in-terfaces: To address the limitations of conventional ORBs,a pluggable protocols framework should define a standard setof APIs to install ESIOPs and their transport-dependent com-ponents. Most applications need not use this interface di-rectly. Therefore, the pluggable protocols interface should beexposed only to application developers interested in definingnew protocols or in configuring existing protocol implementa-tions in novel ways.

2. Use standard CORBA programming and control inter-faces: To ensure application portability, clients should pro-gram to standard application interfaces defined in CORBAIDL, even if pluggable ORB messaging or transport protocolsare used. Likewise, object implementors need not be aware ofthe underlying framework. Developers should be able to setpolicies, however, that control the ORB’s choice of protocolsand protocol properties. Moreover, these interfaces shouldtransparently support certain real-time ORB features, such asscatter/gather I/O, optimized memory management, and strate-gized concurrency models [7].

3. Simultaneous use of multiple ORB messaging and trans-port protocols: To address the lack of support for multi-ple inter-ORB protocols in conventional ORBs, a pluggableprotocols framework should support different messaging andtransport protocolssimultaneouslywithin an ORB endsystem.The framework should configure inter-ORB protocols trans-parently, eitherstaticallyduring ORB initialization [11] ordy-namicallyduring ORB run-time [12].

4. Support for multiple address representations: This re-quirement addresses the lack of support for multiple Inter-ORB protocols and dynamic protocol configurations in con-ventional ORBs. For example, each pluggable protocol imple-mentation can potentially have a different profile and objectaddressing strategy. Therefore, a pluggable protocols frame-work should provide a general mechanism to represent thesedisparate address formats transparently, while also supportingstandard IOR address representations efficiently.

5. Support CORBA standard features and future en-hancements: A pluggable protocols framework should sup-port standard CORBA [13] features, such as object referenceforwarding, connection transparency, preservation of foreignIORs and profiles, and the GIOP 1.2 protocol, in a mannerthat does not degrade end-to-end performance and predictabil-ity. Moreover, a pluggable protocols framework should ac-commodate forthcoming enhancements to the CORBA spec-ification, such as (1)fault tolerance[14, 15], which supports

4

group communication, (2)real-time properties[11], which in-clude features to reserve connection and threading resourceson a per-object basis, (3)asynchronous messaging[16], whichexports QoS policies to application developers, and (4)wire-less access and mobility[6], which defines lighterweight Inter-ORB protocols for low-bandwidth links.

6. Optimized inter-ORB bridging: A pluggable protocolsframework should ensure that protocol implementors can cre-ate efficient, high-performance inter-ORBin-line bridges. Anin-line bridge converts inter-ORB messages or requests fromone type of IOP to another. This makes it possible to bridgedisparate ORB domains efficiently without incurring unneces-sary context switching, synchronization, or data movement.

7. Provide common protocol optimizations and real-timefeatures: A pluggable protocols framework should supportfeatures required by real-time CORBA applications [11], suchas resource pre-allocation and reservation, end-to-end prior-ity propagation, and mechanisms to control properties specificto real-time protocols. These features should be implementedwithout modifying the standard CORBA programming APIsused by applications that do not possess real-time QoS require-ments.

8. Dynamic protocol bindings: To address the limitationswith static, inflexible protocol bindings in conventional ORBs,a pluggable protocols frameworks should support dynamicbinding of specific ORB messaging protocols with specific in-stances of ORB transport protocols. This design permits effi-cient and predictable configurations for both standard and cus-tomized IOPs.

3.3 Architectural Overview of TAO’s Plug-gable Protocols Framework

To meet the requirements outlined in Section 3.2, we iden-tified logical communication component layers within TAO,factored out common features, defined general framework in-terfaces, and implemented components to support differentconcrete inter-ORB protocols. Higher-level services in theORB, such as stubs, skeletons, and standard CORBA pseudo-objects, are decoupled from the implementation details of par-ticular protocols, as shown in Figure 2. This decoupling isessential to resolve several limitations of conventional ORBsoutlined in Section 3.1, as well as to meet the requirements setforth in Section 3.2.

In general, the higher-level components and services ofTAO use the Facade pattern [17] to access the mechanismsprovided by its pluggable protocols framework. Thus, ap-plications can (re)configure custom protocols without requir-ing global changes to the ORB. Moreover, because applica-tions typically access only the standard CORBA APIs, TAO’s

CLIENT

STUBS SKELETONS

TCP

MULTICAST

IOP

VMEUDP

ORB MESSAGING COMPONENT

ORB TRANSPORT ADAPTER COMPONENT

ESIOP

IIOP

REAL -TIME

IOPEMBEDDED

IOP

RELIABLE,BYTE-STREAM

ATMTCP

MEMORY

MANAGEMENT

OTHER

ORB CORE

SERVICES

PLU

GG

AB

LE P

RO

TO

CO

LS F

RA

ME

WO

RK

COMMUNICATION INFRASTRUCTUREHIGH SPEED NETWORK INTERFACE

REAL -TIME I /O SUBSYSTEM

ORB MESSAGE

FACTORY

ORB TRANSPORT

ADAPTER FACTORY

OBJECT ADAPTER

GIOP GIOPLITE

ADAPTIVE Communication Environment (ACE)

OBJECT (SERVANT)operation (args)IN ARGS

OUT ARGS & RETURN VALUE

CONCURRENCY

MODEL

POLICY

CONTROL

CONNECTION

MANAGEMENT

PROFILE

MANAGEMENT

Figure 2: TAO’s Pluggable Protocols Framework Architecture

pluggable protocols framework can be used transparently byCORBA application developers.

The key TAO pluggable protocols framework componentsillustrated in Figure 2 are described below.

3.3.1 ORB Messaging Component

This component is responsible for implementing ORB mes-saging protocols, such as the standard CORBA GIOP ORBmessaging protocol, as well as custom ESIOPs. An ORBmessaging protocol must define a data representation, anORB message format, an ORB transport protocol or transportadapter, and an object addressing format. Within this frame-work, ORB protocol developers are free to implement opti-mized Inter-ORB protocols and enhanced transport adaptors,as long as they respect the ORB interfaces.

Each ORB messaging protocol implementation inheritsfrom a common base class that defines a uniform interface.This interface can be extended to include new capabilitiesneeded by special protocol-aware policies. For example, ORBend-to-end resource reservation or priority negotiation canbe implemented in an ORB messaging component. TAO’spluggable protocols framework ensures consistent operationalcharacteristics and enforces general IOP syntax and semanticconstraints, such as error handling.

When adding a new IOP, it may not be necessary to re-implement all aspects of the ORB’s messaging protocol. Forexample, TAO has a highly optimized CDR implementationthat can be used by new IOPs [7]. TAO’s CDR implemen-tation contains highly optimized memory allocation strategiesand data type translations. Thus, protocol developers can sim-ply identify new memory or connection management strate-gies that can be configured into the existing CDR components.

Message factoriesare another key part of TAO’s ORB mes-saging component. During connection establishment, thesefactories instantiate objects that implement various ORB mes-

5

saging protocols. These objects are associated with a specificconnection and ORB transport adapter component,i.e., the ob-ject that implements the component, for the duration of theconnection.

3.3.2 ORB Transport Adapter Component

This component maps a specific ORB messaging protocol,such as GIOP or DCE-CIOP, onto a specific instance of anunderlying transport protocol, such as TCP or ATM. Figure 2shows an example in which TAO’s transport adapter maps theGIOP messaging protocol onto TCP–this standard mapping iscalled IIOP. In this case, the ORB transport adapter combinedwith TCP corresponds to the transport layer in the Internet ref-erence model. However, if ORBs are communicating over anembedded interconnect, such as a VME bus, the bus driver andDMA controller provide the “transport layer” in the commu-nication infrastructure.

TAO’s ORB transport component accepts a byte streamfrom the ORB messaging component, provides any additionalprocessing required, and passes the resulting data unit to theunderlying communication infrastructure. Additional process-ing that can be implemented by protocol developers includes(1) concurrency strategies, (2) endsystem/network resourcereservation protocols, (3) high-performance techniques, suchas zero-copy I/O, shared memory pools, periodic I/O, and in-terface pooling, (4) enhancement of underlying communica-tions protocols,e.g., provision of a reliable byte stream proto-col over ATM, and (5) tight coupling between the ORB and ef-ficient user-space protocol implementations, such as Fast Mes-sages [18].

3.3.3 ORB Policy Control Component

It is not possible to determinea priori all attributes definedby all protocols. Therefore, TAO’s pluggable protocols frame-work provides an extensiblepolicy controlcomponent, whichimplements the QoS framework defined in the CORBA Mes-saging [16] and Real-time CORBA [11] specifications. Thiscomponent allows applications to control the QoS attributesof configured ORB transport protocols.

In general, the CORBA QoS framework allows applica-tions to specify variouspoliciesto control the QoS attributesin the ORB. The CORBA specification uses policies to de-fine semantic properties of ORB features precisely without (1)over-constraining ORB implementations or (2) increasing in-terface complexity for common use-cases. Example policiesrelevant for pluggable protocols include buffer pre-allocations,fragmentation, bandwidth reservation, and maximum trans-port queue sizes.

Policies in CORBA can be set at the ORB, thread, or ob-ject level. Thus, application developers can set global poli-

cies that take effect for any request issued in a particular ORB.Moreover, these global settings can be overridden on a per-thread basis, a per-object basis, or even before a particular re-quest. In general, CORBA’s Policy framework provides veryfine-grained control over the ORB behavior, while providingsimplicity for the common case.

Certain policies, such as timeouts, can be shared betweenmultiple protocols. Other policies, such as ATM virtual circuitbandwidth allocation, may apply to a single protocol. Eachconfigured protocol can query TAO’s policy control compo-nent to determine its policies and use them to configure itselffor user needs. Moreover, protocol implementations can sim-ply ignore policies that do not apply to it.

TAO’s policy control component enables applications toselect their protocol(s). This choice can be controlled bythe ClientProtocolPolicy defined in the Real-timeCORBA specification [11]. Using this policy, an applicationcan indicate its preferred protocol(s) and TAO’s policy controlcomponent then attempts to match that preference with its setof available protocols. TAO provides other policies that con-trol the behavior of the ORB if an application’s preferencescannot be satisfied. For example, an exception can be raisedor another available protocol can be selected transparently.

3.3.4 Connection Management Services

Connection management services are a fundamental compo-nent of TAO’s pluggable protocols framework. These ser-vices are responsible for creating ORB protocol objects dy-namically and associating them with specific connections.They also interpret profiles and create object references onthe server. By employing patterns and leveraging TAO’s real-time features [19], protocol implementors can design high-performance IOPs that enforce stringent QoS properties.

The connection management services are implemented withconnectors, acceptors, reactorsandregistriesthat keep trackof available protocols, create protocol objects, and interpretprofiles and object addresses. Acceptors and connectors im-plement theAcceptor-Connectorpattern [3], which decouplesthe task of connection establishment and connection handlerinitialization from subsequent IOP message processing. Theconnectors and acceptors register themselves with their corre-sponding registries. The registries in turn keep track of avail-able ORB message and transport protocols and are responsiblefor interpreting object references.

TAO’s connection management services behave differentlydepending on whether the ORB plays the role of a client or aserver, as outlined below.

Client ORB components: In the client ORB, theConnector Registry and Connector establishconnections to server objects and link the constituent objects

6

together statically or dynamically. When a client applicationinvokes an operation, it uses the list of profiles derived fromthe object’s IOR.

For each inter-ORB and transport protocol combinationavailable in the ORB, there is a correspondingConnectorobject responsible for performing the connection. The registrywill cycle through the list of profiles for an object, requestingthe appropriate connector to attempt a connection. If a connectsucceeds, then the search is concluded and the successful pro-file is returned to the client. If no connect succeeds, the ORBthrows atransient exception to the client.

Server ORB components: In the server ORB, anAcceptor waits passively for a connection event usinga Reactor in accordance with the Reactor pattern [3].Different concurrency architectures may be used, suchas single-threaded, thread-per-connection or thread-per-priority [20]. The actual concurrency strategy used isprovided as a service by TAO’s ORB Core and the plug-gable protocols framework. Regardless of the threading andconnection concurrency strategy, the basic steps are the same:

1. AnAcceptor listens to endpoints and waits for connec-tion requests.

2. When a connection is accepted, a connection handler ob-ject and IOP object are created.

The Acceptor Registry creates object references forregistered server objects. When an object is advertised, theregistry will request each registeredAcceptor to create aprofile for this object. TheAcceptor will place in this pro-file the host address, the corresponding transport service ac-cess point (for example, port number for TCP/IP), and objectkey. All profiles are then bundled by theAcceptor into anIOR, which clients can use to access the object.

3.3.5 Multiple Profiles and Location Forwarding

As explained in Section 2, clients obtain interoperable ob-ject references (IORs), which are used to locate the objectsupon which invocations are performed. An object reference in-cludes at least one profile, which contains information for ac-cessing an object through different network interfaces, sharedmemory, security restrictions, or QoS parameters. Multipleprofiles could be used in a situation where an object resides ona server with multiple interfaces,e.g., ATM and Ethernet. Aprofile will then be created for each of the two interfaces.

TAO’s multiple profiles implementation incorporates sup-port for location forwarding, which occurs when an ORBsends a request to a server object, and the server responds witha location forward reply. The location forward reply will in-clude an IOR that the client decodes to get the list of forward-ing profiles. The forwarding profiles will then replace the for-warded profile in the original profile list. Each new profile will

then be tried in turn until one succeeds, is itself forwarded, oruntil all fail. If all forwarding profiles fail, the forwarding listis removed and the ORB continues with the next profile afterthe one that was forwarded initially.

There is no pre-defined limit on the number of location-forward messages that an ORB may receive. For example,if an invocation using a profile from the list of forwarding pro-files should also be forwarded, the process will repeat recur-sively until the operation succeeds or all profiles have beentried. In practice, however, it is advantageous to limit the depthof recursion in case forwarding loops occur.

Multiple profiles can be used for other purposes, such asfault-tolerance [14, 15]. For example, consider an object thatis replicated in three locations,e.g., on different hosts, pro-cesses, or CPU boards in an embedded system. The IOR forthis object would contain three profiles, one for for each objectlocation. If an invocation fails using the first profile, TAO’spluggable protocols framework will transparently retry the in-vocation using the second profile that corresponds to the repli-cated object at a different location. By using some form ofcheckpointing or reliable multicast the state of these object in-stances can be synchronized.

Location forwarding can also be used for load balancing.For example, if one server becomes overloaded, it can migratesome of its objects to another server. Subsequent requests onthe relocated object will then result in a location forward replymessage. The message contains the new IOR for the relocatedobject. In the client ORB, TAO’s pluggable protocols frame-work will then retry the object operation invocation using thenew IOR transparently to the application. When system loadsreturn to normal, the object can migrate back to the originalserver, and if the client performs another operation invocation,the forwarded server can reply with an exception indicating theobject is no longer there. The client then retries at the originallocation transparently to the application.

3.4 Pluggable Protocols Scenarios

To illustrate how TAO’s pluggable protocols framework hasbeen applied in practice, we now describe two scenariosthat require performance-sensitive and real-time CORBA sup-port. These scenarios are based on our experience developinghigh-bandwidth, low-latency audio/video streaming applica-tions [21] and avionics mission computing [22] systems. Inprevious work [20], we addressed the network interface andI/O system and how to achieve predictable, real-time perfor-mance. In the discussion below, we focus on ORB support foralternate protocols.

7

3.4.1 Low-latency, High-bandwidth Multimedia Stream-ing

Multimedia applications running over high-speed networks re-quire optimizations to utilize available link bandwidth, whilestill meeting application deadlines. For example, consider Fig-ure 3, where network interfaces supporting 1.2 Mbps or 2.4

WUGS HIGH- SPEEDNETWORK

TAO QOS-ENABLED ORB

RIO SUBSYSTEM

SUPPLIERCONSUMER

TAO QOS-ENABLED ORB

RIO SUBSYSTEM

Figure 3: Example CORBA-based Audio/Video (A/V) Appli-cation

Mbps link speeds are used for a CORBA-based studio qualityaudio/video (A/V) application [21].

In this example, we use TAO’s pluggable protocols frame-work to replace GIOP/IIOP with a custom ORB messagingand transport protocol that transmits A/V frames using TAO’sreal-time I/O (RIO) subsystem [20]. At the core of RIO isthe high-speed ATM port interconnect controller (APIC) [23].APIC is a high-performance ATM interface card that supportsstandard ATM host interface features, such as AAL5 (SAR). Inaddition, the APIC supports (1) shared memory pools betweenuser and kernel space, (2) per-VC pacing, (3) two levels of pri-ority queues, and (4) interrupt disabling on a per-VC bases.

We have leveraged the APIC features and the underlyingATM network to support end-to-end QoS guarantees for TAOmiddleware. In particular, pluggable ORB message and trans-port protocols can be created to provide QoS services to ap-plications, while the ORB middleware encapsulates the actualresource allocation and QoS enforcement mechanisms. Lever-aging the underlying APIC hardware requires the resolution ofthe following two design challenges:

Custom protocols: The first challenge is to create customORB messaging and transport protocols that can exploit high-speed ATM network interface hardware. A careful examina-tion of the system requirements along with the hardware andcommunication infrastructure is required to determine (1) theset of optimizations required and (2) the best partitioning ofthe solution into ORB messaging, transport and policy com-ponents.

The A/V streaming application is primarily concerned with(1) pushing data to clients via one-way method invocations and(2) meeting a specific set of latency and jitter requirements.Considering this, a simple frame sequencing protocol can beused as the ORB’s ESIOP. Moreover, because multimedia data

has diminishing value over time, a reliable protocol like TCPis not required. The overhead of full GIOP is not required,therefore, nor are the underlying assumptions that require atransport protocol with the semantics of TCP.

A key goal of this scenario is to simplify the ORB messag-ing and transport protocol, while adding QoS-related informa-tion to support timely delivery of the video frames and audio.For example, a CORBA request could correspond to one videoframe or audio packet. To facilitate synchronization betweenendpoints, a timestamp and sequence number can be sent witheach request. The Inter-ORB messaging protocol can performa similar function as the real-time protocol (RTP) and real-timecontrol protocol (RTCP) [24].

The ORB messaging protocol can be mapped onto an ORBtransport protocol using AAL5. The transport adapter is thenresponsible for exploiting any local optimizations to hardwareor the endsystem. For example, conventional ORBs copy userparameters into internal buffers used for marshaling. Thesebuffers may be allocated from global memory or possibly froma memory pool maintained by the ORB. In either case, at leastone system call is required to obtain mutexes, allocate buffers,and copy the data. Thus, not only is an additional data copyincurred, but this scenario is rife with opportunities for prior-ity inversion and jitter while waiting to acquire shared ORBendsystem resources.

Optimized protocol implementations: The second chal-lenge is to implement an optimized pluggable protocol thatimplements the design described above. For example, mem-ory can be shared throughout the ORB endsystem,i.e., be-tween the application, ORB middleware, OS kernel, and net-work interface, by allocating memory from a common bufferpool [23, 7]. This optimization eliminates memory copies be-tween user- and kernel-space when data is sent or received.Moreover, the ORB endsystem can manage this memory,thereby relieving application developers from this responsibil-ity. In addition, the ORB endsystem can manage the APICinterface driver, interrupt rates, and pacing parameters, as out-lined in [20].

Figure 4 illustrates a buffering strategy where the ORB man-ages multiple pools of buffers to be used by applications send-ing multimedia data to remote nodes. These ORB buffers areshared between the ORB and APIC driver in the kernel. Thetransport adapter implements this shared buffer pool on a per-connection and possibly per-thread basis to minimize or re-duce the use of resource locks. For example, in the strategydepicted in Figure 4, each active connection is assigned itsown send and receive queues. Likewise, there are two freebuffer pools per connection, one for receive and one for send.

An ORB can guarantee that only one application thread willbe active within the send or receive operation of the transportadapter. Therefore, buffer allocation and de-allocation can

8

MARSHALFRAMES

CONSUMER

OBJECT ADAPTER

ACTIVE

OBJECT

MAP

IDLSKELETON

OS KERNEL

APICDRIVER

OS KERNEL

APICDRIVER

IDLSTUBS

ORB MESSAGING

ORB TRANSPORT

SUPPLIER

ORB MANAGED

DATA BUFFERS

ATM LINK

ORB MESSAGING

ORB TRANSPORT

DEMARSHALFRAMES

ORBCORE

movie->ship (frame)

DEMUX

SENDFREE

APPENDGET FREE

RECV FREE

GET RECV

CDR

ADD FREE

Figure 4: Shared Buffer Strategy

be performed without locking. A similar buffer managementstrategy is described in [23]

User applications can interact with the buffering strategy de-scribed above as follows:

� Zero-copy: The application requests a set of sendbuffers from the ORB that it uses for video and audio data. Inthis case, application developers must not reuse a buffer afterit has been given to the ORB. When the original set of buffersare exhausted, the application must request additional buffers.

� Single-copy: The ORB copies application data into theORB managed buffers. While this strategy incurs one datacopy, the application developer need not be concerned withhow or when buffers are used in the ORB.

Well-designed ORBs can be strategized to allow applica-tions to decide whether data are copied into ORB buffers ornot. For instance, it may be more efficient to copy relativelysmall request data into ORB buffers, rather than using sharedbuffers within the ORB endsystem. By using TAO’s policycontrol component, this decision can be configured on a per-connection, per-thread, per-object or per-operation basis.

3.4.2 Low-latency, Low-jitter Avionics Mission Comput-ing

Avionics mission computing applications [22] are real-timeembedded systems that manage sensors and operator displays,navigate the aircraft’s course, and control weapon release.CORBA middleware for avionics mission computing applica-tions must support deterministic real-time QoS requirementsinteroperating over shared memory, I/O buses, and traditionalnetwork interfaces. Support for deterministic real-time re-quirements is essential for mission computing tasks, such asweapon release and navigation, that must meet all their dead-lines. Likewise, avionics software must support tasks, such as

built-in-test and low-priority display queues, that can tolerateminor fluctuations in scheduling and reliability guarantees, butnonetheless require QoS support [25].

To enforce end-to-end application QoS guarantees, missioncomputing middleware must reduce overall inter-ORB com-munication latencies, maximize I/O efficiency, and increaseoverall system utilization [8, 26]. A particularly important op-timization point is the inter-ORB protocol itself, and the se-lection of an optimal transport protocol implementation for aparticular platform.

For example, Figure 5 depicts an embedded avionics con-figuration with three CPU boards, each with an ORB instance.Each board is connected via a VME bus, which enables the

INTER-ORB COMMUNICATION VIA VME BUS

1553INTERFACE

TO REMOTE

TERMINALS

Figure 5: Example Avionics Embedded ORB Platform

ORBs on each CPU board to communicate using optimizedinter-board communication, such as DMA between the indi-vidual board address spaces. CPU board 1 has a 1553 businterface to communicate with so-called remote terminals,such as aircraft sensors for determining global position andforward- looking infrared radar [22]. This configuration al-lows ORB A to provide a bridging service that forwards ORBrequests between ORBs B and C and remote terminals con-nected with board 1.

The scenario in Figure 5 motivates the need for multipleORB messaging and transport protocols that can be addedseamlessly to an ORB without affecting the standard CORBAprogramming API. For instance, ORB A could use a 1553transport protocol adapter to communicate with remote termi-nals. Likewise, custom ORB messaging and transport proto-cols can be used to leverage the underlying VME bus hardwareand eliminate sources of unbounded priority inversion. Lever-aging the underlying bus hardware requires the resolution ofthe following two design challenges:

Custom protocols: With TAO’s pluggable protocols frame-work, we can create optimized VME-based and 1553-basedinter-ORB messaging and transport protocols. Moreover, byseparating the IOP messaging from a transport-specific map-ping, we can adapt TAO’s pluggable protocols framework todifferent transmission technologies, such as CompactPCI orFibrechannel, by changing only the transport-specific mappingof the associated inter-ORB messaging protocol.

9

Consider an embedded application that must periodicallyprocess sensor data. The sensor data is collected and for-warded aperiodically to a central, although redundant, proces-sor. The sensor data is sent/received aperiodically. Therefore,the resulting bus transfers, interrupts, and driver processingcan reduce the overall utilization of the system. For example,a DMA transfer between two CPU boards requires that theVMEBus, the source PCI bus and the destination PCI bus beacquired and data copied.

A more efficient protocol could buffer these one-way datatransfers until a predetermined byte count or timeout value isreached. Thus the time required to acquire the different busescould be amortized over a larger data transfer. Additionally,given the periodic nature of the transfers rate monotonic anal-ysis could be used to better predict system performance.

Optimized protocol implementations: To optimize the on-the-wire protocol message footprint we use a lightweight ver-sion of GIOP, called GIOPlite. GIOPlite is a streamlined ver-sion of GIOP that removes�15 extraneous bytes from thestandard GIOP message and request headers.4 These bytesinclude the GIOP magic number (4 bytes), GIOP version (2bytes), flags (1 byte), Request Service Context (at least 4bytes), and Request Principal (at least 4 bytes). GIOPlite re-duces the number of bytes transfered across the backplane peroperation.

Another optimization that pertains to avionics mission com-puting involves the use of buffered one-way operations [19].TAO’s pluggable protocols framework has been optimized tosend a series of queued one-way requests in a smaller num-ber of ORB messages. For example, Figure 6 depicts the casewhere one-way CORBA invocations are buffered in the ORBfor later delivery. In this case, a series of one-way invocations

MARSHALPARAMS

OBJECT (SERVANT)

OBJECT ADAPTER

ACTIVE

OBJECT

MAP

IDLSKELETON

VME BUS

OS KERNEL

VMEDRIVER

OS KERNEL

VMEDRIVER

IDLSTUBS

ORB MESSAGING

ORB TRANSPORT

CLIENT

BUFFERED

DATA COPY

DMA COPY

ORB MESSAGING

ORB TRANSPORT

DEMARSHALPARAMS

DATA COPY

ORBCORE

obj->op (params)

DEMUX

Figure 6: One-way Delayed Buffering Strategy

to the same object and for the same operation are queued in the

4The request header size is variable. Therefore, it is not possible to pre-cisely pinpoint the proportional savings represented by these bytes. In manycases, however, the reduction is as large as 25%.

same buffer and sent via a single ORB message. This resultsin an overall increase in throughput between CPU boards byamortizing key sources of communication overhead, such ascontext switching, synchronization, and DMA initialization.

4 Key Design Challenges and Pattern-based Resolutions

Section 3.3 describedhowTAO’s pluggable protocols frame-work is designed. It does not, however, motivatewhy this par-ticular design was selected. In this section, we explore eachfeature in TAO’s pluggable protocols framework and showhow they achieve the goals described in Section 3.2. To clar-ify and generalize our approach, the discussion below focuseson the patterns [17] we applied to resolve the key design chal-lenges we faced during the development process.

4.1 Adding New Protocols Transparently

Context: The QoS requirements of many applications canbe supported solely by using default static protocol config-urationns,i.e., GIOP/IIOP, described in section 3.1. How-ever, applications with more stringent QoS requirements oftenrequire custom protocol configurations. Implementations ofthese custom protocols require several related classes, such asConnector s,Acceptor s,Transport s, andProfile s.To form a common framework, these classes must all be cre-ated consistently.

In addition, many embedded and deterministic real-timesystems require protocols to be configureda priori, with noadditional protocols required once the application is config-ured statically. These types of systems cannot afford the foot-print overhead associated with dynamic protocol configura-tions.

Problem: It must be possible to add new protocols to TAO’spluggable protocols framework without makingany changesto the rest of the ORB. Thus, the framework must be open forextensions, but closed to modifications,i.e., the Open-Closedprinciple [27]. Ideally, creating a new protocol and configur-ing it into the ORB is all that should be required.

Solution: Use aregistry to maintain a collection ofabstractfactories. In the Abstract Factory pattern [17], a single classdefines an interface for creating families of related objects,without specifying their concrete types. Subclasses of an ab-stract factory are responsible for creating concrete classes thatcollaborate among themselves. In the context of pluggableprotocols, each abstract factory can create theConnector ,Acceptor , Profile , andTransport classes for a par-ticular protocol.

10

Applying the solution in TAO: In TAO, the role of the pro-tocol registry is played by theConnector Registry onthe client and theAcceptor Registry on the server. Thisregistry is created by TAO’sResource Factory , which isan abstract factory that creates all the ORB’s strategies andpolicies [28]. Figure 7 depicts theConnector Registryand its relation to the abstract factories.

ACCEPTOR_REGISTRY

IIOP ATM-IOPIIOP ATM-IOP

CONNECTOR_REGISTRY

CLIENT SIDE SERVER SIDE

Figure 7: TAO Connector and Acceptor Registries

Note that TAO does not use abstract factories directly, how-ever. Instead, these factories are accessed via theFacade[17]pattern to hide the complexity of manipulating multiple facto-ries behind a simpler interface. The registry described aboveplays the role of a facade. As shown below, these patterns pro-vide sufficient flexibility to add new protocols transparently tothe ORB.

Establishing connections, manipulating profiles, and cre-ating endpoints are delegated to theConnector andAcceptor registries respectively. Clients will simply pro-vide the Connector Registry with an opaque profile,which corresponds to an object address for a particular pro-tocol instance. The registry is responsible for locating the cor-rect concrete factory, to which it then delegates the respon-sibility for establishing the connection. The concrete factoryestablishes the connection using the corresponding protocol-specific instance, notifying the client of its success or failure.Thereafter, the client simply invokes CORBA operations usingthe selected protocol.

The server delegates endpoint creation to theAcceptor Registry in a similar manner. The reg-istry is passed an opaque endpoint representation, whichit provides to the corresponding concrete factory for theindicated protocol instance. The concreteAcceptor factorycreates the endpoint and enables the ORB to receive requestson the new endpoint.

4.2 Adding New Protocols Dynamically

Context: When developing new pluggable protocols, it isinconvenient to recompile the ORB and applications just to

validate a new protocol implementation. Moreover, it is oftenuseful to experiment with different protocols,e.g., systemati-cally compare their performance, footprint size, and QoS guar-antees. Moreover, in 24�7 systems with high availability re-quirements, it is important to configure protocols dynamically,even while the system is running. This level of flexibility helpssimplify upgrades and protocol enhancements.

Problem: How to populate the registry with the correct ob-jectsdynamically.

Solution: Use the Component Configurator [3] pattern,which decouples the implementation of a service from its con-figuration into the application. This pattern can be applied ineither of the following ways:

1. The Component Configurator pattern [3] can be used todynamically load the registry class. This facade knows how toconfigure a particular set of protocols. To add new protocols,we must either implement a new registry class or derive froman existing one.

This alternative is well-suited for embedded systems withtight memory footprint constraints since it minimizes the num-ber of objects that are loaded dynamically. Implementationsof the Component Configurator pattern can optimize for use-cases where objects are configured statically. Embedded sys-tems can exploit these optimizations to eliminate the need forloading objects into the pluggable protocols framework dy-namically.

2. Use the Component Configurator pattern to load the setof entries in a registry dynamically. For instance, a registry cansimply parse a configuration script and link the services listedin it dynamically. This design is the most flexible strategy, butit requires more code,e.g., to parse the configuration script andload the objects dynamically.

Applying the solution in TAO: TAO implements a classthat maintains all parameters specified in a configurationscript. Adding a new parameter to represent the list of pro-tocols is straightforward,i.e., the default registry simply ex-amines this list and links the services into the address-spaceof the application, using the Component Configurator patternimplementation provided by ACE [29].5 Figure 8 depicts theConnector Registry and its relation to the ACE Compo-nent Configurator implementation.

4.3 Profile Creation

Context: The contents of a profile must be parsed to deter-mine an object’s location. In general, the format and semantics

5ACE provides a rich set of reusable and efficient components for high-performance, real-time communication, and forms the portability layer ofTAO.

11

IIOP ATM-IOP

REGISTRY

COMPONENT CONFIGURATOR

<<INSTANTIATES>>

<<INSTANTIATES>>

Figure 8: TAO Connector Registry and the ACE ComponentConfigurator Implementation

of the profile contents are protocol-specific. Therefore, a com-pletely generic component for it cannot be written. Parsing thedata is a relatively expensive operation that should be avoidedwhenever possible. It is also useful to support multiple pro-tocols (see section 3.1), each one potentially using a differentaddress representation (see section 3.2).

Problem: As new protocols are added to the system, newprofile formats are introduced. It is essential that the correctparsing function be used for each profile format.

Solution: We use the Factory Method pattern [17] to cre-ate the rightProfile class for each protocol. This patterndefines a fixed interface to create an object, while allowingsubclasses the flexibility to create the correct type of object.Two of our classes play theCreatorrole in this pattern: (1) theConnector , using theProfile ’s CDR representation forinitialization and (2) theAcceptor , using the object key forinitialization.

These two approaches are based on the two use-casesin which a Profile object must be manipulated. In theConnector case TAO interprets aProfile received re-motely, whereas in theAcceptor it builds aProfile for alocal object. As usual, theConnector Registry and theAcceptor Registry are used as facades [17] that locatethe appropriateConnector or Acceptor and delegate thejob of building the object to it.

Applying the solution in TAO: TheProfile class is usedto represent a protocol-specific profile. This class provides anabstract interface for parsing, marshaling, hashing, and com-paring profiles. In addition, it provides a unit of encapsulationto maintain information about forwarding and caching connec-tions established to a particular server.

4.4 Decoupling ORB Messaging and TransportProtocol Implementations

Context: It is desirable to support alternative mappings be-tween different ORB messaging protocols and ORB transport

adaptors. For example, a single ORB messaging protocol,such as GIOP, can be mapped to any reliable, connection-oriented transport protocol, such as TCP. Alternatively, a sin-gle transport protocol can be the basis for alternative instanti-ations of ORB messaging protocols,e.g., different versions ofGIOP differing in the number and types of messages, as wellas in the format of those messages.

An ORB messaging protocol imposes requirements on anyunderlying network transport protocols. For instance, thetransport requirements assumed by GIOP described in Sec-tion 2 require the underlying network transport protocol tosupport a reliable, connection-oriented byte-stream. These re-quirements are fulfilled by TCP, thus leading to the direct map-ping of GIOP onto this transport protocol. However, alterna-tive network transport protocols, such as ATM with AAL5,encapsulation may be more appropriate in some environments.In this case, the messaging implementation must provide themissing semantics, such as reliability, to use GIOP.

Problem: The ORB Messaging protocol implementationsmust be independent of the adaptation layer needed for trans-ports that do not satisfy all their requirements. Otherwise, thesame messaging protocol may be re-implemented needlesslyfor each transport, which is time-consuming, error-prone, andtime/space inefficient. Likewise, for those transports that cansupport multiple ORB Messaging protocols, it must be pos-sible to isolate them from the details of the ORB messagingimplementation. Care must be taken, however, because notall ORB Messaging protocols can be used with all transportprotocols,i.e., some mechanism is needed to ensure that onlysemantically compatible protocols are configured [30].

transport ()iop ()

Profile

IIOP_Profile

transport_iop_

transport ()iop ()

profile_in_use_

STUB_Object

profile_in_use ()next_profile ()

1 1

1

1

IIOP_Transport

receive ()send ()handler ();

handler_

receive ()send ()handler ()

Transport1

IOP

start ()invoke ()

GIOP

start ()invoke ()

1

Figure 9: Client Inter-ORB and Transport Class Diagram

Solution: Use the Layers architectural pattern [4], which de-composes the system into groups of components, each one ata different level of abstraction.6 The Layers architectural pat-tern can be implemented differently, depending on whether theORB plays the role of a client or a server, as outlined below.

6Protocol stacks based on the Internet or ISO OSI reference models arecommon examples of the Layers architectural pattern.

12

� Client ORB: For the client, the ORB uses a particu-lar ORB messaging protocol to send a request. This ORBmessaging protocol delegates part of the work to the trans-port adapter component that completes the message and sendsit to the server. If the low-level transport in use, such as ATM,UDP, or TCP/IP, does not satisfy the requirements of the ORBmessaging protocol, the ORB transport adapter component canimplement them.

� Server ORB: In the server, the transport adapter com-ponent receives data from the underlying communication in-frastructure, such as sockets or shared memory, and it passesthe message up to the ORB messaging layer. As with theclient, this layer can be very lightweight if the requirementsimposed by the ORB messaging layer are satisfied by the un-derlying network transport protocol. Otherwise, it must im-plement those missing requirements by building them into theconcrete transport adapter component.

Applying the solution in TAO: As shown in Figure 9, TAOimplements the messaging protocol and the transport proto-col in separate components. The client ORB uses the currentprofile to find the right transport and ORB messaging imple-mentations. The creation and initialization of these classesis controlled by theConnector (described in Section 4.8),with eachConnector instance handling a particular ORBmessaging/transport tuple.

Figure 10 illustrates how the server’s implementation usesthe same transport classes, but with a different relationship. In

IOP

start ()invoke ()

GIOP

start ()invoke ()

1

IIOP_Transport


handler_


Transport1

transport_iop_

IIOP_Connection_Handler

iop ()transport ()

1

1

Figure 10: Server Inter-ORB and Transport Class Diagram

particular, the transport class calls back the messaging classwhen data is received from the IPC mechanism. As with theclient, a factory–in this case theAcceptor –creates and ini-tializes these objects.

4.5 Exception Propagation and Error Detec-tion

Context: The server and client use the same exceptions toinform the application of failures in the communication me-

dia. The ORB must be able to ignore certain communicationerrors selectively and re-issue the request transparently usingalternative addresses or resources.

Problem: If the ORB uses exceptions to internally commu-nicate failures, it can be confused by remote exceptions gener-ated by the server.

Solution: Once again, apply the Layers architectural pat-tern [4] to limit exceptions to communicate failures only be-tween the higher levels of the ORB and the application. Thus,the lower levels of the pluggable protocols framework simplyuse integral return values to indicate an error. These return val-ues are transformed to the appropriate CORBA exception bythe upper levels of the pluggable protocols framework whenreporting the error to the application.

Applying the solution in TAO: Higher level meth-ods in all components,i.e., Acceptor s, Connector s,Transport s, andProfile s, in TAO’s pluggable protocolsframework raise CORBA exceptions if lower level methodsreturn an integer value that indicates failure.

One drawback of using return codes rather than exceptionsis that it may burden ORB developers, who must explicitlycheck for errors, rather than writing exception handlers. TAOmust run on platforms that do not support native C++ excep-tions, however. Therefore, it is already necessary to checkreturn values, so there is no additional burden on TAO devel-opers.

4.6 Adapting TAO to the ACE Framework

Context: TAO is built largely using the reusable andportable ACE framework [29] components, particularlyReactor s, Acceptor s, Connector s, ServiceHandler s, and ACE IPC wrapper facades [29]. TAO’spluggable protocols framework uses inheritance and dynamicbinding to configure these ACE components to create newprotocols.

Problem: Using the lower-level ACE IPC wrapper facadecomponents directly is infeasible because ACE avoids poly-morphism at this level to eliminate the overhead of virtualmethods by non-optimizing compilers [22]. Thus, the ACEconnectors for UNIX-domain sockets and Internet-domainsockets have no common ancestor that can be used to dispatchmethods in subclasses polymorphically. However, a pluggableprotocols framework must be able to establish connections us-ing any protocol.

Solution: Use the External Polymorphism pattern [31] toencapsulate ACE components behind their TAO counterparts.This pattern enables classes that are not related by inheritance,or have no virtual methods, to be treated polymorphically.

13

Applying the solution in TAO: A TAO Acceptor con-tains an ACEAcceptor , which is registered with an ACEReactor that the ORB uses to demultiplex IOP events to theappropriate transport handlers. Eventually, the ACE IPC com-ponents accept a connection and creates an ACEServiceHandler to handle the communication. Our TAO-levelAcceptor encapsulates thatService Handler in aTransport adapter object and passes it up to the ORB. Asa practical consequence of this solution, there exist two sub-layers within TAO’sTransport object.

4.7 Multiple Profiles and Location Forwarding

Context: Object references may contain multiple profiles,and servers may specify alternate object references in responseto a client’s request. In addition, a CORBA-compliant ORB isrequired to try all object references and profiles until one suc-ceedswithoutany client intervention.

Problem: Retries must occur transparently to the client ap-plication, even though profiles for different ORB protocolsmay be dissimilar and profile lists may be altered dynamicallyas a result of forwarding.

Solution: Apply the Proxy pattern [17] and use polymor-phism and an efficient list processing strategy.

Applying the solution in TAO: Figure 11 depicts the classdiagram for the solution. ASTUBObject is a client’slocal proxy for the (potentially) remote object. All com-munication with the server object is done through the stubproxy. While the server does not require aSTUBObject , theAcceptor Registry will initialize an object’s IOR usingtheMProfile andProfile classes.

IIOP_Profile

hintobject_addrversion

transport ()object_addr ()hash ()


Profile

base_profiles_forward_profiles_

STUB_Object

set_mprofiles ()add_forward_profiles ()set_base_profiles ()get_profiles ()next_profile ()

MProfile

add_profile ()give_profile ()

N

1

profiles_

1 N

Figure 11: Class Diagram for Multiple Profile and ForwardingSupport in TAO’s Pluggable Protocols Framework

Profile lists are maintained by anMProfile object. Theprofile list is stored as a simple array of pointers toProfileobjects. All instances of IOP profiles are derived from thiscommonProfile class. By relying on dynamic binding ofobjects, the base class can be used for both referencing and

performing common method invocations on the concrete pro-file instances. TheMProfile object can therefore maintaina list of Profile proxies to the actual concrete profile in-stances.

The MProfile object keeps track of the current profileand allows a user only to increment and decrement this refer-ence. If the current profile is forwarded, a reference is keptin that profile to the forwardingMProfile object. Like-wise, the forwardingMProfile contains a back pointer totheMProfile object that was forwarded. In this way, a listof MProfile s is maintained, corresponding to the initial andall forwarding profile lists.

When a client decodes the initial IOR, the resulting profilelist is stored in anMProfile object. If the client receives alocation forward, either as a result of aLocate RequestGIOP message or in aLOCATION FORWARD reply, the re-ceived IOR is decoded and added to theSTUBObject us-ing its add forward profiles method. TheMProfileobject that was forwarded keeps track of the current profile,marks it as being in aFORWARDING state, and sets a referenceto the forwarding profile list. TheSTUBObject maintains areference to the initial profile list and to the current forwardingprofile list, i.e., theMProfile object, because the forward-ing MProfile objects each contain a back pointer to the for-wardedMProfile object.

Figure 12 illustrates how forwarding is represented usingthe MProfile objects. Not shown is theSTUBObject ,which maintains references to the initial or unforwarded pro-file list, the current profile in use, and the last forwarding pro-file list. In effect, theSTUBObject andMProfile presentthe ordered profile list–P1, P2, P5, P6, P7, P8, P9, P3, P4–totheConnection Registry .

P7P8P9

MPROFILEP5P6

MPROFILEP1P2P3P4

MPROFILE

current currentcurrent

1) Current profile is P8, P2 and P6 were forwarded

P1P2P3P4

MPROFILE

current

P7P8P9

MPROFILEP5P6

MPROFILEP1P2P3P4

MPROFILE

currentcurrent

current

2) P8 failed, try P9

3) P9 failed, remove forwarding profile lists and try P3

Figure 12: Object Reference Forwarding Example

14

4.8 Establishing Connections Actively

Context: When a client references an object, the ORB mustobtain the corresponding profile list, which is derived from theIOR and a profile ordering policy, and establish a connectionto the server transparently.

Problem: There can be one or more combinations of inter-ORB and transport protocols available in an ORB. For a givenprofile, the ORB must verify the presence of the associatedIOP and transport protocol, if available. It must then locatethe applicableConnector and delegate to it to establish theconnection.

Solution: We use theConnector component in theAcceptor-Connector pattern [3] to actively establish a con-nection to a remote object. This pattern decouples the con-nection establishment from the processing performed after theconnection is successful. Figure 13 shows how multiple pro-files may be used during connection establishment in both theclient and server. This figure shows a connection toObject

CONNECT (PROFILE2)

ATM TRANSPORT

OBJECTCONNECTION HANDLER

OBJECT

CONNECTION HANDLER

OBJECT

ATM-IOPOBJECT

1 - IIOP:// HOSTA:PORT1/OBJECT_KEY

2 - ATM-IOP:// HOSTA_ATM :SAP1/OBJECT_KEY

3 - IIOP:// HOSTB:PORT2/OBJECT_KEY

PROFILE LIST FOR OBJECT A

IIOP ATM-IOP

CONNECTOR_REGISTRYCONNECT (PROFILE_LIST A)

ACCEPTOR_REGISTRY

IIOP ATM-IOPOBJECT A

ATM TRANSPORT

OBJECT

ATM-IOPOBJECT

<<INSTANTIATES >>

CLIENT

SERVER

ESTABLISH CONNECTION

TO OBJECT A

PERFORM INVOCATION

ON OBJECT A

<<INSTANTIATES >>

<<INSTANTIATES >>

<<INSTANTIATES >>

<<INSTANTIATES >>

<<INSTANTIATES >>

Figure 13: Connection Establishment Using Multiple Plug-gable Protocols

A being requested of theConnector Registry . The reg-istry will in turn try the profiles listed in thesuppliedprofile list for Object A . In this figure, the first pro-file is for an IIOP connection toHost A atport 1 . Assum-ing the connect fails for some reason, the registry will try thesecond profile automatically. This profile contains a referenceto the same host via ATM interface using an ESIOP.

Assuming the connect on the second profile succeeds, theConnector andAcceptor create their corresponding con-

nection handlers and ATM-IOP transport objects. The con-nection handlers then create transport objects, which providethe mapping from the chosen transport protocol to a transport-independent interface used by the IOP messaging component.The connection handler is considered as part of the ORB trans-port adapter component.

Applying the solution in TAO: As described in Section 4.6,Connector s are adapters for the ACE implementation ofthe Acceptor-Connector pattern. Thus, they are lightweightobjects that simply delegate to a corresponding ACE com-ponent. Figure 15 shows the base classes and their relationsfor IIOP. This figure shows an explicit co-variance between

IIOP_Transport

receive ();send ();handler ();

handler_

Transport


transport_


handler_input ()svc ()transport ()

open ()close ()accept ()

IIOP_Strategegy_Acceptor

GIOP

start ()invoke ()

IOP

start ()invoke ()

IIOP_Acceptor

create_profile ();acceptor ();

transport_acceptor

Acceptor

create_profile ()acceptor ()


Profile

IIOP_Profile

hint = 0object_addrversion


<<instantiates>>

<<instantiates>>

<<Instantiates>>

N

1

1

1

1

1

11

1

1

<<instantiates>>

Acceptor Registry

add_acceptor ()get_acceptor ()create_ior ()

registered acceptors

1

Figure 14: Server Pluggable Protocols Class Diagram

the Profile and theConnector s for each protocol. Ingeneral, aConnector must downcast theProfile to itsspecific type. This downcast is safe because profile creationis limited to theConnector andAcceptor registries. Inboth cases, the profile is created with a matching tag. Thetag is used by theConnector Registry to choose theConnector that can handle each profile.

As shown in the same figure, theConnector Registrymanipulates only the base classes. Therefore, new protocolscan be added without requiring any modification to the ex-isting pluggable protocols framework. When a connection isestablished successfully, theProfile is passed a pointer tothe particular IOP object and to theTransport objects thatwere created.

4.9 Accepting Connections Passively

Context: A server can accept connections at one or moreendpoints, potentially using the same protocol for all end-points. The set of protocols that an ORB uses to play the clientrole need not match the set of protocols used for the serverrole. Moreover, the ORB can even be a “pure client”,i.e., aclient that only makes requests. In this case it can use several

15

protocols to make requests, but receive no requests from otherclients.

Problem: The server must generate an IOR that includes allpossible inter-ORB and transport-protocol-specific profiles forwhich the object can be accessed. As with the client, it shouldbe possible to add new protocols without changing the ORB.

Solution: Use theAcceptor component in the Acceptor-Connector pattern [3] to accept the connections. AnAcceptor accepts a connectionpassively, rather than be-ing initiatedactively, as with theConnector component de-scribed above.

Applying the solution to TAO: Figure 14 illustrates howTAO’s pluggable protocols framework leverages the design

Connector

open ()close ()connect ()

Connector Registry

open ()close ()add_connector ()get_connector (tag)connect (stubobj)

registered connectors

1 N

IOP

start ()invoke ()

open ()close ()connect ()

IIOP_Strategegy_Connector


Profile

IIOP_Profile

hintobject_addrversion


1

1

GIOP

start ()invoke ()

transport

handler_input ()svc ()transport ()

IIOP_Transport


handler_

1

1


Transport

IIOP_Connector

open ();close ();connect ();

transport_connector


<<instantiates>>

<<instantiates>>

1

1

1

11

1

<<instantiates>>

Figure 15: Client Pluggable Protocols Class Diagram

presented in Section 4.1. The concrete ACEServiceHandler created by the ACEAcceptor is responsible forimplementing the External Polymorphism pattern [31] and en-capsulating itself behind theTransport interface defined inTAO’s pluggable protocols framework.

As discussed in Section 4.6, TAO use the Adapter pat-tern [17] to leverage the ACEAcceptor implementa-tion. This pattern also permits a seamless integration withlower levels of the ORB. In the Acceptor-Connector pattern,the Acceptor object is a factory that createsServiceHandler s, which perform I/O with their connected peers.In TAO’s pluggable protocols framework, theTransportobjects areService Handlers implemented as abstractclasses. This design shields the ORB from variations in theAcceptor s, Connector s, andService Handler s foreach particular protocol.

When a connection is established, the concreteAcceptorcreates the appropriateConnection Handler and IOPobjects. The Connection Handler also creates aTransport object that functions as the implementation rolein the Bridge pattern [17]. As with theConnector , the

Acceptor also acts as the interface role in the Bridge pat-tern, hiding the transport- and strategy-specific details of theAcceptor .

5 The Performance of TAO’s Plug-gable Protocols Framework

Despite the growing demand for off-the-shelf middleware inmany application domains, a widespread belief persists thatOO techniques are not suitable for real-time systems due toperformance penalties [22]. In particular, the dynamic bindingproperties of OO programming languages and the indirectionimplied in OO designs seem antithetical to real-time systems,which require low latency and jitter. The results presented inthis section are significant, therefore, because they illustrateempirically how the choice of patterns described in Section 4enabled us to meet non-functional requirements, such as porta-bility, flexibility, reusability, and maintainability, without com-promising overall system efficiency, predictability, or scalabil-ity.

To quantify the benefits and costs of TAO’s pluggable pro-tocols framework, we conducted several benchmarks usingtwo different ORB messaging protocols, GIOP and GIOPlite,and two different transport protocols, POSIX local IPC (alsoknown as UNIX-domain sockets) and TCP/IP. These bench-marks are based on our experience developing CORBA mid-dleware for avionics mission computing applications [22] andmultimedia applications [21], as described in Section 3.4.

Note that POSIX local IPC is not a traditional high-performance networking environment. However, it does pro-vide the opportunity to obtain an accurate measure of ORBand pluggable protocols framework overhead. Based on thesemeasurements, we have isolated the overhead associated witheach component, which provides a baseline for future work onhigh-performance protocol development and experimentation.

5.1 Hardware/Software Benchmarking Plat-form

All benchmarks in this section were run on a Quad-CPU IntelPentium II Xeon 400 MHz workstation, with one gigabytesof RAM. The operating system used for the benchmarkingwas Debian GNU/Linux “potato” (glibc 2.1) with Linux ker-nel version 2.2.10. GNU/Linux is an open-source operatingsystem that supports true multi-tasking, multi-threading, andsymmetric multiprocessing.

For these experiments, we used the GIOP and GIOPlite [7]messaging protocols. GIOPlite is a streamlined version ofGIOP that removes�15 extraneous bytes from the standard

16

GIOP message and request headers.7 These bytes include theGIOP magic number (4 bytes), GIOP version (2 bytes), flags(1 byte), Request Service Context (at least 4 bytes), and Re-quest Principal (at least 4 bytes).

Our benchmarks were run using the standard GIOP ORBmessaging protocol, as well as TAO’s GIOPlite messagingprotocol. For the TCP/IP tests, the GIOP and GIOPlite ORBmessaging protocols were run using the standard CORBAIIOP transport adapter along with the Linux TCP/IP socketlibrary and the loopback interface.

For the local IPC tests, GIOP and GIOPlite were used alongwith the optimized local IPC transport adapter. This resultedin the following four different Inter-ORB Protocols: (1) GIOPover TCP (IIOP), (2) GIOPlite over TCP, (3) GIOP over localIPC8 (UIOP), and (4) GIOPlite over local IPC. No changeswere required to our standard CORBA benchmarking tool,calledIDL Cubit [32], for either of the ORB messaging andtransport protocol implementations.

5.2 Blackbox Benchmarks

Blackbox benchmarks measure the end-to-end performance ofa system from an external application perspective. In our ex-periments, we used blackbox benchmarks to compute the av-erage two-way response time incurred by clients sending vari-ous types of data using the four different Inter-ORB transportprotocols.

Measurement technique: A single-threaded client is usedin the IDL Cubit benchmark to issue two-way IDL opera-tions at the fastest possible rate. The server performs the oper-ation, which cubes each parameter in the request. For two-waycalls, the client thread waits for the response and checks that itis correct. Interprocess communication is performed over theselected IOPs, as described above.

We measure throughput for operations using a variety ofIDL data types, includingvoid , sequence , andstructtypes. Thevoid data type instructs the server not to per-form any processing other than that necessary to prepare andsend the response,i.e., it does not cube any input parame-ters. Thesequence andstruct data types exercise TAO’s(de)marshaling engine. Thestruct contains anoctet , along , and ashort , along with padding necessary to alignthose fields. We also measure throughput using long and shortsequences of thelong and octet types. Thelong se-quences contain 4,096 bytes (1,024 four bytelong s or 4,096

7The request header size is variable. Therefore, it is not possible to pre-cisely pinpoint the proportional savings represented by these bytes. In manycases, however, the reduction is as large as 25%.

8For historical reasons, TAO retains the expression “UNIX-domain” inits local IPC pluggable protocol implementation, which is where the name“UIOP” derives from.

octet s) and the short sequences are 4 bytes (one four bytelong or fouroctet s).

Blackbox results: The blackbox benchmark results areshown in Figure 16. All blackbox benchmarks were averaged

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

void

shor

toc

tet

long

struc

t

small

seq<

octe

t>

large

seq<

octe

t>

small

seq<

long>

large

seq<

long>

small

seq<

struc

t>

large

seq<

struc

t>

Data TypeC

alls

per

Sec

on

d

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

Per

form

ance

Imp

rove

men

t (%

)

IIOP IIOP/GIOPliteUIOP UIOP/GIOPlitePerformance Increase

Figure 16: TAO’s Pluggable Protocols Framework Perfor-mance Over Local IPC and TCP/IP

over 100,000 two-way operation calls for each data type, asshown in Figure 16.

UIOP performance surpassed IIOP performance for all datatypes. The benchmark results show how UIOP improves per-formance from 20% to 50% depending on the data type andsize. For smaller data sizes and basic types, such asoctetand long , the performance improvement is approximately50%. For larger data payload sizes and more complex datatypes, however, the performance improvements are reduced.This result occurs due to the increasing cost of both the datacopies associated with performing I/O and the increasing com-plexity of marshaling structures other than the basic data types.

For certain data types, additional improvements are ob-tained by reducing the number of data copies required. Sucha situation exists when marshaling and demarshaling data oftype octet and long . For complicated data types, suchas a largesequence of struct s, ORB overhead is par-ticularly prevalent. Large ORB overhead implies lower effi-ciency, which accounts for the smaller performance improve-ment gained by UIOP over IIOP for complex data types.

GIOPlite outperformed GIOP by a small margin. ForIIOP, GIOPlite performance increases over GIOP ranged from0.36% to 4.74%, with an average performance increase of2.74%. GIOPlite performance improvements were slightlybetter over UIOP due to the fact that UIOP is more efficientthan IIOP. GIOPlite over UIOP provided improvements rang-ing from 0.37% to 5.29%, with an average of 3.26%.

Our blackbox results suggest that more substantial changes

17

to the GIOP message protocol are required to achieve sig-nificant performance improvements. However, these resultsalso illustrate that the GIOP message footprint has a rela-tively minor performance impact over high-speed networksand embedded interconnects. Naturally, the impact of theGIOP message footprint for lower-speed links, such as second-generation wireless systems or low-speed modems, is moresignificant.

5.3 Whitebox Benchmarks

Whitebox benchmarks measure the performance of specificcomponents or layers in a system from an internal perspective.In our experiments, we used whitebox benchmarks to pinpointthe time spent in key components in TAO’s client and serverORBs. The ORB’s logical layers, or components, are shownin Figure 17 along with the timeprobe locations used for these

ORB TRANSPORTORB TRANSPORT

RECVRECV

OBJECT (SERVANT)

ORBORBCORECORE

OBJECT ADAPTEROBJECT ADAPTER

IDLIDLSKELETONSKELETON

VME BUSVME BUS

OS KERNELOS KERNEL

VME DRIVERVME DRIVER

OS KERNELOS KERNEL

VME DRIVERVME DRIVER

ORB MESSAGINGORB MESSAGING


ACTIVEACTIVE

OBJECTOBJECT

MAPMAP

GET OBJECTGET OBJECT

REFREF

IDLIDLSTUBSSTUBS

MARSHALMARSHAL

PARMATERSPARMATERS


SENDSEND




SENDSEND

II//O SENDO SEND

II//O RECVO RECV77

88


RECVRECV99

CLIENTCLIENT

DEMARSHALDEMARSHAL

PARAMETERSPARAMETERS

7799

II//O RECVO RECV


RECVRECV


RECVRECV

PARSE OBJECTPARSE OBJECT

KEY KEY

OBJECTOBJECT

DEMUXDEMUX

DEMARSHALDEMARSHAL


USER UPCALLUSER UPCALL


SENDSEND


SENDSEND

II//O SENDO SEND

MARSHALMARSHAL


OPERATIONOPERATION

DEMUXDEMUX

131311

22

44

661111

1010

1212

33

88INITIALIZATIONINITIALIZATION

33

55

44

11

22

66

1010

OUTGOINGOUTGOING INCOMINGINCOMING

INC

OM

ING

INC

OM

ING

OU

TG

OIN

GO

UT

GO

ING

POAPOA

DEMUXDEMUX55

Figure 17: Timeprobe Locations for Whitebox Experiment

benchmarks.

5.3.1 Measurement Techniques

One way to measure performance overhead of operations incomplex CORBA middleware is to use a profiling tool, suchas Quantify [33]. Quantify instruments an application’s bi-nary instructions and then analyzes performance bottlenecksby identifying sections of code that dominate execution time.Quantify is useful because it can measure the overhead of sys-tem calls and third-party libraries without requiring the sourcecode.

Unfortunately, Quantify is not available for Linux kernel-based operating systems on which whitebox measurement ofTAO’s performance was performed. Moreover, Quantify mod-ifies the binary code to collect timing information. It is mostuseful, therefore, to measure therelativeoverhead of differentoperations in a system, rather than measuringabsoluterun-time performance.

To avoid the limitations of Quantify, we therefore used alightweight timeprobe mechanism provided by ACE to pre-cisely pinpoint the amount of time spent in various ORB com-ponents and layers. The ACE timeprobe mechanism provides

highly accurate, low-cost timestamps that record the timespent between regions of code in a software system. Thesetimeprobes have minimal performance impact,e.g., 1-2�secoverhead per timeprobe, and no binary code instrumentationis required.

Depending on the underlying platform, ACE’s timeprobesare implemented either by high-resolution OS timers or byhigh-precision timing hardware. An example of the latter is theVMEtro board, which is a VME bus monitor. VMEtro writesunique ACE timeprobe values to an otherwise unused VMEaddress. These values record the duration between timeprobemarkers across multiple processors using a single clock. Thisenables TAO to collect synchronized timestamps and mea-sure communication delays end-to-end accurately across dis-tributed CPUs.

Below, we examine the client and server whitebox perfor-mance in detail.

5.3.2 Whitebox Results

Figure 17 shows the points in a two-way operation request pathwhere timeprobes were inserted. Each labeled number in thefigure corresponds to an entry in Table 1 and Table 2 below.The results presented in the tables and figures that follow wereaveraged over 1,000 samples.

Client performance: Table 1 depicts the time in microsec-onds (�s) spent in each sequential activity that a TAO clientperforms to process an outgoing operation request and its re-ply.

Direction Client Activities Absolute Time (�s)

Outgoing 1. Initialization 6.302. Get object reference 15.63. Parameter marshal 0.74 (param. dependent)4. ORB messaging send 7.785. ORB transport send 1.026. I/O 8.70 (op. dependent)7. ORB transport recv 50.78. ORB messaging recv 9.259. Parameter demarshal op. dependent

Table 1:�seconds Spent in Each Client Processing Step

Each client outgoing step is outlined below:

1. In theinitialization step, the client invocation is createdand constructors are called for the input and output CommonData Representation (CDR) stream objects, which handle mar-shaling and demarshaling of operation parameters.

2. TAO’s connector caches connections, so even thoughits connect method is called for every operation, existingconnections are reused for repeated calls. For statically config-ured systems, such as avionics mission computing, TAO pre-

18

establishes connections, so the initial connection setup over-head can be avoided entirely.

3. In the parameter marshalstep, the outgoingin andinout parameters are marshaled. The overhead of this pro-cessing depends on the operation signature,i.e., the number ofdata parameters and their type complexity.

4. In thesend operation in theORB messaginglayer, theclient creates a request header and frames the message. Themessaging layer then passes the message to the ORB trans-port component for transmission to the server. If the requestis a synchronous two-way operation, the transport componentwaits for and processes the response.

5. Thesend operation in theORB transportcomponentimplements the connection concurrency strategy and invokesthe appropriate ACE I/O operation. TAO maintains a linkedlist of CDR buffers [7], which allows it to use “gather-write”OS calls, such aswritev . Thus, multiple buffers can be writ-ten atomically without requiring multiple system calls or un-necessary memory allocation and data copying.

6. TheI/O operation represents the time the client spendsin the receive system call. This time is generally dominatedby the cost of copying data from the kernel to user suppliedbuffers.

Each client incoming step is outlined below:

7. TheI/O receiveoperation copies the data from a kernelbuffer to a receive CDR stream and returns control to the ORBtransport component.

8. The recv operation in theORB transportlayer dele-gates the reading of the received messages header and bodyto the ORB messaging component. If the message header isvalid, then the remainder of the message is read. This also in-cludes time when the client is blocked waiting for the serverto read the supplied data.

9. The recv operation in theORB messaginglayerchecks the message type of the reply, and either raises an ap-propriate exception, initiates a location forward, or returns thereply to the calling application.

10. In theparameter demarshalstep, the incoming replyout and inout parameters are demarshaled. The overheadof this step depends, as it does with the server, on the operationsignature.

Server performance: Table 2 depicts the time in microsec-onds (�s) spent in each activity as a TAO server processes arequest.

Each incoming server step is outlined below:

1. TheI/O operation represents the time the server spendsin theread system call.

Direction Server Activities Absolute Time (�s)

Incoming 1. I/O 7.0 (op. dependent)2. ORB transport recv 24.83. ORB messaging recv 4.54. Parsing object key 4.65. POA demux 1.396. Servant demux 4.67. Operation demux 4.528. User upcall 3.84 (op. dependent)

Outgoing 9. ORB messaging send 4.5610. ORB transport send 93.6

Table 2:�seconds Spent in Each Server Processing Step

2. The recv operation in theORB transportlayer dele-gates the reading of the received message header to the ORBmessaging component. If it is a valid message the remainingdata is read and passed to the ORB messaging component.

3. The recv operation in theORB messaginglayerchecks the type of the message and forwards it to the POA.Otherwise, it handles the message or reports an error back tothe client.

4. The Parsing object keystep comes before any otherPOA activity. The time in the table includes the acquisitionof a lock that is held through all POA activities,i.e., POA de-mux, servant demux, andoperation demux.

5. ThePOA demuxstep locates the POA where the servantresides. The time in this table is for a POA that is one-leveldeep, although in general, POAs can be many levels deep [7].

6. The servant demuxstep looks up a servant in the tar-get POA. The time shown in the table for this step is basedon TAO’s active demultiplexing strategy [7], which locates aservant in constant time regardless of the number of objects ina POA.

7. The skeleton associated with the operation resides inthe operation demuxstep. TAO uses perfect hashing [7] tolocate the appropriate operation.

8. In theparameter demarshalstep, the incoming requestin and inout parameters are demarshaled. As with theclient, the overhead of this step depends on the operation sig-nature.

9. The time for theuser upcallstep depends upon the ac-tual implementation of the operation in the servant.

Each outgoing server step is outlined below:

10. In the return value marshalstep, the return ,inout , andout parameters are marshaled. This time alsodepends on the signature of the operation.

19

11. The send operation in theORB messaginglayerpasses the marshaled return data down to the ORB transportlayer.

12. Thesend operation in theORB transportlayer addsthe appropriate IOP header to the reply, sends the reply, andcloses the connection if it detects an error. Also included inthe category is the time the server is blocked in thesend op-eration while the client runs.

13. The I/Osend operation gets the peer I/O handle fromthe server connection handler and calls the appropriatesendoperation. The server uses a gather-write I/O call, just like theclient-side I/Osend operation described above.

Depending on the type and number of operation parame-ters, theORB transport recvstep often requires the most ORBprocessing time. This time is dominated by the required datacopies. These costs can be reduced significantly by using atransport adapter that implements a shared buffer strategy.

Component costs: Figure 18 compares the relative over-

112

111

52 51

49 48

49 47

24 23

27 27

31

27

30

27

0

50

100

150

200

250

IIOP IIOP w/GIOPlite UIOP UIOP w/GIOPlite

Transport Protocol

To

tal T

ime

(use

cs)

OS and I/O ORB Transport Messaging

Figure 18: Comparison of ORB and Transport/OS OverheadUsing Timeprobes

head attributable to the ORB messaging component, transportadaptor, ORB and OS for two-wayIDL Cubit calls to thecube void operation for each possible protocol combina-tion. This figure shows that when using IIOP the I/O and OSoverhead accounts for just over 50% of the total round triplatency. It also shows that the difference in performance be-tween IIOP and UIOP is due primarily to the larger OS andI/O overhead of TCP/IP, compared with local IPC.

The only overhead that depends on size is(de)marshaling,which depends on the type complexity, number, and size ofoperation parameters, anddata copying, which depends on thesize of the data. In our whitebox experiment, only the param-eter size changes,i.e., thesequence s vary in length. More-over, TAO’s (de)marshaling optimizations [8] incur minimaloverhead when running between homogeneous ORB endsys-tems.

In Figure 19, the parameter size is varied and the above testis repeated. It shows that as the size of the operation parame-

64 64 65 65 65 66 73 81 85 93

77 77 77 77 77 77 78 76 76 77

27 27 27 27 28 28 29 31 33

3627 27 28 28 28 28

30

34 34

35

0

50

100

150

200

250

300

2 3 4 5 6 7 8 9 10 11

Bytes in Octet Sequence (powers of two)

To

tal T

ime

(use

cs)

OS and I/O ORB Transport Messaging

Figure 19: ORB and Transport/OS Overhead vs. ParameterSize

ters increases, I/O overhead grows faster than the overall ORBoverhead, including messaging and transport. This result illus-trates that the overall ORB overhead is largely independent ofthe request size. In particular, demultiplexing a request, creat-ing message headers, and invoking an operation upcall are notaffected by the size of the request.

TAO employs standard buffer size and data copy tradeoffoptimizations. This optimization is demonstrated in Figure 19by the fact that there is a slight increase in the time spent bothin the transport component and in the ORB itself when thesequence size is greater than 256 bytes. The data copy tradeoffoptimization is fully configurable via run-time command lineoptions, so it is possible to configure TAO to further improveperformance above the 256 byte data copy threshold.

For the operations tested in theIDL Cubit benchmark, theoverhead of the ORB is dominated by memory bandwidth lim-itations. Both the loopback driver and local IPC driver copydata within the same host. Therefore, memory bandwidth limi-tations should essentially be the same for both IIOP and UIOP.This result is illustrated in Figure 18 by the fact that the timespent in the ORB is generally constant for the four protocolcombinations shown.

In general, the use of UIOP demonstrates the advantagesof TAO’s pluggable protocols framework and how optimized,domain-specific protocols can be deployed.

6 Related Work

We have used TAO to research many dimensions of high-performance and real-time ORB endsystems, includingstatic [2] and dynamic [25] scheduling, request demultiplex-ing [7], dispatching [34], and event processing [22], ORB

20

Core connection [32] and concurrency architectures [35], IDLcompiler stub/skeleton optimizations for synchronous [8] andasynchronous [36] communication, I/O subsystem integra-tion [20], evaluation Real-time CORBA [11] features [19],fault tolerance features [14, 15], reflective QoS techniques theCORBA Component Model [37], multimedia streaming sup-port [21], systematic benchmarking of multiple ORBs [38],and patterns for ORB extensibility [28] and optimization [7].The design of TAO’s pluggable protocols framework is influ-enced by prior research on the design and optimization of pro-tocol frameworks for communication subsystems. This sec-tion outlines that research and compares it with our work.

Configurable communication frameworks: The x-kernel [39], Conduit+ [30], System V STREAMS [40],ADAPTIVE [41], and F-CSS [42] are all configurablecommunication frameworks that provide a protocol back-plane consisting of standard, reusable services that supportnetwork protocol development and experimentation. Theseframeworks support flexible composition of modular protocolprocessing components, such as connection-oriented and con-nectionless message delivery and routing, based on uniforminterfaces.

The frameworks for communication subsystems listedabove focus on implementing various protocol layers beneathrelatively low-level programming APIs, such as sockets. Incontrast, TAO’s pluggable protocols framework focuses onimplementing and/or adapting to transport protocols beneatha higher-level CORBA middleware API,i.e., the standardCORBA programming API. Therefore, existing communica-tion subsystem frameworks can provide building block proto-col components for TAO’s pluggable protocols framework.

Patterns-based communication frameworks: An increas-ing number of communication frameworks are being designedand documented using patterns [28, 30]. In particular, Con-duit+ [30] is an OO framework for configuring network pro-tocol software to support ATM signaling. Key portions of theConduit+ protocol framework,e.g., demultiplexing, connec-tion management, and message buffering, were designed us-ing patterns like Strategy, Visitor, and Composite [17]. Like-wise, the concurrency, connection management, and demulti-plexing components in TAO’s ORB Core and Object Adapteralso have been explicitly designed using patterns such as Re-actor, Acceptor-Connector, and Active Object [3].

CORBA pluggable protocols frameworks: The archi-tecture of TAO’s pluggable protocols framework is in-spired by the ORBacus [43] Open Communications Interface(OCI) [44]. The OCI framework provides a flexible, intuitive,and portable interface for pluggable protocols. The frameworkinterfaces are defined in IDL, with a few special rules to mapcritical types, such as data buffers.

Defining pluggable protocols interfaces with IDL permitsdevelopers to familiarize themselves with a single program-ming model that can be used to implement protocols in differ-ent languages. In addition, the use of IDL makes it possibleto write pluggable protocols that are portable among differentORB implementations and platforms.

However, using IDL also limits the the degree to whichvarious optimizations can be applied at the ORB and trans-port protocol levels. For example, efficiently handling localityconstrained objects, optimizing profile handling, strategizedbuffer allocation, or interfacing with optimized OS abstrac-tion layers/libraries are not generally supported by existingIDL compilers. Additionally, changes to an IDL compiler’smapping rules on a per protocol basis is prohibitive.

In our approach we use C++ classes and optimized frame-work interfaces to allow protocol developers to exploit newstrategies or available libraries. TAO uses the ACE frame-work [29] to isolate itself from non-portable aspects of under-lying operating systems. This design leverages the testing, op-timizations, implemented by ACE, enabling us to focus on theparticular problems of developing a high-performance, real-time ORB.

Our framework allows each protocol implementation to rep-resent a profile as it sees fit. Since these profiles are only cre-ated in a few instances, it is possible for them to parse the octetstream representation and store it in a more convenient format.The parsing can be also done on demand to minimize startuptime. The protocol implementor is free to choose the strategythat best fits the application.

TAO implements a highly optimized pluggable protocolsframework that is tuned for high-performance and real-timeapplication requirements. For example, TAO’s pluggable pro-tocols framework can be integrated with zero-copy high-speednetwork interfaces [23, 45, 20, 9], embedded systems [8],or high-performance communication infrastructures like FastMessages [18].

7 Concluding Remarks

To be an effective development platform for performance-sensitive applications, CORBA middleware must preserveend-to-end application QoS properties across the communica-tion layer. It is essential, therefore, to define a pluggable proto-cols framework that allows custom inter-ORB messaging andtransport protocols to be configured flexibly and transparentlyby CORBA applications.

This paper identifies the protocol-related limitations of cur-rent ORBs and describes a CORBA-based pluggable protocolsframework we developed and integrated with TAO to addressthese limitations. TAO’s pluggable protocols framework con-tains two main components: an ORB messaging component

21

and an ORB transport adapter component. These two com-ponents allows applications developers and end-users to ex-tend their communication infrastructure transparently to sup-port the dynamic and/or static binding of new ORB messagingand transport protocols. Moreover, TAO’s patterns-orientedOO design makes it straightforward to develop custom inter-ORB protocol stacks that can be optimized for particular ap-plication requirements and endsystem/network environments.

This paper illustrates the performance of TAO’s pluggableprotocols framework empirically when running CORBA ap-plications over high-speed interconnects, such as VME. Ourbenchmarking results demonstrate that applying appropriateoptimizations and patterns to CORBA middleware can yieldhighly efficient and predictable implementations, without sac-rificing flexibility or reuse. These results support our con-tention that CORBA middleware performance is largely an im-plementation issue. Thus, well-tuned, standard-based CORBAmiddleware like TAO can replacead hocand proprietary so-lutions that are still commonly used in traditional distributedapplications and real-time systems.

Most of the performance overhead associated with plug-gable protocols framework described in this paper stem from“out-of-band” creation operations, rather operations in the crit-ical path. We have shown how patterns can resolve key designforces to flexibly create and control the objects in the frame-work. Simple and efficient wrapper facades can then be usedto isolate the rest of the application from low-level implemen-tation details, without significantly affecting end-to-end per-formance.

References[1] M. Henning and S. Vinoski,Advanced CORBA Programming With

C++ . Addison-Wesley Longman, 1999.

[2] D. C. Schmidt, D. L. Levine, and S. Mungee, “The Design andPerformance of Real-Time Object Request Brokers,”ComputerCommunications, vol. 21, pp. 294–324, Apr. 1998.

[3] D. C. Schmidt, M. Stal, H. Rohnert, and F. Buschmann,Pattern-Oriented Software Architecture: Patterns for Concurrency andDistributed Objects, Volume 2. New York, NY: Wiley & Sons, 2000.

[4] F. Buschmann, R. Meunier, H. Rohnert, P. Sommerlad, and M. Stal,Pattern-Oriented Software Architecture - A System of Patterns. Wileyand Sons, 1996.

[5] Object Management Group,The Common Object Request Broker:Architecture and Specification, 2.2 ed., Feb. 1998.

[6] Object Management Group,Telecom Domain Task Force Request ForInformation Supporting Wireless Access and Mobility in CORBA -Request For Information, OMG Document telecom/98-06-04 ed., June1998.

[7] I. Pyarali, C. O’Ryan, D. C. Schmidt, N. Wang, V. Kachroo, andA. Gokhale, “Using Principle Patterns to Optimize Real-time ORBs,”Concurrency Magazine, vol. 8, no. 1, 2000.

[8] A. Gokhale and D. C. Schmidt, “Optimizing a CORBA IIOP ProtocolEngine for Minimal Footprint Multimedia Systems,”Journal onSelected Areas in Communications special issue on Service EnablingPlatforms for Networked Multimedia Systems, vol. 17, Sept. 1999.

[9] R. S. Madukkarumukumana and H. V. Shah and C. Pu, “HarnessingUser-Level Networking Architectures for Distributed ObjectComputing over High-Speed Networks,” inProceedings of the 2ndUsenix Windows NT Symposium, August 1998.

[10] Compaq, Intel, and Microsoft, “Virtual Interface Architecture, Version1.0.” http://www.viarch.org, 1997.

[11] Object Management Group,Realtime CORBA Joint RevisedSubmission, OMG Document orbos/99-02-12 ed., March 1999.

[12] F. Kon and R. H. Campbell, “Supporting Automatic Configuration ofComponent-Based Distributed Systems,” inProceedings of the5th

Conference on Object-Oriented Technologies and Systems, (San Diego,CA), USENIX, May 1999.

[13] Object Management Group,The Common Object Request Broker:Architecture and Specification, 2.3 ed., June 1999.

[14] B. Natarajan, A. Gokhale, D. C. Schmidt, and S. Yajnik, “DOORS:Towards High-performance Fault-Tolerant CORBA,” inProceedings ofthe 2nd International Symposium on Distributed Objects andApplications (DOA 2000), (Antwerp, Belgium), OMG, Sept. 2000.

[15] B. Natarajan, A. Gokhale, D. C. Schmidt, and S. Yajnik, “ApplyingPatterns to Improve the Performance of Fault-Tolerant CORBA,” inProceedings of the 7th International Conference on High PerformanceComputing (HiPC 2000), (Bangalore, India), ACM/IEEE, Dec. 2000.

[16] Object Management Group,CORBA Messaging Specification, OMGDocument orbos/98-05-05 ed., May 1998.

[17] E. Gamma, R. Helm, R. Johnson, and J. Vlissides,Design Patterns:Elements of Reusable Object-Oriented Software. Reading, MA:Addison-Wesley, 1995.

[18] M. Lauria, S. Pakin, and A. Chien, “Efficient Layering for High SpeedCommunication: Fast Messages 2.x.,” inProceedings of the 7th HighPerformance Distributed Computing (HPDC7) conference, (Chicago,Illinois), July 1998.

[19] C. O’Ryan, D. C. Schmidt, F. Kuhns, M. Spivak, J. Parsons, I. Pyarali,and D. Levine, “Evaluating Policies and Mechanisms for SupportingEmbedded, Real-Time Applications with CORBA 3.0,” inProceedingsof the6th IEEE Real-Time Technology and Applications Symposium,(Washington DC), IEEE, May 2000.

[20] F. Kuhns, D. C. Schmidt, C. O’Ryan, and D. Levine, “SupportingHigh-performance I/O in QoS-enabled ORB Middleware,”ClusterComputing: the Journal on Networks, Software, and Applications,2000.

[21] S. Mungee, N. Surendran, and D. C. Schmidt, “The Design andPerformance of a CORBA Audio/Video Streaming Service,” inProceedings of the Hawaiian International Conference on SystemSciences, Jan. 1999.

[22] T. H. Harrison, D. L. Levine, and D. C. Schmidt, “The Design andPerformance of a Real-time CORBA Event Service,” inProceedings ofOOPSLA ’97, (Atlanta, GA), ACM, October 1997.

[23] Z. D. Dittia, G. M. Parulkar, and J. R. Cox, Jr., “The APIC Approach toHigh Performance Network Interface Design: Protected DMA andOther Techniques,” inProceedings of INFOCOM ’97, (Kobe, Japan),pp. 179–187, IEEE, April 1997.

[24] H. Schulzrinne, S. Casner, R. Frederick, and V. Jacobson, “Rtp: Atransport protocol for real-time applications,”Network InformationCenter RFC 1889, January 1996.

[25] C. D. Gill, D. L. Levine, and D. C. Schmidt, “The Design andPerformance of a Real-Time CORBA Scheduling Service,”TheInternational Journal of Time-Critical Computing Systems, specialissue on Real-Time Middleware, to appear 2000.

[26] C. O’Ryan and D. C. Schmidt, “Applying a Real-time CORBA EventService to Large-scale Distributed Interactive Simulation,” in5

th

International Workshop on Object-oriented Real-Time DependableSystems, (Monterey, CA), IEEE, Nov 1999.

22

[27] B. Meyer,Object-Oriented Software Construction, Second Edition.Englewood Cliffs, NJ: Prentice Hall, 1997.

[28] D. C. Schmidt and C. Cleeland, “Applying a Pattern Language toDevelop Extensible ORB Middleware,” inDesign Patterns inCommunications(L. Rising, ed.), Cambridge University Press, 2000.

[29] D. C. Schmidt, “Applying Design Patterns and Frameworks to DevelopObject-Oriented Communication Software,” inHandbook ofProgramming Languages(P. Salus, ed.), MacMillan ComputerPublishing, 1997.

[30] H. Hueni, R. Johnson, and R. Engel, “A Framework for NetworkProtocol Software,” inProceedings of OOPSLA ’95, (Austin, Texas),ACM, October 1995.

[31] C. Cleeland, D. C. Schmidt, and T. Harrison, “External Polymorphism– An Object Structural Pattern for Transparently Extending ConcreteData Types,” inPattern Languages of Program Design(R. Martin,F. Buschmann, and D. Riehle, eds.), Reading, MA: Addison-Wesley,1997.

[32] D. C. Schmidt, S. Mungee, S. Flores-Gaitan, and A. Gokhale,“Software Architectures for Reducing Priority Inversion andNon-determinism in Real-time Object Request Brokers,”Journal ofReal-time Systems, special issue on Real-time Computing in the Age ofthe Web and the Internet, To appear 2000.

[33] P. S. Inc.,Quantify User’s Guide. PureAtria Software Inc., 1996.

[34] I. Pyarali, C. O’Ryan, and D. C. Schmidt, “A Pattern Language forEfficient, Predictable, Scalable, and Flexible Dispatching Mechanismsfor Distributed Object Computing Middleware,” inProceedings of theInternational Symposium on Object-Oriented Real-time DistributedComputing (ISORC), (Newport Beach, CA), IEEE/IFIP, Mar. 2000.

[35] D. C. Schmidt, “Evaluating Architectures for Multi-threaded CORBAObject Request Brokers,”Communications of the ACM special issue onCORBA, vol. 41, Oct. 1998.

[36] A. B. Arulanthu, C. O’Ryan, D. C. Schmidt, M. Kircher, andJ. Parsons, “The Design and Performance of a Scalable ORBArchitecture for CORBA Asynchronous Messaging,” inProceedings ofthe Middleware 2000 Conference, ACM/IFIP, Apr. 2000.

[37] N. Wang, D. C. Schmidt, K. Parameswaran, and M. Kircher, “ApplyingReflective Middleware Techniques to Optimize a QoS-enabled CORBAComponent Model Implementation,” in24th Computer Software andApplications Conference, (Taipei, Taiwan), IEEE, Oct. 2000.

[38] A. Gokhale and D. C. Schmidt, “Measuring the Performance ofCommunication Middleware on High-Speed Networks,” inProceedingsof SIGCOMM ’96, (Stanford, CA), pp. 306–317, ACM, August 1996.

[39] N. C. Hutchinson and L. L. Peterson, “Thex-kernel: An Architecturefor Implementing Network Protocols,”IEEE Transactions on SoftwareEngineering, vol. 17, pp. 64–76, January 1991.

[40] D. Ritchie, “A Stream Input–Output System,”AT&T Bell LabsTechnical Journal, vol. 63, pp. 311–324, Oct. 1984.

[41] D. C. Schmidt, D. F. Box, and T. Suda, “ADAPTIVE: A DynamicallyAssembled Protocol Transformation, Integration, and eValuationEnvironment,”Journal of Concurrency: Practice and Experience,vol. 5, pp. 269–286, June 1993.

[42] M. Zitterbart, B. Stiller, and A. Tantawy, “A Model forHigh-Performance Communication Subsystems,”IEEE Journal onSelected Areas in Communication, vol. 11, pp. 507–519, May 1993.

[43] I. Object Oriented Concepts, “ORBacus.” www.ooc.com/ob.

[44] I. Object-Oriented Concepts, “ORBacus User Manual - Version 3.1.2.”www.ooc.com/ob, 1999.

[45] T. v. Eicken, A. Basu, V. Buch, and W. Vogels, “U-Net: A User-LevelNetwork Interface for Parallel and Distributed Computing,” in15thACM Symposium on Operating System Principles, ACM, December1995.

23

Applying Patterns to Develop a Pluggable Protocols ... · speed protocols and networks, real-time embedded system in-terconnects, and standard TCP/IP protocols over the Internet.

Documents