Real-time Middleware for Distributed and Embedded Systemsschmidt/PDF/lucent.pdf · 1999-11-09 · Real-time Middleware for Distributed and Embedded Systems Douglas C. Schmidt Associate

Real-time Middleware forDistributed and Embedded Systems

Douglas C. SchmidtAssociate Professor & Computer Engineering Dept.www.cs.wustl.edu/�schmidt/ University of California, Irvine

SponsorsNSF, DARPA, Bellcore/Telcordia, BBN, Boeing, CDI/GDIS, Experian, Global MT,

Hughes, Kodak, Krones, Lockheed, Lucent, Microsoft, Motorola, Nokia, Nortel, OCI,OTI, Raytheon, SAIC, Siemens SCR, Siemens MED, Siemens ZT, Sprint, USENIX

November 1999

Real-time and Embedded ORBs Douglas C. Schmidt

Motivation: the Telecom Software Crisis

www.arl.wustl.edu/arl/� Symptoms

– Network element hardware getssmaller, faster, cheaper

– Communication software gets larger,slower, more expensive

� Culprits

– Inherent and accidental complexity

� Solution Approach

– Manage & control network elementsvia QoS-enabled middleware

Washington University, St. Louis 1

Real-time and Embedded ORBs Do

Goal: Apply Embedded Middleware toNetwork Element Mgmt & Control

MANAGEMENT

CLIENT

RIO

TAONETWORK OPERATIONS CENTER

VSI

MASTER

RIO

TAOSIGNALING PROCESSOR

VSI

MASTER

RIO

TAOSIGNALING PROCESSOR

VSI

SLAVECONTROL PROCESSOR

TAORIO

WUGSWUGSWUGS

ENDSYSTEM A

UNI

CLIENT

TAO

CONNECT REQUEST

ENDSYSTEM A

UNI

CLIENT

TAO

ATMSWITCH

ATMSWITCH

ATMSWITCH

VSI


TAORIO

VSI


TAORIO

ADD CONNECTION

GET STATS

EVENT

PORT DOWNADD CONNECTION

GET STATS

ADD CONNECTION

CONNECT REQUEST

INTER -ORB PROTOCOL

Domain Challenges

� High-speed (20 Gbps) ATM switchesw/embedded controllers

� Low-latency and statistical real-time deadlines

� COTS infrastructure, standards-based opensystems, and small footprint

Washington University, St. Louis


Problem: Low-level Switch Control& Management ( e.g., GSMP & VSI)

SIGNALING

CLIENT MASTER

CONTROLLER

SIGNALING PROCESSOR

TAO

NETWORK ELEMENT

SLAVE

CONTROLLER

SLAVE

CONTROLLER

CONNECT COMMIT

SWITCH

CONTROL MESSAGES

CONTROL LIBRARY API

Features

� Setup & releaseconnections

� Add & deletepoint-to-multipointleaves

� Manage ATM switchports

� Requestconfigurationinformation &statistics

Drawbacks

� Non-portable, tedious, and error-prone programmingAPIs



Proxy Server Configuration

SIGNALING

PROCESSOR

TAO

SIGNALING

PROCCESSOR

PROXY

MASTER

CONTROLLER

TAO

PROXY CONTROL

PROCESSOR

SLAVE

CONTROLLER

TAO

ATMSWITCH

ATMSWITCH

ATMSWITCH

NETWORK ELEMENT

Control cellsSLAVE

CONTROLLER

config portCONTROL

PROCESSOR

NETWORK ELEMENTNETWORK ELEMENT

CONTROL

PROCESSOR

CONNECTION

REQUEST

SIGNALING

PROCESSOR

TAO

SIGNALING

PROCCESSOR

GSMP: ADD_BRANCHVSI: CONNECT COMMIT

Inter-ORB Protocol

establish_connect()

Features

� Supports standardCORBA programmingAPI

� Can use standard ORB� Transparent to existing

GSMP & VSI servers

� Scales to distributedconfiguration

– i.e., one CP cancontrol multipleswitches



Embedded ORB Configuration

SLAVE

CONTROLLER

TAO

CONTROL PROCESSOR

NETWORK ELEMENT

ATMSWITCH

MANAGEMENT

AGENT

NETWORK OPERATIONS

CENTER

SIGNALING

PROCESSOR

MASTER

CONTROLLER

SIGNALING

PROCESSOR

ADD NEW

CONNECTION

GET PORT

STATUSEVENT

PORT DOWN

TAOTAO

INTER -ORB PROTOCOL

HOST A HOST B

Features

� Leverages middlewareflexibility andstandardization

� Multiple protocols can besupported

– GSMP & VSI in-linebridging, GIOP/IIOP,etc.

� Minimal footprint



Context: Levels of Abstraction inInternetworking and Middleware

HTTPFTP

FDDIATMETHERNET

IP

FIBRE CHANNEL

UDP TCP

TELNETDNS

LYNXOSLINUXWIN NT

CORBA

SOLARIS

CORBA SERVICES

CORBA

APPLICATIONS

VXWORKS

MIDDLEWARE ARCHINTERNETWORKING ARCH



Problem: Lack of QoS-enabled Middleware

LOWLOW--LEVELLEVELMIDDLEWAREMIDDLEWARE

OPERATINGOPERATING

SYSTEMS SYSTEMS &&PROTOCOLSPROTOCOLS

HIGHERHIGHER--LEVELLEVELMIDDLEWAREMIDDLEWARE

COMMONCOMMONSERVICESSERVICES

APPLICATIONSAPPLICATIONS

ConsConsEVENTEVENT

CHANNELCHANNELConsCons

ConsCons

� Many telecom applications requireQoS guarantees

– e.g., call-processing,network/switch management,wireless systems

� Building these applicationsmanually is hard

� Existing middleware doesn’tsupport QoS effectively

– e.g., CORBA, DCOM, DCE, Java� Solutions must be integrated

horizontally & vertically



Candidate Solution: CORBA

INTERFACE

REPOSITORY

IMPLEMENTATION

REPOSITORY

IDLCOMPILER

DII ORBINTERFACE

ORBORB CORECORE GIOPGIOP//IIOPIIOP//ESIOPSESIOPS

IDLIDLSTUBSSTUBS

operation()operation()in args

out args + return value

CLIENTOBJECT(SERVANT)

OBJ

REF

STANDARD INTERFACE STANDARD LANGUAGE MAPPING

ORB-SPECIFIC INTERFACE STANDARD PROTOCOL

INTERFACE

REPOSITORY

IMPLEMENTATION

REPOSITORY

IDLCOMPILER

IDLSKELETON

DSI

OBJECT

ADAPTER

www.cs.wustl.edu/�schmidt/corba.html

Goals of CORBA

� Simplify distributionby automating

– Object location &activation

– Parametermarshaling

– Demultiplexing– Error handling

� Provide foundationfor higher-levelservices



Caveat: Requirements/Limitations ofCORBA for Telecom

NETWORKNETWORKOPERATIONSOPERATIONS

CENTERCENTER

HSMHSM

ARCHIVEARCHIVE

SERVERSERVER

AGENTAGENT

INTERACTIVEINTERACTIVE

AUDIOAUDIO//VIDEOVIDEO

AGENTAGENT ARCHITECTUREARCHITECTURE

SPCSPC

HARDWAREHARDWARE

EMBEDDEDEMBEDDED

TAOTAO

MIBMIB

AGENTAGENT

www.cs.wustl.edu/�schmidt/RT-ORB.ps.gz

Requirements

� Location transparency

� Performancetransparency

� Predictabilitytransparency

� Reliability tranparency

Limitations

� Lack of QoS specifications

� Lack of QoS enforcement

� Lack of real-timeprogramming features

� Lack of performanceoptimizations



Overview of the Real-time CORBA Specification

OS KERNEL

OS I/O SUBSYSTEM

NETWORK ADAPTERS

STANDARD

SYNCHRONIZERS

END-TO-END PRIORITY

PROPAGATION

ORB CORE

OBJECT

ADAPTER

CLIENT

GIOP

PROTOCOL

PROPERTIES

THREAD POOLS

EXPLICIT

BINDING

NETWORK

OS KERNEL

OS I/O SUBSYSTEM

NETWORK ADAPTERS

operation()

out args + return value

in args

OBJECT

REF

OBJECT

(SERVANT)

STUBSSKELETON

Features

1. End-to-end prioritypropagation

2. Protocol properties

3. Thread pools

4. Explicit binding

5. Standardsynchronizers

www.cs.wustl.edu/�schmidt/oorc.ps.gz



Our Approach: The ACE ORB (TAO)

NETWORKNETWORK

ORBORB RUN RUN--TIMETIME

SCHEDULERSCHEDULER

operation()operation()

IDLIDLSTUBSSTUBS

IDLIDLSKELETONSKELETON

in argsin args

out args + return valueout args + return value

CLIENTCLIENT

OS KERNELOS KERNEL

HIGHHIGH--SPEEDSPEED

NETWORK INTERFACENETWORK INTERFACE

REALREAL--TIME ITIME I//OOSUBSYSTEMSUBSYSTEM

OBJECTOBJECT((SERVANTSERVANT))

OS KERNELOS KERNEL

HIGHHIGH--SPEEDSPEED

NETWORK INTERFACENETWORK INTERFACE

REALREAL--TIME ITIME I//OOSUBSYSTEMSUBSYSTEM

ACEACE COMPONENTSCOMPONENTS

OBJOBJ

REFREF

REALREAL--TIMETIME ORBORB CORECOREIOPIOP

PLUGGABLEPLUGGABLE

ORBORB & & XPORTXPORT

PROTOCOLSPROTOCOLS

IOPIOPPLUGGABLEPLUGGABLE

ORBORB & & XPORTXPORT

PROTOCOLSPROTOCOLS

REALREAL--TIMETIME

OBJECTOBJECT

ADAPTERADAPTER

www.cs.wustl.edu/�schmidt/TAO.html

TAO Overview !

� An open-source,standards-based,real-time,high-performanceCORBA ORB

� Runs onPOSIX/UNIX,Win32, & RTOSplatforms– e.g., VxWorks,

Chorus, LynxOS

� Leverages ACE



The ADAPTIVE Communication Environment (ACE)

PROCESSES//THREADSTHREADS

DYNAMICDYNAMIC

LINKINGLINKING

MEMORYMEMORY

MAPPINGMAPPING

SELECTSELECT//IO COMPIO COMP

SYSTEMSYSTEM

VV IPCIPCSTREAMSTREAM

PIPESPIPES

NAMEDNAMED

PIPESPIPES

CAPISS

SOCKETSSOCKETS//TLITLI

COMMUNICATIONCOMMUNICATION

SUBSYSTEMSUBSYSTEM

VIRTUAL MEMORYVIRTUAL MEMORY

SUBSYSTEMSUBSYSTEM

GENERAL POSIX AND WIN32 SERVICES

PROCESSPROCESS//THREADTHREAD

SUBSYSTEMSUBSYSTEM

FRAMEWORKS ACCEPTORACCEPTOR CONNECTORCONNECTOR

SELF-CONTAINED

DISTRIBUTED

SERVICE

COMPONENTS

NAMENAME

SERVERSERVER

TOKENTOKEN

SERVERSERVER

LOGGINGLOGGING

SERVERSERVER

GATEWAYGATEWAY

SERVERSERVER

SOCKSOCK__SAPSAP//TLITLI__SAPSAP

FIFOFIFO

SAPSAP

LOGLOG

MSGMSG

SERVICESERVICE

HANDLERHANDLER

TIMETIME

SERVERSERVER

C++WRAPPER

FACADES

SPIPESPIPE

SAPSAP

CORBACORBA

HANDLERHANDLER

SYSVSYSVWRAPPERSWRAPPERS

SHAREDSHARED

MALLOCMALLOC

THE ACE ORBTHE ACE ORB

((TAOTAO))

JAWS ADAPTIVEJAWS ADAPTIVE

WEB SERVERWEB SERVER

MIDDLEWARE

APPLICATIONS

REACTORREACTOR//PROACTORPROACTOR

PROCESSPROCESS//THREADTHREAD

MANAGERSMANAGERS

STREAMSSTREAMS

SERVICESERVICE

CONFIGCONFIG--URATORURATOR

SYNCHSYNCH

WRAPPERSWRAPPERS

MEMMEM

MAPMAP

OS ADAPTATION LAYER

www.cs.wustl.edu/�schmidt/ACE.html

ACE Overview !

� A concurrentOO networkingframework

� Available inC++ and Java

� Ported toPOSIX, Win32,and RTOSs

Related work !

� x-Kernel

� SysVSTREAMS



ACE and TAO Statistics

� Over 35 person-years of effort

– ACE > 200,000 LOC– TAO > 125,000 LOC– TAO IDL compiler > 100,000 LOC– TAO CORBA Object Services >

150,000 LOC

� Ported to UNIX, Win32, MVS, andRTOS platforms

� Large user community

– www.cs.wustl.edu/�schmidt/ACE-users.html

� Currently used by dozensof companies

– Bellcore, Boeing,Ericsson, Kodak,Lockheed, Lucent,Motorola, Nokia, Nortel,Raytheon, SAIC,Siemens, etc.

� Supported commercially

– ACE !

www.riverace.com– TAO !

www.ociweb.com



Applying TAO to Avionics Mission Computing

REPLICATION

SERVICE

OBJECT REQUEST BROKER

1: SENSORS

GENERATE

DATA

FLIRGPS IFF

3:PUSH (EVENTS)

2: SENSOR PROXIES DEMARSHAL DATA

& PASS TO EVENT CHANNEL

3:PUSH (EVENTS)

EVENT

CHANNEL

HUD NavAir

FrameWTS

4: PULL(DATA)

www.cs.wustl.edu/�schmidt/JSAC-98.ps.gz

Domain Challenges

� Deterministic & statisticalreal-time deadlines

� Periodic & aperiodic processing

� COTS and open systems

� Reusable components

� Support platform upgrades

www.cs.wustl.edu/�schmidt/TAO-boeing.html



Applying TAO to Other Performance-SensitiveApplications

'&

$%

DIAGNOSTIC STATIONSDIAGNOSTIC STATIONS

ATMATMMANMAN

ATM

LAN

ATM

LAN

MODALITIES

(CT, MR, CR) CENTRAL

BLOB STORE

CLUSTER

BLOB

STORE

DX

BLOB

STORE

'&

$%

WIDE AREA

NETWORK

SATELLITESSATELLITESTRACKINGTRACKINGSTATIONSTATION

PEERSPEERS

STATUS INFO

COMMANDS BULK DATA

TRANSFER

LOCAL AREA NETWORK

GROUNDSTATION

PEERS

GATEWAY

Medical Imaging Satellite Surveillance



Problem: Optimizing Complex Software

DIAGNOSTIC STATIONSDIAGNOSTIC STATIONS

ATMATMMANMAN

ATM

LAN

ATM

LAN

MODALITIES

(CT, MR, CR) CENTRAL

BLOB STORE

CLUSTER

BLOB

STORE

DX

BLOB

STORE

www.cs.wustl.edu/�schmidt/JSAC-99.ps.gz

Common Problems !

� Optimizing complex softwareis hard

� Small “mistakes” can be costly

Solution Approach (Iterative) !

� Pinpoint overhead viawhite-box metrics– e.g., Quantify and

VMEtro

� Apply patterns and frameworkcomponents

� Revalidate via white-box andblack-box metrics



Solution 1: Patterns and Framework Components

ACCEPTORCONNECTOR

ABSTRACT

FACTORY

SERVANTCLIENT

OS KERNELOS KERNEL

ACTIVE

OBJECT

THREAD-SPECIFIC

STORAGE

SERVICE

CONFIGURATOR

STRATEGY

REACTORREACTOR

WRAPPER FACADESWRAPPER FACADES

www.cs.wustl.edu/�schmidt/ORB-patterns.ps.gz

Definitions

� Pattern

– A solution to a problem ina context

� Framework

– A “semi-complete”application built withcomponents

� Components

– Self-contained, “pluggable”ADTs



Solution 2: ORB OptimizationPrinciple Patterns

Definition

� Optimization principle patterns documentrules for avoiding common design andimplementation problems that can degrade theperformance, scalability, and predictability ofcomplex systems

Key Principle Patterns Used in TAO

# Principle Pattern1 Optimize for the common case2 Remove gratuitous waste3 Replace inefficient general-purpose

functions with efficient special-purpose ones4 Shift computation in time, e.g., precompute5 Store redundant state to speed-up

expensive operations6 Pass hints between layers and components7 Don’t be tied to reference implementations/models8 Use efficient/predictable data structures



ORB Latency and Priority Inversion Experiments

��

C 1C 0

Requests

C n

��

Client

...

...

2ATM Switch

Ultra 2 Ultra 2

I/O SUBSYSTEM

Server

Object Adapter

ORB Core

Servants RU

NT

IME

SCH

ED

UL

ER

www.cs.wustl.edu/�schmidt/RT-perf.ps.gz

Method

� Vary ORBs, hold OSconstant

� Solaris real-time threads

� High priority client C0

connects to servant S0

with matching priorities� Clients C1 : : : Cn have

same lower priority

� Clients C1 : : : Cn

connect to servant S1

� Clients invoke two-wayCORBA calls that cube anumber on the servantand returns result



ORB Latency and Priority Inversion Results

0

4

8

12

16

20

24

28

32

36

40

44

48

52

1 5 10 15 20 25 30 35 40 45 50

Number of Low Priority Clients

Late

ncy p

er

Tw

o-w

ay R

eq

uest

in M

illiseco

nd

s

CORBAplus High Priority CORBAplus Low Priority

MT-ORBIX High Priority MT-ORBIX Low Priority

miniCOOL High Priority miniCOOL Low Priority

TAO High Priority TAO Low Priority

Synopsis of Results

� TAO’s latency is lowest forlarge # of clients

� TAO avoids priority inversion

– i.e., high priority clientalways has lowest latency

� Primary overhead stemsfrom concurrency andconnection architecture

– e.g., synchronization andcontext switching



ORB Jitter Results

1 5 10 15 20 25 30 35 40 45 50

TAO High Priority

TAO Low Priority

miniCOOL High Priority

miniCOOL Low Priority

MT-ORBIX High Priority

MT-ORBIX Low Priority

CORBAplus High Priority

CORBAplus Low Priority

0

50

100

150

200

250

300

Jitt

er in

mill

isec

on

ds

Number of Low Priority Clients

Definition

� Jitter ! standarddeviation fromaverage latency

Synopsis of Results

� TAO’s jitter islowest and mostconsistent

� CORBAplus’ jitteris highest andmost variable



Problem: Improper ORB Concurrency Models

OBJECTOBJECTADAPTERADAPTER

SERVANTSERVANT DEMUXERDEMUXER

SERVANTSERVANTSKELETONSSKELETONS

U

ORBORB CORECORE

FILTERFILTER

FILTERFILTER

FILTERFILTER

SERVANTSERVANTSKELETONSSKELETONS

U

2: enqueue(data)2: enqueue(data)

3: dequeue,3: dequeue,filterfilter

request,request,&enqueue&enqueue

4: dispatch4: dispatchupcall()upcall()

II//OO SUBSYSTEMSUBSYSTEM

THREADTHREAD

FILTERFILTER

1: recv()1: recv()

CONNECTION THREADSCONNECTION THREADS

Common Problems

� High context switching andsynchronization overhead

� Thread-level andpacket-level priorityinversions

� Lack of application controlover concurrency model

www.cs.wustl.edu/�schmidt/CACM-arch.ps.gz



Problem: ORB Shared Connection Models

SERVERSERVERORBORB CORECORE



COMMUNICATIONCOMMUNICATION LINKLINK


SERVANTSSERVANTS

ONEONE TCPTCP

CONNECTIONCONNECTION

ORBORB CORECORE

SEMAPHORESSEMAPHORES

2: select()2: select()4: release()

3: read()

BO

RR

OW

ED

TH

RE

AD

BO

RR

OW

ED

TH

RE

AD

S

APPLICATION

5: dispatch_return()

1: invoke_twoway()

www.cs.wustl.edu/�schmidt/RTAS-98.ps.gz

Common Problems

� Request-levelpriority inversions

– Sharing multiplepriorities on asingle connection

� Complex connectionmultiplexing

� Synchronizationoverhead



Problem: High Locking Overhead

0

94

199

175

0

231216

599

0

100

200

300

400

500

600

700

TAO miniCOOL CORBAplus MT ORBIX

ORBs Tested

Use

r L

evel

Lo

ck O

per

atio

ns

per

Req

ues

t

client server

Common Problems

� Locking overhead affectslatency and jitter significantly

� Memory managementcommonly involves locking

www.cs.wustl.edu/�schmidt/RTAS-98.ps.gz



Solution: TAO’s ORB Endsystem Architecture

RR

UU

NN

TT

II

MM

EE

S S

CC

HH

EE

DD

UU

LL

EE

RR

HIGH-SPEED NETWORKINTERFACES(e.g., APIC, VME)

ZZ

EE

RR

OO

CC

OO

PP

YY

BB

UU

FF

FF

EE

RR

SSRTRT I/O I/OSUBSYSTEMSUBSYSTEM

RTRT OBJECTOBJECTADAPTERADAPTER

OO(1) (1) REQUESTREQUEST DEMUXERDEMUXER

CLIENTSCLIENTS

STUBSSTUBS

SERVANTSSERVANTS

SKELETONSSKELETONS

U

RTRT ORBORB CORECORE

REACTORREACTOR

((PP11))

REACTORREACTOR

((PP22))

REACTORREACTOR

((PP33))

REACTORREACTOR

((PP44))

SOCKETSOCKET QUEUEQUEUE DEMUXERDEMUXER

PLUGGABLEPLUGGABLE PROTOCOLSPROTOCOLS

Solution Approach !

� Integrate scheduler into ORBendsystem

� Co-schedule threads

� Leader/followers thread pool

Principle Patterns !

� Pass hints, precompute,optimize common case,remove gratuitous waste,store state, don’t be tied toreference implementations &models



Thread Pool Comparison Results

SERVANTSSERVANTS

ORBORB CORECORE

1: select()1: select()


5: dispatch upcall()5: dispatch upcall()

2: read()2: read()

3: enqueue()3: enqueue()

4: dequeue()4: dequeue()

II//OOTHREADTHREAD

WORKER THREADSWORKER THREADS

REQUESTREQUEST

QUEUEQUEUE

Worker Thread Pool

SERVANTSSERVANTS

ORBORB CORECORE

SEMAPHORESEMAPHORE

LEADERLEADER FOLLOWERSFOLLOWERS

1: select()1: select()3: release()3: release()


4: dispatch upcall()4: dispatch upcall()

2: read()2: read()

Leader/FollowerThread Pool

1 2 3 45

6

0

25

5075

100

0

500

1000

1500

2000

2500

3000

Pe

rfo

rma

nc

e Im

pro

ve

me

nt

Threads

Lo

ad



Problem: Reducing Demultiplexing Latency

OP

ER

AT

ION1

OP

ER

AT

ION2

OP

ER

AT

IONK

2: DEMUX TO

I/O HANDLE

...

...

...POA1

OS I/O SUBSYSTEM

NETWORK ADAPTERS

SERVANT 1

5: DEMUX TO

SKELETON

6: DISPATCH

OPERATION

1: DEMUX THRU

PROTOCOL STACK

4: DEMUX TO

SERVANT

ORB COREORB CORE

ROOT POA3: DEMUX TO

OBJECT

ADAPTER

POA2 POAN

SERVANT N

...SKEL 1 SKEL 2 SKEL N

SERVANT 2

OS

KERNEL

LAYER

ORB

LAYER

SERVANT

LAYER

Design Challenges

� Minimize demuxing layers

� Provide O(1) operationdemuxing through all layers

� Avoid priority inversions

� Remain CORBA-compliant

www.cs.wustl.edu/�schmidt/POA.ps.gz



Solution: TAO’s Request Demultiplexing Optimizations

.........

.........IDLIDL

SKEL SKEL 22

OBJECT ADAPTER

SERVANT SERVANT 500500

(A)(A) LAYERED DEMUXING LAYERED DEMUXING,,LINEAR SEARCHLINEAR SEARCH

SERVANT SERVANT 11 SERVANT SERVANT 22

IDLIDL

SKEL SKEL 11

OP

ER

AT

ION

OP

ER

AT

ION

100

100

OP

ER

AT

ION

OP

ER

AT

ION

22

.........

OP

ER

AT

ION

OP

ER

AT

ION

11

IDLIDL

SKEL NSKEL N

... ..................

SE

RV

AN

TS

ER

VA

NT500::

500::

OP

ER

AT

ION

OP

ER

AT

ION

100

100

SE

RV

AN

TS

ER

VA

NT500::

500::

OP

ER

AT

ION

OP

ER

AT

ION

11

SE

RV

AN

TS

ER

VA

NT1::

1::

OP

ER

AT

ION

OP

ER

AT

ION

100

100

SE

RV

AN

TS

ER

VA

NT1::

1::

OP

ER

AT

ION

OP

ER

AT

ION

22

SE

RV

AN

TS

ER

VA

NT1::

1::

OP

ER

AT

ION

OP

ER

AT

ION

11

OBJECT ADAPTER

.........

.........IDLIDL

SKEL SKEL 22

OBJECT ADAPTER

(C) LAYERED DEMUXING,PERFECT HASHING

SERVANT SERVANT 11 SERVANT SERVANT 22

OP

ER

AT

ION

OP

ER

AT

ION

100

100

OP

ER

AT

ION

OP

ER

AT

ION

22

.........

OP

ER

AT

ION

OP

ER

AT

ION

11

IDLIDL

SKEL NSKEL N

search(object key)

hash(operation)

IDLIDL

SKEL SKEL 11

(D) DE-LAYERED ACTIVE DEMUXING

hash(object key)

search(operation)

index(object key/operation)

(B) LAYERED DEMUXING,DYNAMIC HASHING

SERVANT SERVANT 500500

Demuxing

� www.cs.wustl.edu/�schmidt/

fieee tc-97,COOTS-99g.ps.gz

Perfect hashing

� www.cs.wustl.edu/�schmidt/gperf.ps.gz



POA Demultiplexing Results

15

1015

2025

012345

Lat

ency

(us)

POA Depth

TransientPersistent

Synopsis of Results

� Active demux isefficient & predictablefor both transient andpersistent objectreferences.

Principle Patterns

� Precompute, passhints, usespecial-purpose &predictable datastructures



Servant Demultiplexing Results

1 100 200 300 400 500

0

50

100

150

200

Lat

ency

(u

s)

No. of Objects

Active Demux

Dynamic Demux

Linear Demux

Synopsis of Results

� Linear demux iscostly

� Active demux ismost efficient &predictable

Principle Patterns

� Precompute, passhints, usespecial-purpose &predictable datastructures



Operation Demultiplexing Results

110

2030

4050

0

5

10

15

20

25

Lat

ency

(u

s)

No. of Methods

Perfect Hashing

Binary Search

Dynamic Hashing

Linear Search

Synopsis of Results !

� Perfect Hashing– Highly predictable– Low-latency

� Others strategiesslower

Principle Patterns !

� Precompute, usepredictable datastructures, removegratuitous waste



TAO Request Demultiplexing Summary

.........

.........POAPOA11

SERVANT SERVANT 11

ORB COREORB CORE

ROOT POAACTIVE

DEMUXING

POA2 POAN

SERVANT N

...SKEL 1 SKEL 2 SKEL N

SERVANT 2

ACTIVE

DEMUXING

PERFECTHASHING

Demultiplexing Stage Absolute Time ( �s)1. Request parsing 22. POA demux 23. Servant demux 34. Operation demux 25. Parameter demarshaling operation dependent6. User upcall servant dependent7. Results marshaling operation dependent



Real-time ORB/OS Performance Experiments

[P ]0

[P ]1

SCH

ED

UL

ER

0

RU

NT

IME

[P ]

I/O SUBSYSTEM

Server

0 nS1

nC

��Pentium II

S S0C 1C

...

...

Object AdapterServants

ORB Core

Client

...

[P]

Requests

Priority

...

[P ][P ]1

[P ]

n

n

www.cs.wustl.edu/�schmidt/RT-OS.ps.gz

Method

� Vary OS, hold ORBsconstant

� Single-processor IntelPentium II 450 Mhz, 256Mbytes of RAM

� Client and servant run onthe same machine

� Client Ci connects toservant Si with priority Pi

– i ranges from 1 : : : 50

� Clients invoke two-wayCORBA calls that cube anumber on the servant andreturns result



Real-time ORB/OS Performance Results'&

$%

0

200

400

600

800

1000

1200

1400

1600

1800

2000

0 1 2 5 10 15 20 25 30 35 40 45 50

Low Priority Clients

Tw

o-w

ay R

equ

est

Lat

ency

, use

c

Linux

LynxOS

NT

Solaris

VxWorks

'&

$%

0

2000

4000

6000

8000

10000

12000

14000

16000

18000

20000

0 1 2 5 10 15 20 25 30 35 40 45 50

Low Priority ClientsT

wo

-way

Req

ues

t L

aten

cy, u

sec

Linux

LynxOS

NT

Solaris

VxWorks

High-priority Client Latency Low-priority Clients Latency



Real-time ORB/OS Jitter Results'&

$%

0

2

10

20

30

40

50

0

500

1000

1500

2000

2500

3000

3500

Tw

o-w

ay J

itte

r, u

sec


LynxOSNTVxWorksLinuxSolaris

'&

$%

0 2

10

20

30

40

50

0

5000

10000

15000

20000

25000

30000

35000

40000

Tw

o-w

ay J

itte

r, u

sec


LynxOSLinuxSolarisNTVxWorks

High-priority Client Jitter Low-priority Clients Jitter



Problem: Hard-coded ORB Messagingand Transport Protocols

BOARD BOARD 11

VMEVME

15531553

BOARD BOARD 22� GIOP/IIOP are not sufficient, e.g.:

– GIOP message footprint may betoo large

– TCP lacks necessary QoS– Legacy commitments to existing

protocols

� Many ORBs do not support“pluggable protocols”

– This makes ORBs inflexible andinefficient



One Solution: Hacking GIOP

� GIOP requests include fields that aren’t neededin homogeneous embedded applications

– e.g., GIOP magic #, GIOP version, byte order,request principal, etc.

� These fields can be omitted without any changesto the standard CORBA programming model

� TAO’s -ORBgioplite option save 15 bytesper-request, yielding these calls-per-second:

Marshaling-enabled Marshaling-disabledmin max avg min max avg

GIOP 2,878 2,937 2,906 2,912 2,976 2,949GIOPlite 2,883 2,978 2,943 2,911 3,003 2,967

� The result is a measurable improvement inthroughput/latency

– However, it’s so small (2%) that hacking GIOPis of minimal gain except for low-bandwidthlinks



Better Solution: TAO’sPluggable Protocols Framework

CLIENT

STUBS SKELETONS

TCP

MULTICAST

IOP

VMEUDP

ORB MESSAGING COMPONENT

ORB TRANSPORT ADAPTER COMPONENT

ESIOP

REAL -TIME

IOPEMBEDDED

IOP

RELIABLE,BYTE-STREAM

ATMTCP

MEMORY

MANAGEMENT

CONCURRENCY

MODEL

OTHER

ORBCORE

SERVICES

COMMUNICATION INFRASTRUCTUREHIGH SPEED NETWORK INTERFACE

REAL -TIME I /O SUBSYSTEM

ORB MESSAGE

FACTORY

ORB TRANSPORT

ADAPTER FACTORY

OBJECT ADAPTER

GIOP GIOPLITE

ADAPTIVE Communication Environment (ACE)

OBJECT (SERVANT)operation (args)IN ARGS

OUT ARGS & RETURN VALUE

POLICY

CONTROL

Features

� Pluggable ORBmessaging andtransport protocols

� Highly efficient andpredictable behavior

Principle Patterns

� Replacegeneral-purposefunctions(protocols) withspecial-purposeones



CORBA Protocol Interoperability Architecture

ORB MESSAGING

COMPONENT

ORB TRANSPORT

ADAPTER COMPONENT

TRANSPORT LAYER

NETWORK LAYER

GIOP

IIOP

TCP

IP

VME

DRIVER

AAL 5

ATM

GIOPLITE

VME-IOP

ESIOP

ATM -IOPRELIABLE

SEQUENCED

PROTOCOL CONFIGURATIONS

STANDARD CORBA PROGRAMMING API Features !

� Presentation layer– e.g., CDR

� Message formats– e.g., GIOP

� Transportassumptions– e.g., TCP

� Object addressing– e.g., IIOP IOR

www.cs.wustl.edu/�schmidt/pluggable protocols.ps.gz



Embedded System Benchmark Configuration

MARSHALMARSHAL

PARAMSPARAMS

OBJECT OBJECT ((SERVANTSERVANT))

OBJECT ADAPTEROBJECT ADAPTER

ACTIVEACTIVE

OBJECTOBJECT

MAPMAP


VME BUSVME BUS

OS KERNELOS KERNEL

VMEVME

DRIVERDRIVER

OS KERNELOS KERNEL

VMEVME

DRIVERDRIVER

IDLIDLSTUBSSTUBS

ORB MESSAGINGORB MESSAGING

ORB TRANSPORTORB TRANSPORT

CLIENTCLIENT

INC

OM

ING

INC

OM

ING

CDRCDR

DATA COPYDATA COPY

DATA COPYDATA COPY

DMADMA COPY COPY



CDRCDR

DATA COPYDATA COPY

DEMARSHALDEMARSHAL

PARAMSPARAMS

DATA COPYDATA COPY

OU

TG

OIN

GO

UT

GO

ING

ORBORBCORECORE

obj->op (params)

VxWorks running on 200 Mhz PowerPC over a 320 Mbps VME & 10Mbps Ethernet



Ethernet & VME Two-way LatencyResults

0 1000 2000 3000 4000 5000 6000 7000 8000 9000

void

short

octet

long

struct

short seq <octet>

long seq <octet>

short seq <long>

long seq <long>

Data T

ype

Latency (usec)

VME/GIOPlite avg.

VME/GIOP avg.

Ethernet/GIOPlite avg.

Synopsis of Results

� VME protocol ismuch faster thanEthernet

� No application changesare required to supportVME



Pinpointing ORB Overhead with VMEtro Timeprobes


RECVRECV

OBJECT (SERVANT)

ORBORBCORECORE

OBJECT ADAPTEROBJECT ADAPTER


VME BUSVME BUS

OS KERNELOS KERNEL

VME DRIVERVME DRIVER

OS KERNELOS KERNEL

VME DRIVERVME DRIVER



ACTIVEACTIVE

OBJECTOBJECT

MAPMAP

GET OBJECTGET OBJECT

REFREF

IDLIDLSTUBSSTUBS

MARSHALMARSHAL

PARMATERSPARMATERS


SENDSEND




SENDSEND

II//O SENDO SEND

II//O RECVO RECV77

88


RECVRECV99

CLIENTCLIENT

DEMARSHALDEMARSHAL

PARAMETERSPARAMETERS

7799

II//O RECVO RECV


RECVRECV


RECVRECV

PARSE OBJECTPARSE OBJECT

KEY KEY

OBJECTOBJECT

DEMUXDEMUX

DEMARSHALDEMARSHAL


USER UPCALLUSER UPCALL


SENDSEND


SENDSEND

II//O SENDO SEND

MARSHALMARSHAL


OPERATIONOPERATION

DEMUXDEMUX

131311

22

44

661111

1010

1212

33

88INITIALIZATIONINITIALIZATION

33

55

44

11

22

66

1010

OUTGOINGOUTGOING INCOMINGINCOMING

INC

OM

ING

INC

OM

ING

OU

TG

OIN

GO

UT

GO

ING

POAPOA

DEMUXDEMUX55

� Timeprobes use VMEtro monitor,which measures end-to-end time

� Timeprobe overhead isminimal, i.e., 1 �sec



ORB & VME One-way Overhead Results

200

200

198

200

203

205

212

226

257

309

51

49 50

50 49

51

50

51

52

56

24

23 23

22 23

23

22

23

23

23

68

66 66 66 66 66 67

66

69

72

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

4 8 16 32 64 128 256 512 1024 2048Sequence length (bytes)

usecs

VME + OS overhead ORB client pre-write

ORB client post-write ORB server overhead

Synopsis of Results

� ORB overhead is relatively constant and low

– e.g., �110 �secs per end-to-end operation

� Bottleneck is VME driver and OS, not ORB



ORB & Transport Overhead Results

112

111

52 51

49 48

49 47

24 23

27 27

31

27

30

27

0

50

100

150

200

250

IIOP IIOP w/GIOPlite UIOP UIOP w/GIOPlite

Transport Protocol

To

tal T

ime

(use

cs)

OS and I/O ORB Transport Messaging

Synopsis of Results

� ORB overhead is relatively constant and low

– e.g., �49 �secs per two-way operation

� Bottleneck is OS and I/O operation



Lessons Learned Developing Real-time ORBs

� Avoid dynamic connection management

� Minimize dynamic memory managementand data copying

� Avoid multiplexing connections fordifferent priority threads

� Avoid complex concurrency models

� Integrate ORB with OS and I/Osubsystem and avoid reimplementingOS mechanisms

� Guide ORB design by empiricalbenchmarks and patterns

'&

$%'

&

$%



Summary of TAO Research ProjectCompleted work

� First POA and first deployedreal-time CORBA schedulingservice

� Pluggable protocols framework

� Minimized ORB Core priorityinversion and non-determinism

� Reduced latency via demuxingoptimizations

� Co-submitters on OMG’sreal-time CORBA spec

Ongoing work

� Dynamic/hybrid scheduling

� Distributed QoS, ATM I/OSubsystem, & open signaling

� Implement CORBA Real-time,Messaging, and Fault Tolerancespecs

� Tech. transfer via DARPAQuorum program andwww.theaceorb.com

– Integration with Flick IDLcompiler, QuO, TMO, etc.



Concluding Remarks

� Researchers and developers of distributed, real-time telecomapplications confront many common challenges

– e.g., service initialization and distribution, error handling, flow control,scheduling, event demultiplexing, concurrency control, persistence, faulttolerance

� Successful researchers and developers apply patterns,frameworks, and components to resolve these challenges

� Careful application of patterns can yield efficient, predictable,scalable, and flexible middleware

– i.e., middleware performance is largely an “implementation detail”

� Next-generation ORBs for telecom will be highly QoS-enabled,though many research challenges remain



Web URLs for Additional Information

� Real-time CORBA 1.0 spec:www.cs.wustl.edu/ �schmidt/RT-ORB-std-new.pdf.gz

www.cs.wustl.edu/ �schmidt/oorc.ps.gz

� More information on TAO:www.cs.wustl.edu/ �schmidt/TAO.html

� TAO real-time event channel:www.cs.wustl.edu/ �schmidt/JSAC-98.ps.gz

� TAO static scheduling:www.cs.wustl.edu/ �schmidt/RT-ORB.ps.gz

� TAO dynamic scheduling:www.cs.wustl.edu/ �schmidt/dynamic.ps.gz


Real-time Middleware for Distributed and Embedded Systemsschmidt/PDF/lucent.pdf · 1999-11-09 · Real-time Middleware for Distributed and Embedded Systems Douglas C. Schmidt Associate

Documents