Applying a Flexible Middleware Scheduling Framework to Optimize Distributed RT/Embedded Systems

Applying a Flexible Middleware Scheduling Framework to Optimize Distributed RT/Embedded Systems

Christopher D. Gill

Center for Distributed Object Computing

Department of Computer Science

Washington University, St. Louis, MO

[email protected]

Friday, March 23, 2001

Primary ContributionsI. Flexible Middleware Scheduling Framework

• Gill, Levine, Schmidt, “The Design and Performance of a Real-Time CORBA Scheduling Service” Real-Time Systems, 20:2, March 2001

• Early use: ASFD, ASTD I demonstrations (AFRL, Boeing) August, December 1999• TAO, WSOA, SEC, ASTD II Bold Stroke infrastructure (AFRL, DARPA, Boeing) - now

Release as a Separate Open-Source Framework Kokyu – summer 2001

Christopher D. Gill

II. Customized Scheduling Strategies and Optimizations for Multi-Layer Integrated Middleware Resource Management

• Doerr, Venturella, Jha, Gill, and Schmidt, "Adaptive Scheduling for Real-time, Embedded Information Systems”, DASC, October 1999

• Loyall, Gossett, Gill, et al., “Comparing & Contrasting Adaptive Middleware Support in Wide-Area and Embedded Distributed Object Applications”, ICDCS, April 2001

• Gill, Cytron, Schmidt, “Middleware Scheduling Optimization Techniques for Distributed Real-Time and Embedded Systems”, submitted to the Workshop on Optimization of Middleware and Distributed Systems (ACM SIGPLAN), June 2001

• WSOA, SEC, ASTD II Bold Stroke infrastructure (Boeing) – nowTAO, Kokyu – summer 2001

III. Real-Time Adaptive Metrics and Visualization Infrastructure• Gill, Levine, O'Ryan, and Schmidt, "Distributed Object Visualization for Sensor-

Driven Systems”, DASC, October 1999• Gill and Levine, "Quality of Service Management for Real-Time Embedded

Information Systems”, DASC, October 2000• WSOA, SEC, ASTD II Bold Stroke infrastructure (Boeing) – now

TAO, Kokyu – summer 2001

Middleware Scheduling for Mission-Critical Distributed Real-Time & Embedded (DRE) Systems

Historically, mission-critical DRE apps were built directly atop the hardware & OS•Tedious, error-prone, & costly over lifecycles

• Avionics Mission Computing (Boeing)• Medical Information Systems (Siemens Med)• Satellite Control (LMCO COMSAT)• Telecommunications (Motorola, Lucent)• Missile Systems (LMCO Sanders)• Steel Manufacturing (Siemens ATD)• Beverage Bottling Lines (Krones AG)

COTS middleware (e.g. ACE+TAO) is being leveraged to lower cost and cycle times in real-world mission-critical DRE applications

However, previous-generation COTS middleware limits design choices, e.g.:•Customized scheduling policies•“Back-end” dispatching optimizations•Heterogeneous end-to-end dispatching•Optimized integration w/ other key technologies, e.g., RT-ARM, QuO, TNA

Historically, COTS couples functional with QoS aspects• e.g., due to lack of “hooks”

Historically, COTS couples functional with QoS aspects• e.g., due to lack of “hooks”

Research Context: Avionics Mission ComputingBold Stroke Middleware Infrastructure Platform

• ASFD, ASTD I, WSOA, ASTD II• Target: transition to production systems

Event Mediated• RT Enhanced TAO Event Channel• Distributed precedence DAG, scheduler

per endsystem sub-graph

Operations Well Defined• Harmonic rate sets, bounded execution

times, some critical, some non-critical• Need criticality isolation assurances

Next Generation Systems• Highly variable environment• Very large number of system states –

support for fine-grain dynamic modes• Need dynamic and adaptive approaches

to resource management• Need coordinated closed-loop control of

QoS across time-scales, system layers– ACE+TAO, QuO, RT-ARM

Previous Generation Systems• Fixed environment, static modes• Used cyclic or RMS scheduling

Kokyu: a Flexible Middleware Scheduling and Dispatching Framework

•The application specifies characteristics• E.g., criticality, periods, dependencies

•The application specifies characteristics• E.g., criticality, periods, dependencies

•Dispatcher is (re)configurable•Multiple priority lanes•Queue, thread, timers per lane•Starts repetitive timers once•Looks up lane on each arrival

•Dispatcher is (re)configurable•Multiple priority lanes•Queue, thread, timers per lane•Starts repetitive timers once•Looks up lane on each arrival

Scheduler

sub-graph

ratetuples

WCET propagation

selectedrates

rate propagation

propagatedrates

tuplevisitor

operationvisitors Rate and priority

assignment policy

DispatcherDispatching configuration

RMS

LLF

mandatory

optionallaxity

static

static

timers

•The scheduler assigns/stores periods & priorities per topology, scheduling policy• Defines necessary dispatch configuration

•The scheduler assigns/stores periods & priorities per topology, scheduling policy• Defines necessary dispatch configuration

• Implicit projection of particular scheduling policy into a generic dispatching infrastructure

•Implicit projection of particular scheduling policy into a generic dispatching infrastructure

Empirical Costs of Dispatching Primitives

Three Canonical Queue Ordering Disciplines

• Simple test with queue classes• Static fixed sub-priority • Deadline time to deadline• Laxity time to deadline – WCET• Messages in dynamic queues age

monotonically – cancellation only needed at the head of the queue

Basic Empirical Comparison• Randomly ordered enqueues• Average enqueue and dequeue

times for deadline were slightly better than for laxity, and much worse than for static

• Simple result, but useful to guide design of and experiments with richer composite strategies

Domain-Tailored Scheduling Heuristics

Dispatcher

Desired Heuristic Depends on Application-Specific Details:• Scheduling ARM ops: timely adaptive transitions w/out impacting mandatory ops• RMS {mandatory, ARM} + LLF {optional} if feasible (lower mandatory overhead)• LLF {mandatory, ARM} + LLF {optional} (MUF) if feasible (non-harmonic rates)• RMS {mandatory} + LLF {ARM} + LLF {optional}

if {mandatory, ARM} infeasible & cannot separate ARM operations• RMS {mandatory, ARMm} + LLF {ARMo} + LLF {optional} • LLF {mandatory, ARMm} + LLF {ARMo} + LLF {optional}• RMS {mandatory, ARMm} + LLF {ARMo, optional} • LLF {mandatory, ARMm} + LLF {ARMo, optional}• A cancellation discipline for futile operations may help optimize lower partitions

Dispatching configuration

laxity

static

static

laxity

mandatory optionalRT-ARM

Abstract Mapping: Operation Characteristics OS & Middleware Primitives

• Generalizes well-known monolithic and composite heuristics, e.g., RMS, EDF, LLF, MUF, RMS+LLF

• Supports arbitrary composition• Also allows strategized factory-

driven dispatching module (re)configuration at run-time (work-in-progress)

• Empirical & a priori info guides design of domain-specific heuristic

Domain-Tailored Scheduling Example

DispatcherPreserve Invariant, but Optimize

• Given: RT-ARM ops separable into RT-ARMm and RT-ARMo

• Given: criticality values express deadline isolation partitions

• Definition: system schedulable if highest partition feasible

• Invariant: no lower partition makes feasible higher one infeasible

• Precise invariant strength is key:– e.g., 1:1 criticality-priority

over-constrains problem– e.g., 1:1 rate-priority (RMS)

both over-and-under-constrains problem

– Want invariant-preserving optimizations, e.g., RMS{mandatory,ARMm} + LLF{ARMo, optional}

• Decision lattice among distinct mappings from criticality partitions into priority partitions

– Example based on one criticality partition: {mandatory},{ARMm}, {ARMo},{optional}

LLF {mandatory, ARM, optional}

RMS {mandatory, ARM, optional}

RMS {mandatory, ARM}LLF {optional}

RMS {mandatory, ARMm}LLF {ARMo, optional}

feasibility

performancefe

asib

ilit

y

per

form

ance

feas

ibil

ity

per

form

ance

RMS {mandatory}LLF {ARM, optional}

feas

ibil

ity

per

form

ance

LLF {mandatory, ARM}LLF {optional}

LLF {mandatory, ARMm}LLF {ARMo, optional}

feas

ibil

ity

per

form

ance

feas

ibil

ity

per

form

ance

LLF {mandatory}LLF {ARM, optional}

feas

ibil

ity

per

form

ance

feasibility

performance

feasibility

performance

feasibility

performance

Experiments to Quantify Hybrid/Adaptive Scheduling

ASFD• Quantify RMS, MUF, RMS+LLF with realistic OFP

application using The ACE ORB (TAO) on realistic hardware (AV-8B simulator demonstration 1999): AFRL sponsored, directed by Boeing

ASTD I• RT-ARM/Scheduler integration and adaptive scheduling

scalability test (NT desktop demonstration in 1999): AFRL sponsored, directed by Boeing

WSOA• Quantify coordinated multi-layer resource management

(ground and flight demonstrations in 2001): AFRL, OS/JTF, DARPA sponsored, directed by Boeing

ASTD II• Integrate TNA and OFP scheduling and dispatching:

AFRL sponsored, directed by Boeing

Boeing Fellowship Benefits to WSOA• Quantify behavior across a range of scheduling

heuristics: work started under 1999-2000 Boeing Fellowship Grant, leveraged on WSOA, new OEP experiments underway

Real-Time Metrics Monitoring Framework

Data Collection• Probes populate singleton

C++ metrics data cache • Monitor periodically digests

raw data, steams to logger

Logging and Visualization• Logger streams data to

storage, viewers• Remote processing, e.g., on

an NT Workstation

Resource Management• RTARM, QuO (syscond) may

query the monitorEM

BE

DD

ED

BO

AR

DS

RE

MO

TE

WO

RK

ST

AT

ION

SH

AR

ED

ME

MO

RY

METRICS CACHE

REMOTE LOGGER

METRICS MONITOR

RTARM

DOVEBrowser (Java)

STO

RA

GE

QuO

OPERATIONS

PRO

BE

S

DISPATCHER

PROBES

Frame Manager• Singleton manages

deadlines per rate

FRAME MANAGER

ASFD: Measured Improvement over Static Approach

One Problem the Flexible Scheduling Framework Addresses• Transition from an already overloaded state (but with mandatory below the

utilization bound), to an even more overloaded state in the ASFD application• As theory predicts, RMS showed significant degradation of mandatory operation

deadline behavior under CPU overload conditions

ASFD: Hybrid Scheduling Isolated Deadline Misses

MUF (Shown) and RMS+LLF: No Mandatory Operation Misses • Successful use of dynamic queue disciplines: predictable mandatory operation

behavior in MUF and RMS+LLF• Same overload conditions in same state• MUF and RMS+LLF mandatory & optional in separate lanes

Partitioning helps isolate effects of offered overload

ASFD: Empirical Result for Domain-Specific Design

MUF Optional OperationsRMS+LLF Optional Operations

Expectations from Theory and Basic Overhead Tests Confirmed• Slight increase in made deadlines under same overload conditions w/out

cancellation, moving from MUF to RMS+LLF• RMS+LLF: static mandatory queues, 5 threads, lower overhead & messages/queue• MUF: laxity mandatory queue, 2 threads, lower overhead & messages/queue

ASFD: Early Results for Operation Cancellation

MUF Optional Options with cancellation

MUF Optional Operations without cancellation

Cancellation Effects in MUF (RMS+LLF similar)

• Optional tasks may miss deadlines

• Cancellation reduced futile dispatches

• However, pessimistic cancellation based on WCET also reduced deadlines made (i.e., due to false cancellations)

• Different and/or more exact execution time information (e.g., distribution curve with WCET, ACET, BCET) needed

ASTD I: Number of Calls in an Adaptive Transition

Interaction Between RT-ARM and Scheduler during Transitions • RT-ARM iteratively proposed operation to rate bindings, according to a (HTC

proprietary) heuristic for adaptive rate selection• Scheduler assessed feasibility of each proposed binding• Scheduler also gave feedback on the feasibility “sensitivity” of the proposed

binding WRT the feasible bound, to changes to rates in the bound set• Number of scheduler calls was linear in number of destination state operations

ASTD I: Average Call Time in an Adaptive Transition

Combined RT-ARM and Scheduler Behavior• Average time in each scheduler call also linear in # of destination state operations• Combined call count and time curves indicate scheduler sensitivity and feasibility

were quadratic in # of destination state operations• O(n2) adaptation time may be fine for small numbers of operations per state, but for

larger scale applications (e.g., ASTD II, SEC, WSOA) a closer bound is needed

WSOA: Improving Adaptive Scheduling Performance

Integrated Mechanisms for Adaptive Scheduling

• In ASTD I, isolation of mechanisms (RT-ARM vs. scheduler) made it O(n2) to select feasible rate assignment

• Merging mechanisms in scheduler: rate and lane assignment as O(n lg n) or O(n) sorts, O(n) feasibility & max-rate

• Modular visitor functors over dependency graph

• i.e., cycle check, rate choice, WCET & rate propagation)

• Rate selection strategized, like lane assignment: arbitrary mapping from operation characteristics to selection order (may use 2 feasibility thresholds)

Similarly General & Flexible Approach to Rate Selection Strategies • e.g., Fair Admission by Indexed Rate (FAIR)

• Rate index, criticality, mean rate, handle• Each op gets its lowest rate, then each gets its next highest, etc.

• e.g., Criticality-Biased FAIR (CB-Fair)• Criticality, rate index, mean rate, operation handle• Mandatory ops get all rates first, then optional ops

Scheduler

sub-graph

ratetuples

WCET propagationselected

rates

rate propagation

propagatedrates

tuplevisitor

operationvisitors Rate and priority

assignment policy

WSOA: Consistent Time and Frame Management

Singleton Frame ManagerFRAME MANAGER

EMBEDDED BOARD

DISPATCHER

START END IDRATE

0 50 020HZ

0 100 010HZ

0 200 05HZ

50 100 120HZ

0 100 010HZ

0 200 05HZ

100 150 220HZ

100 200 110HZ

0 200 05HZ

150 200 320HZ

100 200 110HZ

0 200 05HZ

200 250 420HZ

200 300 210HZ

200 400 15HZ

250 300 520HZ

200 300 210HZ

200 400 15HZ

300 350 620HZ

300 400 310HZ

200 400 15HZ

350 400 720HZ

300 400 310HZ

200 400 15HZ

400 450 820HZ

400 500 410HZ

400 600 25HZ

ADAPTER RTARM QuO

• Common Reference Point for Deadlines

• Rates are Registered with Frame Manager

• Frame Start, End, and Id for Each Rate

• Timer-Based Call to Update Frames Advances Each Frame Appropriately

• Dispatcher Obtains Deadline Using Period from Operation RT_Info to Manage Laxity, Deadline Queues

• Upcall Adapter Uses Cancellation Deadline, Again Using RT_Info Rate

• RT-ARM and QuO Can Also Obtain Consistent Deadlines from Frame Manager

20HZ45

0

20H

Z45

0 50010H

Z

6005HZ

WSOA: Meeting Embedded & RT Metrics Constraints

Shared Memory Capable• Time & space constraints: shared

memory (e.g., VME) is attractive• However, virtual functions, native

pointers and heap allocation are perilous in shared memory

• Apply offset-based smart pointers, strategized allocators to support run-time cache sizing and use from any address space

• System instrumented with C++ inline probe macros: write to a singleton metrics cache

Process Owned Memory• Upcall Adapter• Frame Manager• DOVE, Metrics Monitor, Logger• All non-Tie CORBA Interfaces

SH

AR

ED

ME

MO

RY

METRICS MONITOR

METRICS CACHE

DIS

PA

TC

HE

R

20H

z D

eque

ue

20Hz E

nqueue

STARTSUSPENDRESUME

STOP

Process Tile

STARTSTOP

EMBEDDED BOARDS

UPCALL ADAPTER

OPERATION

STA

RT

B

SUSP

EN

D

S

RE

SUM

E

R

STO

P

ESTART

B

SUSPEND

S

RESUME

R

STOP

E

B

STO

P

STA

RTE

20Hz EnQ

20Hz DQ

Proc Tile

ASTD II: Domain Integration at the Dispatching Level

TN Task

TN Event Consumer

execute()

push(...)

push(...)Dispatcher(configured for RMS+LLF)

Task Network Scheduler Active

Task Agenda

TAO Scheduler

Prioritized OS Threads

lookup

enqueue

process(...)

Coordinated TNA+OFP Operation Dispatching

• TNS specifies operation characteristics to the scheduler

• TNS manages tasks and execution predicates, processes active agenda by strategized transfer to dispatcher

• Dispatcher manages priority lanes according to specified configuration

• Special dispatch target (in this case implemented as an Event Consumer) invokes execute function of special event as a functor (a.k.a., Command Design Pattern) to call the TN task

Concluding Remarks

Empirical Results• Show adaptive & hybrid scheduling approach can

improve DRE real-time performance

Composable Dispatching

• Enables domain-specific optimizations, especially when design decisions are aided by empirical data

QoS Metrics Framework• Offers diverse modes of interaction with a DRE system:

resource managers, human operators, control systems

OEP Experiments• Will offer a quantitative profile of the benefits of this

approach to flexible real-time scheduling in middleware

Open-Source Code• All software described here that is part of my research

will be made available in the ACE_wrappers distribution– First within TAO, then as a distinct Kokyu directory

(summer 2001)

Future Work

I. Run-Time Re-configurable Dispatching Module + Factory• Strategized adaptation at run-time to changing queue, thread, and timer

configurations: pre-allocation, reallocation, caching of primitives• Paper in progress for submission to Middleware 2001• SEC and ASTD II Bold Stroke infrastructure (Boeing) – in progress

TAO, Kokyu – summer 2001

II. Extremely Small Footprint DRE Middleware• Applying scheduling/dispatching primitives and framework to nodes and

nodelets with different resource niche scales (downward scalability)• “Just enough” middleware: e.g., from Jini-like backbone to an ORB• Proposal for DARPA ITO NEST Program: Open Experimental Platform• 2001-2005 time frame

laxity

static

static

timers

Future Work

III. Aspect-Oriented Techniques for Middleware QoS Management• Using aspect weaving to configure possibly heterogeneous middleware

scheduling and dispatching points, at multiple layers and on multiple paths• Multi-dimensional resource management, i.e., memory, network, and CPU

together, to apply good heuristics and domain-specific optimizations• Domain-specific type systems, generic programming techniques may apply

IV. Coordinated Multi-Layer Multi-Agent QoS Management• Integrated cooperation of resource managers, schedulers, dispatchers in

OS, middleware, and application layers• Infrastructure hybridization and re-factoring to leverage mechanism

composition and integration, while preserving policy separation• Toward generalized techniques, patterns and a theory of QoS composition

in middleware, e.g.., hard real-time + anytime + adaptive control + DAS + …

ApplicationOperations

DynamicScheduling

Dispatch

Sensitivity Measure

Mission States

ARM

Admit Monitor

AdaptationExecutableOperationsHRT

SRT

t

t

t

OperationDispatchesProgress

andSensitivityFeedback