Adaptive & Reflective Middleware for Large-scale Distributed Real-time & Embedded Systems

Adaptive & Reflective Adaptive & Reflective Middleware Middleware

for Large-scale Distributed for Large-scale Distributed Real-time & Embedded Real-time & Embedded

SystemsSystemsDr. Douglas C. Schmidt

dschmidt@uci.eduwww.ece.uci.edu/~schmidt/

2

Accomplishments & New Directions

•Productivity & quality gains from higher-level abstraction mechanisms

•Partially automated solutions for limited domains

Analysis

Generation

Characteristics of DRE Systems• Mission-critical control• Resource constrained • Stringent simultaneous QoS demands

• Network-centric & large-scale• Dynamic operating conditions

Distributed real-time & embedded (DRE) systems are increasingly

pervasive & essential

3

Current Challenges & Limitations

Distributed Active Control:Vibration Damping on Delta-4Rocket Payload Fairing (1,000 nodes)

Gossamer Space Antenna (1,000,000 nodes)

Noiseless sonar onsubmarines to providecamouflage (3,000 nodes)

Total Ship Computing Environments:Global dynamic resource management (1,000 nodes)

Distributed Network of embedded sensor motes for environment monitoring,tracking, & surveillance (10,000 nodes)

Emerging Trends• Large-scale DRE system requirements are increasingly more • Dynamic• Diverse• Demanding

1000 – 1,000,000 node fusion of physical

& informationsystems

4

Applications

Endsystem Networks

Operating System

Applications

Endsystem

Operating System

Applications

Endsystem

Operating System

Networks

Technology ProblemsToday’s DRE systems are:•Stovepiped•Proprietary •Brittle & non-adaptive•Expensive•Vulnerable

Sensors ActuatorsControllers

Util

ity

Resources

Utility “Curve”

“Broken” “Works”

Cultural Problems•Existing research approaches do not scale to address key technology problems

•Non-appreciation of the importance of DRE software

Emerging Trends• Large-scale DRE system requirements are increasingly more • Dynamic• Diverse• Demanding

Current Challenges & Limitations

5

CPUs & networks have increased by 3-7 orders of magnitude in the past decade2,400 bits/sec to

1 Gigabits/sec

10 Megahertz to 1 Gigahertz

These advances stem largely from standardizing hardware & software APIs and protocols, e.g.:

Lessons from IT History

• TCP/IP, ATM• POSIX & JVMs• Middleware & components

• Intel x86 & Power PC chipsets

• Quality of service aspects

Extrapolating this trend to 2010 yields

• ~100 Gigahertz desktops• ~100 Gigabits/sec LANs• ~100 Megabits/sec wireless• ~10 Terabits/sec Internet backbone

DRE software has not improved as rapidly or as effectively as hardware & networks

6

Applications

Endsystem Networks

Operating System

SensorsApplications

Endsystem

Operating System

Applications

Endsystem

Operating System

Actuators

Networks

Controllers

The Road Ahead for DRE Systems

We need to create the new generation of open DRE system middleware technologies to 1. Simultaneously control

multiple QoS properties & 2. Improve software

development quality, productivity, & assurability

Common Services

Distribution Middleware

InfrastructureMiddleware

Domain-Specific Services

Applications

Endsystem Networks

Operating System

SensorsApplications

Endsystem

Operating System

Applications

Endsystem

Operating System

Actuators

Networks

Controllers

Common Services

Distribution Middleware

InfrastructureMiddleware

Domain-Specific Services

Common Services

Distribution Middleware

InfrastructureMiddleware

Domain-Specific Services

Win2K Linux LynxOS

Solaris VxWorks

Middleware

MiddlewareServices

MiddlewareApplications

MIDDLEWARE ARCHRTP

DNS

HTTP

UDP TCP

IP

TELNET

Ethernet ATM FDDI

Fibre Channel

FTPINTERNETWORKING ARCH

TFTP

21st Century20th CenturyBenefits of Middleware• Highly scalable QoS• Enable new resource management capabilities

• Support common & open technology bases

• Leverage & enhance advances in assurance & security

ResourcesU

tility

Desired Utility Curve “Working

Range”

7

Technical Challenge: Real-time Control of Distributed Resources

Ship-wide QoS Doctrine & Readiness Display

Distributed resource management

• Allocation/reservations, caching, scheduling, monitoring, & load balancing

Distributed security Distributed fault tolerance

Network latency & bandwidth

Workload & Replicas

CPU & memory

Connections & priority bands

Network latency & bandwidth

Workload & Replicas

CPU & memory

Connections & priority bands

Control Vars.}

Localmiddleware

QoSQoS

TBMD ApplicationAAW Application

ControlAlgorithmControlAlgorithm

ControlAlgorithmControlAlgorithm

ControlAlgorithmControl

Algorithm

Requested QoS

Measured QoS

GlobalMiddleware

Network latency & bandwidth

Workload & Replicas

CPU & memory

Connections & priority bands

Goal: Create new generation of middleware to simultaneously control multiple QoS properties

Hardware

Middleware

OS & Protocols

Applications

8

Solution Approach• Doctrine-driven middleware to support multi-level management of DRE resources• i.e., dynamic, dependable, & scalable resource analysis, scheduling, allocation, monitoring & balancing

• Guided by online reflective QoS models & empirical evaluation

Research Challenge • Devise middleware to formally specify QoS-constrained global resource management plans; model, reason about and refine them; & monitor/enforce these plans automatically at run-time

Problem• Multiple technology bases make it hard to analyze & control the QoS of distributed resources at multiple system levels dynamically, dependably, & securely

IntServ + Diffserv

RTOS + RT Java

RT/DP CORBA + DRTSJ

Load BalancerFT CORBA

Measured QoS

QoS

TBMD ApplicationAAW Application

ControlAlgorithmControlAlgorithm

ControlAlgorithmControlAlgorithm

ControlAlgorithmControlAlgorithm

Requested QoS

Network latency & bandwidth

Workload & Replicas

CPU & memory

Connections & priority bands

QoS

Hard Problem 1:Secure Multi-level Distributed Resource Management

9

Data Parallel CORBA bridges the gap between traditional CORBA applications & high-performance embedded parallel processing applications as follows: •Enable CORBA applications over clusters of computers •No change required in software technologies, methodologies, or tools •Enable massively parallel applications to integrate easily with distributed systems

•Allow parallel applications to benefit from distributed object methodologies, technologies, & tools

•Add parallelism & data distribution to the transparencies offered by CORBA•Enable a new class of applications e.g., financial, industrial, medical, aerospace, multimedia, and military domains

Client on parallel ORB

Parallel Object Computing Grid

Part 1

Part 2

Part 3

• Airborne HPEC• Distributed shipboard clusters

• CONUS supercomputers

• …Data reorganization strategies

Example: Data Parallel CORBA

10

Hard Problem 2:Meta-programmable DRE Middleware Frameworks

Endsystem

ApplicationsApplications

Endsystem

MiddlewareMiddleware MiddlewareMiddleware

ApplicationsApplications

Mechanism & PropertyManagers

Sys Cond Sys Cond Sys CondInterceptor Interceptor

LocalResourceManagers

Sys Cond

{}QoS Contract QoS Contract

Network latency & bandwidth

Workload & Replicas

CPU & memory

Connections & priority bands

Network latency & bandwidth

Workload & Replicas

CPU & memory

Connections & priority bands

LocalResourceManagers

Problem• Existing DRE systems are rigidly designed with fixed QoS parameters that limit their utility for new operations

Research Challenges• Assuring dynamic flexibility and QoS simultaneously

Solution Approach• Meta-programming techniques that

• Decouple functional & QoS paths to allow more degrees of freedom• Specify QoS doctrine declaratively• Support dynamic QoS adaptation & optimizations

11

Example: Meta-programmed Distributed QoS Aspects

MissionRequirements

Fidelity• Highest fidelity frames

must be delivered

Timeliness• Maintain an out-of-the-window

view of imagery

Importance• Frames must be dropped in

reverse order of importance

Distributor...PBBPBBPBBI I I IDATA FILTERING• Condition: Excessive Network or CPU load• Action: Drop selective frames

IMAGE MANIPULATION• Condition: Excessive

Network load• Action: Scale image to

smaller size

LOAD BALANCING• Condition: Excessive CPU load • Action: Migrate distributor to a

lightly loaded host

XXDistributorI P

BI

II

II

II

IP P PP P

P P PPP

B

B BBBBBB BBBBBB

BBBB

BBBBBBBBBBBBBBBBBBBB

BBBBBBBBBBBBBBBB

BB BBBBBBBBBBBBBBBBBBBBBBBBBBBBBP P P P P P P P P PIBBPBBPBBPBBPBBPBBIBBPB

NETWORK RESERVATION• Condition: Excessive Network load • Action: Use IntServ & DiffServ to reserve bandwidth

QoS Contracts

www.dist-systems.bbn.com/techwww.cs.wustl.edu/~schmidt

12

Hard Problem 3:

DRE Middleware Generation & Optimization Tools

Research Challenge • Minimizing footprint & customizing

standard DRE middleware capabilities without sacrificing key QoS properties

tSTART t2 t4 t5 t6 t7 t9 t10 t12 t13 t14 t15 t16 t17 t18t11 t19 tREQ’Dt3 t8

Impl1

App

licat

ion

Req

uire

men

ts

Impl2 Impl1

Common Semantic Rep.

Plat2

Plat2.pd

Plat3

Plat3.pd

Plat1

Plat1.pd

Middleware Generator

DRE Applications

Optim

ized & verified

DR

E middlew

are

Common Middleware Tools

Problems• COTS middleware is often

unsuited for DRE systems due to insufficient

• QoS specification & enforcement

• Time/space optimizations• Flexibility & customizability

• However, conventional DRE development techniques are

• Tedious• Proprietary• Manual & ad hoc

Solution Approach• Develop Model Driven Architecture (MDA)

tools to reflectively auto-generate, optimize, & verify custom implementations of standard DRE middleware from higher-level specifications

Network latency & bandwidth

Workload & Replicas

CPU & memory

Connections & priority bands

Network latency & bandwidth

Workload & Replicas

CPU & memory

Connections & priority bands

13

•Early compilers required •Separate internal representations hand-written for each programming language &

•Separate hand-written optimizers for each target backend

•Developing, verifying, validating, & evolving all these components separately is costly, time-consuming, tedious, & error-prone

•The problem only gets worse as more languages & target backends emerge

C Compiler

Internal Rep.

Ix86Opt.

Ix86

VAXOpt.

VAX

68KOpt.

68K

C Program

To illustrate the benefits of reflection as an optimization technique, consider the evolution of compiler technology:

Example:Applying Reflection as Compiler Optimization

14

•Early compilers required •Separate internal representations hand-written for each programming language and

•Separate hand-written optimizers for each target backend

•Developing, verifying, validating, & evolving all these components separately is costly, time-consuming, tedious, & error-prone

•The problem only gets worse as more languages & target backends emerge

C Compiler

Internal Rep.

Ix86Opt.

Ix86

VAXOpt.

VAX

68KOpt.

68K

C Program

To illustrate the benefits of reflection as an optimization technique, consider the evolution of compiler technology:

C++ Compiler

Internal Rep.

PPCOpt.

MIPSOpt.

88KOpt.

PPC MIPS 88K

C++ Program

Ada Compiler

Internal Rep.

1751Opt.

32KOpt.

HPPAOpt.

1751 32K HPPA

Ada Program

Example:Applying Reflection as Compiler Optimization

15

C/C++/Ada Compiler

Common Internal Rep.

Ix86Opt.

PPCOpt.

68KOpt.

C/C++/Ada Programs

Ix86Ix86.md

PPCPPC.md

68K68K.md

•Modern compilers, such as GNU GCC, support •A common internal representation (still hand-written) for each programming language •Based on generalizing the language semantics

1. Read the target machine description

Optimizer Generator

2. Use discrimination network to analyze the optimization rules & opportunities

3. Generate an optimizer that is customized for the particular platform/language

•A synthesized compiler optimizer that is customized automatically for each target backend•Based on reflective assessment of algebraic target machine description

Key Benefit of Reflective Optimization•New targets can be supported by writing a new machine description, rather than writing a new code generator/optimizer

Example:Applying Reflection as Compiler Optimization

16

•Developing, verifying, validating, & evolving all these components separately is costly, time-consuming, tedious, & error-prone

•Moreover, it is even harder to hand-configure support for dynamic platform variations & complex application use-cases

•The problem only gets worse as more middleware, target platforms, & complex applications emerge

•Separate hand-written & hand-optimized implementations for each embedded target platform•e.g., various OS/network/HW configurations

•Conventional middleware require•Separate tools and interfaces hand-written for each ORB middleware specification •e.g., CORBA, Java RMI, COM+

CORBA ORB & Assorted Tools

CORBA Application

Conventional middleware for embedded systems is developed & optimized in a manner similar to early compiler technologies:

WinNT

WinNTImpl

LynxOS

LynxOSImpl

VxWorks

VxWorksImpl

New Approach:Applying Reflection to Optimize DRE Middleware

17

•Developing, verifying, validating, & evolving all these components separately is costly, time-consuming, tedious, & error-prone

•Moreover, it is even harder to hand-configure support for dynamic platform variations & complex application use-cases

•The problem only gets worse as more middleware, target platforms, & complex applications emerge

•Separate hand-written & hand-optimized implementations for each embedded target platform•e.g., various OS/network/HW configurations

•Conventional middleware require•Separate tools and interfaces hand-written for each ORB middleware specification •e.g., CORBA, Java RMI, COM+

Java RMI & Assorted Tools

Win2K SolarisLinux

Java Application

Win2KImpl

SolarisImpl

LinuxImpl

COM+ ORB & Assorted Tools

WinNT Win98WinCE

COM+ Application

WinNTImpl

Win98Impl

WinCEImpl

CORBA ORB & Assorted Tools

CORBA Application

Conventional middleware for embedded systems is developed & optimized in a manner similar to early compiler technologies:

WinNT

WinNTImpl

LynxOS

LynxOSImpl

VxWorks

VxWorksImpl

New Approach:Applying Reflection to Optimize DRE Middleware

18

Common ORB + Assorted Tools

Common Semantic Representation

Plat1

Impl

•The functional and QoS-related properties of embedded systems middleware can be improved greatly by advanced R&D on the following topics:•A common internal representation (ideally auto-generated) for each middleware specification •Based on generalizing the middleware semantics

Middleware Generator

2. Use discrimination network to analyze the optimization rules & opportunities

3. Generate middleware that is customized for a particular platform & application use-case

•A synthesized middleware implementation that is optimized automatically for each target platform & application use-case•Based on reflective assessment of platform descriptions & application use-case

Plat2

Plat2

.pd

Plat2

ImplPlat3

Impl

1. Read the target platform description & application requirements

App

licat

ion

Req

uire

men

ts

CORBA/Java/COM+ Applications

Plat3

Plat3

.pd

Plat1

Plat1

.pd

New Approach:Applying Reflection to Optimize DRE Middleware

19

DARPA DRE Software R&D FocusComparable & Customizable Frameworks & Intelligent Middleware

Patterns & Pattern Languages

Model-based Software Development & Domain-specific Languages

GPS IFF FLIR

Object Request Broker

AirFrame

AP Nav WTS

Event Channel

ReplicationService

SynchronizationPersistence

Fault ToleranceMemory Management

Cross-cutting Concerns

Multi-faceted SoftwareDevelopment

Collaborative Software Development Environments

Applications to Embedded Systems