CESAR – APPROACH & MEASUREMENT OF OBJECTIVES

CESAR - Cost-efficient methods and processes for safety relevant embedded systems

CESAR – APPROACH &

MEASUREMENT OF OBJECTIVES

Bourrouilh Quentin, Kundner Ingrid / AVL

Bourrouilh, Kundner / AVL

10-06-302

MOTIVATION – AUTOMOTIVE

1st Vehicles a decade ago� A few embedded systems per

vehicle

Vehicles nowadays� Up to a few hundreds of

computing devices per vehicle

� Multiple networks per vehicle

Advantage: � Safety-critical embedded

systems have been key innovation drivers

� E.g. by-wire systems

Disadvantage: � Enormous complexity is

challenging industry (automotive, aerospace, rail, automation)

� Increasing costs

� Affected product quality �safety-critical

PAST TODAY

FUTURE ?


10-06-303

MOTIVATION - AEROSPACE

Engine equipment and parts

• Integrated engine control systems

• Power transmissions

• Engine modules and components

• Composite engine parts

Engines

• CFM56 family (50/50 with GE)

• SAM146 engine for the Russian

Regional Jet (50/50 with NPO Saturn)

• Participation in programs: CF6, GE90,

GE90-115B, GP7000, PW4000, AS900, CF34

Landing & braking systems

• Landing gear for all types of aircraft

• Braking/landing control systems

• Wheels and carbon brakes

• Control systems and hydraulics

• Maintenance, repair and overhaul

Aircraft equipment

• Network server systems

• Back-up flight control

• Secure data link

• Cockpit control systems

• Electrical wiring systems

• Aircraft condition monitoring systems

Engine services

• Maintenance, repair and overhaul

• Engine testing and test equipment

• Composite aerostructures

• Auxiliary power units

• Hydraulic systems

• Sensors and actuators

• Ventilation/filtration

• Inertial references

Nacelles

Nacelles and components

(thrust reversers,…)


10-06-304

CESAR OBJECTIVES

Improvement of processes and methods for safety-critical embedded systems development

Development of the reference technology platform (RTP) for safety-critical embedded systems development

Motivate technology providers and SMEs to contribute to the RTP

Bring innovations in two most improvable engineering disciplines� Requirements engineering

� Component-based engineering


10-06-305

CESAR OBJECTIVES

Reduce costs for the development of safety critical systems while ensuring the quality and safety properties

Covering the entire System Engineering (SE) disciplines:

� Improve Requirements engineering (RE)

� Improve Component-based development (CBD) and extend it with multi views and multi criteria

� Combine the improved RE (e.g. RMM) and Design System Engineering (incl. CBD)

� A close collaboration between RE and CBD is necessary to satisfy the ambitious CESAR goals

� Only an integration of these disciplines accompanied with an adequate tool support into a seamless tool chain (CESAR RTP) can unleash the full potential of the CESAR approach


10-06-306

CESAR APPROACH

� A main objective is to reduce costs

� CESAR will answer this by improving the SE (System Engineering) discipline

� SE considers the necessary combination of RE, CBD, Safety and PLE

Innovations in tools and methods

RE (Requirements Engineering)

SCADE

Simulink

Tool A

Rhapsody

Tool B

RT-Builder

Tool X

Enovia

Reqtify

Tool Y

Analysis X

Analysis Y

Innovations in tools and methods

CBD (Component Based Development)

Safety

PLE


10-06-307

CESAR PROJECT STRATEGY

� Industrial driven

� Multi-domain approach

� 3 Innovation Cycles

� Industrial needs (requirements)

� Solutions (don’t reinvent the

wheel) provided by technical

Subprojects + common bottom-

up and top-down approach

� Results (Technical Items) back

to Domain SPs (integrated or

not integrated in the RTP)

� Evaluation and feedback

through the end users Pilot

Applications


10-06-308

CESAR FACTS & FIGURES

� Consortium

� 55 Partners

� Further Assisting Parties

� Project Performing

� Duration: 3 years

� Start: 01.03.2009

� Manpower

� Effort: 5124 MM ~ 142 MY/Y

� Project Figures

� Total Budget: 58.535.000 €

� Total Funding: 28.317.000 €


10-06-309

CESAR PROJECT STRUCTURE

AVL

OFFIS

SIEMENS

A-F

VO

LV

O

MB

AB

BEADS-IW

SAGEM

PSB

TPC

AVL

SAGEM

Technical SPs are lead by an academic and an indusrial partner


10-06-3010

SP1: REFERENCE TECHNOLOGY PLATFORM

� Realization of RTP for safety-critical embedded systems development

Task Force Safety and Diagnosability� Appropriate and efficient

approaches to developmentand validation

Task Force Product Lines� Insurance of a consistent

product line engineering process throughout development cycle


10-06-3011

SP2 & SP3

SP2: Requirements engineering� Improve RE methods and

processes

SP3: Component-based development� Facilitation of system

exploration

� Provide means for incremental verification, validation and certification


10-06-3012

SP2: REQUIREMENTS ENGINEERING

WHY: � Proper requirements engineering essential in safety projects

� Provide evidence for achievement of functional safety

� Provide extensive documentation for certification

HOW:� Improve current process dealing with requirements engineering and management

� Develop a formalized multi criteria requirement specification language called RSLand a corresponding RMM:

� High level of abstraction: requirements in textual form

� Low level of abstraction: specified in formal notation

� Tool-independent exchange format

� Ensure complete traceability, completeness and consistency from concepts to products

� Support validation of formal multi-criteria requirements with respect to consistency and completeness

� Implementation of improved requirements engineering methods and tools as well providing corresponding guidelines


10-06-3013

SP3: COMPONENT BASED DEVELOPMENT

WHY:� Safety-critical embedded systems development is costly and

reuse is difficult

HOW:� Focus of SP3 is on system architecture and system detailed

design which relies on component based design

� Support integration of multiple views and multi-criteria in design space exploration

� E.g. functionality, hardware, safety, performance

� Main activities

� Modeling languages and validation techniques

� Methodologies for specifying and validating system architectures as well detailed system design made from components

� Tool chains specification and tool implementation


10-06-3014

SP5, SP6, SP7: DOMAIN SPs

Provision of industrial needs� Pilot Applications

� State of practice

� Safety Design Process

� Requirements Description

Support of existing and evolving domain standards:� ISO 26262 Functional Safety

� IEC61508

� DO178

� ENxxx

Inputs for technical subprojects and support for evaluation and validation

PA examples:� Power train control unit for hybrid

vehicle

� Engine control system for aircraft


10-06-3015

CESAR MULTI DOMAIN APPROACH

4 DOMAINS - INDUSTRIAL PARTNER

� Automotive (SP5)

� Aerospace (SP6)

� Automation & Rail (SP7)


10-06-3016

ACHIEVEMENTS YEAR 1

2nd JU Review on the 17th and 18th of May

Very successful and green light to continue

Examples of Top-Level achievements so far:� Cartography of Pilot Applications (PA) and Technical Innovations

(TI)

� SP1: RTP V1.0 is available and first evaluation starts

� SP2: « RE process » techniques (RSL, RMM) and methods have been identified

� SP3: Selection and identification of methodologies and data model to support the multiviews modelling


10-06-3017

CESAR MEASUREMENT OF OBJECTIVES


10-06-3018

CESAR OBJECTIVES

Scope

Short name

Area Quantitative Objective

Aero

sp

ace

Au

tom

otiv

e

Rail

Au

tom

atio

n

Innovation cutting cost by 30-50%

Process

Introduce in each domain at least one significant innovation in design, integration or validation process, clearly supported by CESAR, acceptable across the supply chain and to certification authorities when appropriate,

resulting in overall reduction of development time or effort, between 30% and 50%, depending on the domain

x x x x

Reduce effort of revalidation by 50%

ProcessDemonstrate, at least in one domain, a reduction by 50% of the effort of re-validation and re-certification after change, the process being acceptable

across the supply chain if appropriate, and to certification authorities

x †[1] x x

100%

complexity increase, 20% effort reduction

ProductProcess

Demonstrate, at least in one domain a 100% complexity increase of the product with 20% engineering effort reduction

x

Product capability improvement without increased cost

ProductIntroduce in each domain at least one major product capability improvement

clearly related with CESAR, without impact on recurring costx x x x

[1] Not applicable for this domain (no certification authority)


10-06-3019

FROM OBJECTIVES TO TECHNICAL ITEMS

CESAR has to track the overall objectives -systematic breakdown from objectives to solutions


10-06-3020

3 STEP PROCESS

� 1. Creation of Baseline

� Metrics definition

� Creation of a harmonized development process, valid for all CESAR domains (5 Phases) with domain-specific activities located in each phase

� 2. Mapping of Pilot Applications to Baseline

� CESAR Cartography

� 3. Measure Impact of Technical Innovations

� Evaluation on 4 different perspectives including their own metrics:

� Efficiency (%)

� Quality (%)

� Complexity (%)

� Cost / Effort (%)


10-06-3021

CREATION OF BASLINE: METRICS DEFINITION (1/5)

Scope

Short name

Area Quantitative Objective

Aero

sp

ace

Au

tom

otiv

e

Rail

Au

tom

atio

n

Innovation cutting cost by 30-50%

Process

Introduce in each domain at least one significant innovation in design, integration or validation process, clearly supported by CESAR, acceptable across the supply chain and to certification authorities when appropriate,

resulting in overall reduction of development time or effort, between 30% and 50%, depending on the domain

x x x x

Reduce effort of revalidation by 50%

ProcessDemonstrate, at least in one domain, a reduction by 50% of the effort of re-validation and re-certification after change, the process being acceptable

across the supply chain if appropriate, and to certification authorities

x †[1] x x

100%

complexity increase, 20% effort reduction

ProductProcess

Demonstrate, at least in one domain a 100% complexity increase of the product with 20% engineering effort reduction

x

Product capability improvement without increased cost

ProductIntroduce in each domain at least one major product capability improvement

clearly related with CESAR, without impact on recurring costx x x x

[1] Not applicable for this domain (no certification authority)

CostComplexity

Cost

Complexity


10-06-3022

CREATION OF BASLINE: METRICS DEFINITION (2/5)

Cost

QualityComplexityEfficiency

×=

Efficiency

QualityComplexityCost

×=

In other words...


10-06-3023

CREATION OF BASELINE: METRICS DEFINITION (3/5)

� A set of metrics (measurable) and indicators (qualitative) have been issued. This list is not exhaustive and will be refined after 1st RTP evaluation tests depending on stated measurability.

WG-Objectives


10-06-3024


# Area Objective Associated goal(s)

A1 Process

Introduce in each domain at least one significant innovation in design, integration or validation process, clearly supported by CESAR, acceptable across the supply chain and to certification authorities when appropriate, resulting in overall reduction of development time or effort, between 30% and 50%, depending on the domain

Cost reduction Efficiency

A2 Process

Demonstrate, at least in one domain, a reduction by 50% of the effort of re-validation and re-certification after change, the process being acceptable across the supply chain if appropriate, and to certification authorities

Efficiency Quality

A4 Product

Introduce in each domain at least one major product capability improvement clearly related with CESAR, without impact on recurring cost

Quality

B4 RTP

Reduce by 50% the cost of integration, configuration, deployment, and maintenance of appropriate tool-chains for all major actors in the supply chain involved in the project

Cost reduction

B5 RTP

Improve the quality of support tools by targeting a level of maturity at EIS consistent with the needs of critical systems engineering: reduce by 50% the number of Problem Reports on support tools after their delivery to system engineering teams.

Quality Efficiency

Coverage versus CESAR Objectives (SP6 related)


10-06-3025


� Fact: All goals are linked together ! The goals are only different views to the entire development to derive metrics:

� Goal 1 Efficiency :

� Sub-goal 1.1: Reduce development cycle time

� Sub-goal 1.2: Find defects as early as possible

� Sub-goal 1.3: Improve requirements stability

� Goal 2 Quality :

� Sub-goal 2.1: Improve reliability

� Sub-goal 2.2: Improve system model maintainability

� Sub-goal 2.3: Improve system’s documents quality

� Goal 3 Complexity :

� Sub-goal 3.1: Improve management of configurability

� Sub-goal 3.2: Improve verification ability

� Sub-goal 3.3: Improve handling of change

� Sub-goal 3.4: Improve system modelling capabilities


10-06-3026

GOAL 1: IMPROVE EFFICIENCY 1/3

� Sub-goal 1: Reduce development cycle time

� Measurement goal: Analyse the impact of RTP tool usage on the development time.

� Questions:

� Does RTP tool reduce development time?

� Indicators:

� Distribution of the normalised elapsed time (normalised by total size) of each development phase and for the whole development process.

� Measures: For each Pilot applications:

� Elapsed time for each development phase

� Total development effort


10-06-3027

3 STEP PROCESS








� Efficiency (%)

� Quality (%)

� Complexity (%)



10-06-3028

CREATION OF BASELINE: HARMONIZATION

� Harmonization of Development process for all CESAR Domains

� 5 Phases:

� User’s needs

� System Design & HW/SW Specification

� HW/SW Design Coding & Build Unit Testing

� System Integration Testing

� Certification / Qualification activities

� Support Activities

� Domain specific activities


10-06-3029

CREATION OF BASELINE: EFFORT ALLOCATION (1/2)

� Allocation of development effort to harmonized development process

� Development effort before impact of CESAR Technical Innovations


10-06-3030

CREATION OF BASELINE: EFFORT ALLOCATION (2/2)

Requirement capture and elicitation

Requirement translation in formal

language

Safety assessmentRequirement management

Validation

Document generation

HW SW integrationRequirement managementVerification

Document generation

Requirement managementDocument generation

Requirement managementSW development and reuseHW development and reuse

Safety assessmentVerificationDocument generation

Requirement management

System architecture definition

Safety assessment Prototyping

Automatic test generation

Validation

HW/SW Specification

Document generation

Support Activities : Collaborative work

Configuration management

Phase 1

User's needs

Phase 2

System Design

& HW/SW

Specification

Phase 3

HW/SW Design,

Coding & Build,

Unit Testing

Phase 4

System Integration

Testing

Phase 5

Certification /

Qualification

activities

X %X %

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

Phase 1 Phase 2 Phase 3 Phase 4 Phase 5

X %X %

X %X %

X %X %

X %X %

Overall baseline represents 100% of cost/effort repartition on our development process

Main activity level allows :• Finest baseline• Mapping of main activities addressed for each PA �baseline for each pilot application.


10-06-3031

3 STEP PROCESS








� Efficiency (%)

� Quality (%)

� Complexity (%)



10-06-3032

MAPPING OF PILOT APPLICATION TO BASELINE

PAs

Phases & Activities

Mapping

� Not all PA cover the entire development process but: the sum of all PA covers the whole System Engineering process.


10-06-3033

3 STEP PROCESS








� Efficiency (%)

� Quality (%)

� Complexity (%)



10-06-3034

MEASUREMENT OF TI-IMPACT

� Parallel Performance or

� Measurement versus Baseline


10-06-3035

BACKUP


10-06-3036


� Sub-goal 1: Reduce development cycle time

� Measurement goal: Analyse the impact of RTP tool usage on the development time.

� Questions:

� Does RTP tool reduce development time?

� Indicators:

� Distribution of the normalised elapsed time (normalised by total size) of each development phase and for the whole development process.

� Measures: For each Pilot applications:

� Elapsed time for each development phase

� Total development effort


10-06-3037


� Sub-goal 2: Find defects as early as possible

� Measurement goal: Analyse the system defects introduction and detection in order to identify opportunities for reducing cost.

� Question:

� How much effort is spent due to discovering defects late in time?

� Indicators:

� Effort spent in fixing failures by introduction phase

� Effort spent in fixing failures by discovering phase.

� Relative effort spent in corrective maintenance (relative to development effort) vs. development effort

� Measures: For each pilot application:

� No. of failures

� For each failure in each pilot application:

� Total effort for fixing a failure

� Phase where failure introduced

� Phase where failure found


10-06-3038


� Sub-goal 1: Improve requirements stability

� Measurement goal: Assess requirements requests for change in order tounderstand their impact on schedule completion date.

� Questions:

� Does the number of changes to requirements decrease with time?

� How many requirements change requests were received in each development phase and for the whole pilot application?

� What are the main reasons for requirements change requests?

� Indicators:

� Profile of current and cumulative No. of requirements changes.

� Distributions of requirements change requests during each development phase.

� Distributions of requirements change requests according to reason.

� Measures: For each pilot application

� No. of requirements change requests.

� No. of requirements change requests per development phase

� No. of requirements change requests per reason of change

CESAR – APPROACH & MEASUREMENT OF OBJECTIVES

Documents