Week 7 - Systems Engineering and Analysis

1

Week 7 - Systems Engineering and Analysis – Making it work together!

Buede

Chapter 10 – Interface Design

Chapter 11 – Integration and Qualification

(Side order of Rapid Development)

2

At some point in big system building…

• Somebody has to tie it all together!

That would be a systems engineering job!

We’ll revisit this figure…

3

Both authors do talk a lot about interfaces to users, vs

systems• E.g., in Wasson, “user” concerns are

mentioned all over:

• Ch 6 – System Acceptability– Client is the “acquirer,” vs the real users

• Ch 10 – There is a “PERSONNEL System Element”

• Ch 11 – The operating environment depends on the missions the User plans to accomplish.

• Ch 17 – all about use cases.

3

4

Interfaces to other “things”• A connection for ‘hooking together’

system components – external or internal.

• Have a logical and physical component responsible for carrying information or electromechanical energy from one component to another.

•Must identify interfaces, evaluate options, and allocate inputs/outputs to them.

5

Approaches to Interface Considerations

• Buede’s Ch. 10 – somewhat limited in scope to computer/software interfaces.– (Good for us software types!)

• Wasson – detailed

• Schindel– Some tips, added to these slides

• Rapid Prototyping

6

Wasson on Interfaces• Interfaces are all about how systems and

subsystems interact with each other and their operating environment.

• Interface Purposes1. Physically link or bind two or more system elements

2. Adapt one or more incompatible system elements

3. Buffer the effects of incompatible system elements

4. Leverage human capabilities

5. Restrain a system element and its usage

7

Broad View of Interfacing

• Wide variety of ‘interfaces’ in all systems and products.– Software

– Communication

– Electrical

– Mechanical

– Optical

– Acoustical

– Chemical

– Biological

– Etc…

8

Logical vs. Physical

• Logical– Direct or indirect

association between two entities

– Who communicates

– What scenarios

– When to communicate

• Physical– How communication

happens

– ‘Specialized’ interfaces

9

Standardized vs Dedicated

• Standardized – standard, modular, interchangeable interfaces – complying with a ‘standard’ like RS-232, USB, Ethernet, etc.

• Dedicated – Unique, dedicated interfaces – often limiting compatibility with other systems

10

Wasson Interface Thoughts

• Consider interactions between system and environment.• Often focus on a few, not all interactions• Environmental effects are natural, induced, and human-

made.

11

Where are the Interfaces? USED AT: CONTEXT:

NODE: TITLE: NUMBER:

AUTHOR:PROJECT:

NOTES: 1 2 3 4 5 6 7 8 9 10

DATE:REV:

WORKING

DRAFT

RECOMMENDED

PUBLICATION

READER DATE

P.

ProvideATM Services

A-0

Perform Customer Activities

A-11

Provide Bank Information to

ATM

A-12 Maintain ATM

A-13

CustomersATM System

BankComputer

BankEmployees

Customer Needs BankingPolicies

General ID,Unique ID

Request for Unique ID,Request for Activity,Request for Account Type,Request for Deposit Type,Physical Means for Insert,Receipt, Request for Amount,Request Denied, Cust Cash,Request for Source Account,Request for Dest Account

Transaction,Request for Fmax,Request for Status Inf..,Input Not Working,Request for Funds,Request for Receipts,Break-in Attempted

Access Opportunity,ATM Opened,Cust Deposits,Cust Payments,Test Results,Fixes AppliedATM Closed

Request to Open,ATM Cash, Blank Receipts,Initialization,Diagnostic Test,ATM Fixes,Request to Close

RobATM

A-14

Robber

Activity Selection,Account Type,Deposit Type,Deposit of Funds,Trans Amount,Source Account,Dest Account,Source of Payment,Payment on Account,Request to Cancel,Choice to End

AudibleAlarm,OperationTerminated

Main Menu

Cust Status Inf..,Fmax

EmployeeID Code

Break-inAttempt

Operational Phase External Systems

Choice, ATM Reset,No Input Device,Request for ID #2,Request for ID #3,Customer Alert, Apology,Request for Paymt Source

Americans withDisabilities Act

SafetyRegulations

Clim

None

1

xAutomatic Teller MachineSYST 520 Student

George Mason University

09/28/99

A-1

At:InputsControlsOutputs

12

Where are the Interfaces ? USED AT: CONTEXT:


AUTHOR:PROJECT:

NOTES: 1 2 3 4 5 6 7 8 9 10

DATE:REV:

WORKING

DRAFT

RECOMMENDED

PUBLICATION

READER DATE

P.




EmployeeID Code

AudibleAlarm,

OperationTerminatedBreak-in

Attempt

SafetyRegulations


Choice, ATM Reset,No Input Device,

Request for ID #2,Request for ID #3,

Customer Alert, Apology,Request for Paymt Source

Request for Unique ID,Request for Activity,

Request for Account Type,Request for Deposit Type,Physical Means for Insert,

Receipt, Request for Amount,Request Denied, Cust Cash,

Request for Source Account,Request for Dest Account

Transaction,Request for Fmax,

Request for Status Inf..,Input Not Working,Request for Funds,

Request for Receipts,Break-in Attempted


Access Opportunity,ATM Opened,

Cust Deposits,Cust Payments,

Test Results,Fixes Applied

ATM Closed

Main Menu

Provide Access to

ATM

A1

Accept Customer

Requests and Provide

Feedback

A2Determine

ATM Responses

A3

Communicate with Bank Computer

A4

Enable Re-Supply and Maintenance

A5

Respond to Hostile

SituationsA6

Request forUnique ID

BankingPolicies

No Input Device,Request for ID #2,Request for ID #3,Customer Alert

Choice, ATM Reset,Apology, Requestfor Paymt Source

Request for Activity,Request for Account Type,Request for Deposit Type,Physical Means for Insert,Receipt, Request for Amount,Request Denied, Cust Cash,Request for Source Account,Request for Dest Account

CustomerValid

EmployeeValid

ID Validation

ID Received

Activity Selected,A/C Type Entered,Deposit Type EnteredDeposit Received,Amount Entered,Source A/C Entered, Dest A/C Entered,Ftrns>Fmax

Need for Fmax,Trans Complete,Receipts<25

Need to Open,Paymts Inserted,Deposits Inserted,Diagnostics,Fixes to ATMNeed to Close

Creq>Cleft

Balance Inf.,Paymt Source Entered,Payment Received,Ptrns>Fmax, Cancel Received,Choice Received

AccountFMax

Account Balance

Cust Activity,Cust A/C Type,Deposit Entered,Cust Amount,Trans Source,Trans Dest,Paymt Source,Payment Entered,Cancel Entered,Choice Entered

Attempted Break-in

Clim

CalculationsforWithdrawal

Input notAvailable

3



08/07/00

Provide ATM ServicesA0

At:InputsControlsOutputs

14

Interface Design Process

Figure 10.7

Define Interface Requirements Identify the Items to Be Transported by the InterfaceDefine the Operational ConceptBound the Problem with an External Systems DiagramDefine the Objectives HierarchyWrite the Requirements

Select a High-Level Interface Architecture of InterfaceIdentify Several Candidate ArchitecturesDefine Trial Interfaces for Each CandidateEvaluate Alternatives against RequirementsChoose High-Level Interface Architecture

Develop Functional Architecture of InterfaceSpecify Functional DecompositionAdd Inputs and OutputsAdd Fault Detection and Recovery Functions

Develop Physical Architecture of InterfaceIdentify Candidates based upon High-Level ArchitectureEliminate Infeasible Candidates

Develop Operational Architecture of InterfaceAllocate Functions to Components of the InterfaceAnalyze Behavior and Performance of AlternativesSelect AlternativeDocument Design and Obtain Approval

Add Functions to Components Connected to Interface as NeededThe same SE process !!

16

Benefits of Standards

• Interchangeability - ability to interchange components with different performance and cost characteristics

• Interoperability - the system can now operate with a wider variety of external systems, systems that have also adopted the same conventions

• Portability - systems can be moved and operate on other systems

• Reduced cost and risk for equivalent performance

• Increased life cycle is possible when long-lived standards are adopted

17

(Bill) Schindel Interface Thoughts

• List of all I/O that pass through it

• List of functions or behaviors at the interface

• Association with a system that owns the interface

• SOA – technologies or systems that support the interface

• An interface does not reside ‘between’ two systems.

18

Schindel Interface Thoughts, cntd






Additional Thoughts:

All I-C-O to a function must be associated with an interface.

In addition to the function/behavior, also list the physical characteristics.

19

Interface Examples

20

Interface TemplateFunctiona

l or Physical Element

Signals Input or

Output

Interface Behavior

Key Interface Attribute

s(Wasson

43.1)

SOA Physical Descripti

on

Applicable

Standards

30

Defining and Managing Interfaces

• IRS – Interface Requirements Specification (not always necessary)

• ICWG – Interface Control Working Group

• ICD – Interface Control Document (Hdwe)

• IDD – Interface Design Description (Sftwe)

31

Schindel Interface Thoughts - Reprise






32

Key Interface Attributes

33

Interface Definition and Control Challenges – Pg 519

34

Oasis Example Source Interface In/Out Frequency Owner SOA

Building Electrical Energy In continuous Building ElectricalBuilding Network In/Out as needed Oasis InternetUser Selection In as needed User KeypadUser Money In as needed User Cash SystemSupport Soda In as needed User ManualRepair Repair In as needed User ManualRepair Materials In as needed User ManualRepair Diagnostic Tests In as needed User Maint KeypadRepair Place into Service In as needed User ManualSupport Temperature Setting In as needed User Maint KeypadUser Out-of-Service Out as needed Oasis Status IndicatorUser Money Received Out as needed Oasis Status IndicatorUser Request Selection Out as needed Oasis Status IndicatorUser Selection not Available Out as needed Oasis Status IndicatorUser Soda Dispensing Out as needed Oasis Status IndicatorUser Fill Status Indicator Out as needed Oasis Status IndicatorUser Money Out on command Oasis Cash SystemUser Change Out as needed Oasis Cash SystemUser Soda Out as needed Oasis Cash SystemRepair Diagnosis Response Out on command Repair Tech Status IndicatorHome Office Soda Quantity Notification Out as needed Oasis InternetHome Office Out-of-Service Notification Out as needed Oasis Internet

What is Good ? What is Bad ?

35

Context of ‘Rapid Development’• ‘For us – “Agile” and “Lean”• Designing Products in Half the Time’,

Reinertsen

• Keys –

• Functional decomposition and allocation– Modularity – more rapid development, but more

modules, more interfaces.

• Performance margin in each module.– More margin, fewer changes, but higher cost.

36

‘Rapid Development’-2

• Keys –

• Interface Design– Link modules and functions together.

– Stable, robust, standard, simple.

– Robust – performance, electrically, mechanically, environmentally, etc.

– Standard – faster, cheaper.

37


• Goal –

• Design ‘Architectures’ fast, early.

• Modular design with known interfaces.

• Flow down requirements to interfaces.

• Then - Work on modules independently, concurrently.

38


• Standard Interface-– Need ‘interface’ to carry 110 VAC power from a power

source to electrical system under development.

• Standard interface – buy at hardware store (COTS) (commercial off the shelf).

• Custom interface – design, develop,

manufacture, safety/life testing, etc.

39

Interface Thoughts

• An interface is a property of a system component, it does not reside between two systems. – Bill Schindel

• The distinction between :– The Interface,

– The Physical Component,

– The ‘Standard’ involved, if any.

A car battery is a standard interface to provide electrical

power to the car….(?)

40

A Car Battery

• What are the external systems ?

• What are the inputs and outputs ?

• An interface at each one !

41

Interface Implications

• More interfaces – more modular, upgradeable, testable, but more expense.

• But, systems tend to fail at an interface – solder joint, connector, bolted joint, data transfer between modules, etc.

• May ‘over-design’ each module

42

The ATM Functional Architecture

• Define the Interface at each I-C-O.

• Define the logical/functional behavior.

• Define the physical and standard/custom instantiations.

USED AT: CONTEXT:


AUTHOR:PROJECT:

NOTES: 1 2 3 4 5 6 7 8 9 10

DATE:REV:

WORKING

DRAFT

RECOMMENDED

PUBLICATION

READER DATE

P.




EmployeeID Code

AudibleAlarm,

OperationTerminatedBreak-in

Attempt

SafetyRegulations


Choice, ATM Reset,No Input Device,

Request for ID #2,Request for ID #3,

Customer Alert, Apology,Request for Paymt Source

Request for Unique ID,Request for Activity,

Request for Account Type,Request for Deposit Type,Physical Means for Insert,

Receipt, Request for Amount,Request Denied, Cust Cash,

Request for Source Account,Request for Dest Account

Transaction,Request for Fmax,

Request for Status Inf..,Input Not Working,Request for Funds,

Request for Receipts,Break-in Attempted


Access Opportunity,ATM Opened,

Cust Deposits,Cust Payments,

Test Results,Fixes Applied

ATM Closed

Main Menu

Provide Access to

ATM

A1

Accept Customer

Requests and Provide

Feedback

A2Determine

ATM Responses

A3

Communicate with Bank Computer

A4

Enable Re-Supply and Maintenance

A5

Respond to Hostile

SituationsA6

Request forUnique ID

BankingPolicies

No Input Device,Request for ID #2,Request for ID #3,Customer Alert

Choice, ATM Reset,Apology, Requestfor Paymt Source

Request for Activity,Request for Account Type,Request for Deposit Type,Physical Means for Insert,Receipt, Request for Amount,Request Denied, Cust Cash,Request for Source Account,Request for Dest Account

CustomerValid

EmployeeValid

ID Validation

ID Received

Activity Selected,A/C Type Entered,Deposit Type EnteredDeposit Received,Amount Entered,Source A/C Entered, Dest A/C Entered,Ftrns>Fmax

Need for Fmax,Trans Complete,Receipts<25

Need to Open,Paymts Inserted,Deposits Inserted,Diagnostics,Fixes to ATMNeed to Close

Creq>Cleft

Balance Inf.,Paymt Source Entered,Payment Received,Ptrns>Fmax, Cancel Received,Choice Received

AccountFMax

Account Balance

Cust Activity,Cust A/C Type,Deposit Entered,Cust Amount,Trans Source,Trans Dest,Paymt Source,Payment Entered,Cancel Entered,Choice Entered

Attempted Break-in

Clim

CalculationsforWithdrawal

Input notAvailable

3



08/07/00

Provide ATM ServicesA0

43

Class Exercise – to think about

• Consider the following systems. Further consider the typical subsystems which make them up and identify interfaces necessary to support the system.

• Use the Interface Template as a guide to describe the interface behavior and characteristics. Note that several signals may pass through the same interface.

1. Elevator System (*)2. Desktop PC3. Machine Tool4. This Classroom5. Digital Camera6. Your Car7. Your Project System …..

44

Interface TemplateFunctiona

l or Physical Element

Signals Input or

Output

Interface Behavior

Key Interface Attribute

s(Wasson

43.1)

SOA Physical Descripti

on

Applicable

Standards

45

Week 7 - Systems Engineering and Analysis,

cntd

Buede Chapter 11 –

Verification/Validation,

Integration and Qualification

46

Definitions

• Integration is the process of assembling the system from its components, which must be assembled from their configuration items (CIs)

• Qualification is the process of verifying and validating the system design and then obtaining the stakeholders’ acceptance of the design.

• Qualification = ‘Testing’.

– Verification is the determination that the system was built right (more of a process focus)

– Validation determines that the right system was built (more of a product focus)

47

Verification, Validation

and Acceptance

DesignEngineering

Systems Engineering

SE VeeTime

Operational Concept

Originating Requirements

System Requirements

Element Specs

Segment Specs

Component Specs

CI Specs

System Delivered

Elements Delivered

Segments Delivered

Components Delivered

CIs Delivered

OperationalValidity

Stakeholders’Needs

Acceptability

DevelopmentalVerification

ConceptualValidity

DesignValidity

RequirementsValidity

Figure 11.1

1

2

3 4

5

6

Verification = Built RightValidation = Right System

Acceptance = Stakeholder OK

50

Verification/Validation/Acceptance

• Do not happen sequentially

• Do not only happen late in the SE process

51

Major Integration

Functions for Component Integration

U S E D A T : C O N T E X T :

N O D E : T I T L E : N U M B E R :

A U T H O R :P R O J E C T :

N O T E S : 1 2 3 4 5 6 7 8 9 1 0

D A T E :R E V :

W O R K I N G

D R A F T

R E C O M M E N D E D

P U B L I C A T I O N

R E A D E R D A T E

P .

I n s p e c t &

V e r i f y C I

A 2 2 1 1

I d e n t i f y & F i x C o r r e c t a b l e

C I D e f i c i e n c i e s

A 2 2 1 2A s s e s s

I m p a c t o f U n c o r r e c t a b l e

C I D e f i c i e n c i e s

A 2 2 1 3

R e d e s i g n C I

A 2 2 1 4M o d i f y C I B a s e l i n e

A 2 2 1 5

I n t e g r a t e w i t h

N e x t C I

A 2 2 1 6

D i s c r e p a n c y R e p o r t s

C IE n g i n e e r i n g C h a n g e s

C l e a r e d C I

C o r r e c t e d C IU n c o r r e c t e d C I

A p p r o v a l t o C o n t i n u e I n t e g r a t i o n

U n a c c e p t a b l e I m p a c t

A c c e p t a b l e I m p a c t

C l e a r e d C I

D e f i c i e n tC I

R e d e s i g n e d C I

Q u a l i f i c a t i o n P r o c e d u r e s ,A c t i v i t i e s , & M o d e l s

" B u i l t - t o "C o m p o n e n t s

" B u i l t - t o "C I s

C I V e r i f i c a t i o nC h a n g e s

C I T e s tR e s u l t s

C o m p o n e n tV e r i f i c a t i o nD o c u m e n t s

C o m p o n e n t L e v e lD e s i g n D o c u m e n t s

S u b s y s t e m -G e n e r a t e dC o m p o n e n tR e g r e s s i o nQ u a l i f i c a t i o n

B a s e l i n eC h a n g e s

I m p a c tS t a t e m e n tS y s t e m -

G e n e r a t e dC o m p o n e n tR e g r e s s i o nQ u a l i f i c a t i o n

1 7

x

E n g i n e e r i n g o f a S y s t e mD e n n i s B u e d e

G M U S y s t e m s E n g i n e e r i n g

P r o g r a m

1 1 / 0 4 / 9 8

P e r f o r m C o m p o n e n t I n t e g r a t i o n & V e r i f i c a t i o nA 2 2 1

N O D E : T I T L E : N U M B E R : P .

V e r i f i c a t i o nD a t a

P e r f o r m C o m p o n e n t

I n t e g r a t i o n & V e r i f i c a t i o n

A 2 2 1

P e r f o r m S u b s y s t e m


A 2 2 2

P e r f o r m S y s t e m


A 2 2 3

Q u a l i f i c a t i o n P r o c e d u r e s ,A c t i v i t i e s , & M o d e l s

" B u i l t - t o "C I s

V e r i f i c a t i o nC h a n g e s

V e r i f i c a t i o nD o c u m e n t

D e r i v e d &O r i g i n a t i n g

R e q u i r e m e n t s

C o m p o n e n t L e v e lD e s i g n D o c u m e n t s

S u b s y s t e m L e v e lD e s i g n D o c u m e n t s

O r i g i n a t i n g & S y s t e mR e q u i r e m e n t sD o c u m e n t s

" B u i l t - t o "C o m p o n e n t s

" B u i l t - t o "S u b s y s t e m s

S y s t e m - L e v e lR e q r e s s i o nQ u a l i f i c a t i o n

S y s t e m -G e n e r a t e dS u b s y s t e mR e g r e s s i o nQ u a l i f i c a t i o n

S y s t e m -G e n e r a t e dC o m p o n e n tR e g r e s s i o nQ u a l i f i c a t i o n

S u b s y s t e m -G e n e r a t e dC o m p o n e n tR e g r e s s i o nQ u a l i f i c a t i o n

C I T e s tR e s u l t s

S u b s y s t e mT e s tR e s u l t s

C o m p o n e n tT e s tR e s u l t s

S y s t e mV e r i f i c a t i o nD o c u m e n t

S u b s y s t e mV e r i f i c a t i o nD o c u m e n t s

C o m p o n e n tV e r i f i c a t i o nD o c u m e n t s

C o m p o n e n tV e r i f i c a t i o nC h a n g e s

S u b s y s t e mV e r i f i c a t i o nC h a n g e s

C I V e r i f i c a t i o nC h a n g e s

1 6C o n d u c t I n t e g r a t i o n & V e r i f i c a t i o nA 2 2

Figure 11.4 The Textbook Approach is ‘Bottom Up’

54

Approaches to Integration

1.Top Down

2.Bottom Up (*)

3.Big Bang

55

Top Down IntegrationIntegration begins with a major or top-level module.

All modules called from the top-level module are simulated by “stubs” (shell or model replica).

Once the top-level module is qualified, actual modules replace the stubs until the entire system has been qualified.

This is most useful for systems using large amounts of COTS components.

Phase Integration: Integration is done from the top down to the lowest level; one peel of the onion at a time.

Incremental Integration: Integration is done for a specific module from top to bottom; one slice of the system at a time.

Advantage

• Early demonstration of the system is allowed.

• Representation of the test cases is easier.

• More productive if major flaws occur toward the top of system.

Disadvantage

• Stubs have to be developed.

• Representation of test cases in the stubs may be difficult.

• Observation of test output may be artificial and difficult.

• This requires a hierarchical system architecture.

Table 11.1

56

Bottom-up IntegrationIntegration begins with the elementary pieces (or CIs) of the system.

After each CI is tested, components comprising multiple CIs are tested.

This process continues until the entire system is assembled and tested.

This is the traditional systems engineering integration approach. Phase Integration: At any point in the integration, all of the subsystems are at the same

stage of integration testing.

Incremental Integration: proceeds one slice of the system at a time.

Advantage

• It is easier to detect flaws in the tiniest pieces of the system.

• Test conditions are easier to create.

• Observation of the test results is easier.

Disadvantage

• “Scaffold” systems must be produced to support pieces as they are integrated.

• System’s control structure cannot be tested until the end.

• Major errors in the system design are typically not caught until the end.

• System does not exist until the last integration test is completed.

• This requires a hierarchical system architecture.

Table 11.1

57

Big Bang Integration

Untested CIs are assembled and the combination is tested.

This is a commonly used and unsuccessful approach. Advantage• Immediate feedback on the status of system elements is

provided.• Little or no pretest planning is required.• Little or no training is required.Disadvantage• Source of errors is difficult to trace.• Many errors are never detected. • Test it until it ‘works’ then stop.• Errors found by customers

Table 11.1

58

Bottom-Up Integration of

RC Car

Controller

Car

Fully IntegratedCar/Controller

System

Directional Interface

Motion Interface

Controller Processor

Processor Logic (Software)

Signal Transmission

Power Control

Power Confirmation

Battery Interface

Signal Receiver

Signal Processor

Processor Logic (Software)

Steering

Alignment

Power Control

Power Confirmation

Battery Interface

Locomotion

Phase 1

Interface Module

CPU Module

Power Module

Integrated ControllerSubsystem

Phase 2 Phase 3

Structural Design

Structural Design

CPU Module

Motion Module

Power Module

Structural Design

Integrated CarSubsystem

Structural Design

Phase 4

59

Exercise: Discussion of Examples

1. How did they do integration on the Denver Airport project.

2. Even when large amounts of time/money are spent on integration/qualification, how are mistakes still made – Genesis and other space vehicle failures ?

3. Why is the Big Bang approach so popular or often used?

4. What integration approach would you recommend for your project ?

60

Qualification Planning During Design

• The design of the qualification system is not only important in terms of finding and defining faults and errors but also in guiding designers to preclude them from introducing faults in the first place. Buede

61

Qualification – The ‘Readers Digest’ Edition

1. If you can't test it, don't build it.

2. If you don't test it, rip it out.

• Boris Beizer, Second edition, chapter 13, section 3.2.5. , "Software testing techniques" by Boris Beizer , ISBN: 0442206720

62

One Way to Look at it USED AT: CONTEXT:


AUTHOR:PROJECT:

NOTES: 1 2 3 4 5 6 7 8 9 10

DATE:REV:

WORKING

DRAFT

RECOMMENDED

PUBLICATION

READER DATE

P.

ProvideATM Services

A-0


A-11


ATM

A-12 Maintain ATM

A-13

CustomersATM System

BankComputer

BankEmployees







RobATM

A-14

Robber



Main Menu


EmployeeID Code

Break-inAttempt




SafetyRegulations

Clim

None

1



09/28/99

A-1

Qualify ATMMachineDevelop a Systems

Model for this Function

63

Circuit Board

Testing

Qualification an Afterthought

64

Circuit Board

TestingQualification planned and designed

65

Failure Definitions• Failure: deviation in behavior between the system and its

requirements. Since the system does not maintain a copy of its requirements, a failure is not observable by the system.

• Error: a subset of the system state, which may lead to a failure. The system can monitor its own state, so errors are observable in principle. Failures are inferred when errors are observed. Since a system is usually not able to monitor its entire state continuously, not all errors are observable. As a result, not all failures are going to be detected (inferred).

• Fault: defects (bugs) in the system that can cause an error. Faults can be permanent (e.g., a failure of system component that requires replacement) or temporary due to either an internal malfunction or external transient. Temporary faults may not cause a sufficiently noticeable error or may cause a permanent fault in addition to a temporary error.

66

Levels of Bug/Fault Severity and Consequences

• Mild

• Moderate

• Annoying

• Disturbing

• Serious

• Very Serious

• Extreme

• Intolerable

• Catastrophic

• Infectious

Levels of coming up “Martin Short”

67

Boris Beizer’s 3 Laws of Software Testing

• First Law: The Pesticide Paradox – Every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffective.

• Second Law: The Complexity Barrier – Software complexity (and therefore that of bugs) grows to the limits of our ability to manage that complexity.– Test developers are no ‘smarter’ than code developers. Errors on

both sides.

• Third Law – Code migrates to data. – More and more of the control and processing functions are stored in

tables. Control tables having hidden programming languages, generalized packages with parametric data to the code, etc.

– Because of this law there is increasing awareness that bugs in code are only half the battle and that data problems should be given equal attention.

68

Rate of Finding Bugs

Time – Months and Years

100% BugsFound

Easy to Find,Obvious,Frequent

Harder to Find

Very Difficult to Find,Intermittent / Infrequent

Time or Configuration Based

70

Qualification Planning Functions

• Plan the qualification process– Acceptance test

– Validation test (Built right system ? – Product)

– Verification test (System built right? – Process)

• Plan the qualification approaches

• Plan qualification activities

• Plan specific tests

Table 11.2

71

What do we need to Qualify ?

• Remember we defined ‘qualification’ requirements. (The ‘what’ to qualify)

• All Input and Output requirements ?

• All Technology and Systemwide requirements ?

72

Plan the qualification process· Review system objectives

· Identify qualification system objectives

· Identify pass/fail thresholds

· Define qualification operational concepts

· Define qualification requirements

· Define qualification functional architecture

· Define qualification generic physical architecture

· Generate qualification coverage matrices (allocate requirements to functional architecture and functions to the generic physical architecture)

· Identify risks and mitigation strategies

· Create master qualification plan

Table 11.2

73

Plan the qualification approaches

· Define qualification resources and organizations (instantiated physical architecture)

· Assign qualification activities to organizations

· Allocate qualification activities to resources

· Develop qualification schedules consistent with development schedule

Table 11.2

74

Plan qualification activities· Develop detailed derived qualification

requirements

· Develop functional architectures for qualification components

· Generate coverage matrices (allocate derived requirements to functional architectures and functions to physical architectures)

· Write activity-level qualification plans for each qualification component

· Assign qualification responsibilities

Table 11.2

75

Plan specific tests

· Identify required stimulation data for each activity

· Create test scenarios

· Write test procedures

· Write analysis procedures

· Define test and analysis schedules

Table 11.2

76

Qualification MethodsMethod Description Used During: Most Effective When:

Inspection (static test)

Compare system attributes to requirements.

During all segments of verification, validation, and acceptance testing for requirements that can be addressed by human examination.

Success or failure can be judged by humans; examples include inspection of physical attributes, code walkthroughs, and evaluation of user’s manuals.

Analysis and simulation

Use models that represent some aspect of the system. Examples of models might address system’s environment, system process, system failures.

Used throughout qualification, but emphasis is early in verification and during acceptance. Often used in conjunction with demonstration.

Physical elements are not yet available. Expense prohibits instrumented test, and demonstration is not sufficient. Issue involves all or most of the system’s life span. Issue cannot be tested (e.g., survive nuclear blast).

Instrumented test

Use calibrated instruments to measure system’s outputs. Examples of calibrated instruments are oscilloscope, voltmeter, LAN analyzer.

Verification testing. Engineering test models through system elements are available. Detailed information is required to understand and trace failures. Life and reliability data is needed for analysis and simulation.

Demonstration or field test

Exercise system in front of unbiased reviewers in expected system environment.

Primarily used for validation and acceptance testing.

Complete instrumented test is too expensive. High-level data/information is needed to corroborate results from analysis and simulation or instrumented test.

Table 11.3

77

Testing Methods Functional testing

Test conditions are set up to ensure that the correct outputs are produced, based upon the inputs of the test conditions. Focus is on whether the outputs are correct given the inputs (also called black box testing).

Structural testing Performance Recovery Interface Stress testing

Examines the structure of the system and its proper functioning. Includes such elements as performance, recovery, stress, security, safety, availability. Some of the key elements are described below. Examination of the system performance under a range of nominal conditions, ensures system is operational as well. Various failure modes are created and the system’s ability to return to an operational mode is determined. Examination of all interface conditions associated with the system’s reception of inputs and sending of outputs. Above-normal loads are placed on the system to ensure that the system can handle them; these above-normal loads are increased to determine the system’s breaking point; these tests may proceed for a long period of time in an environment as close to real as possible.

Table 11.4

78

Black & White Box Testing

Black box testing

Outputs are determined correct or incorrect based upon inputs; inner workings of the module are ignored. Both positive and negative testing have to be employed. This approach is scalable to system-level testing. Positive testing pulls the test data and sequences from the

requirements documents. Negative testing attempts to find input sequences missed in the

requirements documents and then determine how the module reacts. Crash testing is an example.

White box testing

Inner workings of the module are examined as part of the testing to ensure proper functioning. Usually used at the CI level of testing; this method becomes impractical at the system level. Path testing addresses each possible simple functionality and is

based upon a prescribed set of inputs. Path domain testing partitions the input space and then examines the

outputs for each partition of the input space. Mutation analysis injects predefined errors and tests the error

detection and recovery functionalities.

Table 11.6

79

Acceptance Testing• Final step in qualification

• Acceptance testing is conducted by Stakeholders.

• Acceptance criteria must be defined early.

• Outcome : accept, accept with changes, not accept.

• Key issue - How much to test

and what assumptions.

81

Another Way to Look at it USED AT: CONTEXT:


AUTHOR:PROJECT:

NOTES: 1 2 3 4 5 6 7 8 9 10

DATE:REV:

WORKING

DRAFT

RECOMMENDED

PUBLICATION

READER DATE

P.

ProvideATM Services

A-0


A-11


ATM

A-12 Maintain ATM

A-13

CustomersATM System

BankComputer

BankEmployees







RobATM

A-14

Robber



Main Menu


EmployeeID Code

Break-inAttempt




SafetyRegulations

Clim

None

1



09/28/99

A-1

Qualify ATMMachine

•This becomes our new ‘system of interest’•Its external systems are the ATM, Customer, etc.•We have some requirements from the ATM itself•Build a new SE model for it•Places needs and requirements back on the ATM

82

The Qualification System as an External System

• I/C/O from ATM function to Qualification function.

• Can model and decompose the Qualification function.

83

RC Car Exampl

e

84

Test Equipment and Resources

Requirement ID Requirement Description Equipment Resources Timetable

[ORG-SW: 2.6.3.006] The speed of the car shall be 15mph or more. The design goal is 20mph.

o Controllero Caro Speed Radar

o Testero Open Track o 30 minutes of testing

Test completedBy 11/10/07

[ORG-SW: 2.6.3.007] The battery life of the car under constant full throttle shall be 10 minutes or more. The design goal is 15 minutes.

o Controllero Car o Fully Charged Battery

o Testero 20 minutes of testing at full throttle condition

Test completed By 10/28/07

[DER: 2.12.001] The R/C car shall accelerate to maximum speed within 2 sec of engaging full throttle position.

o Controllero Car o Speed Radar o Stop Watch

o Testero 20 minutes of testing at full throttle condition


[ORG-SW: 2.6.3.008] The suspension of the car shall have a spring constant k between 50lb/in and 60lb/in.

o Car Springso Certified Testinstrument for measuring force o Test fixture for springs

o Testero Statistical Sampling of 30Piece capability


85

Discussion

• What would a qualification plan look like for an ATM system?

• What would a qualification plan look like for an ATM manufacturing system?

• What would a qualification plan look like for an elevator?

86

Discussion

• What qualification approach and plan is needed for your project ?

• Do most engineers view qualification as a key part of the development process ?

• What are some of the reasons that make qualification process difficult to do well ?

Week 7 - Systems Engineering and Analysis

Documents