November, 2001 R. Dameron, University of Colorado, ECEN5033, Syst em Test Planning 1 System Test Planning and the usefulness of a “Safety Checklist” ECEN5033
Dec 21, 2015
November, 2001 R. Dameron, University of Colorado, ECEN5033, System Test Planning
1
System Test Planningand the usefulness of a “Safety Checklist”
ECEN5033
November, 2001 R. Dameron, University of Colorado, ECEN5033, System Test Planning
2
State Transition Diagram for “split” routine
Present State
Input Actions Outputs Next State
S0 DS Open F6
Open F7
S1
S1 D11 Write F6 D11; F6 S1
D12 Close F6
Write F7
D12; F7 S2
DE Close F6
Close F7
S0
S2 D12 Write F7 D12; F7 S2
DE Close F7 S0
November, 2001 R. Dameron, University of Colorado, ECEN5033, System Test Planning
3
Present State Input or Event Action Output Next State
ST1. Idle card inserted request for PIN Waiting for PIN
ST2. Waiting for PINPIN entered display asterisks Validating PIN
ST3. Waiting for PINcancel display msg Ejecting
ST4. Validating PINindicates “valid” display choices Waiting for customer
transaction choice
ST5. Validating PINindicates “stolen” display “stolen” confiscating
ST6. Validating PINindicates “invalid” display “invalid” Waiting for PIN
ST7. Waiting for customer transaction choice Cancel display “cancel” Ejecting
ST8. Waiting for customer transaction choice Balance Query selected Processing query
continued on next slide
November, 2001 R. Dameron, University of Colorado, ECEN5033, System Test Planning
4
ST9. Waiting for customer transaction choice
Withdrawal selected Processing w/d
ST10. confiscating Card confiscated terminating
ST11. Processing query Rejected for this user display “rejected” Ejecting
ST12. Processing query Query OK display printing printing
ST13. Processing withdrawal ok amount display ok msg dispensing
ST14. Processing withdrawal not ok amount display refusal Ejecting
ST15. Printing transaction complete print receipt ejecting
ST16. Dispensing sufficient cash in ATM cash printing
ST17. Dispensing insufficient cash in ATM disp “insufficient cash”ejecting
ST18. Ejecting card ejection started display msg to
take card terminating
ST19. terminating card ejection complete display ending msg idle
November, 2001 R. Dameron, University of Colorado, ECEN5033, System Test Planning
5
State Transition Diagram - incomplete
Idle
card inserted/
waiting for PIN
PIN inserted/
validating PIN
ejecting
“cancel”“invalid”
“stolen”
confis-cating
“valid”
waiting for cust xaction
“cancel”
terminat-ing
card ej
complete
card confis’d
November, 2001 R. Dameron, University of Colorado, ECEN5033, System Test Planning
6
2-dimensional event tableEVENT
Mode E13 E37 E45 …
Start-up
A16 ---- A14; A32
Steady X A6, A2 ---
Shut-down
--- --- ---
Alarm --- --- ---
Action;action = sequential actions. Action, action = concurrent actions. X = impossible. --- = no action required.
November, 2001 R. Dameron, University of Colorado, ECEN5033, System Test Planning
7
Checklist regarding Robustness
November, 2001 R. Dameron, University of Colorado, ECEN5033, System Test Planning
8
Robustness
11. If performance degradation is the chosen response, is the degradation predictable?
12. Are there sufficient delays incorporated into the error-recovery responses, e.g. don’t return to normal state prior to managing the error
13. Are feedback loops specified where appropriate to compare the actual effects of outputs on the system with the predicted effects?
November, 2001 R. Dameron, University of Colorado, ECEN5033, System Test Planning
9
Robustness, continued
14. Are all modes and modules reachable (used in some path through the code)? Superfluous? Other logic error?
15. If a hazards analysis has been done, does every path from a hazardous state (a failure-mode) lead to a low-risk state?
16. Are the inputs identified which, if not received, can lead to a hazardous state or can prevent recovery (single-point failures)?
November, 2001 R. Dameron, University of Colorado, ECEN5033, System Test Planning
10
Usefulness?
• Safety checklist has been demonstrated to “ask the right questions”
• Not sufficient to preclude introducing errors
• Necessary although not sufficient
November, 2001 R. Dameron, University of Colorado, ECEN5033, System Test Planning
11
May I have the envelope, please … Not every hazardous state led to a low-risk state.
Error-recovery responses incorporated
insufficient delays.
Input arrived when it shouldn’t and no response
was specified; response defaulted to
unintended behavior.
Response not specified for out-of-range values
that were possible for some inputs
#5 Output produced too fast for interfacing module
#4
#2
#3
#1
November, 2001 R. Dameron, University of Colorado, ECEN5033, System Test Planning
12
Allows tailoring
• Focuses on historically troublesome aspects of safety-critical, embedded software
• Avoids over-specification of well-understood or low-risk requirements
• Tailor to level of technical or historical risk
November, 2001 R. Dameron, University of Colorado, ECEN5033, System Test Planning
13
First step toward safety constraints
• Many items that it identifies are system hazards
• Can be used to identify safety constraints
• Not yet ready for formal prediction– How use for informal prediction of error prone
factors
November, 2001 R. Dameron, University of Colorado, ECEN5033, System Test Planning
14
After Requirements Are Improved …
• How do we ensure that requirements are implemented and maintained?– After code is written (new code or bug fixes);
note: difficult to unit test these issues– After new requirements are added– After old requirements are modified
• Role of reviews• Code the invariants where appropriate• System tests to test use cases and the safety
checklist
November, 2001 R. Dameron, University of Colorado, ECEN5033, System Test Planning
15
Create a system test plan – IEEE Std. 829
1. Test the Success Scenario and conditions that lead to alternative paths of use cases
2. If possible, test to verify the relevant safety checklist items – “safety” may not be main concern but correct interfaces and robustness are.
3. If any resources are shared among processes, review and test for correctness of mutual exclusion. (SW Eng of Multi-program Sys)
4. If “cooperating processes”, verify suspension happens correctly, a suspended process restored when appropriate, restoration correct.
November, 2001 R. Dameron, University of Colorado, ECEN5033, System Test Planning
16
IEEE 829 Standard Test Plan Outline
1.0 Introduction2.0 Test Items3.0 Tested Features4.0 Features Not Tested (per cycle)5.0 Testing Strategy and Approach
5.1 Syntax5.2 Description of functionality5.3 Arguments for Tests5.4 Expected Output5.5 Specific Exclusions5.6 Dependencies5.7 Test Case Success/Failure Criteria
November, 2001 R. Dameron, University of Colorado, ECEN5033, System Test Planning
17
IEEE 829 Standard Test Plan Outline - 11.0 Introduction
2.0 Test Items
3.0 Tested Features
4.0 Features Not Tested (per cycle)
[Repeat 5.0 for each system level test.]
5.0 Testing Strategy and Approach
5.1 Syntax
5.2 Description of functionality
5.3 Arguments for Test
5.4 Expected Output
5.5 Specific Exclusions
5.6 Dependencies
5.7 Test Case Success/Failure Criteria
November, 2001 R. Dameron, University of Colorado, ECEN5033, System Test Planning
18
IEEE 829 Standard Test Plan Outline - 26.0 Pass/Fail Criteria for the Complete Test Cycle
7.0 Entrance Criteria/Exit Criteria
8.0 Test-Suspension Criteria and Resumption Req’s
9.0 Test Deliverables/Status Communications Vehicles
10.0 Testing Tasks
11.0 Hardware and Software Requirements (for testing)
12.0 Problem Determination and Correction Responsibilities
November, 2001 R. Dameron, University of Colorado, ECEN5033, System Test Planning
19
IEEE 829 Standard Test Plan outline - 3
13.0 Staffing and Training Needs/Assignments
14.0 Test Schedules
15.0 Risks and Contingencies
16.0 Approvals
November, 2001 R. Dameron, University of Colorado, ECEN5033, System Test Planning
20
Glass-box – briefly (Need implementation details)
• Test module/process/object-cluster interfaces (process level can be system test)
• Test object/object-cluster contracts
• Create test data to force certain code paths
• Note: if team is doing incremental development, you can begin glass-box testing early
November, 2001 R. Dameron, University of Colorado, ECEN5033, System Test Planning
21
More to consider
• If the system is too large to test thoroughly, what tests should you emphasize?
• Stay tuned …
November, 2001 R. Dameron, University of Colorado, ECEN5033, System Test Planning
22
HAZOP
System Safety: HAZOP and Software HAZOP, by Felix Redmill, Morris Chudleigh, James Catmur, John Wiley & Sons, 1999
November, 2001 R. Dameron, University of Colorado, ECEN5033, System Test Planning
23
What is HAZOP?
• Technique for identifying and analyzing the hazards and operational concerns of a system.
• Central activity – a methodical investigation of a system description (design representation).
November, 2001 R. Dameron, University of Colorado, ECEN5033, System Test Planning
24
What this presentation does not cover:
• The book puts a LOT of emphasis on – Selecting the study initiator– Selecting the study leader– Planning the study– Roles during the study– Questions vs. follow-up– Completion criteria
(P.S. It also tells how to conduct the study itself :-)
November, 2001 R. Dameron, University of Colorado, ECEN5033, System Test Planning
25
Reasonable Limits for this class• This is a human-intensive activity
• As such, the details on the previous page are of extreme importance – authors are experienced and therefore recognize this
• You won’t be able to conduct a HAZOP study on the basis of these slides
• Goal: Understand what it is – set the bar higher
November, 2001 R. Dameron, University of Colorado, ECEN5033, System Test Planning
26
Study process itself in a nutshellIntroductions
Presentation of design notation
Examine design methodically one unit at a timeIs it possible to deviate from design intent here?
Examine both consequences and causes of the possible deviation
YES
NO
Document resultsDefine follow-up work
Time up? Agree on documentationSign off
YES
NO
November, 2001 R. Dameron, University of Colorado, ECEN5033, System Test Planning
27
Examine design methodically each unit in turn
• Suppose the design representation is a collection of state transition tables:
• Units are states, transitions, event/action pairs
• For EACH, list the recommended attributes (see table from the Hazop book)
• For each attribute, use the guide words to trigger the questions about ways to deviate
November, 2001 R. Dameron, University of Colorado, ECEN5033, System Test Planning
28
The suggested guide words– No: negation of design intention; no part of design
intention is achieved but nothing else happens
– More: Quantitative increase
– Less: Quantitative decrease
– As well as: Qualitative increase where all design intention is achieved plus additional activity
– Part of: Qualitative decrease where only part of the design intention is achieved
– Reverse: logical opposite of the intention
– Other than: complete substituion, where no part of the original intention is achieved but something quite different happens
November, 2001 R. Dameron, University of Colorado, ECEN5033, System Test Planning
29
When timing matters• Add the following guide words:
– Early: something happens earlier in time than intended
– Late: something happens later in time than intended
– Before: something happens earlier in a sequence than intended
– After: something happens later in a sequence than intended
November, 2001 R. Dameron, University of Colorado, ECEN5033, System Test Planning
30
Guide words chosen• Match the system being examined to
appropriate table or modify the closest• Match the design representation• Note: not all guide words apply to all attributes
– For attribute “speed” of an electric motor, omit guide word “as well as” and “part of”
– For attribute “data flow” on a dfd, “less” is not used because meaning covered by “part of”
• Generally, study leader selects from the guide words, provides interpretations based on chosen design representation and context, distributes to team in advance of the study
November, 2001 R. Dameron, University of Colorado, ECEN5033, System Test Planning
31
Applications• Originally developed for chemical plants
• Book has detailed examples for– Software using data flow diagrams– Software using state transition diagrams
• Includes timing attributes of response time and repetition time
– Software using various OO models– Digital electronics– Communication systems– Electromechanical systems
• Same guide words, different interpretations
November, 2001 R. Dameron, University of Colorado, ECEN5033, System Test Planning
32
See book excerpts
• More detailed outline of the HAZOP process – Figure 9.2
– For all entities
• For all attributes
–For each guide word
»Is deviation credible?
• Example matrices
November, 2001 R. Dameron, University of Colorado, ECEN5033, System Test Planning
33
Fig 9.2
HAZOP meeting process