8/6/2002 Safeware Engineering Co rporation 1 The Safety Risk of The Safety Risk of Requirements Requirements Incompleteness Incompleteness Jeffrey Howard Patrick Anderson
Mar 31, 2015
8/6/2002 Safeware Engineering Corporation 1
The Safety Risk of The Safety Risk of Requirements IncompletenessRequirements Incompleteness
Jeffrey Howard
Patrick Anderson
8/6/2002 Safeware Engineering Corporation
2
Requirements IncompletenessRequirements Incompleteness
Many incidents and accidents have been linked to flaws in real-time embedded system software
Software-related errors are most often requirements errors, particularly incompleteness
A specification is incomplete if required behavior is omitted or subject to more than one interpretation
8/6/2002 Safeware Engineering Corporation
3
Completeness CriteriaCompleteness Criteria
Professor Nancy Leveson has compiled over 60 completeness criteria to address this problem, covering:
– Human-Computer Interface– Trigger Events– Robustness– Nondeterminism– Values and Timing– Data Age– Feedback– And More
Validated at JPL and used at Safeware
SpecTRM-RL (SpecTRM Requirements Language) enforces these criteria
8/6/2002 Safeware Engineering Corporation
4
Today’s Example AccidentToday’s Example Accident
The importance of the criteria is easily demonstrated when they are ignored
No one wants their embarrassing stories told in a conference session
Everything you see here is falseEverything you see here is trueThe ElectroShear 2000 Accident
8/6/2002 Safeware Engineering Corporation
5
ElectroShear 2000 SchematicElectroShear 2000 Schematic
8/6/2002 Safeware Engineering Corporation
6
ElectroShear 2000 Shearing PenElectroShear 2000 Shearing Pen
Shearing pen, where shearing is done
Entry and exit gates – Gate position sensors– Gate actuators– Gate locks
Four mechanical arms mounted with electric trimmers
Three sheep detection sensors– Digital camera– Weight plate– Thermal sensor
Trimmer head sensors– Wool sensor– Skin flush-fit
sensor
8/6/2002 Safeware Engineering Corporation
7
Normal OperationNormal Operation
The system begins with entry gate open and exit gate closed
Workers load a sheep and close the entry gate
At least two of the three sheep detection sensors agree on the sheep’s presence
The system shears, adjusting trimmer position using the skin flush-fit sensor
The wool detection sensor is ignored - the software detects its own completion
After shearing, the exit gate opens
Collect wool and repeat
8/6/2002 Safeware Engineering Corporation
8
The AccidentThe Accident
A technician replaced the trimmer blades in a pen, then greased the entry gate
While manually moving the gate, he lowered it to the point of closing it
The system exited standby mode and began a shearing cycle
The technician was caught in the pen and sheared
The system behaved erratically during shearing, and three of the four mechanical arms were damaged
8/6/2002 Safeware Engineering Corporation
9
Technician’s StatementTechnician’s Statement
“My next work order was pen #22. The guys working with it had complained that the entrance gate was moving slowly and making some noise. As long as I was there, I was supposed to replace the trimmer heads. They were overdue. I got there and the guys unloaded the sheep they were putting into the pen. They put the pen into standby, so I lifted the exit gate, disconnected the weight plate, and went in to replace the trimmer heads. After that, I sprayed some grease on the gate tracks and worked it by hand a little to get the grease spread out. The machine just went crazy on me. It was a close shave.”
8/6/2002 Safeware Engineering Corporation
10
The InvestigationThe Investigation
ElectroShear’s documentation jumbled requirements and design
Accident investigators used SpecTRM-RL to explore the system’s behavior
SpecTRM-RL uses text attributes and AND/OR tables to represent software behavior
8/6/2002 Safeware Engineering Corporation
11
SpecTRM-RLSpecTRM-RL
8/6/2002 Safeware Engineering Corporation
12
SpecTRM-RL (2)SpecTRM-RL (2)
8/6/2002 Safeware Engineering Corporation
13
Why did the system leave Standby Mode?Why did the system leave Standby Mode?
Gates do not require frequent maintenance
Maintenance procedures require the gates to stay open during maintenance
Designers didn’t anticipate entrance gate closings during standby mode
Entrance gate closing during standby mode moves the pen into loaded mode
8/6/2002 Safeware Engineering Corporation
14
Shearing Pen Mode LogicShearing Pen Mode Logic
8/6/2002 Safeware Engineering Corporation
15
Criterion: NondeterminismCriterion: Nondeterminism
“The behavior of the state machine should be deterministic (only one possible transition out of a state is applicable at any time.”
Automated tools can check this
8/6/2002 Safeware Engineering Corporation
16
Was the technician a ram?Was the technician a ram?
The system classified the technician as a sheep
Two of the three sensors must agree– Digital Camera– Thermal Sensor– Weight Plate
The camera mistook the human on all fours as a sheep
The software still had obsolete input data queued from the disconnected weight plate
8/6/2002 Safeware Engineering Corporation
17
Weight Plate InputWeight Plate Input
8/6/2002 Safeware Engineering Corporation
18
Criterion: Data AgeCriterion: Data Age
“All inputs used in specifying output events must be properly limited in the time they can be used (data age).”
In SpecTRM-RL, all inputs have an Obsolete value
8/6/2002 Safeware Engineering Corporation
19
Why was the exit gate open?Why was the exit gate open?
If the exit gate is open, the shearing cycle shouldn’t start
During the accident, it was open
No escape for the technician
When the system went into standby mode, exit gate position sensors were ignored
The system came out of standby mode with an incorrect system model
8/6/2002 Safeware Engineering Corporation
20
Exit Gate Position LogicExit Gate Position Logic
8/6/2002 Safeware Engineering Corporation
21
Criterion: State CompletenessCriterion: State Completeness
“The internal software model of the process must be updated to reflect the actual process state at initial startup and after temporary shutdown.”
SpecTRM-RL requires states to have an Unknown state value
8/6/2002 Safeware Engineering Corporation
22
What about the wool sensor?What about the wool sensor?
The wool sensor didn’t detect wool being sheared
That didn’t stop the shearing cycle
System engineers provided a wool sensor to detect the end of shearing
The software keeps track of shearing completion as progress along the planned shearing path
The software ignores the sensor, because it’s easier to detect the end of shearing as running out of planned shearing path
8/6/2002 Safeware Engineering Corporation
23
Criterion: Input Variable CompletenessCriterion: Input Variable Completeness
“All information from the sensors should be used somewhere in the specification.”
SpecTRM-RL has an “Appears In:” attribute to identify orphaned inputs
8/6/2002 Safeware Engineering Corporation
24
Why were the arms flailing?Why were the arms flailing?
Mechanical shearing arm motion became increasingly erratic
By the end of the accident, three of the four arms were damaged by the controller’s commands
The shearing arm fine-adjustment sensor doesn’t handle struggling humans well
The data bus was flooded with commands and telemetry
8/6/2002 Safeware Engineering Corporation
25
Criterion: Environmental CapacityCriterion: Environmental Capacity
“For the largest interval in which both input and output loads are assumed and specified, the absorption rate of the output environment must equal or exceed the input arrival rate.”
SpecTRM-RL’s attributes address timing behavior
8/6/2002 Safeware Engineering Corporation
26
Why couldn’t the operator help?Why couldn’t the operator help?
An operator finally noticed the calamity
The operator issued a stop command to the shearing pen
The shearing pen didn’t stop
The designers didn’t anticipate high communication load
The stop command is just another order on the bus
The operator had no way to know the order was lost
8/6/2002 Safeware Engineering Corporation
27
Criterion: Output FeedbackCriterion: Output Feedback This problem actually
touches on a number of criteria– Inadequate display of
state to operators– Inability to preempt
lower priority tasks– Lack of feedback
For the moment, focus on the lack of feedback to the operators
SpecTRM attributes on outputs make feedback paths easy to check
8/6/2002 Safeware Engineering Corporation
28
Why didn’t the entry gate open?Why didn’t the entry gate open?
When the operators realized the system wouldn’t shut down, they commanded the gate open
It didn’t open
Keeping gates closed during shearing is a safety feature
The command that closes the gate isn’t reversible.
No notice was given to the operator.
8/6/2002 Safeware Engineering Corporation
29
Criterion: ReversibilityCriterion: Reversibility
“Output commands should usually be reversible.”
SpecTRM-RL outputs have an attribute linking to the output that reverses their command
8/6/2002 Safeware Engineering Corporation
30
Investigation FindingsInvestigation Findings
There was no operator error in this accident.
There were no component failures in this accident.
Even the software didn’t “fail.” It met its requirements, such as they were.
The Electroshear 2000 was found to be unsafe.
The culprit cited was the shearing pen control software.
Software problems stemmed from incomplete requirements.
8/6/2002 Safeware Engineering Corporation
31
Completeness Criteria (2)Completeness Criteria (2)
The ElectroShear accident demonstrates several completeness critera– Nondeterminism– Data Age– State Completeness– Input Variable
Completeness– Environmental Capacity– Output Feedback– Reversibility
Consideration of these criteria could have prevented and/or reduced the severity of the accident
8/6/2002 Safeware Engineering Corporation
32
SummarySummary
The example may be fanciful, but the problems illustrated are quite real
The completeness criteria were compiled from decades of research, accident and incident reports, and specification review
SpecTRM-RL builds the criteria into a state of the art, analyzable, and executable requirements language
8/6/2002 Safeware Engineering Corporation 33
DiscussionDiscussion
And/Or Questions
8/6/2002 Safeware Engineering Corporation 34
The EndThe End