SSED Application Example Lessons Learned: 100 Questions That Should Be Asked during Technical Reviews Seminar on Aerospace Mishaps and Lessons Learned.

SSED Application Example

Lessons Learned:

100 Questions That Should Be Asked during Technical Reviews

Seminar on Aerospace Mishaps and Lessons Learned2004 MAPLD Conference

7 September 2004

Paul Cheng

(310) 336-8222

[email protected]

Date

Program

Problem/Outcome

04/90 Hubble A defect in the tool used both in manufacturing and in QA misshaped the mirror

07/92 TSS-1 Deployment mechanism jammed by a bolt added after I&T

09/92 Mars Observer

Oxidizer reacted with braze, jamming regulator and bursting tank during pressurization

08/93 NOAA 13 The battery charger had low dimensional tolerance — shorted out by a screw

10/93 Landsat F Pyrovalve ignited fuel nearby

01/94 Clementine CPU froze due to overload, allowing the thruster to deplete fuel

05/94 MSTI 2 Contact lost, probably due to micro meteoroid/debris impact or charging

12/95 Skipper Solar arrays miswired on drawing — I&T did not ascertain current direction

02/96 TSS-1R Contamination within the tether caused arcing

08/97 Lewis Flawed GN&C design caused tumbling — not saved due to inadequate monitoring

10/97 STEP-4 Damage by launch vibration. Ground test strategy improper

10/98 STEX Solar array too hot, fatiguing solder joints. Analysis used wrong configuration

12/98 MCO Unit mix-up in ground soft ware, coupled with vulnerable navigation scheme, caused trajectory error

01/99 Mars Polar Lander

Requirement error prevented touchdown sensors from being protected against de-ployment shock. Engine shut down premature ly

03/99 WIRE A start-up transient in the pyro electronics controller prematurely ejected the telescope cover

08/01 Simplesat Transmitter arcing

07/02 Contour Plume analysis, based on similarity, misled by typo in an AIAA paper

Unclassified U.S. Government Satellite Failures, 1990–Present Engineering

Mistake

Technology Surprise

X X X

X X

X X X

X X X X X X X

X

X X

X X

Count 14 6Since 1995 9 3

Why Do Satellites Fail?

100 Questions: “Driver’s Ed Movie” for Engineers

• Based on lessons extracted from SSED data:– 79 catastrophic failures – 32 major events (e.g., loss of an instrument)– 21 ground problems (e.g., unit damaged during vibe)– 3 recoveries of “dead” missions

• Examine:– How did the mistake occur?– What prevented its detection?– Why did a flaw bring down the system?

Remember Past Mistakes to Avoid Repetition

Fools say that they learn by experience. I prefer to profit by others' experience. Otto Bismarck

Like Susan Lee did for NEAR!

The Thrust of QuestionsQuestions are grouped in:• Requirements• Heritage and Qualification-by-Similarity• Analysis • Fault Management• Embedded Software and Database• Interface• Parts, Materials, and Manufacturing Process • Testing and Evaluation

For example: Q 3-1 (Analysis): Have all critical analyses been placed under configuration control?

See Lessons: 26 (STEX Failure) and 83 (AC 70/71 Failures)

Hyperlinks explain the context

Q 1-3 (Requirements): Are there lumped/nested requirements?

One requirement, one statement

Systems Requirement stated:

The touchdown sensors shall be sampled at 100 Hz. The sample process shall initiate to keep processor demand constant.

However, sensor data shall not begin until 12 m above the surface.

Mars Polar Lander Failure

This requirement did not flow down to software requirements

Software read stored sensor status; shut down engine

Legs deployed;Unprotected sensors

registered shock

Launch Vehicle X Failure

• A dual-payload launcher was used for a single payload.

• Hardware engineers redlined spec drafted by software engineers to facilitate wiring, and designed harness based on redlines

• Systems engineer failed to verify - viewed mission spec as software document and not subject to configuration control

• Generic test masked problem

ForwardPayload

SFC SFC

Aft P/LBW BW

SFC SFC

I/FI/F

MissionUnique

GenericCore

BW BW

BW = Bridge WireSFC = Squib Firing CircuitI/F = Interface Connection

Generic Configuration

Hard Wired

P/LBW BW

SFC SFC

I/FI/F

P/LBW BW

SFC SFC

I/FI/F

Failed Mission Software

Commanded

• Redlines fell through mission spec’s cracks – S/W and H/W incompatible

Q 8-15 (Testing): Does the system being tested represent the flight configuration?

Representative Questions for Electrical Engineers

• Are units and tolerances specified? – See Mars Climate Orbiter failure* and Huygens

launch pad damage

• Do testing independently confirm development results? – See Hubble mirror aberration*

• Are handover procedures between two sources of control well defined?– See START launch failure

• Does the harness design preclude mismating?– See BP-TD launch failure

*: Report available on klabs.org

Some Questions Specifically for Digital Engineers

• Can a momentary glitch cause a crash (will logic devices improperly reset following a brief undervoltage, for example)?– See Delta 178 and Titan A-20 failures

• How are databases verified?• See Centaur TC-14 failure

• Will unexpected inputs cause the computer to freeze, without a way to autonomously reboot? – See Clementine failure and SPIRIT anomaly*

• Can the fault protection logic be set off too easily (e.g., can phantom sensor readings spoof the fault management system into taking precipitous actions)?– See Ariane 501* and Atlas/Mariner 1 failure

More Items EEs Rarely Think of, but Should:• Ambiguous drawing instructions

• Opposite engineering convention (right- or left- hand coordinates? Positive- or negative- ground?)

• Wiring crossover between two drawings

• Commandability after OBC faults disabled receivers

• Revivability of solar array regulator after battery drain

• Fratricide by pyro devices

• In-rush current welding relays shut

• FOD-caused shorting and arcing

• ...

Using “100 Questions” in Practice

Four problems found:

• Constant-voltage firing circuit may fail (SAFER lesson)• Routing both arming and firing relays to one PLD (WIRE) • If deployed wires touch firing circuits, battery can drain;

power distribution board may overheat (Deep Space 1) • Test circuits are constant-current (not flight-like)

A satellite uses many low-shock deployment devices

– Consisting of spools of tightly wound wires

– Actuated by electrically severing restraining wires

ReleasesSolar array, etc.

ArmingRelay

FiringRelay

LogicControl

Power Supply

In Closing

Petroski’s Law of Design: To engineer is human

Akin's Laws of Spacecraft Design: Space is a completely unforgiving environment. If you screw up the engineering, somebody dies!

For additional interesting quotes, see klabs.org