1 Safety Assessment February 2006. 2 SAFETY ASSESSMENT A Safety Assessment is essentially a process for finding answers to three fundamental questions:

1

Safety Assessment

February 2006

2

SAFETY ASSESSMENTSAFETY ASSESSMENT

A Safety Assessment is essentially a process for finding answers to three fundamental questions:

What could go wrong? What would be the consequences? How often is it likely to occur?

Once we know the answers this automatically raises the next question:

Is this acceptable? What can we do if not?

3


Consequently, the objective of Safety Assessments is to:

ensure that the system operates normally and without exposing unacceptable risks to anyone;

reduce and prevent incidents and accidents and;

limit the consequences of any occurrence that might occur.

The Scope of the Safety Assessments includes: Safety Assessment on Air Navigation Systems

covering people, procedures and equipment; … does not address Air Navigation System

“certification” issues; … does not address organisational and

management aspects related to safety assessment.

4


Safety A condition in which the risk of harm or

damages is limited to an acceptable level

Risk The probable rate of occurrence of a hazard

causing harm and the degree of severity of the harm

Risk = Severity * likelihood

Need to define severity and likelihood Need to define acceptability

5

SEVERITY CLASSIFICATIONSEVERITY CLASSIFICATION

Severity Classification Scheme

1 Accident One or more catastrophic accident One or more mid-air collision One of more collisions on ground between two aircraft No independent source of recovery mechanism, such as surveillance or ATC / Flight Crew procedure, can

reasonably be expected to prevent the accident(s)

2 Serious Incident large reduction in separation (e.g. a separation of less than half the separation minima), without crew or ATC fully controlling the situation or able to recover from the situation.

one or more aircraft deviating from their intended clearance, so that abrupt manoeuvre is required to avoid collision with another aircraft or with terrain (or when an avoidance action would be appropriate).

3 Major Incident large reduction in separation (e.g. a separation of less than half the separation minima), without crew or ATC fully controlling the situation or able to recover from the situation.

Minor reduction in separation (e.g. a separation of more than half the separation minima), without crew or ATC fully controlling the situation, or able to recover from the situation, jeopardising the ability to recover without use of collision or terrain avoidance manoeuvres

4 Significant Incident Increased workload on ATCO or Flight Crew or slightly degrading capability of the CSN system Minor reduction in separation (e.g. a separation of more than half the separation minima), without crew or

ATC fully controlling the situation, or able to recover from the situation and fully able to recover the situation

5 No immediate effect on safety

No immediate direct or indirect impact on operations

6

LIKELIHOOD CLASSIFICATIONLIKELIHOOD CLASSIFICATION

Likelihood Classification Scheme

1 Frequently Likely to occur frequently (often)

2 Probable Likely to occur several times during the life-time of the system (2-5 occurrences per year)

3 Occasional Occurs sometimes during the life-time of the system (1 occurrence per year)

4 Remote Unlikely to occur sometimes during the life-time of the system (1 occurrence per 5 years)

5 Improbable Very unlikely to occur (1 occurrence per 20 years)

6 Extremely Improbable Extremely unlikely to occur (1 occurrence per 100 years)

7

RISK CLASSIFICATIONRISK CLASSIFICATION

Risk Classification

Probability Severity

Probability Qualitative Definition Quantitative

Definition 1 2 3 4 5

Frequently Likely to occur frequently. > 5*10-4 A A A A C

Probable Likely to occur several times during system life.

< 5*10-4 A A A B D

Occasional Occurs sometime during system life. < 1*10-5 A A B C D

Remote Unlikely to occur sometimes during system life.

< 1*10-6 A B C D D

Improbable Very unlikely to occur. < 1*10-7 B C D D D

Extremely Improbable

Extremely unlikely to occur. < 1*10-8 C D D D D

Likelihood

Likelihood

8

AS LOW AS REASONABLE PRACTICABLEAS LOW AS REASONABLE PRACTICABLE

The risk is less than the pre-determined unacceptable limit,

The risk has been reduced to a level which is as low as reasonable practicable (ALARP) and

The benefits of the proposed system or changes are sufficient to justify accepting the risk

All three of the above criteria should be satisfied before a risk is classed as tolerable

9


ICAO SEVEN STEP APPROACH Hazard Identification and Estimation steps

Step 1 – System and Environment Description Step 2 – Hazard Identification Step 3 – Hazard Severity Step 4 – Hazard Likelihood

Mitigation steps Step 5 – Risk Evaluation Step 6 – Risk Mitigation

Documentation Step 7 – Safety Assessment Documentation

10

STEP 1 - DESCRIPTIONSTEP 1 - DESCRIPTION

Before a safety assessment can be performed, we need to describe the ATM system being assessed. For that purpose we need (as a minimum):

System Description;

Operational Environment Description.

11


A detailed system description should include:

the purpose of the system; how the system will be used; a description of system functions; the system boundaries and the external interfaces; where appropriate, the transition procedures from the previous

system to the new system, including any hazards associated with the decommissioning of the previous system;

description of contingency procedures and other procedures for non-normal operations;

other input such as other safety assessment results, occurrence and investigation reports, lessons learnt etc.;

regulatory framework and applicable standards.

12


A detailed operational environment description should include:

traffic characteristics; weather characteristics & weather-related factors (e.g. average

frequency of diversions due to severe weather); topography; aircraft performance and equipment; infrastructure modes and limitations including e.g. runway in use,

closed taxiways etc; environmental constraints; characteristics of the users of the system; adjacent centre capabilities; …and other input concerning the environment in which the system

is to be operated.

13

HAZARD IDENTIFICATION AND ESTIMATION PROCESS

hazard

hazard

hazardhazard

hazardhazard

Brainstorming – Hazard Identification

hazard

hazard

hazardhazard

hazardhazard

hazard

hazard

hazardhazard

hazardhazard

hazard

hazard

hazardhazard

hazardhazard

Brainstorming – Hazard Identification

hazard1 can lead to?hazard2 can lead to?hazard3 can lead to?

--

hazard1 can lead to?hazard2 can lead to?hazard3 can lead to?

--

Identification of Hazard Consequences

1. Introduction

2. Methodology

3. Operational Environment

4. Scenario

5. Classification Schemes

6. Example

Briefings what are the potentialwhat are the potentialconsequences?consequences?

what can go wrong?what can go wrong?

Catastrophic ?Major Incident ?

Negligible?--

Catastrophic ?Major Incident ?

Negligible?--

Identification of Severities

How severe can it become?How severe can it become?

Frequently ?Occasionally ?

Negligible ?--

How often can it occur?How often can it occur?

List of 10 most safety-critical hazardsIdentification of Likelihood of Occurrence

1

2 3

4

6

5

14

STEP 2 – HAZARD IDENTIFICATIONSTEP 2 – HAZARD IDENTIFICATION

Purpose

…to identify what could go wrong!(- or anticipate problems before they occur…)

….to identify the consequences (on safety) of the hazards

A hazard is defined as any condition, event or

circumstances which could induce an accident

or incident (ICAO DOC 9422)

The equipment (hardware and software);

The operating environment; The human operators; The human machine interface (HMI); Operational procedures; Maintenance procedures; External services.

15


…to identify the consequences of the hazards on operation!

A hazard consequence is defined as the potential effects on operation that a hazard may create

The operational consequences list the effects the hazard will have on the operation and emphasise the impact / changes the hazard will introduce compared with “normal operation”.

The safety consequences are derived from the operational consequences by deciding the impact on the safe provision of ATS. E.g. potential loss of separation.

- increased receive/transmit- increased co-ordination

- increased receive/transmit- increased co-ordination

- potential loss of separation- potential loss of separation

16

17

18


The hazard identification step should consider all the possible sources of system failure. Depending on the nature and size of the system under consideration these could include:

The equipment (hardware and software); The operating environment (including physical

conditions, airspace and air route design); The human operators; The human machine interface (HMI); Operational procedures; Maintenance procedures; External services.

19


Methodologies

Brainstorming;

Vision Conferences;

Historical Records of Incidents;

Checklists;

Other systematic methods.

20


Preferred Methodology

Brainstorming because: Easy and straightforward process. No need to

complicate or make too academic! Such group sessions are usually good at

generating ideas and identifying issues – mutual inspiration;

The interactions between participants with varying experience and knowledge tend to lead to broader, more comprehensive and more balanced consideration of safety issues.

21


WHAT IF?MODERATOR

ATCO

SYSTEMEXPERT

SAFETYEXPERT

Brainstorming Process

interactive session facilitated by a moderator experts encouraged to bring

forward any safety-related issue they can think of

based upon pre-developed scenarios

first step: identify hazards second step: identify

consequences of the hazards

22


Participants participants should be chosen for their expertise in fields

relevant to the project being assessed.

Such experts usually include System users/operational experts: ATCOs and Flight

Crew (where necessary), to assess the consequences of hazard(s) from an operational perspective;

System technical experts, to explain the system purpose, interfaces and functions;

Safety and human factors experts, to guide in the application of the FHA methodology itself and to bring wider experience of the consequences of hazards.

23


EXAMPLE

24

STEP 3 – SEVERITY ASSESSMENT

The severity expresses the impact on operation or the harm an individual may suffer.

Severity Classification is a gradation, ranging from "worst case/accident" to "no safety impact" – expressing the magnitude of the consequence of the hazard.

Thus, a severity is allocated each hazard consequence in accordance with the agreed severity classification scheme.

25

STEP 3 – SEVERITY ASSESSMENT

Severity Classification Scheme

1 Accident One or more catastrophic accident One or more mid-air collision One of more collisions on ground between two aircraft No independent source of recovery mechanism, such as surveillance or ATC / Flight Crew procedure, can

reasonably be expected to prevent the accident(s)

2 Serious Incident large reduction in separation (e.g. a separation of less than half the separation minima), without crew or ATC fully controlling the situation or able to recover from the situation.

one or more aircraft deviating from their intended clearance, so that abrupt manoeuvre is required to avoid collision with another aircraft or with terrain (or when an avoidance action would be appropriate).

3 Major Incident large reduction in separation (e.g. a separation of less than half the separation minima), without crew or ATC fully controlling the situation or able to recover from the situation.

Minor reduction in separation (e.g. a separation of more than half the separation minima), without crew or ATC fully controlling the situation, or able to recover from the situation, jeopardising the ability to recover without use of collision or terrain avoidance manoeuvres

4 Significant Incident Increased workload on ATCO or Flight Crew or slightly degrading capability of the CSN system Minor reduction in separation (e.g. a separation of more than half the separation minima), without crew or

ATC fully controlling the situation, or able to recover from the situation and fully able to recover the situation

5 No immediate effect on safety

No immediate direct or indirect impact on operations

26

STEP 4 – LIKELIHOOD ASSESSMENT

The likelihood of occurrence expresses how often the consequence of a hazard is likely to occur.

Likelihood Classification is a gradation, ranging from "frequently" to “extremely improbable".

Thus, a likelihood is allocated each hazard consequence in accordance with the agreed likelihood classification scheme.

27

STEP 4 – LIKELIHOOD ASSESSMENT

Likelihood Classification Scheme

1 Frequently Likely to occur frequently (often)

2 Probable Likely to occur several times during the life-time of the system (2-5 occurrences per year)

3 Occasional Occurs sometimes during the life-time of the system (1 occurrence per year)

4 Remote Unlikely to occur sometimes during the life-time of the system (1 occurrence per 5 years)

5 Improbable Very unlikely to occur (1 occurrence per 20 years)

6 Extremely Improbable Extremely unlikely to occur (1 occurrence per 100 years)

28

STEP 3 & 4 – SEVERITY AND LIKELIHOODSTEP 3 & 4 – SEVERITY AND LIKELIHOOD

EXAMPLE

29

STEP 5 & 6 – RISK EVALUATION AND MITIGATION

Is this risk acceptable?

We have a risk

with a defined likelihood

and severity

Acceptablerisks

No

Yes

Notacceptable

risks

One of the causes

training of

Discussion of causes and failures

What are the potential causes

could be insufficientThis consequence

prevented if

How can we resolve it?

Discussion of Risk Mitigation

could be reduced or

Risk Mitigation Plan

Mitigation willremove risk

Mitigation willnot remove risk

Residualrisk

acceptable?

Riskmitigation

impracticable?

Mitigation impracticable

Openrisks

Discussion of acceptability

30

STEP 5 – RISK EVALUATION

Determine what is / is not acceptable Acceptable level of Safety

Determine acceptability of identified risks Clearly unacceptable Clearly acceptable May be / may be not acceptable

Risk Classification






< 5*10-4 A A A B D



< 1*10-6 A B C D D




likelihood

likelihood

31

STEP 5 – RISK EVALUATION

Performed by a small group System users/operational experts: ATCOs and Flight Crew

(where necessary), to assess the consequences of hazard(s) from an operational perspective;

System technical experts, to explain the system purpose, interfaces and functions;

Safety and human factors experts, to guide in the application of the FHA methodology itself and to bring wider experience of the consequences of hazards.

May need to be extended with specialists in areas relevant for the ALARP assessment

32

STEP 5 – RISK EVALUATIONSTEP 5 – RISK EVALUATION

EXAMPLE

33

STEP 6 – RISK MITIGATIONSTEP 6 – RISK MITIGATION

Identify potential causes for a risk to occur Some causes are identified during the hazard

identification Ensure that we have identified all causes

Identify potential mitigation Remove the risk (remove the cause of the risk) Reduce the risk

Reduce severity and/or probability

Identify preferred mitigation approach

34

Risk Classification






< 5*10-4 A A A B D



< 1*10-6 A B C D D




likelihood

likelihood


35


Risk mitigation should be sought in any of the three components of a system:

People Procedures Equipment

The possible approaches to risk mitigation include:

revision of the system (or airport) design; modification of operational procedures; changes to staffing arrangements; and training of personnel to deal with the hazard.

36


To identify causes a number of techniques may be required

Brainstorming sessions Fault tree analysis - Effect tree analysis Common cause failure identification (Single point

failure) Task, Fail-Safe & Error Tolerance Analysis Failure Mode and Criticality Analysis Reliability, Availability and Maintainability Analysis

Focus on components giving: Highest likelihood Highest degree of severity

37


Performed by a small group System users/operational experts System technical experts Safety and human factors experts

Different experts may be required to: Performed detailed studies of the causes of a risk

Study system design to determine component potentially causing, e.g. loss of air situation display

Study procedures to determine where e.g. misunderstandings can arise

Ways to remove those causes

38


SW

Hazard

S

F

S

S

F

F

Effect 1

Effect 2

Effect 3

Effect 4

P=Likelihood

E = Severity

PR

P=Likelihood

Failure Recovery

Fault Tree and Effect Tree Analysis

39


Procedure Assurance Level Procedure development effort should be proportional to the

potential Risk associated with the Procedure. To achieve this, objective PAL should be determined and satisfied.

PAL is setting some objectives to be met during the different phases of the procedure life cycle – Table 1.

PAL objectives are applicable to the entire Procedure, not only to some part of it.

40


Level Definition Design and validation

Implementation Transfer in operations

Operations

3 Other/own experience benchmarking

Specification quality assurance

Fast time simulation Qualitative risk

assessment Pre-implementation

trials

Dedicated training Staff acceptance

argumentation Quality assurance of

implementation

Competency argument for the staff to perform transfer

Contingency plan

Regular proficiency checks

4 Other/own experience benchmarking

Specification quality assurance

Fast time simulation Qualitative risk

assessment Pre-implementation

trials

Quality assurance of implementation

Contingency plan Regular proficiency checks

Procedure Assurance Level

41


Software Assurance Level Software development effort should be proportional to the

potential Risk associated with the Software. To achieve this, objective SWAL should be determined and satisfied.

SWAL is setting some objectives to be met during the different phases of the software life cycle.

SWAL objectives are applicable to the software component is question (only some part of of the total software).

42


LevelRequirement

1 2 3 4

37.3 Unit, integration and system testing

37.3.1 Unit and integration tests shall be conducted on individual units and on partially integrated units to demonstrate that the software is executable and that it produces the expected results for the specified test cases.

M M M M

37.3.3 Integration tests shall as a minimum demonstrate the correctness of all interfaces.

J1 J2 M M

M Mandatory requirement to the development processJ1 Justification is to be provided if the clause or part of the clause is not followedJ2 Justification for the omission or non-compliance is to be provided

Extract from DEF-STAN-55

43


Mitigation actions (safety requirements) should be carefully analysed:

Will the mitigation remove the risk or reduce the risk (what will be remaining risk be)

Will the implementation introduce any new hazards (repeat step 3, 4 and 5)

Mitigation actions shall be documented Risk Mitigation Plan

44


EXAMPLE

45

STEP 7 - SAFETY ASSESSMENT DOCUMENTATION

The purpose: To provide a permanent record of the final result of

the safety assessment To provide the arguments and evidence

demonstrating that the risks associated with the implementation of the proposed system or change:

have been eliminated, or have been adequately controlled and reduced to a

tolerable level.

46

STEP 7 - SAFETY ASSESSMENT DOCUMENTATION

Should contain a summary of: Methods used Safety criteria (the agreed safety levels) Results of the hazard identification process (including Hazard Logs) Risk mitigation required (safety requirements) Follow-up actions Evidence of compliance with safety requirements

References should be included Evidence of validity of assumptions

47

DIFFICULTIES – DIFFICULTIES – SAFETY ASSESSMENTSAFETY ASSESSMENT

General Complex, resource-demanding activity

Target Levels of Safety (Severity and Likelihood) Complexity No guidelines or recommendation – in most cases not even

statistics No guidelines to apportioning Safety Targets to lower levels No guidelines to who does what (Regulator Provider

Supplier)

48

DIFFICULTIES – DIFFICULTIES – SAFETY ASSESSMENTSAFETY ASSESSMENT

Risk Mitigation Very demanding concepts (software assurance

levels, procedure assurance levels) Very demanding activities for risk mitigation Analyses required beyond reach for many

organisation

49

RECOMMENDATIONSRECOMMENDATIONS

Start with low level of ambition Even simple Safety Assessment provides quite

efficient risk mitigation Introduce more advanced features once the simple

version works Start with quantitative likelihood classification while

data are collected to establish qualitative figures Make sure assumptions are well-defined

and traced

50

RECOMMENDATIONSRECOMMENDATIONS

Don’t forget to design a follow-up system for (ICAO 2.26.5)

Hazards (likelihood for different causes) Assumptions, e.g.:

Capacity figures Reliability figures

Should be extracted from the reporting system

51

SUPPORTING SLIDESSUPPORTING SLIDES

52

Target Level of SafetyTarget Level of Safety

MET NAV/Enr NAV/Term

Ground TWR APP ACC

Safety factor for Accidents (1,55 10-8 per Flight hour)

Mid-air collision ÷

Controlled flight into terrain

÷

Accident on ground with fatalities

÷ ÷ ÷

……

Safety Factors for Serious Incidents

Separation minima infringement (less than

50%)

÷

Runway incursion with avoiding action

÷ ÷ ÷

……

1 Safety Assessment February 2006. 2 SAFETY ASSESSMENT A Safety Assessment is essentially a process for finding answers to three fundamental questions:

Documents

description of system

safety assessment safety

detailed system description

system boundaries

proposed system

new system

atm system

safety assessment icao