(ASMM) and Assessment ATM Safety Maturity Model 1 APAC Your Service Provider for Success ATM Safety Maturity Model (ASMM) and Assessment EUROCONTROL Safety R&D Seminar Barcelona 2006
Post on 11-Mar-2018
216 Views
Preview:
Transcript
APA-PHO-AP0A7G-1.0b-0 1
APAC Your Service Provider for Success
ATM Safety Maturity Model (ASMM) and Assessment
EUROCONTROL Safety R&D SeminarBarcelona 2006
ATM Safety Maturity Model (ASMM) and Assessment
EUROCONTROL Safety R&D SeminarBarcelona 2006
Susanne.Lanzerstorfer@apac.at
Hans.Scherzer@apac.at
www.apac.at
APA-PHO-AP0A7G-1.0b-0 2
APAC Your Service Provider for Success
ExperienceExperience• EAD • SES activities
– SESIS– SESFARR– Recognised Organisation
• Safety assessment activities– Assessment of SAM as an AMC with ESARR 4– EAD– OATMS
• Definition of safety management systems– Austro Control– Slovenia Control
• Studies/Research– ASMM– Knowledge management database
APA-PHO-AP0A7G-1.0b-0 3
APAC Your Service Provider for Success
SES ActivitiesSES Activities• SESIS
– Supporting the development of guidance material for• ANSPs for the implementation • NSAs – certification
• SESFARR– Supporting the development of a questionnaire to
assess the status of the implementation of the SES regulations (EU member states and associated states)
– Supporting the development of the report• Recognised Organisation
– APAC was recognised according to Art. 3 Regulation (EEC) No.550/2004 of March 2004 by the Austrian National Supervisory Authority
APA-PHO-AP0A7G-1.0b-0 4
APAC Your Service Provider for Success
End of 2004End of 2004
• Common Requirements ante portas
• ESARRs
• A large number of safety-related standards
• Dozens of GUIs, partially in draft status
• SWAL seems to be something esoteric for suppliers
• ...
APA-PHO-AP0A7G-1.0b-0 5
APAC Your Service Provider for Success
SAFETY (ATM) Maturity Model BasisSAFETY (ATM) Maturity Model Basis
R
E
Q
U
I
R
E
M
E
N
T
S
European CommissionRegulation
No. 550/2004
Common Requirements
EurocontrolESARR 5
Safety Regulatory Requirement for ATM Services' Personnel
ISO 9001
EurocontrolESARR 3
Use of Safety Management
Systems by ATM Service Providers
EurocontrolEATMP
Safety Policy
Eurocontrol & European
CommissionESARR 1
Safety Oversight in ATM
National Supervisory Authority
Air Navigation Service Provider
Supplier and/or ANSP
EurocontrolESARR 2
Reporting and Assessment of
Safety Occurrences in ATM
EurocontrolESARR 7
ATM Procedures(RVSM, GNSS, Data Link, etc.)
EurocontrolESARR 6
Software in ATM Systems
EurocontrolESARR 4
Risk Assessment and Mitigation in ATM
EurocontrolESARR 6
Software in ATM Systems
Guidance Material
EurocontrolSafety
Assessment Methodology
(SAM)and Guidance
Material
EN 61508Functional Safety of electrical/electronic/
programmable electronic safety-related systems
ApplicableICAO SARPs
(Standards and Recommended
Practices)
FullCompliance Determine
Compliance
Scope of the Safety
Capability Maturity Model
Legend:ATM
Extension Capability Maturity Model
System SCMM
Questionnaire
ATM Extension
Questionnaire
ESARR Technical System
Development Questionnaire
EN 61508Questionnaire
Technical System/Service Development
Level
Responsibility
Air Traffic Management Service Provision Level
Supervision Level
EUROCAE ED78A,
(Guidelines for approval of the
provision and use of Air
Traffic Services supported by data communications)
Approved Means of
Compliance
Determine Compliance
Means of Compliance to be taken into Account
APA-PHO-AP0A7G-1.0b-0 6
APAC Your Service Provider for Success
The GoalThe Goal
• Establish clear pass/fail criteria to reach safety objectives
• Repeat ISO9001/Bootstrap (BICO) success by added value to a pass/fail assessment
• Benchmarking of SMS
• Target environments– ANSPs (results re-usable for NSA certification)
– ATM system supplier
APA-PHO-AP0A7G-1.0b-0 7
APAC Your Service Provider for Success
AT
M C
MM
Mod
el
Dev
elop
men
t
Res
earc
h on
ex
istin
g m
ate
rial
Dev
elop
men
t of C
ore
Saf
ety
Cap
abili
ty M
odel This phase includes the
definition of the best practices
1
3
5
7
9
11
13
Months
I
II
III
IV
V
Phase
Dev
elop
men
t of t
he
Sys
tem
SC
MM
Q
ues
tionn
aire
Dev
. of t
he
ES
AR
R
Sys
tem
Dev
elop
men
t Q
ues
tionn
aire
Dev
elop
men
t of t
he
EN
615
08
Qu
estio
nnai
re
AT
M E
xten
sion
Q
uest
ion
naire
Test / Validation
Update
Final Delivery
Update Update Update
Model Development PlanModel Development Plan
Full Common Requirements Scope
APA-PHO-AP0A7G-1.0b-0 8
APAC Your Service Provider for Success
Cooperation with UniversityCooperation with University• Evaluation of available maturity models –
ISO15504 (SPICE) selected as basis
• Combination of ATM/safety requirements with ISO15504
• Draft questionnaires for all areas
• Validation in ANSP environment only partially achieved
• Validation in industrial environment of the ISO15504/EN61508 model
APA-PHO-AP0A7G-1.0b-0 9
APAC Your Service Provider for Success
2005 Target Environment Response2005 Target Environment Response
• ANSPs – ANSP - do not seem to be too interested in numbers and seem to consider benchmarking a threat rather than a help.
• ATM suppliers and automotive suppliers are interested, but– We have to speak a language they can
understand = EN61508
– Focus on projects (system aspects: ISO9001)
– Separation of development and safety maturity is not acceptable (development and safety aspects have to be covered in one assessment)
APA-PHO-AP0A7G-1.0b-0 10
APAC Your Service Provider for Success
The Supplier Safety Maturity Model
The Supplier Safety Maturity Model
• EN 61508 is not a development standard and project oriented but very successfully applied and widely accepted in industry
• Combination of ISO15504 (SPICE) with EN 61508 leads to a sound and robust model capable to become an AMC for SWAL requirements
• Both are well established standards• ISO15504 provides a reference system for
benchmarking and a well defined algorithm for scoring
APA-PHO-AP0A7G-1.0b-0 11
APAC Your Service Provider for Success
The Assessment BasisThe Assessment Basis
• ISO15504 (SPICE)
• EN 61508
• Allow uncertainties to keep the assessment effort acceptable –mitigation of resulting risks by application of common sense
APA-PHO-AP0A7G-1.0b-0 12
APAC Your Service Provider for Success
Capability LevelCapability Level
Planned-and-Tracked Level
Informally-performed Level
Well-defined Level
Quantitatively-controlledLevel
• Performing the process
• Planning performance • Disciplined performance• Verifying performance• Tracking performance
• Defining a standard process
• Performing the defined process
• Establishing measurable quality goals
• Objectively managing performance
Continuously-ImprovingLevel
• Improving organisational capability
• Improving process effectiveness
APA-PHO-AP0A7G-1.0b-0 13
APAC Your Service Provider for Success
Architecture of the QuestionnaireArchitecture of the Questionnaire
12 G
EN
ER
IC
PR
AC
TIC
ES
5 GE
NE
RIC
P
RA
CT
ICE
S
3 GE
NE
RIC
P
RA
CT
ICE
S
5 GE
NE
RIC
P
RA
CT
ICE
S
............
APA-PHO-AP0A7G-1.0b-0 14
APAC Your Service Provider for Success
Example Engineering Level 1Example Engineering Level 1
Engineering Processes
QUESTION REMARKS FOR SCORING SPICE SIL 2 Methods&Techniques 61508
Level 11.1: Perform Processes
2.1 Develop software requirements
Develop software requirements:
Determine software requirements; analyse software requirements;
R: Computer-aided specification tools; Tools without preference for one particular design method;
Establish, analyse and refine the software requirements.
determine operating environment impact; evaluate requirements with customer; update requirements for next iteration
R: Describe some critical parts with semi-formal methods e.g.: Logic-Function Block Diagrams, Sequence Diagrams Dataflow Diagrams, Finite State Machine/State Transition Diagrams, Time Petri Nets, Decision Truth TableR: Formal Methods including for example, CCS (Calculus of Communicating Systems), CSP(Communicating Sequential Processing), HOL, LOTOS, OBJ, temporal logic VDM and Z
APA-PHO-AP0A7G-1.0b-0 15
APAC Your Service Provider for Success
Example Engineering Level 2Example Engineering Level 2Level 2
2.1: Planning Performance2.10 Allocate
resourcesAllocate adequate resources (including people) for performing the process category "engineering".
Evidence of resource allocation exists; records/plan indicate resources are allocated to perform job tasks
2.11 Assign responsibilities
Assign responsibilities for developing the work products and/or providing the services of the process category "engineering".
Assigned responsibilities are recorded; representative understands the process and tasks he is responsible for
2.12 Document the process
Document the approach to performing the process category "engineering" in standards and/or procedures.
Tasks to be performed; inputs and outputs; entry/exit criteria; control points; internal and external interfaces; process measurements
2.13 Provide tools Provide appropriate tools to support performance of the process category "engineering".
Adequate training in the operation of the tool; documentation and/or instructions are available for the tool; support for the tool is available
2.14 Ensure training Ensure that the individuals performing the process category "engineering" are appropriately trained in how to perform the processes.
Training is available for tools;training curriculum covers all tasks;resources are allocated for training
2.15 Plan the process Plan the performance of the process category "engineering".
WBS; project standards; special needs; reuse strategy; resource estimation; risks; schedule
APA-PHO-AP0A7G-1.0b-0 17
APAC Your Service Provider for Success
Rating SchemeRating Scheme
• The rating is represented as follows:N...Not Adequate 0 <= x < 16
P...Partially Adequate 16 >= x < 51
L...Largely Adequate 51 >= x < 86
F...Fully Adequate 86 >= x <= 100
• If a question is not applicable, it is not taken into account for scoring.
APA-PHO-AP0A7G-1.0b-0 18
APAC Your Service Provider for Success
Ratings for the LevelsRatings for the Levels
• 1st Step: all best practices for Level 1 will be evaluated with N, L, P and F.
• 2nd Step: calculation of the evaluation average
• 3rd Step: result will be mapped to the interval of N,L,P and F. You received the values for Level 1.
• The same process will be used for the Generic Practices of Level 2, 3, 4 and 5.
APA-PHO-AP0A7G-1.0b-0 19
APAC Your Service Provider for Success
Example of a Derived RatingExample of a Derived Rating
Leve
l 1
Best Practices Scoring3.1 0.85993.2 0.85993.3 0.85993.5 0.85993.9 1
Results per LevelLevel 1 0.8879Level 2 0.8599Level 3 0.7899Level 4Level 5
Sum Levels 2.75
APA-PHO-AP0A7G-1.0b-0 20
APAC Your Service Provider for Success
Example ResultsExample ResultsAssessments Result Example, October 2005 and October 2006
0,00
0,50
1,00
1,50
2,00
2,50
3,00
3,50
4,00
4,50
5,00
SIL 2
Oct. 05
Oct. 06
APA-PHO-AP0A7G-1.0b-0 21
APAC Your Service Provider for Success
Description Audit Ref ActionProject
Recording reasons of decisions 1) Record reasons of decisions, requirements/(sepcification and design process
Independ safety manager 2) Define an independ safety manager :- Update Quality manual.- List tasks
V&V responsible 3)4.2) - 1.85
company or project level- Update Quality manual- List tasks- Update default PMP/PQP
Indenpendence between tester and developer 4) - Update Quality manuel- Update default PMP/PQP
Configuration management 12) - Update CVS procedure, and a $Name tag- Update source code tempalte
Working environment 4.3) - 1.5 Define of a reuse strategy and processSafety
Quality manual to adress safetyInitiation of a safety-life cycle: policy, persons, activities, documentation, phases
5)1.2) - 0.153.1) - 1.854.1) - 1.854.2 ) - 1.85
Safety management procedures to be integrated in PQP and QS : Defined strategy, vision and culture (management goals) and communication.Identify person, department and organisation if charge of independent safety activities- List tasks- Update Quality manual
Implement safety management during development
1.4) - 0.151.5) - 1.85
- Update SDP (define safety specific analysis of hazardous incidents and operations and maintenance performance, safety validation procedures, periodical functional safety auditsSDP : safety requirements, identification of safety-related functions, clear interface to non-safety-related functions
Safety management after delivery 1.6) - 0.15 Definition of procedures for initiatin and approving modifications to the system incl. Responsibilities and documents
Update reporting system 1.7) - 1.75 Define procedure for maintaining accurate information on potential hazards and safety related system
Results as Basis for Process Improvement
Results as Basis for Process Improvement
• Action Plan
• Charts allow easy visualisation of improvement (“management compatible“)
top related