Using the SMART Method Using the SMART Method to Assess SMS in Aviation to Assess SMS in Aviation Ronald J. Ronald J. Heslegrave Heslegrave , Ph.D. , Ph.D. University Health Network & University Health Network & University of Toronto University of Toronto CASS 2007 CASS 2007 - - COUNTING THE ACCIDENTS COUNTING THE ACCIDENTS YOU DON’T HAVE… YOU DON’T HAVE… May 1, 2007 May 1, 2007 – – Gatineau Gatineau QC QC
76
Embed
Using the SMART Method to Assess SMS in Aviation · Using the SMART Method to Assess SMS in Aviation Ronald J. Heslegrave, ... regard to all aspects of the evaluation. ... Q38 Q39
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Using the SMART Method Using the SMART Method to Assess SMS in Aviationto Assess SMS in Aviation
Ronald J. Ronald J. HeslegraveHeslegrave, Ph.D., Ph.D.University Health Network & University Health Network &
University of TorontoUniversity of Toronto
CASS 2007 CASS 2007 -- COUNTING THE ACCIDENTS COUNTING THE ACCIDENTS YOU DON’T HAVE…YOU DON’T HAVE…
May 1, 2007 May 1, 2007 –– GatineauGatineau QCQC
Successful Implementation of a Safety Management System or Policies Need to:
Develop a Vision for the Organization Develop a Vision for the Organization –– without a without a culture it will failculture it will failDevelop Policies and Procedures designed to Develop Policies and Procedures designed to Support a Safety Culture and the VisionSupport a Safety Culture and the VisionImplement Policies and Procedures through Implement Policies and Procedures through Training, Organizational Strategies, Operational Training, Organizational Strategies, Operational Strategies, Enabling ToolsStrategies, Enabling ToolsDevelop Assessment and Evaluation MechanismsDevelop Assessment and Evaluation MechanismsEvaluate what you have put in placeEvaluate what you have put in place
The Question sometimes is…
How do you do that?How do you do that?
One Answer …AdherenceAdherence
to Recognized Standards or Adopted to Recognized Standards or Adopted Regulations where there is consensus Regulations where there is consensus
that such Standards or Regulations that such Standards or Regulations reflect an appropriate reflect an appropriate Vision and PolicyVision and Policy
for example, ISO standardsfor example, ISO standards
Another Answer …
EvaluationEvaluationof objective measures where most agree of objective measures where most agree that the measures reflect the Vision and that the measures reflect the Vision and
PolicyPolicy
for example, reduced accidents, incidents for example, reduced accidents, incidents or near missesor near misses
Near MissesNear Misses
Or WorseOr Worse
Another Answer …
Accept Accept expert testimony and evaluation that expert testimony and evaluation that
the Vision and Policy have been the Vision and Policy have been achieved in good measureachieved in good measure
for example, Expert External Auditfor example, Expert External Audit
Auditors and Regulators are sometimes Auditors and Regulators are sometimes viewed differently than intendedviewed differently than intended
It depends on your perception?It depends on your perception?
Auditor and Henchmen may be viewed NegativelyAuditor and Henchmen may be viewed Negatively
The Audit Process can be viewed The Audit Process can be viewed as Confrontationalas Confrontational
We should strive for a Reasonable Dialogue with We should strive for a Reasonable Dialogue with Enlightened Views between Auditors and OrganizationsEnlightened Views between Auditors and Organizations
Is there a simple, effective and multidimensional approach to
evaluating SMS in your organization that is value-
added?
Perhaps?
Today I will introduce the SMARTSMARTapproach to safety management
and evaluation?
While this approach has been investigated in the civil Aviation
sector, it should be transferable to other organizations and systems
Evaluates the Understanding ofEvaluates the Understanding ofFundamental System Elements of a Fundamental System Elements of a
Safety Management SystemSafety Management System
Assesses whether the Assesses whether the Safety Management System has Safety Management System has Penetrated the Organization at Penetrated the Organization at
all levelsall levels
Evaluates Supporting Systems:Evaluates Supporting Systems:Training and Measurement, Training and Measurement,
Assessment and Evaluation ToolsAssessment and Evaluation Tools
SMARTSMART method method evaluates key evaluates key elements of safety elements of safety systems or policiessystems or policies
Safety Systems and Policies are Multidimensional -Evaluations Need to be as well
Assess the reasonableness of what is proposedAssess the reasonableness of what is proposedAssess how well it is implemented Assess how well it is implemented Assess whether it is effective Assess whether it is effective Develop a system for continuous quality improvement Develop a system for continuous quality improvement to monitor and promote changeto monitor and promote changeFeedback the evaluation to the Organization to Feedback the evaluation to the Organization to provide continuous quality improvementprovide continuous quality improvement
BASIC EVALUATION APPROACH OF THE SMARTSMART METHOD
Fundamentally at least 3 Criteria or Dimensions:Fundamentally at least 3 Criteria or Dimensions:Understanding of the Policy and ProceduresUnderstanding of the Policy and ProceduresQuality of Implementation of the SystemQuality of Implementation of the SystemPenetrance of Safety Policy and Procedures Penetrance of Safety Policy and Procedures throughout the organizationthroughout the organization
SMART METHOD1)1) Identify the goals of the systemIdentify the goals of the system2)2) Develop a set of tools designed to target the primary Develop a set of tools designed to target the primary
requirements of that systemrequirements of that system3)3) Collect objective and subjective data from relevant Collect objective and subjective data from relevant
stakeholders across the hierarchy of the organizationstakeholders across the hierarchy of the organization4)4) Blindly evaluate and analyze these data in a Blindly evaluate and analyze these data in a
multidimensional contextmultidimensional context5)5) Provide feedback for ongoing quality improvement Provide feedback for ongoing quality improvement
for the organizationfor the organization
Key to the approach is to Quantitatively Assess and
Evaluate the Data collected on a Multidimensional Basis
Can this SMART method be implemented Practically, Easily and Successfully?
You be the Judge?
Recently the SMART method was validated in a major corporation in the Aviation Community with national operations to study the successful implementation of their Safety Management Program and Policies
An Example of the Application of An Example of the Application of the SMART Methodthe SMART Method
SMART APPROACHIdentified Key Policies and Objectives of the Identified Key Policies and Objectives of the Safety Management ProgramSafety Management ProgramDeveloped an Assessment TeamDeveloped an Assessment TeamDeveloped a Survey to Assess Safety Developed a Survey to Assess Safety ManagementManagementImplemented a multidimensional approach to the Implemented a multidimensional approach to the evaluation of written and interview responses to evaluation of written and interview responses to the Survey Toolthe Survey Tool
Expert Team AnalysisAn Audit Team was established with An Audit Team was established with varying expertise including Operational varying expertise including Operational Systems, Safety Management Operations, Systems, Safety Management Operations, Specific Content Experience, and Scientific Specific Content Experience, and Scientific ExpertiseExpertisePrior to assessing the data collected, a Prior to assessing the data collected, a decision was made to “blind” the evaluators decision was made to “blind” the evaluators to each others scoring of responses with to each others scoring of responses with regard to all aspects of the evaluationregard to all aspects of the evaluation
Develop a Survey ToolQuestions were grouped into Common Safety Questions were grouped into Common Safety Dimensions Dimensions Questions were developed along 5 Dimensions Questions were developed along 5 Dimensions representing the Continuum of Safety Management representing the Continuum of Safety Management Systems Systems –– Fundamental Program ElementsFundamental Program Elements–– Implementation of Fundamental Program ElementsImplementation of Fundamental Program Elements–– Training Training –– Assessment and Evaluation Strategies Assessment and Evaluation Strategies –– Measurement SystemsMeasurement Systems
Collect Data from OrganizationSubmit the Survey Tool to the Organization for a Submit the Survey Tool to the Organization for a Formal Written ResponseFormal Written ResponseBased on the Survey, Interview Key Stakeholders Based on the Survey, Interview Key Stakeholders across the Hierarchy of the Organizationacross the Hierarchy of the Organization
Quality of the ResponseQuality of the Response -- How well do the How well do the answers address the questions asked?answers address the questions asked?Policy UnderstandingPolicy Understanding -- How well do the How well do the answers demonstrate an understanding of answers demonstrate an understanding of the Safety Management Policies?the Safety Management Policies?Penetrance Penetrance -- How well does the answer How well does the answer demonstrate the implementation of the demonstrate the implementation of the Safety Management Policies throughout Safety Management Policies throughout the organization?the organization?
Develop a Multidimensional Evaluation StrategyDevelop a Multidimensional Evaluation Strategy
Evaluation ProcessEach Audit Team Member Independently Each Audit Team Member Independently and Blindly assessed each written and and Blindly assessed each written and interview response interview response
All ratings were made on a 5All ratings were made on a 5--point point LikertLikertScale:Scale:–– 5= Good and Full Response5= Good and Full Response–– 3= Reasonable Response3= Reasonable Response–– 1= Inadequate or Low Response1= Inadequate or Low Response
Organizational FeedbackResults are presented in the following fashion:Results are presented in the following fashion:–– Overall ratings of Quality, Policy, and Overall ratings of Quality, Policy, and PenetrancePenetrance
Dimensions Dimensions –– Ratings on a question by question basis to Ratings on a question by question basis to
identify areas of strength and opportunity for identify areas of strength and opportunity for improvement in the Organization improvement in the Organization
–– Summarized data by common dimensions based Summarized data by common dimensions based on the questionson the questions
Findings
Overall Ratings of Audit Team
1
2
3
4
5
WrittenQuestions
InterviewQuestions
Average
QUALITY POLICY PENETRANCE
High
Low
Reasonable
Overall Ratings of Audit Team for Written Responses
1
2
3
4
5
QUALITY POLICY PENETRANCE
High
Low
Reasonable
Opportunities for Opportunities for ImprovementImprovement
Ratings of Audit Team for Ratings of Audit Team for Written Responses who were blinded in their Written Responses who were blinded in their
assessmentsassessments
1
2
3
4
5
Auditor 1 Auditor 2 Auditor 3
QUALITY POLICY PENETRANCE
High
Low
Reasonable
It is often said that:
The Devil is in the DetailsThe Devil is in the Details
In this case…
The Added Value is in the DetailsThe Added Value is in the Details
Individual Question Results
0
1
2
3
4
5
Q1 Q2 Q3 Q5 Q6 Q7 Q8 Q9 Q10
QUALITY POLICY PENETRANCE
High
Low
Reasonable
0
1
2
3
4
5
Q11 Q12 Q13 Q14 Q15 Q16 Q17 Q18 Q19
QUALITY POLICY PENETRANCE
High
Low
Reasonable
Individual Question Results
0
1
2
3
4
5
Q20 Q21 Q22 Q23 Q24 Q25 Q26 Q27 Q28
QUALITY POLICY PENETRANCE
High
Low
Reasonable
Individual Question Results
0
1
2
3
4
5
Q29 Q30 Q31 Q32 Q33 Q34 Q35 Q36 Q37
QUALITY POLICY PENETRANCE
High
Low
Reasonable
Individual Question Results
0
1
2
3
4
5
Q38 Q39 Q40 Q41 Q42 Q43 Q44 Q45 Q46
QUALITY POLICY PENETRANCE
High
Low
Reasonable
Individual Question Results
The SMARTSMART Process:Analysis of Individual Questions
The results of individual questions reveal areas The results of individual questions reveal areas of excellence as well as specific opportunities for of excellence as well as specific opportunities for improvement:improvement:
Questions 4 and 5 show companyQuestions 4 and 5 show company--wide wide excellenceexcellence
The SMARTSMART Process:Analysis of Individual Questions
Questions 8 and 9 show strong enabling Questions 8 and 9 show strong enabling tools and goodtools and good penetrancepenetrance despite less than despite less than adequate policy description adequate policy description -- perhaps due perhaps due to selfto self--evident dataevident data
The SMARTSMART Process:Analysis of Individual Questions
Question 2 shows good policy development Question 2 shows good policy development but poor but poor penetrancepenetrance throughout the throughout the organizationorganizationQuestion 1 and 3 indicate opportunities for Question 1 and 3 indicate opportunities for improvementimprovement
Rather than looking at individual responses to Rather than looking at individual responses to written and interview data, another approach was written and interview data, another approach was to aggregate the information into Safety Processes. to aggregate the information into Safety Processes. Data was aggregated following processes:Data was aggregated following processes:
In general these data show a strong emphasis on training and
implementation
Ratings of Responses by Processes
1 2 3 4 5
Implementation
Measurement
Evaluation
Training
Fundamentals
Penetrance Policy Quality
HighLow Reasonable
**
**
These data show that penetranceinto the organization falls behind
policy understanding and an ability to articulate policy perspectives
Ratings of Responses by Processes
1 2 3 4 5
Implementation
Measurement
Evaluation
Training
Fundamentals
Penetrance Policy Quality
HighLow Reasonable
******
**
Evaluation strategies and measurement tools could use more
emphasis
Ratings of Responses by Processes
1 2 3 4 5
Implementation
Measurement
Evaluation
Training
Fundamentals
Penetrance Policy Quality
HighLow Reasonable
Below acceptableBelow acceptable
Mostly Below acceptableMostly Below acceptable
Interview Findings
Generally Safety Personnel Understood Policy and Could
Articulate the Policy but Generally Agreed that the Penetrance in the Organization could be improved
Ratings of Interviews for Managers and Safety Professionals
2.5
3
3.5
4
Senior Managers SafetyProfessionals
Line Managers
QUALITY POLICY PENETRANCE
High
Low
Reasonable
Comparison of Interviews and Written Responses
1
2
3
4
5
SeniorManagers
Safety LineManagers
Written
QUALITY POLICY PENETRANCE
High
Low
Reasonable
Ratings of Interview Questions
0
1
2
3
4
5
Q25 Q26 Q27 Q28 Q29 Q31 Q32 Q33
QUALITY POLICY PENETRANCE
High
Low
Reasonable
Ratings of Interviews by Processes
1 2 3 4 5
Economic Impact on Safety
Contractor Training
Measurement of Safety
SMS Daily Duties
Penetrance Policy Quality
HighLow Reasonable
**
Are there examples of questions you can ask in your organization?
You bet!!
Example 1 –Non-Punitive Reporting Systems
Does the company have a non-punitive reporting policy?
If so, can you describe it and tell me if it is followed in day-to-day operations?
Non-Punitive Reporting Systems: Answer
The company has a wellThe company has a well--documented and documented and comprehensive noncomprehensive non--punitive reporting policy that can punitive reporting policy that can be found in the flight operations manual and is posted be found in the flight operations manual and is posted on all safety bulletin boards. on all safety bulletin boards. The policy includes easy to use webThe policy includes easy to use web--based and paper based and paper reporting forms, and has an option for anonymity if the reporting forms, and has an option for anonymity if the reporter requests it. reporter requests it. Unfortunately, most people won’t fill out reports Unfortunately, most people won’t fill out reports because they don’t believe they will be protected from because they don’t believe they will be protected from prosecution by management if they report errors.prosecution by management if they report errors.
Non-Punitive Reporting Systems: ScoringQuality of Response = 3Quality of Response = 3•• The answer is of reasonable quality and demonstrates a The answer is of reasonable quality and demonstrates a
fundamental understanding of the company’s nonfundamental understanding of the company’s non--punitive punitive reporting policyreporting policy
Policy Assessment = 4 Policy Assessment = 4 •• The answer reflects a good understanding of the nonThe answer reflects a good understanding of the non--punitive punitive
reporting system, including details such as where it can be reporting system, including details such as where it can be found and how to use itfound and how to use it
Organizational Penetration = 1Organizational Penetration = 1•• Even though the answer is of reasonable quality and Even though the answer is of reasonable quality and
demonstrates a good understanding of the policy, it shows demonstrates a good understanding of the policy, it shows poor adherence and the policy has not penetrated into daypoor adherence and the policy has not penetrated into day--toto--day operationsday operations
Example 2 –Reactive and Proactive
Safety Reporting
What processes or steps do you follow when you have an incident and observe
a hazard?
How do you think it is working?
Reactive and Proactive Reporting Processes: Line Staff Answer
When there is a safety incident, there are specific When there is a safety incident, there are specific procedures that we follow in the operations manual and procedures that we follow in the operations manual and forms that we fill out. forms that we fill out. [Prompt, how do you think this is working][Prompt, how do you think this is working] It works It works pretty well I guess but I don’t get much feedback. I used pretty well I guess but I don’t get much feedback. I used to fill out the forms for everything but now the incident to fill out the forms for everything but now the incident needs to be serious before I do the paperwork.needs to be serious before I do the paperwork.When I observe potential hazards I usually let my When I observe potential hazards I usually let my supervisor know.supervisor know.
Reactive and Proactive Reporting Processes: Line Staff
Good response and understanding of Policy and ProceduresGood response and understanding of Policy and Procedures
Good adherence to procedures but becoming diligence seems Good adherence to procedures but becoming diligence seems to be reducing over time indicating poorer to be reducing over time indicating poorer PenetrancePenetrance
Much better Policy and Procedures for Reactive Reporting of Much better Policy and Procedures for Reactive Reporting of Incidents than for Proactive Assessment of HazardsIncidents than for Proactive Assessment of Hazards
Reactive and Proactive Reporting Processes: Executive Answer
When there is a safety incident, there are specific procedures tWhen there is a safety incident, there are specific procedures that hat need to be followed in the operations manual and forms that we need to be followed in the operations manual and forms that we fill out. All employees are trained on these procedures and formfill out. All employees are trained on these procedures and forms s are completed and analyzed so that corrective action can be are completed and analyzed so that corrective action can be taken. These reporting procedures are flexible enough to cover taken. These reporting procedures are flexible enough to cover both reactive and proactive reporting and are used in both casesboth reactive and proactive reporting and are used in both cases..[Prompt, how do you think this is working][Prompt, how do you think this is working] It works very well. It works very well. We have binders of analyses and act on these analyses to improveWe have binders of analyses and act on these analyses to improvesafety. safety. It must be working well because the reports pick up data on It must be working well because the reports pick up data on serious incidents. Safety is improving as demonstrated by fewer serious incidents. Safety is improving as demonstrated by fewer reports every year.reports every year.
Reactive and Proactive Reporting Processes: Executive
Good response and understanding of Policy and ProceduresGood response and understanding of Policy and Procedures
No demonstration of the need for feedback to ensure No demonstration of the need for feedback to ensure maintenance of implementationmaintenance of implementation
Declining Declining PenetrancePenetrance given that line staff are restricting their given that line staff are restricting their reporting and the Executives interpret this as improved safety.reporting and the Executives interpret this as improved safety.
SummaryThe SMART method produced objective The SMART method produced objective assessments of the Safety Management assessments of the Safety Management System System The approach appears to provide reliable The approach appears to provide reliable ratings and consistent resultsratings and consistent resultsThe findings appear to identify areas of The findings appear to identify areas of excellence and opportunities for excellence and opportunities for improvementimprovement
SummaryClearly there are differences in the Quality Clearly there are differences in the Quality of Responses, Understanding of Policy, and of Responses, Understanding of Policy, and Penetration of Safety into the Organization Penetration of Safety into the Organization that need to be addressedthat need to be addressedThe approach clearly identifies strengths and The approach clearly identifies strengths and weaknesses in the Safety Management weaknesses in the Safety Management System to help focus Continuous Quality System to help focus Continuous Quality Improvement with regard to Safety Improvement with regard to Safety Management SystemsManagement Systems
SummaryImportantly, the feedback from this Importantly, the feedback from this analysis provides the organization analysis provides the organization with a baseline against which to with a baseline against which to measure change (improvement) of measure change (improvement) of future Quality Improvement Strategies future Quality Improvement Strategies and changes in subsequent Policy and changes in subsequent Policy Development Development
SummaryEqually important is that this type of Equally important is that this type of feedback helps target resources to feedback helps target resources to strategic areas strategic areas Using this strategy, decisions can be Using this strategy, decisions can be made empirically on the most made empirically on the most important areas to invest resources important areas to invest resources while protecting resources for areas while protecting resources for areas where accomplishments have occurred where accomplishments have occurred
SummaryThis approach will allow strategic decisions This approach will allow strategic decisions This approach will allow the strategic allocation This approach will allow the strategic allocation of resources to where they might be of most of resources to where they might be of most benefitbenefitThis approach needs to be informed by future This approach needs to be informed by future applications leading to a applications leading to a manualizedmanualized approach approach that can be widely available with appropriate that can be widely available with appropriate consultationconsultation
Allows self evaluationAllows self evaluationComprehensiveComprehensiveSimple to useSimple to useTheory driven Theory driven -- Practically OrientedPractically OrientedTransparencyTransparency
“…“… the overall process was the overall process was helpful in identifying the method helpful in identifying the method of conducting a safety audit of conducting a safety audit ……[our organization] can learn [our organization] can learn significantly from this process.significantly from this process.””
President & Chief Executive Officer President & Chief Executive Officer