Putting the A in ADCL – the work of the Calibration ... · work of the Calibration Laboratory Subcommittee ... Subcommittee initiated a major revision of the accreditation ... based

1

Putting the A in ADCL Putting the A in ADCL –– the the work of the Calibration work of the Calibration

Laboratory SubcommitteeLaboratory Subcommittee

Malcolm McEwen, Donna Stevens, Tom Slowey and Jan Seuntjens

AAPM Calibration Laboratory Sub-committee

2

What is an ADCL?What is an ADCL?

Accredited Dosimetry Calibration LaboratoryProvides calibrations to users for instruments and radioactive sources for dosimetry in radiotherapy and diagnostic imagingCurrently three ADCLs:

University of Wisconsin (Madison) K&S Associates (Nashville) MD Anderson Cancer Centre (Houston)

http://uwrcl.medphysics.wisc.edu/http://www.kslab.comhttp://rpc.mdanderson.org/adcl/

3

What does the CLA do?What does the CLA do?

“Oversee all activities regarding the operations of the ADCLs. The Subcommittee’s task is to accredit, supervise and maintain the highest level of confidence in the quality of the ADCL system, with sufficient capacity in the system to prevent undue delays in satisfying the membership’s calibration needs while providing a choice of ADCLs.”

Main forum for discussion of issues relating to calibration of ion chambers and brachytherapy sources

4

Some historySome history

In 1971 the AAPM formed a task group to develop guidelines for the establishment of a system of secondary standard calibration laboratories.

The laboratories would be accredited by the AAPM to provide high precision dosimetry calibrations.They would be outside of the National Institute of Standards andTechnology (National Bureau of Standards as it was then) but traceable to NIST.

Purpose:To reduce the time required for precision calibrations To create a system of secondary standard laboratories, To establish a technical resource for the membership of the AAPM

5

Time progresses … Time progresses …

A sub-committee of the Radiation Therapy Committee of AAPM has overseen the development of the program to this day.Names have changed but the focus is still the sameAt the summer meeting of the AAPM in 1995, the Subcommittee initiated a major revision of the accreditation protocol which has resulted in the present criteria that now conform to ISO standard 17025:2005.

6


Develops criteria

Accredits laboratories

Carries out assessment visits

Monitors performance

Makes recommendations

7

What does the CLA What does the CLA notnot do?do?

Does not set prices Does not recommend any one ADCL over anotherDoes not distinguish between commercial and academic organizationsDoes not try to “spread the business around”

8

Registration Registration vsvs AccreditationAccreditation

We all see signs saying “An ISO9001 registered company” but what does it mean?Registration:

Procedure used to give written assurance that a system conforms to specified requirements. Such systems include those established for the management of product, process or service quality, and environmental performance

9

Quality System RegistrationQuality System Registration

ISO 9000 quality systems are genericApplies to all types of organizationsRecognition is of a quality system that’s defined in terms the output (products/services) of the quality systemRegistration does not certify or guarantee the quality of products or service for compliance with specific technical specifications

10

AccreditationAccreditation

Accreditation:Procedure by which an authoritative body gives formal recognition that a body or person is competent to carry out specific tasks

11

Laboratory AccreditationLaboratory Accreditation

Formal recognition that a testing laboratory is competent to carry out specific tests or types of testsKey Words:

competentspecific tests (scope of testing)

Accreditation is having a Quality System anddemonstrating competency.Requires assessment (audit) visits to the laboratory to verify competency.

12


Develops criteria





13

Laboratory Accreditation:Laboratory Accreditation:System AuditingSystem Auditing

Based on a course by:PROFESSIONAL QUALITY SYSTEMS

Gary D. Scalise, Instructor Technical Lead Assessor A2LA

Malcolm McEwen, Donna Stevens, Tom Slowey and Jan Seuntjens

AAPM Calibration Laboratory Sub-committee

14

What Really Is An Audit?What Really Is An Audit?

Audit or Assessment: to verify conformancebased on objective evidence

15

Types of AuditsTypes of Audits

First PartyInternalThe laboratory’s own audit

Second PartyExternalUsually by a client to their own specific requirements

Third PartyExternalBy independent organization, e.g. AAPMFull quality system evaluationObjective – done by trained and qualified assessors

16

ISO17025 and the AAPMISO17025 and the AAPM

ISO standard 17025:2005 is the international standard covering calibration laboratories The AAPM has a set of Criteria which are used for the accreditation of ADCLsThe Criteria are based on the requirements set out in ISO17025A lab accredited according to the AAPM Criteria is, by implication, following ISO17025But – the Criteria have additional requirements that must be met. Therefore ISO17025 accreditation is not enough on its own to satisfy use of the ADCL logo

17

The AAPM CriteriaThe AAPM CriteriaThe Criteria are based on the requirements set out in ISO17025In addition there are detailed technicalrequirements for each specific area that will be covered by the scope of accreditation:

Co-60 absorbed dose to waterLDR and HDR brachytherapyDiagnostic radiology dosimetryElectrometer calibration, etc

18

Audits Audits –– the details the details

The slides will focus on the AAPM criteria and what the CLA does

BUTThey can be applied to any assessment of a quality system, quality assurance program, procedure, etcTo both:

external audits (such as carried out by the CLA)internal audits (checking before someone else does)

19

First First -- an audit needs Auditors!an audit needs Auditors!

Understanding of quality system philosophiesIn-depth knowledge of standardsTechnically competent on test methodsExcellent communication skills (written and oral)Ability to probe and analyze – not obnoxiousGood logic, judgement and integrityPoise, tact, persistence and maturityEasy going and positive attitudeNot easy to find!

20

Auditors need to be trained on…Auditors need to be trained on…

The standard (AAPM Criteria, ISO17025…)The organization’s auditing proceduresInformation gathering techniquesThe human relation aspects of auditing

It unfortunately takes more than 1 lectureIt unfortunately takes more than 1 lecture

21

Phases of an AuditPhases of an Audit

Determine ifQuality SystemDocumentationmeets intent ofthe standard

(AAPM Criteria)

Reading beforehand

Evaluate how well Quality System

Policies and Proceduresare put into practice

“Live”

22

PrePre--Audit PreparationsAudit Preparations

Review all pertinent quality system documents• Goal: awareness of specific policies and procedures

Prepare a detailed gap analysis vs. requirements • Goal: understanding of what is there compared to what

should bePrepare questions and checklists for use during auditReview all documents once again• Goal: complete familiarity with the laboratory procedures• Don’t start on the plane

23

Scheduling the AuditScheduling the Audit

Contact the auditeeProvide results of your document review (if possible)Request any additional documentsEstablish the date and duration of the auditProvide checklists you want to have filled outEnsure there is a correct understandingSend auditee an agenda of the audit

24

Typical Audit SequenceTypical Audit SequenceOpening Meeting

Introduction of each auditee

Gathering the information and evidence

Summary with each auditee

Compiling Report

Closing Meeting

25

Opening MeetingOpening Meeting

Gather all the major participantsLaboratory director and quality manager

Supervisors, key personnel, analysts as needed

Discuss the scope of the auditReview the agenda of the auditDiscuss how deficiencies will be handledNo surprises at the closing meeting

26

What’s being looked at? What’s being looked at?

1. Management Requirements From Section 4 of AAPM CriteriaOrganizational Structure, Quality Policy, Technical Management, Records Management, Purchasing Policy, Internal Audit, etc.

2. Technical RequirementsFrom Section 5 of the Criteria (and Appendices)Test methods, Equipment, Analysis, Reporting, etc.

27

Gathering Information and Gathering Information and EvidenceEvidence

Asking QuestionsInspecting Facility and Equipment Examining DocumentsExamining RecordsObserving Activities

28

Limitations of QuestionsLimitations of Questions

Can’t always take answers at face valueAnswers may not be the truthMay be intentionally misleadingCould be coached responsesBut remember:

Auditors are in the evidence businessAuditors are in the evidence businesshunches and opinions don’t counthunches and opinions don’t count

29

Technical ProficiencyTechnical Proficiency

Beyond system auditing, an audit of a laboratory requires demonstrated technical proficiency and competency (Section 5.0)Auditors can verify (see) this four ways:1. Verify suitable equipment2. Verify appropriate procedure/test method3. Interview personnel to verify knowledge4. Observe the test to verify competence

30

Inspect Facility & EquipmentInspect Facility & Equipment

Evaluate design layout of laboratoryProper environmental conditions and monitoring?Sufficient lighting? Temperature?Effective separation between areas?Good housekeeping? Appropriate security?Suitable equipment – range, precision?Status of equipment – maintenance, calibration?

31

Examining DocumentsExamining Documents

Documents available for use? Implemented?Is there proper document control for

Internal documents?External documents?

Document changes properly made?Draft procedures in use?Obsolete documents removed?Verify that testing follows written procedures

32

Examining RecordsExamining Records

Quality and technical records in place?Records allow traceability and repetition?Procedures being followed for archiving?Records include identity of all personnel?Contemporaneous records? Complete?Error corrections properly made?

33

Observation of ActivitiesObservation of Activities

Witness demonstrations of procedures by staffConfirms what we heard from our questionsConfirms that analysts are following proceduresDetermines status of equipment and setupEnables us to judge technical skillsEstablishes the level and depth of trainingGains information on supervision

34

Aside Aside –– CLA surveillance CLA surveillance

There is no requirement in a surveillance visit to witness a procedure (calibration)However, it may be useful:

if there is any confusion with documentationif it’s a simple procedureto help familiarize the auditor with the lab facilities and personnel if the lab requests it

35

Compare Audit Findings to…Compare Audit Findings to…

AAPM Criteria requirements (or relevant standard)

Need specific clause #The internal quality system requirementsTechnical laboratory proceduresAgreed methods with the clientCompare reality to requirements,

But remember the rule…

36

#1 Rule of Auditing #1 Rule of Auditing

If you cannot express the non-conformance in the words of the quality system standard,

the appropriate technical specification or test method, or the company’s own policies

and procedures,

THEN YOU DO NOT HAVE A THEN YOU DO NOT HAVE A NONNON--CONFORMANCECONFORMANCE

37

Audit ReportingAudit Reporting

Reporting has more than one aspectVerbal reporting

Feedback to the auditee at time of finding

Closing meetingSummarize findings to management at close of audit

Written deficiencies statementsDocumented audit report

Documented as soon as possible (same day)

38

RecommendationRecommendation

Don’t keep tight-lipped until the presentation of the report because:

Shock and Awe are not legitimate assessor toolsInformation may be power but you are really there to assist not controlIf a finding is simple to rectify then it can be cleared before you leave (and that’s easier for everyone) You may be mistaken than it’s a non-conformance

39

When Something Seems WrongWhen Something Seems Wrong

Is it really wrong?Do the lab personnel know its wrong?What is their explanation?Is it an isolated event, or a symptom of a deeper problem?Why didn't the quality system detect it?What lapse in the quality system allowed this to happen?

40

Compiling the Audit ReportCompiling the Audit Report

Writing the Non-Conformances (Deficiencies)Assembling the complete audit report

Scope of Audit: What, How, WhyAgenda of Audit: When, Where, How LongIdentification of auditor(s) and auditee(s): WhoNon-ConformancesCompleted checklistsTime frame for corrective action responses

Make sure you leave time for this!Make sure you leave time for this!

41

What is a nonconformity?What is a nonconformity?

Definition:

“non-fulfillment of a requirement”

An equivalent term is “deficiency” “Non-compliance” should be avoided as it has a legal implication

42

What is an “audit finding?”What is an “audit finding?”

Definition:“result of the evaluation of the collected audit evidence against audit criteria”

Findings can be good or “not so good”Can indicate conformity, nonconformity or opportunities for improvement

43

Writing Deficiencies Writing Deficiencies

ClearConciseNon-confrontationalNon-blamingNon-personal

44

Writing a DeficiencyWriting a Deficiency

Structure of a deficiency statement

The Requirement

Description of the deficiency

The objective evidence

45

Writing DeficienciesWriting Deficiencies

Directly related to the relevant quality system standard, technical procedures, laboratory policies and procedures, etc.And based on a clear statement of objective evidence recorded during the audit

46

RecommendationsRecommendations

Auditors are free to make recommendations that result from audit findings or general observations of the laboratoryLaboratories are free to ignore any recommendations!But recommendations can be a real help to the development and improvement of a QS (the lab is effectively getting free consultancy) and therefore should be considered

47

Closing MeetingClosing Meeting

Gather participants – everyone if possibleReview the compliance areas – firstReview the non-compliance issuesBe careful to distinguish between non-compliances and recommendationsComplete any recordsThank people for their assistance and time

48

Corrective ActionsCorrective Actions

A deficiency triggers the need for corrective actionThe auditee is responsible for corrective action

Conduct a root cause investigation for eachIdentify potential corrective actionsDetermine the significance and resourcesEvaluate possible effects on other departments or other elements of the quality systemEstablish a time frame for implementationReference each corrective action to one deficiency

49

FollowFollow--Up on DeficienciesUp on Deficiencies

Auditee should first monitor that the corrective action brought desired resultsVerify that the agreed corrective action has been effective in the long termAdditional audits may be warranted if the laboratory is still out of compliance

50

Audit ClosureAudit Closure

An audit is formally closed when…All reported nonconformities are addressed by corrective action plan

Responsibility of auditee

Corrective action is implementedResponsibility of auditee

Corrective action is verified as effectiveResponsibility of audit program manager (CLA)

51

Audit summaryAudit summary

Audits are a valuable tool for any process-based organizationAudits are compliance basedAn auditor only writes deficiencies when objective evidence has been gatheredAll deficiencies must directly relate to the relevant standard, test method, etc.Remember the #1 Rule of Auditing

52


Develops criteria





53

Thank you for your attentionThank you for your attention