Company Confidential Registration Management Committee (RMC) 1 AQMS Accreditation Programs ANAB Findings Atlanta, GA July 22, 2010 Steve Holladay ANAB,

Company Confidential

Registration Management Committee (RMC)

1

AQMS Accreditation Programs ANAB Findings

Atlanta, GAJuly 22, 2010

Steve HolladayANAB, Accreditation Assessor

Auditor WorkshopAtlanta, GA

July 22-23, 2010


Atlanta, GAJuly 22-23, 2010

Objectives

• Provide an overview of the NCR’s identified from the witness audits.

• Present in process approach.

• Discuss steps that could have been taken to prevent the NCR’s.

• Rules of engagement.

2



Goal

• Provide knowledge/information to the AS auditor pool with the goal to improve auditor/CB performance that will reduce repeat NCR’s which will, in turn, add value to the assessment process and the industry as a whole.

3


Atlanta, GAJuly 22-23, 2010 4

• 120 NCR’s in 54 Witnessed Audits (WA)– 2.22 Average per WA

• 50 NCR’s in approx 40 Office Assessments (OA)– 1.25 Average per OA

Overall



• 3 NCR

• DETAIL:– Assigning Auditors who either did not have the

proper NACE code or AS91XX qualifications.

– Client not made aware of the OP assessor.

• LEARNINGS: – Verify auditors are qualified for the full

dynamics of the audit.

– Communicate clearly with the client.

Pre-Audit Assignment



• 11 NCR’s

• DETAIL: 19011, 6.4.1.c– (4) The audit plan did not ID the organizations

functional units and processes to be audited.

• LEARNINGS:– The audit plan ensure the client will have the

proper resources available to the audit.

– Clearly demonstrates what processes are intended to be audited.

Pre-Audit Planning



• 11 NCR’s

• DETAIL: 17021. 9.3.2.1– (2) Audit Planning not effective in assuring the

AQMS is assessed to the min requirements for surveillance audits.

• LEARNINGS:– Ensure the auditors understand where the full

auditing requirements are for surv audits if not clearly ID in the audit plan.

Pre-Audit Planning



• 11 NCR’s

• DETAIL: MD 5 and AS9104– (5) Insufficient Auditor Days in the planning

without clear justification.

• LEARNINGS:– If deviation…..JUSTIFY.

– If the audit is being witnessed, provide to ANAB in the pre-audit information.

Pre-Audit Planning



• 5 NCR’s, several OFI’s.

• DETAIL:– Information missing or not correct in EQM.

– Client Profile form not complete or accurate. (scope, head count, regulatory, ITAR)

• LEARNINGS:– Ensure the person entering information is

knowledgeable about the information.

– Verify with Client and lead auditor before submitting.

Pre-Audit Preparation



• Total of 24 NCR’s

• DETAIL:– Sampling part of an element and making a

conclusion on the whole requirement.

• LEARNINGS:– Audit plan and report accuracy is critical.

On-site Audit Depth of auditing



• DETAIL: – Auditing to the scope of certification is not

adequate.

• LEARNINGS:– Ensuring the scope of certification, scope of

audit and audit plan is consistent.

– Validate the exclusions. Is scope consistent with the QM.

– Ensure the address and scope on the Certificate is correct.




• DETAIL: – Not following audit trails when objective

evidence suggest otherwise.

• LEARNINGS:– A plan is a plan. Follow the trail to its natural

conclusion if potential findings are evident.

– Keep head on a swivel. Don’t ignore clear findings.




• DETAIL: – Not fully verifying the effectiveness of the

actions taken on nonconformities identified during the previous audit.

• LEARNINGS:– Ensure the evidence is more than a record

review.




• 4 NCR’s

• DETAIL:– Design Applicability

• LEARNINGS– More of an issue with AS9110.

On-site Audit Interpretation



• DETAIL:– Not fully auditing Outsourcing when

applicable.

• LEARNINGS– Standard clearly requires “control of such

outsourced processes shall be identified” and therefore subject to audit. Not just a document verification.




• DETAIL:– Changes to the capability listing of the client.

• LEARNINGS– Similar to the scoping discussion. Verify any

changes to the capability listing as this could affect the scope of certification or introduce new processes and technology.




• 11 NCR’s

• DETAIL: – Calling the Clients CA/PA system effective with

ineffective/inadequate correction, root cause or corrective action.

• LEARNINGS:– Deep dive the CA/PA system. Validate the

information. If it doesn’t make since….make the call.

On-site Audit Decisions



• DETAIL:– Soft Grading 8 NCR’s

– Clearly stating findings during the audit and not raising the NCR or improper categorization.

– Accepting informal correction during the audit.

• LEARNINGS– Learn the definitions and follow them. Keep track

of the “verbal” findings identified during the audit. Accepting correction to verbal findings without reporting is consulting.

– As an auditor it is not your job to justify an NCR down.




• DETAIL:– Continuing the audit when the audit objectives

are clearly unattainable.

• LEARNINGS:– Only the client can make the decision to

continue an audit.

– There should be a clear conclusion that the objectives are unattainable and reported to the client with options.» At the time this is realized not at the end!!




• DETAIL: – Closing meeting does not address all of the

requirements.

• LEARNINGS– Clearly address the CA follow up and impacts

on the existing cert.

– Ensure the CA are presented to the correct requirements.

On-Site Audit Closing



• 22 NCR’s

• DETAIL:– Accuracy of the Audit Report and Checklist to

the observed audit.

• LEARNINGS– Anecdotal evidence should be documented as

such. Same for deductive evidence.

On-site/Off-site Audit Reporting



• DETAIL: – Report not in conformance with

AS9014/AS91XX

• LEARNINGS– Justify/explain all NA.

– Clear evidence to support the conclusion.

– Clear evidence to support the SCOPE.

– Include detailed notes

On Site/Off-Site Audit Reporting



• 3 NCR’s

• DETAIL:– Report did not contain mandatory items.

• LEARNINGS– Ensure the Checklist are complete or

references the location of the information.

– Explain and differences between the information left on site to the published report.

Post Audit Reporting



• 8 NCR’s

• DETAIL: Improper closing of NCR’s and Improper Certification Decision.

• LEARNINGS:– Clear evidence of Correction, RC and CA.

– Ensure persons closing NCR are AEA.

– Ensure the NCR’s are closed with appropriate evidence PRIOR to the cert decision.

Post Audit NCR Closure



Office Audit NCR’s

• OASIS data base admin/accuracy

• Verification of Client OASIS data admin

• Justification of auditor days.

• Improper qualification of AS cert decision maker.

• Improper decision made with certification information.

25



Questions

Company Confidential Registration Management Committee (RMC) 1 AQMS Accreditation Programs ANAB Findings Atlanta, GA July 22, 2010 Steve Holladay ANAB,

Documents

preaudit planning slide

preaudit information

preaudit assignment

preaudit preparation

repeat ncrs

oa overall slide

itar learnings

witnessed audits wa