Proficiency Testing in Microbiology: Statistics ...nla.org.za/webfiles/conferences/2016/Presentations... · •NLA-SA Microbiology PT schemes: z scores z score: A normalised value

Proficiency Testing in Microbiology:

Statistics, performance criteria and the use of proficiency data

What is Proficiency Testing?

• Part of quality assurance

• Demonstration of competence

• Scheduled interlaboratory testing

Proficiency testing: Assigned value

• Assigned value: Known or consensusNLA-SA Microbiology PT schemes: Consensus value

• Influence of outliersOutlier: A member of a set of values which is inconsistentwith other members of the set – extreme high or low

Removal of outliers or robust statistical methods

NLA-SA Microbiology PT schemes: Robust statistical methods

Classical vs robust statistics

Classical statistics Robust statistics

Mean

Robust mean

or

Median

Standard deviation

Robust standard deviation

or

Normalised inter-quartile

range (NIQR)

Robust mean & Standard deviation

• Derived by iterative calculation (or repetition)

• Values of mean and standard deviationupdated several times using modified datauntil the process converges

• No change from one repetition to the next inthe third significant figures

• Example

Median & NIQR:Median explained

• Median: The number separating the upperhalf of a data set from the lower half

– Data arranged from the lowest to the highestvalue and the middle value determined

Example: 9 5 7 1 8

1 5 7 8 9

1 5 7 8 9

Median & NIQR: Median explained (cont.)

• If an even number of data points: Average of two middle items of data

Example: 13 18 13 16 14 21

13 13 14 16 18 21

13 13 14 16 18 21

= 15

Median & NIQR:NIQR explained

• NIQR: The difference between the 3rd

quartile (Q3) and 1st quartile (Q1) of theparticipant laboratories’ results.First quartile (Q1) = First 25% of results when rankedin order

Third quartile (Q3) = First 75% of results whenranked in order

NIQR = 0.7413 x (Q3-Q1) (Assumption: Normaldistribution)

Performance criteria

• Robust statistics used to calculate performance criteria

• NLA-SA Microbiology PT schemes: z scores

z score: A normalised value which gives a “score” to each result, relative to the other results in the data set

Describes closeness of laboratory’s result to consensus value

Close to zero: Result agrees well with rest

Calculation of z scores

Food Microbiology PT scheme:

z score = (result obtained by participant – robust mean)Robust standard deviation

Water Microbiology PT scheme:

z score = (result obtained by participant – median)NIQR

Between & within z scores

• Duplicate results submitted, denoted A and B

• Between laboratory z scoreDemonstrate bias in results: Caused by equipment or operator

1) Calculate standardized sum (S) for each participant: S = (A + B)

2

2) Calculate median & NIQR of all S’s, i.e. median(S) and NIQR(S)

3) ZBW = [standardised sum of participant results (S) – median(S)]

NIQR(S)

Between & within z scores (cont.)

• Within laboratory z scoreReflect laboratory’s ability to reproduce exactly the same result

1) Calculate standardized difference (D) for each participant:

D = (A - B)

2

2) Calculate median & NIQR of all D’s, i.e. median(D) and NIQR(D)

3) ZWI = [standardised difference of participant (D) – median(D)]

NIQR(D)

Interpretation of z scores

• z score close to zero:

lab’s result agrees well with consensusvalue

• Positive z score:

lab’s result > consensus value

• Negative z score:

lab’s result < consensus value

Interpretation of z scores (cont.)

• Conventionally interpreted as follows (ISO 13528):

|z| ≤ 2 Satisfactory

2 < |z| < 3 Questionable

Investigate possible causes to

identify emerging or recurrent

problems

|z| ≥ 3 UnsatisfactoryAction signal indicating a need

for corrective action

How to effectively use PT results

• Set own internal acceptance criteria

• Read the PT report & review your performance:How close to zero is the lab’s z score?

Is the lab’s result higher or lower than the consensus?

Is the result acceptable according to the internal acceptance criteria?

• Trend your performance: Excel spreadsheet or graph

• Give feedback

How to effectively use PT results (cont.)

• Do follow up investigations:Check: reported result = result obtained

correct method used & instructions followed

calibrated equipment used

training of staff

• Implement corrective action

• Verify: perform test on same sample or another unknown sample

Conclusion

• Better understanding:

Microbiology Proficiency Testing Schemes

Interpretation of PT results

• Powerful tool:

Identify problems in testing

Improve the performance of the laboratory

Thank you