www.labsmartservices.com.au CBR - 2016 (67) PROFICIENCY TESTING PROGRAM REPORT Accredited for compliance with ISO/IEC 17043 Copyright: LabSmart Services Pty Ltd CBR Proficiency Testing Program - 2016(67) Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 1 of 80
80
Embed
CBR - 2016 (67) PROFICIENCY TESTING PROGRAM REPORT · Report This report is available on the LabSmart Services website. The issue of this proficiency report was ... CBR Proficiency
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
www.labsmartservices.com.au
CBR - 2016 (67)
PROFICIENCY TESTING
PROGRAM REPORT
Accredited for compliance with ISO/IEC 17043
Copyright: LabSmart Services Pty Ltd
CBR Proficiency Testing Program - 2016(67)
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 1 of 80
Report This report is available on the LabSmart Services website. The issue of this proficiency report was authorised by Peter Young, Director, LabSmart Services Pty Ltd, February 2017. Contact Details
Accredited Proficiency Testing Provider LabSmart Services is accredited by NATA to ISO/IEC 17043, Conformity assessment – General requirements for proficiency testing. Accreditation number 19235. The accreditation provides additional assurance to participants of the quality and importance we place on our proficiency testing programs.
LabSmart Services As well as proficiency testing programs LabSmart Services also offers a wide range of other services including consultancy, training and nuclear gauge calibration. Please see our website for further details.
www.labsmartservices.com.au
Copyright This work is copyright. No part of this publication may be reproduced in any form, transmitted or stored in any repository (e.g. mechanical, digital, electronic or photographic) without prior written permission of LabSmart Services Pty Ltd. Please contact LabSmart Services should you wish to reproduce any part of this report.
Amendment History Reports may be downloaded from the LabSmart Services website.
Version 1 – Issued 28 February 2017
CBR Proficiency Testing Program - 2016(67)
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 2 of 80
CONTENTS PAGE
1. Program Aim
5
2. Performance
2.1 Performance assessment 2.2 Identified outliers 2.3 Focus on improvement 2.4 Program summary
5 5 6 6 9
3. Technical Comment
3.1 General performance
3.1.1 Supply of program information 3.1.2 Errors
3.2 Statistical reality
3.2.1 Accuracy of data 3.2.2 Variation in CBR results 3.2.3 Set s.d limit 3.2.4 Repeatability
3.3 CBR results
3.3.1 Participant assessment 3.3.2 CBR results 3.3.3 Identification of inconsistencies and errors 3.3.4 Repeatability
3.4 Measurement uncertainty 3.5 Direct influences
3.5.1 Load cell 3.5.2 Seating load 3.5.3 Penetration rate 3.5.4 Test (penetration / load) data 3.5.5 Accuracy of the graph prepared 3.5.6 Zero-point correction 3.5.7 Rounding
4.1 CBR Z-scores : Sample A & B 4.2 Recalculated CBR results 4.3 Repeatability 4.4 Set SD limits
36
35 39 43 45
CBR Proficiency Testing Program - 2016(67)
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 3 of 80
5. Program Information
5.1 Z-score summary 5.2 Program design
5.2.1 Design 5.2.2 Selection of material for program 5.2.3 OMC & MDD 5.2.4 Role of proficiency testing 5.2.5 Participant assessment 5.2.6 Reporting of results – significant figures 5.2.7 Additional information requested
5.9.1 Z-score Summary 5.9.2 Comparing statistics from one program to another 5.9.3 Measurement uncertainty 5.9.4 Metrological traceability
47
47 47
47 47 48 48 48 49 49
49 49 50 50 50 50 50
52 52 53 53
6. Participants’ Test Results
55
Appendix A Instructions for testers
Appendix B Results log
Appendix C Graph example
75
77
80
CBR Proficiency Testing Program - 2016(67)
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 4 of 80
1. Program Aim
The proficiency program was conducted in September/October 2016 with participants
throughout Australia. The program involved the performance of:
AS 1289.6.1.1 (2014) – Determination of the California Bearing Ratio of a soil –
Standard laboratory method for a remoulded specimen.
The program’s intention is to provide feedback and confidence to the construction
materials testing industry regarding the competency of participants (and the industry)
to perform this test. Each participant’s performance is statistically assessed and used
as a measure of competency relative to all those who participated. Other measures
of performance are also used.
This report has been prepared using robust statistics. In addition, test data has been
reviewed for consistency and additional feedback regarding aspects of the test are
provided.
Comprehensive technical comment is provided to assist participants improve the
overall performance of this test (Section 3).
Information regarding the conduct and design of the program etc. can be found under
section 5.
2. Performance
2.1 Performance assessment The CBR test is a complex test from a measurement uncertainty perspective despite its apparent technical simplicity. Unfortunately, the CBR test method does not provide guidance about some aspects of the test such as reproducibility. There also appears a lack of guidance on both the performance and the interpretation of the test within the industry. The range of test results obtained in a proficiency program, for any given sample, has been far wider than is generally acceptable to the industry. These aspects add to the difficulty in interpreting CBR proficiency programs. In discussing the outcome of this program the following have broadly been used to determine outliers and areas for investigation/review.
Z-scores based on submitted CBR results (statistical outliers)
Identification of inconsistences, non-adherence to test method and errors
Accuracy of calculations
Repeatability Proficiency testing providers are obligated under their accreditation standard to remove results known to be incorrect or where a participant has not followed the test method including adherence to prescribed limits. Not providing all data requested,
CBR Proficiency Testing Program - 2016(67)
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 5 of 80
particularly where it is used to assess the validity of the results obtained (e.g. compaction, MC) is also a valid reason to reject a CBR result. This is not however ‘black & white’ but where the ‘grey area’ becomes complex. Keeping results that may be suspect in the statistical pool may distort the statistical outcome. However, if all the results found to be inaccurate or not meet the test method etc. were rejected form this program there would be very little to analyse! A balance must be struck. This is discussed in more detail under technical comment (section 3.1 & 3.2).
2.2 Identified outliers
In most proficiency testing programs the identification of outliers is relatively straight
forward. This is not the situation with CBR testing due to the large range in CBR
results obtained.
Industry has expressed concerns that from an engineering “End User” perspective that
such large variations in CBR results are impractical. It is also undesirable from a
laboratory testing perspective.
As has been indicated in previous proficiency programs, it is the middle 50% of
participants results that is far larger than it should be. It is this group of results
therefore that is of primary interest.
Participants with statistical outliers, departure from the test method or errors
(Investigate) and those with significant departures compared to other participants
(Review) are shown in table 2.2A.
Participants where there is a concern regarding accuracy of the results are requested
to investigate their submissions. Others have been identified as able to benefit from
reviewing their submissions where it is felt the quality of testing may be improved.
In table 2.2A there are no participants listed e.g. ‘penetration rate’ and ‘test data’. This
is not because there are no concerns identified only that the test method does not
identify or address the issues affecting accuracy identified in this program.
The more times a participant’s code appears in the table 2.2A the greater the need for
follow up.
2.3 Focus on improvement Have laboratories improved? The answer is yes. The standard of CBR testing has
improved enormously over the last 5 to 7 years. Much of this can be attributed to
laboratories being prepared to participate in PT programs and being prepared to
improve on existing laboratory practices (e.g. move away from hand graphs to
computer generated graphs etc.).
CBR Proficiency Testing Program - 2016(67)
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 6 of 80
Table 2.2A – Participants identified where investigation or review follow up is warranted.
CBR Proficiency Testing Program - 2016(67)
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 7 of 80
To the many out there “well done” and “thank you” for your participation.
The CV over this time has been around 30 % whereas this program has a CV of 21%. Note that the ‘gold’ highlighting shows programs conducted to the current 2014 test method. It shows that the spread (variation) in CBR test results are heading in the right direction.
Year Program Median CV
2016 67 155 21
2015 59 140 20
2014 54 74 31
2013 46 37 29
2012 37 44 20
2011 48 61 35
2009 16 30 32
Table 2.3A Comparison of CBR program results for the last seven years
Why is the middle 50% important? At present the spread of results is extremely large and affects the repeatability and reproducibility of the test. In other words, if it is too large the CBR results become meaningless. Based on this national proficiency program and others the CBR results appear of limited value to the end user. However, if put in context as to where the CBR results are used the situation may not be so bad. On a regional basis, where the CBR results are used, there may be much closer agreement. This may be due to similar training or better overseeing by a technical body etc. (e.g. Road Authorities) Has the CBR test results always been subject to the same large spread in results? Most likely it has. It may have been that it just was either not observed due to no national PT programs or seemed unimportant at the time. There is nothing in the test method that has changed substantially over the years to cause a change in the spread of results obtained. Staff are no better or worse trained then previously. What has changed over the last 50 years however is how results are interpreted and used. Road Authorities did much of the road construction work and testing so CBR results were basically in-house. Results were generally compared amongst those trained the same way using similar or same equipment. There was not the proliferation of small testing laboratories back then or the need to compare results from one State to another. National CBR proficiency programs for CBR were either not run or if so very infrequent. Today there are laboratories with quite diverse views on the CBR method and using quite different equipment. The test method allows considerable latitude in the performance of the test. It is not unexpected that the variation across Australia is as large as it is.
CBR Proficiency Testing Program - 2016(67)
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 8 of 80
How can improving the accuracy help. It was hoped that improving the accuracy of testing would significantly reduce the spread in results shown. This program has shown that this may not be the case. The accuracy of many of the test results was shown that they could be improved but it did not necessarily lead to a significant reduction in the spread of the middle 50% of participants. Accuracy of testing still needs to be improved and most laboratories are trying to do this. Graphing has a significant impact on the overall accuracy achieved. Section 3.5 focus on those aspects most directly affecting CBR accuracy. What aspects of the test affect the outcome indirectly? The test method puts limits on the indirect aspects of the test such as moisture, compaction, LDR and LMR etc. It would appear at present, that while these are important, the control of them and how it affects the outcome is difficult to predict or quantify. Most cannot be examined in isolation nor is it clear how they interact. See section 3.6. Removing those results that did not comply with the limits placed by the test method did not, in this program, lead to an improvement in the spread of results. Can changes to the test method help? The last change to the test method has seen a significant reduction to the spread of results obtained. Based on the technical comments (section 3.7) it appears that while incremental improvements are possible any major improvement may need a substantial rethink of the test method. How can laboratories improve? Proficiency testing programs that provide technical feedback assist laboratories to improve. All of the technical comment detailed in section 3 has this in mind and is aimed at helping laboratories to have a greater understanding of the test. 2.4 Program Summary There has been an observable improvement in CBR testing over the last six years as measured by the coefficient of variation (CV). Previous proficiency programs have highlighted the need to reduce the variation shown by the middle 50 % of participants. The technical comment indicates that there are ways to reduce this slightly but it may be that this spread is just reflecting the accuracy of this test. The program shows that there are several improvements that laboratories can make to improve the accuracy of their individual results. It is highly desirable as it will overall reduce the spread of results. Improvements in repeatability are also needed. Improvements in accuracy however may not directly transfer to a significant reduction in the spread in results.
CBR Proficiency Testing Program - 2016(67)
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 9 of 80
The program identified those aspects of the test that most affect accuracy (direct influences) and those aspects of the test that have less influence (indirect). There are aspects of the test method such as graphing and zero correction where further guidance is needed for the industry. The CBR graph is the only way of checking the validity of the results obtained. In most cases the graphs prepared do not adequately address this issue. Both these and other issues are identified as major contributors to CBR accuracy. The program showed that penetration rates may contribution significantly to the spread in results. Many participants were identified (>65%) as needing to review one or more aspects of testing. Therefore, test results for all sixty-five participants were recalculated based on the test data submitted. This allowed several conclusions to be drawn that previously had not been able to be answered in past programs. What this means is that every participant should review the testing submitted against the technical comments made in this report. Improvements to the test method, by better defining the test process, limits and expected outcomes would significantly improve the accuracy of the test. The statistics used in all proficiency programs is there for guidance. Care however needs to be exercised to avoid incorrect conclusions being drawn. This is one of the reasons that LabSmart Services provides comprehensive technical comment for each program. This proficiency program provides increased understanding of current test practices and potential sources of variation. It also allows monitoring of improvements in testing and provides the opportunity for participants to improve their competency. A summary of the program statistics is shown in Table 2.4A.
Statistic CBR A CBR B
Number of participants 61 63
Median 155 168
Normalized IQR 32 35
Minimum* 81 88
Maximum* 222 258
Range* 141 170
CV (%) 21 21
Table 2.4A Summary of statistics for the CBR program. Some results have been rounded. *Min, Max & Range
are with outliers excluded.
CBR Proficiency Testing Program - 2016(67)
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 10 of 80
The observed variation (based on CV) has improved compared to other proficiency programs conducted over the last decade.
3.0 Technical comment The feedback is extremely lengthy and for this we apologise. It is unlikely that such detailed feedback will be offered in the future. In today’s vernacular, it is a “conversation” that the industry needs to have. It is hoped that this report will contribute to that discussion. 3.1 General performance Proficiency program participants are expected to comply with the requirements of the program and meet basic laboratory standards. General performance covers those aspects of laboratory operations that are expected to be performed as part of good laboratory practice and in keeping with NATA accreditation. Some aspects that are particularly relevant for this program are:
Supervision of testing
Following the test method
Following proficiency testing instructions
Correctly filling out paperwork i.e. PT log sheet
Checking of results
Free of errors i.e. calculations correct
Reality check of results i.e. does it fit the type of material submitted
Compared to earlier CBR proficiency testing programs there has been significant improvement in most of the above areas. However as detailed in subsequent sections there is still a considerable way to go to improve the accuracy of testing. It raises the question that if participants are not meeting the above basic requirements then what other omissions or errors are occurring during testing that remain undetected. 3.1.1 Supply of program information
Several participants supplied all the testing details requested. This information is used
to provide the feedback given in the following sections and is an important part of the
program.
Participants are always welcome to contact the program coordinator if they require
further explanation as to what information is required or how to proceed.
The information requested is also used to validate the results obtained.
CBR Proficiency Testing Program - 2016(67)
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 11 of 80
Proficiency testing program providers may reject results on this basis if they do not
conform to the program requirements.
3.1.2 Errors
Errors may arise from several sources, an incorrect calculation, transcription error, wrong methodology used, not following the test method etc. Many of the comments in the following sections relate to errors. For example, participant 5-U9-A load /penetration results may have been entered the wrong way around on the log sheet. Or it could be that the CBR values are the wrong way around. Either way it corrupts the statistics used to assess the program outcome as well as being incorrect. Although some of these may have only a small impact they do accumulate and should not occur. Others can have a large impact such as incorrect graphing technique and zero correction.
Reduction in the number of errors detected in this program would significantly improve
the credibility of CBR testing and possibly reduce the variation (spread) in CBR results
obtained.
3.2 Statistical reality
The use of statistics are a very useful and practical means of analysing test data.
Those that use statistics everyday know the limitations and shortcomings of their use.
Unfortunately, they often do not get aired sufficiently to remind those who only
occasionally use statistics. This section addresses some of these aspects associated
with this report.
3.2.1 Accuracy of data
If the test data is in error, then any statistics calculated may also be error. Any
interpretations made, based on the statistics, may also be in error. Most proficiency
programs can handle a few inaccurate results without any concern about the validity
(accuracy and precision of the outcome) of the program. Most of the technical
comment concerns the accuracy of the CBR test results. The number of results that
are questionable does raise significant concern about the validity of many of the test
results.
3.2.2 Variation in CBR results
Often proficiency testing programs tend to focus on feedback concerning those with results that seem either too high or too low (outliers). While this is important sometimes other areas become just as important. The fundamental issue with CBR test results is that there is too large a spread shown by the middle 50% of participants. Within this group the results are too spread out.
CBR Proficiency Testing Program - 2016(67)
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 12 of 80
This means that identifying accurately a median value or outliers may be seriously compromised. It seriously affects the accuracy and precision of the test results. Without improvement or better understanding of CBR test results there is a strong risk
that the test may no longer be credible.
3.2.3 “Set s.d limit”
In previous CBR proficiency programs the z-score statistics have been recalculated
using a “Set s.d limit” or “Target s.d”. The purpose of which was to bring the variation
(spread) in results back to something useful to geotechnical engineers and other
clients. It is a very practical way of approaching the problem. There is no reason to
suspect that, based on the outcome of this program, any result within the middle 50%
is better than another result from the middle 50%. The “Set s.d limit” outcome then
does not give much useful information other than an exercise in setting a limit. It
becomes a bit of a ‘Round peg in a square hole’ approach.
If the accuracy of most test results is questionable along with the median value, then
results may lie above or below the set limits. It does not identify problems or inaccurate
results, worse, it will indicate results as being satisfactory when they are not.
It has still been used in this program mainly as an indicator as to where a reasonable
variation limit may lie. So long as it is not taken that anyone test result is more accurate
them another then it is reasonable to use this approach.
In previous CBR proficiency programs LabSmart Services has given some guidance
as to what could be used as an acceptable spread from an “End User” perspective. A
variation considered reasonable has been based on ± 9.5 % (coefficient of variation)
of the median. Calculated target values are shown in table 3.2.3A below for this
program. Results as submitted have been used.
See Section 4.4
Median CV (%) S.D. Limit
Sample A 155.2 % 9.5 ± 14.75 %
Sample B 167.8 % 9.5 ± 15.94 %
Table 3.2.3A Target S.D Limits. Bearing Ratio at 5.0 mm
Please note that the following “Set s.d limit” outcomes do not constitute an
outlier in this program but provide one approach to determining which participant
results may be improved on further review.
CBR Proficiency Testing Program - 2016(67)
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 13 of 80
Results were recalculated using the above ‘Set SD Limits’ as detailed in section 4.4
Based on these limits, participants exhibiting statistically significant variation are
shown in table 3.2.3B below.
Sample A - Review Sample B - Review
Z-score> 3 Z-score> 2.75 Z-score> 3 Z-score> 2.75
M8, U9, F2, X5, U7, B8, T2, U3, S7, T5,
N3, Z3, Y3, V5 K8
L8, J6, J7, T2, R5, N8, S7, K2, N3, P6,
Y3, V5 U7
Table 3.2.3B Statistically significant unrounded CBR results based on a set “SD Limit”.
Participants shown above may find it beneficial to review the performance of this test.
Also as all the CBR results shown in table 3.2.3B are very large from an engineering
perspective these may still be considered acceptable depending on the use of the
material.
3.2.4 Repeatability
If the spread of results both within and between participants is large it is hard to arrive
at a sensible repeatability and reproducibility outcomes. The accuracy of the results
also has a large impact. See section 3.3.4 for more detail on repeatability. Hopefully
future programs may result in better estimates of repeatability.
3.3 CBR Results
3.3.1 Participant assessment
Participant performance has been assessed in three ways:
Z-scores based on submitted CBR results (statistical outliers)
Identification of inconsistences, non-adherence to test method and errors
Accuracy of calculations
Repeatability
Participants need to be aware that the program coordinator performing the checks
may not have access to the full set of results for each participant. Also, due to the
large amount of data associated with this program it is entirely possible that the
coordinator may not have recalculated some participants results correctly.
Participants are asked to “investigate” matters that are statistical outliers and where
the test method has not been followed or are outside the limits set in the test method.
CBR Proficiency Testing Program - 2016(67)
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 14 of 80
Other maters identified are shown as “Review”. These are matters that would help
improve testing and it most cases would be considered outside normal testing
parameters. It is sometimes difficult to determine as the test method often does not
provide sufficient guidance.
As an aid to identifying usable limits “Set sd limits’ have been calculated but do not
form part of the assessment.
3.3.2 CBR results
Z-scores and associated statistics were calculated on the CBR results as submitted
and are detailed in section 4.1. The following statistical outliers were identified:
Sample A Sample B
Investigate Review Investigate Review
V5 - S7 -
Table 3.3.2A Participants identified as having statistical outliers in the program
For the CBR test the spread of results is very large over the middle 50% of participants.
Both sample A and B were the same material. The repeatability outcome was
statistically good but the accuracy of the data called the outcome into question (see
section 3.3.4 for more detail).
It makes it extremely difficult to draw accurate conclusions from the statistics. For
example, participant V5 had a high CBR result and was identified as an outlier for
sample A. However participant V5 had an even larger CBR result for sample B but it
was not identified as an outlier! The statistics are correct it is just a reflection of very
poor reproducibility and repeatability.
The construction industry may need to acknowledge, if improvement cannot be
established, that this is the accuracy and precision of the CBR test.
Sections 3.5, 3.6 and 3.7 explores this further.
3.3.3 Identification of inconsistences and errors
There are many steps within the conduct of the test (methodology) that can become a
source of error or where inconsistencies can occur. As well there are limits posed by
the test method itself that may also contribute. For example, compaction and moisture
content.
CBR Proficiency Testing Program - 2016(67)
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 15 of 80
To better understand the influence that these sources have on the variation of the test
they have been broken up into those that directly affect the CBR result and can be
measured and those where the impact on the CBR result cannot be easily measured
i.e. indirect. See section 3.4 for more information on determining direct and indirect
influences.
Direct influences generally involve participant errors in testing. These are discussed
in more detail in section 3.5.
The following calculation was used to determine if a submitted result required further
appraisal.
[ ( Load5.0 - Load2.5 )*100 ] / 2.5 where Load is in kN.
This could also be used by supervisors as a quick means of checking test data as it
does not need the graph.
This check was undertaken. As it showed a large number to check it was decided in
the end to recalculate everyone’s results. The outcome of this exercise is shown in
section 4.2.
The recalculated results have been used to determine performance in many instances.
This gave a more accurate appraisal and did not change the outlier outcome except
to show a better performance for some individuals.
Indirect influences generally involve non – compliance to the test method requirements
or limits. These are discussed in section 3.6.
3.3.4 Repeatability
The spread of results was much larger in earlier programs. Previously it was felt that
investigating methodology yielded better information then duplicate samples would
have. The test method was revised in 2014 and addressed some shortcomings in
relation to the compaction process. This led to changes in methodology for some
laboratories and improvement in the CBR variation i.e. CBR Proficiency Program
2015(59) CV was 20%.
The new test method has now had a reasonable period to ‘bed in’. As consequence
if was felt that it was appropriate to use duplicate samples for this program to measure
repeatability.
Unfortunately, the large spread in results obtained affects the repeatability outcome.
The number of participants where the CBR value calculated by participants was shown
to be inaccurate was quite high. Considering both these factors makes the estimate
of repeatability shaky. Z-score statistics are shown in section 4.3
CBR Proficiency Testing Program - 2016(67)
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 16 of 80
An alternative approach would be to use the homogeneity data as an estimate.
However, this may also be a shaky estimate. The precision may be good (same
machine and pace rate) but the accuracy may be poor.
The repeatability was calculated (sample A and B combined) with the participants
results (section 3.3.2) as submitted and as corrected (section 3.3.3). Only the original
participant’s results are shown in section 4.3, repeatability for the recalculated results
has been omitted from the report (to reduce the report volume).
By using the recalculated CBR results it reduces the number of participants shown as
needing to either investigate or review their outcomes, table 3.3.4A.
Based on submitted CBRs Based on recalculated CBRs
Between laboratories Within laboratory Between laboratories Within laboratory
At high CBRs the seating load has only minimal effect on the CBR obtained but does
influence where the penetration points fall. For this type of material any effect of
incorrect assignment of the zero penetration is usually cancelled out with the zero-
point correction offset if performed correctly.
Setting the seating load to zero was done by most participants except (B8, S6, X8,
Y9, R5, U3, N5, C2). Not setting ‘back to zero’ again can lead to an inaccuracy in the
load scale creating an offset.
However, errors in both processes (seating load applied and resetting back to zero)
may influence the CBR result by up to ± 1.3% CBR. This may not seem much but in
the rounding process when reporting this may cause a difference of 10%.
3.5.3 Penetration rate
Test method indicates that the machine used must be capable of “….forcing the
penetration piston into the specimen at uniform (not pulsating) rate of 1.0± 0.2 mm/min
during the complete test….”. The penetration rate had in the past not been routinely
checked until NATA in recent years required it be checked every two years.
It is not entirely clear, based on input from participants, if the standard means an
‘average rate’ of if it means it must be met at ‘all times’. If it is taken as an average
rate then theoretically you could have half the penetration at 0.5 mm/min and the other
half at 1.5 mm/min and still arrive at the average rate of 1.0 mm/min.
For ‘hand’ operated devices it is hard to check other than an overall average. A
motorised platform was used by the majority of participants with seven participants
(K8, L5, U3, N3, Q5, K4, V5) using a hand operated unit.
With load cell units, they usually allow the rate to be checked as you go on a ‘per 0.5
mm of travel’ etc. This can be done on a ‘test by test’ basis so is a very good record
of meeting the requirements of the standard.
Participant J8 reported the penetration rates as detailed in table 3.5.3A. The average
value obtained of 0.84 mm/min meets the test method requirements. All participants
reported a penetration rate within the Australian Standard specified range of 1.0 ± 0.2
mm/min.
Very few reported penetration rates to this accuracy most participants appear to have
quoted just the test method requirements (i.e.1 mm/min) rather than stating a
calculated or measured average penetration rate. The test method does not ask that
this be done. It raises however the question as to whether most participants had
something different, more in keeping with what is shown in the table 3.5.3A.
CBR Proficiency Testing Program - 2016(67)
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 21 of 80
A load cell or ring may compress around 0.4 mm under high loads which contributes
to the penetration rate achieved. In future, the NATA ‘penetration checks’ may need
to be requested.
Penetration Penetration Rate
mm mm/min
0.50 0.69
1.00 0.86
1.50 0.82
2.00 0.79
2.50 0.78
3.00 0.79
3.50 0.80
4.00 0.80
4.50 0.87
5.00 0.91
5.50 0.88
6.00 0.87
6.50 0.89
7.00 0.95
7.50 0.95
Average = 0.84
Table 3.5.3A Penetration rates for participant J8
The penetration rate is linked to the slope of the force/penetration curve. It is therefore
significant in determining the CBR and hence the set limits placed on the rate of travel
by the test method.
Graph 3.5.3B Effect of applying the tolerance on the penetration rate 1.0 ± 0.2 mm/min
0
5
10
15
20
25
30
35
40
0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0
Load
Penetration
Penetration at 4.8, 5.0 and 5.2 mm for CBR 160
1.2 mm/min 1.0 mm/min
0.8 mm/min
CBR Proficiency Testing Program - 2016(67)
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 22 of 80
The rate can be interpreted in terms of time or penetration. If it takes 5 minutes to
penetrate 5 mm. It takes 6.25 minutes to penetrate 5 mm at a rate of 0.8 mm/min and
4.17 minutes to penetrate 5 mm at a rate of 1.2 mm/min. Alternatively, it can be viewed
as the load at 4.8, 5.0 and 5.2 mm as shown in graph 3.5.3B.
Regardless of the approach the outcome remains the same, for 5 mm penetration the
accuracy is therefore ± 0.2 mm. For a CBR of 162% this translates into approximately
± 6.6%. This is detailed in table 3.5.3C.
Penetration mm
Load kN
CBR %
Difference CBR %
CV %
4.8 30.7 155.0 -6.6 -4.1
5.0 32.0 161.6 0.0 -
5.2 33.3 168.2 6.6 4.1
Table 3.5.3C Variation in CBR for rate of 1.0 ± 0.2 mm/min.
For this program where the CV was 21 % a possible ± 4 % CV is significant. Improving
the accuracy of penetration would also reduce the spread of results obtained. Even
at lower CBR values the CV variation remains significant.
There is an assumption that the slope associated with penetration remains constant
regardless of the rate of penetration. This means that the soil offers the same
resistance no matter how quick or slow you push.
It is possible that for some soils pushing slowly may allow the soil to “move out of the
way” easier than if pushed faster against the soil. The stresses in the soil may be
different. The effect cannot be easily calculated. Some experimentation would be
needed to determine if there is a significant effect or not.
3.5.4 Test (penetration / load) data
The number of penetration points selected is extremely important. Many
laboratories recorded the requested additional load/penetration data. A few took less
than that requested for this proficiency program or terminated the test before 12.5 mm.
The test method specifies a minimum data set (0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 4.0, 5.0, 7.5,
10.0 and 12.5 mm penetrations).
The key word in the test method is “at least”. In other words if you know the material
well (i.e. have a CBR history of the material) then you should be able to use fewer
points otherwise you need to record loads at more points.
CBR Proficiency Testing Program - 2016(67)
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 23 of 80
Additional points are needed to:
Allow for the discount of an abnormal data value Have sufficient points left so that the discounting of a point does not
compromise the test result Have sufficient points to fit a straight line and a curve Have sufficient points above the straight section of the graph. Have sufficient points to be able to tell that you have an abnormal data point
Participant E5 was selected but there were many other participants with small data
sets. The penetration/load values submitted for participant E5 were used to produce
a graph and a zero-correction line. The results are shown below Graph 3.5.4A.
3.5.4A Load / penetration graph for participant E5, Sample B
Only the minimum data points have been collected. The graph shows that there was
one data point only between 5 mm and 10 mm (i.e. circled data in the graph above).
If the data point at 7.5 is out it can have a large effect on the CBR obtained. If more
points had been taken, then the uncertainty about this point would be removed.
It is evident that two few data points can have a measurable difference on the result
that is obtained. Greater confidence in the result and accuracy is obtained when more
points are taken.
3.5.5 Accuracy of the graph prepared
Graphing has been singled out due to its importance in deriving an accurate result and
being able to check the CBR result obtained. The CBR test method does not emphasis
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 24 of 80
Graphing of results has been an issue for the last seven CBR proficiency programs (7
years). Overall graphing has improved vastly over this time but there is still
considerable room for improvement.
In nearly every case you can take the raw data from a AS 1289 test method and
everyone would calculate the same result. With CBR the raw data can be given to 12
or more laboratories and possibly get 12 different answers!
From a testing perspective, this is unacceptable but within the test method. This
is not the fault of laboratories but it is within their scope and ability to improve on
this.
Regardless of what graph is submitted to the client a detailed graph for use by the
laboratory is important as it is the primary method of checking that a reasonable
result has been obtained.
This is also how the program coordinator checked if participant’s results matched the
data submitted. Unfortunately, this is not conveyed by the test method. The test
method is also not very descriptive regarding the quality of the graph prepared.
In previous proficiency programs, considerable feedback has been given. An example
graph is given in Appendix C. This is the level of detail and quality that laboratories
should be aiming.
Participants are welcome to contact LabSmart Services and request a copy of the
recalculated graph.
The following sections expand on issues where better graphing practice would have
improved the accuracy of the result obtained.
3.5.6 Zero-point correction
Overall, most participants correctly calculated the zero-point correction and applied it. Seating load and zero correction combined generally result in small changes.
However sometimes small changes can have a significant effect and particularly when
a BR value is to be rounded either up or down to the nearest 10%. A variation of ±
20 % CBR is not unrealistic.
Not applying the zero-point correction does have a significant impact. For example,
participant (J7) did not correct the raw data back to a zero point. Participant (V5)
appears to have overestimated the correction needed.
Participants J7 and V5 data have been re-graphed (3.5.6A & B) and the bearing ratio
at 5 mm recalculated.
CBR Proficiency Testing Program - 2016(67)
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 25 of 80
3.5.6A Load / penetration graph for participant J7, Sample B
3.5.6B Load / penetration graph for participant V5, Sample A
C2, E4, R6, X7, G2, R7). Unrounded was required by the program due to the impact
on the statistics generated. These participants need to investigate the reason for
rounding the results.
Sample A Sample B
Unrounded Rounded Unrounded Rounded
Number 61 60 63 60
Median 155.2 160 167.8 165
N-IQR 32.3 29.7 34.8 38.9
Minimum 81.0 80 60.0 60
Maximum 252.4 250 258.1 230
Range 171.4 170 198.1 170
CV (%) 20.8 18.5 20.8 23.6
Table 3.5.7A Effect on statistics for rounded and unrounded CBR results
3.6 Indirect Influences
As mentioned previously the following aspects of testing methodology are difficult to
relate to the final CBR test result. They can be measured individually but the influence
it has on the CBR result is more difficult due to the ‘unknown interactions’ they have
on each other.
Section 4.2 identifies, by colour coding, the ‘indirect influences’ that are discussed in
this section (i.e. compaction blows, LMC, LDR etc.). Results have been flagged that
show a significant departure from the test method. If these were to be deleted and
those left (not many) are analysed statistically the variation (spread) in results remains
much the same.
CBR Proficiency Testing Program - 2016(67)
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 28 of 80
Sample A Sample B
Submitted Less indirect Submitted Less indirect
Number 60 43 62 42
Median 157.4 161.7 168.9 163.9
Third Quartile 136.6 137.2 150.0 153.6
First Quartile 180.2 179.3 197.5 192.7
IQR 43.6 42.2 47.5 39.0
N-IQR 32.3 31.2 35.2 28.9
Minimum 81.0 100.6 60.0 108.5
Maximum 252.4 222.7 258.1 232.8
Range 171.4 122.1 198.1 124.3
CV (%) 20.5 19.3 20.8 17.7
Table 3.6 Effect on statistics for submitted CBR results and those less indirect
There is some change to the overall spread of results by deleting suspect results but
by no means large, due to the accuracy of results themselves, making the outcome
less assured.
It is clear however that more accurate measurement of these aspects of the test in
conjunction with better definition within the test method would assist with improving
the overall accuracy of the test.
3.6.1 Pre-compaction curing
There were a range of curing times used by participants. The majority used 48 hours and above. The curing times specified by the test method are minimums. More curing is better than less.
Participants (Z6, S6, X8, X6) used curing times of less than 24 hours. These curing
times are at odds with the rest of the participants and should be reviewed.
The process to be used when curing material does not appear to be documented.
Thus, there is a strong probability that many laboratories use quite different
approaches, some being more effective than others.
Samples need to be in sealed containers and the material broken up and mixed
regularly during curing. Water should be added as a mist to the largest surface area
possible. Condensation on the container side and lid needs to be monitored. Regular
mixing should avoid material on the bottom being wetter than the rest of the material.
3.6.2 CBR compaction
The proficiency program required participants to perform the CBR compaction using the OMC and MDD values provided and 100 % standard compaction.
CBR Proficiency Testing Program - 2016(67)
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 29 of 80
Test methods relating to compaction are very specific about the energy input into the process. This is largely governed by the spread of hammer blows and the number of blows used. The revised CBR method now stipulates the pattern to be used when compacting the CBR mould. The test method however does not specifically require the number of blows delivered to be recorded. As it is an important part of the test it should be recorded. It is expected that by compacting a calculated amount of material to a set height that the desired density will be achieved. The blows will vary depending on the material type and moisture. Depending on how this is done a variation in the number of blows per layer is the typical outcome. However, between layers these should remain reasonably close. For determination of OMC/MDD using standard compaction 25 blows per layer is used. To achieve the same energy input around 53 blows is required for the larger CBR mould. More or less blows than 53 may be needed for a variety of reasons.
Inaccuracy of the OMC and MDD initially
Blows not delivered in a regular pattern
Nature of the material may cause it to move around the mould excessively
Material added is higher or lower than the prescribed layer depth The blows delivered provides an insight into whether any of the above issues may have had an effect. Relying on the dry density calculated is useful but it is a calculated value and dependant largely on how representative and accurate was the moisture determination. How much variation is reasonable. This is at present unknown but for this program a variation of 40 to 60 has been used with a variation between layers of 5 blows. The following participants shown in table 3.6.2A do not meet this criterion. Most shown did so for both sample A and B except for those underlined.
Sample A Sample B
< 40 Blows per layer > 60 blows per layer < 40 Blows per layer > 60 blows per layer
W8, S7 D8, C6, K8, J6, F2,
X6, N3, N9, K4 W8, S7
D8, C6, K8, J6, F2, X6, J8, N3, N9, X3,
K4, M3
Table 3.6.2A Participants with high or low number of compaction blows.
CBR Proficiency Testing Program - 2016(67)
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 30 of 80
The bulk of participants, outside the limits set, used more than 60 blows per layer. It may not affect the dry density obtained but there is concern that it may have an effect such as;
orienting the soil particles,
segregation of particles
causing fissures,
breaking up of particles
uneven compaction, all of which could influence the CBR without affecting the dry density value achieved. CBR results may be higher or lower depending on the influence. As far as is known these issues have not been investigated in recent times. There is also the possibility that these results do not belong to the general population of test results for this program. If a modified compaction hammer was used, then the number of blows needed for equivalent energy using 3 layers is approximately 20 blows. Two participants (N4, T8) used modified compaction over 3 layers (instead of 5). Two others (E5, X7) indicated standard compaction but used blows consistent with modified compaction. As the results, do not comply with the test method or proficiency testing program requirements the results obtained do not belong to the general population of test results. 3.6.3 OMC & MDD
Different determinations of OMC & MDD by different laboratories will give rise to a
spread of results (Variation). To limit the effect of this variation on the CBR testing in
this proficiency program the OMC & MDD have been predetermined. This information
was supplied to participants (See instructions Appendix A) so that all participants used
the same OMC & MDD values.
3.6.4 LDR and LMR
Calculation of LDR & LMR
Participants were requested to submit:
The sample moisture immediately prior to compaction (w1) in accordance with
clause 6(c) of the standard.
Moisture content variation (wv)
The Laboratory Moisture Ratio (LMR)
The Laboratory Density Ratio (LDR) and
Dry Density (before soaking)
CBR Proficiency Testing Program - 2016(67)
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 31 of 80
These intermediate results are noted in the test method as needing to be reported or
required to determine compliance with the test method.
The reported LDR and LMR values were re-calculated using the reported moisture
from clause 6(c) and density (before soaking). It is suspected that several participants
had incorrectly reported the moisture of the sample as being that of ‘as received’.
There were several participants that had difficulty in calculating the intermediate
results detailed above. There were also many participants that did not report LNR &
LDR to the correct number of decimals or had rounded incorrectly.
The participants listed in Table 3.6.4A showed inconsistencies in the values submitted
throwing doubt on compliance with the test method and should be investigated.
Note: A # indicates an outlier where the z-score obtained is either greater then 3 or less than -3. Codes for all participates are shown. Results where only one value
was submitted have been left out. The results column shows a blank entry for those participants that did not perform this test. Minimum, Maximum and Range are
calculated with outliers excluded.
WithinSample A Sample B
Between Within
Laboratories Laboratory Laboratories Laboratory
4.3 CBR - Repeatability: Z - Scores
Code
Test Results (%) Z Score
Code
Test Results (%) Z Score
Sample A Sample BBetween
CBR Proficiency Testing Program - 2016(67)
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 43 of 80
Statistic Sample A Sample B
Number of results 61 61
Median 155 161
First Quartile 136 150
Third Quartile 180 196
IQR 43.6 46.0
Normalised IQR 32.3 34.1
CV (%) 20.8 21.2
Minimum 81 60
Maximum 222 258
Range 141 198
4.3 CBR - Repeatability: Z - Scores
CBR Proficiency Testing Program - 2016(67)
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 44 of 80
M8 205.5 3.41 # T2 87.3 -4.60 # X3 122.4 -2.22
D6 143.9 -0.77 Y9 140.5 -1.00 N5 173.0 1.21
D8 165.2 0.68 A2 146.0 -0.62 Z3 199.7 3.02 #
P7 154.6 -0.04 X6 159.6 0.30 P6 NR
U9 204.3 3.33 # A7 191.9 2.49 C4 138.9 -1.11
C6 150 -0.35 R5 136.7 -1.25 Z9 NR
T9 171.7 1.12 N8 188.8 2.28 Y3 212.1 3.86 #
Q7 170.7 1.05 U3 218.6 4.30 # Q5 135.2 -1.36
L8 115.2 -2.71 S7 100 -3.74 # T7 NR
C8 140 -1.03 Q2 NR K4 162.3 0.48
Z6 171.1 1.08 M4 155.2 0.00 V5 252.4 6.59 #
K8 111.7 -2.95 D3 180 1.68 N4 126.1 -1.97
J6 180 1.68 G4 130.0 -1.71 T8 161.1 0.40
L5 167.8 0.85 U6 NR C2 180 1.68
F2 81.0 -5.03 # T5 107.6 -3.23 # E4 130 -1.71
X5 222.2 4.54 # V3 125.8 -1.99 M3 180.8 1.74
D9 136.4 -1.27 M7 NR R6 150 -0.35
W8 189.9 2.35 E9 NR E5 189.9 2.35
U7 103.9 -3.48 # J8 150.6 -0.31 X7 170 1.00
L6 NR A3 164 0.60 S4 NR
J7 152.0 -0.22 K2 181.7 1.80 G2 140.0 -1.03
B8 200 3.04 # N3 216.7 4.17 # R7 140.0 -1.03
S6 130 -1.71 L7 130.7 -1.66
X8 180 1.68 N9 145.5 -0.66
Number of results 61
Median 155.2
Median MU 2.4
First Quartile 136.4
Third Quartile 180.0
IQR 43.6
Set SD 14.8
CV (%) 9.5
Minimum 111.7 (81.0)
Maximum 191.9 (252.4)
Range 80.2 (171.4)
4.4 CBR - Sample A - Set SD: Z - Scores
Statistic Value
Note: A # indicates an outlier where the z-score obtained is either greater then 3 or less than -
3. Codes for all participates are shown. The results column shows a blank entry for those
participants that did not submit a result for this test. Minimum, Maximum and Range are
calculated with outliers excluded, those in brackets include outliers.
Code
Test
Result
%
Z Score Code
Test
Result
%
Z ScoreCode
Test
Result
%
Z Score
CBR Proficiency Testing Program - 2016(67)
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 45 of 80
M8 195.2 1.72 T2 88.3 -4.99 # X3 196.0 1.77
D6 129.1 -2.43 Y9 150.0 -1.12 N5 170.6 0.18
D8 208 2.52 A2 150.0 -1.12 Z3 186.4 1.17
P7 154.6 -0.83 X6 151.5 -1.02 P6 226.3 3.67 #
U9 157.4 -0.65 A7 202.0 2.15 C4 170.7 0.18
C6 140 -1.74 R5 229.7 3.88 # Z9 NR
T9 171.8 0.25 N8 216.6 3.06 # Y3 218.7 3.19 #
Q7 186.9 1.20 U3 190.1 1.40 Q5 143.8 -1.51
L8 112.0 -3.50 # S7 60 -6.76 # T7 NR
C8 150 -1.12 Q2 NR K4 204.9 2.33
Z6 171.1 0.21 M4 167.8 0.00 V5 258.1 5.66 #
K8 131 -2.31 D3 120 -3.00 N4 157.9 -0.62
J6 220 3.27 # G4 160.0 -0.49 T8 157.5 -0.65
L5 155.5 -0.77 U6 NR C2 170 0.14
F2 160.9 -0.43 T5 126.5 -2.59 E4 130 -2.37
X5 202.0 2.15 V3 126.1 -2.62 M3 198.0 1.89
D9 141.4 -1.66 M7 NR R6 160 -0.49
W8 210 2.65 E9 NR E5 151.5 -1.02
U7 213.3 2.85 J8 203.8 2.26 X7 180 0.77
L6 184.6 1.05 A3 188 1.27 S4 NR
J7 119.1 -3.06 # K2 229.7 3.88 # G2 140 -1.74
B8 180 0.77 N3 218.2 3.16 # R7 160.0 -0.49
S6 130 -2.37 L7 160.0 -0.49
X8 150 -1.12 N9 176.2 0.53
Number of results 63
Median 167.8
Median MU 2.5
First Quartile 150.0
Third Quartile 197.0
IQR 47.0
15.9
CV (%) 9.5
Minimum 120.0 (60.0)
Maximum 213.3 (258.1)
Range 93.3 (198.1)
Note: A # indicates an outlier where the z-score obtained is either greater then 3 or less than -
3. Codes for all participates are shown. The results column shows a blank entry for those
participants that did not submit a result for this test. Minimum, Maximum and Range are
calculated with outliers excluded, those in brackets include outliers.
Code
Test
Result
%
Z Score Code
Test
Result
%
Z ScoreCode
Test
Result
%
Z Score
4.4 CBR - Sample B - Set SD: Z - Scores
Statistic Value
Set SD
CBR Proficiency Testing Program - 2016(67)
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 46 of 80
5. Program information 5.1 Z score Summary The proficiency program was conducted in September/October 2016. A ‘Z-score Summary’ summary was issued on the 12 December 2016. A copy was e-mailed to all participants who submitted results. The summary is intended as an early indicator of participant performance. This program report supersedes the z - score summary. Further information can be found in section 5.9 ‘Statistics’. The z-scores generally do not vary significantly between the “summary” and the “Final Report”. Following review of the submitted results several results were rejected but it did not significantly affect the statistics. 5.2. Program Design 5.2.1 Design
This program is one of a series of CBR programs conducted by LabSmart Services over the last ten years. Proficiency testing programs have shown that the CBR test produces a wide variation in results. Part of the design of each program involves asking for the right information. The correct analysis of the data collected then allows feedback to be offered to enable participants to improve in the performance of this test. The program was designed to provide technical feedback regarding performance as well as possible improvements in performance. Other considerations involving the design of the program are detailed below. 5.2.2 Selection of material for the program
The test in this proficiency program is operator skill/experience dependant. Different materials are selected for each program to mirror the range of materials encountered in practice and hence the results obtained. This program provides a sample that gives results in the range that would be commonly tested by laboratories. The higher the CBR value the greater the variation encountered. Although a lower CBR material was desirable preliminary testing can be misleading as shown by the program outcome. For this material CBR results varying from 80 to 250% were possible. Comments under “Comparing Statistics” should be considered. It is expected that the level of experience/skill need to perform these tests will present a reasonable assessment of the overall competency of the tester and industry performance.
CBR Proficiency Testing Program - 2016(67)
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 47 of 80
5.2.3 OMC & MDD
The determination of OMC and MDD is usually an initial stage undertaken prior to performing a CBR test. The determination of these two parameters can show a significant variation. In turn having an impact on the variation obtained for CBR results. The intention of the program is to minimise the influence on the CBR results that could arise from laboratories determining these values in-house and reduce the likelihood of different OMC and MDD values being applied. To assist in reducing this variation, participants were requested to use 100% standard compaction and use:
OMC = 8.6%
MDD = 2.260 t/m3. Although this has been the approach to try and minimise variation other aspect may still contribute to the variation observed. OMC/MDD values may vary from person to person but this may not be so important if the same person determines OMC/MDD and CBR. That is a low compaction on the OMC/MDD should give the same compaction on the CBR. Overall it is still considered that a set OMC/MDD will contribute the least variation. 5.2.4 Role of proficiency testing
The determination of outliers is an important task of this proficiency program. A secondary function is to provide feedback that can help those with outliers identify possible areas to investigate as well as assist all participants to improve. In addition to the statistics, proficiency programs often obtain other information that is not normally available to a laboratory. It allows for a better understanding of the testing and can provide information that can lead to improvements in the testing process or test method. Proficiency testing enables participants to measure competency against others. It is also a measure of staff performance and the equipment used. Apart from ‘measurement uncertainty’ it is the next most useful tool a laboratory has in better understanding the performance of a test. 5.2.5 Participant assessment
Assessment of each participant is based on a z-score that is related to the program consensus value (median). This is used to determine any statistical outliers. Compliance to proficiency program requirements including the correct calculation of results and adherence to program and test method requirements may also be used as part of the assessment process. Participants may also be asked to investigate any discrepancies detected with the paperwork submitted.
CBR Proficiency Testing Program - 2016(67)
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 48 of 80
2.2.6 Reporting of results - Significant figures
The number of decimal places (significant figures) reported for a test has a bearing on the statistical analysis and therefore the interpretation of the results. There is a need to strike a balance between what is desirable from a statistical viewpoint and test method accuracy while recognising how the results are used in practice. Too few decimal places (e.g. due to rounding) can cause an increase in the observed spread of results. Increasing the number of decimal places (with respect to normal reporting) can distort the observed spread of results compared to that encountered in actual practice. Large numbers of similar, rounded results can also cause a distortion in the analysis. For example rounding to 10 % means that any number between 45 and 55 will become 50%. If the largest value is 45 in a set of results it is pushed out to 50 through rounding. Rounded results may better reflect the repeatability and reproducibility of the test according to the rounding in the test method but are not as useful when considering laboratory performance. For this program, it was decided that the benefits of using additional decimal places would complement the aim of the proficiency program. Participants results were analysed as received regardless of whether there were more or less significant figures than the number requested by the program. 2.2.7 Additional information requested
This program requested additional information as detailed in Section 6 not usually reported. The additional information is however consistent with the performance of the test and the records the test method requires laboratories to maintain. The additional information is used to interpret participant’s performance and assist with providing technical comment including feedback on outliers and possible participant improvement.
5.3. Sample preparation Sufficient material of a homogeneous appearance was obtained for the proficiency program. The lot was partially dried then mixed to ensure, as far as possible, a homogeneous material throughout. The material was sampled and placed into numbered plastic bags. Ten samples were drawn at regular intervals from the lot for homogeneity testing. Each participant received a randomly drawn sample from the remaining samples. A unique program code was assigned to each sample.
5.4. Packaging and instructions Each plastic bag was sealed with a rubber band and placed into a sturdy box. Each participant received one box with a sealed sample labelled ‘2016 (67) CBR Sample’.
CBR Proficiency Testing Program - 2016(67)
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 49 of 80
The sample weighed approximately 14.8 kg. Instructions and a ‘results log’ sheet were enclosed (See Appendix A & B). Participants were instructed to test according to the nominated test method and report to the accuracy indicated on the ‘Results Log’.
5.5. Quarantine There were three samples that required additional preparation to meet WA quarantine requirements.
5.6. Sample despatch Samples were dispatched to participants between the 13 & 16 September 2016 via courier. Dispatched samples were tracked from ‘despatch to delivery’ for each participant.
5.7. Homogeneity testing Homogeneity samples were selected, evenly spaced, from the prepared participant samples. Samples for homogeneity testing were packed in the same way as those for all participants. The homogeneity samples were tested by an independent NATA accredited laboratory. To approximate the same conditions the same instructions were given to the laboratory performing the homogeneity testing. Ten samples were tested for homogeneity. The wide variation in results that arise from CBR testing can also occur with homogeneity testing particularly at high CBRs. One of the ten samples (H3) was very low and rejected on a statistical level. The result may have been affected by the soaking process. The overall variability associated with the homogeneity samples was considered satisfactory. The average of the homogeneity samples also lies within 1 s.d of the participant’s median value. This provides confidence that any outliers identified in the program represent statistically valid outliers. A statistical analysis of the homogeneity testing results is provided in table 5.7A.
5.8. Participation Seventy participants from around Australia entered the program. Sixty-five participants returned results. Participants were requested to return results by 12 October 2016.
5.9. Statistics Z-Scores were calculated for each test and used to assess the variability of each participant relative to the consensus median. A corresponding z-score graph was produced for each test.
CBR Proficiency Testing Program - 2016(67)
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 50 of 80
The use of median and quartiles reduces the effect that outliers have on the statistics and other influences. Therefore, z-scores provide a more realistic or robust method of assessment.
Code
Bearing Ratio
CBR Rounded
CBR 2.5 mm 5.0 mm
% %
H1 101.7 136.0 136.0 140
H2 108.2 145.1 145.1 150
H3 R R
H4 112.4 150.6 150.6 150
H5 95.9 123.9 123.9 120
H6 101.1 135.8 135.8 140
H7 104.9 138.2 138.2 140
H8 101.8 137.7 137.7 140
H9 99.1 131.6 131.6 130
H10 94.1 122.3 122.3 120
Mean 102.1 135.7 135.7 137
Standard Deviation 5.7 9.1 9.1 11
Range 30.6 46.4 46.4 50
Coefficient of Variation (%) 5.6 6.7 6.7 8
Table 5.7A Homogeneity results
Some results were reported by participants to more decimal places than requested as part of the proficiency program and by others to fewer decimal places. In all instances test results have been used as submitted by participants. Assessment of participant’s data is undertaken to ensure data is statistically comparable. Checks are undertaken to ensure the data calculated matches that reported by the participant and that the appropriate corrections etc. have been applied if required. The level of checking required varies from program to program. If inconsistencies are identified the data may be removed or amended with the discrepancy highlighted.
A z-score is one way of measuring the degree of consensus with respect to the grouped test results. The z-scores in this report are an approximate of the standard deviation. For each test a z-score graph is shown. Use the graph to visually check statistically how you compare to other participants.
CBR Proficiency Testing Program - 2016(67)
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 51 of 80
The following bar (Figure 5.9A) is shown at the bottom of each graph. This helps to quickly visualize where each participant’s results falls.
Review Weak
Consensus Strong Consensus
Weak Consensus
Review
Figure 5.9A Z-score interpretation bar
For example:
A strong consensus (i.e. agreement) means that your test result is close i.e. within
1 standard deviation of the median.
A weak consensus means that your test result is satisfactory and is within 2
standard deviations of the median.
If you have obtained a test result that is outside 2 standard deviations then it may be
worth reviewing your testing processes to ensure that all aspects are satisfactory.
Only those obtaining a z-score approaching 3 (I.e. outside 2.75 range) have been highlighted in the report for review.
If you have obtained a test result that is outside 3 standard deviations then you will need to investigate your testing processes to ensure that all aspects are satisfactory. Participant assessment is not based purely on statistical analysis. Compliance to proficiency program requirements including the correct calculation of results and adherence to program requirements may also be used as part of the assessment process. Participants may also be asked to investigate any discrepancies detected with the paperwork submitted. For further details on the statistics used in this proficiency program can be obtained from LabSmart Services or download the ‘Participant Guide’ from the LabSmart Services website. 5.9.1 Z-score summary
A “Z-Scores Summary” is issued soon after most results are received. It gives participants early feedback as to any program outliers. The summary is available on the LabSmart Services website up until the final report is issued. The final report supersedes the z-score summary. The final report contains detailed technical feedback regarding the performance of tests and revised z-scores. The inclusion of late results or corrections are at the discretion of the program coordinator. In some instances, this may change some of the z-scores slightly but generally the performance outcome remains the same. If there is any impact it will be discussed within section 5.1 of the report. 5.9.2 Comparing statistics from one program to another
The statistics generated from one proficiency program are not usually comparable to those from another proficiency testing program. Only very general comparisons may be possible. The reason statistics from one program may not be compared to another is due to the range of variables that differ from one proficiency program to another.
CBR Proficiency Testing Program - 2016(67)
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 52 of 80
These variables include:
Type of material selected,
The number of participants,
Experience of participants,
Test methodology variations,
Equipment used,
Test methods used,
Experience of supervisors,
Range of organisations involved.
Program design and the statistics employed. The program outcome represents a ‘snap shot’ of the competency within the industry and hence provides an overview of the industry. The more participants involved in the program then the more representative the overview. 5.9.3 Measurement uncertainty
The statistics detailed in this program do not replace the need for laboratories to separately calculated measurement uncertainties associated with each test when required by the client or NATA. The proficiency program does give information useful for calculating the MU and bench marking the MU calculated. 5.9.4 Metrological traceability
The assigned median value used in this proficiency testing program is derived from participant performance and is not metrologically traceable.
CBR Proficiency Testing Program - 2016(67)
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 53 of 80
This page has been left blank intentionally for formatting purposes.
CBR Proficiency Testing Program - 2016(67)
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 54 of 80
16. Please retain the completed “Results Log” as this contains your confidential
participation code. You will need this code to identify your results in the technical report covering the proficiency testing program. It is also recommended that a copy of completed worksheets be kept with the results log in your proficiency file.
17. Proficiency testing can also form part of a laboratories training records for the technician who performed the test.
Thank you for participating in this proficiency testing program.
Page 2 of 2
CBR Proficiency Testing Program - 2016(67)
Copyright: LabSmart Services Pty Ltd Report Issued - February 2017 Page 76 of 80
67 App B CBR PT Results Log.docx
LabSmart Services
Proficiency Testing Program - California Bearing Ratio – 2016 (67)
RESULTS LOG for xxxx
Participation Code: xx
Please fax or e-mail the completed results log by 12 October 2016
------------------------------------ ---------------------------------- --------------- Supervisor Name (Please Print) Signature Date
In signing the above, I acknowledge that the above results are approved and have been checked. I will also ensure that the results are kept confidential both internal and external to the laboratory until the issue of the final technical report covering this program.
Thank you for participating. Please retain these sheets for your records.