Interlaboratory test performance study (TPS) for the ...

1

Interlaboratory test performance study (TPS) for the evaluation of molecular methods

to detect Xylella fastidiosa in the vector Philaenus spumarius

Ref.: 17-XFAST-EU

May 2017-February 2018

2

CONTENT:

1 GENERAL INFORMATION ................................................................................................................................ 3

1.1 Objectives ................................................................................................................................................ 3

1.2 Organizers ................................................................................................................................................ 5

1.3 Participating Laboratories ....................................................................................................................... 5

1.4 Documents and instructions .................................................................................................................... 7

1.5 Timeline of the test performance study .................................................................................................. 7

2. PANEL OF EXPERIMENTAL SAMPLES ............................................................................................................. 7

3. DIAGNOSTIC PROCEDURES EVALUATED ........................................................................................................ 9

4. ANALYSIS OF THE RESULTS .......................................................................................................................... 15

5. RESULTS ....................................................................................................................................................... 17

5.1 Homogeneity and stability..................................................................................................................... 17

5.1.1 Homogeneity .................................................................................................................................. 17

5.1.2 Stability ........................................................................................................................................... 21

5.2 RESULTS OF THE TPS .............................................................................................................................. 23

5.2.1 On spiked insects ............................................................................................................................ 23

5.2.2 On natural insects ........................................................................................................................... 38

6. SUMMARY OF THE OVERALL VALUES RECOVERED FOR EACH PERFORMANCE CRITERIA .......................... 41

7 DISCUSSION AND CONCLUSION ................................................................................................................... 42

8. REFERENCES ................................................................................................................................................. 44

8. ACKNOWLEDGMENTS.................................................................................................................................. 44

Annex I ............................................................................................................................................................. 45

Annex II ............................................................................................................................................................ 46

Annex III ........................................................................................................................................................... 51

Annex IV ........................................................................................................................................................... 55

This report has been prepared based on the analyses of the results received by all participant

laboratories.

Authors:

Bruno Legendre, Anses

Dimitri Molusson, Anses

Valerie Olivier, Anses

Aude Chabirand, Anses

Françoise Poliakoff, Anses

3 1 GENERAL INFORMATION

1.1 Objectives The test performance study (TPS) is a way in which the performances of one or more methods are

assessed.

For this test, standardized samples were prepared with known status regarding the presence of the target

pathogen. In addition naturally infected samples were also used.

These were sent out to participating laboratories that analyze them using the provided protocols. The

organizer analyzed the results and provided a report detailing all participants’ results in confidential

manner together with actual sample status.

This test performance study aims to assess the efficiency and accurateness of different molecular methods

used for the detection of the bacterium Xylella fastidiosa into the insect vector Philaenus spumarius. The

study is part of the research activities aiming at implementing and validating reliable and sensitive

detection procedures for X. fastidiosa in the framework of the following ongoing European projects:

- EUPHRESCO project (2015-F-146) “Harmonized protocol for monitoring and detection of Xylella

fastidiosa in its host plants and its vectors”

- H2020 “POnTE – Pest Organisms Threatening Europe (635646)” – Work package 4: “Implementation

and validation of diagnostic kits for early and rapid detection of target pathogens in host plants and

vectors”

This test performance study has been organized in accordance with the EPPO 7/122 guideline and the

following performance criteria have been considered in the data analyses:

Diagnostic sensitivity: proportion of positive samples giving a positive result.

Diagnostic specificity: proportion of negative samples giving a negative result.

Accuracy: the accuracy is the closeness of agreement between a test result and the accepted

reference value.

Repeatability (or accordance): level of agreement between replicates of a sample tested under the

same conditions.

Reproducibility: ability of a test to provide consistent results when applied to aliquots of the same

sample tested under different conditions (time, persons, equipment, location, etc.).

4

Limit of detection (analytical sensitivity): Smallest amount of target that can be detected reliably in matrix.

The diagnostic methods evaluated during this TPS included several molecular approaches described

in the EPPO diagnostic protocol 7/24 (2). The detailed laboratory procedures tested in the different

laboratories are reported in the technical sheet in Annex I. The samples were processed either

manually or, according to the kit or protocol, using automatized platforms.

IMPORTANT NOTES: In order to provide to all laboratories the same materials and to avoid any risk

related to the movement of infective materials, samples were prepared by spiking inactivated

bacterial suspensions of Xylella fastidiosa subsp. fastidiosa (ATCC35879) into crushed insects

prepared according to each extraction method. This condition may have affected the yield of

bacterial target DNA using the different extraction methods. Thus, the results obtained may not

reflect the performances of these methods achieved when using fresh infected samples.

5

1.2 Organizers This study has been conceived and promoted by the French Agency for food, environmental and

occupational health and safety (Anses), Plant health laboratory, Bacteriology Virology and GMO unit,

Angers (France), and implemented in collaboration with Anses - Plant health laboratory, RAPT, Saint Pierre

de La Réunion, France.

1.3 Participating Laboratories Participant laboratories are listed below. During the test they have been identified with an anonymous

identification code, ensuring results confidentiality.

Laboratories Country

AGES, Department for Molecular Diagnostics of Plant Diseases Austria

Biology and Ecology of Soil Microbiota, Institute for Sustainable

Agriculture ( CSIC) Spain

Central laboratory for plant quarantine Bulgaria

Centro di difesa e certificazione, CREA Italy

CIHEAM-IAMB Italy

Danish Veterinary and Food Agency, Ringsted Laboratory Denmark

DEFRA (FERA Science LTD) United Kingdom

Departement for Agriculture and Forestry Science (DAFNE),

University of Tuscia Italy

French Agency for food, environmental and occupational health

and safety (Anses), LSV, UBVO France

ILVO Plant Health Belgium

Instituto por la protezione sostenibile, CNR

UNIBA, Dipartimento di Scienze del Suolo, della Planta e delli

Alimenti

Italy

Institute Valenciano de investigaciones agrarias (IVIA),

Bacteriology Spain

Julius Kuehn-Institute - (JKI), Institute for national and

international plant health, Braunschweig Germany

6

Laboratories Country

Julius Kühn-Institut (JKI), Institute AG, Kleinmachnow Germany

Laboratorio Oficial de Sanitad Vegetal de las Islas Baleares (OSVIB) Spain

National institute of agrarian and veterinarian research (INIAV) Portugal

Plant Health Laboratories, Department of agriculture, food and the

marine Ireland

Science and Advice for Scottish Agriculture (SASA) United Kingdom

USDA-ARS USA- California

WUR, Wageningen Plant Research The Netherlands

7

1.4 Documents and instructions Participants received the following documents containing the information for the contract and the

instructions describing the protocols and the panel of samples to be used for each diagnostic method.

Technical sheet (Annex I)

Participant contract (Annex II)

Excel files with spreadsheet to register and report the results (Annex III)

1.5 Timeline of the test performance study The panels of samples were prepared from September 13th to October 03rd 2017; this also included the

panel of samples to be used for the homogeneity and stability tests.

The homogeneity tests for all diagnostic methods considered in this TPS were performed on the October

04th and 12th 2017, immediately after preparing the different batches of spiked samples.

The stability tests were performed December 11th, 12th and 13th, 2017.

Aliquoted samples were then kept at -20°C prior to be shipped. Shipment was organized during the week of

23 - 27 October, 2017.

Participants were requested to perform the selected diagnostic tests and send the results to the organizers

by the week 48 (November 27th – December 1st), 2017.

All the raw data (qualitative results: positive, negative) received from the different participating

laboratories were collected in separate excel files.

This final report was delivered in February, 2018.

2. PANEL OF EXPERIMENTAL SAMPLES Spiked samples

The samples consisted of crushed heads of insects (Philaenus spumarius collected from Xylella-free areas)

macerated in the appropriate extraction buffers according to the different protocols under evaluation:

- one crushed head in 200 µL demineralized sterile water for the QuickPick™ extraction

- one crushed head in 500 µL CTAB buffer for the CTAB extraction.

Samples have been spiked with a calibrated heat inactivated bacterial suspension in water of Xylella

fastidiosa subsp. fastidiosa (ATCC35879). The panel included 13 samples with three different bacterial

concentrations (5.103 bact./head, 5.104 bact./head and 5.105 bact./head), Xylella free samples and one lure.

8

The panel of samples included:

Type of samples Number of replicates

5.105 bact./head 2

5.104 bact./head 2

5.103 bact./head 4

Healthy sample 4

Plus one tube of Positive Amplification Control (PAC)

Samples were aliquoted and shipped in microcentrifuge safe lock tubes (2 mL) and sent frozen with ice

pack. Any laboratories reported any sample degradation or apparent alteration.

Naturally infected and no-infected samples

Panel of 20 natural samples consisted in:

- 15 insects (Philaenus spumarius) produced in containment facilities of CNR and sent to organizer

by Maria Saponari, CNR, Italy and

- 5 insects collected in Xylella free areas (sent to organizer by Maria Saponari, CNR, Italy; Michael

Maixner, Julius Kühn-Institut, Germany; Gudrun Strauss, AGES, Austria; FREDON Nord-Pas-de-Calais,

France).

Insects were stored in 96% (v/v) ethanol at -20°C.

Insects were shipped in microcentrifuge safe lock tubes (2 mL) and sent frozen in 96% (v/v) ethanol.

Panels were provided for the 3 following extraction procedures: Yaseen et al, 2015 extraction, CTAB

extraction and Bio-Nobile QuickPick™ extraction.

Maria Saponari (CNR, Italy) who provided insects from contaminated area into 2 batches, estimated the

contamination rate of her specimens of about 15% for the first batch and > at 40% for the second batch.

The part of 15 specimens of the panels were composed of 12 specimens from the first batch (estimated

contamination rate = 15%) and of 3 specimens of the second batch (contamination rate estimated > 40%).

So the theoretical contamination rate of the panels is estimated > 19%, so an average > 2.85 positive

insects per panel. In addition, the contamination rate was also appreciated trough the homogeneity and

stability tests results (section 5.1).

9

3. DIAGNOSTIC PROCEDURES EVALUATED Molecular tests were consisted of:

DNA extraction:

QuickPick™ Plant DNA kit (Bio-Nobile)

CTAB DNA extraction protocol (EPPO PM7/24)

Yaseen et al., 2015 DNA extraction (Tris HCl; Triton x-100; EDTA extraction buffer) (for natural

insects only)

Amplification:

real time PCR Harper et al., 2010 (erratum 2013)

real time PCR Harper et al., 2010 (erratum 2013) in duplex with primers Ioos et al., 2009 (internal

control)

real time PCR Francis et al., 2006

PCR Francis et al., 2006 using SYBR dying

LAMP Harper et al., 2010 modified by Yaseen et al., 2015

10

Table 1. Laboratory following procedures for conducting analysis for DNA extraction and real time PCR and/or LAMP on spiked crushed insects

Laboratory

Real-Time PCR SYBR LAMP

Harper et al, 2010 Duplex

Harper et al, 2010 Ioos et al., 2009

Francis et al., 2006 Francis et al., 2006 Harper et al, 2010 modified

by Yaseen et al., 2015

QuickPickTM

CTAB QuickPickTM

CTAB QuickPickTM

CTAB QuickPickTM

CTAB QuickPickTM

CTAB

L01 X (StratageneMX

3000)

L02 X (AB Step One Plus)

X (AB Step One Plus)


L03 X (AB 7500 Fast) X (AB 7500 Fast) X (AB 7500 Fast)

X (Genie® II (Optigène))




L05 X (AB 7900HT) X (AB 7900HT) X (AB 7900HT) X (AB 7900HT) X (AB 7900HT) X (AB 7900HT) X (Genie® II

(Optigène)) X (Genie® II (Optigène))

L06 X (BioRad CFX96)

X (BioRad CFX96)

X (BioRad CFX96)

X (BioRad CFX96) X (BioRad CFX96)

X (BioRad CFX96)

X (BioRad CFX96)

X (BioRad CFX96)

L07 X (AB 7900) X (AB 7900) X (BioRad

CFX96) X (BioRad CFX96) X (AB 7900) X (AB 7900) X (BioRad

CFX96) X (BioRad

CFX96)

L08 X (BioRad

C1000 + CFX96) X (BioRad

C1000 + CFX96) X (BioRad

C1000 + CFX96)

X (Genie® II (Optigène))

L09 X (Bioneer

Exicycler96) X (Bioneer

Exicycler96) X (Bioneer

Exicycler96)

L10 X (Roche Lightcycler 480

I)

X (Roche Lightcycler 480

I)


I)

X (Roche Lightcycler 480 I)


I)


I)


I)


I)

L11 X (Eppendorf Realplex 4

Mastercycler S)

X (Eppendorf Realplex 4

Mastercycler S)


Mastercycler S)

X (Eppendorf

Realplex 4 Mastercycler S)


X (BioRad CFX96)

X (BioRad CFX96)


X (BioRad CFX96)

X (BioRad CFX96)

X (BioRad CFX96)

X (BioRad CFX96)

X (BioRad CFX96)

L13

X (ECO-ILLUMINA)

X (ECO-ILLUMINA) X (ECO-ILLUMINA)

X (ECO-ILLUMINA) + (ICGENE mini (Embiotech)

11

L14 X (AB QanStudio 5)

X (AB QanStudio 5)

X (AB QanStudio 5)

X (AB QanStudio 5)

X (AB QanStudio 5)

X (AB QanStudio 5)

L15 X (AB 7500) X (AB 7500) X (AB 7500) X (Genie® II

(Optigène))


X (BioRad CFX96)

X (BioRad CFX96)

X (BioRad CFX96)

X (BioRad CFX96)

X (BioRad CFX96)

L17 X (Qiagen

RotorgeneQ) X (Qiagen

RotorgeneQ) X (Qiagen

RotorgeneQ)

L18 X (BioRad CFX) X (BioRad CFX) X (BioRad CFX) X (BioRad CFX)

L19 X (BioRad

CFX96) X (BioRad CFX96) X (BioRad

CFX96)

X (ICGENE mini

(Embiotech)

L20 X (AB Step One

Plus) X (AB Step One

Plus) X (AB Step One

Plus)

X (AB Step One

Plus)

12

Table 2. Laboratory following procedures for conducting analysis for DNA extraction and real time PCR and/or LAMP on natural insects (DNA extraction QuickPick™ Plant DNA kit and/or CTAB)

Laboratory

Real-Time PCR SYBR LAMP

Harper et al, 2010 Duplex

Harper et al, 2010 Ioos et al., 2009

Francis et al., 2006 Francis et al., 2006 Harper et al, 2010 modified by Yaseen et al., 2015

QuickPickTM CTAB QuickPickTM CTAB QuickPickTM CTAB QuickPickTM CTAB QuickPickTM CTAB

L01

L02

L03





L05 X (AB 7900HT) X (AB 7900HT)

X (AB 7900HT) X (AB 7900HT)

X (AB 7900HT) X (AB 7900HT)

X Genie® II (Optigène)

X Genie® II (Optigène)


X (BioRad CFX96)

X (BioRad CFX96)

X (BioRad CFX96)

X (BioRad CFX96)

X (BioRad CFX96)


L07 X (AB 7900) X (AB 7900) X (BioRad CFX96)

X (BioRad CFX96)

X (AB 7900) X (AB 7900) X (BioRad CFX96) X (BioRad CFX96)

L08

L09

L10 X (Roche Lightcycler 480

I)

X (Roche Lightcycler

480 I)


I)


480 I)


I)


480 I)



L11 X (Eppendorf Realplex 4

Mastercycler S)


Mastercycler S)


Mastercycler S)


Mastercycler S)


X (BioRad CFX96)

X (BioRad CFX96)

X (BioRad CFX96)

X (BioRad CFX96)

X (BioRad CFX96)

X (BioRad CFX96)

X (BioRad CFX96)


L13 X (ECO ILUMINA)

X (ECO ILUMINA)

X (ECO ILUMINA)

X (ECO ILUMINA) + (ICGENE mini

(Embiotech)

L14

L15


X (BioRad CFX96)

X (BioRad CFX96)

X (BioRad CFX96)

X (BioRad CFX96)

X (BioRad CFX96)

L17

L18

L19

L20

13

Table 2 bis. Laboratory following procedures for conducting analysis for DNA extraction and real time PCR

and/or LAMP on natural insects (DNA extraction protocol Yaseen et al., 2015)

Laboratory

LAMP

DNA extraction protocol Yaseen et al., 2015 Harper et al, 2010 modified by Yaseen et al., 2015

Thermocycleur Portable device

Genie® II (Optigene)

portable device ICGENE mini (Embiotech)

L01

L02

L03

L04 X X

L05 X

L06

L07 X

L08

L09

L10 X X

L11

L12 X X

L13 X X

L14

L15

L16 X

L17

L18 X X

L19

L20

14

Table 3. DNA extraction with QuickPick™ Plant DNA kit with robot or by hand

Laboratory

QuickPick™ extraction with robot or by hand

L01 Robot

L02 By hand (with magnet pipette)

L03 Not applicable


L05 Robot KingFisher™ mL

L06 By hand (with magnetic rack)

L07 Robot BioSprint 15

L08 Not applicable

L09 Not applicable



L12 Robot KingFisher™ mL

L13 Not applicable


L15 Robot


L17 Not applicable

L18 By hand (with magnetic rack / magnet pipette)

L19 Not applicable

L20 Not applicable

15 4. ANALYSIS OF THE RESULTS

Diagnostic results were primarily based on qualitative data, i.e. each sample was assigned to one of the

following categories: negative or positive, based on the interpretation criteria for each diagnostic method.

Upon receiving all results, for each sample the following values were assigned after decrypting the sample

code:

PA= positive agreement; PD= positive deviation; ND = negative deviation; NA = negative agreement

Table 4. Definition of the parameters adapted from ISO 16140

Laboratory results Assigned value

Positive Negative

Positive PA= positive agreement PD= positive deviation

Negative ND= negative deviation NA= negative agreement

These assigned values were then used to retrieve the performance criteria evaluated as a function of the

qualitative results and expressed as percentage values (please see Chabirand et al., 2014 for more detailed

information).

Table 5. Details on the performance criteria

Performance criteria Definition Calculation

Sensitivity (SE) Closeness of agreement between the laboratory result and the assigned value for samples for which the assigned value is positive

SE= NPA/N+

Specificity (SP) Closeness of agreement between the laboratory result and the assigned value for samples for which the assigned value is negative

SP=NNA/N-

Accuracy (AC) Closeness of agreement between the laboratory result and the assigned value

AC= (NPA+NNA)/N

Repeatability (DA) Closeness of agreement between independent test results obtained under conditions of repeatability, i.e. conditions under which independent test results are obtained by the same method, on identical test samples in

DA denotes the percentage chance of obtaining the same result (positive, negative or indeterminate) from two identical samples analyzed in the same laboratory

16

the same laboratory, by the same operator, using the same equipment, within a short period of time

Reproducibility The reproducibility is the probability of finding the same result for two identical samples analyzed in two different laboratories

The % of reproducibility is calculated based on the number of interlaboratory pairs of same results/total number of interlaboratory pairs

17 5. RESULTS

5.1 Homogeneity and stability The homogeneity and the stability were assessed for the artificially, naturally contaminated and healthy

samples.

For studying the homogeneity and the stability of the spiked samples, the real time PCR Harper et al., 2010 have

been used after DNA extraction using the QuickPick™ Plant DNA kit and the CTAB protocol.

For studying the homogeneity and the stability of the natural samples, the real time PCR Harper et al., 2010

have been used after DNA extraction using the QuickPick™ Plant DNA kit.

Homogeneity and stability detailed calculation are presented in Annex IV.

5.1.1 Homogeneity

For the spiked samples, the homogeneity study covers 10 samples at the concentration of 5.103 bact./head, 10

healthy samples, 5 samples at the concentration of 5.104 bact./head and 5 samples at the concentration of 5.105

bact./head.

For evaluating the average rate of contamination of the natural insects, 3 panels of 20 insects have been

analyzed.

18

Table 6 Homogeneity results for spiked samples using the QuickPick ™Plant DNA kit and real

time PCR Harper et al., 2010

Panel QuickPick™ Ct Value Average Standard deviation

Healthy sample N/A Negative




















5.103 b/head 31,32 Positive




















30,77 0,45

N/A N/A

Panel QuickPick™ Ct Value Average Standard deviation





















0,1727,20

23,62 0,15

19

Table 6 bis Homogeneity results for spiked samples using CTAB protocol and real time PCR

Harper et al., 2010

According to the ISO 13528 standard, all the quantitative and qualitative results obtained with the spiked

samples are homogeneous, either with the QuickPick™ extraction or either with the CTAB extraction, regardless

of the different target concentrations. Detailed calculations are presented in Annex IV.

Panel CTAB Ct Value Average Standard deviation









































32,58 0,24

N/A N/A

Panel CTAB Ct Value Average Standard deviation





















28,90 0,14

25,48 0,17

20

Table 7 Homogeneity results for natural panels using the QuickPick ™Plant DNA kit and real

time PCR Harper et al., 2010

Homogeneity results for natural samples gave 12 positive insects out of a total of 45 potentially positive insects,

so a contamination rate of 26.67%. The assigned value being not known for the naturally infected samples, the

homogeneity of the panel is not verifiable according to ISO 13528 standard.

Panel 2 QuickPick Ct 1 Ct 2 Result

Healthy sample N/ A N/ A Negative





Proba + about 40% N/ A N/ A Negative


Proba + about 40% 38,32 N/ A Negative


Proba + about 15% 30,75 30,65 Positive





















































21

5.1.2 Stability

Stability tests were conducted after the deadline indicates for the results delivery by each participant

laboratory.

For the spiked samples, the stability covers 3 samples at the concentration of 5.103 bact./head, 2 samples at the

concentration of 5.104 bact./head, 2 samples at the concentration of 5.105 bact./head and 3 healthy samples.

For the stability study on natural insects, 2 panels of 20 insects have been analyzed.

Table 8 Stability results for spiked samples using the QuickPick™ Plant DNA kit and real time PCR Harper et al.,

2010

Table 8 bis Stability results for spiked samples using the CTAB protocol and real time PCR Harper et al., 2010

According to the ISO 13528 standard, all the quantitative and qualitative results obtained with the spiked

samples are stable, either with the QuickPick™ extraction or either with the CTAB extraction, regardless of the

Panel QuickPick Ct 1 Ct 2 Result Average

Standard

deviation

Healthy sample N/A N/A Negative N/A N/A



5.103 b/head 30,53 30,49 Positive

5.103 b/head 32,17 32,14 Positive

5.103 b/head 31,34 31,17 Positive

5.104 b/head 27,65 27,54 Positive

5.104 b/head 27,18 27,2 Positive

5.105 b/head 23,89 23,89 Positive

5.105 b/head 23,32 23,22 Positive

31,31 0,67

27,39 0,21

23,58 0,31

Panel CTAB Ct 1 Ct 2 Result Average

Standard

deviation





5.103 b/head 34,12 34,37 Positive

5.103 b/head 32,11 32,27 Positive


5.104 b/head 27,94 27,95 Positive

5.105 b/head 24,63 24,73 Positive

5.105 b/head 24,55 24,44 Positive

32,85 0,99

28,04 0,10

24,59 0,11

22

different target concentrations. Detailed calculations are presented in Annex IV.

Table 9 Stability results for natural samples using the QuickPick™ Plant DNA kit and real time PCR Harper et al.,

2010

Stability results for naturally infected samples gave 7 positive insects on a total of 28 potentially positive insects,

so a contamination rate of 25%. The assigned value being not known for the naturally infected samples, the

stability of the panel is not verifiable according to ISO 13528 standard. However no significant differences are

observed between the contamination rate calculated for the homogeneity (26.5%) study and the rate calculated

for the stability study (25.0%). However this rate of contamination is higher than the theoretical expected one

with a rate of about 19% (section 2).


Healthy sample N/A N/A Negative





Proba + about 40% N/A N/A Negative




Proba + about 15% 39,02 N/A Negative

























Proba + about 15% 37,2 N/A Negative

Proba + about 15% 39,94 39,17 Negative




23

5.2 RESULTS OF THE TPS It is important to note that the following rules have been applied to determine the positivity threshold of

the samples for the different methods:

Method Cut-off rule Other rule Real time PCR Harper et al. 2010 Sample positive if Ct value ≤ 38 -

Duplex Real time PCR Harper et al. 2010, Ioos et al., 2009

Sample positive if Ct value ≤ 38 -

Real time PCR Francis et al. 2006 Sample Positive if Ct value ≤ 38 -

PCR Francis et al. 2006 using SYBR Green dye Sample positive if Ct value ≤ 35 (as recommended in PM7/24)

Melting peak between 83°C-85°C

Lamp Harper et al. 2010 modified by Yaseen et al., 2015

No cut-off Sample positive if presence of an exponential amplification curve

The following decision rules for determining the positivity or negativity of the samples are also applied:

2 amplification replicates per sample

Result for replicate 1 Result for replicate 2 Final decision

Positive Positive Positive

Positive Undetermined (e.g. Ct value between 38 and 40 for qPCR)

Positive

Positive Negative Negative

Negative Negative Negative

Note: Following the previous rules, results (positive/negative) sent by the participants have needed in some

cases to be harmonized.

5.2.1 On spiked insects

5.2.1. 1 Qualitative results

Diagnostic sensitivity

24

Table 10. Diagnostic sensitivity calculated using the results obtained in real time PCR, PCR (SYBR Green) and

LAMP tests using the DNA extracts prepared following 2 different protocols.

Figure 1. Diagnostic sensitivity calculated using the results obtained in real time PCR tests using the DNA

extracts prepared following 2 different protocols (restricted series: L18 results excluded due to systematic

false negative results).

In the figure 1, the unrepresentative results of Lab18 are excluded of results series because of systematic

false negative results.

Diagnostic sensitivity (%)

QuickPickTM CTAB QuickPickTM CTAB QuickPickTM CTAB QuickPickTM CTAB QuickPickTM CTAB QuickPickTM CTAB

L01 100 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A

L02 100 N/A 100 N/A 87,5 N/A N/A N/A N/A N/A N/A N/A

L03 N/A 100 N/A 100 N/A 100 N/A N/A N/A N/A N/A 100

L04 100 N/A N/A N/A 100 N/A N/A N/A 100 N/A N/A N/A

L05 100 100 100 100 100 100 N/A N/A N/A N/A 100 100

L06 100 100 100 100 50 100 N/A N/A 87,5 87,5 N/A N/A

L07 100 100 100 100 100 87,5 N/A N/A 100 87,5 N/A N/A

L08 N/A 100 N/A N/A N/A 100 N/A 100 N/A N/A N/A 100

L09 N/A 100 N/A 100 N/A 100 N/A N/A N/A N/A N/A N/A

L10 50 100 50 100 62,5 100 N/A N/A 87,5 100 N/A N/A

L11 100 N/A 100 N/A 0 N/A N/A N/A 100 N/A N/A N/A

L12 100 87,5 100 100 0 62,5 25** 37,5 100 62,5 N/A N/A

L13 N/A 100 N/A 100 N/A 100 N/A N/A N/A 100 N/A 87,5

L14* 100 100 100 100 50 87,5 N/A N/A N/A N/A N/A N/A

L15 100 N/A 100 N/A 100 N/A N/A N/A N/A N/A 100 N/A

L16 100 100 N/A N/A 75 100 87,5 100 N/A N/A N/A N/A




L20 N/A 100 N/A 100 N/A 100 N/A N/A N/A 100 N/A N/A

All laboratories 88,46 99,11 84,81 100,00 60,42 95,54 56,25 79,17 82,14 89,58 100,00 97,50

Restricted series*** 95,83 99,11 94,37 100,00 65,91 95,54 56,25 79,17 95,83 89,58 100,00 97,50

N 13 14 10 12 12 14 2 3 7 6 2 5

* Only one replicate for L14

** L12 with cut-off value of 40: amplification performed on DNA dilution because of lack of DNA extract - Melting peak between 81,5 and 82°C

*** L18 results excluded (systematic false negatives)

Duplex Harper et al , 2010 Ioos et

al. , 2009Cut-off = 38

Real-Time PCR SYBR Green

LaboratoryFrancis et al ., 2006

Cut-off = 38

Francis et al ., 2006Cut-off = 35 / Melting peak 83-85°C

Harper et al , 2010Cut-off = 38

LAMP


Harper et al , 2010 modified by Yaseen et a l ., 2015

No cut-off

25

Data show at first look a better sensitivity with CTAB protocol for the majority of the different amplification

methods. Further analysis shows that for the 3 different real time PCR the degradation of the performance

is entirely due to laboratories performing QuickPick™ extraction using a by hand protocol. In that case, the

5 laboratories performing the automated protocol aim 100% of sensitivity (Figure 1 bis).

For the real time PCR using SYBR Green dye, the melting peak has to be in the range 83°C - 85°C (according

to EPPO PM7/24). However the laboratory Lab12 got a melting peak in the range 81.5°C – 82°C, even with

the PAC control. So for this laboratory, the results are validated in these conditions.

Note: It is important to keep in mind that the differences in cut off values for the different methods are

sources to disadvantage (e.g. Real time PCR Francis using SYBR Green dye with a Ct value of 35) or

advantage the sensitivity of the methods (LAMP method without cut off value). The cut off value are those

indicated into EPPO PM7/24 if mentioned. For the real time PCR Francis using SYBR Green dye, when

applying a cut-off of 38, the sensitivity rate rises from 56% to 75% for spiked samples using QuickPick™

extraction and rises from 79% to 83% for spiked samples using CTAB protocol.

In this TPS, the real time PCR Francis et al., 2006 using SYBR Green dye presents a lesser sensitivity than the

real time PCR Harper et al., 2010 (in simplex or duplex) but the results are not really significant due to the

low number of participants (QuickPick™ N = 2 / CTAB N = 3). So, no conclusion must be taken without

caution.

26

Figure 1 bis. Diagnostic sensitivity calculated using the results obtained in real time PCR tests using the DNA

extracts prepared following 2 different protocols (restricted series: L18 results excluded due to systematic

false negative results) – For QuickPick™ extraction: series restricted to the 5 laboratories which have used a

robot.

Real time PCR methods and LAMP method present comparable performance, except real time PCR Francis

et al., 2006 for which the sensibility is lower, especially after QuickPickTM extraction (see. note on cut-off

effect).

27

Tableau 11 Ct value comparison for the real time PCR Harper et al., 2010 - DNA extraction with the

QuickPick™ method versus CTAB method

Quantitative results

Real time PCR Harper et al ., 2010

Laboratory Sample

Ct value average

extract. QuickPick

Standard

deviation Ct

extract. QuickPick

Ct value average

extract. CTAB

Standard deviation

Ct extract. CTAB

∆ Ct [QuickPick-

CTAB]

5.105 bact/head 24,82 0,66 29,27 0,34 -4,45

5.104 bact/head 27,96 0,16 30,82 1,52 -2,86

5.103 bact/head 31,52 0,37 33,73 1,59 -2,21

5.105 bact/head 23,62 0,35 23,37 0,20 0,25

5.104 bact/head 27,45 0,17 26,76 0,10 0,69

5.103 bact/head 31,19 0,55 32,78 2,32 -1,59

5.105 bact/head 25,66 1,05 22,34 0,16 3,32

5.104 bact/head 31,25 0,13 26,21 0,10 5,04

5.103 bact/head 33,24 0,19 30,00 0,19 3,24

5.105 bact/head 23,88 0,21 27,97 0,64 -4,09

5.104 bact/head 27,24 0,15 29,87 2,48 -2,63

5.103 bact/head 30,57 0,50 36,72 1,38 -6,15

5.105 bact/head 26,87 2,69 24,86 0,14 2,01

5.104 bact/head 27,46 0,52 28,28 0,20 -0,82

5.103 bact/head 36,45 2,11 31,33 0,70 5,12

5.105 bact/head 27,77 2,64 22,73 0,28 5,04

5.104 bact/head 28,79 0,85 26,21 0,12 2,58

5.103 bact/head 33,27 0,86 29,54 0,33 3,73

Lab06 results excluded

Lab14

(QuickPick by

hand)

Lab05

(Quickick with

robot)

Lab07

(Quickick with

robot)

Lab10

(QuickPick by

hand)

Lab12

(Quickick with

robot)

Lab16

(QuickPick by

hand)

28

Figure 2 Ct value comparison for the real time PCR Harper et al., 2010 - DNA extraction with the QuickPick™

method versus CTAB method

Tableau 11 bis Ct value comparison for the real time PCR Harper et al., 2010 - DNA extraction with the

QuickPick™ method versus CTAB method – Averages and standard deviations

In the figures 2 are compared the Ct values of the laboratories which performed both QuickPick™ (with

robot N=3; by hand method N=3) and CTAB extractions (N=6). No significant differences are observed with

the real time PCR Harper et al., 2010 regardless of the extraction method performed and the target

concentrations (about minus 0,3 Ct in favor of the CTAB method).

Sample

(Ct value average

extract.

QuickPick)

average

Standard

deviation

(Ct value extract.

QuickPick)

(Ct value

average extract.

CTAB)average

Standard

deviation

(Ct value extract.

CTAB)

∆ Ct [QuickPick-

CTAB] average

5.105 bact/head 25,44 1,51 25,09 2,64 0,35

5.104 bact/head 28,36 1,39 28,03 1,80 0,33

5.103 bact/head 32,71 1,95 32,35 2,44 0,36

29

Diagnostic specificity

Table 12. Diagnostic specificity calculated using the results obtained in real time PCR, PCR (SYBR Green) and


Figure 3. Diagnostic specificity calculated using the results obtained in real time PCR using the DNA extracts

prepared following 2 different protocols.

Diagnostic specificity (%)



L02 100 N/A 100 N/A 100 N/A N/A N/A N/A N/A N/A N/A



L05 100 100 100 100 100 100 N/A N/A N/A N/A 100 100

L06 75 0 0 0 100 100 N/A N/A 100 75 N/A N/A

L07 100 100 100 100 100 100 N/A N/A 100 100 N/A N/A

L08 N/A 100 N/A N/A N/A 100 N/A 100 100 N/A N/A 100


L10 100 100 100 100 100 100 N/A N/A 100 100 N/A N/A


L12 100 100 100 100 100 100 100 100 100 100 N/A N/A

L13 N/A 100 N/A 100 N/A 100 N/A N/A N/A 100 N/A 100

L14* 100 100 100 100 100 100 N/A N/A N/A N/A N/A N/A


L16 75 100 N/A N/A 100 100 75 100 N/A N/A N/A N/A






Restricted series** 97,92 100,00 100,00 100,00 100,00 100,00 87,50 100,00 100,00 100,00 100,00 100,00

N 13 14 10 12 12 14 2 3 7 6 2 5

* Only one amplification replicate for L14

** L06 results excluded (high number of false positives / about 40% on healthy insects)

Portable device

Laboratory

Real-Time PCR SYBR Green LAMP


al. , 2009Cut-off = 38

Francis et al ., 2006Cut-off = 38



No cut-off

Thermocycleur


30

Figure 3 bis. Diagnostic specificity calculated using the results obtained in real time PCR using the DNA

extracts prepared following 2 different protocols (restricted series: L06 results excluded due to high

number of false positive results).

If regarding the figure 3 bis, excluding laboratory L06 which obtained 40% of false positive on healthy

samples, no significant difference of performance are observed between QuickPick™ and CTAB protocols,

except for Francis et al., 2006 SYBR Green method (QuickPick™: N= 2; CTAB: N=3).

31

Accuracy

Table 13. Accuracy calculated using the results obtained in real time PCR, PCR (SYBR Green) and LAMP tests

using the DNA extracts prepared following 2 different protocols.

Figure 4. Accuracy calculated using the results obtained in real time PCR, PCR (SYBR Green) and LAMP tests

using the DNA extracts prepared following 2 different protocols.

Accuracy (%)



L02 100 N/A 100 N/A 91,67 N/A N/A N/A N/A N/A N/A N/A



L05 100 100 100 100 100 100 N/A N/A N/A N/A 100 100

L06 91,67 66,67 72,73 66,67 66,67 100 N/A N/A 91,67 83,33 N/A N/A

L07 100 100 100 100 100 100 N/A N/A 100 91,67 N/A N/A



L10 66,67 100 66,67 100 75 100 N/A N/A 91,67 100 N/A N/A

L11 100 N/A 100 N/A 33,33 N/A N/A N/A 100 N/A N/A N/A

L12 100 91,67 100 91,67 33,33 75 50 58,33 100 75 N/A N/A

L13 N/A 100 N/A 100 N/A 100 N/A N/A N/A 100 N/A 91,67

L14* 100 100 100 100 66,67 91,67 N/A N/A N/A N/A N/A N/A


L16 91,67 100 N/A N/A 83,33 100 83,33 100 N/A N/A N/A N/A


L18 33,33 N/A 33,33 N/A 33,33 N/A N/A N/A 33,33 N/A N/A N/A




Restricted series** 96,21 99,36 95,83 99,24 78,33 97,44 66,67 86,11 98,33 93,33 100,00 98,33

N 13 14 10 12 12 14 2 3 7 6 2 5

* Only one amplification replicate for L14

** L06 results excluded (high number of false positives / about 40% on healthy insects) and L18 results excluded (systematic false negatives)

Laboratory



al. , 2009Cut-off = 38




No cut-off



32

Figure 4 bis. Accuracy calculated using the results obtained in real time PCR, PCR (SYBR Green) and LAMP

tests using the DNA extracts prepared following 2 different protocols (restricted series: L06 results excluded

(high number of false positives) and L18 results excluded (systematic false negatives))

Figure 4 ter Accuracy calculated using the results obtained in real time PCR, PCR (SYBR Green) and LAMP

tests using the DNA extracts prepared following 2 different protocols (restricted series: L06 results excluded

(high number of false positives)and L18 results excluded (systematic false negatives)) – For QuickPick™

extraction: series restricted to the 5 laboratories which have used a robot

33

The accuracy data show similar results with Harper et al.2010 with real-time PCR in simplex and duplex and

with LAMP, whatever the extraction methods CTAB or QuickPick™ Plant DNA kit. In contrast false results

are got by Francis et al. 2006 PCR with Taqman® or SYBR Green mainly after QuickPick™ kit extraction.

These results (figures 4, 4bis, 4ter) show a lab effect on the result reliability: loose of target during the

process, cross contamination. The use of QuickPick™ kit by hand increases a risk of false negative results. So

these points must be considered as very critical in a procedure.

Repeatability

Table 14. Repeatability calculated using the results obtained in real time PCR, PCR (SYBR Green) and LAMP

tests using the DNA extracts prepared following 2 different protocols.

Repeatability (%)




L03 N/A 95,83 N/A 95,83 N/A 100 N/A N/A N/A N/A N/A 100


L05 100 100 100 100 100 100 N/A N/A N/A N/A 100 100

L06 95,83 100 93,75 100 87,50 95,83 N/A N/A 100 100 N/A N/A

L07 100 100 100 95,83 100 95,83 N/A N/A N/A 91,67 N/A N/A



L10 100 95,83 100 87,5 100 100 N/A N/A 95,83 100 N/A N/A

L11 100 N/A 100 N/A 100 N/A N/A N/A 95,83 N/A N/A N/A

L12 100 95,83 100 91,67 87,50 100 100 100 100 91,67 N/A N/A

L13 N/A 100 N/A 100 N/A 100 N/A N/A N/A 91,67 N/A 100

L14* N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A


L16 100 100 N/A N/A 100 100 100 100 N/A N/A N/A N/A



L19 N/A 95,83 N/A 95,83 N/A 100 N/A N/A N/A N/A N/A N/A**


All laboratories* 99,65 98,72 99,31 96,97 97,73 99,36 100,00 100,00 98,61 95,83 100,00 100,00

N 12 13 9 11 11 13 2 3 6 6 2 4

* L14 results excluded (only one amplification replicate)

** L19 results excluded (no details on replications)

Laboratory



al. , 2009Cut-off = 38




No cut-off



34

Figure 5. Repeatability calculated using the results obtained in real time PCR, PCR (SYBR Green) and LAMP


Results are very repeatable inside laboratories. No significant differences between the different methods

are observed in term of repeatability.

Reproducibility

Table 15. Reproducibility calculated using the results obtained in real time PCR, PCR (SYBR Green) and


Reproducibility (%)



Restricted series* 93,54 99,06 91,60 98,89 68,41 94,60 72,66 81,94 97,63 92,00 100,00 95,50

N 13 14 10 12 12 14 2 3 7 6 2 5

*L6 (high number of false positives / about 40% on healthy insects) and L18 (systematic false negative) results excluded




al. , 2009Cut-off = 38




No cut-off


35

Figure 6. Reproducibility calculated using the results obtained in real time PCR, PCR (SYBR Green) and LAMP


Figure 6 bis. Reproducibility calculated using the results obtained in real time PCR, PCR (SYBR Green) and

LAMP tests using the DNA extracts prepared following 2 different protocols (restricted series: L06 results

excluded (high number of false positives) and L18 results excluded (systematic false negatives))

36

The reproducibility values are better for the real time PCR methods after a CTAB extraction for real time

PCR. Here again, we can suspect the by hand QuickPick™ method to damage the performance of the

methods. In fact, for example, for the real time PCR Harper et al., 2010, if we hold only the results of the

laboratories which have used a robot (N=5), the reproducibility aim 100%. On the contrary, if we hold only

the results of the laboratories which have used a by hand protocol (N= 8), the reproducibility value is of

72.77% (figure 6 ter).

The real time PCR Francis et al., 2006 using TaqMan® technology or SYBR Green dye presents a lesser

reproducibility than the real time PCR Harper et al., 2010 (in simplex or duplex). But for the PCR Francis et

al., 2006 using SYBR Green the results are not really significant due to the low number of participants

(QuickPick™ N = 2 / CTAB N = 3).

Figure 6 ter. Reproducibility calculated using the results obtained in real time PCR, PCR (SYBR Green) and

LAMP tests using the DNA extracts prepared following 2 different protocols (restricted series: L06 results

excluded (high number of false positives) and L18 results excluded (systematic false negatives)) – For

QuickPick™ extraction: series restricted to the 5 laboratories which have used a robot

37

Analytical sensitivity (limit of detection)

Table 16. Analytical sensitivity (limit of detection) calculated using the results obtained in real time PCR,

PCR (SYBR Green) and LAMP tests using the DNA extracts prepared following 2 different protocols.

For all the methods and for the great majority of the laboratories, low target concentration samples (5.103

bact./insect head) are detected (see the Table 16 for the details).

Analytical sensitivity = Limit of detection (probability of detection 100%) Sample concentration range: 5.103, 5.104 and 5.105 bact./head


L01 5.103 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A

L02 5.103 N/A 5.103 N/A 5.104 N/A N/A N/A N/A N/A N/A N/A

L03 N/A 5.103 N/A 5.103 N/A 5.103 N/A N/A N/A 5.103 N/A 5.103

L04 5.103 N/A N/A N/A 5.103 N/A N/A N/A 5.103 N/A N/A N/A

L05 5.103 5.103 5.103 5.103 5.103 5.103 N/A N/A N/A N/A 5.103 5.103

L06 5.103 5.103 5.103 5.103 5.105 5.103 N/A N/A 5.103 5.103 N/A N/A

L07 5.103 5.103 5.103 5.103 5.103 5.103 N/A N/A 5.103 5.104 N/A N/A

L08 N/A 5.103 N/A N/A N/A 5.103 N/A 5.103 N/A N/A N/A 5.103

L09 N/A 5.103 N/A 5.103 N/A 5.103 N/A N/A N/A N/A N/A N/A

L10 5.105 5.103 5.105 5.103 5.104 5.103 N/A N/A 5.104 5.103 N/A N/A

L11 5.103 N/A 5.103 N/A No positive N/A N/A N/A 5.103 N/A N/A N/A

L12 5.103 5.104 5.103 5.103 - 5.104 5.105 5.105 5.103 5.104 N/A N/A

L13 N/A 5.103 N/A 5.103 N/A 5.103 N/A N/A N/A 5.103 N/A 5.103

L14* 5.103 5.103 5.104 5.103 5.105 5.103 N/A N/A N/A N/A N/A N/A

L15 5.103 N/A 5.103 N/A 5.103 N/A N/A N/A N/A N/A 5.103 N/A

L16 5.103 5.103 N/A N/A 5.104 N/A 5.103 5.103 N/A N/A N/A N/A

L17 N/A 5.103 N/A 5.103 N/A 5.103 N/A N/A N/A N/A N/A N/A

L18 No positive N/A No positive N/A No positive N/A N/A N/A No positive N/A N/A N/A

L19 N/A 5.103 N/A 5.103 N/A 5.103 N/A N/A N/A N/A N/A 5.103

L20 N/A 5.103 N/A 5.103 N/A 5.103 N/A N/A N/A 5.103 N/A N/A

All laboratories

Lowest LOD 5.103 5.103 5.103 5.103 5.103 5.103 5.103 5.103 5.103 5.103 5.103 5.103

Harper et al , 2010Cut-off = 38Laboratory



al. , 2009Cut-off = 38




No cut-off


38

5.2.2 On natural insects

5.2.2.1 On natural insects (DNA extraction QuickPick™ Plant DNA and/or CTAB)

Table 17 Results on natural insects (DNA extraction QuickPick™ Plant DNA and/or CTAB)

Homogeneity and stability studies allow estimating the contamination rate of potentially contaminated

insects at about 25% (DNA extraction using QuickPick only), so 3.75 positive insects per 15 potentially

contaminated insects. A panel contains 5 healthy insects and 15 potentially contaminated insects. The

percentage of positive insects per laboratory ranges from 0% to 60% with average ranging from 3.33% to

28.57% according to the methods and laboratories (L06 results excluded due to high number of false

positives also on the spiked samples).

On the basis of a binomial distribution, with a probability of success of 25% and a number of tests of 15

(number of potentially contaminated insects), with an alpha risk of 5% (confidence level of 95%), observed

values should range into the interval of [1; 7]. On the table 16, values under the lower bound of probability

are highlighted in blue and values above the highest bound of probability are highlighted in orange. Values

within the interval are highlighted in green.

With the QuickPick™ kit extraction, all the data got by laboratories except L06 are in the interval of [1; 7]

with PCR Harper et al. 2010 by real-time PCR in simplex/duplex and by LAMP. The number of positive

QuickPickTM Plant DNA kit and CTAB protocol

Number % Number % Number % Number % Number % Number % Number % Number % Number % Number % Number % Number %

L01 - - - - - - - - - - - - - - - - - - - - - - - -

L02 - - - - - - - - - - - - - - - - - - - - - - - -

L03 - - - - - - - - - - - - - - - - - - - - - - - -

L04 2 13,3 8 53,3 - - - - 2 13,3 6 40,0 - - - - 2 13,3 3 20,0 - - - -

L05 3 20,0 3 20,0 3 20,0 3 20,0 3 20,0 3 20,0 - - - - - - - - 3 20,00 2 13,33

L06 9 60,0 18 120,0 14 93,3 17 113,3 0 0,0 3 20,0 - - - - 1 6,7 0 0,0 - - - -

L07 2 13,3 4 26,7 2 13,3 4 26,7 1 6,7 4 26,7 - - - - 1 6,7 2 13,3 - - - -

L08 - - - - - - - - - - - - - - - - - - - - - - - -

L09 - - - - - - - - - - - - - - - - - - - - - - - -

L10 5 33,3 1 6,7 3 20,0 1 6,7 4 26,7 1 6,7 - - - - 3 20,0 1 6,7 - - - -

L11 2 13,3 - - 2 13,3 - - 0 0,0 - - - - - - 0 - - - - - - -

L12 1 6,7 9 60,0 1 6,7 8 53,3 0 0,0 5 33,3 0 0,0 1 6,7 1 6,7 3 20,0 - - - -

L13 - - 3 20,0 - - 3 20,0 - - 1 6,7 - - - - - - 1 6,7 - - 3 20,00

L14 - - - - - - - - - - - - - - - - - - - - - - - -

L15 - - - - - - - - - - - - - - - - - - - - - - - -

L16 4 26,7 2 13,3 - - - - 2 13,3 2 - 1 6,7 1 6,7 - - - - - - - -

L17 - - - - - - - - - - - - - - - - - - - - - - - -

L18 - - - - - - - - - - - - - - - - - - - - - - - -

L19 - - - - - - - - - - - - - - - - - - - - - - - -

L20 - - - - - - - - - - - - - - - - - - - - - - -

MIN 1 6,67 1 6,67 1 6,67 1 6,67 0 0,00 1 6,67 0 0,00 1 6,67 0 6,67 0 0,00 3 20,00 2 13,33

MAX 9 60,00 18 120,00 14 93,33 17 113,33 4 26,67 6 40,00 1 6,67 1 6,67 3 20,00 3 20,00 3 20,00 3 20,00

Average 3,50 23,33 6,00 40,00 4,17 27,78 6,00 40,00 1,50 10,00 3,13 21,90 0,50 3,33 1,00 6,67 1,33 10,67 1,67 11,11 3,00 20,00 2,50 16,67

MIN* 1 6,67 1 6,67 1 6,67 1 6,67 0 0,00 1 6,67 0 0,00 1 6,67 0 6,67 1 6,67 3 20,00 2 13,33

MAX* 5 33,33 9 60,00 3 20,00 8 53,33 4 26,67 6 40,00 1 6,67 1 6,67 3 20,00 3 20,00 3 20,00 3 20,00

Average* 2,71 18,10 4,29 28,57 2,20 14,67 3,80 25,33 1,71 11,43 3,14 22,22 0,50 3,33 1,00 6,67 1,40 11,67 2,00 13,33 3,00 20,00 2,50 16,67

N = 8 8 6 6 8 8 2 2 6 6 1 2

Number of potentially contaminated insects tested: 15

* L06 results excluded due to high number of false positives

LAMP

Positive samples

Real-Time PCR SYBR Green

Thermocycleur


QuickPickTM CTAB QuickPickTM CTAB

Duplex Harper et al , 2010 Ioos et al. ,

2009Cut-off = 38

QuickPickTM CTAB QuickPickTM QuickPickTMCTAB CTAB



QuickPickTM CTAB

Portable device


No Cut-off

Laboratory

39

insects is a little lower with LAMP. By testing with Francis et al., 2006 PCR, false negative results are

obtained in 3 laboratories. These results are in line with those obtained on spiked sample results in the first

part of this report.

With CTAB extraction, 2 laboratories others than L06 obtained results over 7 positive insects per panel. In

this case, we do not have the results of homogeneity /stability data on natural insects because the number

of natural insects being low and as the results of Ct obtained in homogeneity results between CTAB and

QuickPick™ kit were very close, we considered do not test panel of natural insects with CTAB extraction for

homogeneity and stability tests. In this case, it is impossible to conclude if the number of positive insects is

due to cross contamination or a better sensitivity after CTAB extraction on natural insects.

5.2.2.2 On natural insects (DNA extraction protocol Yaseen et al., 2015)

Table 18 Results on natural insects (DNA extraction protocol Yaseen et al., 2015)

Yaseen et al , 2015 DNA extraction

% Number % Number %

L01 - - - - -

L02 - - - - -

L03 - - - - -

L04 0,0 0 0,0 - -

L05 - 0 - -

L06 - - - - -

L07 0,0 - - - -

L08 - - - - -

L09 - - - - -

L10 0,0 - - 0 0,0

L11 - - - - -

L12 0,0 0 0,0 - -

L13 0,0 - - 0 0,0

L14* - - - - -

L15 - - - - -

L16 - - - 2 13,3

L17 - - - - -

L18 53,3 - - 3 20,0

L19 - - - - -

L20 - - - - -

Number of potentially contaminated insects tested: 15

Laboratory

-

-

0

-

-

0

-

-

-

-

ThermocycleurGenie II

Portable device


No Cut off

Positive samples

LAMP

ICGENE

Number

0

-

0

0

-

-

-

-

8

-

40

Surprisingly, only 2 laboratories, using the ICGene device, got positive results with this method on a total of

8 laboratories. These results are maybe due to a problem with experience of laboratories working for the

first time on this protocol and with transferability of the method. No others explanations have been found.

41

6. SUMMARY OF THE OVERALL VALUES RECOVERED FOR EACH PERFORMANCE CRITERIA In table 18 are summarized the values of the performance criteria obtained for all tested diagnostic methods tested in this proficiency test.

Table 19. Values of the performance criteria obtained for all tested diagnostic methods on spiked samples.

Values of the performance criteria obtained for all tested diagnostic methods (on artificially spiked insects)


% 88,46 99,11 84,81 100,00 60,42 95,54 56,25 79,17 82,14 89,58 100,00 97,50

% (restricted series)* 95,83 99,11 94,37 100,00 65,91 95,54 56,25 79,17 95,83 89,58 100,00 97,50

Nbr of lab with false negative 2 1 3 1 8 3 2 1 3 3 0 1

% 96,15 92,86 90,00 91,67 100,00 100,00 87,50 100,00 100,00 95,83 100,00 100,00

% (restricted series)** 97,92 100,00 100,00 100,00 100,00 100,00 87,50 100,00 100,00 100,00 100,00 100,00

Nbr of lab with false positive 2 1 1 1 0 0 1 0 0 0 1 0

% 91,03 97,02 87,27 96,53 73,61 97,62 66,67 86,11 88,10 91,67 100,00 98,33

% (restricted series)*** 96,21 99,36 95,83 99,24 78,33 97,44 66,67 86,11 98,33 93,33 100,00 98,33

Repeatability % 99,65 98,72 99,31 96,97 97,73 99,36 100,00 100,00 98,61 95,83 100,00 100,00

% 82,49 95,51 76,37 94,76 64,77 96,45 72,66 81,94 78,13 87,24 100,00 95,50

% (restricted series)**** 92,94 98,98 91,90 98,78 69,14 96,18 72,66 81,94 97,63 92,00 100,00 95,50

13 14 10 12 12 14 2 3 7 6 2 5

* L18 results excluded (systematic false negatives)

** L06 results excluded (high number of false positives / about 40% on healthy insects)

*** L06 results excluded (high number of false positives / about 40% on healthy insects) and L18 results excluded (systematic false negatives)

**** L6 (high number of false positives / about 40% on healthy insects), L14 (one repetition only for amplification) and L18 (systematic false negative) results excluded

N =

Reproducibility

performance criteria

Portable device

Diagnostic sensitivity

Diagnostic specificity

Accuracy



Duplex Harper et al , 2010

Ioos et al. , 2009Cut-off = 38


Francis et al ., 2006Cut-off = 35 / Melting peak 83-

85°C


No cut-off

Thermocycleur

42 7 DISCUSSION AND CONCLUSION

On spiked insects

The real time PCR Francis et al., 2006 using SYBR Green dye presents a lesser sensitivity than the real time

PCR Harper et al., 2010 (in simplex or duplex) but the results are not really significant due to the low

number of participants for this method (QuickPick™ N = 2 / CTAB N = 3). Moreover, the cut-off value for

this method is 35 (value indicated in EPPO standard PM7/24) against 38 for the real time PCR of this TPS

using TaqMan® technique, and so the method is also “disadvantaged”. No difference appears between the

results got by PCR Harper et al., 2010 in duplex and simplex, so it could be recommended to use the duplex

integrating an internal control.

For this TPS, insect spiked samples are composed of head only and eyes were not removed. The results

obtained in term of sensitivity and Ct values are very similar to results obtained by the organizer in a

previous characterization and validation study. So, the presence of eyes, elsewhere known to contain PCR

inhibitors, seems for this species (Philaenus spumarius), in order to not damage the performance of the

methods.

No significant differences between the different methods are observed in term of repeatability.

The real time PCR Francis et al., 2006 using TaqMan® technology or SYBR Green dye presents a lesser

reproducibility than the real time PCR Harper et al., 2010 (in simplex or duplex). But for the PCR Francis et

al., 2006 using SYBR Green the results are not really significant due to the low number of participants

(QuickPick™ N = 2 / CTAB N = 3).

For the all the methods and for the great majority of the laboratories, low target concentration samples

(5.103 bact./insect head) are detected.

It is important to note that for the QuickPick™ DNA extraction method, this TPS shows some significant

differences between the results obtained by the laboratories using a robot and laboratories using a by hand

protocol, especially in term of sensitivity and reproducibility. So it is recommend the use of the QuickPick™

kit preferably in association with the use of a robot.

Results delivered by LAMP method are very similar to thus obtain with real time PCR for the different

performance criteria. But it should be noted that no cut-off value are applied for LAMP because of the

absence of cycle during the amplification phase, and so this method is advantaged.

43

On potentially naturally contaminated insects

With these samples, the major difficulty for the analysis of the results is that the real status of each

specimen is not known but only a probable contamination rate of the panel. This explains the difficulty to

extricate a tendency from the analysis of the results on natural insects.

For the Yaseen et al., 2015 simplified DNA extraction method followed by the LAMP Harper et al.,2010

modified by Yaseen et al., 2015, only 2 participants got positives results on the 15 insect panel on a total of

8 laboratories. These results are maybe due to a problem with experience of laboratories working for the

first time on this protocol and with transferability of the method. No others explanations have been found.

In conclusion, this TPS shows the real-time method PCR and LAMP Harper et al. 2010 allow to detect X.

fastidiosa on heads of Philaenus spumarius with eyes with reliability in terms of:

- good sensitivity (detection limit on spiked samples in this TPS is 5.103 b/mL),

- good specificity,

- good repeatability and reproducibility.

For the PCR Francis et al., 2006(Taqman and SYBR-Green), the results show a lesser sensitivity. The

differences between cut-off calculation can be questionnable with the method using SYBR Green.

This TPS put in light some critical points and recomendations for the laboratories : be vigilent on the risk of

false negative results during the extraction step, mainly by using the QuickPick™ Plant DNA kit by hand and

not the robot. The use of the robot increases the sensitivity and the reproducibility. This device can be

recomended.

CTAB is the most used extraction method by laboratories of this TPS. Nevertheless, differences in

experience with this method can impact on the results.

It is also necessary to avoid DNA cross contamination during the procedures.

44

8. REFERENCES - Chabirand A, Anthoine G, Pierson O, Hostachy B, 2014. The organization of proficiency testing in plant pathology (qualitative methods of analysis) according to the ISO/IEC 17043: example of the French national reference laboratory. Accred Qual Assur (2014) 19: 111–125 DOI 10.1007/s00769-014-1034-y. - Francis M., Hong Lin, Cabrera-La Rosa J., Harshavardhan Doddapaneni and Cirevelo E. L. (2006). Genome-based PCR Primers for Specific and Sensitive Detection and Quantification of Xylella fastidiosa, European Journal of Plant Pathology. Volume115, Number 2, 203 - Harper SJ, Ward LI, Clover GRG, 2010. Development of LAMP and real-time PCR methods for the rapid detection of Xylella fastidiosa for quarantine and field applications. Phytopathology 100, 1282–1288. - Ioos R., Fourrier C., Iancu G., Gordon T.R. 2009. Sensitive Detection of Fusarium circinatum in Pine Seed by Combining an Enrichment Procedure with a Real-Time Polymerase Chain Reaction Using Dual-Labeled Probe Chemistry. Phytopathology 99-5-0582 - PM 7/24 (2) (2016), Xylella fastidiosa. EPPO Bull, 46: 463–500. doi:10.1111/epp.12327. - Yaseen T., Drago S., Valentini F., Elbeaino T., Stampone G., Digiaro M.and D’onghioa A.M., On-site

detection of Xylella fastidiosa in host plants and in “spy insects” using the real-time loop-mediated

isothermal amplification method, Phytopathologia Mediterranea (2015) 54, 3, 488−496

8. ACKNOWLEDGMENTS Maria Saponari, Instituto por la protezione sostenibile, CNR, Bari, Italy for providing insect specimens and

for its advice and informations,

Michael Maixner, Julius Kühn-Institut, Siebeldingen, Germany for providing insect specimens,

Gundrun Straus, AGES, Vienna, Austria for providing insect specimens,

Philippe Reynaud and his staff from the Anses Plant health laboratory Entomology unit, Montferrier-sur-Lez, France, for the identification of insect specimens

45 Annex I

Technical sheet: detailed protocols

Please see file attached.

46

Annex II Participant contract

47

48

49

50

51

Annex III Result sheets form

TPS 17-XFAST-EU

Test performance study for the evaluation of

molecular methods to detect Xylella fastidiosa in

Philaenus spumarius

Instructions to complete the results form

Organism Xylella fastidiosa

Participant Laboratory

1/ Five sheets (from 2 to 6) can be used to fill out this result file. First, delete the ones you

don't need (according to the panels you have choosen).

2/ Please, in participant laboratory, fill out your name, your code, the number of panels you

have received.

3/ For each sheet, please enter the panel code

4/ In each sheet, you will find a square for one PCR method (# 1 for Harper et al., 2010 for

example). As PCR Harper (simplexed and duplexed), Francis (Taqman version) and LAMP

Yaseen are compulsory, these squares must be filled out. Squares #4 and #6 must be filled

out if the PCR are performed.

5/ Blue cells must be filled out (may be not for the optional methods), and white cells can be

used to give more information, or provide observations.

6/ Please, return this result file by e-mail in Excel form, and in PDF at these three electronic

addresses :

[email protected] ; [email protected] ; [email protected]

Number of naturally panel(s) received

Number of artificially contaminated panel(s) received

Laboratory code (L__)

Laboratory's name

52

# 1 Real-time PCR Harper

Sample ID Cq 1 Cq 2 Mean Result (positive or negative) potential comments

1

2

3

4

5

6

7

8

9

10

11

12

13

PAC

NAC

# 3 LAMP Real-time PCR Harper modified by Yaseen

Sample ID "Cq 1" "Cq 2" Mean Result (positive or negative) potential comments

1

2

3

4

5

6

7

8

9

10

11

12

13

PAC

NAC

# 5 Real-time PCR Francis


1

2

3

4

5

6

7

8

9

10

11

12

13

PAC

NAC

Panel code

on 40 cycles, positive if Cq mean < 38. Real-time curve

Observations :

Thermocycler :

on 60 cycles of 30 secondes, positive if exponential curve. Real-time curve

Observations :

Thermocycler :

on 45 cycles, positive if Cq mean < 38. Real-time curve

Observations :

Thermocycler :

Date of validation

Additional

comments

Validation by the test correspondent

Name

Date of analysis

Used robot or magnet

Signature

53

# 2 Real-time PCR Harper duplexed Ioos


1

2

3

4

5

6

7

8

9

10

11

12

13

PAC

NAC

# 4 LAMP Real-time PCR with Portable Device


1

2

3

4

5

6

7

8

9

10

11

12

13

PAC

NAC

# 6 Real-time SYBRgreen PCR Francis (optional)


1

2

3

4

5

6

7

8

9

10

11

12

13

PAC

NAC

Sample ID Tm 1 Tm 2 Mean

1

2

3

4

5

6

7

8

9

10

11

12

13

PAC

NAC

Real-time curveCq from HARPER (FAM) on 40 cycles, positive if Cq mean < 38.

Observations :

Thermocycler :

Used protocol (Enbiotech or Optigene for example) : Real-time curve

Observations :

Portable device (brand)

on 40 cycles, positive if Cq mean < 35 and 83°C<Tm<85°C. Real-time curve

Observations :

Thermocycler :

Melting peaks

Observations :

54

# 3 LAMP Real-time PCR Harper modified by Yaseen


1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

PAC

NAC

# 4 LAMP Real-time PCR with Portable Device


1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

PAC

NAC

on 60 cycles of 30 secondes, positive if exponential curve. Real-time curve

Used protocol (Enbiotech or Optigene for example) : Real-time curve

Observations :

Observations :

Thermocycler :

Portable device (brand)

Name

Signature

Date of validation

Date of analysis

Additional

comments

Validation by the test correspondent

55

Annex IV Homogeneity and stability calculation

Please see file attached (in French)