Studying a Study and Testing a Test: Sensitivity Training, “Don’t Make a Good Test Bad”, and “Analyze This” Borrowed Liberally from Riegelman and Hirsch,

Studying a Study andTesting a Test:

Sensitivity Training, “Don’t Make a Good Test Bad”,

and “Analyze This”Borrowed Liberally from

Riegelman and Hirsch, 2nd Edition

What you need to know:

• When will ordering a given test be helpful in making a decision on therapy?

• When is a test more likely to be misleading?

• Which test should I order when?

• How do I evaluate the literature regarding the value of different tests?

Sensitivity and Specificity:

• Sensitivity = TP/(TP+FN)

• Specificity =TN/(TN+FP)

NEWTEST

GOLD STANDARDDiseased

GOLD STANDARD

Disease-free

Positive

True Positive

False positive

Negative

False negative

True negative

Scenario #1

• You read an article describing a new rapid test for diagnosis of Herpes Simplex virus infection. It was used on samples from 1000 neonatal patients with CSF pleocytosis (>30 wbc/mm3) elevated protein (>50), with negative gram stain, and the results were compared with a gold-standard test. The following results were obtained.

Sensitivity and Specificity:

• Question 1:

What are the Sensitivity and Specificity of this test?

• Question 2:

Is this a Good Test?

NEWTEST

GOLD STANDARDDiseased

GOLD STANDARD

Disease-free

Positive

400 50

Negative

100 450

500 500

Sensitivity and Specificity:Answer to Questions 1 & 2

• Sensitivity = True Positives Divided by Gold Standard Diseased (True positives plus false negatives) = 400/500 = 80%

• Specificity = True Negatives Divided by Gold Standard Disease-free (True negatives plus false positives = 450/500 = 90%

• Good Test?

Take Home Message #1

• Sensitivity and Specificity are Properties of the Test!!!

Scenario #2

• The same diagnostic test becomes commercially available and your hospital decides that for “medicolegal reasons”, it should be done on all CSF samples, regardless of cell count and protein results. The results of the next 1,000 tests are shown on the following table.

Sensitivity and Specificity

• Question 3:

What are the sensitivity and specificity now?

• Question 4:

Is this a good test?

NEWTEST

GOLD STANDARDDiseased

GOLD STANDARD

Disease-free

Positive 80 90

Negative

20 810

100 900

Answers to 3 and 4

• Sensitivity = 80/100=80%

• Specificity = 810/900=90%

• Specificity and Sensitivity do not change when you overuse the test, but the value of the positive result is less

• Good test?– …. 90 of the 170 patients with positive tests

are actually disease-free…

Predictive Values

• Predictive value of a positive test = TP/(TP+FP)

• Predictive value of a negative test = TN/(TN+FN)

NEWTEST

GOLD STANDARDDiseased

GOLD STANDARD

Disease-free

Positive

80 90

Negative

20 810

100 900

Positive and Negative Predictive Value

• Predictive value of a positive test = Proportion of those with a positive test who have the disease = 80/170 = 47.1%

• Predictive value of a negative test = Proportion of those with a negative test who are disease-free = 810/830 = 97.6%

Take Home Message #2

• The Predictive Value of a test depends upon the prevalence of the disease in the population in which it is applied!!

• Corollary: You can make a good test into a bad test by using it in a population with a very low prevalence of the disease

Scenario #3

• A new extended screening test is being piloted for medium chain acyl-CoA dehydrogenase deficiency (MCAD) on newborn blood spots. The following results are obtained:

Predictive Value

• Question 3:

What are the sensitivity, specificity, positive and negative predictive values now?

• Question 4:

Is this a good screening test?

NEWTEST

GOLD STANDARDDiseased

GOLD STANDARD

Disease-free

Positive 99 1800

Negative

1 8100

100 9900

Answers to Questions 5 and 6

• Sensitivity = 99/100 = 99%

• Specificity = 8100/9900 = 82%

• Predictive value of positive = 99/1899 = 5.2%

• Predictive value of negative= 8100/8101= 99.99%

Characteristics of a Good Screen

• Very high NEGATIVE PREDICTIVE value (implies high sensitivity)

• Availability of follow-up Gold Standard test to confirm (may be repetition of original test)

• Availability of counseling and education

• Intervention that affects outcome

Risks and Odds

• Sometimes you don’t know the prevalence of the disease within a given situation, so you need to evaluate that from the literature

• Two questions:– What is the risk of a given pathology in the

context of a given risk factor?– What are the odds before and after you

assess for the presence of a risk factor?

Scenario #4: You see a febrile infant in the midst of an aseptic meningitis outbreak, but this one has never had

Prevnar. You have access to some raw prospective data on the use of Prevnar in your area:

Invasive Pneumococcal disease

No Invasive disease

No Prevnar 30 970

Prevnar 3 997

What is the relative risk of Invasive disease in an unimmmunized patient?

Relative Risk

• Relative Risk = probability of disease with the risk factor ÷ probability without the risk factor

• 0.030/0.003 = 10

• BUT, this only works if the data is generated PROSPECTIVELY

What if this data were generated RETROSPECTIVELY, i.e., taking kids who had invasive disease and selecting

matched controls.

Invasive Pneumococcal disease

No Invasive disease

No Prevnar 90 45

Prevnar 10 55

What is the odds ratio of Invasive disease in an unimmmunized patient?

Odds ratio

• Odds ratio: odds of having a risk factor in the diseased population ÷ odds of having the risk factor in a matched control population

• 90/10 ÷ 45/55 = 9 ÷ 0.82 = 11

Scenario #5

• An article outlines guidelines for referral to a pediatric cardiologist for infants and children with cardiac murmurs. The article describes a decision analysis methodology that allows you to compare the effectiveness of a two different paradigms, one in which EKG is combined with CXR at the same time, the other in which only abnormal quality murmurs are sent on for Echo.

Decision Analysis:Serial Application of Tests

Decision Analysis:Parallel Application of Tests

Which of the following is true?

A. In a serial testing approach, the initial test must meet most criteria of a good screening test.

B. In a parallel approach, more diagnostic value is gained if the outcomes of the 2 tests are independent [e.g., CXR is less useful in asthmatics, since wheeze and atelectasis occur together in a non-random fashion]

C. A and B are both true.

Interventional Studies

• Assignment to Groups

• Assessment of outcomes

• Analysis

• Interpretation

Assignment (Avoid Bias)

• Prospective vs. Retrospective

• Randomized vs. Selected

• Blinded vs. Open-label

• Placebo

Assessment

• Define variables prior to initiation of study

• Choose a good test!!!

Analysis

• State Hypothesis as null: “There is no difference between treatment and control groups”

• Type I error: Falsely reject the null hypothesis, P value (0.05) is likelihood of type I

• Type II error: Falsely accept the null hypothesis, Power = 1.0 - type II error

• Power of 0.8 is standard

3 determinants of Power

• Variability of the test

• Incremental change

• Sample size

Interpretation

• What was the study population?

• Do the results apply to your population?