What is wrong with What is said? Hasan Yazici, MD University of Istanbul.

What is wrong with

What is said?

Hasan Yazici, MDUniversity of Istanbul

Vitamin-E in goutthe Hypothesis

• Several recent reports hint that vitamin E can lower serum uric acid levels (1-3).

• These are either single case reports or uncontrolled, open studies in small groups of patients.

• Thus we tested the hypothesis to see whether vitamin E was effective treatment for gout in a 6 month, double blind, placebo controlled study.

What is wrong?

A hypothesis should not be formulated as a question.

Methods - 1

• The patients were randomized into the vitamin E (n = 60) and placebo (n= 60) groups.

• After randomization each patient signed a written informed consent.

• The patients were assessed monthly with serum uric acid, hematocrit, serum creatinine, cholesterol and testosterone levels.

• The number of gout attacks were also noted.

What is wrong?

• The randomization follows, not precedes, the informed consent.

Methods -2 • 30 patients had to leave the study for various

reasons. – 20 patients from the active drug group and 10 from

the placebo left the study before its completion (ns). – Thus the final analysis for drug efficiency was made

among 90 patients.

• Among the 20 patients who dropped out from the treatment group 3 moved to another city.

• 15 complained of severe indigestion and no information was available for the remaining 2.

• Dyspepsia was the reason for withdrawal in 7 patients in the placebo group while headaches were the main reason for discontinuation of treatment in the remaining 3.

What is wrong?

• The intention to treat principle is neglected.

Intention to treatanother study

• 90 patients were initially randomized into 30 patients each of placebo, QXY 2mg qd and QXY 5mg qd groups.

• One patient in the QXY 5mg group developed pneumonia and had to be hospitalized 10 days after treatment started.

• The patient was withdrawn and, according to the protocol, another patient was recruited.

• Thus, the intention to treat analysis brought the number analyzed to 31 in the QXY 5mg group and the total number to 91.

What is wrong?

• How was randomization achieved in the newcomer?

• The newcomer inflates the

denominator when looking at harm.

Methods-3

• The allocation of 60 patients to the active drug and the placeo arms ensured a 86% power to detect a difference rate of 25% in the uric acid levels between the 2 groups using a 2-sided Fisher’s exact test with α set at 0.05

What is wrong?

• In a power calculation it is not enough to state the magnitude of change, the anticapated event rates should also be given.

• Ensured 86% power?

Sample size calculations 4 components

I. Type I (alpha) error

II. Type II (beta) error

III. Event rate in the control group

IV. Event rate in the treatment group

Power calculations for efficacy and harm

• The primary end point was the achievement of at least 25% improvement according to the ACR criteria (ACR20) at week 20.

• The sample size of 180 patients per group was chosen to ensure an adequate safety evaluation.

• The sample size also ensured that there was 90% power to detect a significant difference in the proportion of ACR20 responders between the treatment groups using a significance level of 0.05, assuming 20% and 42% of the patients in the control and the treatment groups respectively achieved ACR 20 responses.

What is wrong?

• No basis for the assumptions related to harm

• Post hoc power calculations

Powering for safety

• A sample size of 300 patients each in the study drug and the control groups was determined to demonstrate a specific adverse event rate of 1% or less with 95% confidence.

What is wrong?

The probability of an event not happening does not give us information about the likelihood of events happening in the different arms of the study.

The zero patient method

• If one screens n individuals and does not find an attribute y among this group;

• Then one can conclude that the frequency of y is less than 1/0.33n among the n individuals surveyed, with 95% confidence.

• This approximation is true for prevalances < 0.02 of the attribute surveyed.

H Yazici et al Rheumatology Oxford (2001)

Efficacya subgroup analysis

• The uric acid levels decreased to 5.2 ± 1.5mg/dL from a baseline of 9.3 ± 2.3 mg/dL in the treatment and from 9.7 ± 1.9 mg/dL in the placebo groups.

• There were no differences in the lowering of uric acid between the 2 groups (p>0.05).

• There were also no differences in the number of gout attacks between the 2 groups of patients. However a subgroup analysis was also done.

• Among those patients in the treatment group who gave a history of more than 5 gouty attacks per year, as compared to those who had less than 3, the uric acid levels were significantly lower after treatment (p< 0.05).

Reviewer comments

• The findings in the subgroup analysis, conducted among a small number of patients, should be interpreted with caution.

• Even though the authors found a statistically significant difference in efficacy the study was not primarily planned to assess this difference.

• This lessens the external validity of these results.

What is wrong?

The main problem with this subgroup analysis is not that the subgroup is small. It is an effort in the direction of proving that the new drug at hand is effecacious.

Harm another subgroup analysis

• Table 3 gives the rate of observed adverse events. Indigestion was rather frequent in either group; 15/45 patients in the treatment and 14/45 in the placebo groups.

• A subgroup analysis was also done after the patients who participated in the trial were questioned about a pre-trial history of indigestion. It turned out that 20/45 patients in the treatment and 18/45 in the placebo group had pre-trial indigestion.

• A further analysis among these patients revealed that 14/20 in the treatment and 2/18 in the placebo group (p< 0.03) with a history of pre-trial indigeston also reported indigestion during the trial.

Reviewer comments • The findings in the subgroup analysis,

conducted among a small number of patients, should be interpreted with caution.

• Even though the authors found a statistically significant difference in harm the study was not primarily planned to assess this difference. This lessens the external validity of these results.

What is wrong?

Again the main issue is not the size of the subgroup. This subgroup analysis looks at the possibility that this new drug might not be all that harmless. Thus the exercise is in the direction of falsification and therefore is justified.

Dr. Pincus study

• Recently, in a placebo controlled withdrawal study, Dr. Pincus showed that small doses of prednisone (1- 4 mg/day) were significantly effective in the management of rheumatoid arthritis.

• The study was conducted among 31 patients

Reviewer comments

This beneficial effect of prednisone described among a small number of patients should be interpreted with extreme caution even if the authors found a statistically significant difference. The number of patients studied, thus the study power, was simply too small.

What is wrong?• The reviewer is partially right. • The issue of power applies only if we

are missing a more important outcome that would have been more evident in a larger group.

• External validity? The small number of patients might be quite different from the real life RA patients.

• However (!) this is not simply an issue of numbers but of patient selection.

The coxib study

• 6000 patients with osteoarthritis of the knee were randomized to recieve either the new coxib (NCB) or the traditional (TCB).

• 40% of the NCB and 35% of the TCB patients found total pain relief.

• Hypertension was a problem in 2.5 % of the NCB and 4.5 % of the TCB patients.

• It was concluded:a. NCB decreased pain by 13% b. There was 45% less hypertension with NCB

What is wrong?

• Nothing. Needs more interpretation. • A NNT analysis will tell you that you

have to treat 10 patients to see the superiority of NCB over TCB in 1 patient.

• A NNH analysis will say that you have to treat 50 patients with TCB to harm 1 more patient with hypertension as compared to using NCB.

NNT & NNH • Relative risk (RR): outcome ratio in the control

group/ outcome ratio in the control group

• Absolute risk reduction (ARR): difference in the ratios

• Relative risk reduction (RRR): ARR/outcome in the control group OR 1-RR

• NNH: 1/ARR (frequency of hypertension in the control group 0.045, frequency of hypertension in the study group 0.025, ARR= 0.020; NNT = 50.)

A withdrawal study

• Two previous double blind studies of colchicine in BS had shown no superiroity of this agent over placebo in treating the oral ulcers in this condition.

• Recently a withdrawal study was done among those patients who claimed benefit from colchicine. They were randomized to contiune to receive the active drug or placebo.

• After 3 months those who stopped taking colchicine had significantly more ulcers (sign test, p = 0.02).

Reviewer comments• This study is interesting but is of

limited use. A problem with all withdrawal trials, they seldom represent real life use.

• The authors used the “sign test” to analyze the differences in oral ulcers between the 2 groups. Is this a new test? I suspect it is not quite powerful. Why not use the more powerful tests of significance to show the real differences, if any?

What is wrong? (I)• The main problem with withdrawal

studies is that they are not done often enough.

• They provide excellent information about possible a type II error in previous, traditional RCT’s.

• An important issue is that they do not give a fair picture of drug associated harm.

What is wrong? (II)

• The sign test is a time honored, conservative tool.

• Significance in a conservative test gives more validity to the significant differences observed.

The mighty “p”

Among the 96 patients allocated to the new medication there were 3 cases of myocardial infarction while the same was true for 2/94 patients allocated to placebo (p=0.86).

What is wrong?

• Better give it with the statistic used (sign test, p = 0.02).

• Do not use it to mesmerize the reader.

The Commandments

• Hypothesis as an affirmative statement • Intention to treat• Components of the power calculation• Post hoc power analyses• Powering for harm, no short cuts• Justifications for subgroup analyses• Small but significant• Meaning of treatment effects• Withdrawal studies to be cherished• The “mighty p”

Summary

•In a RCT, like in all scientific endeavor, all efforts aimed at proving the hypotheses are wrong.

•The aim is falcification.

Vitamin-E in goutthe Conclusion

Vitamin E is an innocuous agent and with the possible additional benefit of an antihypertensive effect, as was also serendipitously noted in this study, such studies are surely warranted. We are about to start a study along those lines at our institution.

What is wrong?

Do not preempt!