…when the conditions of an experiment are once established and worked up according to the mind's preconceived idea, an induced or premeditated observation.

“…when the conditions of an experiment are once established and worked up according to the mind's preconceived idea, an induced or premeditated observation will, as we said, result.”

“If men discuss and experiment … to prove a preconceived idea …, they no longer have freedom of mind, and they no longer search for truth.”

An experiment done to prove something is different from an experiment done to test something.

A Favorite Example (10/2003):

Patients with uncomplicated DVT/PE

Randomize between

Unfractionated heparin (“standard”)

New drug (easier to use)

Compare rates of DVT, PE, sudden death

Open label, >2,000 patients/subjects

Non-inferiority design

The results:

Similar outcomes in DVT/PE (slight NS advantage to new drug)

Similar outcomes re: mortality

Patients getting new drug tended to leave the hospital earlier (sub-q rather than i.v.)

Conclusion: Non-inferior in efficacy; dosing convenience make it an attractive drug

What’s Wrong?

New drug was compared against old approach of known efficacy

Everybody got treated (no placebo gremlins)

Trial design was standard; sample size was appropriate; few drop-outs

Monitoring was appropriate (Incl. DSMB)

Reputable researchers, sites

The trial was ethical and well-done ... but …

The “standard” therapy was no longer usual practice … drugs similar to the test drug had already largely replaced unfractionated heparin, at least in academic centers

The trial outcome can be restated as “This drug is probably no worse than what we used to do ten years ago”

A strong finding in favor of licensing the drug, but very little help in deciding how to use it wisely

A Non-so-benign Example:

Dan Markingson and his mother, Mary WeissDan’s death in an industry-sponsored clinical trial raises questions about how we do them and monitor them,

Dan’s medical history:

Bright kid; got a B.A. in English from the University of Michigan … sought employment in California.

Over the following three years, developed paranoia, changed his name, adopted cult beliefs.

Indicated to his mother that there was an apocalyptic event approaching, that he might be instructed to kill and would do so.

The story gets complex:

Several maternal interventions fell short

Hospitalized on a hold, given antipsychotic meds; commitment proceedings begun

All examiners agreed that he lacked full decisional capacity and was a danger to himself and others

Result: Stay of commitment

Stay of commitment:

The court finds in favor of commitment, but stays the order on conditions

Typically: “As long as you comply with treatment and do OK with it, you won’t actually be committed and will still have control over your own life”

Conceptually like a revokable parole

No guardian is named

The treatment chosen for Dan:

CAFÉ: Clinical Trial Comparing Atypicals in First Episode

Randomly assign among the three most common “atypical” antipsychotic agents for patients having their first psychotic episodeSeroquel (quetiapine), Zyprexa (olanzapine), and Risperdal (risperidone.)

Study designed, funded by AstraZeneca

Digression: Atypical antipsychotics

All three were licensed and approved;

Very successful drugs commercially, having largely supplanted earlier antipsychotic drugs on the strength of claims that they had fewer autonomic side effects and a lower incidence of tardive dyskinesia;

One of these would very probably have been chosen if not for study entry

How would the study alter Dan’s Rx?

Double-blind;

Limits on other agents; discontinuing or crossing over would require study withdrawal;

Treatment in other respects would be pretty much the same as getting one of the drugs non-randomly out of study.

Concerns/Oddnesses:

Should he be his own decisional authority on study matters?

The trial forbade entry of patients who were suicide risks, but not of patients who were homicide risks(!) Dan had threatened to kill others, but not himself

45 CFR§46.111 (b); 21 CFR§56.111 (b)

The story continues:

Dan’s mother objected to his study enrollment

She was deemed not to have status in the matter (because Dan’s commitment had been stayed, she was not named guardian)

Dan was randomized to receive Seroquel™, and was discharged to a halfway house

The story continues:

Dan seemed improved at his psychiatrist visits, but they were infrequent

Social workers and Dan’s mother recorded concerns that he was still delusional and still at risk to harm himself or others

Dan’s mother asked that he be removed from the study and tried on a different drug - again deemed to have no status

The story continues:

At the end of the six-month stay, his psychiatrist asked that the stay be extended (rather than effecting or vacating the commitment) -- might be at risk if he stops treatment

Mother warned in writing that she feared he would kill himself or someone else; study participation continued

Dan killed himself six months into study participation

Postscript:

(Civil legal ugliness I won’t get into…)

FDA review: Not clear that Dan died because of study participation; no action taken

Prosecutorial review: Not clear that Dan died because of study participation; no criminal liability found; no action taken

Questions:

Would his drug regimen have been changed if he were not in a study?

Would his protections have been better if he had in fact been committed and had a guardian?

Was study participation in fact a contributor to his suicide (or was his schizophrenia beyond help)?

Postscript ii:

Several faculty members asked for an outside review, arguing that “nothing worth prosecuting” isn’t a high enough standard…

Third example:

• Popular anti-inflammatory drug about to have its patent expire

• It has an aspirin-like effect on platelets, so might prevent certain types of clots

• If so, it may make an intravenous form of the drug useful

• New formulation new patent $$$$

• Let’s try it!

The experiment worked:

• Intravenous drug prevented experimental clots in veins that had been chemically, thermally or abrasionally damaged

• Worked better than having the animals on warfarin for a week in advance of the injury

• Worked almost as well as heparin (the standard i.v. anticoagulant)

• The sponsor decided not to try the study in people

The experiment worked TOO WELL:

• The drug levels that prevented clots could be achieved with the oral form already on the market

• No new form no new patent no big bucks (Schillinge, actually – this was in Vienna before the Euro came in)

• When we now add this drug to a failing anticoagulant regimen, we’re doing so on the strength of unpublished bunny data

The three examples show several of the problems one might encounter in industry-sponsored research.As Claude Bernard warned 150 years ago,

going into research with a desired outcome in mind distorts the entire scientific process:

• What questions you ask• How you ask them• How you select subjects• How you analyze results• How (even whether) you report results

The three examples also suggest that the incentive structure might be important:If your desired outcome is support of product

development or marketing, it further influences:

• What questions you ask• How you ask them• How you select subjects• How you analyze results• How (even whether) you report results

How big a deal is this?

Marketing has become a much larger fraction of drug-company budgets;

Firewalls between marketing and research components of drug companies have largely been taken down;

This may now be a bigger concern than it was 25 years ago Carl Elliott, M.D.

The questions you ask:

• Questions that will likely gain evidence in support of your desired outcome (“counterproof”)

• Questions that will support the licensure and marketing of your product

• Questions that will demonstrate a niche in which your product might outperform others

The questions you might not ask:

• Questions that will critically challenge the hypotheses that underpin your desired outcome (“counterproof”)

• Questions that will sensitively detect the limitations of your product (toxicities; drug interactions)

• Questions that will demonstrate a niche in which your product might be outperformed by others

• Questions supporting strategies that do not lead to profit

How you ask the question:

• Choose subjects who are most likely to benefit; least likely to have problems

• Structure the competitor arms so that they are unlikely to be as good as they could be

• Choose a sample size and study interval that make it unlikely that you will detect/document any uncommon problems

• Make the primary endpoint something that advantages your product

Reality check:

• It’s not automatically evil to do a study that evaluates a product under the most favorable circumstances … you have to find out if it CAN work

• It becomes a problem when those studies are presented as though they are more representative and more definitive

• It becomes a problem when the more definitive and more generalizable studies never happen, or never get published

Reality check ii:

• It’s not just industry-sponsored research• The same sort of intellectual traps and

adverse incentive structures are encountered when someone is pursuing an idea on which he’s built his career, or when he’s pursuing his next major grant

How (or whether) you report results:

• Shout from the rooftops when you hypothesis (or product) is supported

• Publish in a high-profile journal• Distribute zillions of reprints

• Put the caveats in a separate publication with a lower profile

• Publish unfavorable studies obscurely or not at all

Is this a theoretical concern or real?

Let’s look at the CAFÉ study, the one that Dan was enrolled in:

• Authors included 3 AstraZenica employees and seven extramural psychiatrists

• 400 subjects• Primary endpoint: Still taking drug at one

year? (A “combined endpoint” for efficacy and tolerance)

• Non-inferiority design

CAFÉ study

• Only 119 of the 400 subjects were still taking the drug at one year

• Publication doesn’t tell why those who stopped the drug did so (side effects vs. ineffectiveness)

• Conclusion: “Non-inferior”

CAFÉ study

• With three groups and only 119 outcomes, a huge difference could fail to be detected statistically.

• For example, if the three groups had 25, 47 and 47 (rather than equal numbers of favorable outcomes), that would not have reached the customary statistical significance threshold of p < 0.05.

CAFÉ study• The study was underpowered to detect a

difference, but its primary outcome was “no difference.”

• That “no difference” is then described with phrases like “comparable efficacy,” obscuring the fact that the study was pretty insensitive.

• This turned out to be really important, because an unpublished study suggested that the sponsor’s drug might actually be inferior!

CAFÉ study

• No comparisons were made to older antipsychotic agents, such as haloperidol

• With the newer agents’ superiority being challenged, even a good study comparing the three newcomers might have been asking the wrong basic question.

• Carl Elliott (and others) asks: Was this study a marketing exercise or a scientific one?

Carl Elliott (reviewing several critiques of the CAFÉ study):

“If these experts are right, then the study in which Dan Markingson committed suicide was not simply a matter of inadequate informed consent, or financial conflicts of interest, or even failure to monitor a subject's care. The ethical breach was built into the study from the start. It is one thing to ask people to take risks for science, or the common good, or to help other people. It is another thing entirely to ask them to risk their lives for the marketing goals of AstraZeneca.”

One Study, or a Pattern?

Meta-analyses have been done both for newer antidepressants and for newer antipsychotic medications

Invariably, more than 3/4 of the published studies have reported results favoring the product of the sponsoring pharmaceutical company

This appears NOT just to be publication bias; the meta-analysts in some studies have looked for – and found – study design bias in a substantial proportion of reports

So Where Are WE?

The balance is tricky:

Pharmaceutical and device manufacturers have a need to test their products

Proof-of-concept studies (CAN it work?) have a legitimate place in product development – but often are limited in their generalizability

Real-world studies (will it work in practice?) may be more practical, but may miss niche utility

Studies comparing products may be very useful, but may also be misleading if designed (or promulgated) from a marketing perspective.

The balance is tricky:

Concealing study results that do not favor your product is wrong if it puts anybody at risk

Potential subjects have a right to know if the risk they are being asked to bear is to answer a marketing question, a curiosity question, or a question with immediate implications for best clinical practice

An IRB is charged to make the call whether the benefit of a study is enough to warrant the risks subjects are asked to bear -- who’s to help them decide the tough cases?

And just to make sure you’re challenged:

The Markingson case led to some lawsuits.One finding might surprise you: The drug

manufacturer’s fiduciary responsibility is to its shareholdersNo legal (as opposed to ethical) responsibility to put the welfare of the subjects ahead of that of the shareholders

Thus, no grounds for tort action if regulations complied with

If we could turn back the clock:

Direct marketing of drugs and devices to consumers has led to huge marketing juggernauts in many companies; the old firewalls between marketing and science were probably a good thing.

We can’t turn back the clock, but perhaps a regulatory structure could be devised to make it harder for marketing considerations to distort the science.

What we can do now:

We can be more diligent in trying to identify studies that are likely to be market driven, and make certain that their real nature is frankly disclosed to subjects.

We can support proposals that would require product-comparison studies to be independently designed and conducted, rather than being the property of someone with a stake in the outcome.

It’s not just the sponsors:

Comprehensive Cancer Centers are scored on the proportion of patients entered on study protocols – that creates a pressure to have a study (ANY study) for every patient

My own division has come under pressure to implement more Phase-I studies, because of their favorable impact on hospital and clinic revenue streams.

Claude Bernard saw it coming

But I’ll bet he didn’t foreseethe enormity of the financial

traps that are there toaugment the intellectual

ones. His warnings arejust as timely now as

in 1865 … but theirchallenge has grown.