Are most positive findings in psychology false or exaggerated? An activist's perspective

Are most positive findings in psychology false or exaggerated? An activist's perspective

Colloquium at ANU Research School of Psychology

Sept 10, 2014

James C. Coyne, Ph.D.University of Groningen University Medical Center Groningen (UMCG),

Groningen, the Netherlands

jcoynester@gmail.com

John Ioannidis documented many positive findings in the biomedical literature do not replicate and many apparent “discoveries” turn out to be exaggerated or simply false.

Many apparent discoveries are created by a combination of confirmatory bias, flexible rules of design, data analysis and reporting and significance chasing.

“It can be proven that most claimed research findings are false”

Ioannidis, JPA. (2005). Why most published research findings are false. PLOS Medicine 2: 696-701.

Ioannidis, JPA. (2005). Contradicted and initially stronger effects in highly cited clinical research. JAMA 294: 218-228.

Young, NS., Ioannidis, JPA. et al. (2008). Why Current Publication Practices May Distort Science. PLOS Medicine 5: 1418-1422.

Ben Goldacre

Drugs are tested by the people who manufacture them, in poorly designed trials, on hopelessly small numbers of weird, unrepresentative patients, and analysed using techniques which are flawed by design, in such a way that they exaggerate the benefits of treatments. Unsurprisingly, these trials tend to produce results that favour the manufacturer. When trials throw up results that companies don't like, they are perfectly entitled to hide them from doctors and patients.

Efforts at reform

Preregistration of clinical trials make it more difficult to hide negative trials or alter analytic plans after results are known.

Reporting standards ensure more transparent and detailed article so results can be independently validated.

Reporting standards and pre-registration of plans for meta-analyses and systematic reviews.

Making data available for independent reanalysis.

Our modest contribution to reform

Roseman, M, Milette, K, Bero, LA, Coyne, JC, Lexchin, J., Turner, EH, & Thombs, BD. (2011). Reporting of conflicts of interest in meta-analyses of trials of pharmacological treatments. JAMA, 2011;305(10):1008-17.

Roseman, M, Turner, EH, Lexchin, J., Coyne, JC, Bero, LA, & Thombs, BD. (2012). Reporting of conflicts of interest from drug trials in Cochrane reviews: cross sectional study. BMJ, 2012; 345.

We documented with association of source of funding with claimed size of effects.

We proposed considering conflict of interest as a moderator, source of heterogeneity, and another risk of bias in Cochrane risk of bias assessment.

The Cochrane Collaboration agreed.

Cochrane Risk of Bias

Sequence generation.

Allocation concealment.

Blinding of participants, investigators, outcome assessors.

Incomplete outcome data.

Selective outcome reporting.

Other threats to validity.

Trickle down of reform

Changes forced upon Pharma slowly and inconsistently reach clinical psychology.

Larger data sets allow exploration of issues such as choice of control groups and investigator allegiance.

Behavioral Medicine adopts changes occurring in clinical psychology later and inconsistently.

Psychology's crisis of credibility

Instances of outright fraud are unusual, but confirmatory bias is rampant and supported by diffuse institutional forces.

On the other hand, when fraud is consistent with prevailing biases, it can be tough to root out.

Getting Hans Eysenck’s fraudulent Getting Hans Eysenck’s fraudulent data out of the literaturedata out of the literature

● Co-authored papers with Grossarth-Maticek that claimed improbably high associations between personality and cancer and personality and smoking and published in own journal.

● Later it was revealed that he was paid substantially high amounts of money by attorneys for US tobacco companies to publish these data.

In a literature dominated by hazard ratios (HRs) clustered around 1.0, Eysenck/Grossarth-Maticek’s were extreme outliers, absurdly so.

Their influence nonetheless lives on in inflated overall estimates of the effects of negative affect on cancer in tertiary sources that do not acknowledge their tainted origins.

Grossarth-Maticek/Eysenck Fake Data

Coyne JC, Ranchor AV, Palmer SC. Meta-analysis of stress-related factors in cancer. Nat Rev Clin Oncol. 2010;7(5).

Ranchor AV, Sanderman R, Coyne JC. Invited Commentary: Personality as a Causal Factor in Cancer Risk and Mortality-Time to Retire a Hypothesis? American Journal of Epidemiology. 2010;172(4):386-8.

Coyne JC, Johansen C. Confirmatory bias and the persistent influence of discredited data in interpreting the stress-cancer link: Commentary on Michael et al. (2009). Health Psychol. 2011;30(3):374-5

Confirmatory bias entrenched, preached to be a virtue.

Advice of a Famous Social Advice of a Famous Social PsychologistPsychologist

“There are two possible articles you can write: (1) the article you planned to write when you designed your study or (2) the article that makes the most sense now that you have seen the results. They are rarely the same, and the correct answer is (2).”

-Bem, 2003, pp. 171-172

Breakthrough “discoveries” are created and perpetuated by a combination of

confirmatory bias, flexible rules of design, data analysis and reporting and

significance chasing.

APS Journals Have not adopted CONSORT.

Violate COPE in not having procedure for appeal of negative decisions.

Strong confirmatory bias, newsworthiness over science.

Aversion to null findings and replications.

Don’t adhere to Pottery Barn Rule.

From: jcoyne@mail.med.upenn.edu <jcoynester@gmail.com>Date: Wed, Jun 11, 2014 at 4:26 PMSubject: Fwd: Appeal re Manuscript PSCI-13-2172To: Alan Kraut <akraut@psychologicalscience.org

I respect the autonomy of APS publications from the CEO. But I think that the new editor of Psychological Science is only going to compound his losses if he continues to fight giving us an appeal consistent with the standards of Committee on Publication Ethics (COPE). I think that COPE and the larger scientific community will be solidly on our side, and will generate considerable outrage. Hopefully, you would not consider it inappropriate to give him some sage advice on this.

Sorry to be writing to you under these circumstances.

Skepticism about the replication movement.

May only serve to ghettoize null findings and failures to replicate.

Protect bias in prestigious journals toward false-positive findings and discrimination against attempted replication and now findings.

Sleeping with the enemy.

The spin cycle in scientific publishing: The spin cycle in scientific publishing: is it necessary to have positive results is it necessary to have positive results to get published?to get published?

SpinSpin

“Spin”-- Distortion to impress readers that something is noteworthy.

Objective: To identify nature and frequency of spin in 62 published reports of RCTs with statistically nonsignificant results.

Conclusion: Reporting and interpretation of findings frequently inconsistent with the results.

SpinSpin

Title was reported with spin in 13 articles (18.0%).

Spin was identified in the Results (37.5%) and Conclusions (58.3%) with conclusions of 17 (23.6%) focusing only on treatment effectiveness.

Spin was identified in the main-text Results (29.2%), Discussion (43.1%),, and Conclusions (50.0%).

More than 40% of reports had spin in at least 2 of these sections in the main text.

The greatest threat [to the credibility of clinical research] may come from the poor relevance and scientific rationale and thus low pre-study odds of success of research efforts. Given that we currently have too many research findings, often with low credibility, replication and rigorous evaluation becomes as important or even more important than discovery. Credibility, replication, and translation are all desirable properties of research findings, but only modestly correlated.

Significance chasing biasSignificance chasing bias

“It is a common failing – and one I have myself suffered from – to fall in love with a hypothesis and to be unwilling to take no for an answer. A love affair with a pet hypothesis can waste years of precious time. There is very often no finally decisive yes, though quite often there can be a decisive no.”

-P. B. Medawar

Copyright ©2005 American Heart Association

Loscalzo, J. Circulation 2005;112:3026-3029

Incentives and pressures for a Incentives and pressures for a confirmatory biasconfirmatory bias

Additional sources of confirmatory Additional sources of confirmatory biasbias

Professional organizations need positive findings to support guidelines and recommendations requiring services of their members.

Journals need positive findings to raise impact factor.

Gurus and advocates of particular treatments need positive findings to promote their interventions and workshops ($$$) and feed their vanity.

Authors need positive findings to get published and advance their careers.

Pressures from journals

Competition for subscriptions, advertising, and prestige depend on the JIF.

JIF depends on number of citations for articles within 2 years of publication.

Newsworthy, citable articles with clock starting at point of “early release.”

Articles with media buzz are more likely to be immediately cited.

Null findings, replications less newsworthy.

JIF and Retractions Correlated

Hot areas plagued by confirmatory Hot areas plagued by confirmatory biasbias

Mind-body relations--psychoneuroimmunology (PNI)

Gene-environment interactions--particularly serotonin transport genes—GWAS, and genomics

Neuroscience

Screening for depression and distress

Claims that psychotherapies are evidence-based

In approaching a hot area of research, you have to keep stopping and asking,

“Are a particular set of results interesting because they advance the field?”

OR

“Are they interesting results only because of some vague relevance claimed to interesting ideas that are not actually being tested?”

Need for activism

Cannot depend on journals to clean up editorial policies and review processes.

Journals inadequately support reform or self-correct.

Institutionalization of confirmatory bias and routine acceptance of bad science.

Promoting reforms

Strengthen post publication peer review. Set standards for conducting and reporting

clinical trials and meta analyses/systematic reviews

Greater transparency and sharing of data. Preregistration of observational studies and

hypotheses, as well as clinical trials. Outing of conflicts of interest. Use of social media to level the playing field.

Targeted takedowns of bad science highlight the problem and model solutions. Social media campaigns: Critique by blogging,

Twitter and Facebook.

Outing and confrontation with bad editorial practices.

Letters to the editor are ineffective, but PubMed Commons is promising.

Calls for retractions and erratum.

.

Pet peeves

Hype and hokum calculated to attract interest despite being premature or exaggerated.

Ideas and practices which are potentially harmful to consumers.

Undisclosed conflicts of interest, both financial and nonfinancial.

Repeat offenders.

Pet peeves

Privileged access to supposedly peer-reviewed outlets.

Editors and reviewers' abuse of the review process to suppress alternative points of view and embarrassing data.

Gurus who undermine consumers' sense of self efficacy and increase their need for experts.

Gurus who sell products based on claims of being scientists.

““Official” scienceOfficial” science

Paradigms or ideas insulated from full critical scrutiny and standards of evidence by endorsement by funding agencies, professional organizations, and flagship journals.

Beware of Professional Organization Involvement as a Risk

of Bias

AHA Advisory on Screening for Depression Not Guideline-Congruent!

Ziegelstein RC, Brett D, Thombs BD, Coyne JC, de Jonge P. Routine Screening for Depression in Patients with Coronary Heart Disease: Never Mind. Journal of the

American Academy of Cardiology. 2009;54(10):886-90.

  Thombs, B. D., Jewett, L. R., Knafo, R., Coyne, J. C., &

Ziegelstein, R. C. (2009). Learning from history: a commentary on the American Heart Association Science Advisory on depression screening. American heart journal, 158(4), 503.

What to watch for in meta analyses commissioned by professional organizations such as Society of

Behavioral Medicine (SBM)

Precommitment to conclusion that interventions work and are ready for dissemination and reimbursement.

Ignoring of preponderance of methodologically flawed, unpowered sources of bias.

Broad inclusion of diverse interventions into one category.

Ignoring statistical heterogeneity.

Weak assessment of risk of bias (Cochrane criteria are seldom used).

Most positive findings are false or Most positive findings are false or exaggerated: Tools for reformexaggerated: Tools for reform

A general heightened awareness of the problem and a healthy skepticism to what is published in peer-reviewed journals.

Reviews and editors educated about the problem and the need for enforcement of higher standards, conformity to established guidelines and transparency.

Readers/consumers armed with critical tools to evaluate for themselves claims of the literature.

Education of science journalists.

Most positive findings are false or Most positive findings are false or exaggerated: Tools for reformexaggerated: Tools for reform

Strengthened post-publication review.

Reduced resistance to publishing null findings and failure to replicate.

High-profile examples of particularly bad articles being retracted.

Open access journals, blogs, and Twitter.

Going after…

Horrible meta-analyses written by American antiabortion activist, and accepted for publication by devout Catholic editor, Peter Tyrer.

American Anti-Abortionist Gets Flawed Meta Analysis into British Journal of Psychiatry

Going after bad science that misleads cancer patients

• Claimed women with metastatic breast cancer who received supportive-expressive group psychotherapy survived almost twice as long as women in the control group.

• Cited 1164 times

Spiegel D, Bloom JR, Kraemer HC, Spiegel D, Bloom JR, Kraemer HC, Gottheil E (1989). Gottheil E (1989). LancetLancet 2:888-891. 2:888-891.

Takedown of Pseudoscience of Positive Psychology and Health

Blogged at PLOS Mind the Brain

Led to exchange of letters in PNAS

Which led to detailed re-analyses exposing claims as nonsense

Unfair editorial practices

Articles published in peer-reviewed journals that are not peer-reviewed or only minimally.

Use of invited commentaries to selectively highlight or dampen evidence and arguments relevant to politicized topics.

Special issues with restricted calls for papers. Authors’ control of whether critical letters to

the editor of published. Restrictions on letters to the editor,

Conflicts of interest

Psychology editors neither solicit nor routinely publish statements of conflict of interest.

Strong risk of bias associated with investigator allegiance, particularly when a particular treatment is being marketed.

Non-financial conflicts of interest are real and powerful, but not always easy to detect.

Exposures of conflicts of interests (COIs) associated with promoters of treatments conducting original studies and meta-analyses has led to numerous erratum rectifying nondisclosure and exposure of association of COI with confirmatory bias and bad science.

Erratum: one, two, many more

Moving From Criticism and Activism to Proposals for Constructive Change

Democratizing Post-Publication Peer Review

PubMed Commons is a system that enables researchers to share their opinions about scientific publications. Researchers can comment on any publication indexed by PubMed, and read the comments of others.

PubMed Commons is a forum for open and constructive criticism and discussion of scientific issues.

It will thrive with high quality interchange from the scientific community.

Join the Pubmed Commons Revolution

No longer will a few people grant an irrevocable judgment of “must stand because it is

peer reviewed.”

You can take post publication peer review out of the hands

of editors.

No criticism without offering solutions?

It takes a lot of work to explicate bad science and the more general context that supports and rewards it.

Critics should not be required to offer constructive solutions or positive steps, particularly when there is substantial opposition from the journals.

But when a receptive audience has been created, identification of positive steps become more timely.

Thank you

Follow me on Twitter

@CoyneoftheRealm

Blogging at Mind the Brain