Top Banner
Are most positive findings in psychology false or exaggerated? An activist's perspective Colloquium at ANU Research School of Psychology Sept 10, 2014 James C. Coyne, Ph.D. University of Groningen University Medical Center Groningen (UMCG), Groningen, the Netherlands [email protected]
74

Are most positive findings in psychology false or exaggerated? An activist's perspective

Dec 18, 2014

Download

Health & Medicine

James Coyne

Colloquium at Australian National University Research School of Psychology Sept 10, 2014
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Are most positive findings in psychology false or exaggerated? An activist's perspective

Are most positive findings in psychology false or exaggerated? An activist's perspective

Colloquium at ANU Research School of Psychology

Sept 10, 2014

James C. Coyne, Ph.D.University of Groningen University Medical Center Groningen (UMCG),

Groningen, the Netherlands

[email protected]

Page 2: Are most positive findings in psychology false or exaggerated? An activist's perspective

John Ioannidis documented many positive findings in the biomedical literature do not replicate and many apparent “discoveries” turn out to be exaggerated or simply false.

Many apparent discoveries are created by a combination of confirmatory bias, flexible rules of design, data analysis and reporting and significance chasing.

Page 3: Are most positive findings in psychology false or exaggerated? An activist's perspective

“It can be proven that most claimed research findings are false”

Ioannidis, JPA. (2005). Why most published research findings are false. PLOS Medicine 2: 696-701.

Ioannidis, JPA. (2005). Contradicted and initially stronger effects in highly cited clinical research. JAMA 294: 218-228.

Young, NS., Ioannidis, JPA. et al. (2008). Why Current Publication Practices May Distort Science. PLOS Medicine 5: 1418-1422.

Page 4: Are most positive findings in psychology false or exaggerated? An activist's perspective

Ben Goldacre

Drugs are tested by the people who manufacture them, in poorly designed trials, on hopelessly small numbers of weird, unrepresentative patients, and analysed using techniques which are flawed by design, in such a way that they exaggerate the benefits of treatments. Unsurprisingly, these trials tend to produce results that favour the manufacturer. When trials throw up results that companies don't like, they are perfectly entitled to hide them from doctors and patients.

Page 5: Are most positive findings in psychology false or exaggerated? An activist's perspective

Efforts at reform

Preregistration of clinical trials make it more difficult to hide negative trials or alter analytic plans after results are known.

Reporting standards ensure more transparent and detailed article so results can be independently validated.

Reporting standards and pre-registration of plans for meta-analyses and systematic reviews.

Making data available for independent reanalysis.

Page 6: Are most positive findings in psychology false or exaggerated? An activist's perspective

Our modest contribution to reform

Roseman, M, Milette, K, Bero, LA, Coyne, JC, Lexchin, J., Turner, EH, & Thombs, BD. (2011). Reporting of conflicts of interest in meta-analyses of trials of pharmacological treatments. JAMA, 2011;305(10):1008-17.

Roseman, M, Turner, EH, Lexchin, J., Coyne, JC, Bero, LA, & Thombs, BD. (2012). Reporting of conflicts of interest from drug trials in Cochrane reviews: cross sectional study. BMJ, 2012; 345.

Page 7: Are most positive findings in psychology false or exaggerated? An activist's perspective
Page 8: Are most positive findings in psychology false or exaggerated? An activist's perspective

We documented with association of source of funding with claimed size of effects.

We proposed considering conflict of interest as a moderator, source of heterogeneity, and another risk of bias in Cochrane risk of bias assessment.

The Cochrane Collaboration agreed.

Page 9: Are most positive findings in psychology false or exaggerated? An activist's perspective

Cochrane Risk of Bias

Sequence generation.

Allocation concealment.

Blinding of participants, investigators, outcome assessors.

Incomplete outcome data.

Selective outcome reporting.

Other threats to validity.

Page 10: Are most positive findings in psychology false or exaggerated? An activist's perspective

Trickle down of reform

Changes forced upon Pharma slowly and inconsistently reach clinical psychology.

Larger data sets allow exploration of issues such as choice of control groups and investigator allegiance.

Behavioral Medicine adopts changes occurring in clinical psychology later and inconsistently.

Page 11: Are most positive findings in psychology false or exaggerated? An activist's perspective

Psychology's crisis of credibility

Page 12: Are most positive findings in psychology false or exaggerated? An activist's perspective
Page 13: Are most positive findings in psychology false or exaggerated? An activist's perspective
Page 14: Are most positive findings in psychology false or exaggerated? An activist's perspective

Instances of outright fraud are unusual, but confirmatory bias is rampant and supported by diffuse institutional forces.

On the other hand, when fraud is consistent with prevailing biases, it can be tough to root out.

Page 15: Are most positive findings in psychology false or exaggerated? An activist's perspective

Getting Hans Eysenck’s fraudulent Getting Hans Eysenck’s fraudulent data out of the literaturedata out of the literature

● Co-authored papers with Grossarth-Maticek that claimed improbably high associations between personality and cancer and personality and smoking and published in own journal.

● Later it was revealed that he was paid substantially high amounts of money by attorneys for US tobacco companies to publish these data.

Page 16: Are most positive findings in psychology false or exaggerated? An activist's perspective

In a literature dominated by hazard ratios (HRs) clustered around 1.0, Eysenck/Grossarth-Maticek’s were extreme outliers, absurdly so.

Their influence nonetheless lives on in inflated overall estimates of the effects of negative affect on cancer in tertiary sources that do not acknowledge their tainted origins.

Page 17: Are most positive findings in psychology false or exaggerated? An activist's perspective

Grossarth-Maticek/Eysenck Fake Data

Page 18: Are most positive findings in psychology false or exaggerated? An activist's perspective

Coyne JC, Ranchor AV, Palmer SC. Meta-analysis of stress-related factors in cancer. Nat Rev Clin Oncol. 2010;7(5).

Ranchor AV, Sanderman R, Coyne JC. Invited Commentary: Personality as a Causal Factor in Cancer Risk and Mortality-Time to Retire a Hypothesis? American Journal of Epidemiology. 2010;172(4):386-8.

Coyne JC, Johansen C. Confirmatory bias and the persistent influence of discredited data in interpreting the stress-cancer link: Commentary on Michael et al. (2009). Health Psychol. 2011;30(3):374-5

Page 19: Are most positive findings in psychology false or exaggerated? An activist's perspective

Confirmatory bias entrenched, preached to be a virtue.

Page 20: Are most positive findings in psychology false or exaggerated? An activist's perspective

Advice of a Famous Social Advice of a Famous Social PsychologistPsychologist

“There are two possible articles you can write: (1) the article you planned to write when you designed your study or (2) the article that makes the most sense now that you have seen the results. They are rarely the same, and the correct answer is (2).”

-Bem, 2003, pp. 171-172

Page 21: Are most positive findings in psychology false or exaggerated? An activist's perspective

Breakthrough “discoveries” are created and perpetuated by a combination of

confirmatory bias, flexible rules of design, data analysis and reporting and

significance chasing.

Page 22: Are most positive findings in psychology false or exaggerated? An activist's perspective
Page 23: Are most positive findings in psychology false or exaggerated? An activist's perspective
Page 24: Are most positive findings in psychology false or exaggerated? An activist's perspective

APS Journals Have not adopted CONSORT.

Violate COPE in not having procedure for appeal of negative decisions.

Strong confirmatory bias, newsworthiness over science.

Aversion to null findings and replications.

Don’t adhere to Pottery Barn Rule.

Page 25: Are most positive findings in psychology false or exaggerated? An activist's perspective

From: [email protected] <[email protected]>Date: Wed, Jun 11, 2014 at 4:26 PMSubject: Fwd: Appeal re Manuscript PSCI-13-2172To: Alan Kraut <[email protected]

I respect the autonomy of APS publications from the CEO. But I think that the new editor of Psychological Science is only going to compound his losses if he continues to fight giving us an appeal consistent with the standards of Committee on Publication Ethics (COPE). I think that COPE and the larger scientific community will be solidly on our side, and will generate considerable outrage. Hopefully, you would not consider it inappropriate to give him some sage advice on this.

Sorry to be writing to you under these circumstances.

Page 26: Are most positive findings in psychology false or exaggerated? An activist's perspective

Skepticism about the replication movement.

May only serve to ghettoize null findings and failures to replicate.

Protect bias in prestigious journals toward false-positive findings and discrimination against attempted replication and now findings.

Sleeping with the enemy.

Page 27: Are most positive findings in psychology false or exaggerated? An activist's perspective

The spin cycle in scientific publishing: The spin cycle in scientific publishing: is it necessary to have positive results is it necessary to have positive results to get published?to get published?

Page 28: Are most positive findings in psychology false or exaggerated? An activist's perspective

SpinSpin

“Spin”-- Distortion to impress readers that something is noteworthy.

Objective: To identify nature and frequency of spin in 62 published reports of RCTs with statistically nonsignificant results.

Conclusion: Reporting and interpretation of findings frequently inconsistent with the results.

Page 29: Are most positive findings in psychology false or exaggerated? An activist's perspective

SpinSpin

Title was reported with spin in 13 articles (18.0%).

Spin was identified in the Results (37.5%) and Conclusions (58.3%) with conclusions of 17 (23.6%) focusing only on treatment effectiveness.

Spin was identified in the main-text Results (29.2%), Discussion (43.1%),, and Conclusions (50.0%).

More than 40% of reports had spin in at least 2 of these sections in the main text.

Page 30: Are most positive findings in psychology false or exaggerated? An activist's perspective

The greatest threat [to the credibility of clinical research] may come from the poor relevance and scientific rationale and thus low pre-study odds of success of research efforts. Given that we currently have too many research findings, often with low credibility, replication and rigorous evaluation becomes as important or even more important than discovery. Credibility, replication, and translation are all desirable properties of research findings, but only modestly correlated.

Significance chasing biasSignificance chasing bias

Page 31: Are most positive findings in psychology false or exaggerated? An activist's perspective

“It is a common failing – and one I have myself suffered from – to fall in love with a hypothesis and to be unwilling to take no for an answer. A love affair with a pet hypothesis can waste years of precious time. There is very often no finally decisive yes, though quite often there can be a decisive no.”

-P. B. Medawar

Page 32: Are most positive findings in psychology false or exaggerated? An activist's perspective

Copyright ©2005 American Heart Association

Loscalzo, J. Circulation 2005;112:3026-3029

Incentives and pressures for a Incentives and pressures for a confirmatory biasconfirmatory bias

Page 33: Are most positive findings in psychology false or exaggerated? An activist's perspective

Additional sources of confirmatory Additional sources of confirmatory biasbias

Professional organizations need positive findings to support guidelines and recommendations requiring services of their members.

Journals need positive findings to raise impact factor.

Gurus and advocates of particular treatments need positive findings to promote their interventions and workshops ($$$) and feed their vanity.

Authors need positive findings to get published and advance their careers.

Page 34: Are most positive findings in psychology false or exaggerated? An activist's perspective

Pressures from journals

Competition for subscriptions, advertising, and prestige depend on the JIF.

JIF depends on number of citations for articles within 2 years of publication.

Newsworthy, citable articles with clock starting at point of “early release.”

Articles with media buzz are more likely to be immediately cited.

Null findings, replications less newsworthy.

Page 35: Are most positive findings in psychology false or exaggerated? An activist's perspective

JIF and Retractions Correlated

Page 36: Are most positive findings in psychology false or exaggerated? An activist's perspective

Hot areas plagued by confirmatory Hot areas plagued by confirmatory biasbias

Mind-body relations--psychoneuroimmunology (PNI)

Gene-environment interactions--particularly serotonin transport genes—GWAS, and genomics

Neuroscience

Screening for depression and distress

Claims that psychotherapies are evidence-based

Page 37: Are most positive findings in psychology false or exaggerated? An activist's perspective

In approaching a hot area of research, you have to keep stopping and asking,

“Are a particular set of results interesting because they advance the field?”

OR

“Are they interesting results only because of some vague relevance claimed to interesting ideas that are not actually being tested?”

Page 38: Are most positive findings in psychology false or exaggerated? An activist's perspective

Need for activism

Cannot depend on journals to clean up editorial policies and review processes.

Journals inadequately support reform or self-correct.

Institutionalization of confirmatory bias and routine acceptance of bad science.

Page 39: Are most positive findings in psychology false or exaggerated? An activist's perspective

Promoting reforms

Strengthen post publication peer review. Set standards for conducting and reporting

clinical trials and meta analyses/systematic reviews

Greater transparency and sharing of data. Preregistration of observational studies and

hypotheses, as well as clinical trials. Outing of conflicts of interest. Use of social media to level the playing field.

Page 40: Are most positive findings in psychology false or exaggerated? An activist's perspective

Targeted takedowns of bad science highlight the problem and model solutions. Social media campaigns: Critique by blogging,

Twitter and Facebook.

Outing and confrontation with bad editorial practices.

Letters to the editor are ineffective, but PubMed Commons is promising.

Calls for retractions and erratum.

.

Page 41: Are most positive findings in psychology false or exaggerated? An activist's perspective

Pet peeves

Hype and hokum calculated to attract interest despite being premature or exaggerated.

Ideas and practices which are potentially harmful to consumers.

Undisclosed conflicts of interest, both financial and nonfinancial.

Repeat offenders.

Page 42: Are most positive findings in psychology false or exaggerated? An activist's perspective

Pet peeves

Privileged access to supposedly peer-reviewed outlets.

Editors and reviewers' abuse of the review process to suppress alternative points of view and embarrassing data.

Gurus who undermine consumers' sense of self efficacy and increase their need for experts.

Gurus who sell products based on claims of being scientists.

Page 43: Are most positive findings in psychology false or exaggerated? An activist's perspective

““Official” scienceOfficial” science

Paradigms or ideas insulated from full critical scrutiny and standards of evidence by endorsement by funding agencies, professional organizations, and flagship journals.

Page 44: Are most positive findings in psychology false or exaggerated? An activist's perspective

Beware of Professional Organization Involvement as a Risk

of Bias

Page 45: Are most positive findings in psychology false or exaggerated? An activist's perspective

AHA Advisory on Screening for Depression Not Guideline-Congruent!

Ziegelstein RC, Brett D, Thombs BD, Coyne JC, de Jonge P. Routine Screening for Depression in Patients with Coronary Heart Disease: Never Mind. Journal of the

American Academy of Cardiology. 2009;54(10):886-90.

  Thombs, B. D., Jewett, L. R., Knafo, R., Coyne, J. C., &

Ziegelstein, R. C. (2009). Learning from history: a commentary on the American Heart Association Science Advisory on depression screening. American heart journal, 158(4), 503.

Page 46: Are most positive findings in psychology false or exaggerated? An activist's perspective

What to watch for in meta analyses commissioned by professional organizations such as Society of

Behavioral Medicine (SBM)

Precommitment to conclusion that interventions work and are ready for dissemination and reimbursement.

Ignoring of preponderance of methodologically flawed, unpowered sources of bias.

Broad inclusion of diverse interventions into one category.

Ignoring statistical heterogeneity.

Weak assessment of risk of bias (Cochrane criteria are seldom used).

Page 47: Are most positive findings in psychology false or exaggerated? An activist's perspective

Most positive findings are false or Most positive findings are false or exaggerated: Tools for reformexaggerated: Tools for reform

A general heightened awareness of the problem and a healthy skepticism to what is published in peer-reviewed journals.

Reviews and editors educated about the problem and the need for enforcement of higher standards, conformity to established guidelines and transparency.

Readers/consumers armed with critical tools to evaluate for themselves claims of the literature.

Education of science journalists.

Page 48: Are most positive findings in psychology false or exaggerated? An activist's perspective

Most positive findings are false or Most positive findings are false or exaggerated: Tools for reformexaggerated: Tools for reform

Strengthened post-publication review.

Reduced resistance to publishing null findings and failure to replicate.

High-profile examples of particularly bad articles being retracted.

Open access journals, blogs, and Twitter.

Page 49: Are most positive findings in psychology false or exaggerated? An activist's perspective

Going after…

Horrible meta-analyses written by American antiabortion activist, and accepted for publication by devout Catholic editor, Peter Tyrer.

Page 50: Are most positive findings in psychology false or exaggerated? An activist's perspective

American Anti-Abortionist Gets Flawed Meta Analysis into British Journal of Psychiatry

Page 51: Are most positive findings in psychology false or exaggerated? An activist's perspective

Going after bad science that misleads cancer patients

Page 52: Are most positive findings in psychology false or exaggerated? An activist's perspective

• Claimed women with metastatic breast cancer who received supportive-expressive group psychotherapy survived almost twice as long as women in the control group.

• Cited 1164 times

Spiegel D, Bloom JR, Kraemer HC, Spiegel D, Bloom JR, Kraemer HC, Gottheil E (1989). Gottheil E (1989). LancetLancet 2:888-891. 2:888-891.

Page 53: Are most positive findings in psychology false or exaggerated? An activist's perspective
Page 54: Are most positive findings in psychology false or exaggerated? An activist's perspective
Page 55: Are most positive findings in psychology false or exaggerated? An activist's perspective

Takedown of Pseudoscience of Positive Psychology and Health

Blogged at PLOS Mind the Brain

Led to exchange of letters in PNAS

Which led to detailed re-analyses exposing claims as nonsense

Page 56: Are most positive findings in psychology false or exaggerated? An activist's perspective
Page 57: Are most positive findings in psychology false or exaggerated? An activist's perspective

Unfair editorial practices

Articles published in peer-reviewed journals that are not peer-reviewed or only minimally.

Use of invited commentaries to selectively highlight or dampen evidence and arguments relevant to politicized topics.

Special issues with restricted calls for papers. Authors’ control of whether critical letters to

the editor of published. Restrictions on letters to the editor,

Page 58: Are most positive findings in psychology false or exaggerated? An activist's perspective
Page 59: Are most positive findings in psychology false or exaggerated? An activist's perspective
Page 60: Are most positive findings in psychology false or exaggerated? An activist's perspective

Conflicts of interest

Psychology editors neither solicit nor routinely publish statements of conflict of interest.

Strong risk of bias associated with investigator allegiance, particularly when a particular treatment is being marketed.

Non-financial conflicts of interest are real and powerful, but not always easy to detect.

Page 61: Are most positive findings in psychology false or exaggerated? An activist's perspective

Exposures of conflicts of interests (COIs) associated with promoters of treatments conducting original studies and meta-analyses has led to numerous erratum rectifying nondisclosure and exposure of association of COI with confirmatory bias and bad science.

Page 62: Are most positive findings in psychology false or exaggerated? An activist's perspective
Page 63: Are most positive findings in psychology false or exaggerated? An activist's perspective

Erratum: one, two, many more

Page 64: Are most positive findings in psychology false or exaggerated? An activist's perspective
Page 65: Are most positive findings in psychology false or exaggerated? An activist's perspective
Page 66: Are most positive findings in psychology false or exaggerated? An activist's perspective

Moving From Criticism and Activism to Proposals for Constructive Change

Page 67: Are most positive findings in psychology false or exaggerated? An activist's perspective

Democratizing Post-Publication Peer Review

Page 68: Are most positive findings in psychology false or exaggerated? An activist's perspective

PubMed Commons is a system that enables researchers to share their opinions about scientific publications. Researchers can comment on any publication indexed by PubMed, and read the comments of others.

Page 69: Are most positive findings in psychology false or exaggerated? An activist's perspective

PubMed Commons is a forum for open and constructive criticism and discussion of scientific issues.

It will thrive with high quality interchange from the scientific community.

Page 70: Are most positive findings in psychology false or exaggerated? An activist's perspective

Join the Pubmed Commons Revolution

No longer will a few people grant an irrevocable judgment of “must stand because it is

peer reviewed.”

You can take post publication peer review out of the hands

of editors.

Page 71: Are most positive findings in psychology false or exaggerated? An activist's perspective

No criticism without offering solutions?

It takes a lot of work to explicate bad science and the more general context that supports and rewards it.

Critics should not be required to offer constructive solutions or positive steps, particularly when there is substantial opposition from the journals.

But when a receptive audience has been created, identification of positive steps become more timely.

Page 72: Are most positive findings in psychology false or exaggerated? An activist's perspective
Page 73: Are most positive findings in psychology false or exaggerated? An activist's perspective
Page 74: Are most positive findings in psychology false or exaggerated? An activist's perspective

Thank you

Follow me on Twitter

@CoyneoftheRealm

Blogging at Mind the Brain