Top Banner
Forskningsdesigns og evidens Bobby Zachariae Professor, dr.med., cand.psych. Aarhus Universitetshospital og Aarhus Universitet
73

Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Jul 03, 2020

Download

Documents

dariahiddleston
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Forskningsdesignsogevidens

BobbyZachariaeProfessor,dr.med.,cand.psych.AarhusUniversitetshospitalog

AarhusUniversitet

Page 2: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Aimsofthepresentation

• Evaluatingtheavailableevidence– challengesandavailablesolutions– Problemsinevery-daythinking– Aimsofscientificmethod– Typesofevidence– Threatstovalidity–Whenonestudyisnotenough– Aspectstoconsiderwhenevaluating“effectiveness”

Zachariae

Page 3: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

UnitforPsychooncologyandHealthPsychology - EPoS

• PsychooncologyResearchUnitestablishedin2000attheDept.OfOncology,AUHbasedonagrantfromtheDanishCancerSociety

• EPoS establishedin2011incollaborationbetweenAUH,Dept.OfOncology,BSS,AU,andDeptofPsychologyandBehavioural Science

• Currentstaff:17(1professor,2assoc.prof.1assistprof,1seniorresearcher,4post-docs,7PhD’s,1adm.)+8-10researchassistants.

Page 4: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Mybackground

• Researchareas– Psychoneuroimmunology– Painresearch– Psycho-socialcancerresearch– Patient-healthprofessionalinteractions– Healthpsychology– Psychosocialinterventions– Hypnosis,guidedimagery,mindfulness-basedintervention,expressivewriting

– Internet-deliveredinterventions– Efficacyofcomplementaryandalternativetreatments

4Zachariae

Page 5: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Mybackground• Researchmethodologies– Experimentalstudies(psychophysiology)– RandomizedControlledTrials(RCT)– Pragmatictrials–Mixed(qualitativeandquantitative)methods– Cross-sectionalandcohortstudies– Psychometrics– Systematicreviewsandmeta-analysis

5Zachariae

Page 6: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

• Intellectualadvisetofuturegenerations:

• “Askyourselfonlywhatarethefactsandwhatisthetruththatthefactsbearout.Neverletyourselfbedivertedeitherbywhatyouwishtobelieve,orbywhatyouthinkwouldhavebeneficentsocialeffectsifitwerebelieved.Butlookonly,andsolely,atwhatarethefacts.”

Zachariae6

BertrandRussell,1872-1970

Page 7: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Whyscientificmethod?

Personalexperiencevs

Scientificevidence

Page 8: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Zachariae

”Manpreferstobelievewhathepreferstobetrue”

FrancisBacon,1561-1626

Page 9: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Automaticvsmanualprocessing“Fast”vs“slow”thinking

Zachariae

Manualmode

Dorsolateral prefrontal cortexVentromedial prefrontal cortex

Automaticmode

Shivetal.2005;Kahneman,2011

System2SlowEffortfulLogicalInfrequentCalculatingConscious

System1FastAutomaticEmotionalFrequentStereotypicSubconscious

Evolutionhasequippedhumans withtwo types

ofthinking

Page 10: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Zachariae

ProblemsinEveryday- (fast)Thinking

• Thepracticalvalueofeveryday(”fast”)thinkingisobtainedatthecostofbiasandprecision,e.g.,:– Wetendtoseepatterns,evenwheretherearenone– Weseecausalrelationships,evenwheretherearenone– Wetendtofocusonandrememberpositiveevidence– Wetendtooverestimateevidenceconfirmingourposition– Ourjudgmentsareinfluencedbythejudgmentsofour

surroundings(conformity)– Wetendtobelievethatpositiveandnegativetraits,

respectively,areassociated(clustering)– Wetendtooverestimatetheprobabilityofdramaticevents

Page 11: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Problemsinfastthinking• Thepracticalvalueofeveryday(”fast”)thinkingis

obtainedatthecostofbiasandprecision,e.g.,– Wehaveatendencytoseepatterns,alsowhenthereareno

patterns:• E.g.incompletelyrandomsequences:122212221221112112211.• E.g.Whenrollingdice:Ifwehavenotobtaineda”six”inmanyrolls,wetendtobelievethattheprobabilityincreases(althoughthechanceremains1/6evenafter100rolls)

– Wetendtoperceivecausalrelationships,alsowhentherearenone:• E.g.,wetendtoimaginecausalitybetweenXandY,ifYtakesplaceafterX(anecessarybutnotsufficientconditionforcausality)

Zachariae

Page 12: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Theechochamberproblem

• Closedideologyechochamber• Appliestopoliticsaswellasotherdomains

Zachariae

QualitativeMethods

QuantitativeMethods

Researchcommunity

ClinicalPractice

Thesolutionisano-brainer– butdifficulttopractice

Page 13: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Zachariae

”Welovetopredictthings– andwearen’tverygoodatit”

Nate Silver(1978-)

Sept.30

Page 14: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Thedomainsofscience• Meta-physics:Philosophy,epistemology:deductionand

reasoning

• Theoriesofscience:Meta-theoriesaboutmethod

• Theoreticalscience:Collecting,condensing,discussing,andinterpretingexistingtheoreticalandempiricalresearchresults

• Empiricalscience:Measuringphenomenaandtestinghypotheses

• Observinganddescribing

• Predicting

• Determiningcauses

• Explaining

Zachariae

Page 15: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Theaimofscientificmethod

• Generalaim:Togeneratemeasurableandtestabledata,graduallyaddingtotheaccumulationofhumanknowledge– Toproducereliable knowledge

– Toproducevalid knowledge

– Aboutcausal relationships

– Byaddressingpossiblesourcesoferror

Zachariae

Page 16: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Thefalsificationprinciple

Zachariae

KarlPopper:AScientificHypothesis MustBe"Falsifiable".• Wesupportahypothesisbyfalsifyingthenull-hypothesis• Ageneralapproach:Wedonot“prove”hypotheses– butmaximizeour

attemptstofalsifystatementsaboutobservations,associations,causality,andmechanisms

Page 17: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Typesofevidence

Zachariae

Page 18: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Evidencehierarchy

Zachariae18

Anecdote

Expertopinion

Casereports

Crosssectionalstudy

Case-controlstudy

Cohortstudy

Experimentaldesign(RCT)

Primaryresearch

Clinicalexperience

Challenge:themodelfavors internal validity

Page 19: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Typesofvalidity• Validity

– Internalvalidity(causality,excludingalternativeexplanations,sourcesoferror)

– Externalvalidity(generalizability)

– Ecologicalvalidity(pragmaticvalidity)

• Thethreetypesofvaliditysupplementeachother• Aredifficulttoobtainwithonesinglemethod• Internalvalidityaprerequisiteforexternaland

ecologicalvalidity• Reliabilityaprerequisiteforvalidity– butnotthe

reverse

Zachariae

Page 20: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Mistakingreliabilityforvalidity

Zachariae20

Page 21: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Researchquestions• Internalvalidity– Doesitwork(statisticalsignificance,superiority)?– Howwelldoesitwork(practicalsignificance)?– Doesitworkaswellassomethingelse(non-inferiority)?– Howdoesitwork(mechanisms,specific,non-specific)?

• Externalvalidity– Forwhomdoesitwork?– Forwhomdoesitnotwork?

• Ecologicalvalidity– Doesitworkintheclinicalpracticalcontext?– Clinicianandpatientadherence

Zachariae

Page 22: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Exampleofdesignmaximizinginternalvalidity:Randomizedcontrolledtrial

Page 23: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

MBCTforpersistentpaininwomentreatedforbreastcancer

• 16-20%ofwomentreatedforbreastcancerexperiencepainafter5-9years

• Limitedpharmacologicaltreatmentefficacy

• Painisamultidimensionalphenomenonconsistingofsensory,cognitive,andaffectivefactors

• Mindfulness-basedtherapyteacheswaysofrelatingtobodilysensationsandemotionaldiscomfortwithhigherdegreeofacceptanceandopenness

• Mindfulness-BasedCognitiveTherapymaybeeffectiveforcancer-relatedpain

Zachariae

Page 24: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

MBCTforpersistentpaininwomentreatedforbreastcancer

Zachariae

Page 25: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

MBCTforpersistentpaininwomentreatedforbreastcancer

• Mediators oftheeffect ofMBCTonpain intensity:– Mindfulnessnon-reactivity facet;Pain catastrophizing

Zachariae

Johannsen,O’Toole,O’Connor,Jensen&Zachariae(underreview)

Page 26: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

MBCTforpersistentpaininwomentreatedforbreastcancer

Zachariae

Cost-effectiveness ofMBCTforpersistent pain inwomen treated forbreast cancer

Johannsen,Sørensen,O’Connor,O’Toole,Zachariae(inpreparation)

Page 27: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

ExampleofdesignPragmatictrialmaximizingbothinternal,external,andecological

validity

Zachariae

Page 28: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Evaluatingcomplementaryandalternativetreatments

• LabexperimentsandRCTsofenergyhealingyieldnegativeresults

• Complementaryandalternativemedicinesandtreatments(CAMs)areassociatedwithincreasedsymptomsofdepressioninbreastcancerpatients

• ActiveacupunctureisnotmoreeffectivethanplaceboacupunctureinadoubleblindedRCT

Zachariae

Zachariaeetal.2005

Pedersenetal.2013

Vaseetal.2013

Page 29: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Effectivenessofenergyhealingonqualityoflifeincolorectalcancerpatients

• CAMpractitionerscriticizevalidityoftraditionalresearchmethodologies,e.g.,RCTs

• Criticismsinclude:– Maynotbegeneralizedtothegeneralpopulation– Peoplemayhavestrongtreatmentpreferences– Standardizedoutcomemeasuresmaynotcoverpatients’individualconcerns

– Patientsmayprefersomepractitionerstoothers– Standardizationoftreatmentcontextmaycancelouteffects

Zachariae

Page 30: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Effectivenessofenergyhealingonqualityoflifeincolorectalcancerpatients

• Studydesignedtomaximizeinternal,external,andecologicalvalidity

• Colorectalcancerpatientsrandomizedto:– ARandomization

• Healingorcontrol

– BSelf-selection• Healingorcontrol

• Patients– Selectedtheirhealerfromalist– Treatmenttookplaceinhealer’sclinic– CompletedstandardizedQoLmeasures– Prioritizedpreferredoutcome– CompletedmeasureofattitudetowardsCAM

Zachariae

Page 31: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Effectivenessofenergyhealingonqualityoflifeincolorectalcancerpatients

• Nooveralleffectsonanyoutcomes• SmalleffectonQoLinsubgroup:Patientsinself-selectedhealinggroupwho

hadratedQoLasimportant,andwhohadapositiveattitudetowardsCAM

Zachariae

Page 32: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

ExampleofdesignPragmaticevaluationofdailypractice

Zachariae

Page 33: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Pragmaticevaluation• Fagligt Selskab forPsykologer i Palliationog Onkologi• 11psychologiststreating92patientsorcaregivers• Psychologistquestionnaires:psychotherapeuticmodelsandtoolsused

• Patientquestionnairespre- andpostconsulation:– MYCaW (MeasureYourselfConcernsandWell-being):Primaryandsecondaryconcernandgeneralwell-being

– WorkingAllianceInventory• Nof1statistics:ReliableChangeIndex(RCI)

– Determineswhetherachangeisbeyondastatisticalerror

Zachariae

Pedersenetal.(unpublished)

Page 34: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

PragmaticevaluationSelvvalgtproblemområdeogalmentvelbefindende

Sign.(RCI)Forbedring Ingenændring

Sign.(RCI)Forværring Datamangler

N(%) N(%) N(%) N(%)Primærtproblemområde

22(23,9) 64(69,6) 0(0,0) 6(6,5)

Sekundærtproblemområde

20(21,7) 43(46,7) 0(0,0) 29(31,5)

Almentvelbefindende

28(30,4) 60(65,2) 1(1,1) 3(3,3)

Zachariae

Statisticallysignificantpredictorsofsign.improvement:• Positiveexpectancies:Thesessionwillimprovemy

understandingofmyreactionsandemotions• Perceivedworkingalliance• Highereducationallevel

Pedersenetal.(unpublished)

Page 35: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Interpretingresults

Page 36: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Whatisaneffectsize?

• Study1:MeanscoreandstandarddeviationofHamiltonDepressionRatingScale(range:0-49):

• Intervention:16.5(13.0)• Control:20.5(14.0)• Cohen’sd=

• Study2:MeanscoreandstandarddeviationofBeck’sDepressionInventory(range:0-63)

• Intervention:17.5(7.0)• Control:21.5(8.0)• Cohen’sd=

Zachariae

Astandardizedeffect,e.g.,standardizedmeandifference,enablingcomparisonsacrossmeasuresandstudies

Cohen’s d=(Mean1– Mean2)/SD(pooled)

Whichinterventionismosteffective?

0.530.29

*)Todetectthedifferenceind betweenstudy1and2requiresasampleof610ineachgroup

BDIMCID(17%)2 =3.7=SD:0.49HDRSMCID(0,5SD)1

1)NICE,2004; 2)Buttonetal.2015

Page 37: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Significanceandprecision

Zachariae

Botheffectsizesarestatisticallysignificantlydifferentfrom”0”,arenotdifferentfromeachother

Onlytheeffectsizeofstudy1issignificantlydifferentfrom”0”.Thetwoeffectsizesarenotdifferentfromeachother

Botheffectsizesarestatisticallysignificantlydifferentfrom”0”,andtwoeffectsizesaresign.differentfromeachother(p=0.003)

Significance:p-values<0.05;Precision:95%Confidenceinterval

MCID

Page 38: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Challenges:Non-replicationpublicationbias“cherrypicking”

Onestudyisnotenough!

Page 39: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Interpretingnon-replicatedresults

Zachariae

Page 40: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

”Replicationcrisis”

Zachariae

Collaboration,OpenScience(2015-08-28)."Estimatingthereproducibilityofpsychologicalscience".Science.349

Page 41: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Zachariae

IstreatmentXbetterthancontrol?(Note:Asmallervalueisbetter)

Page 42: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Evidencehierarchy

Zachariae42

Anecdote

Expertopinion

Casereports

Crosssectionalstudy

Case-controlstudy

Cohortstudy

Experimentaldesign(RCT)

Narrativesystematicreview

Quantitativesystematicreview

Primaryresearch

Clinicalexperience

Secondaryresearch

Challenge:themodelfavors internal validity

Page 43: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Systematicreviewsandmeta-analysis

Hyp

Method

Result

Hyp

Method

Result

Hyp

Method

Result

Hyp

Method

Result

Singlestudies

Hyp.

MethodResult

Systematicreviewandmeta-analysis

Zachariae

Page 44: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Zachariae

Resultsofmeta-analysis:

Page 45: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

• Cantest(falsify)hypotheses• Testreproducibility(wereinitialresultsrandomorreliable?)

• Controlforrandomerror(variation)betweenstudies

• Testsystematicvariationbetweenstudies• Generalizeresults

Systematicreviewsandmeta-analysis

Zachariae

Page 46: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Numberofpublishedmeta-analysesperyear-PubMED1990-2013

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

1990 1992 1994 1996 1998 2000 2002 2004 2006 2008 2010 2012

Zachariae

Page 47: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Zachariae

Historyofmeta-analysis• Pearson(1904)averagedcorrelationsbetweenmortalityand

inoculationfortyphoidfever.• Firstmedical“meta-analysis”onplaceboeffects(Beecher,1955)• Eysenck (1952)arguedthatpsychotherapywasineffective• Glassstandardizedandaveragedtreatment-controldifferencesfrom

375studies,namingit”meta-analysis”(Smith&Glass,1977)• ”Anexerciseinmega-silliness”(Eysenck,1978)• SimilarmethodsdevelopedbyRosenthalandRubin(1978)• Today(1977-2014):PubMed:60460”hits”- PsychINFO:13833”hits”• CochraneCollaboration(1993):Medicine• CampbellCollaboration(1999):Socialsciences• Handbookofresearchsynthesis(Cooper&Hedges,1994)

Page 48: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Zachariae

Narrativevs Systematic review:Amatteroflife or death

• From1972-81,7studiesinvestigatedtheeffectofsteroid-injektionsonprematuredelivery(associatedwithincreasedinfantmortality)

• Twostudiesshowedaweakpositiveeffect– theremainingstudieswerenon-significant

• Thetreatmentwasabandoned• Alater1989meta-analysisoftheoriginaldata• revealedasignificantpositiveeffectoninfantmortality(OR:0.50)• TheCochraneCollaborationlogoshowsdatafromthe1989meta-

analysiswww.cochrane.org

Page 49: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Riskofbias

Zachariae

Page 50: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Studyquality• Validity:”Theapproximatetruthofaninferenceorclaimabouta

relationship”

• Internalvalidity– Threats:allalternativemechanismsthatcouldexplainresults,e.g.,

”placebo”,group-differencesatbaseline,unevendropout

• Externalvalidity– Areresultsgeneralizabletootherintendedparticipantsandcontexts?

• Constructvalidity– Dotheoperationalcharacteristicsofinterventionandmeasures

adequatelyrepresentintendedabstractcategories?

• Statisticalconclusionvalidity– Thevalidityofthestatisticalinferencesregardingthestrengthofthe

relationship.Threatsincludeinsufficientstatisticalpower,regressiontowardsthemean,incorrectassumptionsabouttheunderlyingvariance

Zachariae

Shadish,Cook&Campbell,2002

Page 51: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Qualityassessment

• Allstudiesshouldbesubjectedapre-definedqualityassessment

• Alreadydevelopedormodifiedexistingchecklist

• Anewlydevelopedchecklist

E.g.,Jadad checklist(Jadad,1996)

Zachariae

Page 52: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Qualityassessment

Possiblescore:0-15Meanscore=11.3(SD=2.4;range:8-15)Inter-rateragreement:89.9%of225individualqualityratings

Masking conditions Poweranalysis Manipulationcheck

Zachariae

Zachariae&O’Toole,2015

Page 53: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Zachariae

ESofpublishedvsunpublished

Lipsey &Wilson,1993

Page 54: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Zachariae

Examplesofpublicationbias

• MedicaljournalsfromChinaalmostneverpublishnegativeresults(e.g.Panetal.2005)

• Only5%ofarticlesinjournalsfocusingonAlternativeandComplementaryMedicinepresentnegativeresults(Schmidtetal.2001)

• StudiesoriginatingfromEuropehavemorepositiveresultsthanstudiesfromtheUS(Soodetal.2007)

Page 55: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Publication bias

Zachariae

• CONCLUSION:• Theefficacyof

psychologicalinterventionsfordepressionhasbeenoverestimatedinthepublishedliterature

• Justasithasbeenforpharmacotherapy.

• Bothareefficaciousbutnottotheextentthatthepublishedliteraturewouldsuggest.

Page 56: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Preregistering

• ClinicalTrials.gov• Aims:– Increasetransparency

– Reduce fishingexpedition bias

– Presentationofpost-hochypotheses asapriory

– Enableassessment ofpublication bias

56Zachariae

Page 57: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Zachariae

Publication biasassessmentResearcher bias against submitting negative resultsPublisher bias against publishing negative results

Cooper,H.,DeNeve,K.&Charlton,K.(1997).

Begg,C.B.&Berlin,J.A.(1988).

Page 58: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Preregistering

• PROSPERO– CentreforReviewsandDissemination

58Zachariae

Page 59: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Examples

Page 60: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Internet-deliveredCBTforinsomnia• Annualprevalenceofinsomnia:10-20%• 6%withachronictrajectory• Pharmacologicaltreatmentisnon-curativeandlong-termuseis

associatedwithdependence,tolerance,side-effects,andincreasedmortality

• CognitiveBehavioralTherapyforInsomnia(CBT-I)isrecommendedasfirstchoice– basedonevidencefromsystematicreviewsandmeta-analyses

• LimitedavailabilityofCBT-I(trainedtherapists,geography,financialreasons)

• OnepossibilityisInternet-deliveredCBT-I(eCBT-I)• IseCBT-Ieffectiveandareeffectscomparabletoface-to-face

deliveredCBT-I?

Zachariae

Vidensråd forforebyggelse(2015)Søvnogsundhed;AmericanAcademyofSleep Medicine

Page 61: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Internet-deliveredCBT-I

Zachariae

Statistically significant effects found forprimary outcomes:Insomnia severity:Hedges’sg=1.09,p<0.001Sleep efficiency:Hedges’sg=0.58,p<0.001

Page 62: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Internet-deliveredCBT-I

Zachariae

Comparing withface-to-facedelivery

Nostatisticallysignificantdifferencesbetweeninternet-deliveredandface-to-face-deliveredCBT-I.Needfornon-inferioritytrialsdirectlycomparingeCBT-IandFtF

Page 63: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Psychologicalinterventionfordistressininformalcancercaregivers

• Informalcancercaregivers(ICCs)reportincreasedlevelsofpsychologicalandphysicalmorbidityandhighermortality

• PsychologicalinterventionssuchasCognitiveBehavioralTherapies(CBTs)havebeenshownefficaciousfordistress(anxietyanddepression)

• Aim:toevaluatetheavailableevidencefortheefficacyofCBTsfordistressandphysicalsymptomsamongICCs

Zachariae

Page 64: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Psychologicalinterventionfordistressininformalcancercaregivers

• Results– 36uniquetrials– 27RCTs,9– TotalN=4746ICCs– Smallsign.effectforalltrialscombined(g=0.08,p=0.014)

– NoeffectinRCT´s(g=0.04,p=0.200)

Zachariae

O’Toole,Zachariae,Penna,Mennin,Applebaum,2016

Page 65: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Criticismsofmeta-analysis

Page 66: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Criticisms ofmeta-analysis

• Mixesapplesandoranges

• Garbagein– garbageout

• File-drawerproblem

• Reductionism(onenumber)

Zachariae

d =2,7(95%CI:1,7– 3,7)

”Exerciseinmega-silliness”(H.Eysenck,1978)”Statisticalalchemy”(Feinstein,1995)

Page 67: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Zachariae

SystematicreviewsTransparency Riskofbias Effectestimation

Non-systematicreview

Low High None

Narrativesystematicreview

Medium-high Medium Qualitative“Votecounting”

Quantitativesystematicreview

(meta-analysis)

High Low MagnitudeDirectionPrecisionSub-groupcomparisons

Page 68: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Zachariae

Lessons learned

Page 69: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Zachariae

”Onemustseekthetruthwhereitis,notwhereonewouldlikeittobe”

Abbé deFaria,1746-1819

Page 70: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Lessonslearned• Whenevaluatingevidence– Onestudyisnotenough(replicability)– Avoidcherrypicking– focusonthecombinedevidenceofallavailableevidence

– Lessemphasisonp-valuesofindividualstudies– moreemphasisonmagnitude(effectsize)

– Moreemphasisonpracticalsignificance(e.g.,MCID)• Ahighlystatisticallysignificanteffectcouldbeofairrelevantmagnitude

• Anon-statisticallysignificanteffectcouldpotentiallybeclinicallyrelevant

Zachariae

Page 71: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Lessonslearned• Consider– StatisticalpowerandriskofType-2error– Theprecisionoftheeffect(theconfidenceinterval)– Homogeneityoftheexistingevidence– Studyquality– potentialbiasandthreatstovalidity– Publicationbias– thetendencytounderreportnull-findings

– Cost-effectiveness– relativetotreatmentasusual• Establishingevidence– Isacomplexcumulativeprocess

Zachariae

Page 72: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Futuretasks• Thefuturewillbringpublicandpolicy-baseddemandsforevaluationofclinicalefficacyandcost-effectiveness

• Psychologistsareadvisedto:– Worktoestablishaproactive,evidence-basedprofessional

culture– Nottotaketheeffectivenessofpsychologicalapproachesas

self-evidentbuttofocusonthebestavailableevidence– Acceptwhenpsychologicalapproachesarenoteffective,be

transparentaboutit,andworktoimprovethesituation– Promoteresearch-basedpracticeandpractice-relevant

research,establishcollaborationbetweenresearchersandclinicians,andconductresearch-basedevaluationincollaborativenetworks

Zachariae

Page 73: Forskningsdesigns og evidens - d p · The echo chamber problem • Closed ideology echo chamber • Applies to politics as well as other domains Zachariae Qualitative Methods Quantitative

Zachariae

Examples