Forskningsdesigns og evidens Bobby Zachariae Professor, dr.med., cand.psych. Aarhus Universitetshospital og Aarhus Universitet
Forskningsdesignsogevidens
BobbyZachariaeProfessor,dr.med.,cand.psych.AarhusUniversitetshospitalog
AarhusUniversitet
Aimsofthepresentation
• Evaluatingtheavailableevidence– challengesandavailablesolutions– Problemsinevery-daythinking– Aimsofscientificmethod– Typesofevidence– Threatstovalidity–Whenonestudyisnotenough– Aspectstoconsiderwhenevaluating“effectiveness”
Zachariae
UnitforPsychooncologyandHealthPsychology - EPoS
• PsychooncologyResearchUnitestablishedin2000attheDept.OfOncology,AUHbasedonagrantfromtheDanishCancerSociety
• EPoS establishedin2011incollaborationbetweenAUH,Dept.OfOncology,BSS,AU,andDeptofPsychologyandBehavioural Science
• Currentstaff:17(1professor,2assoc.prof.1assistprof,1seniorresearcher,4post-docs,7PhD’s,1adm.)+8-10researchassistants.
Mybackground
• Researchareas– Psychoneuroimmunology– Painresearch– Psycho-socialcancerresearch– Patient-healthprofessionalinteractions– Healthpsychology– Psychosocialinterventions– Hypnosis,guidedimagery,mindfulness-basedintervention,expressivewriting
– Internet-deliveredinterventions– Efficacyofcomplementaryandalternativetreatments
4Zachariae
Mybackground• Researchmethodologies– Experimentalstudies(psychophysiology)– RandomizedControlledTrials(RCT)– Pragmatictrials–Mixed(qualitativeandquantitative)methods– Cross-sectionalandcohortstudies– Psychometrics– Systematicreviewsandmeta-analysis
5Zachariae
• Intellectualadvisetofuturegenerations:
• “Askyourselfonlywhatarethefactsandwhatisthetruththatthefactsbearout.Neverletyourselfbedivertedeitherbywhatyouwishtobelieve,orbywhatyouthinkwouldhavebeneficentsocialeffectsifitwerebelieved.Butlookonly,andsolely,atwhatarethefacts.”
Zachariae6
BertrandRussell,1872-1970
Whyscientificmethod?
Personalexperiencevs
Scientificevidence
Zachariae
”Manpreferstobelievewhathepreferstobetrue”
FrancisBacon,1561-1626
Automaticvsmanualprocessing“Fast”vs“slow”thinking
Zachariae
Manualmode
Dorsolateral prefrontal cortexVentromedial prefrontal cortex
Automaticmode
Shivetal.2005;Kahneman,2011
System2SlowEffortfulLogicalInfrequentCalculatingConscious
System1FastAutomaticEmotionalFrequentStereotypicSubconscious
Evolutionhasequippedhumans withtwo types
ofthinking
Zachariae
ProblemsinEveryday- (fast)Thinking
• Thepracticalvalueofeveryday(”fast”)thinkingisobtainedatthecostofbiasandprecision,e.g.,:– Wetendtoseepatterns,evenwheretherearenone– Weseecausalrelationships,evenwheretherearenone– Wetendtofocusonandrememberpositiveevidence– Wetendtooverestimateevidenceconfirmingourposition– Ourjudgmentsareinfluencedbythejudgmentsofour
surroundings(conformity)– Wetendtobelievethatpositiveandnegativetraits,
respectively,areassociated(clustering)– Wetendtooverestimatetheprobabilityofdramaticevents
Problemsinfastthinking• Thepracticalvalueofeveryday(”fast”)thinkingis
obtainedatthecostofbiasandprecision,e.g.,– Wehaveatendencytoseepatterns,alsowhenthereareno
patterns:• E.g.incompletelyrandomsequences:122212221221112112211.• E.g.Whenrollingdice:Ifwehavenotobtaineda”six”inmanyrolls,wetendtobelievethattheprobabilityincreases(althoughthechanceremains1/6evenafter100rolls)
– Wetendtoperceivecausalrelationships,alsowhentherearenone:• E.g.,wetendtoimaginecausalitybetweenXandY,ifYtakesplaceafterX(anecessarybutnotsufficientconditionforcausality)
Zachariae
Theechochamberproblem
• Closedideologyechochamber• Appliestopoliticsaswellasotherdomains
Zachariae
QualitativeMethods
QuantitativeMethods
Researchcommunity
ClinicalPractice
Thesolutionisano-brainer– butdifficulttopractice
Zachariae
”Welovetopredictthings– andwearen’tverygoodatit”
Nate Silver(1978-)
Sept.30
Thedomainsofscience• Meta-physics:Philosophy,epistemology:deductionand
reasoning
• Theoriesofscience:Meta-theoriesaboutmethod
• Theoreticalscience:Collecting,condensing,discussing,andinterpretingexistingtheoreticalandempiricalresearchresults
• Empiricalscience:Measuringphenomenaandtestinghypotheses
• Observinganddescribing
• Predicting
• Determiningcauses
• Explaining
Zachariae
Theaimofscientificmethod
• Generalaim:Togeneratemeasurableandtestabledata,graduallyaddingtotheaccumulationofhumanknowledge– Toproducereliable knowledge
– Toproducevalid knowledge
– Aboutcausal relationships
– Byaddressingpossiblesourcesoferror
Zachariae
Thefalsificationprinciple
Zachariae
KarlPopper:AScientificHypothesis MustBe"Falsifiable".• Wesupportahypothesisbyfalsifyingthenull-hypothesis• Ageneralapproach:Wedonot“prove”hypotheses– butmaximizeour
attemptstofalsifystatementsaboutobservations,associations,causality,andmechanisms
Typesofevidence
Zachariae
Evidencehierarchy
Zachariae18
Anecdote
Expertopinion
Casereports
Crosssectionalstudy
Case-controlstudy
Cohortstudy
Experimentaldesign(RCT)
Primaryresearch
Clinicalexperience
Challenge:themodelfavors internal validity
Typesofvalidity• Validity
– Internalvalidity(causality,excludingalternativeexplanations,sourcesoferror)
– Externalvalidity(generalizability)
– Ecologicalvalidity(pragmaticvalidity)
• Thethreetypesofvaliditysupplementeachother• Aredifficulttoobtainwithonesinglemethod• Internalvalidityaprerequisiteforexternaland
ecologicalvalidity• Reliabilityaprerequisiteforvalidity– butnotthe
reverse
Zachariae
Mistakingreliabilityforvalidity
Zachariae20
Researchquestions• Internalvalidity– Doesitwork(statisticalsignificance,superiority)?– Howwelldoesitwork(practicalsignificance)?– Doesitworkaswellassomethingelse(non-inferiority)?– Howdoesitwork(mechanisms,specific,non-specific)?
• Externalvalidity– Forwhomdoesitwork?– Forwhomdoesitnotwork?
• Ecologicalvalidity– Doesitworkintheclinicalpracticalcontext?– Clinicianandpatientadherence
Zachariae
Exampleofdesignmaximizinginternalvalidity:Randomizedcontrolledtrial
MBCTforpersistentpaininwomentreatedforbreastcancer
• 16-20%ofwomentreatedforbreastcancerexperiencepainafter5-9years
• Limitedpharmacologicaltreatmentefficacy
• Painisamultidimensionalphenomenonconsistingofsensory,cognitive,andaffectivefactors
• Mindfulness-basedtherapyteacheswaysofrelatingtobodilysensationsandemotionaldiscomfortwithhigherdegreeofacceptanceandopenness
• Mindfulness-BasedCognitiveTherapymaybeeffectiveforcancer-relatedpain
Zachariae
MBCTforpersistentpaininwomentreatedforbreastcancer
Zachariae
MBCTforpersistentpaininwomentreatedforbreastcancer
• Mediators oftheeffect ofMBCTonpain intensity:– Mindfulnessnon-reactivity facet;Pain catastrophizing
Zachariae
Johannsen,O’Toole,O’Connor,Jensen&Zachariae(underreview)
MBCTforpersistentpaininwomentreatedforbreastcancer
Zachariae
Cost-effectiveness ofMBCTforpersistent pain inwomen treated forbreast cancer
Johannsen,Sørensen,O’Connor,O’Toole,Zachariae(inpreparation)
ExampleofdesignPragmatictrialmaximizingbothinternal,external,andecological
validity
Zachariae
Evaluatingcomplementaryandalternativetreatments
• LabexperimentsandRCTsofenergyhealingyieldnegativeresults
• Complementaryandalternativemedicinesandtreatments(CAMs)areassociatedwithincreasedsymptomsofdepressioninbreastcancerpatients
• ActiveacupunctureisnotmoreeffectivethanplaceboacupunctureinadoubleblindedRCT
Zachariae
Zachariaeetal.2005
Pedersenetal.2013
Vaseetal.2013
Effectivenessofenergyhealingonqualityoflifeincolorectalcancerpatients
• CAMpractitionerscriticizevalidityoftraditionalresearchmethodologies,e.g.,RCTs
• Criticismsinclude:– Maynotbegeneralizedtothegeneralpopulation– Peoplemayhavestrongtreatmentpreferences– Standardizedoutcomemeasuresmaynotcoverpatients’individualconcerns
– Patientsmayprefersomepractitionerstoothers– Standardizationoftreatmentcontextmaycancelouteffects
Zachariae
Effectivenessofenergyhealingonqualityoflifeincolorectalcancerpatients
• Studydesignedtomaximizeinternal,external,andecologicalvalidity
• Colorectalcancerpatientsrandomizedto:– ARandomization
• Healingorcontrol
– BSelf-selection• Healingorcontrol
• Patients– Selectedtheirhealerfromalist– Treatmenttookplaceinhealer’sclinic– CompletedstandardizedQoLmeasures– Prioritizedpreferredoutcome– CompletedmeasureofattitudetowardsCAM
Zachariae
Effectivenessofenergyhealingonqualityoflifeincolorectalcancerpatients
• Nooveralleffectsonanyoutcomes• SmalleffectonQoLinsubgroup:Patientsinself-selectedhealinggroupwho
hadratedQoLasimportant,andwhohadapositiveattitudetowardsCAM
Zachariae
ExampleofdesignPragmaticevaluationofdailypractice
Zachariae
Pragmaticevaluation• Fagligt Selskab forPsykologer i Palliationog Onkologi• 11psychologiststreating92patientsorcaregivers• Psychologistquestionnaires:psychotherapeuticmodelsandtoolsused
• Patientquestionnairespre- andpostconsulation:– MYCaW (MeasureYourselfConcernsandWell-being):Primaryandsecondaryconcernandgeneralwell-being
– WorkingAllianceInventory• Nof1statistics:ReliableChangeIndex(RCI)
– Determineswhetherachangeisbeyondastatisticalerror
Zachariae
Pedersenetal.(unpublished)
PragmaticevaluationSelvvalgtproblemområdeogalmentvelbefindende
Sign.(RCI)Forbedring Ingenændring
Sign.(RCI)Forværring Datamangler
N(%) N(%) N(%) N(%)Primærtproblemområde
22(23,9) 64(69,6) 0(0,0) 6(6,5)
Sekundærtproblemområde
20(21,7) 43(46,7) 0(0,0) 29(31,5)
Almentvelbefindende
28(30,4) 60(65,2) 1(1,1) 3(3,3)
Zachariae
Statisticallysignificantpredictorsofsign.improvement:• Positiveexpectancies:Thesessionwillimprovemy
understandingofmyreactionsandemotions• Perceivedworkingalliance• Highereducationallevel
Pedersenetal.(unpublished)
Interpretingresults
Whatisaneffectsize?
• Study1:MeanscoreandstandarddeviationofHamiltonDepressionRatingScale(range:0-49):
• Intervention:16.5(13.0)• Control:20.5(14.0)• Cohen’sd=
• Study2:MeanscoreandstandarddeviationofBeck’sDepressionInventory(range:0-63)
• Intervention:17.5(7.0)• Control:21.5(8.0)• Cohen’sd=
Zachariae
Astandardizedeffect,e.g.,standardizedmeandifference,enablingcomparisonsacrossmeasuresandstudies
Cohen’s d=(Mean1– Mean2)/SD(pooled)
Whichinterventionismosteffective?
0.530.29
*)Todetectthedifferenceind betweenstudy1and2requiresasampleof610ineachgroup
BDIMCID(17%)2 =3.7=SD:0.49HDRSMCID(0,5SD)1
1)NICE,2004; 2)Buttonetal.2015
Significanceandprecision
Zachariae
Botheffectsizesarestatisticallysignificantlydifferentfrom”0”,arenotdifferentfromeachother
Onlytheeffectsizeofstudy1issignificantlydifferentfrom”0”.Thetwoeffectsizesarenotdifferentfromeachother
Botheffectsizesarestatisticallysignificantlydifferentfrom”0”,andtwoeffectsizesaresign.differentfromeachother(p=0.003)
Significance:p-values<0.05;Precision:95%Confidenceinterval
MCID
Challenges:Non-replicationpublicationbias“cherrypicking”
Onestudyisnotenough!
Interpretingnon-replicatedresults
Zachariae
”Replicationcrisis”
Zachariae
Collaboration,OpenScience(2015-08-28)."Estimatingthereproducibilityofpsychologicalscience".Science.349
Zachariae
IstreatmentXbetterthancontrol?(Note:Asmallervalueisbetter)
Evidencehierarchy
Zachariae42
Anecdote
Expertopinion
Casereports
Crosssectionalstudy
Case-controlstudy
Cohortstudy
Experimentaldesign(RCT)
Narrativesystematicreview
Quantitativesystematicreview
Primaryresearch
Clinicalexperience
Secondaryresearch
Challenge:themodelfavors internal validity
Systematicreviewsandmeta-analysis
Hyp
Method
Result
Hyp
Method
Result
Hyp
Method
Result
Hyp
Method
Result
Singlestudies
Hyp.
MethodResult
Systematicreviewandmeta-analysis
Zachariae
Zachariae
Resultsofmeta-analysis:
• Cantest(falsify)hypotheses• Testreproducibility(wereinitialresultsrandomorreliable?)
• Controlforrandomerror(variation)betweenstudies
• Testsystematicvariationbetweenstudies• Generalizeresults
Systematicreviewsandmeta-analysis
Zachariae
Numberofpublishedmeta-analysesperyear-PubMED1990-2013
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
1990 1992 1994 1996 1998 2000 2002 2004 2006 2008 2010 2012
Zachariae
Zachariae
Historyofmeta-analysis• Pearson(1904)averagedcorrelationsbetweenmortalityand
inoculationfortyphoidfever.• Firstmedical“meta-analysis”onplaceboeffects(Beecher,1955)• Eysenck (1952)arguedthatpsychotherapywasineffective• Glassstandardizedandaveragedtreatment-controldifferencesfrom
375studies,namingit”meta-analysis”(Smith&Glass,1977)• ”Anexerciseinmega-silliness”(Eysenck,1978)• SimilarmethodsdevelopedbyRosenthalandRubin(1978)• Today(1977-2014):PubMed:60460”hits”- PsychINFO:13833”hits”• CochraneCollaboration(1993):Medicine• CampbellCollaboration(1999):Socialsciences• Handbookofresearchsynthesis(Cooper&Hedges,1994)
Zachariae
Narrativevs Systematic review:Amatteroflife or death
• From1972-81,7studiesinvestigatedtheeffectofsteroid-injektionsonprematuredelivery(associatedwithincreasedinfantmortality)
• Twostudiesshowedaweakpositiveeffect– theremainingstudieswerenon-significant
• Thetreatmentwasabandoned• Alater1989meta-analysisoftheoriginaldata• revealedasignificantpositiveeffectoninfantmortality(OR:0.50)• TheCochraneCollaborationlogoshowsdatafromthe1989meta-
analysiswww.cochrane.org
Riskofbias
Zachariae
Studyquality• Validity:”Theapproximatetruthofaninferenceorclaimabouta
relationship”
• Internalvalidity– Threats:allalternativemechanismsthatcouldexplainresults,e.g.,
”placebo”,group-differencesatbaseline,unevendropout
• Externalvalidity– Areresultsgeneralizabletootherintendedparticipantsandcontexts?
• Constructvalidity– Dotheoperationalcharacteristicsofinterventionandmeasures
adequatelyrepresentintendedabstractcategories?
• Statisticalconclusionvalidity– Thevalidityofthestatisticalinferencesregardingthestrengthofthe
relationship.Threatsincludeinsufficientstatisticalpower,regressiontowardsthemean,incorrectassumptionsabouttheunderlyingvariance
Zachariae
Shadish,Cook&Campbell,2002
Qualityassessment
• Allstudiesshouldbesubjectedapre-definedqualityassessment
• Alreadydevelopedormodifiedexistingchecklist
• Anewlydevelopedchecklist
E.g.,Jadad checklist(Jadad,1996)
Zachariae
Qualityassessment
Possiblescore:0-15Meanscore=11.3(SD=2.4;range:8-15)Inter-rateragreement:89.9%of225individualqualityratings
Masking conditions Poweranalysis Manipulationcheck
Zachariae
Zachariae&O’Toole,2015
Zachariae
ESofpublishedvsunpublished
Lipsey &Wilson,1993
Zachariae
Examplesofpublicationbias
• MedicaljournalsfromChinaalmostneverpublishnegativeresults(e.g.Panetal.2005)
• Only5%ofarticlesinjournalsfocusingonAlternativeandComplementaryMedicinepresentnegativeresults(Schmidtetal.2001)
• StudiesoriginatingfromEuropehavemorepositiveresultsthanstudiesfromtheUS(Soodetal.2007)
Publication bias
Zachariae
• CONCLUSION:• Theefficacyof
psychologicalinterventionsfordepressionhasbeenoverestimatedinthepublishedliterature
• Justasithasbeenforpharmacotherapy.
• Bothareefficaciousbutnottotheextentthatthepublishedliteraturewouldsuggest.
Preregistering
• ClinicalTrials.gov• Aims:– Increasetransparency
– Reduce fishingexpedition bias
– Presentationofpost-hochypotheses asapriory
– Enableassessment ofpublication bias
56Zachariae
Zachariae
Publication biasassessmentResearcher bias against submitting negative resultsPublisher bias against publishing negative results
Cooper,H.,DeNeve,K.&Charlton,K.(1997).
Begg,C.B.&Berlin,J.A.(1988).
Preregistering
• PROSPERO– CentreforReviewsandDissemination
58Zachariae
Examples
Internet-deliveredCBTforinsomnia• Annualprevalenceofinsomnia:10-20%• 6%withachronictrajectory• Pharmacologicaltreatmentisnon-curativeandlong-termuseis
associatedwithdependence,tolerance,side-effects,andincreasedmortality
• CognitiveBehavioralTherapyforInsomnia(CBT-I)isrecommendedasfirstchoice– basedonevidencefromsystematicreviewsandmeta-analyses
• LimitedavailabilityofCBT-I(trainedtherapists,geography,financialreasons)
• OnepossibilityisInternet-deliveredCBT-I(eCBT-I)• IseCBT-Ieffectiveandareeffectscomparabletoface-to-face
deliveredCBT-I?
Zachariae
Vidensråd forforebyggelse(2015)Søvnogsundhed;AmericanAcademyofSleep Medicine
Internet-deliveredCBT-I
Zachariae
Statistically significant effects found forprimary outcomes:Insomnia severity:Hedges’sg=1.09,p<0.001Sleep efficiency:Hedges’sg=0.58,p<0.001
Internet-deliveredCBT-I
Zachariae
Comparing withface-to-facedelivery
Nostatisticallysignificantdifferencesbetweeninternet-deliveredandface-to-face-deliveredCBT-I.Needfornon-inferioritytrialsdirectlycomparingeCBT-IandFtF
Psychologicalinterventionfordistressininformalcancercaregivers
• Informalcancercaregivers(ICCs)reportincreasedlevelsofpsychologicalandphysicalmorbidityandhighermortality
• PsychologicalinterventionssuchasCognitiveBehavioralTherapies(CBTs)havebeenshownefficaciousfordistress(anxietyanddepression)
• Aim:toevaluatetheavailableevidencefortheefficacyofCBTsfordistressandphysicalsymptomsamongICCs
Zachariae
Psychologicalinterventionfordistressininformalcancercaregivers
• Results– 36uniquetrials– 27RCTs,9– TotalN=4746ICCs– Smallsign.effectforalltrialscombined(g=0.08,p=0.014)
– NoeffectinRCT´s(g=0.04,p=0.200)
Zachariae
O’Toole,Zachariae,Penna,Mennin,Applebaum,2016
Criticismsofmeta-analysis
Criticisms ofmeta-analysis
• Mixesapplesandoranges
• Garbagein– garbageout
• File-drawerproblem
• Reductionism(onenumber)
Zachariae
d =2,7(95%CI:1,7– 3,7)
”Exerciseinmega-silliness”(H.Eysenck,1978)”Statisticalalchemy”(Feinstein,1995)
Zachariae
SystematicreviewsTransparency Riskofbias Effectestimation
Non-systematicreview
Low High None
Narrativesystematicreview
Medium-high Medium Qualitative“Votecounting”
Quantitativesystematicreview
(meta-analysis)
High Low MagnitudeDirectionPrecisionSub-groupcomparisons
Zachariae
Lessons learned
Zachariae
”Onemustseekthetruthwhereitis,notwhereonewouldlikeittobe”
Abbé deFaria,1746-1819
Lessonslearned• Whenevaluatingevidence– Onestudyisnotenough(replicability)– Avoidcherrypicking– focusonthecombinedevidenceofallavailableevidence
– Lessemphasisonp-valuesofindividualstudies– moreemphasisonmagnitude(effectsize)
– Moreemphasisonpracticalsignificance(e.g.,MCID)• Ahighlystatisticallysignificanteffectcouldbeofairrelevantmagnitude
• Anon-statisticallysignificanteffectcouldpotentiallybeclinicallyrelevant
Zachariae
Lessonslearned• Consider– StatisticalpowerandriskofType-2error– Theprecisionoftheeffect(theconfidenceinterval)– Homogeneityoftheexistingevidence– Studyquality– potentialbiasandthreatstovalidity– Publicationbias– thetendencytounderreportnull-findings
– Cost-effectiveness– relativetotreatmentasusual• Establishingevidence– Isacomplexcumulativeprocess
Zachariae
Futuretasks• Thefuturewillbringpublicandpolicy-baseddemandsforevaluationofclinicalefficacyandcost-effectiveness
• Psychologistsareadvisedto:– Worktoestablishaproactive,evidence-basedprofessional
culture– Nottotaketheeffectivenessofpsychologicalapproachesas
self-evidentbuttofocusonthebestavailableevidence– Acceptwhenpsychologicalapproachesarenoteffective,be
transparentaboutit,andworktoimprovethesituation– Promoteresearch-basedpracticeandpractice-relevant
research,establishcollaborationbetweenresearchersandclinicians,andconductresearch-basedevaluationincollaborativenetworks
Zachariae
Zachariae
Examples