RoB 2.0: A revised tool to assess risk of bias in randomized trials Matthew Page University of Bristol, UK With special thanks to Julian Higgins, Jelena Savović, Asbjørn Hróbjartsson, Isabelle Boutron, Barney Reeves, Roy Elbers, Jonathan Sterne
RoB 2.0:Arevisedtooltoassessriskofbiasinrandomizedtrials
MatthewPageUniversityofBristol,UK
WithspecialthankstoJulianHiggins,Jelena Savović,Asbjørn Hróbjartsson,IsabelleBoutron,BarneyReeves,RoyElbers,JonathanSterne
Overview
• ReminderoftheCochraneriskofbiastoolforrandomizedtrials• Theneedforanewtool• Developmentofthenewtool• Keyinnovationstothetool• Someexcerptsfromthetool• Someunresolvedissues
3
BMJ 2011; 343: d5928
4
Foam dressings for venous leg ulcers
CurrentCochranetoolforriskofbiasinrandomizedtrials
• Sixsourcesofbias(withoptional‘Other’)
• Foreachsource,• Freetexttodescribewhathappened• Judgement:Lowrisk/Unclearrisk/Highriskofbias
• Somesourcesofbiascanberepeatedfordifferentendpoints
CurrentCochranetoolforriskofbiasinrandomizedtrials
• CochraneRoB toolisverywidelyused(Jørgensen 2016)• 100outof100Cochranereviewsfrom2014(100%)• 31outof81non-Cochranereview(38%)
• >2700citationsfromnon-Cochranesources
• Thescientificdebateonriskofbiashascontinued
• Evaluationstudiesofthetool• Userexperience:surveyandfocusgroups(Savovic 2014)• Inter-agreementstudies(e.g.Hartling2009&2013)• Actualuseinreviewsandpublishedcomments(Jørgensen2016)
Someissuesraisedwithexistingtool
• Usedsimplistically
• Usedinconsistently (domainsaddedorremoved)
• Modestagreement rates
• Only5-10%oftrialsinCochranereviewsarescoredasLowriskofbias
• overuseof“unclearrisk”?
• RoB judgementsaredifficult forsomedomains,particularlyincompleteoutcomedataandselectivereporting
• Challengeswithunblindedtrials
• Notwellsuitedtocross-overtrialsorcluster-randomizedtrials
• Notwellsetuptoassessoverallriskofbias
Funding
• Therevisedtoolforrandomizedtrials(RoB2.0)wassupportedbytheUKMedicalResearchCouncil NetworkofHubsforTrialsMethodologyResearch(MR/L004933/1- N61)
RoB2.0:developmentchronology
• RevisionoftheRoBtoolstartedinMay2015• 1st DevelopmentmeetingheldinBristolinAugust2015• 1st ‘workingdraft’ofthetoolcompletedJanuary2016• PilotingphaseFeb– March2016• Revised‘workingdraft’• 2nd DevelopmentmeetingheldinBristolon21-22April2016• Developmentoffurtherguidanceandpiloting• ReleasedforSeoulColloquium
RoB2.0:contributors
• Coregroup:• JulianHiggins,Jelena Savović,MatthewPage,AsbjørnHróbjartsson,Isabelle
Boutron,BarneyReeves,RoyElbers,JonathanSterne• WorkingGroupmembers:
• DougAltman,NatalieBlencowe,MikeCampbell,ChristopherCates,RachelChurchill,MarkCorbett,NickyCullum,FrancoisCurtin,AmyDrahota,SandraEldridge,JonathanEmberson,BrunoGiraudeau,JeremyGrimshaw,ShareaIjaz,SallyHopewell,AsbjørnHróbjartsson,PeterJüni,JamieKirkham,TobyLasserson,TianjingLi,StephenSenn,SashaShepperd,IanShrier,NandiSiegfried,LesleyStewart,PennyWhiting
• And:HenningKeinke Andersen,MikeClarke,JonDeeks,GeraldineMacDonald,RichardMorris,MonaNasser,Nishith Patel,JaniRuotsalainen,HolgerSchünemann, JayneTierney
Keyinnovations
• Result-focussed assessments• Fixed(inclusive)biasdomains,notmodifiable• “Signallingquestions”tofacilitateriskofbiasjudgements• Newresponseoptionsforriskofbias,without‘Unclear’option• Formaloverall riskofbiasjudgement
• Somerethinkingoftheassessment:• Importantdistinctionbetweeneffectsofinterest• Selectivereportingfocusesonreportedresult
RoB1.0 RoB2.0
Randomsequencegeneration(selectionbias) Biasarisingfromtherandomization
processAllocationconcealment(selectionbias)
Blindingofparticipantsandpersonnel(performancebias)
Biasduetodeviationsfromintendedinterventions
Incompleteoutcomedata(attritionbias) Biasduetomissingoutcomedata
Blindingofoutcomeassessment(detectionbias) Biasinmeasurementoftheoutcome
Selectivereporting(reportingbias) Biasinselectionofthereportedresult
Otherbias N/A
N/A Overallbias
Fundingandvestedintereststobeaddressed,butnotwithinthispartofthewiderframeworkWorkinggroupledbyAsbjørn Hróbjartsson and
IsabelleBoutron
Signallingquestionsandjudgements
• Signallingquestionsareintroducedtomakethetooleasier(andmoretransparent)• ‘Yes’,‘Probablyyes’,‘Probablyno’,‘No’,‘Noinformation’
• Riskofbiasjudgementsfollowfromanswerstosignallingquestions(canbeover-ridden)• ‘Lowriskofbias’,‘Someconcerns’,‘Highriskofbias’
• Achangeintheinterpretationofthejudgements,sothata‘Highriskofbias’judgementinonedomainputsthewholestudyathighriskofbias
• Overallriskofbiasjudgementcanthenbecompletedautomatically(canbeover-ridden)
Overallriskofbiasjudgement
Lowriskofbias Thestudyisjudgedtobeatlowriskofbias foralldomainsforthisresult.
Someconcerns Thestudyisjudgedtobeatsomeconcernsinatleastonedomainforthisresult.
Highriskofbias Thestudyisjudgedtobeathighriskofbias inatleastonedomainforthisresult.ORThestudyisjudgedtohavesomeconcernsformultipledomains inawaythatsubstantiallylowersconfidenceintheresult.
riskofbias.info
Someexcerptsfromthetool
Examplealgorithm
4.1Wereoutcomeassessorsawareoftheinterventionreceivedby
studyparticipants?
4.2Wastheassessmentoftheoutcomelikelyto
beinfluencedbyknowledgeofintervention
received?
Highrisk
Someconcerns
Lowrisk
Lowrisk
Y/PY/NI
N/PN
Y/PY
N/PN
NI
Biasarisingfromtherandomizationprocess
Biasarisingfromtherandomizationprocess
• Currenttoolincludestwoseparatedomains:• sequencegeneration• allocationconcealment(bothunder“selectionbias”)
• Botharerelatedtorandomization/allocationofparticipatesintotreatmentarms
• Failuretoimplementeitherprocessadequatelycreatesopportunitiesforeithertheenrolmentintothestudyortheallocationofenrolledparticipantsintogroupstobeinfluencedbyprognosticfactors
• Theendresultisthesame– unbalanced(biased)distributionofpatientsbetweengroups(notafaircomparison,confounding)
Ø ItmakessensetocombineSGandACintoasingledomain
Biasarisingfromtherandomizationprocess
• EvaluationstudiesoftheuseoftheRoB toolinCochraneshowthatreviewersoftenconsiderbaselineimbalanceas“Otherbias”
• Butthisisrelatedtothesuccessofrandomization
Ø Itmakessensetoincludebaselineimbalanceinthesamebiasdomain
• Indicatorsthatrandomizationwasnotperformedadequately:• unusuallylargedifferencesbetweeninterventiongroupsizes;• asubstantialexcessinstatisticallysignificantdifferencesinbaselinecharacteristics;
• asubstantialexcessinclinicallyimportantdifferencesinbaselinecharacteristics
Biasarisingfromtherandomizationprocess
1.1Wastheallocationsequencerandom?1.2Wastheallocationsequenceconcealeduntil
participantswererecruitedandassignedtointerventions?
1.3Weretherebaselineimbalancesthatsuggestaproblemwiththerandomizationprocess?
Randomizationmethods
Additionalevidenceofproblems
Biasduetodeviationsfromintendedinterventions
Theeffectofinterest
• Thecurrenttoolhasverylittletosayaboutsituationsinwhichblindingisnotfeasible• (otherthantoclassifyasnotblindhencehighriskofbias)
• Issuesofperformancebiasverydifferentfordifferenteffectsofinterest,yetpoorlyaddressedincurrentRoB tool
Theeffectofinterest
• Thecurrenttoolhasverylittletosayaboutsituationsinwhichblindingisnotfeasible• (otherthantoclassifyasnotblindhencehighriskofbias)
• Issuesofperformancebiasverydifferentfordifferenteffectsofinterest,yetpoorlyaddressedincurrentRoB tool
• effectofassignmenttointervention• e.g.doesreferraltophysicaltherapyincreasepost-operativemobility?(thequestionofinteresttoahospitalmanageraboutwhethertointroduceareferralprogramme)
• effectofstartingandadheringtointervention• e.g.doesattendingaphysicaltherapyprogramincreasepost-operativemobility?(thequestionofinteresttoanindividualaboutwhethertoattendphysicaltherapy)
Theeffectofinterest
• Wheninterestedineffectofassignment tointervention• Deviationsfromintendedinterventionarenotimportantprovidingthesedeviationsreflectusualpractice
• e.g.itisusualpracticeforsomereferredpatientstonotattendphysicaltherapy,ortocompleteonlysomesessions
• thisdifferstobehaviourthatreflectsexpectationsofadifferencebetweeninterventionandcomparator
• Wheninterestedineffectstartingandadheringtointervention• Deviationssuchaspooradherence,poorimplementationandco-interventionsmayleadtoriskofbias
• Wethereforehavedifferenttoolsforthesetwoeffectsofinterest
Biasduetodeviationsfromintendedinterventions
Effectofassignment tointervention2.1.Wereparticipantsawareoftheirassignedinterventionduring
thetrial?2.2.Werecarersandtrialpersonnelawareofparticipants'assigned
interventionduringthetrial?2.3.IfY/PY/NIto2.1or2.2:Weretheredeviationsfromthe
intendedinterventionbeyondwhatwouldbeexpectedinusualpractice?
2.4.IfY/PYto2.3:Werethesedeviationsfromintendedinterventionunbalancedbetweengroupsand likelytohaveaffectedtheoutcome?
2.5Wereanyparticipantsanalysedinagroupdifferentfromtheonetowhichtheywereassigned?
2.6IfY/PY/NIto2.5: Wastherepotentialforasubstantialimpact(ontheestimatedeffectofintervention)ofanalysingparticipantsinthewronggroup?
Blinding
Deviationsreflectusual
practice?
Firstprincipleof
ITT
Biasduetodeviationsfromintendedinterventions
Effectofstartingandadheringtointervention2.1.Wereparticipantsawareoftheirassignedintervention
duringthetrial?2.2.Werecarersandtrialpersonnelawareofparticipants'
assignedinterventionduringthetrial?2.3.IfY/PY/NIto2.1or2.2:Wereimportantco-interventions
balancedacrossinterventiongroups?2.4.Wastheinterventionimplementedsuccessfully?2.5.Didstudyparticipantsadheretotheassignedintervention
regimen?2.6.IfN/PN/NIto2.3,2.4or2.5:Wasanappropriateanalysis
usedtoestimatetheeffectofstartingandadheringtotheintervention?
Blinding
Specificdeviations
Overcomebyanalysis?
Biasduetomissingoutcomedata
Missingoutcomedata
• Whencompleteoutcomedataforallparticipantsisnotavailableforyourreview• attrition- losstofollowup,withdrawals,othermissingdata• exclusions– someavailabledatanotincludedinreport
• Considerations• howmuchdataismissingfromeachgroup?(includenumbersinyourdescription)
• whyisitmissing?• howwerethedataanalysed?
Source:CochraneTraininghttp://training.cochrane.org/resource/assessing-risk-bias-included-studies
Biasduetomissingoutcomedata
3.1.Wereoutcomedataavailableforall,ornearlyall,participantsrandomized?
3.2.IfN/PN/NIto3.1:Aretheproportionsofmissingoutcomedataandreasonsformissingoutcomedatasimilaracrossinterventiongroups?
3.3.IfN/PN/NIto3.1:Isthereevidencethatresultswererobusttothepresenceofmissingoutcomedata?
Anymissingdata?
Amountandreasons?
Resultsrobust?
Biasinmeasurementoftheoutcome
Biasinmeasurementoftheoutcome
• Systematicdifferencesbetweengroupsinhowoutcomesareassessed
• Someoutcomesaremorepronetobiasthanothers• Patient-reportedoutcome(e.g.pain,qualityoflife)• Observer-reportedinvolvingjudgement(e.g.clinicalexamination)
• Observer-reportednotinvolvingjudgement(e.g.all-causemortality)
Biasinmeasurementoftheoutcome
4.1.Wereoutcomeassessorsawareoftheinterventionreceivedbystudyparticipants?
4.2.IfY/PY/NIto4.1:Wastheassessmentoftheoutcomelikelytobeinfluencedbyknowledgeofinterventionreceived?
Blinding?
Assessmentinfluenced?
Biasinselectionofthereportedresult
Selectivereporting
• Currenttooltakesabroadapproachtoselectivereporting• Anyevidenceofitinthetrialreports?
38
Results
• Selectivenon-reportingbiasestheresultofthemeta-analysiswhichcannotincludethetrialthatomittedtheoutcome;itdoesnotbiasthetrialresult
• Thisissimilartopublicationbias(non-reportingofastudy)
WeincludeonlyselectionofthereportedresultintheRoB 2.0tool
...andconsiderselectivenon-reporting inotherways
Biasinselectionofthereportedresult
Trialresultisbiasedbecauseithasbeenselectedonthebasisoftheresultsfrommultiple:• Outcomemeasurements
• Scales• Definitionsof/criteriaforanevent• Timepoints
• Analyses• Unadjustedvsadjustedmodels• Differentsetsofcovariatesinadjustedmodels• Finalvaluesvschangefrombaselinevsanalysisofcovariance• Continuousscaleconvertedtocategoricaldatawithdifferentcut-
points
Biasinselectionofthereportedresult
Arethereportedoutcomedatalikelytohavebeenselected,onthebasisoftheresults,from...
5.1....multipleoutcomemeasurements(e.g.scales,definitions,timepoints)withintheoutcomedomain?
5.2...multipleanalysesofthedata?
Selectiveoutcomereporting
Selectiveanalysisreporting
Piloting
• RoB 2.0hasundergonemultiplephasesofpiloting• informeddevelopmentandrefinement• moreisalwayswelcome
• Formalstudiesofinter-rateragreementnotyetperformed
• Fullguidanceavailableatriskofbias.info• initialdraft,subjecttominorrefinements
Someunresolvedissues
• Howmanyresultstoassessperstudy?• Howtointegrateintodatacollectionprocess?• Howtopresentassessmentsinareview?
• Implementation• RoB 2.0willneedcarefulconsiderationtomaketheprocessefficientformultipleoutcomes
• DiscussionsinitiatedwithRevMan andCovidence teamatSeoulColloquium
Concludingremarks
• WebelieveRoB 2.0offersconsiderableadvantagesovertheexistingtool
• Onceprogrammedintosoftware,weexpectthetoolwillbeeasytouseandintegrateintotheinterpretationofresults
• WeareextremelygratefultoallthosewhohavecontributedtothedevelopmentofRoB 2.0
• RoB 2.0isavailableatriskofbias.info