RTI International is a trade name of Research Triangle Institute www.rti.org ancy Berkman , PhD Meera Viswanathan, PhD mail: berkman@r ti.org Email: [email protected]Development of a Tool to Evaluate the Quality of Non-randomized Studies of Interventions or Exposures Presented by Nancy D Berkman, PhD & Meera Viswanathan, PhD Presented at AHRQ 2009 Annual Conference Bethesda, Maryland September 15, 2009
12
Embed
RTI International is a trade name of Research Triangle Institute Nancy Berkman, PhDMeera Viswanathan, PhD
Context for the Project Increasing demand to include non-randomized studies in systematic literature reviews and comparative effectiveness reviews to capture –The effects of interventions or exposures on a more broadly defined population than can be observed through RCTs –Topics where RCTs would be logistically or ethically inappropriate –Longer term outcomes and harms (side effects) The trade-off for wider applicability of findings among observational studies, compared with RCTs, is a potentially wider range of sources of bias, including in selection, performance, detection of effects, and attrition.
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
RTI International is a trade name of Research Triangle Institute
– Phase 1:• Grant from RTI Independent Research and Development
(IR&D) funds
– Phase 2:• Contract from Agency for Healthcare Research and
Quality (AHRQ), U.S. Department of Health and Human Services, through the Evidence-based Practice Centers program (EPC)
www.rti.org
Context for the Project
• Increasing demand to include non-randomized studies in systematic literature reviews and comparative effectiveness reviews to capture– The effects of interventions or exposures on a more broadly defined
population than can be observed through RCTs– Topics where RCTs would be logistically or ethically inappropriate– Longer term outcomes and harms (side effects)
• The trade-off for wider applicability of findings among observational studies, compared with RCTs, is a potentially wider range of sources of bias, including in selection, performance, detection of effects, and attrition.
www.rti.org
Background: Rating the Quality of Non-randomized Studies
The quality (internal validity) of each study included in a review needs to be evaluated:
• Well-established criteria and instruments exist for evaluating the quality of RCTs, but not non-randomized (observational) studies
• PIs conducting systematic reviews generally lack access to validated and adaptable instruments for evaluating the quality of observational studies.
• Each new review often develops its own quality rating tool, “reinvents the wheel”, leading to inconsistent standards within and across reviews.
www.rti.org
Project Goals
To create a practical and validated tool for evaluating the quality of non-randomized studies of interventions or exposures that is:
• Reflects a comprehensive theoretical framework: captures all relevant domains
• Broad applicability: can be used "off the shelf" by different PIs • Modifiable: can be adapted to different topic areas • Easy to use and understand: can be used by reviewers with
varying levels of expertise or experience • Validated: users can be confident of their evaluation of study
quality • Advances the methodology in the field• Disseminated widely
www.rti.org
Methods: Phase 1
Item development
• Reviewed the literature on the evaluation of the quality of observational studies
• Collected quality review items used in early tools to evaluate non-RCTs through – Published literature– 90 AHRQ-sponsored EPC reviews
• Categorized all potential items into the 12 quality domains identified in Evaluating non-randomized intervention studies (Deeks et al., 2003)
www.rti.org
Methods: Phase 1 (continued)
Item Bank development
• Selected the best items for measuring each of the included domains
• Modified selected items where necessary to ensure that critical domains were included and to improve readability
• Developed a pre-specified set of responses
• Developed explanatory text to be used by PIs and abstractors to individualize as well as standardize interpretation
www.rti.org
Methods: Phase 2
Technical Expert Panel input– Conceptual framework to ensure that we included all
relevant domains– Face validity
• Cognitive interviews with potential users– Readability– Conceptualization
Evaluation of quality can rely on either a description of methods or an assessment of validity and precision
• Methods description approach– Follows the reporting structure of many manuscripts– Relies less on judgment than on reporting
• Validity and precision approach– What we really care about– More challenging to evaluate– Greater reliance on judgment
www.rti.org
Domains for quality evaluation approaches
Methods description approach• Background/context• Sample definition and selection• Intervention/exposure• Creation of treatment groups• Follow-up• Specification of outcomes• Analysis: comparability of
groups• Analysis: outcomes• Interpretation
Validity and precision approach • Selection bias• Performance bias• Information bias• Detection bias• Attrition bias• Reporting bias• Precision
www.rti.org
Tool Results
• Comprehensive: bank of 39 questions• Modifiable: includes relevant items appropriate for all
non-randomized study types• Easy to use: instructions for PIs and abstractors to assist
in appropriate interpretation of questions. Example:
What is the level of detail in describing the intervention or exposure? [PI: specify which details need to be stated, e.g., intensity, duration, frequency, route, setting, and timing of intervention/exposure. For case-control studies, consider if the condition, timing, frequency, and setting of symptoms is provided in the case definition]
www.rti.org
Next Steps
• Finalize inter-rater reliability results• Publish findings and disseminate the tool
• Proposed Phase III:– Design specific validation including inter-rater reliability
testing by study type– Reduce the number of questions needed to address
specific domains– Develop a web-based platform for generating design and