Critical Appraisal of Health Care Intervention Studies Dr. Maureen Markle-Reid, RN, MScN, PhD Associate Professor and Acting Assistant Dean (Research), School of Nursing, Associate Member, Clinical Epidemiology and Biostatistics Nursing Graduate Seminars October 4, 2011
50
Embed
Critical Appraisal of Health Care Intervention Studies
Critical Appraisal of Health Care Intervention Studies. Dr. Maureen Markle-Reid, RN, MScN, PhD Associate Professor and Acting Assistant Dean (Research), School of Nursing, Associate Member, Clinical Epidemiology and Biostatistics Nursing Graduate Seminars October 4, 2011. Objectives. - PowerPoint PPT Presentation
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Critical Appraisal of Health Care Intervention Studies
Dr. Maureen Markle-Reid, RN, MScN, PhDAssociate Professor and Acting Assistant Dean (Research),
School of Nursing, Associate Member, Clinical Epidemiology and Biostatistics
Nursing Graduate SeminarsOctober 4, 2011
Objectives Define evidence-informed decision-making and
describe its importance to nursing practice Identify the parts of a relevant, answerable
question Describe the different types of questions
relevant to nursing practice Define the term critical appraisal and discuss its
relevance to nursing practice, policy and research
Identify criteria appraisal criteria for health care intervention studies
Develop skill in applying the criteria for critical appraisal of an intervention study to determine the quality and applicability of the research.
What is evidence-informed decision-making?
“The conscientious, explicit and judicious use of the current best evidence in making decisions about the care of individual patients”
(Sackett et al, 1996)
Health care resources
Patient preferences and actions
Research evidence
Clinical state, setting, and circumstances
Clinical Expertise
Clinical Decision
A Model for Evidence-Based Clinical Decisions
[Haynes, R.B., Devereaux, P.J., & Guyatt, G.H. (2002). Clinical expertise in the era of evidence-based medicine and patient choice. ACP J Club, 136, A11-14]
Why is evidence-based practice important?
Demonstrate that nursing actions and decisions are clinically appropriate and result in positive outcomes for clients, their families and health care system as a whole
Demonstrate professional accountability to clients Provide evidence for the cost-effectiveness of
nursing care
Steps to Evidence-Informed Decision-Making:
1.1. Define: Define: Formulate a focused answerable question Formulate a focused answerable question from the practice situation based on your information from the practice situation based on your information needs needs
2.2. Search:Search: Efficiently search for research Efficiently search for research
3.3. Appraise:Appraise: Critically and efficiently appraise the Critically and efficiently appraise the research sourcesresearch sources
4.4. Synthesize:Synthesize: Interpret/form recommendations for Interpret/form recommendations for practice based on the literaturepractice based on the literature
5.5. Apply:Apply: Apply the results to the client/population Apply the results to the client/population
6.6. Implement: Implement: Decide whether (and plan how) to Decide whether (and plan how) to implement the adapted evidence into practiceimplement the adapted evidence into practice
7.7. Evaluate: Evaluate: Evaluate the effectiveness of Evaluate the effectiveness of implementation effortsimplementation efforts
Formulate a Focused Answerable QuestionFormulate a Focused Answerable Question
P opulation / Situation
I ntervention / Exposure
C ounter intervention
O utcome T imeframe
Population/Situation:
Client group or clinical scenario of interest Single patient or group of patients with a
particular condition or health care problem, e.g.,: A person with a health condition People with hypertension (a group of people
with a particular condition Primary health care for the elderly (an aspect
of health care delivery)
Intervention/Exposure:
Interventions can be: Therapeutic Preventive Diagnostic Organizational
If exploring the meaning of a phenomena, the question may involve a situation rather than an intervention
Counter Intervention: Clinical decisions involve choosing between alternative
courses of action (or no action)
Outcome:Outcome: What is the outcome, or the effect we are hoping to
achieve by the using the intervention? May be more than one outcome that is important to the
question
Time Frame:
What is the period of time over which the question occurs?
What is the optimal time to measure a change in the outcome(s)?
Step 1:Formulate a focused answerable questionStep 1:Formulate a focused answerable question
Formulating Answerable Clinical Questions (Centre for Evidence- (Centre for Evidence-Based Medicine, Mount Sinai Hospital)Based Medicine, Mount Sinai Hospital)
The Well-Built Clinical Question (Duke University Medical Center (Duke University Medical Center Library and Health Sciences Library, UNC-Chapel Hill) Library and Health Sciences Library, UNC-Chapel Hill)
Formulating Patient Centered Questions (University Library, (University Library, University of Illinois at Chicago)University of Illinois at Chicago)
Asking Focused Questions (Centre for Evidence-Based Medicine, (Centre for Evidence-Based Medicine, Oxford) Oxford)
Types of questions relevant to nursing practice Therapy/Health care interventions: What is the effectiveness
of different interventions? Causation and harm: What might be causing disease/ill
health/adverse effects? Diagnosis or assessment: Does a diagnostic test differentiate
between people with and without a condition? Prognosis: What are potential future outcomes of a condition? Economic evaluation: What is the cost-effectiveness of
different interventions? Meaning: Describing, exploring and explaining aspects of health
and illness.
Nursing Graduate Seminar 200916
Different clinical questions require Different clinical questions require evidence from different research evidence from different research
designsdesignsType of Question Suggested Best Type of Study Design
Therapy/Health care Therapy/Health care intervention intervention
RCT>cohort>case control>case seriesRCT>cohort>case control>case series
DiagnosisDiagnosis Prospective, blind comparison to a gold standardProspective, blind comparison to a gold standard
PrognosisPrognosis Cohort study>case control>case seriesCohort study>case control>case series
Harm (Causation or Harm (Causation or etiology)etiology)
RCT>cohort>case control>case seriesRCT>cohort>case control>case series
Meaning of illnessMeaning of illness QualitativeQualitative
Qualitative Survey(cross sectional)
(Randomised)crossover
(Randomised)Parallel group
Cohort study
Cross-sectional(analytic)
Case-control study
Q1
Q3
Q2
All studies
Experimental
Analytic
(PICO or PECO) Descriptive
(PO) Theoretical
(PS)
Observational
analytic
What type of research design will you use?
[Glasziou, P., & Heneghan, C. (2009). A spotter’s guide to study designs. EBN Notebook, 12, 71-72]
Step 2: Collect the best evidence: Hierarchy of Pre-Processed Evidence
[Adapted from Haynes, R.B. (2007). Of studies, summaries, synopses, and systems: The “5S” evolution of information services for evidence-based healthcare decisions. Evidence-Based
Nursing, 10, 6-7]
EXAMPLES: SYSTEMS:
Computerized decision support
SUMMARIES::Evidence-Based Guidelines
~ Evidence-Based Texts
SYNOPSES OF SYNTHESES:DARE ~ Evidence-Based
Abstract Journals
SYNTHESES (Systematic Reviews): Cochrane Database
of Systematic Reviews SYNOPSES OF SINGLE
STUDIES:Evidence-Based Abstract
Journals
SINGLE STUDIES:Clinical Queries
Step 3: Critically appraise the literature for Step 3: Critically appraise the literature for
validity and applicabilityvalidity and applicability
What is critical appraisal?What is critical appraisal?
Critical appraisal is the process of assessing and Critical appraisal is the process of assessing and interpreting evidence by systematically considering its interpreting evidence by systematically considering its validity, results and relevance to an individual's work.validity, results and relevance to an individual's work.
Relevance of Critical Appraisal to Nursing Relevance of Critical Appraisal to Nursing
Practice, Policy and ResearchPractice, Policy and Research
Use the literature more effectively in Use the literature more effectively in answering clinical questions to guide answering clinical questions to guide clinical practiceclinical practice
Distinguish stronger evidence from Distinguish stronger evidence from weaker evidence – identify high quality weaker evidence – identify high quality research research
Identify the methodological strengths and Identify the methodological strengths and limitations, results and relevance of the limitations, results and relevance of the studies in answering a clinical questionstudies in answering a clinical question
Three Steps in Using an Article from the
Health Care Literature
Are the results valid?
What are the results?
Are the results applicable (and useful) to my client care population?
[DiCenso, A., Guyatt, G., & Ciliska, D. (2005). Evidence-based nursing: A guide to clinical practice]
Are the results valid?Are the results valid?
Are the study methods sufficiently Are the study methods sufficiently rigorous to ensure that the study results rigorous to ensure that the study results represent an unbiased estimate of the represent an unbiased estimate of the true effect?true effect?
OROR
Are the study methods sufficiently biased Are the study methods sufficiently biased to lead to a false conclusion?to lead to a false conclusion?
Are the results valid?Are the results valid?
Final assessment of validity is never Final assessment of validity is never a yes/no decisiona yes/no decision
Validity as a continuum ranging Validity as a continuum ranging from strong studies to weak studiesfrom strong studies to weak studies
Evaluation of the validity of a study Evaluation of the validity of a study involves some subjectivity involves some subjectivity
What are the results?What are the results?
Size and precision of the estimate of Size and precision of the estimate of effecteffect
Are the results applicable (and useful) to my client care population?
Can you apply the results to patients Can you apply the results to patients in your clinical setting?in your clinical setting?
Were all important outcomes Were all important outcomes considered? considered?
Are the likely intervention benefits Are the likely intervention benefits worth the potential harm and costs?worth the potential harm and costs?
User Guides to Evidence-Based PracticeUser Guides to Evidence-Based Practice
1993-2000: Evidence based medicine working group: “User guides 1993-2000: Evidence based medicine working group: “User guides to the Medical Literature” in JAMAto the Medical Literature” in JAMA
2002: Guyatt et al. “User guides to the medical literature: A manual 2002: Guyatt et al. “User guides to the medical literature: A manual for evidence based clinical practice”for evidence based clinical practice”
2005: DiCenso, Guyatt, & Ciliska. “Evidence-based nursing: A 2005: DiCenso, Guyatt, & Ciliska. “Evidence-based nursing: A guide to clinical practice”guide to clinical practice”
Critical Appraisal Forms:Critical Appraisal Forms: Critical Appraisal of Articles on Critical Appraisal of Articles on CAUSATIONCAUSATION Critical Appraisal of Articles on Critical Appraisal of Articles on
THERAPY/INTERVENTIONSTHERAPY/INTERVENTIONS Critical Appraisal of Critical Appraisal of SYSTEMATIC REVIEWSSYSTEMATIC REVIEWS Critical Appraisal of Articles on Critical Appraisal of Articles on PREVALENCE AND PREVALENCE AND
INCIDENCEINCIDENCE Critical Appraisal of Critical Appraisal of QUALITATIVE RESEARCHQUALITATIVE RESEARCH Critical Appraisal of Articles on Critical Appraisal of Articles on PROGNOSISPROGNOSIS
http://www.cche.net/usersguides/prognosis.asphttp://www.cche.net/usersguides/prognosis.asp Critical Appraisal of Critical Appraisal of GUIDELINES GUIDELINES [AGREE: Appraisal of [AGREE: Appraisal of
Guidelines Research and Evaluation Instrument] Guidelines Research and Evaluation Instrument] http://www.agreecollaboration.org/pdf/agreeinstrumentfinalhttp://www.agreecollaboration.org/pdf/agreeinstrumentfinal.pdf.pdf
Example: The Effectiveness of a Nurse-Led Example: The Effectiveness of a Nurse-Led Interprofessional Team Approach to Fall Prevention Interprofessional Team Approach to Fall Prevention
in Older Home Care Clients at Risk of Fallingin Older Home Care Clients at Risk of Falling
Markle-Reid, M., Browne, G., Gafni, A., Roberts, J., Weir, R., Markle-Reid, M., Browne, G., Gafni, A., Roberts, J., Weir, R., Thabane, L., Miles, M., Vaitonis, V., Hecimovich, C., Baxter, P., & Thabane, L., Miles, M., Vaitonis, V., Hecimovich, C., Baxter, P., & Henderson, S. (2010). The effects and costs of a multifactorial and Henderson, S. (2010). The effects and costs of a multifactorial and interdisciplinary team approach to fall prevention for older home interdisciplinary team approach to fall prevention for older home care clients “at risk” for falling: A randomized controlled trial. care clients “at risk” for falling: A randomized controlled trial. Canadian Journal on Aging, 29Canadian Journal on Aging, 29(1), 139-161(1), 139-161
Critical Appraisal Criteria Critical Appraisal Criteria for Health Care for Health Care Intervention StudiesIntervention StudiesScreening Questions:Screening Questions:
1.1. Did the study ask a clearly focused question? Did the study ask a clearly focused question?
Consider if the question is ‘focused’ in terms of:Consider if the question is ‘focused’ in terms of: The population studiedThe population studied The intervention givenThe intervention given The outcomes consideredThe outcomes considered
Research Question:Research Question:
`What is the effectiveness of a 6-month nurse-led `What is the effectiveness of a 6-month nurse-led interprofessional team approach to fall prevention compared interprofessional team approach to fall prevention compared with usual home care services in older home care clients “at with usual home care services in older home care clients “at risk” of falling with respect to the number of falls and fall risk” of falling with respect to the number of falls and fall risk factors (slip or trip, health-related quality of life and risk factors (slip or trip, health-related quality of life and function, depressive symptoms, nutritional status, gait and function, depressive symptoms, nutritional status, gait and balance, cognitive function, fear of falling)? balance, cognitive function, fear of falling)?
PP Frail older adults using home care services at risk for Frail older adults using home care services at risk for fallingfalling
II Nurse-led interprofessional team approach to fall Nurse-led interprofessional team approach to fall preventionprevention
CC Usual home care servicesUsual home care services OO Number of falls, slip or trip, Number of falls, slip or trip, health-related quality health-related quality
of life and function, depressive symptoms, nutritional of life and function, depressive symptoms, nutritional status, gait and balance, cognitive function, fear of falling status, gait and balance, cognitive function, fear of falling
TT 6 months6 months
Critical Appraisal Criteria Critical Appraisal Criteria for Health Care for Health Care Intervention StudiesIntervention Studies
Screening Questions:Screening Questions:2.2. Was this a randomized controlled trial (RCT) and was it Was this a randomized controlled trial (RCT) and was it
appropriately so?appropriately so?
Consider:Consider: Why this study was carried out as an RCTWhy this study was carried out as an RCT If this was the right research approach for the question If this was the right research approach for the question
being askedbeing asked
Critical Appraisal Criteria Critical Appraisal Criteria for Health Care for Health Care Intervention StudiesIntervention Studies
Is it worth continuing? Is it worth continuing?
Critical Appraisal Criteria Critical Appraisal Criteria for Health Care for Health Care Intervention StudiesIntervention Studies3.3. How were participants allocated to How were participants allocated to
intervention and control groups? intervention and control groups?
Consider:Consider: How participants were allocated to intervention and How participants were allocated to intervention and
control groups. Was the process truly random?control groups. Was the process truly random? Whether the method of allocation was describedWhether the method of allocation was described How the randomization schedule was generatedHow the randomization schedule was generated Were the groups comparable at baseline on Were the groups comparable at baseline on
characteristics that might explain the outcome? characteristics that might explain the outcome?
Population Population
Eligible SampleBaseline MeasuresEligible Sample
Baseline Measures
IneligibleIneligible
R n=
Treatment Control
Follow-up Measures Follow-up Measures
StrengthsStrengths
Random allocation of participants to Random allocation of participants to groups ensures that groups are similar in groups ensures that groups are similar in all respects except exposure to the all respects except exposure to the outcome; outcome;
Prospective design ensures that exposure Prospective design ensures that exposure to the intervention precedes the to the intervention precedes the development of the outcome;development of the outcome;
Greater likelihood that participants, health Greater likelihood that participants, health care providers, and outcome assessors care providers, and outcome assessors can be blinded. can be blinded.
LimitationsLimitations
May not be ethical or feasible;May not be ethical or feasible; Volunteer bias limits generalizability; Volunteer bias limits generalizability; Problems associated with a Problems associated with a
Final results may not be available for Final results may not be available for several years. several years.
Critical Appraisal Criteria Critical Appraisal Criteria for Health Care for Health Care Intervention StudiesIntervention Studies4.4. Were participants, staff and study Were participants, staff and study
personnel ‘blind’ to participants’ study personnel ‘blind’ to participants’ study group?group?
Consider:Consider: The fact that blinding is not always possibleThe fact that blinding is not always possible If every effort was made to achieve blindingIf every effort was made to achieve blinding If you think this matters to the studyIf you think this matters to the study The fact that we are looking for ‘observer The fact that we are looking for ‘observer
bias’bias’
Critical Appraisal Criteria Critical Appraisal Criteria for Health Care for Health Care Intervention StudiesIntervention Studies5.5. Was follow-up complete? Was follow-up complete?
Consider:Consider: If any intervention-group participants got a control If any intervention-group participants got a control
group option or vice versagroup option or vice versa If all participants were followed up in each study groupIf all participants were followed up in each study group Was there loss to follow-up bias?Was there loss to follow-up bias? If all of the participants’ outcomes were analysed by the If all of the participants’ outcomes were analysed by the
groups to which they were originally allocated (intention groups to which they were originally allocated (intention to treat analysis)to treat analysis)
Study FlowStudy Flow
R
Eligible Clientsn=267
Informed consent from 40.8% of eligible applicants
n=109
Refused 141Unable to Contact 13
Language 4158
Group 1n=54
Group 2 n=55
n=5 (9.3%)
n=49 (90.7%)
Interprofessional Falls Prevention Team
Usual Home Care Services
n=12 (21.8%)
n=43 (78.2%)
Group Size
Interventions
Dropouts @ 6 month follow-up: 17 clients (15.6%)
Analyzed @ 6 month follow-up: 92 clients (84.4%)
Critical Appraisal Criteria Critical Appraisal Criteria for Health Care for Health Care Intervention StudiesIntervention Studies6.6. Were participants in both groups followed up and data Were participants in both groups followed up and data
collected in the same way?collected in the same way?
Consider:Consider: Was data collected in the same way and at the same Was data collected in the same way and at the same
time interval for both groups? time interval for both groups?
Data Collection Data Collection
Multiple sources of data:Multiple sources of data: In-person interviewsIn-person interviews CCAC dataCCAC data RAI-HC dataRAI-HC data Data from service provider agenciesData from service provider agencies
Measurement of Clinical Outcomes: Measurement of Clinical Outcomes: Baseline and 6-monthsBaseline and 6-months
OutcomesOutcomesEffectsEffects::Frequency and Context of Falls (Falls Surveillance Report)Frequency and Context of Falls (Falls Surveillance Report)
Functional Health Status and Quality of Life (SF-36 Health Functional Health Status and Quality of Life (SF-36 Health Survey)Survey)
CostsCosts::Health Services Utilization (HSSU and CCAC Utilization Health Services Utilization (HSSU and CCAC Utilization Data) Data)
Critical Appraisal Criteria Critical Appraisal Criteria for Health Care for Health Care Intervention StudiesIntervention Studies7.7. How are the results presented and what is the main How are the results presented and what is the main
result?result?
Consider:Consider: What are the results?What are the results? How large this size of result and how meaningful it isHow large this size of result and how meaningful it is How you would sum up the bottom-line result of the trial How you would sum up the bottom-line result of the trial
in a few sentencesin a few sentences
ResultsResults At 6 months, there was noAt 6 months, there was no difference in the mean difference in the mean
number of falls between groups;number of falls between groups;
Subgroup analyses showed that the intervention Subgroup analyses showed that the intervention was effective in reducing falls was effective in reducing falls in menin men, 75-84 , 75-84 years years of age, with a fear of falling, or a negative history of age, with a fear of falling, or a negative history of fallsof falls;;
Greater reduction in number of slips and trips in Greater reduction in number of slips and trips in the intervention group; the intervention group;
Greater improvement in role functioning related to Greater improvement in role functioning related to emotional health in the intervention group; emotional health in the intervention group;
No additional cost from a societal perspective, e.g. No additional cost from a societal perspective, e.g. both interventions cost the same. both interventions cost the same.
Critical Appraisal Criteria Critical Appraisal Criteria for Health Care for Health Care Intervention StudiesIntervention Studies8.8. How precise are these results?How precise are these results?
Consider:Consider: If the result is precise enough to make a decisionIf the result is precise enough to make a decision If a confidence interval were reported. Would your If a confidence interval were reported. Would your
decision about whether or not to use this intervention decision about whether or not to use this intervention be the same at the upper confidence limit as at the be the same at the upper confidence limit as at the lower confidence limit?lower confidence limit?
Critical Appraisal Criteria Critical Appraisal Criteria for Health Care for Health Care Intervention StudiesIntervention Studies9.9. Were all important outcomes considered so the results Were all important outcomes considered so the results
can be applied?can be applied?
Consider whether:Consider whether: How the participants could be different from your population in How the participants could be different from your population in
ways that would produce different resultsways that would produce different results Your local setting differs from that of the trialYour local setting differs from that of the trial Is the intervention feasible in your setting?Is the intervention feasible in your setting?
Consider outcomes from the point of view of the:Consider outcomes from the point of view of the: IndividualIndividual Policy maker and professionalPolicy maker and professional Family caregiversFamily caregivers Wider community Wider community
Critical Appraisal Criteria Critical Appraisal Criteria for Health Care for Health Care Intervention StudiesIntervention Studies9.9. Were all important outcomes considered so the results Were all important outcomes considered so the results
can be applied?can be applied?
Consider whether:Consider whether: Any benefit reported outweighs any harm and/or costAny benefit reported outweighs any harm and/or cost Policy or practice should change as a result of the Policy or practice should change as a result of the
results of this trialresults of this trial
Methodological Strengths
Study design: randomized controlled trialStudy design: randomized controlled trial Follow-up period was appropriateFollow-up period was appropriate Follow-up was complete (<20% loss to Follow-up was complete (<20% loss to
follow-up)follow-up) Groups comparable at baselineGroups comparable at baseline
Methodological Limitations
Small sample sizeSmall sample size
Volunteer biasVolunteer bias
Potential for recall bias related to reporting fallsPotential for recall bias related to reporting falls
Strategies for collecting data regarding the number of Strategies for collecting data regarding the number of falls falls
Other Resources Evidence-Informed Decision-Making Resource Modules posted on Avenue to Learn Bandolier Evidence-Based Health Care http://www.medicine.ox.ac.uk/bandolier/ Tutorials:
Introduction to Evidence-Based Medicine (Duke University Medical Center Library) http://www.hsl.unc.edu/services/tutorials/ebm/index.htm
Directories: Evidence Based Medicine (EBM) Tookit (New York University School of Medicine.
The Frederick L. Ehrman Medical Library): www.urmc.rochester.edu/hslt/miner/digital_library/evidence_based_resources.cfm Resource Guide for Evidence-Based Practice (University of Alberta Libraries)
http://www.library.ualberta.ca/subject/evidence/guide/index.cfm McKibbon, A. (1999). PDQ: Evidence-based principles and practice. Hamilton, Ontario:
B.C. Decker Inc. Evidence Based Practice Tips http://www.ebmtips.net/risk001.asp