Pt 650 1A

PT650 1AScientific Inquiry I:Critical Appraisal of Medical LiteratureLinda [email protected] Sciences Librarian, Rinker Health Science CampusJune 30, 2016

Today’s Agenda

• Review of health science research process• Types of research studies and levels of evidence• Critical appraisal of research studies

• What is a Critical Appraisal?• How to appraise a research paper

• In-class appraisal activity

ASK ACQUIRE APPRAISE

Formulate a researchable question, find relevant articles, and evaluate articles for suitability

Review of the health science research process…

ASK: Formulate a focused question

Patient / Problem / PopulationInterventionComparisonOutcome

PICO is a mnemonic that helps one remember the key components of a well focused question. The question needs to identify the key problem of the patient, what treatment or tests you are considering for the patient, what alternative treatment or tests are being considered (if any) and what is the desired outcome to promote or avoid.

Our PICOA sedentary 82 year old woman living alone has been falling frequently. Her doctor has referred her to PT to help strengthen her legs and improve her balance.

Patient / Problem / Population – 82 year old white female who is fallingIntervention – physical therapy, movement exercisesComparison – do nothing (modify home environment)Outcome – prevent falls at all times of day

Can physical therapy interventions reduce falls in elderly patients?

Developing a search strategy

1. Frame the question – use PICO2. Find background information – use AccessMedicine

or 5Minute Consult or…3. Develop a search strategy using keywords and

phrases4. Perform PubMed search, and refine search terms

as you retrieve results5. Use the tools in the database to help with your

search!6. Evaluate the results obtained, and further refine if

necessary

ASK ACQUIRE APPRAISE Find relevant articles

PubMed Advanced Search – choosing ‘Publication Type’

ASK ACQUIRE APPRAISE

Evaluate articles for suitability

Critical Appraisal of Research Topics

Critical Appraisal Learning Outcomes

By the end of this section you should:

• Understand the principles of criticalappraisal and why you should undertake it

• Be able to appraise published research and judge its reliability

• Be able to assess the relevance of published research to your own work

What is Critical Appraisal?

‘Critical appraisal is the assessment of evidence by systematically reviewing its relevance, validity and results to specific situations’Chambers, R (1998)

A balanced assessment of strengths of research against its weaknesses. It is a skill that needs to be practiced!

Before we appraise medical studies, let’s review levels of evidence and study types

Dartmouth Biomedical Libraries: Resources. http://www.dartmouth.edu/~biomed/resources.htmld/guides/ebm_resources.shtml. Accessed April 29, 2016

Quality of evidence

Study Types Defined meta-analysis: a quantitative approach in which individual study findings addressing a

common problem are statistically integrated and analyzed to determine the effectiveness of interventions.

systematic review: a process by which a body of literature is reviewed and assessed using systematic methods which are intended to reduce bias in the review process and improve understandability.

randomized controlled trial: an experiment in which investigators randomly allocate participants into (e.g. treatment and control) groups to receive or not to receive one or more interventions that are being compared.

controlled clinical trial: any study which compares two groups by virtue of different therapies or exposures.

cohort study: an observational study in which a cohort is followed over time.case-control studies: retrospective research design that compares individuals with a

specific condition to those without it.case report: a detailed report of the symptoms, signs, diagnosis, treatment, and

follow-up of an individual patient. case series: a report on a series of patients with an outcome or condition of interest.

Susan Fowler, MLIS Washington University in St. Louis 2012

Levels of Evidence in Medical Studies Systematic Revie

ws and

Meta-analyses

Randomised Controlled Trials (RCTs)

Cohort Studies

Case Control Studies

Case-Control Studies• Conducted after outcome has occurred• Used mainly for causation studies• Typically retrospective (looks back in time)• Patient with outcome matched to control• Investigations made into possible causes in both

patients• May be only option in rare conditionsExample: Study used a matched design, matching infants who had persistent pulmonary hypertension with infants who did not have it, and compared the rates of exposure to SSRIs.

Cohort Studies

• Two or more populations (cohorts) are followed prospectively to see which risk factors are associated with respect to disease or outcome

• One group has received an intervention or exposure • Groups otherwise closely matched• Groups followed over time• Can be used for causation, diagnostic, harms and

therapeutic studiesExample: To determine the long-term effectiveness of influenza vaccines in elderly people, cohorts of vaccinated elderly and unvaccinated community-dwelling elderly were studied. The results suggest that the elderly who are vaccinated have a reduced risk of hospitalization for pneumonia or influenza.

Randomized Controlled Trials (RCT)

• Treatment group and ‘control’ group• Random assignment to groups• May involve ‘blinding’ of participants and

researchers• Used for therapeutic or diagnostic interventions• Some interventions unsuitable for RCTs• Expensive

Example: This research studied the effect of raloxifene on fracture risk in postmenopausal women, and found that the women who took raloxifene over the same five year period of time as the women who did not reduced their risk of clinical vertebrate fracture.

Systematic Reviews and Meta-Analyses

• Combination of results of several RCTs or other types of evidence

• Meta-analyses includes a statistical combination of the results

• Considered to be highest-level evidence if conducted properly

• Relies on enough RCTs being available• Can include ‘Odds-Ratio’ diagrams

Example: Respiratory muscle training increases respiratory muscle strength and reduces respiratory complications after stroke: a systematic review.

Other Types of Clinical Research

These are lesser forms of evidence, but for some interventions, exposures or conditions they may be the only form available

• Case studies / Case series - compares patients who have a disease or outcome of interest (cases) with patients who do not have the disease or outcome (controls), and looks back retrospectively to compare how frequently the exposure to a risk factor is present in each group to determine the relationship between the risk factor and the disease. These studies are designed to estimate odds.

• Cross-sectional surveys – Data collected once in a short period of time

Quiz

What Type of clinical research is this?

1. Two groups of doctors, one group smokers, the other non-smokers are followed over the course of 20 years to see whether which group are more likely to develop lung cancer

Answer

Cohort study

Quiz

What Type of clinical research is this?

2. Two groups of patients are studied, one group given physiotherapy for low back pain, the other given advice only. Patients are randomly assigned to either group and followed up after six months

Answer

Randomised controlled trial

Quiz

What Type of clinical research is this?

3. One hundred sets of twins, where one had developed melanoma and the other had not, were studied for possible causation factors

Answer

Case-control study

(As each twin should be closely matched in most aspects other then one having developed

melanoma)

Break!

Critical Appraisal of Medical Research

• Balanced assessment of strengths of research against its weaknesses

• Assessment of research process and results• Consideration of quantitative and qualitative

aspects of research• To be undertaken by all health professionals as

part of their work

Critical Appraisal is NOT:

• Negative dismissal of a piece of research• Assessment of results alone• Based entirely on detailed statistical

analysis• To be undertaken by expert researchers

only

Why Should We Bother?• Published research is not

always reliable or applicable

• Patients are informed by the news and media reporting the results of scientific studies

Drinking more coffee may stave off multiple

sclerosis!!

Gluten triggers strange

delusions!

Low fat diet bad for your health and cutting back on meat, dairy and eggs a disastrous mistake!

Questions to ask when appraising medical literature

Appraising medical literature1. Is the study question relevant? Study should address an

important topic, add to what is already known and be relevant to one’s work.

2. Does the study add anything new? Scientific research is built upon previous research – subsequent studies should extend research and enhance knowledge.

3. What type of research question is being asked? A well-developed research question usually identifies three components: the group or population of patients, the studied parameter (e.g. a therapy or clinical intervention) and the outcomes of interest. Typically questions address: Effectiveness of Treatment or the Frequency of Events.

Appraising medical literature

4. Was the study design appropriate for the research question? The research question determines the most appropriate study design. Randomized Controlled Trials (RCT) are the gold standard for intervention studies. Cohort studies are common for diagnostic tests. There are many different types of studies.

5. Did the study methods address the most important potential sources of bias? Bias in this context does not mean investigator bias, but rather bias attributed to chance (e.g. a random error) or to the study methods (systematic bias). Systematic biases arise from how participants were selected, how data was collected, or through the researchers' analysis or interpretation.

Appraising medical literature

6. Was the study performed according to the original protocol? Deviations from the planned protocol can affect the validity or relevance of a study – often planned participation is lower than anticipated or drop-outs occur.

7. Does the study test a stated hypothesis? A hypothesis is a clear statement of what the investigators expect the study to find and states the research question in a form that can be tested and refuted

8. Were the statistical analyses performed correctly? Difficult to ascertain; read the Methods section, and pay attention to how missing data are treated. Ideally, this section should be understandable to non-statisticians.

Appraising medical literature

9. Do the data justify the conclusions? Are the conclusions that the authors present reasonable on the basis of the accumulated data? Be wary of generalizations and size of populations studied.

10. Are there any conflicts of interest? Personal factors, including financial, career and outcome-based conflicts. This is a judgement call on the part of the reader.

Questions?

How to do Critical Appraisal

• Checklists are available that ask simple questions

• There are different checklists for different types of studies (systematic reviews, RCTs etc)

• These checklists help you focus on the most important aspects of the article

Key Steps in Critical Appraisal

The checklists help us answer these questions

• What are the results? Are they significant?

• Are the results valid? How was the research done?

• How will these results help me work with my patients?

Diagnostic Research Studies• PT Diagnosis process:

• Patient history• Systems review• Tests & measures

• Diagnostic tests have a range of values• Should be compared to the Reference or Gold Standard test

• e.g. MRI compared to PT musculoskeletal test• ‘Index test’ or ‘test of interest’ are the tests being investigated

Appraising Diagnostic Studies• Appraising diagnostic literature

• Determining applicability – relevance of clinical population to your patient

• Quality – Was there an independent, blind comparison with the gold standard diagnostic tool? Were examiners unaware of results of gold std test? Did all subject receive both tests?

• Results: Clinically useful statistics reported and interpreted?• Sensitivity – clinical test’s accuracy in correctly identifying problem as

compared to gold standard.• Specificity – test’s ability to correctly identify absence of problem (no

problem)

• Summarizing clinical bottom line of diagnostic study• Is the test accurate and clinically relevant to PT practice?• Will the results of this test (knowing it’s limitations) affect treatment plan

and help patient?

Sensitivity: clinical test’s accuracy in correctly identifying a problem as established by the Gold Standard.

Specificity: clinical test’s ability to correctly identify the absence of a problem.

Intervention Studies• The intervention (the preventative or therapeutic measure)

being tested is allocated by the investigator to a group of two or more study populations

• Subjects are followed prospectively to compare the intervention vs. the control (standard treatment, no treatment or placebo)

• The main intervention study design is the randomized controlled trial (RCT)

Appraising Intervention Studies• Inclusion and exclusion criteria should be clearly defined AND

applicable• Realistic interventions and controls• Outcome measures relevant to the clinical question and

conducted realistically• Appropriate random assignment of participants• Bias minimized?

• Randomization• Recruitment of participants• Minimal dropout rate• Masking or blinding of researchers, evaluators, participants,

therapists

p-value

The P value, or calculated probability, is the probability of finding the observed, or more extreme, results when the null hypothesis (H 0) of a study question is true – the definition of 'extreme' depends on how the hypothesis is being tested.

p-value < 0.05Can state with 95% confidence that the result is not due to chance

Most authors refer to statistically significant as P < 0.05 and statistically highly significant as P < 0.001 (less than one in a

thousand chance of being wrong).

Diagostic Study Appraisal RubricYes No ? Comments

1. Is the entire spectrum of patients represented in the study sample?

2. Is my patient represented in thisspectrum of patients?

3. Was there an independent blindcomparison with a reference (gold) standard of diagnosis?

4. Do the authors provide evidence of the “quality” of the gold standard?

5. Were the diagnostic tests performedby one or more reliable examiners whowere masked to the results of the reference (gold) test? (requires that some level of evidence of the qualifications of the examiners is provided)

6. Did all subjects receive both (index andgold standard tests) regardless of test outcomes?

7. Was there minimal time delaybetween the administration of the index tests and the "gold" standard?

Yes No ? Comments

8. Was the diagnostic test interpreted independently of all clinical information?

9. Do the authors provide clinically useful measures of the diagnostic test’s utility, for instance likelihood ratios (patient centered) to support the Sensitivity and Specificity (test centered)?

10. Is the test accurate (when comparedto the Gold standard)?

11. is the test clinically relevant tophysical therapy practice?

12. Will the resulting post-testprobabilities affect my management and help my patient?

Summary Comments (take home message):

danielcip32013-11-01 22:21:13--------------------------------------------Summary Comments (take home message):

Additional sources of Checklists

Critical appraisal checklists can be downloaded from:

Critical Appraisal Skills Programmewww.casp-uk.net

Centre for Evidence-Based Medicinewww.cebm.net/index.aspx?o=1157

The Good News Is…

Some healthcare research has already been appraised

Cochrane Library (good quality systematic reviews conducted

by the Cochrane Collaboration)Database of Abstracts of Reviews of Effects

(critical appraisals of published systematic reviews)

In-class appraisal activity

Link to myNCBI bibliography

Questions?Contact Linda Galloway, [email protected]