Introduction to Critical appraisals of the medical literature Partini Pudjiastuti, Sudigdo Sastroasmoro Department of Child Health Medical School University.

Introduction to Critical appraisals of the medical literaturePartini Pudjiastuti, Sudigdo Sastroasmoro

Department of Child Health Medical School University of Indonesia

Evidence based medicine 5 steps

Formulate question

Efficiently track down bestavailableevidence

Critically review thevalidity and usefulnessof the evidence

Implement changes in clinical practice

Evaluate performance

>25,000 journals worldwide>2 million published articles per yearMany published articles have methodological (including statistical) flaws – even in most respected journalsNot all results can be applied due to many reasons, a.o. dissimilarities of study subjects with our patientsLimited time of physicians: focus on articles relevant to your clinical practice

Rationale

Errors and inappropriateness may occur in any part of medical research:

in choosing appropriate design in choosing population in selecting study subjects in the details of the designin intervention and measurements in use of statistical methods in analysis in interpreting statistical analysisand in writing research report

The effect of dietary habit on calcium level in pediatric patients with nephrotic

syndromeJ Agric Soc Sci 2006;2

• r = - 0.1213, p = 0.015• There was a significant inverse relationship

between the age and the total calcium level ……

IMPORTANT!!!

Statistical significance vs. clinical importance

Negligible clinical difference may be statistically very significant if the number of subjects >>>. e.g., difference in reduction of cholesterol level of 3 mg/dl, n1=n2 = 10,000; p = 0.00002

Large clinical difference may be statistically non- significant if the no of subjects <<<, e.g. 30% difference in cure rate, if n1 = n2 = 10, p = 0.74

R

300 mg/dl

300mg/dl

Standardtreatment

New treatment

Cholesterol level, mg/dl

t = df = 9998 p = 0.00002

200

197

Clinical

Statistical

Clinical importance vs. statistical significance

n=10000

n=10000

Cured Died

Standard Rx 0 10 (100%)

New Rx 3 7 (70%)

Fischer exact test: p = 0.211

Clinical importance vs. statistical significance

Absolute risk reduction = 30% Clinical

Statistical

Critical appraisal (Making Reading More

Worthwhile)

What is Critical Appraisal?1. Critical appraisal = quality assessment2. ….process of weighing up evidence to

see how useful it is in decision making3. .…a process of assessing the validity,

reliability and usefulness of evidence4. …..is about considering, evaluating and

interpreting information in a systematic and objective way

Critically Appraise What You Read.

• Separating the wheat from the chaff.

• Time is limited – you should aim to quickly stop reading the dross.

• Others contain useful information mixed with rubbish.

• Simple checklists enable the useful information to be identified.

Critical Appraisal – Critical Thinking

• Appraising the available evidence to construct clinical reasoning and to make decisions

• Finding strengths and limitations of written ‘evidence’

• Deciding what evidence to pay attention to versus what to ignore

Why critically appraise?

• Supports sound decision making based on best available evidence

• Helps us determine (three R’s):•How rigorous a piece of research is - Validity

•What the results are telling us - Importance

•How relevant it is to our patient -Applicability

Value =

MortalityMorbidity

QoL

PatientSatisfacti

on

Health Status

Resources used

Quality

Cost

What is “Evidence”?

• People disagree on what constitutes “evidence”

• Evidence – generally = scientific fact

• Evidence - a combination of information obtained from 3 sources: research,

clinical experience, and client preferences (Kitson, Harvey, & McCormack, 1998)

Why do we need evidence?

– Resources should be allocated to things that are EFFECTIVE

– The only way of judging effectiveness is EVIDENCE

– “In God we trust – all others need evidence”

Sources of Evidence

• Primary sources– Based on experiments and published

research• Secondary sources

– Systematic reviews– Clinical guidelines– Journals of secondary publication

e.g. Evidence Based Medicine

Levels of evidence

1. Syst reviews of RCTs and high quality RCTs

2. Syst reviews of cohort studies, lower quality RCTs, outcomes research

3. Syst reviews of case controls, case control studies

4. Case Series

5. Expert opinion www.cebm.net

Types of EvidenceQuestion Types

Type of Question Best EvidenceHealth care interventions: treatment, prevention

Quantitative: Systematic Review of RCTs or RCT

Harm or Etiology Quantitative: Observational Study - Cohort or Case Control

Prognosis Quantitative: Observational Study - Cohort, Case Control

Diagnosis or Assessment Quantitative: Comparison to Gold Standard

Economics Quantitative: Cost-effectiveness Study

Meaning Qualitative: case study, ethnography, grounded theory, phenomenologic approach

Key quality parameters

• Validity

• Reliability

• Importance

Validity

• InternalIs the study designed in such a way that I Can trust the findings?

• ExternalIs the study designed in such a way that ICan generalize the findings?

Reliability

If the study was conducted again,would the results be the same?

Usually interpreted as the accuracyof measurement.

Importance

What was the effect sizeor magnitude of effect?

Clinical vs. statistical significance.

Tools for Critical Appraisal

• Are the results valid?

• What are the results?

• Will the results help me in patient care?

EBM “simplified” approach:

V

I

A

Format of research reports

• Title• Authors and Institutions• Abstract & keywords• IMRAD

– Introduction - why did I start?– Methods - what did I do?– Results - what did I find?– Discussion - what does it mean?

• References

Check list for medical literature

1. Title2. Authors & Institutions3. Abstract: Structured? Informative? Abbreviation?4. Introduction: Length? Relevant ref? Objective?5. Methods:

– Design, time and place– Inclusion criteria – Exclusion criteria– Sample size, sampling method– Randomization technique– Intervention: masking?– Outcome measurement: blinding?

• Primary outcome: type of variable • Secondary outcome: type of variable

– Analysis: Clinical, statistical

6. Results– Baseline characteristics– Main outcome– Secondary outcome

7. Discussion– General– Strength and weakness– Conclusions

8. References– Vancouver style– Constant

9. Acknowledgments10. Ethics approval11. Conflict of interest

Check list for medical literature

What to assess?(in study of cause-effect

relationship)

A. General description• Type of design• Target population, source

population, sample• Sampling method• Dependent and independent

variables• Main results?

B. Internal validity, non-causal relationship– Influence of bias– Influence of chance– Influence of confounders

What to assess?(in study of cause-effect

relationship)

Bias

What is a bias? A process that tends to produce results that depart systematically from the true values existing in the study population

Types of bias1. Sample (subject selection) biases, which may

result in the subjects in the sample being unrepresentative of the population which you are interested in

2. Measurement (detection) biases, which include issues related to how the outcome of interest was measured

3. Intervention (performance) biases, which involve how the treatment itself was carried out.

C. Internal validity, causal relationship• Temporality (cause precedes effect)• Strength of association (large difference, RR, OR, etc)

or small p value or narrow confidence interval• Biological gradient (dose dependence)• Consistency among studies (diff. populations or

designs)• Specificity (certain factor results in certain effect)• Coherence (does not conflict with current knowledge)• Biological plausibility: can be explained with current

knowledge (at least in part)

What to assess?(in study of cause-effect

relationship)

D. External validity

– Applicable to study subjects– Applicable to source population– Applicable to target population

What to assess?(in study of cause-effect

relationship)

11 items, each with 3 sections

1. Can you find this information in the paper?

2. Is the way this was done a problem?

3. Does this problem threaten the validity of the study?

11 items1. What is the research question?2. What is the study type?3. What are the outcome factors and how are they measured?4. What are the study factors and how are they measured?5. What important confounders are considered?6. What are the sampling frame and sampling method?7. In an experimental study, how were the subjects assigned

to groups? In a longitudinal study, how many reached final follow-up? In a case control study, are the controls appropriate? (Etc)

8. Are statistical tests considered?9. Are the results clinically/socially significant?10. Is the study ethical? 11. What conclusions did the authors reach about the

question?

1. What is the research question?

• (Is the way this was done a problem?)

– Is it concerned with the impact of an intervention, causality or determining the magnitude of a health problem?

• (Does this problem threaten the validity of the study?)

– Is it a well stated research question/hypothesis?

2. What is the study type?

• (Is the way this was done a problem?)

– Is the study type appropriate to the research question?

• (Does this problem threaten the validity of the study?)

– If not, how useful are the results produced by this type of study?

3. What are the outcome factors and how are they measured?

• (Is the way this was done a problem?)– a) are all relevant outcomes assessed– b) is there measurement error?

• (Does this problem threaten the validity of the study?)– a) how important are omitted

outcomes– b) is measurement error an important

source of bias?

4. What are the study factors and how are the measured?

• (Is the way this was done a problem?)– Is there measurement error?

• (Does this problem threaten the validity of the study?)– Is measurement error an important

source of bias?

5. What important potential confounders are considered?

• (Is the way this was done a problem?)– Are potential confounders examined and

controlled for?

• (Does this problem threaten the validity of the study?)– Is confounding an important source of bias?

6. What are the sampling frame and sampling method?

• (Is the way this was done a problem?)– Is there selection bias?

• (Does this problem threaten the validity of the study?)– Does this threaten the external

validity of the study?

7. Questions of internal validity

• (Is the way this was done a problem?)– In an experimental study, how were the

subjects assigned to groups?– In a longitudinal study, how many

reached follow-up?– In a case control study, are the controls

appropriate?• of the study?

8. Are statistical tests considered?

• (Is the way this was done a problem?)– Were the tests appropriate for the

data?– Are confidence intervals given?– Is the power given if a null result?– In a trial, are results presented as

absolute risk reduction as well as relative risk reduction?

• How useful are the results?

9. Are the results clinically/socially significant?

• (Is the way this was done a problem?)– Was the sample size adequate to detect

a clinically/socially significant result?– Are the results presented in a way to

help in health policy decisions?• (Does this problem threaten the validity of

the study?)– Is the study useful?

10. Are ethical issues considered?

• (Is the way this was done a problem?)– Does the paper indicate ethics approval?– Can you identify potential ethical issues?

• (Does this problem threaten the validity of the study?)– Are the results or their application

compromised?

11. What conclusions did the authors reach about the study

question?

• (Is the way this was done a problem?)– Do the results apply to the population in

which you are interested?

• (Does this problem threaten the validity of the study?)– Will you use the results of the study?

Appraisal Tools

• Tools from the Critical Appraisal Skills Programme (CASP)– Systematic Reviews– Randomised Controlled Trials– Qualitative Research Studies– Cohort Studies– Case-Control Studies– Diagnostic Test Studies– Economic Evaluation StudiesAvailable at

http://www.phru.nhs.uk/casp/critical_appraisal_tools.htm

Study Designs Recap

Effectiveness of Therapy

Risk Factors / Prognosis

Diagnosis

Attitudes & Beliefs

Randomised Controlled Trial

Cohort Study

Survey using Gold Standard

Qualitative (Interviews, Observations, etc)

Critical appraisal

- Valid- Important- Applicable

Methods Results Discussion

50 massa pemuda dan mahasiswa berunjuk rasa memprotes pelecehan seksual yang makin marak di depan Mabes PolriMenyadari pentingnya Panduan Pelayanan Medis (PPM), dibentuklah Panitia Penyusunan SPM di RSCM

Sekuens dan hubungan subyek-predikat

• Dalam pertemuan ilmiah yang diselenggarakan setiap tahun yang merupakan ajang untuk menyajikan perkembangan mutakhir dalam bidang ilmu penyakit dalam di tanah air tersebut menyimpulkan bahwa pertemuan tersebut disamping dilakukan oleh Universitas selayaknya juga dilakukan oleh cabang-cabang PAPDI di setiap propinsi, bahkan kalau mungkin disetiap kabupaten

Plagiarisme adalah tindakan yang dapat diartikan sebagai pencurian ide atau hasil pemikiran dan tulisan orang lain yang digunakan dalam tulisan seolah-olah ide atau tulisan orang lain tersebut adalah ide atau hasil tulisannya sendiri untuk keuntungannya sendiri sehingga merugikan orang lain baik materiil maupun non-materiil, atau plagiarisme dapat berupa pencurian sebuah kata, frase, kalimat, atau alinea, atau bahkan pencurian suatu bab dari sebuah tulisan atau buku seseorang, tanpa menyebut sumber yang dicuri. (Draft SK Rektor UI)

RCT (Pragmatic trials): Validity Were the study participants randomized?Was the randomization technique described?Was the randomization table concealed?Were the characteristics of the subjects similar at the start of the intervention?Were all participants given equal treatment apart from the intervention?Were all relevant outcomes considered?Were the results analyzed correctly?

RCT (Pragmatic trials): Importance

• Calculate: EER, CER, RRR, ARR, NNT

a b

c d

E

C

Success Failure

EER = a/(a+b)CER = c/(c+d)

RRR = (CER-EER)/CERARR = CER-EERNNT = 1/ARR

RCT: Applicability• Were the participations similar to your

patients?• May be intuitively concluded or use f

(factor indicating how much severe your patient compared to the study participation in terms of prognostic factor)

DIAGNOSTIC TEST: Validity

Was independent and blind comparison to gold standard applied?

Was the diagnostic test include spectrum of disease similar to your real practice?

Was the gold standard applied regardless of the diagnostic result?

Diagnostic Test: Importance

• Calculate: Sensitivity, specificity, predicitive values, likelihood ratios

a b

c d

+

-

+ -

Se = a/(a+c)Sp = d/(b+d)PPV = a/(a+b)NPV = d/(c+d)

LR+ = se/(1-sp)LR - = (1-se) /spPosttest odds = Pretest odds x LR+

Test

Diagnostic Test: Applicability

Were the participations similar to your patients?Is the diagnostic test applicable, acceptable, and affordable in your setting?Will the result of the test help your patient?

Cohort Studies: Validity

Was the inception cohort assembled in usual point of course of the disease?Was the follow-up sufficient & complete?Were outcome criteria applied in blind fashion?Was there any validation in other group of patients?Was subgroup analysis performed after adjustment for prognostic factors?

Cohort studies: Importance

Calculate: Relative risk (RR) and 95% CI

a b

c d

+

-

+ -

Exposure

RR = a/(a+b) : c/(c+d)RR > 1: risk factorRR < 1 : protective factorRR = 1 : not a risk factor

Cohort studies: Applicability

Were the patients similar to yours?Wil the evidence make a clinically important impact for the care of your patient?

Case control study: Validity

Was the selection of patients represent the common patients of yours? (Spectrum of the patients)Was the selection of control group sensible and reasonable?Have sufficient attempts been made to minimize recall bias?

Case control studies: Importance

• Calculate: Odds ratio (OR) and 95%CI

a b

c d

+

-

+ -

Exposure

OR = a/b : c/dOR > 1 : risk factorOR < 1 : protective factorOR = 1 : not a risk factor

Case-control studies: Applicability

• Were the patients similar to yours?• Wil the evidence make a clinically

important impact for overall care (diagnosis, treatment, prognosis) of your future patients?b

From Data to Wisdom

• Data are what researchers collect• Information results when data is

analyzed and interpreted (EVIDENCE)• Knowledge results when information

is shared, acquired, and used• Wisdom is the ability to make the

right use of knowledge

THANKS