Quality Education for a Healthier Scotland Pharmacy Critical Appraisal Skills Critical Appraisal Skills Basic I Basic I NHS Education Scotland Produced in collaboration with the Association of Scottish Medicines Information Pharmacists Group
Jan 26, 2015
Quality Education for a Healthier Scotland
Pharmacy
Critical Appraisal SkillsCritical Appraisal SkillsBasic IBasic I
NHS Education ScotlandProduced in collaboration with the
Association of Scottish Medicines Information
Pharmacists Group
Quality Education for a Healthier Scotland
Pharmacy
What is critical appraisal?
• This is the term given to describe the skills used when reading a paper to enable one to assess the validity (i.e. how close to truth) and usefulness (i.e. can the results be applied to your practice) of the results.
• Forms an integral part of evidence based medicine (EBM).
Quality Education for a Healthier Scotland
PharmacyWhat is EBM?
• EBM is the judicious use of current best evidence, combined with clinical experience, to make decisions about patient care.
Quality Education for a Healthier Scotland
Pharmacy
5 key steps that underpin EBM
• Define the specific question to be answered.• Find the best evidence to answer the question.• Critically evaluate the evidence to assess it validity
and usefulness.• Apply the results of the critical evaluation to practice.• Evaluate the performance of the intervention.
Quality Education for a Healthier Scotland
PharmacyWhy do we need evidence?
• Resources should only be allocated to interventions that are effective.
• The only way of judging effectiveness is EVIDENCE!
Quality Education for a Healthier Scotland
PharmacyWhat are “good” sources of evidence?
Less reliable sources
Glossy literature from pharmaceutical companies
Press releases from pharmaceutical companies
Magazines such as Pulse
Advertisements in medical journals
Conference abstracts about clinical trials
Trusted sources
Scottish Medicines Consortium
SIGN
NICE
National Library for Health
Peer reviewed clinical journals
Summaries of information published by NHS bodies (e.g. National Prescribing Centre)
Quality Education for a Healthier Scotland
PharmacyHierarchy of evidence
• Evidence comes in different forms and can be ranked in terms of importance.
• Quite often there may not be any high-levels of evidence to support a clinical intervention. In these cases it may be necessary to use evidence from the lower end of the hierarchy scale.
Quality Education for a Healthier Scotland
Pharmacy
Expert opinion/ clinical experience from respected sources
Case series then Case reports
Cross sectional surveys
Case control studies
Cohort studies
RCTs
Systematic reviewsMeta-analyses
Hierarchy of evidence
Quality Education for a Healthier Scotland
PharmacyFormat of clinical trials
Clinical trials are usually written in a standard format. This normally consists of:
» Title» Authors» Abstract – contains a brief summary of the trial.» Introduction» Methods» Results» Discussion» Conclusion
Quality Education for a Healthier Scotland
PharmacyKey questions to ask when assessing how valid
and useful a clinical trial is
• Patient or population – does the trial assess a commonly seen clinical condition and were the patients studied similar to your patient?
• Intervention – what was the medicine being tested? Was it used at the normal dose?
• Comparison – What was the experimental medicine compared to? Trials that compare a new medicine to a placebo are not useful when trying to decide where the medicine fits into current practice.
• Outcome – Was the end-point relevant to the patient (i.e. was it patient-oriented like a reduction in risk for a having a heart attack)?
Quality Education for a Healthier Scotland
PharmacyOutcomes
Patient orientated
Outcomes that directly improve the outcome for patients.
e.g. Reduction in hip fracture; improved cardiovascular mortality; prevention of a stroke
Disease orientated
Outcomes that are the result of a change to a disease characteristic
e.g. Change in bone mineral density; lowering of blood pressure; a reduction in cholesterol
Quality Education for a Healthier Scotland
PharmacySurrogate outcomes
• Outcomes that are not patient-orientated are surrogate outcomes.
• Often physiological or biochemical markers
• Cannot always assume that they are always a good indication of disease progression or improved survival.
– For example a modest decrease in blood pressure may not be clinically relevant even though the result may be statistically significant.
Quality Education for a Healthier Scotland
PharmacyThree key factors affecting the
results of a trial
The intervention
Bias – researchers take steps to minimise by use of a control group, randomisation and blinding but some biases can still exist.
Chance – statistical tests are used to assess this.
Quality Education for a Healthier Scotland
PharmacyTypes of Statistics
Descriptive
Summarises or describes the sample
[Please note that for purposes of critical appraisal this type will not be discussed in this training package].
Inferential
Concerned with generalising from the sample to make inferences and estimates about a wider population.
Quality Education for a Healthier Scotland
Pharmacy
Inferences and Estimates
Inferences
Can conclusions be drawn from the sample be generalised to the population?
e.g. If a better response is seen with a medicine in a sample will it hold true in the population
Help answer whether results may have occurred by chance in the trial.
Estimates
Given an observed size of effect in the sample, what is the likely value (or range of values) you will see in the population?
Help assess usefulness of a trial.
Quality Education for a Healthier Scotland
Pharmacy
Inferential Statistics
Quality Education for a Healthier Scotland
Pharmacy
• There are 1 of 2 assumptions made for interventions in clinical trials:– Null hypothesis (i.e. no difference between the control group
and the experimental group).– Alternative hypothesis (i.e. there is a difference between the
control group and the experimental group).• Generally it is the null hypothesis that is assumed however.
Hypothesis Testing
Quality Education for a Healthier Scotland
Pharmacy
• The probability that a difference will be seen between 2 interventions in a clinical trial.
• Measured on a scale of 0 (impossible for event to happen) to 1 (the event will certainly happen)
• i.e. P = 1 would always happen• P = 0.05 would happen 1 time in 20• P = 0.02 would happen 1 time in 50• P = 0.01 would happen 1 time in 100• P = 0.001 would happen 1 time in 1000
Probability (P)
Quality Education for a Healthier Scotland
Pharmacy
• If p-value is less than 1 in 20 (p<0.05) then the result is regarded as being statistically significant; and the possibility of the difference observed arising by chance is low
=> If this is the case then one can reject
the null Hypothesis
P values
Quality Education for a Healthier Scotland
Pharmacy
• The probability that a test will detect a real difference in treatment outcomes in a sample if it is present in the population
• Usually expressed as a percentage and often set at 80-90%
Power
Quality Education for a Healthier Scotland
Pharmacy
Power and Sample Size
Sample size determinants:
Size of the difference being investigated
Level of significance
Quality Education for a Healthier Scotland
Pharmacy
Errors that can arise when drawing conclusions from data
• Type 1 error (alpha)• The data suggests a difference between the
groups when there is really no difference = False positive
• Often called significance level• A level of significance of p<0.05 represents
a 5% probability of making a type 1 error
Quality Education for a Healthier Scotland
PharmacyErrors that can arise when drawing conclusions from data
Type 2 error (beta)• The results fail to pick up a real difference that exists between the groups, and a conclusion is made that no difference exists = False negative• 100-(power)% is the probability of making a type 2 error
Quality Education for a Healthier Scotland
Pharmacy
Population Estimates
Quality Education for a Healthier Scotland
Pharmacy
• A tool for inferring the characteristics / parameters of a whole population from the measurements in one sample
• One of the most widely misused terms in statistics
• In 95% of cases the ‘True Population Mean’ will lie within +/-2 SEM of the sample mean
• Should NEVER be used instead of SD to indicate dispersion of measurements
SEM = SD
------
n
Standard Error of the Mean (SEM)
Quality Education for a Healthier Scotland
Pharmacy
• Represents the range of values within which the true population mean lies.
• Indicate the precision (or imprecision) with which a study sample estimates the true population value for the whole population.
• Important role whenever we wish to apply results of a clinical study to the general population
• Narrower the range the more reliable the results
Confidence Intervals (CI)
Quality Education for a Healthier Scotland
PharmacyConfidence Intervals
Calculated by adding and subtracting multiples of the standard error of the mean to and from the sample mean
95% confidence interval normally used (i.e. can be 95% confident that the population value lies within this interval); or alternatively stated that there is a 1 in 20 chance (5%) that the true value lies outside the range quoted.
The narrower the CI, the more confident you can be the sample represents the population
Quality Education for a Healthier Scotland
Pharmacy
Comparing means:No difference if CI overlap (i.e. even though 2 mean values may
differ, extensive overlap of their respective CIs may suggest that the difference is not statistically significant)
When comparing differences between means:No difference if CI includes 0
For proportions (e.g. RR):No differences if CI includes 1
For further information on this topic:-Statistics in divided doses: Number 8 (July 2005).Produced by the
North West Medicines Information Service. Available at http://www.ukmi.nhs.uk/filestore/misc/StatsinDivDose8.pdf
How to Interpret?
Quality Education for a Healthier Scotland
Pharmacy
Estimation Statistics
Help assess “usefulness of the trial” by determining clinical importance and magnitude of the benefit by using data to estimate a range of probable values for the population.
Quality Education for a Healthier Scotland
Pharmacy
Example study
Group Total Number of patients in each group
Pain free within 2 hours
Intervention Group (Received Drug X)
2073 845
Control Group (Received Placebo)
1128 96
Quality Education for a Healthier Scotland
Pharmacy
Definition:
The proportion of patients in whom an event is observed
Control Event Rate (CER)
Vs
Experimental Event Rate (EER)
Event Rates
Quality Education for a Healthier Scotland
Pharmacy
Control Event Rate (CER) =
Event Rate in control group
Total patients in control group
Example
CER = 96/1128 = 0.085 (9%)
Control Event Rate
Quality Education for a Healthier Scotland
Pharmacy
Experimental Event Rate (EER) =
Event Rate in experimental group
Total patients in experimental group
Example
EER = 845/2073 = 0.41 (41%)
Experimental Event Rate
Quality Education for a Healthier Scotland
Pharmacy
Absolute Risk Reduction (ARR) is way of expressing differences between groups.
It is the difference in the event rate between the control event rate (CER) and the experimental event rate (EER).
ARR = CER-EER
Example
ARR = 9-41 = -32
Absolute Risk Reduction
Quality Education for a Healthier Scotland
Pharmacy
• Is an alternative means of expressing the difference between groups as a percentage• The Relative Risk Reduction (RRR) is the percent reduction in events in the experimental event rate (EER) and the control
event rate (CER).
RRR = (CER-EER) X 100
CER
Example
RRR = (9-41/9) X 100 = 356%
Relative Risk Reduction
Quality Education for a Healthier Scotland
Pharmacy
Why calculate?
Sometimes the trial may just state “the treatment reduced the risk” but does not state whether this is relative risk reduction or absolute risk reduction. Obviously the relative risk reduction looks more impressive since a larger number. Be aware of this and use the figures given to calculate both.
Neither RRR or ARR are intuitive ways to look at data. Numbers needed to Treat (NNT) is the more relevant way to look at the figures.
Quality Education for a Healthier Scotland
Pharmacy
Definition:
The number of people who needed to be treated to produce one particular clinical outcome
(e.g. How many patients need to receive Drug X instead of placebo to allow one patient to be pain free at 2 hours?)
NNT = ____1____ or ____1____
CER-EER ARR
Example
NNT = 1/32 = 3
Numbers Needed to Treat (NNT)
Quality Education for a Healthier Scotland
Pharmacy
Numbers Needed to Harm (NNH)
• This value can be similarly calculated when looking at adverse effects in a clinical trial.• It is the number of patients you would need to treat with the
experimental medicine (rather than the placebo or control) for one additional patient to suffer an adverse effect.
» NNH = 1/(EER-CER)
Quality Education for a Healthier Scotland
PharmacyExample NNH
Medicine Y is given to patients for treatment of hypertension. 12 of the 4000 patients given medicine Y experience a rash compared with 2 out of 3000 given placebo.
CER = 2/3000 = 0.00066
EER = 12/4000 = 0.003
NNH = 1/ (0.003-0.00066) = 428
Therefore 428 patients must be treated with medicine Y
rather than placebo for an additional 1 patient to have an
adverse effect.
Quality Education for a Healthier Scotland
PharmacyRelative Risk
The relative risk (RR) is the size of the effect in the experimental group relative to the size of the effect in the control group. The relative risk is often quoted in a clinical trial paper.
RR = CER/EER
Example RR= 9/41 = 0.21
A relative risk of 1.0 means that there is no difference between the experimental and control groups. This result shows a RR < 1.0 indicating that the patients on the medication are more likely to be pain free at 2 hours than those receiving placebo.
Quality Education for a Healthier Scotland
Pharmacy
The ratio of patients in the treatment group succumbing to a particular end point compared to the control group
Compares the probability of the event occurring with the probability that it will not occur.
If >1 = event more likely to happen
If <1 = event less likely to happen
Odds Ratio (OR)
Quality Education for a Healthier Scotland
Pharmacy
The odds ratio must be calculated first for control and treatment group:
Control = 1032/1128 = 0.915
Treatment = 1228/2073 = 0.592
Odds Ratio = treatment / control = 0.592/0.915 = 0.647
In isolation it is difficult to imagine what this figure means but the smaller the odds ratio the more effective Drug X is in allowing a patient to be pain free within 2 hours.
Odds Ratio (II)
Quality Education for a Healthier Scotland
PharmacySummary
• Don’t always believe everything you read!
• Choose your source of evidence wisely and systematically to answer your question.
• Use estimation statistics to help evaluate usefulness and clinical importance of a trial.
• Utilise population estimates to determine how reflective of the true population the trial results are likely to be.
• Statistical significance does not always equate to clinical significance.• There is a lot of information out available but you have to choose the best
evidence available. Remember that all evidence is not equal!
Quality Education for a Healthier Scotland
PharmacyReferences
Brignell J. How do Relative Risk and Odds Ratio compare? (April 2006). Available at http://www.numberwatch.co.uk/rr&or.htm
Burls A. What is Critical Appraisal in Evidenced Based Medicine 2nd ed. Oxford: University of Oxford. Available at www.whatisseries.co.uk
DeCaro, S. A. A student's guide to the conceptual side of inferential statistics (2003). Available from http://psychology.sdecnet.com/stathelp.htm.
Easton V, McColl JH. Confidence Intervals in Statistics Glossary V1.1. Available from http://www.stats.gla.ac.uk/steps/glossary/confidence_intervals.html#confinterval
Greenhalgh T. How to read a paper: The basics of evidence based medicine 2nd edition. London: BNJ Books 2001.
Swinscow TDV, Campbell MJ. Statistics at Square One 10th edition. London: British Medical Association 2002. Statistics in Divided Doses, Assessing the reliability of a sample (Number 3). North West Medicines Information
Service (September 2001). Available at http://www.ukmi.nhs.uk/filestore/misc/StatsinDivDose3.pdf Statistics in Divided Doses, Variability, probability and power (Number 4). North West Medicines Information
Service (May 2002). Available at http://www.ukmi.nhs.uk/filestore/misc/StatsinDivDose4.pdf Statistics in Divided Doses, First steps in analysis - comparing the means of large samples (Number 5). North
West Medicines Information Service (November 2002). Available at http://www.ukmi.nhs.uk/filestore/misc/StatsinDivDose5.pdfStatistics in Divided Doses, Confidence intervals (Number 8). North West Medicines Information Service (July
2005). Available at http://www.ukmi.nhs.uk/filestore/misc/StatsinDivDose8.pdf Wills S et al. Critical Appraisal of Clinical Trials E-learning Module via NHS Education South Central. Available
for free registration at http://www.learning.nesc.nhs.uk/