Evidence Based Medicine Meta-analysis and systematic reviews Ross Lawrenson.

Evidence Based Medicine

Meta-analysis and systematic reviews

Ross Lawrenson

A systematic review

• This is a systematic collection of the relevant primary papers in human populations that deal with a focused question and includes a summary of the evidence from the primary sources.

A systematic review• This is a systematic collection of the relevant primary

papers in human populations that deal with a focused question and includes a summary of the evidence from the primary sources.

• This is in contrast to an overview which often tackles a whole topic rather than a focused question, may include opinion based articles and whose summary may be influenced by clinical experience or evidence from other sources such as animal experiments.

Meta-analysis

• A particular type of systematic review that uses quantitative methods to combine the results from a number of studies

Meta-analysis

• Meta analysis is a statistical anlysis which combines the results of several independent studies considered by the analyst to be combinable.

• Huque 1988

Meta-analysis• A particular type of systematic review that uses

quantitative methods to combine the results from a number of studies

• Since meta-analysis is a retrospective look at data, it is important to make the process rigorous and well defined to prevent opportunities for bias to distort the results. Only in this way can it achieve the status of a scientific discipline. This necessitates blinding the selection of papers, extraction of data and quality assessment in duplicate following an established protocol at the start of the study.

Critically appraising an overview paper

Critical Appraisal of an overview - methodology

• When critically appraising a paper ask yourself three questions:– Are the results valid?– What are the results?– Will the results help me in caring for my

patients?

• Go to the overview worksheet for the complete list of questions

I. Are the results valid?

• In other words was this thorough and rigorously conducted overview?

• Go through the worksheet questions 1- 6 to help you decide whether you are likely to believe the results of the paper you are considering.

Are the results valid?

• Did the overview address a clearly focused question?

Are the results valid?

• Did the overview address a clearly focused question?

• Is there a defined population, a clearly identified treatment or exposure and one or more clearly defined outcomes?

Are the results valid?

• Were the criteria used to select articles for inclusion appropriate?

Are the results valid?• Were the criteria used to select articles for inclusion

appropriate?• The article should clearly set out the criteria for inclusion of

studies. This usually includes the type of study (for therapy questions ideally an RCT), the populations included in the studies, treatments or exposures and relevant outcomes.

• The criteria should not be so restrictive that important studies are likely to be missed. Ideally the authors should list all trials reviewed with a reject log and reasons for exclusion

Are the results valid?

• Is it likely that important, relevant studies were missed?

Are the results valid?

• Is it likely that important, relevant studies were missed?

• What was the search strategy used in identifying papers. Is it clearly stated. What databases were searched? Was the grey literature searched? What about secondary sources? Was it a wide (sensitive) search?

Is it likely that important, relevant studies were missed?

• Sensitivity of search– The search strategy

– Sources used

– Foreign language papers

– Grey and unpublished

The search strategy• This should be very sensitive.• Ideally the paper will include in the methods section the

words that were used in the interrogation of the Medical Databases and which terms were combined and which intersected. (e.g. When looking for randomised controlled trials the terms “random allocation, randomised, double blind and controlled” should be used.)

• The time period over which papers could be included should also be included.

Sources used

• All the sources used should be listed. These could include Medline, Cochrane, Embase, BIDS, Cinahl, etc.

• This should also include secondary references I.e. the references cited by the papers that were retrieved from Medline etc.

Foreign language papers

• Many well conducted studies are published in languages other than English. Researchers undertaking systematic reviews are often tempted to exclude these papers because of the cost of having them translated (as well as the delay). However this means that some perfectly valid studies that deal with the question are not included.

• This can introduce bias

Grey Literature• A comprehensive overview should also look for grey

and unpublished literature. Studies with negative findings may be difficult to get published. If only studies with positive results are published then a meta-analysis of the published papers will give a positive result. Suggest contacting researchers known to be active in the field of interest. Also drug companies often have unpublished studies which they are now releasing to Cochrane and other interested parties.

Control and Measurement of Bias.

• Data extraction bias

• Selection bias

• Source of support

Data extraction bias

• To ensure that the correct papers have been included, and that bias in the selection of papers is minimised, duplicate extraction is recommended. Ideally this should occur blinded. I.e. The reviewers should be blinded to the authors, the location of study, the journal and the finding. This can best be achieved by copying only the methods section of the papers being selected.

Selection bias• One way of looking for selection bias is to

construct a funnel plot where the largest studies (with the most precision) are close to the top and smaller studies are nearer the X axis. If there is no bias then the studies selected should be symmetrically distributed around the results from the larger studies which should be closer to the true result and less influenced by random error.

Include diagrams of four funnel plots - two with bias and two that are symmetrical.

Sources of support

• The funding of the original studies should be mentioned to allow the reader to make conclusions as to any conflict of interest although not to disqualify a study from inclusion.