What we work on
Assessing the relevance of a metaanalysis or any other systematic review, is no trivial task. Ultimately, its quality depends on the reliability of the included studies and a reasonable selection of studies for analysis. However, there are some fundamental points to consider, which at least give some idea about the quality of a metaanalysis:
Planning:
Were the following issues determined in advance?
• Procedure for study selection, e.g. according to the PRISMA guidelines: Prisma statement
• Research question – one who searches randomly, will eventually find something
• Consistent criteria for inclusion and exclusion of studies to analyse
Methods:
• Exact description of the literature search: which databases were used, who did the search and who selected the studies?
• Study selection performed transparently, systematically and adequately?
• Description of the statistical analysis?
• Heterogeneity between studies tested and considered?
• Results of the studies combined in a methodically adequate way?
For example, if pain is measured in different studies using different continuous scales, it cannot be compared directly. Instead, for each study the standardized mean difference needs to be calculated (mean difference between the groups divided by the mean deviation). This standardized mean difference (SMD) is independent of the original scale and allows comparison between different studies.
Results:
• Differences in effect clinically relevant? A difference in blood loss of 70 ml may be significant statistically, but clinically irrelevant for adults.
• Does the result of a single study influence the total result of the metaanalysis significantly because of its sample size? Has it been checked how the overall result changes if this study is excluded?
Discussion:
• Discussion of the limitations of the metaanalysis? Do the authors explain how these limitations limit the statement of the study?
• Are the conclusions of the authors based on the presented results?
If several questions have to be negated, the results of the metaanalysis should be taken with a pinch of salt. However, if adequately done, a metaanalysis may be very revealing, e.g. if there are many very small studies regarding a specific question, which do not reach statistical significance by themselves, but may in combination indeed be of great relevance.
Author: Dr. Miriam Imo