Bias in meta-analysis detected by a simple, graphical test. Prospectively identified trials could be used for comparison with meta-analyses.

P Langhorne - BMJ: British Medical Journal, 1998 - ncbi.nlm.nih.gov
BMJ: British Medical Journal, 1998ncbi.nlm.nih.gov
Editor—Egger et al report that they “found bias in 38% of meta-analyses published in four
leading journals.” 1 This is misleading, at least insofar as our meta-analysis of inpatient
geriatric consultations is concerned. 2 Firstly, the bias observed in our metaanalysis was not
a retrospective detection of bias, as one might infer from Egger et al's statements. We knew
that there was evidence of heterogeneity for the pooled effect estimates of geriatric
consultation programmes and reported this finding. 2 Secondly, the asymmetry detected in …
Editor—Egger et al report that they “found bias in 38% of meta-analyses published in four leading journals.” 1 This is misleading, at least insofar as our meta-analysis of inpatient geriatric consultations is concerned. 2 Firstly, the bias observed in our metaanalysis was not a retrospective detection of bias, as one might infer from Egger et al’s statements. We knew that there was evidence of heterogeneity for the pooled effect estimates of geriatric consultation programmes and reported this finding. 2 Secondly, the asymmetry detected in the funnel plot of the meta-analysis of inpatient consultation programmes was probably due not to bias (distortion of true effect) but to true heterogeneity (true difference of effects between trials). We took the presence of heterogeneity as an opportunity to examine whether we could identify the programme elements that might have resulted in the observed effect differences between geriatric consultation programmes. Using a multivariate logistic regression approach, we found that both geriatric assessment programmes in which the consultant controlled the implementation of the recommendations and those that included long term follow up resulted in better outcomes than did programmes in which this was not the case. Thus, the meta-analytical methods of testing heterogeneity or drawing funnel plots should not be considered absolute criteria for separating good from bad metaanalyses. Meta-analyses reporting effect estimates that may contain bias should continue to be published in leading medical journals, as long as the possibility of heterogeneity is stated and potential underlying reasons for heterogeneity are addressed. This is especially true for meta-analyses of complex interventions. Although they are methodologically difficult to deal with, variations in effect estimates give us the opportunity to disentangle the black box of complex interventions, such as of geriatric assessment, and identify what the necessary ingredients of these programmes are. 3 A third issue concerns the “mega-trial” to which our meta-analysis was being compared. 4 This trial was different from any of the trials considered in our meta-analysis. Among other things, it was based in a health maintenance organisation system that had incorporated considerable geriatric expertise into its usual care for older people. Another important factor was that it involved four hospital sites, each with different characteristics, populations, and survival rates. If Egger et al had taken the same pains as we did in recovering unpublished data from primary trials, they would have found that the mega-trial they used in questioning our meta-analysis was a multicentre trial with unreported variability in intervention components and outcomes across study sites. Analysts must consider rigorously any methodological issues unique to each trial, particularly when considering complex interventions.
ncbi.nlm.nih.gov