Adjusting for Publication Bias in Meta-Analysis: An Evaluation of Selection Methods and Some Cautionary Notes

Published on Sep 1, 2016in Perspectives on Psychological Science8.19
· DOI :10.1177/1745691616662243
Blakeley B. McShane16
Estimated H-index: 16
(NU: Northwestern University),
Ulf Böckenholt32
Estimated H-index: 32
(NU: Northwestern University),
Karsten T. Hansen15
Estimated H-index: 15
(UCSD: University of California, San Diego)
View in Source
We review and evaluate selection methods, a prominent class of techniques first proposed by Hedges (1984) that assess and adjust for publication bias in meta-analysis, via an extensive simulation study. Our simulation covers both restrictive settings as well as more realistic settings and proceeds across multiple metrics that assess different aspects of model performance. This evaluation is timely in light of two recently proposed approaches, the so-called p-curve and p-uniform approaches, that can be viewed as alternative implementations of the original Hedges selection method approach. We find that the p-curve and p-uniform approaches perform reasonably well but not as well as the original Hedges approach in the restrictive setting for which all three were designed. We also find they perform poorly in more realistic settings, whereas variants of the Hedges approach perform well. We conclude by urging caution in the application of selection methods: Given the idealistic model assumptions underlying selec...
  • References (43)
  • Citations (63)
#1Robbie C. M. van Aert (Tilburg University)H-Index: 8
#2Jelte M. Wicherts (Tilburg University)H-Index: 39
Last.M.A.L.M. van Assen (UU: Utrecht University)H-Index: 25
view all 3 authors...
46 CitationsSource
#1Blakeley B. McShaneH-Index: 16
#2David Gal (UIC: University of Illinois at Chicago)H-Index: 18
24 CitationsSource
10 CitationsSource
#1Andrew Gelman (Columbia University)H-Index: 79
45 CitationsSource
#1Zhichao Jin ('SMMU': Second Military Medical University)H-Index: 12
#2Xiao-Hua Zhou (UW: University of Washington)H-Index: 46
Last.Jia He ('SMMU': Second Military Medical University)H-Index: 28
view all 3 authors...
59 CitationsSource
#1M.A.L.M. van Assen (Tilburg University)H-Index: 25
#2Robbie C. M. van Aert (Tilburg University)H-Index: 8
Last.Jelte M. Wicherts (Tilburg University)H-Index: 39
view all 3 authors...
69 CitationsSource
#1Uri Simonsohn (UPenn: University of Pennsylvania)H-Index: 26
#2Leif D. Nelson (University of California, Berkeley)H-Index: 26
Last.Joseph P. Simmons (UPenn: University of Pennsylvania)H-Index: 24
view all 3 authors...
181 CitationsSource
#1Blakeley B. McShane (NU: Northwestern University)H-Index: 16
#2Ulf Böckenholt (NU: Northwestern University)H-Index: 32
41 CitationsSource
#1Richard A. Klein (UF: University of Florida)H-Index: 7
#2Kate A. Ratliff (UF: University of Florida)H-Index: 13
Last.Brian A. Nosek (UVA: University of Virginia)H-Index: 61
view all 51 authors...
337 CitationsSource
#1Tommy Stanley (Hendrix College)H-Index: 32
#2Hristos Doucouliagos (Deakin University)H-Index: 30
191 CitationsSource
Cited By63
#1Jordane Boudesseul (University of Lima)H-Index: 3
#2Kelly A. Gildersleeve (UM: University of Michigan)H-Index: 11
Last.Laurent BègueH-Index: 20
view all 4 authors...
#1Claire Noble (University of Liverpool)H-Index: 5
#2Giovanni Sala (Osaka University)H-Index: 9
Last.Julian M. Pine (University of Liverpool)H-Index: 33
view all 7 authors...
#1Andrew W. Brown (IU: Indiana University)H-Index: 20
#2Douglas G. Altman (University of Oxford)H-Index: 209
Last.David B. Allison (IU: Indiana University)H-Index: 114
view all 18 authors...
#1Matthew J. Page (Alfred Hospital)H-Index: 24
#2Julian Pt Higgins (UoB: University of Bristol)H-Index: 3
Last.Jonathan A C Sterne (UoB: University of Bristol)H-Index: 91
view all 3 authors...
1 CitationsSource
#1Ming Fang (ZJU: Zhejiang University)H-Index: 1
#2Guilin Liu (ZJU: Zhejiang University)H-Index: 1
Last.Tianyu Wu (ZJU: Zhejiang University)H-Index: 1
view all 4 authors...
#1Stefan Wiens (Stockholm University)H-Index: 24
#2Erik van Berlekom (Stockholm University)H-Index: 1
Last.Rasmus Eklund (Stockholm University)H-Index: 3
view all 4 authors...
#1Eran Amsalem (HUJI: Hebrew University of Jerusalem)H-Index: 2
#2Lilach Nir (HUJI: Hebrew University of Jerusalem)H-Index: 12
#1Malte Elson (RUB: Ruhr University Bochum)H-Index: 11
#1Chiungjung Huang (NCUE: National Changhua University of Education)H-Index: 12
1 CitationsSource