Match!

REVISITING THE FILE DRAWER PROBLEM IN META-ANALYSIS: AN ASSESSMENT OF PUBLISHED AND NONPUBLISHED CORRELATION MATRICES

Published on Jun 1, 2012in Personnel Psychology6.93
· DOI :10.1111/j.1744-6570.2012.01243.x
Dan R. Dalton59
Estimated H-index: 59
(IU: Indiana University Bloomington),
Herman Aguinis56
Estimated H-index: 56
(IU: Indiana University Bloomington)
+ 2 AuthorsCharles A. Pierce30
Estimated H-index: 30
(U of M: University of Memphis)
Abstract
The file drawer problem rests on the assumption that statistically non-significant results are less likely to be published in primary-level studies and less likely to be included in meta-analytic reviews, thereby resulting in upwardly biased meta-analytically derived effect sizes. We conducted 5 studies to assess the extent of the file drawer problem in nonexperimental research. In Study 1, we examined 37,970 correlations included in 403 matrices published in Academy of Management Journal (AMJ), Journal of Applied Psychology (JAP), and Personnel Psychology (PPsych) between 1985 and 2009 and found that 46.81% of those correlations are not statistically significant. In Study 2, we examined 6,935 correlations used as input in 51 meta-analyses published in AMJ, JAP, PPsych, and elsewhere between 1982 and 2009 and found that 44.31% of those correlations are not statistically significant. In Study 3, we examined 13,943 correlations reported in 167 matrices in nonpublished manuscripts and found that 45.45% of those correlations are not statistically significant. In Study 4, we examined 20,860 correlations reported in 217 matrices in doctoral dissertations and found that 50.78% of those correlations are not statistically significant. In Study 5, we compared the average magnitude of a sample of 1,002 correlations from Study 1 (published articles) versus 1,224 from Study 4 (dissertations) and found that they were virtually identical (i.e., .2270 and .2279, respectively). In sum, our 5 studies provide consistent empirical evidence that the file drawer problem does not produce an inflation bias and does not pose a serious threat to the validity of meta-analytically derived conclusions as is currently believed.
  • References (86)
  • Citations (69)
References86
Newest
#1Herman Aguinis (IU: Indiana University Bloomington)H-Index: 56
#2Ryan K. Gottfredson (IU: Indiana University Bloomington)H-Index: 12
Last.Thomas A. Wright (KSU: Kansas State University)H-Index: 33
view all 3 authors...
78 CitationsSource
#1Herman Aguinis (IU: Indiana University Bloomington)H-Index: 56
#2Charles A. Pierce (U of M: University of Memphis)H-Index: 30
Last.Catherine M. Dalton (IU: Indiana University Bloomington)H-Index: 16
view all 5 authors...
113 CitationsSource
34 CitationsSource
#1Herman Aguinis (IU: Indiana University)H-Index: 56
#2Brian K. Boyd (ASU: Arizona State University)H-Index: 28
Last.Jeremy C. Short (TTU: Texas Tech University)H-Index: 37
view all 4 authors...
114 CitationsSource
103 CitationsSource
#1Chad A. Hartnell (ASU: Arizona State University)H-Index: 9
#2Amy Y. Ou (ASU: Arizona State University)H-Index: 7
Last.Angelo J. Kinicki (ASU: Arizona State University)H-Index: 39
view all 3 authors...
329 CitationsSource
#1Winny Shen (UMN: University of Minnesota)H-Index: 13
#2Thomas Kiger (UMN: University of Minnesota)H-Index: 5
Last.Deniz S. OnesH-Index: 49
view all 6 authors...
62 CitationsSource
#1Herman Aguinis (IU: Indiana University Bloomington)H-Index: 56
#2Dan R. Dalton (IU: Indiana University Bloomington)H-Index: 59
Last.Catherine M. Dalton (IU: Indiana University Bloomington)H-Index: 16
view all 5 authors...
121 CitationsSource
#1Guido Schwarzer (University Medical Center Freiburg)H-Index: 37
#2James R. Carpenter (Lond: University of London)H-Index: 41
Last.Gerta Rücker (University Medical Center Freiburg)H-Index: 33
view all 3 authors...
38 CitationsSource
#1Harris Cooper (Duke University)H-Index: 64
#2Larry V. HedgesH-Index: 73
31 Citations
Cited By69
Newest
#1Matt C. Howard (USA: University of South Alabama)H-Index: 8
#2Melissa B. Gutworth (MSU: Montclair State University)H-Index: 3
Source
Source
#1Heather R. Parola (University of Evansville)H-Index: 3
#2Michael B. Harari (FAU: Florida Atlantic University)H-Index: 6
Last.Palina Prysmakova (FAU: Florida Atlantic University)H-Index: 2
view all 4 authors...
2 CitationsSource
#1Fallon A. Kane (Adelphi University)H-Index: 1
#2Robert F. Bornstein (Adelphi University)H-Index: 40
Source
#1Jeremy D. Mackey (AU: Auburn University)H-Index: 11
Last.Jack Emery Carson (AU: Auburn University)H-Index: 2
view all 4 authors...
Source
Source
#1Wolfgang G. Weber (University of Innsbruck)H-Index: 10
#2Christine Unterrainer (University of Innsbruck)H-Index: 7
Last.Thomas Höge (University of Innsbruck)H-Index: 8
view all 3 authors...
1 CitationsSource
#1Jeremy D. Mackey (AU: Auburn University)H-Index: 11
#2Philip L. Roth (Clemson University)H-Index: 42
Last.Lynn A. McFarland (USC: University of South Carolina)H-Index: 21
view all 4 authors...
2 CitationsSource
#1Antonia J. Kaluza (Goethe University Frankfurt)H-Index: 1
#2Diana Boer (University of Koblenz and Landau)H-Index: 14
Last.Rolf van Dick (Goethe University Frankfurt)H-Index: 47
view all 4 authors...
1 CitationsSource
#1Caroline Knight (Curtin University)H-Index: 2
#2Malcolm Patterson (University of Sheffield)H-Index: 23
Last.Jeremy Dawson (University of Sheffield)H-Index: 33
view all 3 authors...
Source