Evaluating Replicability of Laboratory Experiments in Economics

Published on Mar 25, 2016in Science41.04
· DOI :10.1126/science.aaf0918
Colin F. Camerer102
Estimated H-index: 102
(California Institute of Technology),
Anna Dreber30
Estimated H-index: 30
(HHS: Stockholm School of Economics)
+ 15 AuthorsHang Wu2
Estimated H-index: 2
(NUS: National University of Singapore)
View in Source
The reproducibility of scientific findings has been called into question. To contribute data about reproducibility in economics, we replicate 18 studies published in the American Economic Review and the Quarterly Journal of Economics in 2011-2014. All replications follow predefined analysis plans publicly posted prior to the replications, and have a statistical power of at least 90% to detect the original effect size at the 5% significance level. We find a significant effect in the same direction as the original study for 11 replications (61%); on average the replicated effect size is 66% of the original. The reproducibility rate varies between 67% and 78% for four additional reproducibility indicators, including a prediction market measure of peer beliefs.
Figures & Tables
  • References (63)
  • Citations (201)
#1Abel Brodeur (U of O: University of Ottawa)H-Index: 4
#2Mathias Lé (PSE: Paris School of Economics)H-Index: 2
Last.Yanos ZylberbergH-Index: 4
view all 4 authors...
98 CitationsSource
#1Anna Dreber (HHS: Stockholm School of Economics)H-Index: 30
#2Thomas Pfeiffer (Massey University)H-Index: 26
Last.Magnus Johannesson (HHS: Stockholm School of Economics)H-Index: 84
view all 8 authors...
54 CitationsSource
#1Brian A. NosekH-Index: 61
#2George AlterH-Index: 20
Last.Tal YarkoniH-Index: 35
view all 39 authors...
443 CitationsSource
#1Leonard P. FreedmanH-Index: 57
#2Iain M. Cockburn (BU: Boston University)H-Index: 41
Last.Timothy Simcoe (BU: Boston University)H-Index: 17
view all 3 authors...
346 CitationsSource
#1Uri Simonsohn (UPenn: University of Pennsylvania)H-Index: 26
220 CitationsSource
19 CitationsSource
#1Alexander A. AartsH-Index: 1
#2Joanna E. Anderson (DRDC: Defence Research and Development Canada)H-Index: 3
Last.Kellylynn Zuni (Adams State University)H-Index: 2
view all 270 authors...
2,235 CitationsSource
#1Geoffroy de Clippel (Brown University)H-Index: 12
#2Kfir EliazH-Index: 20
Last.Brian G. Knight (Brown University)H-Index: 19
view all 3 authors...
23 CitationsSource
#1Josine Verhagen (UvA: University of Amsterdam)H-Index: 18
#2Eric-Jan Wagenmakers (UvA: University of Amsterdam)H-Index: 67
94 CitationsSource
28 CitationsSource
Cited By201
#1Ben Anderson (University of Southampton)H-Index: 21
#2Tom Rushby (University of Southampton)H-Index: 1
Last.Patrick James (University of Southampton)H-Index: 20
view all 4 authors...
#1Rotem Botvinik-Nezer (TAU: Tel Aviv University)H-Index: 2
#2Roni Iwanir (TAU: Tel Aviv University)H-Index: 1
Last.Tom Schonberg (TAU: Tel Aviv University)H-Index: 14
view all 10 authors...
#1Stephan B. Bruns (GAU: University of Göttingen)H-Index: 6
#2Igor Asanov (University of Kassel)H-Index: 1
Last.Guido Buenstorf (University of Kassel)H-Index: 17
view all 14 authors...
#1Stefano DellaVigna (University of California, Berkeley)H-Index: 24
#2Devin G. Pope (U of C: University of Chicago)H-Index: 20
Last.Eva Vivalt (ANU: Australian National University)H-Index: 1
view all 3 authors...
#1Richard M. Walker (CityU: City University of Hong Kong)H-Index: 43
#2Gene A. Brewer (UGA: University of Georgia)H-Index: 33
Last.Witteloostuijn van Arjen (VU: VU University Amsterdam)H-Index: 35
view all 5 authors...
3 CitationsSource
#1Ryan Martin (NCSU: North Carolina State University)H-Index: 15
4 CitationsSource
#1HanssonSture (Stockholm University)H-Index: 28
#2Ulf Larsson (Stockholm University)H-Index: 31
Last.Jakob Walve (Stockholm University)H-Index: 11
view all 3 authors...
#1Jordi Brandts (CSIC: Spanish National Research Council)H-Index: 29
#2Arno Riedl (UM: Maastricht University)H-Index: 28
View next paperEstimating the reproducibility of psychological science