On the Need for a Replication Journal

Published on Jan 1, 2015
· DOI :10.2139/ssrn.2647280
Christian Zimmermann1
Estimated H-index: 1
There is very little replication of research in economics, particularly compared with other sciences. This paper argues that there is a dire need for studies that replicate research, that their scarcity is due to poor or negative rewards for replicators, and that this could be improved with a journal that exclusively publishes replication studies. I then discuss how such a journal could be organized, in particular in the face of some negative rewards some replication studies may elicit.
  • References (14)
  • Citations (4)
📖 Papers frequently viewed together
3 Citations
1 Author (Maarten Boudry)
1 Author (Luke McCrohon)
1 Citations
78% of Scinapse members use related papers. After signing in, all features are FREE.
#1Annie Franco (Stanford University)H-Index: 4
#2Neil Malhotra (Stanford University)H-Index: 26
Last. Gabor Simonovits (Stanford University)H-Index: 6
view all 3 authors...
We studied publication bias in the social sciences by analyzing a known population of conducted studies—221 in total—in which there is a full accounting of what is published and unpublished. We leveraged Time-sharing Experiments in the Social Sciences (TESS), a National Science Foundation–sponsored program in which researchers propose survey-based experiments to be run on representative samples of American adults. Because TESS proposals undergo rigorous peer review, the studies in the sample all...
342 CitationsSource
#1Thomas Herndon (Political Economy Research Institute)H-Index: 1
#2Michael Ash (Political Economy Research Institute)H-Index: 15
Last. Robert Pollin (Political Economy Research Institute)H-Index: 21
view all 3 authors...
Herndon, Ash and Pollin replicate Reinhart and Rogoff and find that coding errors, selective exclusion of available data, and unconventional weighting of summary statistics lead to serious errors that inaccurately represent the relationship between public debt and GDP growth among 20 advanced economies in the post-war period. They find that when properly calculated, the average real GDP growth rate for countries carrying a public-debt-to-GDP ratio of over 90 percent is actually 2.2 percent, not ...
438 CitationsSource
#1Zacharias ManiadisH-Index: 6
#2Fabio Tufano (University of Nottingham)H-Index: 5
Last. John A. List (U of C: University of Chicago)H-Index: 75
view all 3 authors...
Some researchers have argued that anchoring in economic valuations casts doubt on the assumption of consistent and stable preferences. We present new evidence that questions the robustness of certain anchoring results. We then present a theoretical framework that provides insights into why we should be cautious of initial empirical findings in general. The model importantly highlights that the rate of false positives depends not only on the observed significance level, but also on statistical po...
93 CitationsSource
#1Simon Johnson (Peterson Institute for International Economics)H-Index: 47
#2William D. Larson (GW: George Washington University)H-Index: 7
Last. Arvind Subramanian (Johns Hopkins University)H-Index: 38
view all 4 authors...
This paper sheds light on two problems in the Penn World Table (PWT) GDP estimates. First, we show that these estimates vary substantially across different versions of the PWT despite being derived from very similar underlying data and using almost identical methodologies; that this variability is systematic; and that it is intrinsic to the methodology deployed by the PWT to estimate growth rates. Moreover, this variability matters for the cross-country growth literature. While growth studies th...
153 CitationsSource
We study economic growth and inflation at different levels of government and external debt. Our analysis is based on new data on forty-four countries spanning about two hundred years. The dataset incorporates over 3,700 annual observations covering a wide range of political systems, institutions, exchange rate arrangements, and historic circumstances. Our main findings are: First, the relationship between government debt and real GDP growth is weak for debt/GDP ratios below a threshold of 90 per...
1,345 CitationsSource
#1A. Talha Yalta (TOBB University of Economics and Technology)H-Index: 8
#2A. Yasemin Yalta (Hacettepe University)H-Index: 6
We survey the literature on the accuracy of econometric software. We also assess the advantages of open source software from the point of view of reliability and discuss its potential in applied economics, which has now become fully dependent on computers. As a case study, we apply various accuracy tests on GNU Regression, Econometrics and Time-series Library (gretl) and demonstrate that the open source nature of the program made it possible to see the cause, facilitated a rapid fix, and enabled...
21 CitationsSource
Most economics journals take no substantive measures to ensure that the results they publish are replicable. To make the data and code available so that published results can be checked requires an archive. Top economics journals have been adopting mandatory data+code archives in the past few years. The movement toward mandatory data+code archives has yet to reach the open access journals. This is paradoxical; given their emphasis on making articles readily available, one would think that open a...
35 CitationsSource
#1Bruce D. McCullough (Drexel University)H-Index: 22
#2Kerry Anne McGeary (Drexel University)H-Index: 7
Last. Teresa D. Harrison (Drexel University)H-Index: 8
view all 3 authors...
can rarely be reproduced using the datapcode in the journal archive. Recently created archives at top journals should avoid the mistakes of their predecessors. We categorize reasons for archives' failures and identify successful policies.
82 CitationsSource
Undergraduate students in my economics classes tend to have an unswerving trust that markets can solve any economic problem. And so it is in development economics, where they mainly tend to come down on the side of “development through growth” when we address whether there is a role for aid, for example, in poverty alleviation in developing economies. The issues, of course, are thorny. How do we define the poverty level? How often is growth pro-poor? What is pro-poor growth? How do we evaluate g...
2 CitationsSource
#1Esther Duflo (MIT: Massachusetts Institute of Technology)H-Index: 67
#2Rachel Glennerster (MIT: Massachusetts Institute of Technology)H-Index: 25
Last. Douglas E. Levy (NBER: National Bureau of Economic Research)H-Index: 200
view all 3 authors...
This paper is a practical guide (a toolkit) for researchers, students and practitioners wishing to introduce randomization as part of a research design in the field. It first covers the rationale for the use of randomization, as a solution to selection bias and a partial solution to publication biases. Second, it discusses various ways in which randomization can be practically introduced in a field settings. Third, it discusses designs issues such as sample size requirements, stratification, lev...
686 CitationsSource
Cited By4
#1Andrew C. ChangH-Index: 4
#2Phillip LiH-Index: 5
Last. Shawn M. MartinH-Index: 1
view all 3 authors...
Chotikapanich and Griffiths (2002) introduced the Dirichlet distribution to the estimation of Lorenz curves. This distribution naturally accommodates the proportional nature of income share data and the dependence structure between the shares. Chotikapanich and Griffiths (2002) fit a family of five Lorenz curves to one year of Swedish and Brazilian income share data using unconstrained maximum likelihood and unconstrained non-linear least squares. We attempt to replicate the authors' results and...
#1Michael A. Clemens (CGD: Center for Global Development)H-Index: 27
28 CitationsSource
#1Tyler Cowen (GMU: George Mason University)H-Index: 25
#2Alexander Tabarrok (GMU: George Mason University)H-Index: 18
We can imagine a plausible case for government support of science based on traditional economic reasons of externalities and public goods. Yet when it comes to government support of grants from the National Science Foundation (NSF) for economic research, our sense is that many economists avoid critical questions, skimp on analysis, and move straight to advocacy. In this essay, we take a more skeptical attitude toward the efforts of the NSF to subsidize economic research. We offer two main sets o...
2 CitationsSource
#1Verónica De Jesús Romo (UNAM: National Autonomous University of Mexico)H-Index: 1
E l objetivo de este articulo es argumentar que para que la economia se fundamente plenamente como una ciencia, y no simplemente como un arte basado en la intuicion, se deben seguir ciertos principios cientificos. El articulo contribuye al debate sobre la materia, especificamente en dos aspectos metodologicos: 1) el adecuado analisis de datos para la verificacion de las hipotesis de una investigacion y 2) la replicacion de los hallazgos cientificos con la finalidad de evaluar a los mismos. El pr...
1 CitationsSource