Transparency, Reproducibility, and the Credibility of Economics Research

Published on Sep 1, 2018in Journal of Economic Literature
· DOI :10.1257/jel.20171350
Garret Christensen4
Estimated H-index: 4
Edward Miguel44
Estimated H-index: 44
There is growing interest in enhancing research transparency and reproducibility in economics and other scientific fields. We survey existing work on these topics within economics, and discuss the evidence suggesting that publication bias, inability to replicate, and specification searching remain widespread in the discipline. We next discuss recent progress in this area, including through improved research design, study registration and pre-analysis plans, disclosure standards, and open sharing of data and materials, drawing on experiences in both economics and other social sciences. We discuss areas where consensus is emerging on new practices, as well as approaches that remain controversial, and speculate about the most effective ways to make economics research more credible in the future.
Figures & Tables
  • References (120)
  • Citations (18)
📖 Papers frequently viewed together
39 Citations
70 Citations
48 Citations
78% of Scinapse members use related papers. After signing in, all features are FREE.
#1Yihui XieH-Index: 5
70 CitationsSource
#1Marianne BertrandH-Index: 42
62 CitationsSource
#1Kevin Croke (World Bank)H-Index: 8
#2Joan Hamory Hicks (University of California, Berkeley)H-Index: 10
Last. Edward Miguel (NBER: National Bureau of Economic Research)H-Index: 44
view all 5 authors...
The WHO has recently debated whether to reaffirm its long-standing recommendation of mass drug administration (MDA) in areas with more than 20 percent prevalence of soil-transmitted helminths (hookworm, whipworm, and roundworm). There is consensus that the relevant deworming drugs are safe and effective, so the key question facing policymakers is whether the expected benefits of MDA exceed the roughly $0.30 per treatment cost. The literature on long run educational and economic impacts of deworm...
18 CitationsSource
#1Tyler Cowen (GMU: George Mason University)H-Index: 25
#2Alexander Tabarrok (GMU: George Mason University)H-Index: 18
We can imagine a plausible case for government support of science based on traditional economic reasons of externalities and public goods. Yet when it comes to government support of grants from the National Science Foundation (NSF) for economic research, our sense is that many economists avoid critical questions, skimp on analysis, and move straight to advocacy. In this essay, we take a more skeptical attitude toward the efforts of the NSF to subsidize economic research. We offer two main sets o...
2 CitationsSource
#1Edward E. Leamer (UCLA: University of California, Los Angeles)H-Index: 44
This paper proposes a context-minimal range of alternative regression models that is used to generate a range of alternative estimates. A prior distribution is assumed with a zero mean but an ambiguous covariance matrix. The choice of the prior covariance matrix is facilitated by transformation to standardized variables which makes the prior expected R2 equal to the sum of the prior variances. Three different ranges of the prior expected R2 are used to define three different intervals of prior c...
5 CitationsSource
Nearly 1 year ago, a group of researchers boldly suggested that the standards for research quality, transparency, and trustworthiness could be improved if journals banded together to adopt eight standards called TOP (Transparency and Openness Promotion).* Since that time, more than 500 journals have been working toward their implementation of TOP. The editors at Science have held additional retreats and workshops to determine how best to adapt TOP to a general science journal and are now ready t...
12 CitationsSource
#1Justin McCrary (University of California, Berkeley)H-Index: 15
#2Garret Christensen (University of California, Berkeley)H-Index: 4
Last. Daniele Fanelli (Stanford University)H-Index: 19
view all 3 authors...
Publication bias leads consumers of research to observe a selected sample of statistical estimates calculated by producers of research. We calculate critical values for statistical significance that could help to adjust after the fact for the distortions created by this selection effect, assuming that the only source of publication bias is file drawer bias. These adjusted critical values are easy to calculate and differ from unadjusted critical values by approximately 50%—rather than rejecting a...
4 CitationsSource
#1Uri Simonsohn (UPenn: University of Pennsylvania)H-Index: 26
#2Joseph P. Simmons (UPenn: University of Pennsylvania)H-Index: 24
Last. Leif D. Nelson (University of California, Berkeley)H-Index: 26
view all 3 authors...
Empirical results often hinge on data analytic decisions that are simultaneously defensible, arbitrary, and motivated. To mitigate this problem we introduce Specification-Curve Analysis, which consists of three steps: (i) identifying the set of theoretically justified, statistically valid, and non-redundant analytic specifications, (ii) displaying alternative results graphically, allowing the identification of decisions producing different results, and (iii) conducting statistical tests to deter...
41 CitationsSource
#1Alexander M AikenH-Index: 14
#2Calum DaveyH-Index: 11
Last. Richard J. Hayes (Lond: University of London)H-Index: 86
view all 4 authors...
Background: Helminth (worm) infections cause morbidity among poor communities worldwide. An influential study conducted in Kenya in 1998‐99 reported that a schoolbased drug-and-educational intervention had benefits regarding worm infections and school attendance. Effects were seen among children treated with deworming drugs, untreated children in intervention schools and children in nearby non-intervention schools. Combining these effects, the intervention was reported to increase school attenda...
55 CitationsSource
#1Benjamin A. Olken (MIT: Massachusetts Institute of Technology)H-Index: 28
The purpose of this paper is to help think through the advantages and costs of rigorous pre-specification of statistical analysis plans in economics. A pre-analysis plan pre-specifies in a precise way the analysis to be run before examining the data. A researcher can specify variables, data cleaning procedures, regression specifications, and so on. If the regressions are pre-specified in advance and researchers are required to report all the results they pre-specify, data-mining problems are gre...
57 CitationsSource
Cited By18
#2Tomas HavranekH-Index: 21
Last. Roman HorvathH-Index: 22
view all 3 authors...
Expatriate workers’ remittances represent an important source of financing for low- and middle-income countries. No consensus, however, has yet emerged regarding the effect of remittances on economic growth. In a quantitative survey of 538 estimates reported in 95 studies, we find that approximately 40% of the studies report a positive effect, 40% report no effect, and 20% report a negative effect. Our results indicate publication bias in favor of positive effects. Correcting for the bias using ...
#1Juan Carlos Guzman (ND: University of Notre Dame)H-Index: 1
#2Lila Kumar Khatiwada (ND: University of Notre Dame)H-Index: 3
Last. Danice Guzmán (ND: University of Notre Dame)
view all 3 authors...
Abstract Women experiencing domestic violence make tradeoffs and negotiations within their constrained environments to reduce the risk they and their children face. In societies where failure to fulfill domestic duties is considered justification for abuse, women may reduce their risk by ensuring adequate and quality food is available when needed. Improved cookstoves can help women prepare quality food at lower cost and in a timely manner. When food preparation is a trigger for domestic violence...
#1Sara PedronH-Index: 1
#2Werner MaierH-Index: 14
Last. Lars Schwettmann (MLU: Martin Luther University of Halle-Wittenberg)H-Index: 5
view all 8 authors...
Abstract Retirement is a major life event potentially associated with changes in relevant risk factors for cardiovascular and metabolic conditions. This study analyzes the effect of retirement on behavioral and biomedical risk factors for chronic disease, together with subjective health parameters using Southern German epidemiological data. We used panel data from the KORA cohort study, consisting of 11,168 observations for individuals 45–80 years old. Outcomes included health behavior (alcohol,...
#1Jerome Geyer-Klingeberg (University of Augsburg)H-Index: 4
#2Markus Hang (University of Augsburg)H-Index: 4
Last. Andreas W. Rathgeber (University of Augsburg)H-Index: 8
view all 3 authors...
Abstract The number of empirical research studies in finance exhibits a strong upward trajectory, producing large differences in empirical results, which often impedes the drawing of consistent conclusions in relation to the phenomenon under examination. This creates demand for methods like meta-analysis that objectively consolidate and evaluate previous empirical findings. Meta-analysis is a group of statistical methods to aggregate prior empirical studies, to discover and explain consistencies...
#1Yang Liu (UCSC: University of California, Santa Cruz)H-Index: 16
Last. domenico viganolaH-Index: 2
view all 8 authors...
The last decade saw the emergence of systematic large-scale replication projects in the social and behavioral sciences, (Camerer et al., 2016, 2018; Ebersole et al., 2016; Klein et al., 2014, 2018; Collaboration, 2015). These projects were driven by theoretical and conceptual concerns about a high fraction of "false positives" in the scientific publications (Ioannidis, 2005) (and a high prevalence of "questionable research practices" (Simmons, Nelson, and Simonsohn, 2011). Concerns about the cre...
#2Kuntal K. DasH-Index: 5
Last. W. Robert ReedH-Index: 20
view all 4 authors...
Abstract This study uses meta-analysis to study the effect of spillovers on exports. It collects 3291 estimated spillover effects from 99 studies. The estimated spillover effects in the literature span a large number of types and measures of both exports and spillovers. As a result, we transform estimates to partial correlation coefficients. We analyze these transformed effects using four different versions of Weighted Least Squares (WLS) estimators, incorporating both meta-analytic “Fixed Effec...
#1Alan M. JacobsH-Index: 1
1 CitationsSource
#1Jeremy FreeseH-Index: 32
#2David PetersonH-Index: 1
#1Garret ChristensenH-Index: 4
#1Garret ChristensenH-Index: 3
Last. Edward MiguelH-Index: 44
view all 2 authors...