Promises and Perils of Pre-Analysis Plans †

Published on Aug 1, 2015in Journal of Economic Perspectives6.45
· DOI :10.1257/jep.29.3.61
Benjamin A. Olken28
Estimated H-index: 28
(MIT: Massachusetts Institute of Technology)
The purpose of this paper is to help think through the advantages and costs of rigorous pre-specification of statistical analysis plans in economics. A pre-analysis plan pre-specifies in a precise way the analysis to be run before examining the data. A researcher can specify variables, data cleaning procedures, regression specifications, and so on. If the regressions are pre-specified in advance and researchers are required to report all the results they pre-specify, data-mining problems are greatly reduced. I begin by laying out the basics of what a statistical analysis plan actually contains so those researchers unfamiliar with it can better understand how it is done. In so doing, I have drawn both on standards used in clinical trials, which are clearly specified by the Food and Drug Administration, as well as my own practical experience from writing these plans in economics contexts. I then lay out some of the advantages of pre-specified analysis plans, both for the scientific community as a whole and also for the researcher. I also explore some of the limitations and costs of such plans. I then review a few pieces of evidence that suggest that, in many contexts, the benefits of using pre-specified analysis plans may not be as high as one might have expected initially. For the most part, I will focus on the relatively narrow issue of pre-analysis for randomized controlled trials.
  • References (31)
  • Citations (57)
#1Abel Brodeur (U of O: University of Ottawa)H-Index: 4
#2Mathias Lé (PSE: Paris School of Economics)H-Index: 2
Last.Yanos ZylberbergH-Index: 4
view all 4 authors...
98 CitationsSource
#1Alessandro Tarozzi (Barcelona Graduate School of Economics)H-Index: 16
#2Aprajit Mahajan (UCLA: University of California, Los Angeles)H-Index: 11
Last.Joanne Yoong (AN: RAND Corporation)H-Index: 4
view all 6 authors...
97 CitationsSource
#1Nava Ashraf (Harvard University)H-Index: 21
#2Erica Field (Duke University)H-Index: 20
Last.Jean N. Lee (NYU: New York University)H-Index: 6
view all 3 authors...
132 CitationsSource
#1Dean Karlan (Yale University)H-Index: 46
#2Robert Osei (University of Ghana)H-Index: 14
Last.Christopher Udry (Yale University)H-Index: 27
view all 4 authors...
285 CitationsSource
#1Richard A. Klein (UF: University of Florida)H-Index: 7
#2Kate A. Ratliff (UF: University of Florida)H-Index: 13
Last.Brian A. Nosek (UVA: University of Virginia)H-Index: 61
view all 51 authors...
337 CitationsSource
#1Christopher Blattman (Columbia University)H-Index: 21
#2Nathan Fiala (UConn: University of Connecticut)H-Index: 9
Last.Sebastian Martinez (BID: Inter-American Development Bank)H-Index: 18
view all 3 authors...
130 CitationsSource
170 CitationsSource
145 CitationsSource
#1Edward Miguel (University of California, Berkeley)H-Index: 44
#2Colin F. Camerer (California Institute of Technology)H-Index: 102
Last.M. J. van der Laan (University of California, Berkeley)H-Index: 12
view all 19 authors...
156 CitationsSource
#1Francesco Avvisati (OECD: Organisation for Economic Co-operation and Development)H-Index: 6
#2Marc Gurgand (PSE: Paris School of Economics)H-Index: 14
Last.Eric Maurin (PSE: Paris School of Economics)H-Index: 28
view all 4 authors...
50 CitationsSource
Cited By57
#1Stephan B. Bruns (GAU: University of Göttingen)H-Index: 6
#2Igor Asanov (University of Kassel)H-Index: 1
Last.Guido Buenstorf (University of Kassel)H-Index: 17
view all 14 authors...
#1Elizabeth Levy Paluck (Princeton University)H-Index: 18
#2Seth A. GreenH-Index: 1
Last.P GreenDonald (Columbia University)H-Index: 63
view all 3 authors...
18 CitationsSource
1 CitationsSource
#1Erick Gong (Middlebury College)H-Index: 6
#2Damien de Walque (World Bank)H-Index: 26
Last.William H. Dow (University of California, Berkeley)H-Index: 23
view all 3 authors...
#1James Chu (Stanford University)H-Index: 18
#2Guirong Li (Henan University)H-Index: 3
Last.Yanyan Li (Henan University)
view all 6 authors...
#1Sophie Thoyer (University of Montpellier)H-Index: 13
#2Raphaële Préget (University of Montpellier)
#1Leah H. Palm-Forster (UD: University of Delaware)H-Index: 4
#2Paul J. Ferraro (Johns Hopkins University)H-Index: 42
Last.Kent D. Messer (UD: University of Delaware)H-Index: 19
view all 5 authors...
#1Laura Dague (A&M: Texas A&M University)H-Index: 6
#2Joanna N. Lahey (A&M: Texas A&M University)H-Index: 12
2 CitationsSource