Specification Curve: Descriptive and Inferential Statistics on All Reasonable Specifications

Published on Nov 24, 2015
· DOI :10.2139/ssrn.2694998
Uri Simonsohn26
Estimated H-index: 26
(UPenn: University of Pennsylvania),
Joseph P. Simmons24
Estimated H-index: 24
(UPenn: University of Pennsylvania),
Leif D. Nelson26
Estimated H-index: 26
(University of California, Berkeley)
Empirical results often hinge on data analytic decisions that are simultaneously defensible, arbitrary, and motivated. To mitigate this problem we introduce Specification-Curve Analysis, which consists of three steps: (i) identifying the set of theoretically justified, statistically valid, and non-redundant analytic specifications, (ii) displaying alternative results graphically, allowing the identification of decisions producing different results, and (iii) conducting statistical tests to determine whether as a whole results are inconsistent with the null hypothesis. We illustrate its use by applying it to three published findings. One proves robust, one weak, one not robust at all.
  • References (9)
  • Citations (20)
#1Jeffrey Czajkowski (Austin College)H-Index: 11
#2Kevin M. Simmons (Austin College)H-Index: 7
Last.Daniel Sutter (UTPA: University of Texas–Pan American)H-Index: 19
view all 3 authors...
Cited By20
#1Evan C. Carter (ARL: United States Army Research Laboratory)H-Index: 13
#2Felix D. Schönbrodt (LMU: Ludwig Maximilian University of Munich)H-Index: 19
Last.Joseph Hilgard (ISU: Illinois State University)H-Index: 10
view all 4 authors...
May 2, 2019 in CHI (Human Factors in Computing Systems)
#1Pierre Dragicevic (IRIA: French Institute for Research in Computer Science and Automation)H-Index: 29
#2Yvonne Jansen (University of Paris)H-Index: 13
Last.Fanny Chevalier (U of T: University of Toronto)H-Index: 17
view all 5 authors...
May 2, 2019 in CHI (Human Factors in Computing Systems)
#1Alex Kale (UW: University of Washington)H-Index: 4
#2Matthew Kay (UM: University of Michigan)H-Index: 17
Last.Jessica Hullman (NU: Northwestern University)H-Index: 12
view all 3 authors...
#1Tomás Lejarraga (MPG: Max Planck Society)H-Index: 8
#2Renato Frey (University of Basel)H-Index: 9
Last.Ralph Hertwig (MPG: Max Planck Society)H-Index: 46
view all 4 authors...
#1Patrick S. Forscher (UA: University of Arkansas)H-Index: 7
#2William T. L. Cox (UW: University of Wisconsin-Madison)H-Index: 10
Last.Patricia G. Devine (UW: University of Wisconsin-Madison)H-Index: 42
view all 4 authors...
View next paperFalse-Positive Psychology Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant