Limitations of PET-PEESE and Other Meta-Analysis Methods:

Published on Feb 28, 2017in Social Psychological and Personality Science
· DOI :10.1177/1948550617693062
T. D. Stanley33
Estimated H-index: 33
(Hendrix College)
A novel meta-regression method, precision-effect test and precision-effect estimate with standard errors (PET-PEESE), predicts and explains recent high-profile failures to replicate in psychology. The central purpose of this article is to identify the limitations of PET-PEESE for application to social/personality psychology. Using typical conditions found in social/personality research, our simulations identify three areas of concern. PET-PEESE performs poorly in research areas where there are only a few studies, all studies use small samples, and where there is very high heterogeneity of results from study to study. Nonetheless, the statistical properties of conventional meta-analysis approaches are much worse than PET-PEESE under these same conditions. Our simulations suggest alterations to conventional research practice and ways to moderate PET-PEESE weaknesses.
  • References (29)
  • Citations (42)
📖 Papers frequently viewed together
4,938 Citations
236 Citations
4 Authors (Evan C. Carter, ..., Joseph Hilgard)
61 Citations
78% of Scinapse members use related papers. After signing in, all features are FREE.
#1T. D. Stanley (Hendrix College)H-Index: 33
#2Hristos Doucouliagos (Deakin University)H-Index: 31
Last. John P. A. Ioannidis (Stanford University)H-Index: 157
view all 3 authors...
The central purpose of this study is to document how a sharper focus upon statistical power may reduce the impact of selective reporting bias in meta-analyses. We introduce the weighted average of the adequately powered (WAAP) as an alternative to the conventional random-effects (RE) estimator. When the results of some of the studies have been selected to be positive and statistically significant (i.e. selective reporting), our simulations show that WAAP will have smaller bias than RE at no loss...
20 CitationsSource
#1T. D. Stanley (Hendrix College)H-Index: 33
#2Hristos Doucouliagos (Deakin University)H-Index: 31
Our study revisits and challenges two core conventional meta-regression estimators: the prevalent use of ‘mixed-effects’ or random-effects meta-regression analysis and the correction of standard errors that defines fixed-effects meta-regression analysis (FE-MRA). We show how and explain why an unrestricted weighted least squares MRA (WLS-MRA) estimator is superior to conventional random-effects (or mixed-effects) meta-regression when there is publication (or small-sample) bias that is as good as...
56 CitationsSource
#1Blakeley B. McShane (NU: Northwestern University)H-Index: 16
#2Ulf Böckenholt (NU: Northwestern University)H-Index: 33
Last. Karsten T. Hansen (UCSD: University of California, San Diego)H-Index: 16
view all 3 authors...
We review and evaluate selection methods, a prominent class of techniques first proposed by Hedges (1984) that assess and adjust for publication bias in meta-analysis, via an extensive simulation study. Our simulation covers both restrictive settings as well as more realistic settings and proceeds across multiple metrics that assess different aspects of model performance. This evaluation is timely in light of two recently proposed approaches, the so-called p-curve and p-uniform approaches, that ...
100 CitationsSource
#1Robbie C. M. van Aert (Tilburg University)H-Index: 11
#2Jelte M. Wicherts (Tilburg University)H-Index: 42
Last. Marcel A.L.M. van Assen (UU: Utrecht University)H-Index: 27
view all 3 authors...
Because of overwhelming evidence of publication bias in psychology, techniques to correct meta-analytic estimates for such bias are greatly needed. The methodology on which the p-uniform and p-curve methods are based has great promise for providing accurate meta-analytic estimates in the presence of publication bias. However, in this article, we show that in some situations, p-curve behaves erratically, whereas p-uniform may yield implausible estimates of negative effect size. Moreover, we show ...
65 CitationsSource
#1Martin S. Hagger (Curtin University)H-Index: 58
#2Nikos L. D. Chatzisarantis (Curtin University)H-Index: 50
Good self-control has been linked to adaptive outcomes such as better health, cohesive personal relationships, success in the workplace and at school, and less susceptibility to crime and addictions. In contrast, self-control failure is linked to maladaptive outcomes. Understanding the mechanisms by which self-control predicts behavior may assist in promoting better regulation and outcomes. A popular approach to understanding self-control is the strength or resource depletion model. Self-control...
348 CitationsSource
#1Stephan B. Bruns (University of Kassel)H-Index: 7
#2John P. A. Ioannidis (Stanford University)H-Index: 157
The p-curve, the distribution of statistically significant p-values of published studies, has been used to make inferences on the proportion of true effects and on the presence of p-hacking in the published literature. We analyze the p-curve for observational research in the presence of p-hacking. We show by means of simulations that even with minimal omitted-variable bias (e.g., unaccounted confounding) p-curves based on true effects and p-curves based on null-effects with p-hacking cannot be r...
65 CitationsSource
#1Alexander A. AartsH-Index: 1
#1Alexander A. AartsH-Index: 1
Last. Kellylynn Zuni (Adams State University)H-Index: 2
view all 270 authors...
Reproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. Replication effects were half the magnitude of original effects, representing a substantial decline. Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of ...
2,768 CitationsSource
#1Evan C. Carter (UM: University of Miami)H-Index: 15
#2Lilly M. Kofler (UM: University of Miami)H-Index: 1
Last. Michael E. McCullough (UM: University of Miami)H-Index: 67
view all 4 authors...
Failures of self-control are thought to underlie various important behaviors (e.g., addiction, violence, obesity, poor academic achievement). The modern conceptualization of self-control failure has been heavily influenced by the idea that self-control functions as if it relied upon a limited physiological or cognitive resource. This view of self-control has inspired hundreds of experiments designed to test the prediction that acts of self-control are more likely to fail when they follow previou...
235 CitationsSource
#1T. D. Stanley (Hendrix College)H-Index: 33
#2Hristos Doucouliagos (Deakin University)H-Index: 31
This study challenges two core conventional meta-analysis methods: fixed effect and random effects. We show how and explain why an unrestricted weighted least squares estimator is superior to conventional random-effects meta-analysis when there is publication (or small-sample) bias and better than a fixed-effect weighted average if there is heterogeneity. Statistical theory and simulations of effect sizes, log odds ratios and regression coefficients demonstrate that this unrestricted weighted le...
99 CitationsSource
#1Tim Bogg (WSU: Wayne State University)H-Index: 18
#2Leanne Lasecki (WSU: Wayne State University)H-Index: 2
In recent years, cognitive scientists and commercial interests (e.g., Fit Brains, Lumosity) have focused research attention and financial resources on cognitive tasks, especially working memory tasks, to explore and exploit possible transfer effects to general cognitive abilities, such as fluid intelligence. The increased research attention has produced mixed findings, as well as contention about the disposition of the evidence base. To address this contention, J. Au and colleagues (2014; doi:10...
22 CitationsSource
Cited By42
#1Megan de Linde Leonard (Hendrix College)H-Index: 3
#2T. D. Stanley (Deakin University)H-Index: 33
#1Ye Sun (UofU: University of Utah)H-Index: 1
#2Zhongdang Pan (UW: University of Wisconsin-Madison)
#1Christopher R. Rakes (UMBC: University of Maryland, Baltimore County)H-Index: 7
#2Robert N. Ronau (Johns Hopkins University)
Last. David Pugalee (UNCC: University of North Carolina at Charlotte)H-Index: 12
view all 6 authors...
#1Susanne Buecker (RUB: Ruhr University Bochum)H-Index: 2
Last. Bianca A. SimonsmeierH-Index: 5
view all 5 authors...
Longstanding research suggests a positive relation between physical activity and health. However, when investigating this relation most studies focused on the absence of disease or infirmity as hea...
#1Philipp A. Schroeder (University of Tübingen)H-Index: 9
#2Tobias Schwippel (University of Tübingen)H-Index: 3
Last. Jennifer Svaldi (University of Tübingen)H-Index: 19
view all 4 authors...
BACKGROUND: Inhibitory control refers to a central cognitive capacity involved in the interruption and correction of actions. Dysfunctions in these cognitive control processes have been identified as major maintaining mechanisms in a range of mental disorders such as ADHD, binge eating disorder, obesity, and addiction. Improving inhibitory control by transcranial direct current stimulation (tDCS) could ameliorate symptoms in a broad range of mental disorders. OBJECTIVE: The primary aim of this p...
#1Deming Adam Wang (JCU: James Cook University)H-Index: 1
Last. Nikos L. D. Chatzisarantis (Curtin University)H-Index: 50
view all 3 authors...
The ironic effect of thought suppression refers to the phenomenon in which individuals trying to rid their mind of a target thought ironically experience greater levels of occurrence and accessibility of the thought compared to individuals that deliberately concentrate on the thought (Wegner, 1994). Ironic effects occurring after thought suppression, also known as rebound effects, have been consistently detected by previous meta-analyses. However, ironic effects that occur during thought suppres...
1 CitationsSource
#1Amanda Kvarven (University of Bergen)H-Index: 2
#2Eirik Strømland (University of Bergen)H-Index: 3
Last. Magnus Johannesson (HHS: Stockholm School of Economics)H-Index: 86
view all 3 authors...
Many researchers rely on meta-analysis to summarize research evidence. However, there is a concern that publication bias and selective reporting may lead to biased meta-analytic effect sizes. We compare the results of meta-analyses to large-scale preregistered replications in psychology carried out at multiple laboratories. The multiple-laboratory replications provide precisely estimated effect sizes that do not suffer from publication bias or selective reporting. We searched the literature and ...
11 CitationsSource
#1James G. Field (WVU: West Virginia University)H-Index: 8
#2Frank A. Bosco (VCU: Virginia Commonwealth University)H-Index: 13
Last. Sven Kepes (VCU: Virginia Commonwealth University)H-Index: 15
view all 3 authors...
Although systematic reviews are considered the primary means for generating cumulative knowledge and their results are often used to inform evidence-based practice, the robustness of their meta-analytic summary estimates is rarely investigated. Consequently, the results of published systematic reviews and, by extension, our cumulative knowledge have come under scrutiny. Using a comprehensive approach to sensitivity analysis, we examined the impact of outliers and publication bias, as well as the...
1 CitationsSource
#1Marco Colagrossi (UCSC: Catholic University of the Sacred Heart)H-Index: 1
#2Domenico Rossignoli (UCSC: Catholic University of the Sacred Heart)H-Index: 2
Last. Mario A. Maggioni (UCSC: Catholic University of the Sacred Heart)H-Index: 14
view all 3 authors...
Abstract The relationship between democracy and economic growth has been widely debated in the social sciences with contrasting results. We apply a meta-analytical framework surveying 188 studies (2047 models) covering 36 years of research in the field. We also compare the effect of democracy on growth with the effect of human capital on growth in a sub-sample of 111 studies (875 models). Our findings suggest that democracy has a positive and direct effect on economic growth beyond the reach of ...
1 CitationsSource
The purpose of this paper is to examine the factors affecting the academic outcome of real estate students in a specialized Federal University in Nigeria. Furthermore, this paper investigates the phenomenon of publication bias in the extant literature as such evidence poses severe threats to the validity of empirical findings on factors affecting the degree outcome of undergraduate students.,The standard statistical approach adopted was to examine whether the reported coefficient estimates from ...