Meta-Analytic Choices and Judgment Calls: Implications for Theory Building and Testing, Obtained Effect Sizes, and Scholarly Impact

Published on Jan 1, 2011in Journal of Management
· DOI :10.1177/0149206310377113
Herman Aguinis56
Estimated H-index: 56
(IU: Indiana University Bloomington),
Dan R. Dalton59
Estimated H-index: 59
(IU: Indiana University Bloomington)
+ 2 AuthorsCatherine M. Dalton16
Estimated H-index: 16
(IU: Indiana University Bloomington)
The authors content analyzed 196 meta-analyses including 5,581 effect-size estimates published in Academy of Management Journal, Journal of Applied Psychology, Journal of Management, Personnel Psychology, and Strategic Management Journal from January 1982 through August 2009 to assess the presumed effects of each of 21 methodological choices and judgment calls on substantive conclusions. Results indicate that, overall, the various meta-analytic methodological choices available and judgment calls involved in the conduct of a meta-analysis have little impact on the resulting magnitude of the meta-analytically derived effect sizes. Thus, the present study, based on actual meta-analyses, casts doubt on previous warnings, primarily based on selective case studies, that judgment calls have an important impact on substantive conclusions. The authors also tested the fit of a multivariate model that includes relationships among theory-building and theory-testing goals, obtained effect sizes, year of publication of...
  • References (78)
  • Citations (121)
📖 Papers frequently viewed together
749 Citations
3,568 Citations
213 Citations
78% of Scinapse members use related papers. After signing in, all features are FREE.
#1Herman Aguinis (IU: Indiana University Bloomington)H-Index: 56
#2Ryan K. Gottfredson (IU: Indiana University Bloomington)H-Index: 12
Last. Thomas A. Wright (KSU: Kansas State University)H-Index: 33
view all 3 authors...
One of the key advantages of meta-analysis (i.e., a quantitative literature review) over a narrative literature review is that it allows for formal tests of interaction effects—namely, whether the relationship between two variables is contingent upon the value of another (moderator) variable. Interaction effects play a central role in organizational science research because they highlight boundary conditions of a theory: Conditions under which relationships change in strength and/or direction. T...
78 CitationsSource
#1Herman Aguinis (IU: Indiana University Bloomington)H-Index: 56
#2Charles A. Pierce (U of M: University of Memphis)H-Index: 30
Last. Catherine M. Dalton (IU: Indiana University Bloomington)H-Index: 16
view all 5 authors...
113 CitationsSource
#1Herman Aguinis (IU: Indiana University)H-Index: 56
#2Steve Werner (UH: University of Houston)H-Index: 26
Last. Donna Kohlhausen (UH: University of Houston)H-Index: 1
view all 6 authors...
In response to the ongoing concern regarding a science-practice gap, we propose a customer-centric approach to reporting significant research results that involves a sequence of three interdependent steps. The first step involves setting an alpha level (i.e., a priori Type I error rate) that considers the relative seriousness of falsely rejecting a null hypothesis of no effect or relationship (i.e., Type I error) relative to not detecting an existing effect or relationship (i.e., Type II error) ...
129 CitationsSource
#1Herman Aguinis (IU: Indiana University Bloomington)H-Index: 56
#2Steven Andrew Culpepper (University of Colorado Denver)H-Index: 15
Last. Charles A. Pierce (U of M: University of Memphis)H-Index: 30
view all 3 authors...
We developed a new analytic proof and conducted Monte Carlo simulations to assess the effects of methodological and statistical artifacts on the relative accuracy of intercept- and slope-based test bias assessment. The main simulation design included 3,185,000 unique combinations of a wide range of values for true intercept- and slope-based test bias, total sample size, proportion of minority group sample size to total sample size, predictor (i.e., preemployment test scores) and criterion (i.e.,...
61 CitationsSource
#1James G. Combs (FSU: Florida State University)H-Index: 33
69 CitationsSource
#1Herman AguinisH-Index: 1
#2Erika E. HardenH-Index: 1
This cautionary note provides a critical analysis of a statistical practice that is used pervasively by researchers in strategic management and related fields in conducting covariance structure analyses: The argument that a “large” sample size renders the χ2 goodness-of-fit test uninformative and a statistically significant result should not be an indication that the model does not fit the data well. Our analysis includes a discussion of the origin of this practice, what the attributed sources r...
5 CitationsSource
#1Inge Geyskens (Tilburg University)H-Index: 22
#2Rekha Krishnan (SFU: Simon Fraser University)H-Index: 5
Last. Paulo V. CunhaH-Index: 1
view all 4 authors...
213 CitationsSource
#1Frank L. Schmidt (UI: University of Iowa)H-Index: 75
#2In-Sue Oh (UI: University of Iowa)H-Index: 25
Last. Theodore L. Hayes (Gallup)H-Index: 4
view all 3 authors...
To da ym os tc onclusion sa bout cumulative knowledg ei np sycholog ya re base do nm etaanalysis .W efi rs tp resent an examinatio no ft he important statistica ld ifferences betwee n fixed-effects (FE) an dr andom-effect s( RE )m odel si nm eta-analysis an db etween tw o differen tR Ep rocedures, du et oH edge sa nd Vevea, an dt oH unte ra nd Schmidt. Th e implications of thes ed ifferences fo rt he appropriat ei nterpretatio no fp ublished meta analyses ar ee xplore db ya pplyin gt he tw oR Ep...
329 CitationsSource
#1Jose M. CortinaH-Index: 30
#2Ronald S. LandisH-Index: 24
81 Citations
#1Herman Aguinis (University of Colorado Denver)H-Index: 56
#2Charles A. Pierce (U of M: University of Memphis)H-Index: 30
Last. Ivan S. Muslin (U of M: University of Memphis)H-Index: 4
view all 4 authors...
The authors conducted a content analysis of the 193 articles published in the first 10 volumes (1998 to 2007) of Organizational Research Methods (ORM). The most popular quantitative topics are surveys, temporal issues, and electronic/Web research (research design); validity, reliability, and level of analysis of the dependent variable (measurement); and multiple regression/correlation, structural equation modeling, and multilevel research (data analysis). The most popular qualitative topics are ...
125 CitationsSource
Cited By121
Abstract Using gender homophily and gender socialization as theoretical foundations, the current study takes the position that both funder-driven (supply-side) and entrepreneur-driven (demand-side) processes perpetuate the gender gap in venture funding. Using this positional anchor, I performed a meta-analysis on gender-funding associations. The results show that gender-funding associations are different across funding contexts, which is consistent with what gender homophily and a funder-driven ...
#1Esther MaassenH-Index: 1
#2M.A.L.M. van AssenH-Index: 25
Last. Jelte M. WichertsH-Index: 39
view all 5 authors...
To determine the reproducibility of psychological meta-analyses, we investigated whether we could reproduce 500 primary study effect sizes drawn from 33 published meta-analyses based on the information given in the meta-analyses, and whether recomputations of primary study effect sizes altered the overall results of the meta-analysis. Results showed that almost half (k = 224) of all sampled primary effect sizes could not be reproduced based on the reported information in the meta-analysis, mostl...
1 CitationsSource
#1Antonio Sartal (University of Vigo)H-Index: 6
#2Miguel González-Loureiro (University of Vigo)H-Index: 10
Last. Xosé H. Vázquez (University of Vigo)H-Index: 10
view all 3 authors...
We analyze the weaknesses of meta-analyses (MAs) in management research using as benchmark a scientific field where this technique shows a longer tradition: clinical research. We suggest four areas...
#1Cort W. Rudolph (SLU: Saint Louis University)H-Index: 13
#2Caelen K. Chang (SLU: Saint Louis University)
Last. Hannes Zacher (Leipzig University)H-Index: 32
view all 4 authors...
Abstract Meta-analysis is a powerful tool for the synthesis of quantitative empirical research. Overall, the field of vocational behavior has benefited from the results of meta-analyses. Yet, there is still quite a bit to learn about how we can improve the quality of meta-analyses reported in this field of inquiry. In this paper, we systematically review all meta-analyses published in the Journal of Vocational Behavior (JVB) to date. We do so to address two related goals: First, based on guidanc...
Meta-analysis is frequently combined with multiple regression or path analysis to examine how the Big Five/Five-Factor Model (FFM) personality traits relate to work outcomes. A common approach in such studies is to construct a synthetic correlation matrix by combining new meta-analyses of FFM-criterion correlations with previously published meta-analytic FFM intercorrelations. Many meta-analytic FFM intercorrelation matrices exist in the literature, with 3 matrices being frequently used in indus...
1 CitationsSource
About 25 years ago, Kaplan and Norton (1992) introduced the Balanced Scorecard (BSC) with the aim of remedying the inadequacies of traditional performance measurement approaches. From then on, the ...
#1Frank A. Bosco (VCU: Virginia Commonwealth University)H-Index: 13
#2James G. Field (WVU: West Virginia University)H-Index: 7
Last. Krista L. Uggerslev (Northern Alberta Institute of Technology)H-Index: 12
view all 5 authors...
In this article, we provide a review of research-curation and knowledge-management efforts that may be leveraged to advance research and education in psychological science. After reviewing the appr...
1 CitationsSource
#1James G. FieldH-Index: 7
#2Frank A. BoscoH-Index: 13
Last. Sven KepesH-Index: 13
view all 3 authors...
Although systematic reviews are considered the primary means for generating cumulative knowledge and their results are often used to inform evidence-based practice, the robustness of their meta-analytic summary estimates is rarely investigated. Consequently, the results of published systematic reviews and, by extension, our cumulative knowledge have come under scrutiny. Using a comprehensive approach to sensitivity analysis, we examined the impact of outliers and publication bias, as well as the...
1 CitationsSource
#1Claudia Cristina Bitencourt (Universidade do Vale do Rio dos Sinos)H-Index: 6
#2Fernando de Oliveira Santini (Universidade do Vale do Rio dos Sinos)H-Index: 8
Last. Eduardo Kunzel Teixeira (PUCRS: Pontifícia Universidade Católica do Rio Grande do Sul)H-Index: 3
view all 5 authors...
Abstract This paper investigates, from meta-analytic approach, some topics that were unexplored in others dynamic capabilities meta-analysis. Hence, our research advances to identify and test the main antecedents of dynamic capabilities, the possible moderation effects of economic and cultural contexts in the relationship between dynamic capabilities and firm performance and the mediation effects between the antecedents of dynamic capabilities and firm performance. (1) The elements that stimulat...
3 CitationsSource