Match!

On doing relevant and rigorous experiments: Review and recommendations

Published on Nov 1, 2018in Journal of Operations Management7.78
· DOI :10.1016/j.jom.2018.10.003
Sirio Lonati1
Estimated H-index: 1
(UNIL: University of Lausanne),
Bernardo F. Quiroga2
Estimated H-index: 2
(UC: Pontifical Catholic University of Chile)
+ 1 AuthorsJohn Antonakis30
Estimated H-index: 30
(UNIL: University of Lausanne)
Cite
Abstract
Abstract Although experiments are the gold standard for establishing causality, several threats can undermine the internal validity of experimental findings. In this article, we first discuss these threats, which include the lack of consequential decisions and outcomes, deception, demand effects and unfair comparisons, as well as issues concerning statistical validity (e.g., minimum sample size per cell, estimating variance correctly). We expose each problem, show potential solutions, and bring to the fore issues of relevance of the findings (i.e., external and ecological validity). Thereafter, we take stock of the state-of-the-science regarding validity threats using a representative sample of 468 recent experiments from 258 articles published in top-tier journals. We compare research practices in three fields of study—management, social psychology, and economics, which regularly use experimental research—to operations management, which has more recently begun to use the experimental paradigm. Our results underscore the importance for journals and authors to follow what we identify to be best-practice methodological suggestions (i.e., the “ten commandments” of experimental research). We show that—on average—markers of methodological rigor and generalizability positively and significantly predict the citations received by published articles. Finally, given that experiments are infeasible in some settings, we conclude with a brief review of often overlooked quasi-experimental designs, which are useful for generating strong counterfactuals and hence allow making causal claims in the field.
  • References (205)
  • Citations (2)
Cite
References205
Newest
Published on Feb 1, 2019in Leadership Quarterly5.63
Philip M. Podsakoff5
Estimated H-index: 5
(College of Business Administration),
Nathan P. Podsakoff21
Estimated H-index: 21
(UA: University of Arizona)
Abstract Despite the renewed interest in the use of experimental designs in the fields of leadership and management over the past few decades, these designs are still relatively underutilized. Although there are several potential reasons for this, chief among them is misunderstanding the value of these designs. The purpose of this article is to review the role of laboratory, field, and quasi-experimental designs in management and leadership research. We first discuss the primary goals of experim...
Michael Schaerer7
Estimated H-index: 7
(Singapore Management University),
Christilene du Plessis1
Estimated H-index: 1
(Singapore Management University)
+ 1 AuthorsStefan Thau15
Estimated H-index: 15
(Ad: INSEAD)
We examine the role of low-power individuals in social power research. A multi-method literature review reveals that low-power individuals may be insufficiently understood because many studies lack necessary control conditions that allow drawing inferences about low power, effects are predominantly attributed to high power, and qualitative reviews primarily focus on how high-power individuals feel, think, and behave. Challenging the assumption that low power tends to produce opposite consequence...
Published on Oct 1, 2018in Leadership Quarterly5.63
Paulo Roberto Arvate1
Estimated H-index: 1
,
Gisele Walczak Galilea1
Estimated H-index: 1
(Insper),
Isabela Todescat1
Estimated H-index: 1
Abstract We investigate the effect of female leadership on gender differences in public and private organizations. Female leadership impact was constructed using a quasi-experiment involving mayoral elections, and our research used a sample of 8.3 million organizations distributed over 5600 Brazilian municipalities. Our main results show that when municipalities in which a woman was elected leader (treatment group) are compared with municipalities in which a male was elected leader (control grou...
Published on Jul 1, 2018in American Journal of Political Science4.35
Jacob M. Montgomery9
Estimated H-index: 9
(WashU: Washington University in St. Louis),
Brendan Nyhan16
Estimated H-index: 16
(Dartmouth College),
Michelle Torres1
Estimated H-index: 1
(WashU: Washington University in St. Louis)
Published on Feb 1, 2018in Leadership Quarterly5.63
Lindie H. Liang4
Estimated H-index: 4
(WLU: Wilfrid Laurier University),
Douglas J. Brown29
Estimated H-index: 29
(UW: University of Waterloo)
+ 3 AuthorsLisa M. Keeping14
Estimated H-index: 14
(WLU: Wilfrid Laurier University)
Abstract When a subordinate receives abusive treatment from a supervisor, a natural response is to retaliate against the supervisor. Although retaliation is dysfunctional and should be discouraged, we examine the potential functional role retaliation plays in terms of alleviating the negative consequences of abusive supervision on subordinate justice perceptions. Based on the notion that retaliation following mistreatment can restore justice for victims, we propose a model whereby retaliation fo...
Published on Jan 1, 2018in Organizational Research Methods6.55
Juan Carlos Bou3
Estimated H-index: 3
(James I University),
Albert Satorra Brucart29
Estimated H-index: 29
(UPF: Pompeu Fabra University)
Two approaches are commonly in use for analyzing panel data: the univariate, which arranges data in long format and estimates just one regression equation; and the multivariate, which arranges data in wide format, and simultaneously estimates a set of regression equations. Although technical articles relating the two approaches exist, they do not seem to have had an impact in organizational research. This article revisits the connection between the univariate and multivariate approaches, elucida...
Published on Nov 1, 2017in Journal of Operations Management7.78
James D. Abbey8
Estimated H-index: 8
(A&M: Texas A&M University),
Margaret G. Meloy14
Estimated H-index: 14
(PSU: Pennsylvania State University)
Abstract This paper examines attention checks and manipulation validations to detect inattentive respondents in primary empirical data collection. These prima facie attention checks range from the simple such as reverse scaling first proposed a century ago to more recent and involved methods such as evaluating response patterns and timed responses via online data capture tools. The attention check validations also range from easily implemented mechanisms such as automatic detection through direc...
Published on Aug 1, 2017
Nicolas Bastardoz3
Estimated H-index: 3
,
Philippe Jacquart5
Estimated H-index: 5
(EMLYON Business School),
John Antonakis30
Estimated H-index: 30
When observations are not randomly assigned to conditions, assumptions of traditional estimation methods such as ANOVA or linear regression will be violated and can lead to biased and inconsistent estimators. However, there exist quasi-experimental designs that can be used to infer causality. One of these designs, the Regression Discontinuity (RD) design, allows for the drawing of proper causal conclusions when observations are not randomized to groupings, as long as the selection process is cor...
Published on Jul 1, 2017in Journal of Management9.06
Thomas Fischer1
Estimated H-index: 1
(UNIL: University of Lausanne),
Joerg Dietz19
Estimated H-index: 19
(UNIL: University of Lausanne),
John Antonakis30
Estimated H-index: 30
(UNIL: University of Lausanne)
In organizational research, studying “processes” is important for uncovering and understanding the underlying causal mechanisms in a predictor-mediator-outcome logic. Processes answer “how” and “why” questions and provide more complete explanations about phenomena. Our focus in this review is on studies of leadership processes, which we systematically analyze to report on the state of the science. In doing so, we present a two-dimensional target-centric taxonomy to integrate previous research: T...
Cited By2
Newest
Published on Aug 1, 2019in Leadership Quarterly5.63
Sirio Lonati1
Estimated H-index: 1
(UNIL: University of Lausanne)
Abstract Why do we observe either participative or directive leadership in organizations? I test an evolutionary-informed theory suggesting that organizational leadership is currently less participative (i.e., close supervision, rare delegation) among societies that used intensive forms of agriculture in the past. Intensification caused increased social complexity and skewed power distribution, promoting the emergence of directive leaders and eventually shaping followers' preferences for and per...
Published on Feb 1, 2019in Leadership Quarterly5.63
Philip M. Podsakoff5
Estimated H-index: 5
(College of Business Administration),
Nathan P. Podsakoff21
Estimated H-index: 21
(UA: University of Arizona)
Abstract Despite the renewed interest in the use of experimental designs in the fields of leadership and management over the past few decades, these designs are still relatively underutilized. Although there are several potential reasons for this, chief among them is misunderstanding the value of these designs. The purpose of this article is to review the role of laboratory, field, and quasi-experimental designs in management and leadership research. We first discuss the primary goals of experim...
View next paperOn the Generalizability of Experimental Results in Economics: With a Response to Camerer