On doing relevant and rigorous experiments: Review and recommendations

Published on Nov 1, 2018in Journal of Operations Management7.776
· DOI :10.1016/J.JOM.2018.10.003
Sirio Lonati1
Estimated H-index: 1
(UNIL: University of Lausanne),
Bernardo F. Quiroga4
Estimated H-index: 4
(UC: Pontifical Catholic University of Chile)
+ 1 AuthorsJohn Antonakis34
Estimated H-index: 34
(UNIL: University of Lausanne)
Abstract Although experiments are the gold standard for establishing causality, several threats can undermine the internal validity of experimental findings. In this article, we first discuss these threats, which include the lack of consequential decisions and outcomes, deception, demand effects and unfair comparisons, as well as issues concerning statistical validity (e.g., minimum sample size per cell, estimating variance correctly). We expose each problem, show potential solutions, and bring to the fore issues of relevance of the findings (i.e., external and ecological validity). Thereafter, we take stock of the state-of-the-science regarding validity threats using a representative sample of 468 recent experiments from 258 articles published in top-tier journals. We compare research practices in three fields of study—management, social psychology, and economics, which regularly use experimental research—to operations management, which has more recently begun to use the experimental paradigm. Our results underscore the importance for journals and authors to follow what we identify to be best-practice methodological suggestions (i.e., the “ten commandments” of experimental research). We show that—on average—markers of methodological rigor and generalizability positively and significantly predict the citations received by published articles. Finally, given that experiments are infeasible in some settings, we conclude with a brief review of often overlooked quasi-experimental designs, which are useful for generating strong counterfactuals and hence allow making causal claims in the field.
  • References (206)
  • Citations (21)
📖 Papers frequently viewed together
739 Citations
103 Citations
18 Citations
78% of Scinapse members use related papers. After signing in, all features are FREE.
#1Marcus Credé (Iowa State University)H-Index: 23
Cuddy, Schultz and Fosse (2018) present the results of p-curve analyses that are interpreted as providing "clear evidential value for power posing effects”. This commentary highlights that the vast majority of the studies included in the p-curve analyses were not designed in a way that could speak to the efficacy of power poses relative to a normal or neutral pose. Further, I discuss how the few studies that were designed to shed light on this issue indicate that any overall effect of physical p...
3 CitationsSource
#1Philip M. Podsakoff (College of Business Administration)H-Index: 55
#2Nathan P. Podsakoff (UA: University of Arizona)H-Index: 22
Abstract Despite the renewed interest in the use of experimental designs in the fields of leadership and management over the past few decades, these designs are still relatively underutilized. Although there are several potential reasons for this, chief among them is misunderstanding the value of these designs. The purpose of this article is to review the role of laboratory, field, and quasi-experimental designs in management and leadership research. We first discuss the primary goals of experim...
18 CitationsSource
#1Marcus CredeH-Index: 1
2 CitationsSource
#1Michael Schaerer (Singapore Management University)H-Index: 9
#2Christilene du Plessis (Singapore Management University)H-Index: 3
Last. Stefan Thau (Ad: INSEAD)H-Index: 17
view all 4 authors...
Abstract We examine the role of low-power individuals in social power research. A multi-method literature review reveals that low-power individuals may be insufficiently understood because many studies lack necessary control conditions that allow drawing inferences about low power, effects are predominantly attributed to high power, and qualitative reviews primarily focus on how high-power individuals feel, think, and behave. Challenging the assumption that low power tends to produce opposite co...
16 CitationsSource
Last. Isabela TodescatH-Index: 1
view all 3 authors...
Abstract We investigate the effect of female leadership on gender differences in public and private organizations. Female leadership impact was constructed using a quasi-experiment involving mayoral elections, and our research used a sample of 8.3 million organizations distributed over 5600 Brazilian municipalities. Our main results show that when municipalities in which a woman was elected leader (treatment group) are compared with municipalities in which a male was elected leader (control grou...
13 CitationsSource
#1Lindie H. Liang (WLU: Wilfrid Laurier University)H-Index: 7
#2Douglas J. Brown (UW: University of Waterloo)H-Index: 33
Last. Lisa M. Keeping (WLU: Wilfrid Laurier University)H-Index: 14
view all 6 authors...
Abstract When a subordinate receives abusive treatment from a supervisor, a natural response is to retaliate against the supervisor. Although retaliation is dysfunctional and should be discouraged, we examine the potential functional role retaliation plays in terms of alleviating the negative consequences of abusive supervision on subordinate justice perceptions. Based on the notion that retaliation following mistreatment can restore justice for victims, we propose a model whereby retaliation fo...
10 CitationsSource
#1Jacob M. Montgomery (WashU: Washington University in St. Louis)H-Index: 11
#2Brendan Nyhan (Dartmouth College)H-Index: 22
Last. Michelle Torres (WashU: Washington University in St. Louis)H-Index: 2
view all 3 authors...
86 CitationsSource
#1Juan Carlos Bou (James I University)H-Index: 4
#2Albert Satorra (UPF: Pompeu Fabra University)H-Index: 30
Last. Albert Satorra Brucart (UPF: Pompeu Fabra University)
view all 2 authors...
Two approaches are commonly in use for analyzing panel data: the univariate, which arranges data in long format and estimates just one regression equation; and the multivariate, which arranges data...
4 CitationsSource
#1Nicolas BastardozH-Index: 5
#2Philippe Jacquart (EMLYON Business School)H-Index: 6
Last. John AntonakisH-Index: 34
view all 3 authors...
When observations are not randomly assigned to conditions, assumptions of traditional estimation methods such as ANOVA or linear regression will be violated and can lead to biased and inconsistent ...
2 CitationsSource
#1James D. Abbey (A&M: Texas A&M University)H-Index: 8
#2Margaret G. Meloy (PSU: Pennsylvania State University)H-Index: 16
Abstract This paper examines attention checks and manipulation validations to detect inattentive respondents in primary empirical data collection. These prima facie attention checks range from the simple such as reverse scaling first proposed a century ago to more recent and involved methods such as evaluating response patterns and timed responses via online data capture tools. The attention check validations also range from easily implemented mechanisms such as automatic detection through direc...
31 CitationsSource
Cited By21
#1Brian Cooper (Monash University)H-Index: 29
#2Nathan Eva (Monash University)H-Index: 7
Last. Martin Obschonka (QUT: Queensland University of Technology)H-Index: 23
view all 6 authors...
Abstract For various reasons, research on vocational behavior often has to rely on self-report measures. Although it is well known that the exclusive use of self-report measures in a given study can pose major threats to the validity of that study, for example via common method variance (CMV) or endogeneity, we still have to witness a systematic literature review of study designs particularly prone to CMV and endogeneity, and of procedural and statistical remedies used in the field of vocational...
#1Philippe P.F.M. van de Calseyde (TU/e: Eindhoven University of Technology)H-Index: 2
#2Anthony M. Evans (Tilburg University)H-Index: 13
Last. Evangelia Demerouti (TU/e: Eindhoven University of Technology)H-Index: 66
view all 3 authors...
Abstract Decision speed is emerging as an important topic to organizations, yet its consequences for leaders have received little research attention. The present research builds on this notion by examining how the speed with which leaders come to decisions shapes observers' perceptions and behaviors. In three incentivized experiments, participants evaluated leaders who decided whether to include or exclude followers from participating in consequential decisions. Leaders were seen as more honest ...
#1Lars van Tuin (UU: Utrecht University)
#2Wilmar B. Schaufeli (UU: Utrecht University)H-Index: 107
view all 4 authors...
The present quasi-experimental study tested the business impact of a leadership development program focusing on psychological well-being through the satisfaction of basic psychological needs. Based on the concept of engaging leadership and self-determination theory, the 8-month program targeted midlevel team leaders of the customer fulfilment center of a health systems multinational organization. The program was designed in co-creation between senior leadership and the team leaders that particip...
#1M. Ghufran Ahmad (LUMS: Lahore University of Management Sciences)H-Index: 1
#2Jawad Syed (LUMS: Lahore University of Management Sciences)H-Index: 21
This article presents a theory of how the consumption of Islamic products may promote morally consistent behavior over time. We identify and examine three unique meta-characteristics (MCs) of Islam...
#1Nicoleta Meslec (Tilburg University)H-Index: 6
#2Petru Lucian Curşeu (OU: Open University)H-Index: 22
Last. Renata Kenda (Tilburg University)H-Index: 2
view all 4 authors...
Abstract We report two replication attempts for the positive main effects of charismatic leadership and performance-based rewards on individual performance from a field experiment in a charitable context (Antonakis, d'Adda, Weber, and Zehnder, 2015). Using video based treatments - instead of the live treatments in the original study - we only replicate the effect of performance-based rewards in a sample of 118 participants; we do not find an effect of charisma on performance. In a second study, ...
#1Harry Garretsen (UG: University of Groningen)H-Index: 38
#2Janka I. Stoker (UG: University of Groningen)H-Index: 16
Last. Roberto A. Weber (UZH: University of Zurich)H-Index: 30
view all 3 authors...
Abstract The fields of leadership and economics have interacted very little until now. The aim of this special issue on “economics and leadership” is to demonstrate the potential benefits of paying attention to and incorporating economic perspectives and methods into leadership research. Specifically, we argue that the field of economics can advance leadership research on three main topics: concepts, causality and context. For each topic, we summarize the state-of-the-art literature and showcase...
ABSTRACTIn behavioural interviews, past-behaviour questions invite applicants to tell a story about a past job-related situation. Nevertheless, applicants often do not produce stories on demand, re...
Abstract Omitted variables create endogeneity and thus bias the estimation of the causal effect of measured variables on outcomes. Such measured variables are ubiquitous and include perceptions, attitudes, emotions, behaviors, and choices. Even experimental studies are not immune to the endogeneity problem. I propose a solution to this challenge: Experimentally randomized instrumental variables (ERIVs), which can correct for endogeneity bias via instrumental variable estimation. Such ERIVs can b...
4 CitationsSource
#1Sirio Lonati (UNIL: University of Lausanne)H-Index: 1
Abstract Why do we observe either participative or directive leadership in organizations? I test an evolutionary-informed theory suggesting that organizational leadership is currently less participative (i.e., close supervision, rare delegation) among societies that used intensive forms of agriculture in the past. Intensification caused increased social complexity and skewed power distribution, promoting the emergence of directive leaders and eventually shaping followers' preferences for and per...