A Tale of Two Sample Sources: Do Results from Online Panel Data and Conventional Data Converge?

Published on Aug 1, 2019in Journal of Business and Psychology2.582
· DOI :10.1007/s10869-018-9552-y
Sheryl L. Walter4
Estimated H-index: 4
(IU: Indiana University),
Scott E. Seibert2
Estimated H-index: 2
(UO: University of Oregon)
+ 1 AuthorsErnest H. O’Boyle10
Estimated H-index: 10
(IU: Indiana University)
Samples drawn from commercial online panel data (OPD) are becoming more prevalent in applied psychology research, but they remain controversial due to concerns with data quality. In order to examine the validity of OPD, we conduct meta-analyses of online panel samples and compare internal reliability estimates for scales and effect size estimates for IV–DV relations commonly found in the field with those based on conventionally sourced data. Results based on 90 independent samples and 32,121 participants show OPD has similar psychometric properties and produces criterion validities that generally fall within the credibility intervals of existing meta-analytic results from conventionally sourced data. We suggest that, with appropriate caution, OPD are suitable for many exploratory research questions in the field of applied psychology.
  • References (122)
  • Citations (14)
📖 Papers frequently viewed together
26.7k Citations
5 Citations
78% of Scinapse members use related papers. After signing in, all features are FREE.
#1Ryan M. Vogel (PSU: Pennsylvania State University)H-Index: 7
#2Marie S. Mitchell (UGA: University of Georgia)H-Index: 15
This research considers two theoretical perspectives on employees’ motivation associated with diminished self-esteem from abusive supervision. The self-defense view of diminished self-esteem suggests that abusive supervision motivates destructive behavior in an attempt to reassert personal control and protect victims’ self-image. The self-presentational view of diminished self-esteem suggests abusive supervision motivates behavior that attempts to signal fit with and value to the workgroup and o...
16 CitationsSource
#1Jeremy D. Mackey (AU: Auburn University)H-Index: 11
#2Rachel E. Frieder (ODU: Old Dominion University)H-Index: 6
Last. Mark J. Martinko (FAMU: Florida A&M University)H-Index: 4
view all 4 authors...
We conducted a meta-analysis and empirical review of abusive supervision research in order to derive meta-analytic population estimates for the relationships between perceptions of abusive supervision and numerous demographic, justice, individual difference, leadership, and outcome variables. The use of psychometric correction enabled us to provide weighted mean correlations and population correlation estimates that accounted for attenuation due to measurement error and sampling error variance. ...
107 CitationsSource
#1Joseph Goodman (Max M. Fisher College of Business)H-Index: 12
#2Gabriele Paolacci (EUR: Erasmus University Rotterdam)H-Index: 13
Data collection in consumer research has progressively moved away from traditional samples (e.g., university undergraduates) and toward Internet samples. In the last complete volume of the Journal of Consumer Research (June 2015–April 2016), 43% of behavioral studies were conducted on the crowdsourcing website Amazon Mechanical Turk (MTurk). The option to crowdsource empirical investigations has great efficiency benefits for both individual researchers and the field, but it also poses new challe...
79 CitationsSource
#1Kathryn Sharpe Wessling (UPenn: University of Pennsylvania)H-Index: 1
#2Joel Huber (Duke University)H-Index: 41
Last. Oded Netzer (Columbia University)H-Index: 16
view all 3 authors...
This tutorial provides evidence that character misrepresentation in survey screeners by Amazon Mechanical Turk Workers (“Turkers”) can substantially and significantly distort research findings. Using five studies, we demonstrate that a large proportion of respondents in paid MTurk studies claim a false identity, ownership, or activity in order to qualify for a study. The extent of misrepresentation can be unacceptably high, and the responses to subsequent questions can have little correspondence...
54 CitationsSource
#1Jan-Willem van Prooijen (VU: VU University Amsterdam)H-Index: 21
#2Reinout E. de Vries (UT: University of Twente)H-Index: 28
Purpose Belief in conspiracy theories about societal events is widespread among citizens. The extent to which conspiracy beliefs about managers and supervisors matter in the micro-level setting of organizations has not yet been examined, however. We investigated if leadership styles predict conspiracy beliefs among employees in the context of organizations. Furthermore, we examined if such organizational conspiracy beliefs have implications for organizational commitment and turnover intentions. ...
14 CitationsSource
#1Jade S. Jenkins (NIU: Northern Illinois University)H-Index: 5
#2Camille J. Heneghan (NIU: Northern Illinois University)H-Index: 1
Last. Larissa K. Barber (NIU: Northern Illinois University)H-Index: 16
view all 4 authors...
In this investigation, we draw from the job demands–resource model and conservation of resources theory to examine the relationship between job demands, the work–family interface and worker behaviours. Data collected from an online survey of workers revealed that hindrance demands indirectly increase interpersonal and organizational deviance through work interference with family and family interference with work. Challenge demands indirectly predict interpersonal and organizational deviance thro...
7 CitationsSource
#1Mindy E. Bergman (A&M: Texas A&M University)H-Index: 20
#2Vanessa A. Jean (A&M: Texas A&M University)H-Index: 4
In this article, we demonstrate that samples in the industrial and organizational (I-O) psychology literature do not reflect the labor market, overrepresenting core, salaried, managerial, professional, and executive employees while underrepresenting wage earners, low- and medium-skill first-line personnel, and contract workers. We describe how overrepresenting managers, professionals, and executives causes research about these other workers to be suspect. We describe several ways that this under...
40 CitationsSource
#1Michael A. Gillespie (University of South Florida Sarasota–Manatee)H-Index: 10
#2Jennifer Z. Gillespie (University of South Florida Sarasota–Manatee)H-Index: 3
Last. William K. Balzer (BGSU: Bowling Green State University)H-Index: 20
view all 4 authors...
3 CitationsSource
#1Jesse S. Michel (AU: Auburn University)H-Index: 14
#2Kerry Newness (Florida Southern College)H-Index: 3
Last. Kris Duniewicz (FIU: Florida International University)H-Index: 2
view all 3 authors...
Purpose This research examines the linking mechanisms and conditional processes underlying the abusive supervision and workplace deviance relationship. Based primarily on Affective Events Theory, it was hypothesized that work-related negative affect would mediate the relationship between abusive supervision and workplace deviance, and that this indirect effect would be moderated by employee-based and organization-based aggressiveness.
25 CitationsSource
#1Caitlin M. Porter (Purdue University)H-Index: 6
#2Sang Eun Woo (Purdue University)H-Index: 15
Last. Jinkook Tak (Kwangwoon University)H-Index: 9
view all 3 authors...
The Protean and Boundaryless Career Attitudes scales developed by Briscoe and colleagues have facilitated research on career attitudes and mindsets, but they are unnecessarily lengthy and somewhat redundant in their content. To address these concerns, this article presents three studies that develop and validate short forms of the Protean and Boundaryless Career Attitudes scales (i.e., PCA-SF and BCA-SF). Study findings suggest that the PCA-SF and BCA-SF provide a more efficient assessment of th...
11 CitationsSource
Cited By14
#1Marc Schreiber (Zurich University of Applied Sciences/ZHAW)H-Index: 2
#2Gregor J. Jenny (UZH: University of Zurich)H-Index: 12
Abstract Positive and negative activation (PA/NA) represent two general activation systems of affect that are of importance for studying personality. Hereby, many studies focus on state assessment of PA and NA in everyday situations, using the ‘Experience Sampling Method’ (ESM) performed via mobile devices. ESM studies require short, reliable and validated non-verbal scales for immediate and fast capturing of personality and situation characteristics. In this study we present the non-verbal ‘Leb...
#1Breda McCarthy (JCU: James Cook University)H-Index: 12
#2Ariadne Beatrice Kapetanaki (Ebor: University of York)
Last. Pengji Wang (JCU: James Cook University)H-Index: 4
view all 3 authors...
Abstract Addressing the social, environmental and economic consequences of food waste has become an urgent issue for governments and industry stakeholders. This study investigates the factors influencing consumer acceptance of novel value-added surplus products (VASP) which could advance industry action to reduce food waste. Consumers' perceptions and potential acceptance of VASP is under-explored in the academic literature. A survey with a panel of 687 consumers in Australia and the UK was cond...
#1Justin M. Shepherd (UH: University of Houston)H-Index: 2
#2Jafar Bakhshaie (BCM: Baylor College of Medicine)H-Index: 1
Last. Michael J. Zvolensky (UH: University of Houston)H-Index: 66
view all 6 authors...
Although anxiety symptoms/disorders are consistently implicated in smoking processes, no empirical work has identified individual difference factors that may underlie such relations among Latinx smokers. The current study examined anxiety sensitivity as an explanatory factor underlying the relationship between anxiety symptoms and smoking expectancies among Spanish-speaking Latinx smokers. Participants included 363 Spanish-speaking Latinx daily smokers (58.7% female, Mage = 33.3 years, SD = 9.81...
#1Kandice J. Benallie (USU: Utah State University)
#2Maryellen Brunson McClain (USU: Utah State University)H-Index: 1
Last. Sarah Schwartz (USU: Utah State University)H-Index: 10
view all 4 authors...
Parents play a critical role in the early identification of ASD because of their experiential knowledge and frequent observations of their children. Being knowledgeable about ASD may help parents recognize early signs and symptoms, know to which professionals to express their concerns, and better navigate systems of care. An appropriate measure of ASD knowledge for parents is essential to further understand the importance of ASD knowledge in this population. This study sought to validate the Aut...
#1Christopher M. Barnes (UW: University of Washington)H-Index: 24
#2Eli Awtrey (UC: University of Cincinnati)H-Index: 3
Last. Gretchen M. Spreitzer (UM: University of Michigan)H-Index: 40
view all 4 authors...
OBJECTIVES: The objectives of this study is to examine the effect of leader sleep devaluation (which we define as leader behaviors that signal to employees that sleep should be sacrificed for work) on the sleep and unethical behavior of subordinates. DESIGN: Across 2 studies (with 3 total samples of participants), we use a cross-sectional survey, a diary study completed by employees, and a diary study completed by employees and their leaders. SETTING: Study 1 - a convenience sample of working ad...
#1Chloe A. Wilson (Clemson University)H-Index: 1
#2Thomas W. Britt (Clemson University)H-Index: 35
ABSTRACTThe present study examined how occupational calling affects the relationships between challenge and hindrance stressors, work motivation, and mental health symptoms. Individuals experience ...
#1Nubia A. Mayorga (UH: University of Houston)H-Index: 5
#2Lorra Garey (UH: University of Houston)H-Index: 9
Last. Michael J. Zvolensky (UH: University of Houston)H-Index: 66
view all 5 authors...
BACKGROUND AND OBJECTIVES: Despite greater rates of cannabis use among those that smoke combustible cigarettes, it is currently unknown whether cannabis use is related to e-cigarette dependence or maladaptive beliefs about combustible cigarettes. Therefore, the current study sought to identify whether adult dual users of combustible and e-cigarettes (ie, dual users) who also used cannabis differed from dual users who did not use cannabis on e-cigarette dependence severity, perceived barriers to ...
#1Michael J. Zvolensky (UH: University of Houston)H-Index: 66
#2Justin M. Shepherd (UH: University of Houston)H-Index: 2
Last. Jodi Berger-Cardoso (UH: University of Houston)H-Index: 1
view all 7 authors...
AbstractObjectives: Although acculturative stress is a known individual difference factor related to poor health, little is understood about its relationship to smoking among Latinx adults. One tra...
#1Michael J. Zvolensky (UH: University of Houston)H-Index: 66
#2Johann D'Souza (UH: University of Houston)H-Index: 1
Last. Matthew W. Gallagher (UH: University of Houston)H-Index: 7
view all 7 authors...
Abstract Introduction Although subjective sleep quality has been associated with combustible cigarette use, little is known about its role in electronic cigarette (e-cigarette) use or beliefs about use. To address this gap, the current study examined subjective sleep quality among adult e-cigarette users in relation to e-cigarette dependence, perceived risks of e-cigarette use, and perceived barriers for quitting e-cigarettes. Methods A cross-section design was employed. Participants included 30...
1 CitationsSource