Match!

Conducting behavioral research on Amazon's Mechanical Turk

Published on Mar 1, 2012in Behavior Research Methods4.06
· DOI :10.3758/s13428-011-0124-6
Winter Mason17
Estimated H-index: 17
(Yahoo!),
Siddharth Suri24
Estimated H-index: 24
(Yahoo!)
Abstract
Amazon’s Mechanical Turk is an online labor market where requesters post jobs and workers choose which jobs to do for pay. The central purpose of this article is to demonstrate how to use this Web site for conducting behavioral research and to lower the barrier to entry for researchers who could benefit from this platform. We describe general techniques that apply to a variety of types of research and experiments across disciplines. We begin by discussing some of the advantages of doing experiments on Mechanical Turk, such as easy access to a large, stable, and diverse subject pool, the low cost of doing experiments, and faster iteration between developing theory and executing experiments. While other methods of conducting behavioral research may be comparable to or even better than Mechanical Turk on one or more of the axes outlined above, we will show that when taken as a whole Mechanical Turk can be a useful tool for many researchers. We will discuss how the behavior of workers compares with that of experts and laboratory subjects. Then we will illustrate the mechanics of putting a task on Mechanical Turk, including recruiting subjects, executing the task, and reviewing the work that was submitted. We also provide solutions to common problems that a researcher might face when executing their research on this platform, including techniques for conducting synchronous experiments, methods for ensuring high-quality work, how to keep data private, and how to maintain code security.
Figures & Tables
  • References (64)
  • Citations (1833)
References64
Newest
#1Michael D. Buhrmester (University of Texas at Austin)H-Index: 14
#2Tracy Kwang (University of Texas at Austin)H-Index: 5
Last.Samuel D. Gosling (University of Texas at Austin)H-Index: 53
view all 3 authors...
#1Andrew Mao (Harvard University)H-Index: 10
#2David C. Parkes (Harvard University)H-Index: 45
Last.Haoqi Zhang (Harvard University)H-Index: 15
view all 4 authors...
Jul 25, 2010 in KDD (Knowledge Discovery and Data Mining)
#1Lydia B. Chilton (UW: University of Washington)H-Index: 15
#2John J. Horton (Harvard University)H-Index: 14
Last.Shiri Azenkot (UW: University of Washington)H-Index: 18
view all 4 authors...
Jul 25, 2010 in KDD (Knowledge Discovery and Data Mining)
#1Greg Little (MIT: Massachusetts Institute of Technology)H-Index: 19
#2Lydia B. Chilton (UW: University of Washington)H-Index: 15
Last.Robert C. Miller (MIT: Massachusetts Institute of Technology)H-Index: 49
view all 4 authors...
Jul 25, 2010 in KDD (Knowledge Discovery and Data Mining)
#1Panagiotis G. Ipeirotis (NYU: New York University)H-Index: 36
#2Foster Provost (NYU: New York University)H-Index: 53
Last.Jing Wang (NYU: New York University)H-Index: 8
view all 3 authors...
Cited By1833
Newest
#1Yukuan Xu (PolyU: Hong Kong Polytechnic University)H-Index: 1
#2Hengyun Li (PolyU: Hong Kong Polytechnic University)H-Index: 10
Last.Ziqiong Zhang (HIT: Harbin Institute of Technology)H-Index: 12
view all 4 authors...
#1D Sathya (Kumaraguru College of Technology)
#2V Sudha (Kumaraguru College of Technology)
Last.D Jagadeesan (Science College)
view all 3 authors...
#1Rita Orji (Dal: Dalhousie University)H-Index: 15
#2Kiemute Oyibo (U of S: University of Saskatchewan)H-Index: 7
Last.Fidelia A. Orji (U of S: University of Saskatchewan)H-Index: 3
view all 4 authors...
#1Siyu Yu (NYU: New York University)H-Index: 4
#2Lindred L. Greer (Stanford University)H-Index: 22
Last.Lisanne van Bunderen (UvA: University of Amsterdam)H-Index: 3
view all 4 authors...
#1Maricarmen Vizcaino (ASU: Arizona State University)
#2Matthew P. Buman (ASU: Arizona State University)H-Index: 29
Last.Christopher Wharton (ASU: Arizona State University)H-Index: 16
view all 4 authors...
#1Sarah Lageson (RU: Rutgers University)H-Index: 8
#2Suzy McElrath (UMN: University of Minnesota)H-Index: 4
Last.Krissinda Ellen Palmer (RU: Rutgers University)H-Index: 1
view all 3 authors...
#1Valerio Capraro (Lond: University of London)H-Index: 18
#2Glorianna Jagfeld (University of Stuttgart)H-Index: 2
Last.Iris van de Pol (UvA: University of Amsterdam)H-Index: 2
view all 5 authors...
View next paperCheap and Fast -- But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks