Crowdsourcing: Bringing psychological science online

The use of online sampling and crowdsourcing is a rapidly growing trend in psychological science. Broadly speaking, crowdsourcing refers to the process of obtaining goods, needs or services through the solicitation of large groups of people. In particular, crowdsourcing refers to the use of the internet to achieve these ends, with popular examples including Kickstarter (i.e., using crowds to fund your idea or crowdfunding) and Amazon.com’s Mechanical Turk (mTurk). But what is crowdsourced research, why use it and how can you get started with your own study?

Crowdsourced research uses online platforms to recruit large and diverse groups of research participants, with mTurk being the primary platform for psychological research. On mTurk “requesters” (i.e., researchers) can advertise tasks (i.e., studies) that “workers” (i.e., participants) can choose to complete. In this way, mTurk may be thought as similar to PSY101 participation pools in which students can choose to participate in various research studies. Unlike undergraduate samples, however, crowdsourced research can sample from a pool of participants with greater demographic variability. Although most studies conducted using crowdsourcing are surveys, crowdsourced platforms can be adapted for other, more complex tasks. For example, delay discounting tasks have been presented on mTurk with outcomes consistent with traditional laboratory samples (Johnson et al. 2015).¹

The benefits of using crowdsourcing are clear, and many of these advantages are particularly valuable for graduate students. First, crowdsourcing research dramatically cuts the cost associated with in-person laboratory studies while facilitating the rate of data collection. Depending on the inclusion criteria of a study, sample sizes in the hundreds can be achieved within days. Second, researchers can efficiently screen participants helping to increase overall data quality and decrease wasted resources on nonqualifying participants. Third, the diversity of the mTurk population provides an opportunity to reach historically hard-to-sample populations. These may include individuals with specific psychiatric diagnoses (e.g., personality disorders) and/or substance-use histories (e.g., illicit drug use). Finally, the use of online samples can increase the external validity of research studies by providing more geographically and demographically diverse samples than traditional research. This strength is particularly notable for researchers historically limited to sampling from PSY101 pools.

There are many resources to turn to if you want to use crowdsourcing in your research. Numerous internet articles and blogs describe tips, tricks and tutorials on using mTurk.² Recent studies in psychopharmacology and related fields using mTurk can also serve as a guide for designing your own study (for examples see Koffarnus et al. 2015;³Rass et al. 2015;⁴Strickland & Stoops 2015⁵). Finally, you can look to other researchers at your institution who might have used crowdsourcing techniques (whether in psychology or not). This step can be particularly helpful given that IRB requirements for conducting online research may vary institution to institution. Hopefully with these tools in hand to designing and conducting your own crowdsourced study.

References

¹ Johnson, P.S., Herrmann, E.S., & Johnson, M.W. (2015). Opportunity costs of reward delays and the discounting of hypothetical money and cigarettes. Journal of the Experimental Analysis of Behavior, 103, 87-107.

² http://turkrequesters.blogspot.com

³ Koffarnus, M.N., Franck, C.T., Stein, J.S., Bickel, W.K. (2015). A modified exponential behavioral economic demand model to better describe consumption data. Experimental and Clinical Psychopharmacology, 23, 504-512.

⁴ Rass, O., Pacek, L.R., Johnson, P.S., & Johnson, M.W. (2015). Characterizing use patterns and perceptions of relative harm in dual users of electronic and tobacco cigarettes. Experimental and Clinical Psychopharmacology, 23, 494-503.

⁵ Strickland, J.C., & Stoops, W.W. (2015). Perceptions of research risk and undue influence: Implications for ethics of research conducted with cocaine users. Drug and Alcohol Dependence, 156, 304-310.