Researching through Mechanical Turk

Over the past four decades, countless Calvin students taking “Introduction to Psychology” have served as subjects for faculty research projects. Now Calvin’s Center for Social Research has a new tool available to dramatically broaden the base for social-science research through an Amazon service: Mechanical Turk.

Mechanical Turk is an online job board that connects employers with “Turkers” who pick up human intelligence jobs to make some extra money. Typical Turker tasks include tagging photos, writing product descriptions and other small jobs that computers cannot do.  As the popularity of Mechanical Turk has grown and the number of Turkers increased, social scientists adopted the application for conducting research through the site.

This year, Calvin’s Center for Social Research (CSR) began using Mechanical Turk to obtain a broader survey base for several faculty research projects. CSR Research Associate Tom Sherwood explains, “Instead of relying on students from introductory psychology classes as subjects, we get thousands of survey responses from a wide variety of people in just a few weeks.” Studies have shown that the population of Turkers is close enough to the general population to produce valid results.

Surveying wide

Turkers are paid a minimal amount of money (micropayments), typically less than half a dollar for completing surveys. Payments are distributed through Amazon, eliminating the cost and hassle it would require to send small payments to a large number of survey respondents.

Calvin has ben intentional about paying survey-takers fair wages, said CSR director Neil Carlson: “We have tried to ensure that the average pay per hour is at least the U.S. federal minimum of $7.25, but those who work slowly may make less.” Carlson recognizes that paying for survey completion is not the norm. “It reveals the tension between traditional research, which has often been done by volunteers for no pay or a small gratuity and research-as-a-business. We may have to choose which model we adhere to for each study and communicate clearly to Turkers whether the study pays a wage or just a gratuity,” he said.

“Professor Jon Hill introduced us to the idea of using Mechanical Turk for social research,” said Sherwood of the Calvin assistant professor of sociology. “Last summer, he proposed using Mechanical Turk for his survey on how wording and question ordering affect results on religious surveys.” CSR staff worked with Professor Hill, and his survey resulted in nearly 3,500 responses, yielding high-quality data.

Prof. Hill hypothesized that the way a question is asked may alter responses. For example, “Do you pray regularly? How often do you pray?” would result in a significantly different response than simply asking “How often do you pray?” In addition, his survey contained questions in three sections: religious, demographic and political. Hill concluded that the question asked first impacted the results.

Yonker’s research revisits Science article

Following in Hill’s footsteps, assistant psychology Professor Julie Yonker wanted to dig deeper after reading an article in Science magazine entitled “Analytical Thinking Promotes Religious Disbelief,” which found that people who thought more analytically tend to be less religious.

In collaboration with several other researches, Yonker’s desire to replicate the experiment used in the Science article necessitated surveying a broad slice of the population. “I conducted face-to-face interviews with Calvin students,” said Yonker, “but I couldn’t replicate the results that the Science article found, so I designed a large-scale survey to supplement the data.”

Sherwood added, “More than 5,000 people responded to Professor Yonker’s survey on Mechanical Turk, so she has a large data set to analyze.” Yonker and psychology student Lauren Bender are currently cleaning up the data. She explained, “There are some invalid responses that have to be removed. For example, some Turkers completed the survey too quickly, indicating that they were just randomly answering questions rather than reading them thoroughly, so I am eliminating their responses from the data.” The results of Yonker’s research will be released in May.

History and future

The name “Mechanical Turk” originates from an 18^th-century machine, designed to look like a Turkish person. The machine was heralded as a rival to any human chess opponent. The Mechanical Turk toured the world, beating many rivals, including Napoleon Bonaparte and Benjamin Franklin and fascinating audiences who were dumfounded by how a machine could play chess. In reality, there was a human chess player hidden inside a compartment within the machine, controlling the Turk’s moves.

With two projects conducted through Mechanical Turk, CSR has worked out many of the bugs. “We figured out a way to prevent Turkers from taking a survey more than once,” said Sherwood. “We are using our new Qualtrics survey software to require that respondents provide their Turk user ID for authentication.”

This innovative approach to ensuring each survey is unique and will be useful to other researches as well. Sherwood explained, “From our research, it appears that others haven’t come up with this solution to the problem of duplicate survey responses, so we’re planning to share what we’re doing with the broader community.”

“Using Mechanical Turk has great potential for future research projects as well,” said Yonker. “It has opened up a wonderful new avenue for conducting large-scale social science surveys.”