David Rand at Yale, who originally wrote this document,
and the TurkShop Google Group which surfaced it.
There are a lot of MTurk workers who just want to click through the study as fast as
possible, and you don’t want these people in your study. One way to screen them out is to
begin the HIT by having subjects transcribe a few sentences of handwritten nonsense text
into a text box.
It’s important to include comprehension questions about the game/task/stimuli to make
sure they actually read/understood it (you can either exclude based on comprehension, or
just include a control for it; the latter is what I typically do these days).
If you repost a HIT, many of the same people who did the previous posting(s) will try to
do it again. So it’s very important that you have some mechanism in place to screen out
repeat participants. One such way is to have the first question in the survey ask them to
enter their MTurk WorkerID, and then match that against a list of previous participants’
WorkerIDs. See the attached appendix put together by Alex Peysakhovich showing how
to do this in Qualtrics using embedded data. [Ed note: This appendix is not included in this
page. For alternate solutions see our pages on preventing workers from participating multiple times.]
MTurk tries hard to prevent people from having multiple IDs. But sometimes they fail.
So once you get all your data, it’s important to filter out repeated observations from the
same IP address (and so that means it’s also important to capture the IP address of each
subject – most survey software should have this functionality).
Workers are willing to wait a few days to receive their bonuses without getting annoyed.
But they hate waiting to have their HITs approved (that is, to get their showup fees). So I
try to accept submitted HITs as they come in (I check every 30 min or so and approve all
the new HITs) and then pay the bonuses within a couple of days of the last data getting
Building up a reputation as a good requester (via ratings received from workers on sites
like Turkopticon) can be good for getting lots of workers to do your HITs quickly. But I
think it can also be bad, in that it encourages the same workers to do lots of your HITs,
and may also change the subset of Turkers willing to do your HITs. So I recommend
changing your requester name with some regularity.
Another reason to change your name / actually make a new account with some regularity
is that people can also sign up to receive notifications every time you post a new HIT. If
you run lots of similar things and don’t want to same people in all your studies, this can
Related to both of these issues, it’s a good idea to include a question in the demographics
questionnaire about previous experience with similar studies, again to use either as a
control or an exclusion criterion.
Also, you should be aware that if you are running a high paying HIT, it may get posted
on forums (for example reddit has a section on good HITs). This effects both the type of
person that takes the HIT (redditers are weird) and also opens the possibility of crosstalk
between subjects in different conditions. So it might not be a bad idea to ask in the
demographics questionnaire if they found the HIT through a forum, and again either
exclude or control for this.
If you do experiments that involve interacting with other MTurk workers, it may be a
good idea to ask at the end to what extent they believed the other person was real. But at
the same time, it’s hard to take these responses as face believe because of things like ex-
post justification of being selfish etc.
Keep in mind that MTurk data is typically noisier than data from the lab, so you want
more subjects than you might in a lab study (I’d say 50-100 per condition as a vague rule