Using images and Qualtrics options to discourage participating in an experiment multiple times

Although we tried for a while to use auto-granting mTurk qualifications to disallow workers from participating in an experiment multiple times, we found that the addition of a qualification discouraged workers from attempting to accept our HIT so drastically that batches which would have once finished in hours now took days to complete. As such, we've moved away from mTurk qualifications and towards using a combination of images and built-in features of the Qualtrics survey platform to discourage participating in experiments multiple times.

Using an "image survey" to filter participants

Most experimental surveys on mTurk look the same -- a bland consent form, or maybe just a link to an external survey. Nothing is distinctive, so workers may well not remember having previously participated in a survey. To combat this problem, we created a separate survey which was a single page with a picture on it, as well as text explaining that if participants recognized the picture, they had likely participated already. We used a whale, and the following text: 

You may have already participated in this study, as our lab has run similar studies before. Some studies are completely independent, but others are related.

If you have already done this study, please return this HIT and do not do the experiment again – we cannot analyze data from someone who has already taken the experiment.

If you have seen a page like this before, and the image below, you have *definitely* participated in this study before.

We hope that the image is distinctive enough to be remembered by most workers, but of course, this method relies on the conscientiousness of workers. However, creating it as a separate survey has two advantages:

  1. By using a separate survey, we can use the Qualtrics "Prevent Ballot Box Stuffing" feature, which prevents workers from participating multiple times based on IP address. This is of course imperfect (most people have dynamic IP addresses, and if someone really wants to bypass it they can use a proxy), but it is another barrier to multiple participation.
  2. Using custom End of Survey messages in combination with setting embedded data via URL, we can pipe people to different surveys. this is useful if, for instance, we are running multiple iterations of experiments, or two experiments which are similar enough that we would not want workers participating in both.

Although it is difficult to be certain whether workers have participated in an experiment multiple times, we believe that these measures have been effective in curbing multiple participation in our experiments.