Creating auto-grant qualifications to exclude workers from participating multiple times

Note: We have moved away from this method, because we found that adding a qualification drastically decreased the number of workers willing to participate in our experiment. 

What are Qualifications?


Mturk Qualifications are a useful way to keep tabs on workers and restrict workers' ability to work on your HITs based on previous work they have done. Mturk provides a number of built-in qualifications, like a worker's country of origin, their overall HIT acceptance rate, or the number of HITs they have had approved. However, mturk also allows requesters to create their own qualifications, which can have any value from 0 to 100 inclusive. New workers can then be screened according to whether their qualification score is greater than, less than, equal to, or not equal to any given score for a qualification. This means that you can create essentially arbitrary conditions for including or excluding workers. 

Unfortunately, mturk does not provide a qualification test equivalent to "Worker has/does not have a value for this qualification". You can test whether a worker has a particular score, but you cannot test for if they have a score at all. And once a qualification is added to an HIT, workers cannot participate in that HIT unless they have some value for that qualification. Workers can, however, request that a qualification be granted to them. This can happen two ways: 

1) workers can search qualifications. This is useful if a worker knows that a specific qualification will be required in order to complete an HIT they have their eye on. Alternatively, this method allows workers to browse qualifications and request any or all of the qualifications available to them.

2) Perhaps more commonly: when a worker sees an HIT they are not qualified for, as long as the qualification is one created by the requester, and not one of mturk's built-in qualifications (which depend on a worker's location and performance), the worker will see a "request qualification" button in place of the "accept HIT" button. If they expand the hit box, they will also see a description of the qualification and their value for that qualification (see screenshot below).



(Automatically) Granting Qualifications


On the requester side, qualification requests are, by default, added to a queue. Requests for qualification must be manually approved by the requester, and a value must be manually added each time a qualification is granted.

There are two different ways to deal with granting qualifications: by automatically granting qualifications, or by creating a qualification test, which creates a specific qualification test HIT type and automatically assigns workers a qualification value based on test results. Unfortunately, neither of these options is available via the requester GUI -- to create either of these qualification types, you'll need to use the command line.

Using the command line with mturk requires an Amazon Web Services account, which can be created at aws.amazon.com. You might even have one set up initially just by setting up an mturk requester account. In addition to an AWS account, you will need to download either the Mechanical Turk Command Line Interface (documentation here), or a library for a programming language to interact with AWS, like the python boto repository (documentation here). You will have to either create or update a qualification type using either method (the command line interface requires you to create a file, while the boto repository is just a function with a number of different arguments). For either method, you are going to want to have the value autogranted=true, and set a value for the qualification when it is automatically granted (either as a single line in your CLI file, or as an argument in your boto command -- see here and here, respectively).



Updating Qualification Scores to exclude Workers



Using these tools, you can set up a qualification to be automatically granted to workers who request it, allowing them to take your HIT. But what about after they've done it, and you want to exclude them from participating in further HITs?

One method is to reassign values to every worker by hand, which can be managed by going to the results page for a batch, clicking on each worker's ID, and assigning them a new score individually. However, this is an arduous process. 


Using a batch .csv


Another solution is to batch upload a .csv (comma separated values) file containing updated scores for all theworkers to whom you wish to assign the qualification. For more details on doing this, see p. 39 of this .pdf. This can also be relatively arduous, as you must either convert a results file to the format required for uploading new qualification scores, or match workers from batches with your master list of all workers (which can be obtained in the requester GUI by going to "manage", then "workers", then "Download .csv".


Programmatic methods


Finally, you can update workers' scores by using either the CLI or boto. 

Using the CLI



Using the CLI, you can update qualification scores for workers using a tab-delimited file with just the worker ID and the score you wish to assign to the worker (that makes it very easy to create these files in excel, especially if you want to assign the same score to many workers). See here for more detailed instructions on the specific commands and file types involved with this process.


Using Boto



To perform the same operation using boto, you would simply write a for loop which loops through a list of worker ID's and uses the function update_qualification_score(qualification_type_id, worker_id, valueto update your scores.