Quality control in job building is an important step. When quality control mechanisms are utilized, they help attain higher quality results. There are two quality control settings under the Quality Control Settings page in Settings: Minimum Time per Page and Contributor Answer Distribution Rules.
Fig. 1: Quality Control Settings
Minimum Time per Page
This is the minimum amount of time a contributor must spend on a page. This is used to ensure that the contributors spend a certain amount of time on a page before they can submit.
- This is a speed trap -- it's the minimum number of seconds it should take for a contributor to complete one page of work. If a contributor is taking less time than this, they're likely not paying close enough attention in your job.
- The default value is 10 seconds.
- Contributor's who submit a page of work under this limit will not be allow to submit any further work. The page that was submitted to flag the worker for going too quickly will be submitted. None of their work will be tainted and or recollected.
Max Judgments per Contributors
This sets the maximum number of judgments that an individual contributor can submit on a job (based on contributor ID).
This does not prevent individuals from submitting more judgments than are specified here on other accounts that they own.
Max Judgments Per Contributor is typically used to help manage quality (prevent individual workers from over-contributing or engaging in abusive behavior), or to restrict contributions when otherwise conceptually necessary (e.g., prevent contributors from completing the same survey multiple times).
This setting is used to ensure that no single contributor can view a Test Question more than once. The Max Judgments Per Contributor is automatically calculated according to the following criteria:
- Max Judgments Per Contributor is a multiple of the amount of Rows per Page and the amount of Test Questions in a job. The number of Rows per Page is effectively the same as the number of judgments per page because an individual contributor submits only one judgment per row of data.
- If there are test questions in the job, by default, each page contains at least one Test Question. Therefore, the Max Judgments Per Contributor is a multiple of the amount of Rows per Page and the amount of Test Questions in a job.
The reason for why the Max Judgments per Contributor has an automatic fallback number is because:
- A contributor can only complete as many pages as there are Test Questions (at one Test Question per Page) before they start to see repeated Test Questions. It is at this point that we want to prevent contributors from completing any more judgments.
- Contributors could learn to recognize Test Questions if they are repeated, and potentially cheat.
An example of Max Judgments Per Contributor is calculated as follows:
Given a page with the following settings:
- 4 Rows per Page
- 1 Test Question Per Page
- 25 Test Questions in the job
- 4 Minimum Test Questions in Quiz Mode (Default)
One page will have 3 Rows and 1 Test Question. After Quiz Mode, there are 21 Test Questions left in the job so the worker can complete 21 (Test Questions in job - Minimum Test Questions) x 4 (Rows per Page) or 84 units before they see repeated Test Questions. The Max Judgments Per Contributor would be 84 in this example.
Max Judgments Per Contributor always has a ceiling amount, or the highest number possible for the job. Therefore, to apply a custom number, the number can only be set to an amount lower than the ceiling amount. If left blank, the number will be automatically set to the ceiling amount.
Contributor Answer Distribution Rules
The Answer Distribution settings allow you to monitor the distribution of answers that a contributor submits in a job. Monitoring a contributor’s Answer Distribution helps ensure that they are removed from the job if they favor a particular answer in a job. This is a tell-tale sign of bad behavior from a contributor and it is important to remove anyone acting this way.
Fig. 2: Answer Distribution setting
Monitor answer patterns to remove contributors whose answers are outside the expected distribution
Check this to enable Answer Distribution monitoring on your job.
Note: Enabling Answer Distribution monitoring on a running job will enforce the Answer Distribution rules for all new judgments collected. Answer Distribution monitoring will not be calculated retroactively.
Minimum rows submitted by a contributor before we start monitoring
This is the number of rows a contributor must submit before their Answer Distribution is monitored. This can be set from 20 to 100 rows.
This column displays the label associated with each CML element in a job. These are the questions that can be enabled for Answer Distribution monitoring. Enable Answer Distribution monitoring on a question by selecting "Add Rule".
Answer Distribution Rules
Answer distributions can be set by clicking the Add Rule button. Once enabled, you will set the rules that will be enforced here. The settings are described in more detail below:
The Regular Expression field requires an expected answer that will be evaluated when looking for violations of the answer distribution. Enter the expected answer by entering the CML name, label, or value associated with the expected answer. Answer distribution will then use the Regular Expression input to enforce the maximum and minimum percentage that we expect to see that input as an answer.
Example: Question - What type of category best describes this business?
The possible answers, set in the CML, are:
- Hotel / Resort / Lodge
- Bar / Cafe
- Cosmetic / Salon
To enforce an answer distribution rule to check if contributors are answering 'Other' too often, add 'Other' as the Regex match.
Use the sliders to set the minimum and maximum percentage that the Regular Expression answer should be submitted by the contributor.
Example: Being familiar with the data helps in determining the Acceptable Range. For this dataset, less than 15% of the data will be 'Not Relevant'. We can create a rule with a Regular Expression of 'Not Relevant'. Then move the right-most slider to 15%.
With these settings, contributors will be removed from a job if they answer 'Not Relevant' on more than 15% of judgments they submit.
Fig. 3: Contributors will be removed if they enter 'Other' more than 10% of the time
To delete a rule, click the trash can button on the right-hand side of each rule.
Click the Add Rule button to add an additional rule for a question. In cases where there are multiple rules, all rules will be enforced.