Quality control in job building is an important step. When quality control mechanisms are utilized, they help attain higher quality results. There are three quality control settings under the Quality Control page in Settings: Minimum Time per Page, Maximum Judgments per Contributor and Answer Distribution Rules.
Fig. 1: Quality Control Settings
Minimum Time per Page
This setting is used to ensure that the contributors spend a certain amount of time on a page before they can submit.
- This is a speed trap -- it's the minimum number of seconds it should take for a contributor to complete one page of work. If a contributor is taking less time than this, they're likely not paying close enough attention in this job.
- The default value is 10 seconds.
- None of their work will be tainted and or recollected.
Max Judgments per Contributors
This sets the maximum number of judgments that an individual contributor will be able to submit on a job.
This setting is typically used to help manage quality (e.g., prevent individual workers from over-contributing or engaging in abusive behavior), or to restrict contributions when otherwise conceptually necessary (e.g., prevent contributors from completing the same survey multiple times).
The Max Judgments Per Contributor is automatically calculated according to the following criteria:
- If Test Questions are enabled on the job, each page of work mode contains one Test Question. Therefore, the maximum number of judgments a contributor can submit is limited by the number of active test questions.
- When a job does not have Test Questions enabled, a contributor can either complete 50 rows or 10% of the rows launched, whichever amount that is higher.
An example of Max Judgments Per Contributor is calculated as follows:
Given a page with the following settings:
- 4 Rows per Page
- 1 Test Question Per Page
- 25 Test Questions in the job
- 4 Minimum Test Questions in Quiz Mode (Same amount as the Rows per Page setting)
A page of work will have 3 Rows and 1 Test Question. After Quiz Mode, there are 21 Test Questions left in the job so the worker can complete 21 (Test Questions in job - Test Questions used in Quiz Mode) x 4 (Rows per Page) or 84 units before they see repeated Test Questions. The Max Judgments Per Contributor would be 84 in this example.
Max Judgments Per Contributor always has a ceiling amount, or the highest number possible for the job. When Test Questions are enabled, to apply a custom number, the number can only be set to an amount lower than the ceiling amount. If left blank, the number will be automatically set to the number of rows there is available for contributors to work on.
Contributor Answer Distribution Rules
This Setting helps monitor the distribution of answers that a contributor submits in a job. Monitoring a contributor’s Answer Distribution helps ensure that they are removed from the job if they favor a particular answer in a job and when they're deviating away from the expected answer distribution of the data.
Note: Flagging a contributor via answer distribution rules is a permanent action and cannot be reversed.
Fig. 2: Answer Distribution setting
A. Monitor answer patterns to remove contributors whose answers are outside the expected distribution
Check this to enable Answer Distribution monitoring on a job.
Note: Enabling Answer Distribution monitoring on a running job will enforce the Answer Distribution rules for all new judgments collected. Answer Distribution monitoring will not be calculated retroactively.
B. Minimum rows submitted by a contributor before we start monitoring
This is the number of rows a contributor must submit before their Answer Distribution is monitored. This can be set from 20 to 100 rows.
This column displays the label associated with each CML question in a job. These are the questions that can be enabled for Answer Distribution monitoring. Enable Answer Distribution monitoring on a question by selecting "Add Rule".
The Regular Expression field requires an expected answer that will be evaluated when looking for violations of the answer distribution. Enter the expected answer by entering the CML value associated with the expected answer. Answer distribution will then use the Regular Expression input to enforce the maximum and minimum percentage that we expect to see that input as an answer.
Example: cml:radios- What type of category best describes this business?
The possible answers, set in the CML, are:
- <cml:radio label="Restaurant" value="restaurant" />
- <cml:radio label="Hotel" value="hotel" />
- <cml:radio label="Bar" value="bar" />
- <cml:radio label="Medical" value="medical" />
- <cml:radio label="Salon" value="salon" />
- <cml:radio label="Sport" value="sport" />
- <cml:radio label="Restaurant" value="restaurant" />
- <cml:radio label="Other" value="other" />
To enforce an answer distribution rule to check if contributors are answering 'Other' too often, add 'other' as the Regex match.
Use the sliders to set the minimum and maximum percentage that the Regular Expression answer should be submitted by the contributor.
Fig. 3: Contributors Will Be Removed If They Enter "other" More Than 15% Of The Time
Example: Being familiar with the data helps in determining the Acceptable Range. If no more than 15% of the source data is expected to be 'other' for the question "What type of category best describes this business", create a rule with a Regular Expression of 'other' then move the right-most slider to 15%.
With these settings, contributors will be removed from a job if they answer 'other' on more than 15% of the judgments they submit.
To delete a rule, click the trash can button on the right-hand side of each rule.
Click the Add Rule button to add an additional rule for a question. In cases where there are multiple rules, all rules will be enforced.