This article explains the characteristics of useful test questions and best practices when creating them.
Why use Test Questions?
Testing
Test Questions will be used to test a Contributor’s performance both before they enter a job (in Quiz Mode) and on an on-going basis while they work in a job. Test Questions allow you to understand the performance of each contributor that works on a job and automatically remove contributors that perform poorly. By setting a accuracy threshold in a job, it will only allow contributors who have performed above the threshold to complete any further work on a job.
Note: Set accuracy threshold Settings > Quality Control > Test Question Settings. These settings as well as others are described in detail in the Test Question Settings article.
Training
An important step in creating Test Questions is to write very clear reasons explaining why you have selected that answer. These reasons will be shown to contributors if they get a Test Question incorrect. This is a great tool to explain to your contributors how you’ve reached your answer. By explaining the answer, contributors can learn the rules and intricacies of a job in more depth than is provided in the instructions.
How many Test Questions do I need?
- It is recommended to have between 50-100 test questions in a job. Because contributors can only see a test question once, the more that are created, the faster the job will complete
- To enable Quiz Mode and test questions in a job, the total number of test question >= (Rows Per Page setting + 3)
Characteristics of Ideal Test Questions
- Ideally, contributors will behave exactly the same on test questions as they do on the regular rows of data in the job. To accomplish this, test questions must be indistinguishable from the overall data set
- Test questions should follow the rules set in the job instructions. As well, test question reasoning should explain exactly how an answer was reached
-
Test questions can accept multiple possible answers. This is especially useful on questions that have a degree of subjectivity
- Example: The job is to rate the sentiment of a tweet. In this type of job, what someone understands as "Slightly Positive", others may think as "Highly Positive". Accepting both answers will allow contributors to get the test question correct when either answer is selected
Fig. 1: Both Highly Positive and Slightly Positive are enabled as correct answers
- Test questions should be hard enough to test a contributor’s performance but easy enough for those who are following the job instructions
- Every test question should be equipped with an instructive reason that explains the process of getting a correct answer
- Do's:
- Write full sentences and use proper grammar
- Refer directly to the instructions
- Explain why the answer is correct and also why the incorrect answers are not. Find points for them to improve in their work
- Pay attention to the answer distributions of test questions, for example if you have Yes/No answers try to provide an equal number of test questions with Yes and No as correct answers.
- The method of answering test questions varies depending on how many questions there are in a job.
- For jobs that only ask a few questions, it’s a good idea to answer every field.
- For jobs that ask a lot of questions and include logic, the questions that are answered should vary. Varying the answer makes it easier for contributors to get the question correct because they are tested on fewer fields. If contributors get one question wrong they get the entire test question wrong.
Note: Contributors will only be tested against questions where an answer has been provided. If a question isn't answered, the contributor will not be tested against that particular question
- If the test question being submitted contains impossible logic. A warning message will appear to warn that a piece of the logic tree is not correct and the questions need to be updated before saving.
Fig. 2: Warning message in regards to answering the first question as ‘No’ and then answering the second question which should only be asked if the answer to the first question is ‘Yes’
What if my test questions are correct but contributors are still missing them?
- Update the instructions to account for areas of confusion
- Consider rephrasing questions to be more clear
- Add additional instruction tags to the CML
- Provide clearer examples
- Contact Platform Support
Auto Pause
If too many people are missing and contesting your Test Questions, we will automatically pause your job and send you an email. You should review/edit the most controversial Test Questions as described above before resuming your job.