Once your job is launched you’ll start to see contributor’s responses on your Test Questions from the Quality page. These responses are extremely valuable feedback on how contributors are understanding and completing your job. This feedback not only gives you insight into the quality of your test questions but also the effectiveness of your job design and instructions. Reviewing the responses to your test questions and updating them when needed is the most important aspect of monitoring your job and is often the deciding factor in whether job is successful or not.
Please see our Guide to: Test Question Review page for definitions and explanations of each of the features discussed in this article.
Steps to Monitoring Test Questions
- 1. Go to Quality page, test questions will be automatically listed by % Contested:
Fig. 1: Test Questions sorted by % Contested
- 2. Select Test Questions that are highly missed
Fig. 2: A highly missed test question that needs review and updating
- 3. Review the answers that are highly missed and make sure they follow your instructions
- 4. Review distribution of contributor answers on highly missed radio buttons
Note: As you can see in this example, 100% of contributors have selected ‘Negative’ when ‘Neutral’ was chosen as the correct answer. Referring to the tweet, it is indeed negative sentiment. This Test Question should be update to have the correct answer.
- 5. Also review distribution of answers on text box
Fig. 3: A highly missed text box test question that needs review and updating
Note: Here you can see 100% of contributors answered that the tweet had ‘11’ likes. This indicates that since the test question has been made there may have been likes added to this tweet. Reviewing the tweet, you can accept the answer submitted by contributors and forgive them.
- 6. After reviewing a test question you have the option to select “Pass Review” which will save any changes to your test question and also update it on the Quality page to avoid you reviewing the same question more than once:
Fig. 4: A test question that has been reviewed and “Passed”
Note: Passed Review is particularly useful for those teams with multiple individuals creating and reviewing test questions because now you can easily identify highly-contested or missed test questions and know which have already been verified, avoiding redundant reviews.
- 7. Repeat steps above until for all highly missed/contested test questions
What if my test questions are correct but contributors are still missing them?
- Update the instructions to account for areas of confusion
- Consider rephrasing questions to be more clear
- Add additional instruction tags to the CML
- Provide clearer examples
- Contact Customer Support at firstname.lastname@example.org
If too many people are missing and contesting your Test Questions, we will automatically pause your job and send you an email. You should review/edit the most controversial Test Questions as described above before resuming your job.