Test Questions are Appen’s quality control mechanism within jobs. By providing the answers to a small percentage of rows, you can tell how accurately a contributors is working and will automatically remove any who do not meet your accuracy threshold. This article covers the interface used to create Test Questions. If you've already created test question in your job, they will appear in a list under 'Quality' tab from the top bar. Here, you can review all of your Test Questions. For more information on reviewing Test Questions, please see the Review Test Questions article.
Figure. 1: List of existing Test Questions
Note: You will not be able to create any Test Questions if you don’t have any data.
The following options are for creating and editing Test Questions will become available after you select “Create Test Questions” or “Create More” if there are existing Test Questions in your job.
Test Question Creation
The Test Question creation page will closely resemble the interface you created in the 'Design' step with 2 exceptions:
- Radio/drop-down inputs will be replaced with checkboxes to allow multiple correct answers
- A text box area will populate under each question so you can provide a reason explaining why your answer is correct.
Selecting 'Save' will create a copy of the row and mark it as a Test Question, containing your answers and reasons.
Figure. 2: Test Question Interface
Note: As shown in this example, you can allow the job to accept multiple correct answers by selecting more than one checkbox.
Figure 3. Important Features within the Test Question Interface
A. Show Job Instructions
- This displays the job instructions within the Test Question
B. Skip
- You can choose to skip a row it does not follow along the with the Test Question’s Best Practices or the answer distributions.
C. Save and Create Another
- Save the row as a Test Question and continue with creating Test Questions. Keep in mind, the row will be copied to a Test Question row in your job's dataset.
- D. Mode
- You can select if you want the test question to appear in 'Quiz Mode', Work Mode' or 'Both'
E. Test Question Reason Textbox
- This textbox will automatically appear when an answer is selected.
Show Unit Page
This page will present you with more detailed data concerning the row selected.
Figure. 4: Important features within ‘Show Unit Page’
A. Show Data
- Test Question and Source Data in JSON format.
Figure 5. The JSON format of the selected Test Question
B. Show Statistics
- A specific CML question can be selected to display the answer distribution associated with it.
Figure 6. The answer distribution across all judgements for the CML Question ‘sentiment’
C. Show Contributor Info
- Display data belonging to each individual contributor who has submitted a judgment on the row. This data includes:
- Worker ID
- Trust Score
- If their judgment was correct
- If their judgment was tainted
- Name of the channel the worker belongs to
- If they contested the incorrect judgement
- Each answer given by the contributor, defined by the CML name
- Each answer given by the contributor, defined by the CML name
D. Edit Test Question
- Brings you back to the Test Question editing interface
Secret Test Question Feature
The primary function of Test Questions is to test the quality of contributor responses to pre-annotated data and provide feedback to them when they are annotated as incorrect, to that end Test Questions are meant to be indistinguishable from the dataset. Ideally Test Questions should not be reused across multiple jobs because long term contributors may eventually learn the specific answers they get wrong. If Test Questions will need to be reused, however, it can be useful to reserve a subset of them for which the correct answers are not revealed.
The Secret Test Question feature allows you to hide the answer to certain Test Questions in your job. After creating Test Questions, toggle the "Hide Answer" option on the Quality page for the Test Questions that you would like to hide.
Figure 7: Hide Answer on Quality page
It is recommended that Secret Test Questions are used in conjunction with the gold pool functionality. Secret Test Questions cannot be used in quiz mode to qualify contributors into the job, thus if you hide Test Questions that are set to quiz and have less than 5 Test Questions allocated to quiz, quiz mode will not work. As Secret Test Questions are only meant for Work, it is recommended that they are used in addition to regular work Test Questions. It's crucial to note that Secret Test Questions do not impact contributors' accuracy in the task, nor do they trigger any flags, rejections, or affect contributors' overall performance during the job.
To assess and calculate the accuracy of contributors on Secret Test Questions, manual review via the full report is necessary at this time. To check the rows with Secret Test Questions, see Figure 8 below where column _answers_hidden equals TRUE.
Figure 8: Full Report featuring field _answers_hidden
Hide Test Question Answers
You can choose to hide all of the Test Question answers from contributors by clicking on the toggle button on the Quality page.
Figure 9: Hide All test questions answers on the Quality Page
Interpreting the Quality Page
While your job is running or after your job has completed, you are able to obtain some key information on how the contributors performed in your Test Questions.
Teal Colored Bars
Teal colored bars show acceptable answers for the Test Question. Percentage indicates the percentage of contributors that selected the acceptable answers.
Figure 10: Contributor performance on test questions
Gray Colored Bars
Gray colored bars indicate that the answer shown was not included as an acceptable answer for that Test Question. The percentage indicates the percentage of contributors that select that answer.
Figure 11: Contributor incorrect answer percentage for test question
Note: By clicking on the + sign next to an answer, you can make that answer 'acceptable'. This would mean, that contributor answers that were previously marks as 'unacceptable' will now be re-marked as 'acceptable'.