Guide to: Creating Test Questions

Overview

In this article we will go through the steps to creating Test Questions within the interface, as well as how to upload pre-annotated gold data to be used as Test Questions. Once some work has been completed you also have the option to create test questions from high confidence finalized rows, see this article. 

For more complex usecases, also see these articles: Test Question Matching and Guide to: Regular Expression with Test Questions.

Once you have created your test questions, you will also need to further specify some additional settings, see this article.

For more information on reviewing Test Questions after some work has been completed, please see How to: Monitoring Test Questions and Review Test Questions.

Create Test Questions: Interface

Step 1: After you have finished designing your job, from the Design page navigate to the Quality Tab, or click on "Next: Create Test Questions"

• Make sure you have your CML design completed and your source data uploaded

Step 2: Click "Create"

Step 3: You will see the Test Questions interface, here is where you will provide answers and reasons for the answers and also decide on various options for your test questions, see Guide to: Test Question Creation Interface for details

• Radio/drop-down inputs will be replaced with checkboxes to allow multiple correct answers

Step 4:  Proceed through each your data rows, answer the rows you selected as Test Questions correctly and provide a reason for each answer, see Guide to: Test Question Creation Interface for details

Create Test Questions: Upload

It is possible to create Test Questions in bulk via a data file by uploading all of your Test Questions at the same time. You can also update existing Test Questions in bulk using a Test Question Report.

Upload New Test Questions (Via Data Page)

It is possible to upload Source Data that already contains Test Questions.

To accomplish this the Source Data must include columns:

• {fieldname_gold}: Test Question answer - lower-case with underscores instead of spaces
• {fieldname_gold_reason} Test Question reason - plain text
• {_golden} - set to ‘TRUE’ for each Test Question

Fig. 6: Source Data with ‘ _golden’, ‘what_is_the_sentiment_of_this_tweet_gold’, ‘what_is_the_sentiment_of_this_tweet_gold_reason’

Steps:

1. Go to the Data page within your job
2. Select “Add More Data”
3. A message may appear letting you know that your column headers do not match the data already uploaded to the job. Select ‘Proceed Anyways’ as we know this is a Test Question report and does not follow the data exactly
4. Once your test questions have been added to your data page, click “Convert Uploaded Test Questions”
5. Go to the Quality page to see your newly uploaded Test Questions

Tip: Test Questions can also contain multiple correct answers. This can occur in scenarios where the response of the contributor is more subjective than objective. When specifying Test Questions manually, multiple values in the Source Data will be determined by the newline character.

Fig. 7: Column 'what_is_the_sentiment_of_this_tweet_gold’ all have multiple values stored in one cell separated by the newline character

Update Existing Test Questions (via Quality Page)

If you would like to batch update Test Questions already in your job, you may do so via a Test Question report.

Steps:

• 1. Go to the Quality page and select ‘Download test question report’

Fig. 8: Test Question report location

• 2. Edit Test Question answers (any fieldname_gold columns)
• Note: Do not edit the ‘_id’ column as this will reference each specific Test Question upon upload
• 3. Select ‘Upload test question report’ from the Reports drop-down menu
• 4. Upload edited report
• Note: The page will automatically refresh with updated Test Questions