Dynamic Judgments can be enabled by navigating to Dynamic Judgments under the Quality Control tab within Settings. It is currently an experimental feature and should be used with caution, as it may result in the unnecessary collection of additional judgments.
Using Dynamic Judgments along with Auto Launch can result in inflated costs in your job. When using these two together, decrease the number of rows per page in order to avoid extra costs.
Dynamic Judgments is a feature that allows each row within your job to collect a variable number of judgments dependent upon the agreement of previous judgments for that row. In other words, this feature will enable you to collect fewer judgments on rows where contributors agree on the answer, and more judgments on rows where contributors have submitted answers that conflict with one another. This can be a powerful tool for decreasing overall costs in your job as well as improving accuracy on rows that are difficult to answer.
There are 2 modes for Dynamic Judgments: Disabled (unchecked) and Enabled (checked). These determine the behavior of judgment collection in the job.
Dynamic Judgments is disabled. Rows will not collect additional judgments beyond the Judgments per Row setting.
Rows may collect additional contributor judgments beyond what you order (judgments per row) based on the confidence of a row (agreement between contributors). When the confidence on one or more of the page fields is below a minimum confidence , the job will collect additional judgments until a judgment ceiling is hit.
Let's take a look at the configuration options available for Dynamic Judgments. First we must determine the amount of Judgments per Row.
Fig. 1: Judgment per row setting on the Launch Page
Fig. 2: Dynamic judgments setting unchecked
Fig. 3: Enabled dynamic judgments with 3 max judgments, 0.9 confidence score for the question named "sentiment"
Turning on Dynamic Judgments will add a number of options to this page. In the above example, instead of collecting 2 judgments per row, each row will collect a maximum of 3 judgments. If the answers submitted by those two contributors for the CML Field named "sentiment" are not identical, a third judgment will be collected.
- Max Judgments per row determines the maximum number of Judgments that will be collected for each row
- Minimum Confidence controls the necessary agreement required to determine whether a row is Finalized or whether it requires an additional Judgment. If the Confidence for this row is above this value, the row will finalize and stop collecting judgments. Take a look at this article for help with understanding how Confidence is calculated.
- Confidence Fields determines upon which fields agreement is measured when deciding to either finalize a row or collect an additional judgment.
Here is another example with different settings:
Fig. 4: Enabled dynamic judgments with 5 max judgments, 0.7 confidence score for the question named "sentiment"
The only Confidence Field that is selected is "sentiment", so only answers for this particular question will be considered when determining Contributors' agreement.
- The first Contributor to see the row answers "Positive" for "sentiment"
- The second Contributor answers "Negative" - we now have a Confidence measure of approximately 0.50
- The third Contributor answers "Positive" - we now have a Confidence measure of approximately 0.67
- 0.67 is still less than our Minimum Confidence of 0.70 and the platform distributes the row to a fourth Contributor:
- If the fourth Contributor answers "Positive" - we now have a Confidence measure of approximately 0.75 and the row becomes Finalized
- If the fourth Contributor answers "Negative" - we now have a Confidence measure of approximately 0.50 and the row is sent to a fifth Contributor. The fifth Contributor will submit the last and final Judgment on the row no matter what he/she answers since Max Judgments per row is set to 5.