Judgment Overcollection FAQ

In some cases, rows may collect more judgments than you expected. This article answers the question "Why might my rows have more judgments than I thought?"

Forgiving Test Questions

  • If the correct answer for a Test Question is changed, contributors have their scores recalculated.
  • Please note: if those contributors were untrusted before and were actually correct on those Test Questions, they can become trusted. When they become trusted, the rows they judged now have an additional trusted judgment which may lead to them having "too many" trusted judgments.

No Quiz Mode

  • When there's no quiz mode, contributors can judge lots of rows before seeing enough Test Questions to become trusted. Because Figure Eight only counts trusted judgments on a row, if after a certain time contributors become trusted, those judgments that were previously untrusted become trusted, and the row can overcollect judgments.

Looking at the Data tab

  • On the data tab (jobs/{{job_id}}/units), the judgments displayed are for both trusted and untrusted judgments. A user sets how many trusted judgments they want in a job in the Job Settings menu.

Looking at an Individual Unit

  • The judgments here are total judgments. Click on "Show Worker Info" to see how many workers are trusted.

Dynamic Judgments Mode

  • Dynamic Judgments allows the user to request a range of judgments in the hopes of achieving a minimum confidence on each row.
  • For example, if a job is set to collect 3 judgments per row and dynamic judgments mode is turned on with max judgments per row set to 6, then rows should collect 3-6 trusted judgments.

Units Remain Finalized - Under-collecting judgments

  • If a job has "rows remain finalized" checked, rows can potentially under-collect.
    Example: when a job is set to collect 3 judgments/row, and a row has collected 3, if one of the contributors later become untrusted, because they fall below the trust threshold, then that row will only show 2 trusted judgments. The third judgment will not be collected again in this case.

Was this article helpful?
1 out of 1 found this helpful

Have more questions? Submit a request
Powered by Zendesk