CML allows you to specify how the CrowdFlower platform will summarize the result for each CML form element in your form. The consolidation of one or more contributor responses into a summarized result is referred to here as 'aggregation'. The output from this process can be found in your job's aggregated report or in the results hash of a unit's JSON payload. All parent CML form elements accept the
aggregation attribute. This attribute moderates the aggregation method used to create your results. Below is an overview of the different types of aggregation we offer and the CML Form Elements that use that type of aggregation by default.
- Default aggregation for: cml:checkbox, cml:radios, cml:taxonomy, cml:checkboxes and cml:select
- Returns a single "top" result - AKA the contributor response with the highest confidence (agreement weighted by contributor trust) for the given field. All other responses will be ignored. A numerical confidence value between 0 & 1 is also returned.
- Default aggregation for: cml:text and cml:textarea
- Returns every response submitted by all trusted contributors for the given field. The result will be a newline '\n' delimited list in the Aggregated report.
- If the given field allows multiple answers, each answer in a contributor's response will be pipe delimited '|'.
- Default aggregation for: cml:rating
- Returns a numeric average calculated based on all responses. Variance will also be returned.
- Returns the top
xresponses (based on confidence) for the given field. This is useful when there are multiple correct responses to a question (e.g., 'Select all that apply' question formats). If you want the top three responses ranked by confidence, then specify
agg_3etc. It is important to note that this setting will result in up-to
xresponses whenever that many unique responses are available, even if the last response has very low confidence or no agreement. The result will be a newline '\n' delimited list in the Aggregated report.
- For Bounding Box questions only, returns boxes that have been successfully clustered with other contributor responses via an Intersection over Union (IoU) value greater than
x, which can be any floating point number between 0 & 1. Always prefix '0' for decimals; for example,
bagg_0.7will generate boxes with an IoU greater than 0.7. For more detail, read our article about Bounding Box aggregation.
- Returns all responses with a confidence greater than the specified value for
xfor the given field.
xcan be any floating point number between 0 & 1. Always prefix '0' for decimals; for example,
cagg_0.4will return all responses with a confidence score greater than 0.4. The result will be a newline '\n' delimited list in the Aggregated report. The aggregate result will be empty if no responses feature a confidence score higher than the specified value.
Note: You can adjust the aggregation for each CML form element on the Reports - Options page on the platform. For more on the Reports Options page, please view this page.