Guide to: Additional CML Attributes

All CML questions can contain additional attributes that determine different behavior. The following attributes pertain to test questions and aggregation behavior. Some of these attributes are implemented by default and can be overwritten, however, understanding the default settings can help you better customize your results to fit your needs.


Test Question Settings:


Remember gold stands for “gold standard responses” which is identical to test questions. All questions types have gold= “true” by default. Remember the gold attribute, if set to true , will link the CML questions with the data specified in the CML question name gold column of your uploaded data file. Basically, this means that a gold or test question column is created in the back end so that your test questions can be stored. However, because this is set by default, you really don’t need to worry about this attribute.


In other words:

<cml:text label="Sample text field:" />

Is identical to

<cml:textlabel="Sample text field:" gold=“true”/>


Test Question Matching Settings:


            By changing the test question matching attributes, you can dictate how contributor responses are matched against test questions. The default setting will pass a contributor as long as one of their answers matches one or more of your answers even if they've also selected a wrong answer. For example, if you select A and B as the correct answer, a contributor will pass the test question if they select “A” or “B”. In this example, the contributor will be marked correct even if he/she selected a wrong answer. Because of this, when using checkboxes you should always use one of the following attributes:



The exact attribute requires that the set of contributor responses and the set of test question standard responses for a CML field are identical. A contributor's responses will not pass test question if they include any item not found in the set of test question responses (e.g., if a, b, and c are set in your test question data, then a contributor will only be correct if he submits a, b, and c). Order does not matter.



The strict attribute requires that every response submitted is included in the set of test question standard responses. Unlike exact, however, strict allows contributors to omit responses that are part of the test question set (e.g., if a, b, and c are set in your gold data, a contributor will be considered correct if he submits only a and b). 


Thus, for cml:checkboxes you should always set strict=“true” or exact=“true”, do not use the default setting.



Allows you to redefine how a contributor's responses to test questions are evaluated. Normally, a contributor's responses must exactly match that of the stored test question answers. This attribute can be set to the following:

  • not - This will reverse the normal behavior so that a contributor must answer anything but the pre-defined test question answer to be considered correct. For example, if answer "a" was stored in the test question data and the contributor selects answer "b" (or anything other than "a"), the contributor would get the test question correct.
  • range - This allows you to specify a range that a contributor response can fall between. Specifying a minimum and maximum range will allow a greater threshold of acceptability when contributors answer test questions that may have some variances. To define the minimum and maximum, you may set this within the test question interface, or within a data file with two headers named "column_name_min" and "column_name_max". Use of a number validator is recommended in conjunction with this matcher.



<cml:checkboxes validates="required"label="Sample checkboxes:" strict=“true”>
<cml:checkboxlabel="Checkbox 1"/>
<cml:checkboxlabel="Checkbox 2"/>
<cml:checkboxlabel="Checkbox 3"/>


Regular Expressions

Regular expressions can be used to create flexible criteria for test questions on cml:text and cml:textarea fields. Suppose you want to allow any response that contains the phrase ‘foo bar' to be submitted in a text field.

Example of cml:text using regex:

<cml:text label="Sample text field:"regex="\\d+\." flags="i" no_escape="true">
</cml:text >

Here's how you would go about setting up the regular expression match for the text field:

1. Before your CML text field closes, had any additional attributes. 

<cml:text label="Please enter the user's Facebook URL:" name="facebook_url" validates="required" src="facebook_url_gold" regex="{seed}" flags="im" no_escape="true"/>

You can set the following regular expression attributes:

regex: The value of this attribute is the regular expression that a correct response should satisfy. The token {seed} will be replaced with the value of the field in the data specified in src="" when the regex attempts to match the response. For example, if you're trying to match ‘foo bar', or another simple phrase, set the the regex to {seed} (regex="{seed}") in your question, and set ‘response_gold' to ‘foo bar'.

flags: With this attribute, you can set modifiers that will change the behavior of the regular expression. Currently, there are two values that are accepted.

  • flags="i" - Makes the regex insensitive to case. E.g., If you set the 'response_gold' field to 'foo bar' but also want to accept 'Foo bar', the capital F will not cause the match to fail.
  • flags="m" - If you are expecting to receive multiple line responses, the 'm' flag will ensure that the entire response is evaluated. This is essential for copy/paste jobs.

If you would like utilize both flags simultaneously, use flags="im"

no_escape: When the ‘response_gold' for your unit is substituted for {seed} in the regular expression, any special or illegal regular expression characters are escaped. E.g., "|" is considered a special character in regular expressions. By default, it will be escaped and lose its special meaning unless ‘no_escape' contains the value "true".

Example: If you wanted to match either ‘first base','second base' or ‘third base', you might try setting ‘response_gold' to '(first|second|third) base'. However, this would not work since the special characters would be escaped and only the literal string '(first|second|third) base' would be matched, parentheses, pipe and all. Setting no_escape="true" would have the desired effect, allowing any of the three options as a correct response.

2. Add the test question data column to your input data file or in the test question creation interface.

Let's say you name your regex text field ‘response'. In your input data, you will need to add a 'response_gold' field (header) and populate it with the phrase or pattern to match for each unit. Your data file might look like this:


This would allow a contributor to get this test question correct according to Regular Expression Rules:

The carrot means ‘must start with’

.* means 0 or more instances of any character

.+ means 1 or more instances of any character

$ means ‘must end with’


For example. a contributor could input or This is very useful for URL finding.


Creating test questions can be done as usual, except you have to be conscious of using regular expression special characters and escaping characters if needed.


Aggregation Settings:


Each CML question has a default aggregation method. Each default aggregation is based on the nature of the question, however you can alter the default aggregation for specialized needs.


Here is a review of the 3 aggregation methods that are set by default:


agg (aggregation="agg")

Default aggregation for: cml:checkbox, cml:radios, cml:taxonomy, cml:checkboxes and cml:select

Returns a single "top" result - AKA the contributor response with the highest confidence (agreement weighted by contributor trust) for the given field. All other responses will be ignored. A numerical confidence value between 0 & 1 is also returned.

all (aggregation="all")

Default aggregation for: cml:text and cml:textarea

Returns all unique responses submitted by all trusted contributors for the given field. The result will be a newline '\n' delimited list in the Agreggated report.

avg (aggregation="avg")

Default aggregation for: cml:rating

Returns a numeric average calculated based on all responses. Variance will also be returned.



Note: agg_{x} (aggregation="agg_x") and cagg_{x} (aggregation="cagg_x") are alternative aggregation methods you can use, however they are not set on any CML question by default.


Here is a list of each CML question with their default aggregation methods:


  • cml:text – all(aggregation="all")
  • cml:textarea - all(aggregation="all")
  • cml:checkbox - agg(aggregation="agg")
  • cml:checkboxes - agg(aggregation="agg")
  • cml:radios - agg(aggregation="agg")
  • cml:select - agg(aggregation="agg")
  • cml:ratings - avg(aggregation="avg")



Because these aggregations are set by default and are not explicitly written in the raw code:

<cml:radios label="Sample radio buttons:"> 
<cml:radio label="Radio 1"/>
<cml:radio label="Radio 2"/>
<cml:radio label="Radio 3"/>

Is identical to:

<cml:radios label="Sample radio buttons:" aggregation=“agg”>
<cml:radio label="Radio 1"/>
<cml:radio label="Radio 2"/>
<cml:radio label="Radio 3"/>

Remember you can always change the default aggregation by setting in manually in the code. For example, if you want to have a cml:text with “agg” instead of “all” you simply write:

<cml:text label="Sample text field:"aggregation =“agg”/>

Was this article helpful?
1 out of 1 found this helpful

Have more questions? Submit a request
Powered by Zendesk