Using polytomous scoring rules

This article explains the difference between Dichotomous and Polytomous scoring in Cirrus. 

When questions in Cirrus allow for multiple answers, you can apply Dichotomous and Polytomous scoring rules:

• Dichotomous:the candidate must answer everything correct to get the max score. When answered partially correct, the candidate will score 0.
• Polytomous:gives a partial score when the candidate answers the question partially correct.

Examples with and without polytomous scoring

Let's take a look at some questions that support multiple correct answers. 

These examples clarify the difference between polytomous and dichotomous scoring rules.

Order

Picture a simple Order question with 4 items to put in the correct order. The example has a maximum score of 4.

For more complex questions like Multiple Response (MR) or Extended Match however, the amount of options a candidate can select affect the scoring model used. 

Multiple Response (MR)

When using polytomous scoring on MR, the author is offered two scoring models. These are based on the number of answer options the candidate can select:

1. The number of answers to select is equal to the correct options
2. The number of answers to select has no limit.

You will find these options using the Options tab, when authoring a MR question.

Polytomous with option 1

In this scenario the candidate 'knows' how many options will be correct. Candidate will receive partial scoring for each option selected correctly. 

Consider the following example question:

Select the 2 cities with the most tourist visiting in 2018

• Barcelona
• Paris
• Amsterdam
• New York
• Moscow

Determining the candidate's score:

• 2 points are divided across 2 correct alternatives
• So points per correctly selected option 2/2 = 1

Polytomous with option 2

When using polytomous scoring together with 'no limit' the candidate does not know the number of correct options to select. 

He is awarded points for:

• Selecting the correct answers
• Avoiding incorrect alternatives. I.e. you are also awarded points for not selecting incorrect alternatives.

Consider the following example question:

Which cities had the most tourists in 2018?

Determining the candidate's score:

• In this case 2 points are divided across 5 alternatives. Meaning each alternative is valued at 0.4 point.
• So points per correctly selected answer + avoided alternative: 2/5 = 0.4.

Say your candidate selected 2 alternatives but only 1 is correct. This means the candidate selected 1 alternative correctly and avoided 2 incorrect alternatives. 

Result: this candidate scores 3*0.4=1.2

Extended Match (ExtMatch)

When using polytomous scoring for ExtMatch, the author is offered two scoring models. Both based on the number of answer options the candidate can select:

1. The number of answers to select is equal to the correct options
2. The number of answers to select has no limit.

You will find these options using the Options tab, when authoring an ExtMatch question.

Polytomous with option 1

In this scenario the candidate 'knows' how many options will be correct. He will receive partial scoring for each option selected correctly. 

Consider the following example question:

Connect the options on the left with the correct numbers on the right

Determining the candidate's score:

• 4 points are divided across 8 correct alternatives
• So points per correctly selected connection: 4/8 = 0.5.

Polytomous with option 2

For ExtMatch questions in Cirrus using polytomous with 'no limit' the candidate does not know the number of correct options to select. He is awarded points for:

• Selecting the correct alternatives
• Avoiding incorrect alternatives, i.e. you are also awarded points for not selecting incorrect alternatives.

Consider the same example question:

Connect the options on the left with the correct numbers on the right

Determining the candidate's score: 

This means that for determining the candidate's score 5 points are divided across 15 alternatives. This means each alternative is valued at 4/15.
In other words: points per correctly selected / avoided alternative: 4/15 =0.26667.

Hotspot

Consider a simple Hotspot question with 4 items to click. Do realise this gives a number of answer options:

• 4 hotspots clicked
• 3 hotspots clicked, one not clicked
• 2 hotspots clicked, two not clicked
• 1 hotspot clicked, three not clicked
• No hotspots clicked

The candidate will be scored on correctly checked options and avoided incorrect options. With a maximum score of 5 points this leads to the following scoring table: 

Taking 'result by chance' into account

For many question types candidates could opt to just simply guess the correct answer. This is why many organisations take the chance of guessing (result by chance) into account. This can be done during scoring and subsequently in determining a 'pass' or 'fail' or grade.

Example

Picture an assessment with 80 MC4 questions (i.e. only Multiple Choice questions with four answer options). Your scoring / assessment scale should take into account the chance to score of your questions.

80 MC4 questions means that a candidate has a 1 in 4 chance of simply guessing the answers correctly. Meaning on average 0,25*80=20 points can be scored by simply guessing! 

What does this mean for the pass mark? 

Let's assume the uncorrected pass mark is set at 50%. This means you need to score 50% of 80 points = 40 points to pass. 

But when taking result by chance into account, you can adjust this. For example:

• Score 20 points + 50% of the remaining points > 20 + (0.5*60) points) = 50 points out of 80 to pass. 

Now your percentage to pass mark is adjusted to 62,5% to avoid chance score.

Some useful notes:

• You set the pass mark when creating an assessment Form, using the 'Options-tab'.
• You can add and adjust pass marks in the Admin part of Cirrus.
• Adjusts percentage to pass and grade scale percentages using chance scores are to be used with dichotomous questions only.
• Be sure to understand how this works before using this with dichotomously scored questions.
• When using a mix of different question types the calculation become a lot more complex.
• If you use other than Multiple Choice questions you will use the Chance score calculations as used in Cirrus. It is imperative you understand the implications before using this.
• If these do not match your needs you can simply create a custom Assessment Scale for your Assessment Form.
• Feel free to contact support in case you want to know more.