Moderation

To ensure the final score is fair, valid, and reliable, you can add up to three moderation rounds to a marking workflow. This allows one or more moderators to review some or all scripts again based on specific rules set up by the coordinator.

This article will look at

• The Role of the Moderator in Cirrus
• Setting Up Moderation Rounds
• How to Moderate in Cirrus
• Bulk Moderation: Moderating Multiple Candidates or Items

The Role of the Moderator in Cirrus

A moderator in a marking process ensures accuracy, fairness, and consistency by:

• Quality Assurance: Checking and maintaining consistent marking standards.
• Standardization: Aligning markers' understanding through meetings and feedback.
• Dispute Resolution: Resolving marking disputes and queries.
• Training and Support: Providing training and ongoing guidance to markers.
• Monitoring: Keeping the marking process on schedule and reporting on quality.
• Feedback Improvement: Collecting feedback for process improvements.
• Annotation Review: Ensuring annotations and feedback are clear and constructive.
• Score Review: Double-checking scores for accuracy, particularly for candidates near the pass mark, to ensure fair and consistent grading.

In Cirrus you can add one or more more moderation rounds to a schedule if you want to make sure that scoring by your assessors was done in a fair, valid and/or reliable way.

The following steps will help you with Feedback Improvement, Annotation review, Score Review and Dispute resolution

Setting Up Moderation Rounds

In some cases the moderator is also the coordinator in Cirrus. A coordinator in Cirrus is considered to make sure the candidates/items are allocated and by who and how.

On of the steps is adding moderation rounds when setting up the marking workflow. This requires a couple of steps that will be explained here.

Step 1: Add a moderation round

1. Navigate to the "Coordinator" section and select the schedule you want to set up.
2. Click on "Settings."
3. Click "Add Moderation Round." You can add up to three moderation rounds

Step 2: Set up Moderation triggers:

• Difference in scoring attempt -This feature applies exclusively to the "Two Assessors" workflow type. In this setup, each script is evaluated by two different assessors. If the scores from the two assessors deviate significantly, you can use this parameter to ensure that a moderator reviews these scripts.

• Proximity to pass-mark - When a candidate's score falls within a certain percentage of the pass mark, it is considered best practice to have a moderator review and determine the final grade. You can set up this percentage to ensure that candidates who are near the pass/fail boundary receive a thorough evaluation by the moderator
• Marking result range -This option allows coordinators to define a range of scores that need to be reviewed by a moderator. The range can be set from 0% to 100%.
• Sample Moderation - With sample moderation, you can randomly check the markers based on a specified percentage.
• Show Assessor Annotations: -When checked, the moderator will be able to view the annotations from the assessor(s).

How to Moderate using Cirrus

If you are setup as Moderator for a schedule you can:

1. Navigate to Marking > To be Assessed-tab.
2. Locate the assessment with the status "Needs moderation."
3. Select the schedule and choose a candidate by clicking their name.
4. Review the overview, which includes the questions.
5. Click on 'Needs Assessment' next to the first question requiring moderation.
6. The 'Essay Marking' page opens, which is identical for both assessors and moderators.

Bulk Moderation: Moderating Multiple Candidates Or Items

In some cases it is up to the moderator themselves to choose which handed in exams to moderate. For example when you want to pick a random sample or double check the top, bottom and middle scores as awarded by your Markers. This is where bulk moderation comes in handy:

• Set the rule to moderate all scripts by setting the proximity to pass mark to 100%.
• The moderator will see all handed in exams that have been marked and submitted
• The moderator selects which ones to moderate and handles these first.
• The moderator selects the remaining candidates and clicks "Bulk Moderate"-button.

This opens a new window (see below) showing the number candidates selected to moderate answers for.

Tips on Bulk Moderation:

• The window will show what scores will be used in case of one or two assessors.
• If you've selected candidate that you already manually moderated a warning will show.
• Multiple Assessors and no Moderator appointed: When a scheduled assessment does not have a pre-appointed moderator, in the case of multiple assessors scoring differently, the answer will receive the average of both scores.