Creating an Assessment Form

In step 4 on the 'Forms'-tab you will build your assessment form (i.e. the exam script for your candidates, preview and publish it.

Creating an Assessment Form

On the 'Forms-tab' you first create an assessment form.

1. Click 'Add form' to create a new form. You will be prompted to choose the amount of questions to be used in your form out of the total selected questions available and need to choose the type of assessment to be Fixed or Random / LOFT. Read more about this further down.
2. You can share, publish and preview your assessment form from this screen.

Defining your type of assessment: Fixed or Random/LOFT and the number of questions to be used

Upon creating an assessment form you will be presented with a few pivotal choices in the 'Generate form modal window':

The first choice is the amount of questions you want to use from your total pool of selected questions. As a general rule of thumb for manually assembled assessments you will choose all (i.e. 50 out 50 questions for example). However: when using a large pool you can opt to use 50 out of 200 questions at random.

The second choice is between a fixed of a randomly generated assessment:

• 'Fixed' - the Assessment will be generated in advance. All candidates will get the same set of questions. You can have multiple forms
• 'Random'. - If you select 'random' , the test will use Linear On the Fly Test generation (LOFT) which means each candidate will get a unique set of questions

Examples:
Fixed 50 out of 100 means all candidates will get to same set of 50 questions. (Whether or not they are in the same order is determined by the Assessment Option 'Question navigation - Display questions in random order).

Random 50 out of 100 means that that each candidate would get 50 questions out of those 100 at random. In this automatically means that the questions will be different and cannot be in the same order.

Difficulty Method

Making sure assessments are valid and reliable, fair and defensible is a vital part of the assessment process. Cirrus uses psychometric data to help our users to validate your assessments.

The main methods are p-values and Angoff values. Both are measurements of the difficulty of the items and assessments. P-values are great if you have already used the items in previous assessments. Cirrus automatically calculates the difficulty of items every time the item is responded to by a candidate. But there are cases where you have never used the item before. You have no data on how the item performed. That’s where Angoff comes in. In short, a panel of Subject Matter Experts sits down and together manually sets the difficulty values. Both P-values and Angoff use a scale from 0 (very hard question) to 1 (very easy question).

Cirrus is taking the use of difficulty values a step further. When creating forms (either fixed forms or random forms) you can define the weighing in the different difficulty levels (easy, medium, hard). Before you do that, you need to make sure the thresholds are set in Admin, see screenshot below.

Then make sure that each item you want to use has either a P- or Angoff value. You can control this within your collection(s) on the Statistics tab.

The next step is to create a form within your assessment. The form can be either Fixed (all candidates will get the same questions), or Random (every candidate will get a different set of questions).

The Difficulty method allows you to select whether you want the system to calculate the difficulty based on the P-values or Angoff values. If you don’t want the system to use difficulty, you can ignore these selection options.

Difficulty allows you to set the weighing between the 3 difficulty levels. The total must be 100%. You can for example have 50% easy questions, and 50% hard questions, or 20% easy, 50% medium and 30% hard questions.

Once you are satisfied with your choices, click on Create form. This will create either a fixed form or random form, depending on your choice. If you have set the desired difficulty level, the system will create a form that takes the difficulty level into account.

Tips for using difficulty levels

This functionality allows you to have control over the difficulty each form has. You can for instance choose to create an easy form A, and a difficult form B for second attempts.

You can also leave it to the system to generate a unique set of questions for each individual candidate, and at the same time make sure that they are balanced and defensible. This is a great way to prevent cheating and content theft since no candidate assessment will look the same.