Programming

You can create Programming questions in Cirrus. Simply go to your Library, create or open a collection and click ‘+ Add’ or use the drop-down arrow to select this question type. You will be presented with the screen below.

The Programming question type requires the candidate to write or debug code. Candidates can be offered the option to run and evaluate their code.

This evaluation uses testcase(s) defined by the question author. The question is auto-scored based on the number of test cases (written by the author) that are successfully passed.

Setting up a Programming question

Click ‘+ Add’ or use the drop-down arrow to select this question type. You will be presented with a start screen, showing 5 tabs.

1. Question
2. Test cases
3. Execution options
4. Requested files
5. Execution files

Question tab

On this tab you can set up the basic elements for your Programming question.

Enter the Title
Write down your Item title here. This is for internal use only; this title will be visible in your Library and further in the process. This title is not visible for candidates taking their exam.

Enter the question
Click inside the text box to enter (or edit) your question. For example: “What is the most common line in PHP?”.  

Use the text editor for a range of text editing options. You can also add media and resource files here.

Set maximum score for this question
You will also have to set a maximum score for the question. Simply enter a number into the box.

Save, close, preview & cancel

• Click the ‘Save’ button to save the item.
• Click ‘Save and Close’ if you wish to return to your collection overview.
• You can use the ‘Preview’ button to see what the question will look like to the candidate and make sure that the correct answer receives the correct marks.
• Hit 'Cancel' to exit the question editor. You will be see a warning about saving the page, otherwise your data will get lost.

Test cases tab

The default test case file is vpl_evaluate.cases - which is empty by default. This is where your author defines the test cases to check the code. Also see scoring method further below.

Execution options tab

 This is where the question author sets what is used to run the coded scripts, and if the candidate can run and/or evaluate their code:

1. Run script - using this drop down the author selects which programming language is used to execute scripts.
2. Debug script - will open a drop down to select which programming language is used to debug scripts.
3. Run -  (yes/no) using this setting the author determines if candidate sees (and can use) a button to run script(s).
4. Evaluate -   (yes/no) using this setting the author determines if the candidate gets an evaluate button to check the code against the test-cases, and to see what score his code will receive. For formative assessments you would typically turn evaluate ‘on’. For summative assessments you obviously do not want to allow the candidate to run his script and see his scores.

Requested files tab

This contains the actual files to code / fix. The author can either pre-define these or ask the candidate to add files themselves.

The  file and it’s content entered here will show for the candidate when he opens the question:

The author can determine the maximum amount of files that can be created  via the options button of the question.

Execution files tab

 By default there are three files here. In most cases you do not need to change anything here. These files define how to evaluate code/script. By default the programming question type (VPL - uses predefined scripts).

Programming Options

The Options button for Programming questions opens a window where the author can determine the following settings:

• Maximum execution time
• Maximum memory used
• Maximum execution file size,
• Maximum number of processes
• Files to keep when running: if option is turned on then files won’t be deleted after compilation

The default values are shown beneath each dropdown (as set by your Admins). It is not possible to select values higher than the maximum values as setup under Admin > Global Settings > VPL settings tab.

Supported languages in Programming questions

Writing test-cases for Programming questions

In order to achieve test cases, candidate code must be written proper way. The code should contains the needed count of ReadLine (receive value from console) and WriteLine (write value in console) operations like it defined in the test cases. These operations are different for different languages, for example in С it can be: scanf(), printf(). In c# it is Console.ReadLine(), Console.WriteLine(), etc.

The programming question is automatically scored. To score it automatically author has to create/write/prepare test cases (Test cases tab). Every test case in file should have the following format:

Example testcase:

case = Test 7+5
input = 5
7
output = 12

case = Test 10+18
input = 10
18
output = 28

case = Test 13+(-17)
input = 13
-17
output = -4

case = Test 1024+1024
input = 1024
1024
output = 2048

Programming question in candidate delivery

When answering the programming question the candidate has access to a code editor, where syntaxes will be highlighted. This editor can be maximized for optimal space for writing code.

Create: Allows the candidate to create a new page
Run: Allows the candidate to run the code within a console
Debug: Allows the candidate to debug the code
Evaluate: Allows the candidate to compile the code. If compilation fails, points will be subtracted from the max score.

Programming question: showing / viewing results in Reports

Admin: setup for executing programming questions

Programming question setup in your Cirrus environment

Go to Admin > Global settings > "VPL-tab"

Enter the address of your Execution server, something like https://vpl.jail.yourserver.com. There can be multiple execution servers, a load-balancer will check which execution server will be available.

Setting up an execution server for your programming questions

In order for your programming question to work you need to setup an execution server.
When a candidate The following steps are performed:

• Files submitted by the student are taken
• Files set by the teacher in "Execution files" are taken (this can replace files of student with same name)
• Depending on the action (run or evaluate) the script file set by the author are taken. If there is no script, the system take a default script by detecting the programming language used based on the extension of filenames.
• In the case of evaluation, if there is no script, the VPL program for automatic evaluation is also added. This program is based on the input and output of the program and requires that you specify the test cases in the file "vpl_evaluate.cases".
• This collected files are sent to an execution server
• The module informs the browser that the execution has begun
• If the request is an evaluation, when the task is finished, the result of the evaluation is retrieved from the execution server.

Marking programming question

In most situations a programming question will be automatically scored by the system. However Cirrus allows you to manually score auto-score questions.

When you score the question you will see the following information:

• Test #
• Description
• Error
• Input
• Program output
• Expected output (number)
• Summary of tests

Note: The actual code (file) of the candidate is not possible to see in marking