You can create Programming questions in Cirrus. Simply go to your Library, create or open a collection and click ‘+ Add’ or use the drop-down arrow to select this question type. You will be presented with the screen below.
The Programming question type requires the candidate to write or debug code. Candidates can be offered the option to run and evaluate their code.
This evaluation uses testcase(s) defined by the question author. The question is auto-scored based on the number of test cases (written by the author) that are successfully passed.
Execution server required:
This question type only works in conjunction with an execution server, to be set up by the customer. This execution server is running on 3rd party solution Virtual Programming Lab (VPL). Admin setup for executing programming questions explains how to set this up.
Setting up a programming question
Click ‘+ Add’ or use the drop-down arrow to select this question type. You will be presented with a start screen, showing 5 tabs.
- Question
- Test cases
- Execution options
- Requested files
- Execution files
Question tab
On this tab you can set up the basic elements for your Programming question.
Enter the Title
Write down your Item title here. This is for internal use only; this title will be visible in your Library and further in the process. This title is not visible for candidates taking their exam.
Enter the question
Click inside the text box to enter (or edit) your question. For example: “What is the most common line in PHP?”.
Use the text editor for a range of text editing options. You can also add media and resource files here.
Set maximum score for this question
You will also have to set a maximum score for the question. Simply enter a number into the box.
Save, close, preview & cancel
- Click the ‘Save’ button to save the item.
- Click ‘Save and Close’ if you wish to return to your collection overview.
- You can use the ‘Preview’ button to see what the question will look like to the candidate and make sure that the correct answer receives the correct marks.
- Hit 'Cancel' to exit the question editor. You will be see a warning about saving the page, otherwise your data will get lost.
NOTE:
Changes that trigger a new version of a Programming question that is in use in an assessment are: changes in score, changes in test cases and changes in requested files changes.
Test cases tab
The default test case file is vpl_evaluate.cases - which is empty by default. This is where your author defines the test cases to check the code. Also see scoring method further below.
Execution options tab
This is where the question author sets what is used to run the coded scripts, and if the candidate can run and/or evaluate their code:
- Run script - using this drop down the author selects which programming language is used to execute scripts.
- Debug script - will open a drop down to select which programming language is used to debug scripts.
- Run - (yes/no) using this setting the author determines if candidate sees (and can use) a button to run script(s).
- Evaluate - (yes/no) using this setting the author determines if the candidate gets an evaluate button to check the code against the test-cases, and to see what score his code will receive. For formative assessments you would typically turn evaluate ‘on’. For summative assessments you obviously do not want to allow the candidate to run his script and see his scores.
Requested filestab
This contains the actual files to code / fix. The author can either pre-define these or ask the candidate to add files themselves.
The file and it is content entered here will show for the candidate when he opens the question:
The author can determine the maximum amount of files that can be created via the options button of the question.
Execution files tab
By default there are three files here. In most cases you do not need to change anything here. These files define how to evaluate code/script. By default the programming question type (VPL - uses predefined scripts).
TIP:
When you hit ‘Check’ when previewing a programming question in the Library this triggers the same as hitting evaluate + score.
Programming options
The Options button for Programming questions opens a window where the author can determine the following settings:
- Maximum execution time
- Maximum memory used
- Maximum execution file size,
- Maximum number of processes
- Files to keep when running: if option is turned on then files will not be deleted after compilation
The default values are shown beneath each dropdown (as set by your Admins). It is not possible to select values higher than the maximum values as setup under Admin > Global Settings > VPL settings tab.
Supported languages in programming questions
| Summary | In order to run code (or debug code - the latter has not been implemented yet in Cirrus) scripts are used. For some languages default scripts are available. For others the author will need to develop these without using VNC as this is not supported in Cirrus. In the overview below you can see the different options per programming language in Cirrus. Out of the box the programming question lists 43 programming languages in total. For 11 languages you need to keep into account the default VNC option is not available, amongst which Csharp en Java. Three (3) languages are not support because they require VNC scripts: html, r and python-3-pudb). |
| - | There is not a default script for this action. So if the author does not write his own script, the code will not work |
| Console | There is default script, that runs the code in console |
| Console or VNC | There is default script, that runs the code in Console or VNC, it depends on many things (see scripts). If the author uses VNC, this will not work in Cirrus. |
| VNC | There is default script, but it always uses VNC. This will not work in Cirrus |
| Programming Language | Code | Run | Debug (Not implemented yet) |
|---|---|---|---|
| Ada | ada, ada-12 | Console | Console |
| Assembler x86 | asm | Console | - |
| C | c, c-11, c-99, c-ansi | Console | Console or VNC |
| Clojure | clojure | Console | - |
| C++ | cpp, cpp-11, cpp-14, cpp-17, cpp-98 | Console | Console or VNC |
| C# | csharp | Console or VNC | Console |
| D | d | Console | - |
| Erlang | erlang | Console | - |
| Fortran | fortran | Console | Console |
| Go | go | Console | - |
| Haskell | haskel | Console | Console |
| Java | java, java-jgrasp | Console or VNC | Console or VNC |
| Javascript | java-script | Console | - |
| Lisp | lisp | Console | - |
| Lua | lua | Console | - |
| Matlab | matlab | Console or VNC | - |
| Pascal | pascal | Console | Console |
| Perl | perl | Console | Console |
| PHP | php | Console or VNC | - |
| Prolog | prolog | Console | - |
| Python | python, python3, (python-2) | Console or VNC | Console |
| python-3-pudb | Console | - | |
| R | r | VNC | - |
| Ruby | ruby | Console | Console |
| Scala | scala | Console or VNC | - |
| Schema | scheme | Console | - |
| Bash Shell | shell | Console | - |
| SQL | sql | Console | - |
| Verilog | verilog | Console | - |
| VHDL | vhdl | Console | - |
Writing test-cases for programming questions
In order to achieve test cases, candidate code must be written proper way. The code should contains the needed count of ReadLine (receive value from console) and WriteLine (write value in console) operations like it defined in the test cases. These operations are different for different languages, for example in С it can be: scanf(), printf(). In c# it is Console.ReadLine(), Console.WriteLine(), etc.
The programming question is automatically scored. To score it automatically author has to create/write/prepare test cases (Test cases tab). Every test case in file should have the following format:
| Field | Mandatory | Description | Example |
|---|---|---|---|
| Case | Yes | Name of the case. Case must start with this value | case = Test 7+5 |
| Input | Yes | At least one value should be set. New line for each value | input = 5 7 |
| Output | Yes | At least one value should be set. New line for each value. Possible types of outputs: only numbers (only numbers are checked out and the other characters are ignored) text (case-insensitive and the other characters are ignored) exact text (text is enclosed in double quotes) regular expressions (*) (It begin with / and ends with / and optionally one or several modifiers. Format is similar to JavaScript literal, but POSIX regex is used.) | output = 12 |
Example testcase:
case = Test 7+5
input = 5
7
output = 12
case = Test 10+18
input = 10
18
output = 28
case = Test 13+(-17)
input = 13
-17
output = -4
case = Test 1024+1024
input = 1024
1024
output = 2048
Programming question in candidate delivery
When answering the programming question the candidate has access to a code editor, where syntaxes will be highlighted. This editor can be maximised for optimal space for writing code.
Create: Allows the candidate to create a new page
Run: Allows the candidate to run the code within a console
Debug: Allows the candidate to debug the code
Evaluate: Allows the candidate to compile the code. If compilation fails, points will be subtracted from the max score.
Programming question: showing / viewing results in Reports
Admin: setup for executing programming questions
.png)
Programming question setup in your Cirrus environment
Go to Admin > Global settings > "VPL-tab"
Enter the address of your Execution server, something like https://vpl.jail.yourserver.com. There can be multiple execution servers, a load-balancer will check which execution server will be available.
Setting up an execution server for your programming questions
In order for your programming question to work you need to setup an execution server.
When a candidate The following steps are performed:
- Files submitted by the candidate are taken
- Files set by the teacher in "Execution files" are taken (this can replace files of candidate with same name)
- Depending on the action (run or evaluate) the script file set by the author are taken. If there is no script, the system take a default script by detecting the programming language used based on the extension of filenames.
- In the case of evaluation, if there is no script, the VPL program for automatic evaluation is also added. This program is based on the input and output of the program and requires that you specify the test cases in the file "vpl_evaluate.cases".
- This collected files are sent to an execution server
- The module informs the browser that the execution has begun
- If the request is an evaluation, when the task is finished, the result of the evaluation is retrieved from the execution server.
Marking programming question
In most situations a programming question will be automatically scored by the system. However Cirrus allows you to manually score auto-score questions.
When you score the question you will see the following information:
- Test #
- Description
- Error
- Input
- Program output
- Expected output (number)
- Summary of tests
Note: The actual code (file) of the candidate is not possible to see in marking