repobee-csvgrades

Important: repobee-csvgrades v0.2.0 and later require RepoBee 3.

A plugin for reporting grades into a CSV file based on issue titles. repobee-csvgrades adds the record category to repobee, along with the grades command, which operates on the JSON file produced by running repobee list-issues with the --hook-results-file option. The idea is pretty simple: find issues in student repositories that match certain user-defined conditions (title + who opened it) and report the corresponding grades into a (preferably version controlled) CSV file.

Example use case

Say that you have three students, slarse, glassey and glennol, three tasks task-1, task-2 and task-3, and that you (and any colleagues) open issues with titles Pass for pass and Fail for fail. repobee-csvgrades can then be configured to look for issues with those titles (using a regex), and write them into a grades sheet that looks something like this:

            name,username,task-1,task-2,task-3
    Simon Larsén,  slarse,     P,     P,     F
 Richard Glassey, glassey,     P,     P,     P
    Glenn Olsson, glennol,     P,     P,     P

GitHub and CSV files: GitHub renders CSV files really nicely, they are even searchable!

Only grades from issues that were opened by authorized teachers, as specified by you, are written to the file. Grades are mapped from issue title to a symbol, so the CSV file becomes more readable (here there's for example P for Pass). You also assign grades a priority, so if for example there's both a fail and a pass issue in a repo, it will pick the one that you've specified is most important. Additionally, each time new grades are reported, a summary message will be produced that looks something like this:

Record grades for task-3

@ta_a
slarse task-3 F

@ta_b
glassey task-3 P

The @ta_a and @ta_b are mentions of the teachers/TAs (with usernames ta_a and ta_b) that opened the issues. The intention is that this message should be used as a commit message for the grades file, so that the people who reported grades get notified of which grades have been written to the grades sheet.

That's just a quick introduction to what the plugin can do, see Usage for how to actually use it in practice.

Install

As of Repobee 3.0, installing a plugin is easier than ever.

First, make sure that you have the latest version of Repobee installed, instructions can be found on the Repobee website.

To install repobee-csvgrades, simply run the command repobee plugin install, and this will open up a bullet-list where you can select csvgrades.

If repobee-csvgrades does not show up when running the grades record command. Run repobee plugin list, this will present a list with all the availiable plugins. If you cannot find repobee-csvgrades in the list, try updating repobee using repobee repobee manage upgrade.

Usage

repobee-csvgrades is easy to use and highly customizable. First of all, you need to know how to use a plugin for RepoBee, see the RepoBee plugin docs. Then, there are a few key parts to familiarize yourself with before using it. The Quickstart section provides a brief introduction on how to use the plugin, without going into too much detail. The following sections explain the command line options in depth. Also don't miss the fact that you can configure all options in the configuration file.

Quickstart

To give you a very brief idea of how to use this plugin, let's have a look at the commands required to report grades. Some of what you see here won't make sense until you read the rest of the usage instructions, but it will give you an overview of the general usage and a good quick-reference once you've used the plugin once or twice.

Before we get started, let's activate the plugin. As it only adds commands, it's suitable for persistent activation.

$ repobee plugin activate
# follow the instructions on-screen to activate the plugin
$ repobee -h

Now, assume that we want to report grades for our three students slarse, glassey and glennol on their three tasks task-1, task-2 and task-3, as shown in the grades file from the example use case. We need to run two commands: one to fetch issues from the platform (e.g. GitHub or GitLab), and one to report grading issues from the fetched issues into the grades CSV file. So let's first fetch the issues.

$ repobee issues list --assignments task-1 task-2 task-3 \
        --students slarse glassey glennol \
        --all \
        --hook-results-file results.json

Note: This assumes the platform settings are configured, if they are not RepoBee will ask you for more options.

The issues are stored in the hook results file, which repobee-csvgrades then uses to report issues. The reason reporting is split in two steps like so is to avoid partially reporting grades in the case of a network failure.

Now, it's time to actually report the grades, and we can do that with the grades record command. It would look something like so.

$ repobee grades record --assignments task-1 task-2 task-3 \
        --students slarse glassey glennol \
        --hook-results-file results.json \
        --grade-specs '1:P:[Pp]ass' '2:F:[Ff]ail '3:c:[Cc]orrection' \
        --edit-msg-file edit_msg.txt \
        --teachers ta_a ta_b

And that should pretty much do the trick! In the following sections, the different options used here are described in more detail. We also highly recommend configuring some of the options, which you can find out more about in the Configuration section.

The grade specification (--grade-specs option)

The grade specification (or grade spec) is the most important part of this plugin. Grade specs tell the grades record command which issues to consider as grading issues, and which grading issues outweigh others if several are found. A typical grade spec looks like this: 1:P:[Pp]ass. There are three parts to the grade spec, delimited by :. First is the priority. A lower priority outweighs a higher priority. Second is the symbol that is written to the CSV grades file. Third is a regex to match against issue titles. Any issue whos title matches the regex is considered a grading issues of that spec.

Important: Your grade spec regexes should not overlap (i.e. be able to match the same strings). If they do, behavior is undefined.

Grade specs are specified by the --grade-specs option. Example:

--grade-specs '1:P:[Pp]ass' '2:F:[Ff]ail' '3:C:[Cc]orrection'

The hook results file (--hook-results-file option)

grades record operates on a file with a JSON database produced by the issues list command (one of RepoBee's core commands). The file is produced by supplying the --hook-results-file FILEPATH option to issues list. You should additionally supply issues list with the --all flag, to get both open and closed issues (so as to avoid missing grading issues). If you try to use grades record with a hook results file that's been produced without the --all flag, it will exit with an error. If you really want to run with that file, you can supply the --allow-other-states flag to grades record, which disregards how the hook results were collected.

The hook results file is specified by the --hook-results-file option. Example:

--hook-results-file ~/some_course/2019/hook_results_jan.json

The grades file (--grades-file option)

grades record writes grades to a CSV file that we refer to as the grades file. Each row represents one student, except for the first row which is a header row. The following requirements are placed on the CSV file format:

• Commas must be used as the delimiter
• The first line of the file must be a row of headers
• One of the headers must be username, and the column must include the usernames of all students to be graded
• There must be one column for each master repo, and the header for that column must exactly match the master repo's name

Below is an example grades file that has been partially filled in by the grades record command. As it is a CSV file, it is rendered very nicely on GitHub (see for example this test file), and it is strongly recommended that you keep this file in version control.

            name,username,task-1,task-2,task-3,task-4,task-5,task-6
    Simon Larsén,  slarse,     P,     P,     F,      ,      ,  
 Richard Glassey, glassey,     P,     P,     P,      ,      ,  
    Glenn Olsson, glennol,     P,     P,     P,      ,      ,  

There are a few additional things to keep in mind with the grades file.

• You should not manually edit the file with grade symbols for which there are no grade specifications, as this may cause grades record to exit because it can't find a priority for the grade.
• You can't have a task called username.
• You can't have duplicate column headers.
• You can have any additional columns that you want. grades record will only look at the username column, and the columns corresponding to the master repo names that you specify when calling the command. Additional columns will simply not be inspected.
• grades record formats the diff file such that every cell of the same column has the same width, which makes diffs easy to inspect.
  • Because of this formatting, it is recommended to keep grade spec symbols shorter than the master repo names, to avoid resizing of columns when grades are entered.

The grades file is specified by the --grades-file option. Example:

--grades-file ~/some_course/2019/grades.csv

The edit message file (--edit-msg-file option)

Each time you run grades record, a file is produced specifying what new grades were recorded, and tags the teachers who opened the grading issues. The intention is that this edit message should be used as a Git commit message. For example, if slarse has teacher ta_a, and glassey has teacher ta_b, and they both got new grades for task-3, the edit message might look like ths:

Record grades for task-3

@ta_a
slarse task-3 F

@ta_b
glassey task-3 P

The reason this edit message file exists is that some of our TAs felt a bit nervous about not knowing when their reported grades were collected. If this edit message is posted as the commit message, every teacher/TA whos grades have been collected will be notified, and the extra thorough ones can even inspect the diff to make sure everything is OK.

The destination for the edit message file is specified by the --edit-msg-file option. Example:

--edit-msg-file edit_msg.txt

Authorized teachers (--teachers option)

The grades record command requires you to specify a set of teachers that are authorized to open grading issues. This is to avoid having students trick the system. If an grading issue by an unauthorized user is found, a warning is emitted. This is both to alert the user about potential attempts at foul play, but also to help identify teachers that are actually authorized, but have not been included in the list.

Teachers are specified with the --teachers option. Example:

--teachers ta_a ta_b

Configuration

repobee-csvgrades can fetch information from the RepoBee configuration file, under the csvgrades section. All of the command line options can be configured. Use the config wizard command to configure csvgrades.

Important: The plugin must be active when running config wizard, or otherwise it will not be configurable. Here, we activate it with -p csvgrades.

$ repobee -p csvgrades config wizard
Editing config file at /home/slarse/.config/repobee/config.ini
Select a section to configure:
 repobee
●csvgrades # make sure to select the csvgrades section to configure

Note that some of the configurable options are lists, such as the teachers and grade-specs options. Simply separate each item in the list with a space, like you would on the commmand line. If an item contains a space (such as a regex in a grade spec), the whole item must be surrounded with single quotes.

License

repobee-csvgrades is released under the MIT license. See LICENSE for details.