CS 125 Gradle Grading Plugin

This Gradle plugin provides autograding for Java-based assignments.

Features

• Parses output from JUnit-compatible test tasks and checkstyle

• Assigns test case point values from annotations

• Manufactures needed Gradle tasks to help reduce build script size

• Pretty-printed output to standard out for student inspection

• Multiple reporting options for resulting JSON, including local file dump and remote POST

• Can require students to commit their work after earning more points

• Supports Android modules tested by Robolectric

• Compatible with Gradle 5+

Installing

GradleGrader is available from Maven Central. To apply the Gradle plugin, add something like this to the project you need to grade, or to the root project if you’re grading multiple subprojects:

buildscript {
    repositories {
        mavenCentral()
    }
    dependencies {
        classpath 'com.github.cs125-illinois:gradlegrader:2020.1.3'
    }
}

apply plugin: 'edu.illinois.cs.cs125.gradlegrader'

gradlegrader {
    // Grading configuration goes here - see the Configuring section below
}

To allow using GradleGrader annotations in test suites, add the same package as a dependency to the project(s) containing graded test suites:

testImplementation 'com.github.cs125-illinois:gradlegrader:2020.1.3'

To apply the test dependency to all subprojects, add that to a subprojects → afterEvaluate → dependencies block in the root project’s build script.

If you are using a Kotlin build script or would like to use the new plugins block to apply the plugin, you will need to fill in the settings script (settings.gradle) to tell Gradle where to find our artifact:

pluginManagement {
    repositories {
        gradlePluginPortal()
        mavenCentral()
    }
    resolutionStrategy {
        eachPlugin {
            if (requested.id.namespace == "edu.illinois.cs.cs125" && requested.version != null) {
                useModule("com.github.cs125-illinois:${requested.id.name}:${requested.version}")
            }
        }
    }
}

Writing Tests

To mark a test case as graded, add the @Graded annotation on the test method and specify the point value with the points property. The optional friendlyName property sets the description of the test case in the report.

You can use the (repeatable) @Tag annotation to add tags to tests, which will be added to their result object in the output JSON. The name property becomes the JSON property name; the value property becomes the (string) value.

Configuring

All other configuration goes in the gradlegrader block in the Gradle script. That block has the following properties:

• assignment (optional) is the string ID of the assignment in your system. It produces an assignment property on the result JSON object.

• captureOutput (default false) sets whether to record the textual output of the checkstyle, compile, and test tasks. The output is combined and written to the output property of the final JSON object.

• forceClean (default true) sets whether to clean all graded subprojects before grading, i.e. require a completely fresh compilation.

• keepDaemon (default true) sets whether to leave the Gradle daemon running after the grade task completes. Setting this to false (i.e. killing the Gradle process) helps suppress Gradle spew but produces an IDE warning the first time, which may alarm students.

• maxPoints (optional) sets an overall points cap.

• subprojects (optional) sets the list of Gradle projects to grade. Each must have a test or testDebugUnitTest task. If this property is not specified, only the project to which the plugin is applied is graded.

The systemProperty function takes a name and value (both strings) to pass to the test JVM(s) as a Java system property (-D). The function can be used multiple times to add multiple system properties. To skip applying custom system properties, run the grade task with the -Pgrade.ignoreproperties command-line option.

gradlegrader also contains several blocks.

gradlegrader → checkpoint

This block allows you to manipulate test takes so that you can grade one "checkpoint" of a large project at a time.

• yamlFile (required to enable checkpointing) is a File that specifies where to find the student’s checkpoint selection. This file must have at least a checkpoint property.

• testConfigureAction (default no-op) is a BiConsumer that will be called for each test task, passed the currently selected checkpoint name and a test task to configure with it. You could use this to apply a test name include pattern based on the current checkpoint, for example.

You can also specify the test configuration action by passing an action/closure to the configureTests function.

gradlegrader → checkstyle

This block sets the checkstyle grading options. You must also apply the checkstyle plugin if you use checkstyle integration.

• configFile (required if this block is present) is the File representing the checkstyle rules XML file.

• exclude (defaults to **/gen/**, **/R.java, **/BuildConfig.java) is the list of file patterns to exclude from style checks.

• include (defaults to **/*.java) is the list of file patterns to check for style.

• points is the number of points earned for the absence of checkstyle violations.

• version (defaults to 8.18) is the checkstyle tool version to use.

gradlegrader → identification

This block sets the student identification policy.

• txtFile (required if this block is present) is the File that students need to fill out with their identities. If there are multiple students working on the project, this file should have one identifier per line.

• validate (defaults to always-true) is a Spec that each student identifier must pass. To require university email addresses, you can use { email -> email.endsWith('@university.edu') }.

• countLimit (defaults to must-be-one) is a Spec that the number of provided student identifiers must pass.

• message (optional) is a string to specify a custom error message for invalid contributor formats or counts.

gradlegrader → reporting

This block sets the grade reporting settings.

• jsonFile (optional) is the File where a JSON report will be saved.

• printJson (default false) sets whether the grade task will print the JSON result to standard output.

The tag function takes a name (string) and value (string or integer) and produces a property on the report JSON object.

gradlegrader → reporting → post

This block configures a POST request that submits the JSON report.

• endpoint (required if this block is present) sets the URL to make the request to.

The includeFiles function sets a list of Files (or a file collection) to be read and included in a files object on the POST JSON report. That information is not included in the local report.

A simple report logging server is included in this repository. See the Reporting Server section below for how to configure it.

gradlegrader → reporting → printPretty

This block configures the pretty grade report table printed to standard output for student inspection.

• enabled (default true) sets whether the pretty-printed table is shown.

• notes (optional) is the extra text to show below the grade report. This will be automatically word-wrapped to the appropriate width unless you add newlines yourself.

• showTotal (default true) sets whether the total score row is displayed.

• title (optional) sets a caption at the top of the table.

gradlegrader → vcs

This block enables VCS (currently only Git) integration.

• git (default false) enables Git integration, adding information about the student’s Git repository and identity to a git object on the JSON report.

• requireCommit (default false) requires students to commit their changes after their score increases. When this is enabled and the student’s best score increases, a note about committing is included in the pretty-printed table (if enabled) and the grade task will refuse to run again until the changes are committed. If checkpointing is enabled, checkpoint-specific best scores will be maintained.

Example

This is an example GradleGrader configuration block for an Android project with two modules:

gradlegrader {
    assignment 'Spring2019.MP0'
    checkstyle {
        points = 10
        configFile = file('config/checkstyle.xml')
    }
    identification {
        txtFile = file('email.txt')
        validate = { email -> email.endsWith('@example.edu') }
    }
    reporting {
        jsonFile = file('grade.json')
        post {
            endpoint = "https://example.com/progress"
        }
        printPretty {
            title = "MP0: Location"
            notes = "Note that the maximum local grade is 90/100. 10 points will be awarded during " +
                    "official grading if you have submitted code that earns at least 40 points by " +
                    "Monday, May 20."
        }
    }
    subprojects project(':app'), project(':lib')
    vcs {
        git = true
        requireCommit = true
    }
}

It has JUnit test suites with methods declared like this:

@Test(timeout = 300)
@Graded(points = 10)
@Tag(name = "difficulty", value = "simple")
@Tag(name = "function", value = "beenHere")
public void testBeenHereSimple() {
    // tests the student's beenHere function
}

If the student has an app module that fails to compile and a lib module that is partially correct, the human-readable output may look like this:

--------------------------------------------------------------------------------
MP0: Location
--------------------------------------------------------------------------------
Compiler                       0    app didn't compile
testFarthestNorthRandom        10   testFarthestNorthRandom passed
testFarthestNorthSimple        0    testFarthestNorthSimple failed
testNextRandomLocationRandom   10   testNextRandomLocationRandom passed
testNextRandomLocationSimple   10   testNextRandomLocationSimple passed
testBeenHereRandom             10   testBeenHereRandom passed
testBeenHereSimple             10   testBeenHereSimple passed
checkstyle                     0    checkstyle found style issues
--------------------------------------------------------------------------------
Total                          50
--------------------------------------------------------------------------------
Note that the maximum local grade is 90/100. 10 points will be awarded during
official grading if you have submitted code that earns at least 40 points by
Monday, May 20.
--------------------------------------------------------------------------------

The JSON report may look like this:

{
  "modules": [
    {
      "name": "app",
      "compiled": false
    },
    {
      "name": "lib",
      "compiled": true
    }
  ],
  "scores": [
    {
      "module": "app",
      "description": "Compiler",
      "pointsPossible": 0,
      "pointsEarned": 0,
      "explanation": "app didn't compile",
      "type": "compileError"
    },
    {
      "difficulty": "simple",
      "function": "farthestNorth",
      "module": "lib",
      "className": "edu.illinois.cs.cs125.spring2019.mp0.lib.LocatorTest",
      "testCase": "testFarthestNorthSimple",
      "passed": false,
      "pointsPossible": 10,
      "pointsEarned": 0,
      "failureStackTrace": "java.lang.AssertionError: expected:<-1> but was:<1>\n\t[truncated for brevity]",
      "description": "testFarthestNorthSimple",
      "explanation": "testFarthestNorthSimple failed",
      "type": "test"
    },
    {
      "difficulty": "simple",
      "function": "beenHere",
      "module": "lib",
      "className": "edu.illinois.cs.cs125.spring2019.mp0.lib.LocatorTest",
      "testCase": "testBeenHereSimple",
      "passed": true,
      "pointsPossible": 10,
      "pointsEarned": 10,
      "description": "testBeenHereSimple",
      "explanation": "testBeenHereSimple passed",
      "type": "test"
    },
    {
      "ran": true,
      "passed": false,
      "explanation": "checkstyle found style issues",
      "description": "checkstyle",
      "pointsEarned": 0,
      "pointsPossible": 10,
      "type": "checkstyle"
    },
    // some test cases removed for brevity...
  ],
  "git": {
    "remotes": {
      "origin": "https://github.com/example/MP0.git"
    },
    "user": {
      "name": "Example Person",
      "email": "example@users.noreply.github.com"
    },
    "head": "7b1df1a592b0959bf402673572fb3079435bf768"
  },
  "pointsEarned": 50,
  "pointsPossible": 70,
  "assignment": "Spring2019.MP0",
  "contributors": [
    "someone@example.edu"
  ]
}

All possible JSON properties are described below.

Output Format

The root JSON object has these properties:

• pointsEarned is the total number of points the submission earned, capped if directed by the maxPoints configuration property.

• pointsPossible is the total number of points available in all scored components.

• assignment is the value of the assignment configuration property or null if that property is not specified.

• checkpoint is the current checkpoint name from the student’s checkpoint selection YAML file, if checkpointing is enabled.

• output is the textual output from the checkstyle, compilation, and test tasks, if the captureOutput configuration property is true.

• contributors is an array of identifiers, if identification is enabled in the configuration.

Tags created by the tag directive in the reporting block appear as properties (with either string or integer values).

modules

This property is an array of objects, one for each tested module/subproject. Each object has these properties:

• name is the name of the Gradle subproject.

• compiled is whether any test results could be found for that module.

scores

This property is an array of objects, one for each scoring component (i.e. one line in the human-readable table report). Each object has these properties:

• description is a short summary of the item (shown on the left in the human-readable table):

  • "Compiler" if the item notes a module’s failure to compile

  • "checkstyle" for the checkstyle report

  • The test method name, or friendly name if set in @Graded, for test methods

• explanation is an explanation of why credit was or was not given (shown on the right in the human-readable table).

• pointsPossible is the number of points possible to earn from this item.

• pointsEarned is number of points earned: all points possible if the check/test succeeded, or zero if failed.

• type identifies what kind of item this is: compileError, checkstyle, testInitializationError, or test.

compileError items have this additional property:

• module is the name of the Gradle subproject that failed to compile.

checkstyle items have these additional properties:

• ran is whether checkstyle processed the sources without crashing and generated a report.

• passed is whether the report indicated no style violations.

testInitializationError items have these additional properties: * module is the name of the Gradle subproject containing the crashed test class. * className is the fully qualified class name of the test class that failed to initialize. * failureStackTrace is the stack trace of the exception that caused initialization to fail.

test items have these additional properties:

• module is the name of the Gradle subproject containing this test.

• className is the fully-qualified class name of the test suite.

• testCase is the test method name.

• passed is whether the test completed successfully.

• failureStackTrace is the stack trace of the exception that caused the test case to fail, if passed is false.

Tags created by @Tag annotations on test methods appear as additional properties with string values.

git

This property, present only if Git integration is enabled, is an object with these properties:

• remotes is an object with a property for each Git remote. The property name is the remote ID; the value is the URL.

• user is an object with these properties:

  • name is the Git user name.

  • email is the Git email address (which is not necessarily an organization address).

• head is the hash of the HEAD commit.

files

This property, present only in the POST report and only if there are any includeFiles configuration directives, is an array with an object for each included file. Each object has these properties:

• name is the unqualified file name.

• path is the full path of the file.

• data is the textual content of the file, if it could be read.

Reporting Server

The server module in this repository is a simple web server that logs POST reports into a MongoDB database. Before storing the POSTed document, it adds these properties:

• receivedVersion (string) is the current reporting server version

• receivedTime (date/time) is the time the report was received

• receivedIP (string) is the IP of the host submitting the POST report

• receivedSemester (string) is the configured semester

The server can be configured with environment variables or a config.yaml file from the working directory of the application. It has these configuration options:

• mongo (required) is the MongoDB connection string.

• mongoCollection (default gradlegrader) is the MongoDB collection to save reports in.

• http (default http://0.0.0.0:8888) is the host and port to listen on.

• semester (optional) is the semester to tag reports with.

A Docker container can be set up using the Dockerfile in the server module.