Refactor grader, fix results.json overwrite

README.md

@@ -12,7 +12,7 @@ Follow the instructions above and in the other `README`s listed above.
 
 ## How to Use As a Student
 
-Thank you for using this framework! It is actually really simple to use. It allows you to submit your own test cases that are then uploaded, and you can run your code against any other students' uploaded tests as well! To get started, you can upload your test cases in a file called `tests.json` just with your normal Gradescope submission. The structure of this file is listed below, but you can also look at the file itself for more details:
+Thank you for using this framework! It is actually really simple to use. It allows you to submit your own test cases that are then uploaded, and you can run your code against any other students' uploaded tests as well! To get started, you can upload your test cases in a file called `tests.json` just with your normal Gradescope submission. The structure of this file is listed below, but you can also look at the file itself for more details. The folder `example-student-submission` contains an example submission on a hypothetical assignment to create a simple express server.
 
 ```json
 [

example-autograder/README.md

@@ -4,9 +4,9 @@ This is an example autograder, to upload it to gradescope, zip these files toget
 
 To setup the autograder, follow the following steps:
 
-1. You may start either with an existing autograder, or just copy everything from this directory (`example-autograder`). If you already have an existing grader, you don't need to copy the `run_autograder` and `setup.sh`. Instead, add on the single line of each of these to the end of your corresponding files. Note that you don't need to copy `sample-java-server`, as this is just a demonstration folder giving an example. If you want to copy this structure, as explained later, just rename it to `sample-server` and delete/overwrite the other `sample-server` folder.
+1. You may start either with an existing autograder, or just copy everything from this directory (`example-autograder`). If you already have an existing grader, you don't need to copy the `run_autograder` and `setup.sh`. Instead, add on the single line of each of these to the end of your corresponding files. Note that you don't need to copy `example-java-server`, as this is just a demonstration folder giving an example. If you want to copy this structure, as explained later, just rename it to `sample-submission` and delete/overwrite the other `sample-submission` folder.
 
-2. In the `test-grader` folder, modify the `config.json` to match what you want. You can find a description of the variables below. It is important that you include the `assignmenTitle` field (and preferably it should match the Gradescope assignment title, but this isn't required and it should still work).
+2. In the `test-grader` folder, modify the `config.json` to match what you want. You can find a description of the variables below. It is important that you include the `assignmenTitle` field (and preferably it should match the Gradescope assignment title, but this isn't required and it should still work) (e.g. for Homework 1 Milestone 1 it might be `hw1_ms1`).
 
 3. Again, in the `test-grader` folder, add a `.env` file that includes 3 variables:
 
@@ -18,13 +18,22 @@ To setup the autograder, follow the following steps:
 
     These correspond to the server IP and port that the backend server and database is running on (probably an EC2 instance or something similar). The `AUTH_TOKEN` is the token that this server requires to authenticate Gradescope. Set it to be the value of the token required by the server (which is set in the `.env` file in the `testit-server` server directory when it's started). Make sure this `.env` file is included when zipping the autograder, but it should be by default.
 
-4. In the `sample-server` directory, add in all files required to run your sample solution to the assignment. You may delete/replace the already existing `index.js` and `package.json` that show an example. Note that you can find an example of what a Java project would look like in the `sample-java-server` folder, but if you actually wanted to use something like this, you would need to rename the folder to `sample-server`.
+4. In the `sample-submission` directory, add in all files required to run your sample solution to the assignment. You may delete/replace the already existing `index.js` and `package.json` that show an example. Note that you can find an example of what a Java server project would look like in the `example-java-server` folder, but if you actually wanted to use something like this, you would need to rename the folder to `sample-submission`.
 
-5. Modify `setup-server.sh` to contain the command(s) needed to setup the environment of your sample solution. For example, when running an express server, it would just be `npm install` to install the required libraries. For running a Java server, it may be `mvn install`, `mvn clean install -DskipTests`, or maybe compiling some classes.
+5. Modify `pre-test.sh` to contain the command(s) needed to setup the environment of your sample solution (before running the tests). Note that this script doesn't necessarily need to terminate, see the `scriptTimeout` variable below for more details. For example, when running an express server, it would just be
+
+    ```
+    npm install
+    node index.js
+    ```
+
+    For running a Java server, it may include commands like `mvn install`, `mvn clean install -DskipTests`, `java -jar server-name.jar`, or possibly compiling some classes and running `java -cp classpath the.main.Class`.
 
     NOTE: This should NOT include any setup for the entire autograder docker container. For example, things like `apt-get update`, `apt-get install ...`, `pip install ...`, etc. should NOT be included in this file. These should be at the top of `setup.sh` at the root of the autograder. If you're confused about this distinction, think about it like this: would I need to run this command for every project/assignment I do, or just once on my entire computer? If it's the former, include it, if it's the later, don't.
 
-6. Modify `run-server.sh` to contain the command(s) to run your server. For example, with an express server, it might just be `node index.js`. For a Java server, if you have a JAR file, it may be something like `java -jar server-name.jar`, or if you're running compiled classes, it might be `java -cp classpath the.main.Class`. It may even be empty in some cases such as for Maven JUnit 4 test cases (see the `sample-java-server` example). Note that these commands should be identical to the ones that the students would run to start their servers, so make sure to instruct them that their solutions must run with these commands.
+6. Modify `post-test.sh` to contain the command(s) that need to execute after the tests are run. Normally this file can be left empty, but for an example where it might not be, consider the following: The assignment is for students to use DynamoDB and upload tests to a table, so the `pre-test.sh` creates the table (if the students aren't responsible for that), and then `post-test.sh` might delete the table.
+
+    Note that these commands should be identical to the ones that the students would run to start/use their submissions, so make sure to instruct them that their submissions must run with these commands. The `sample-submission` directory should exactly match the file structure of what a student submits, except with the addition of the pre and post test scripts (and the renaming of `tests.json` to `default-tests.json`) (and any other extra files your example submission might need).
 
 7. Finally, put your default test cases in `default-tests.json`, if you have any. For a description of the format and fields of these tests, please see the README of the parent (root) directory.
 
@@ -47,3 +56,5 @@ Here are a list of the config variables and what they affect. Note that all of t
 - `pomPath`: When using Maven and a Java server, this is the path to the `pom.xml` file. (In future versions, this is subject to removal in favor of the `pom.xml` file always being at the root). If you are using a different server, then this variable can be set to anything.
 
 - `jUnitTestLocation`: When using Maven and JUnit 4 tests, this is the path to the directory that contains all of the tests. (In future versions, this is subject to removal in favor of a field in each JUnit 4 test case written in the test file). If you are using a different server, then this variable can be set to anything.
+
+- `scriptTimeout`: **IMPORTANT:** The amount of time given to `pre-test.sh` and `post-test.sh` to run before timing them out. Note that `pre-test.sh` is NOT expected to terminate, for example in the case where its last command is to start a server (in the foreground). In this case, the timeout is the amount of time waited before running the tests. If an error is thrown in this time, the process exists and the error will appear, otherwise the tests will continue to be executed.

example-autograder/sample-submission/post-test.sh

@@ -0,0 +1 @@
+#!/bin/bash

example-autograder/sample-server/run-server.sh → example-autograder/sample-submission/pre-test.sh

@@ -1,3 +1,5 @@
 #!/bin/bash
 
+npm install
+
 node index.js

example-autograder/test-grader/config.json

@@ -4,6 +4,7 @@
   "maxTestsPerStudent": 10,
   "maxNumReturnedTests": 100,
   "weightReturnedTests": false,
+  "scriptTimeout": 10,
   "pomPath": "sample-java-project/pom.xml",
   "jUnitTestLocation": "sample-java-project/src/test/java/nets2120"
 }