Skip to content

Grading Features

Jump to bottom
Gabriel Chang edited this page May 18, 2024 · 36 revisions

Table of Contents

2019 CP3108A: Team GRADING (cadet-frontend)

The project is a combination of multiple features, distributed across the client (cadet-frontend), server (cadet) and autograder (grader) repositories.

The improvements are:

  • Editor template reset button (client)
  • Integration of new XML format and autograder improvements (client and server)
  • Refactoring of autograder (server and grader)
  • Client-side public test case support (client)
  • Grading overview overhaul (client and server)
  • Server-side pagination (client and server)

Scroll below to view the documentation for the changes made to the client.

Feature 1: Editor template reset button (Completed)

Rationale: Enable students to quickly restore the editor state to the template code in assessments.

Pull requests

Changelog
  • Added a template reset button in the assessment workspace (excluding the grading workspace and playground) to allow users to reset the editorValue to the assessment template
Outstanding issues
  • Add tests for reset button in assessment workspace

Feature 2: Integration of new XML format (Completed)

Rationale: Update frontend to support the updated XML format for creation and deployment of assessments - the updated specifications are available here.

Pull requests

Changelog
  • Added editorPrepend, editorPostpend, editorTestcases attributes to StateProps of assessment and grading workspaces
  • Added new ITestcase interface to represent testcases
Outstanding issues
  • Integrate new workspace properties with Mission Control feature to allow editing of prepend, postpend code fragments and testcases during XML creation
  • Add tests for mission questions with varying combinations of prepend, postpend and testcases

Feature 3: Refactoring of autograder (Completed)

Please proceed to the grader repository reference.

Feature 4: Client-side public test case support (In progress)

Rationale: Where intended by mission authors, expose public test cases to expedite students' basic testing of their programs - not an intended replacement for self-design of test cases, instead serving as a convenience feature by collating test cases previously in mission briefings.

Pull requests

  • Client: PR 527 (bundled with XML changes)
Changelog
  • Added a new Test case tab that parses and displays data stored in the editorTestcases attribute of the assessment and grading workspaces
    • Displays test case (function call), expected output and actual output (if ran)
    • Add new EVAL_TESTCASE action to allow each test case to be run individually
    • Updated workspaceSaga to handle EVAL_TESTCASE actions and added new evalTestCode saga
Outstanding issues
  • Add additional button to run all test cases concurrently
  • Standardise and clean up UI elements
  • Automatically compare expected
  • Add tests for test cases from different mission varieties

Feature 5: Grading overview overhaul (Completed)

Rationale: Improve the UX for graders by cleaning up the UI of the grading overview page and providing more more granular information on each submission.

Pull requests

Changelog
  • Added check to disable grading status display for non-graded assessments (i.e. paths)
  • Corrected ag-grid CSS stylesheet import and amended CSS to fix styling inconsistencies and persistent scrollbar bug
  • Collated all control buttons above the datagrid
  • Amended handling of FETCH_GRADING_OVERVIEWS actions in backendSaga
  • Added gradingStatus, questionCount and gradedCount attributes to GradingOverview type
  • Added grading status display for submissions with on-hover grading progress tooltip
  • Enabled sorting and filtering by grading status and group
  • Updated mocks
Outstanding issues
  • Add additional tests for grading overview page

Feature 6: Server-side pagination (Completed)

Rationale: Implement server-side pagination for the /grading endpoint (retrieval of submissions) to reduce amount of data transmitted between client and server.

Pull requests

Changelog
  • Disabled ag-grid's native pagination and added custom pagination controls
  • Added button to select pagination page size (#submisions to retrieve)
  • Amended /grading endpoint (async function getGradingOverviews) from using a HTTP GET to POST request
    • Pagination details, search, filter and sort details are all contained in the transmitted JSON body
  • Replicated search functionality server-side
    • Replaced quick filter searchbar with tag-based searching by name, assessment, category and group
  • Replicated sorting functionality server-side
  • Replicated column filter functionality server-side
    • Text-based column filters disabled
    • Number-based column filters enabled for XP and grade columns
Outstanding issues
  • Fix UI issue with outline CSS with :focus pseudo-selector of the page size dropdown
  • Update mocks and add tests for POST request

2024 CP3108: Grading Features team

1: Include assessment with answers in submission

Implementation features and reasoning:

  • This backend change was to allow for assessment briefing to be included while grading.

2: Pagination from frontend to backend

Previously, due to changes under Frontend PR 2437, pagination was done by the client. All submissions had to be loaded first before the frontend table shows 10 entries at a time - a waste of time later into the course as the submissions table increased to several hundred thousand entries. This change unfortunately also introduced major dependencies as they used the tanstack and tremor libraries that were not used elsewhere in the frontend.

This feature reverts some of the damage of the PR - it shifts work of filtering from the frontend back to the backend. The revert of the libraries used to reduce dependencies have been pursued by the Grading UI/UX team under Frontend PR 2893.

With locally-tested benchmarking, this change managed to cut loading times by about 13x (from ~30s to ~2s), and load sizes by 4100x (from the order of megabytes to kilobytes) for loading the first page.

Implementation features and reasoning:

  • Limit number of entries requested by the backend by using OFFSET and LIMIT, which is calculated by the table state within the frontend (page number, number of entries per page)
  • The previous frontend feature, which showed the grading summaries for each avenger, was removed. This is because the pagination change from the frontend to the backend broke this feature as it did not retrieve all submissions. However, we discussed that this feature was unlikely to be useful to avengers as the main use case would be to ensure that all submissions were graded - something that could have been done using the "ungraded" filter.
image
  • We have also decided to reduce the functionality of the search bar. Previously, the search bar was able to filter by all categories. However, we decided to limit the scope of the search bar, which can now only filter by assessment name. We made this decision to simplify the backend calls made. This restriction allowed us to greatly simplify the complexity of backend work as it avoided the need for OR conditional querying, complementary querying (filtering by values NOT equal to something), or comparative querying (return all rows whereby graded questions < total questions). This regression can be partially reverted, and will be a good QoL improvement: Frontend issue 2820
  • Other frontend feature enhancements:
    • additional dropdown option to allow the user to set the number of entries per page that they see.
    • Introduces 2 additional page control buttons "<<" and ">>" that set the page to the minimum and maximum respectively.

Backend Implementation

  • Uses Ecto and the main function is submissions_by_grader_for_index.
  • Builder functions (helpers) are used to separate the filters by tables, e.g. build_submission_filter creates filters for the submission table. The format can be thought of as if statements and if {desired_key, desired_value} are present in the query parameter, we will add the following SQL query.

Issues:

  • It breaks the export grading CSV function, which can only export off the limited backend queries it gets. However, we deemed that this was a minor regression as the previous export CSV function did not fit a useable standardized format. If interested, this issue is described under https://github.com/source-academy/frontend/issues/2814

3: Auto-seeding feature

Implementation features and reasoning:

  • To allow for easier testing the impact of pagination and to have a variety of dummy data available, the seeding was changed from hardcoded to a scalable seed. This allows us to easily customise the amount of data to be seeded.
  • By default, the database is populated with 10 students and 5 assessments. Each student will have a submission to the corresponding submission.
  • In priv/repo/seeds.exs there are variables number_of_students, number_of_assessments and number_of_questions. These can be changed to the desired amount.

4: Publishing of grading feature, streamlining of grading workflows, auto-publishing, admin bulk publishing

Previously, graded submissions were immediately shown to students, even if the avenger was in the middle of grading it. Additionally, this also caused an issue for avengers as it was harder to find re-submitted work: Issue 2522.

This major feature change thus implements a 'publish' button for both autograded and manually graded work. Other related features are borne out of the consequences of having the publishing feature.

  • Introduces bulk publishAll and unpublishAll controls within GroundControl (admins only), allowing for batch publishing.
  • Introduces isAutoPublished that permits autograded submissions to be automatically published by the autograder, preventing trivial assessments like Paths from wasting avengers' time
  • Revamps columns in grading submissions table + publishing status into a single column for simplified tracking
  • Tweak notifications to reflect published and unpublished actions instead of graded / ungraded actions

Implementation details and reasoning:

Frontend reasoning is detailed within the pull request itself here, and is copied here for reference:

Introduce isGradingPublished This pull request implements the isGradingPublished feature as described on #2522, but in a modified manner. The original intention was focused on manually graded work which had been resubmitted. This pull request introduces this new field to the frontend to help in this - grading will not be shown until publishing. However, this pull request additionally implements the published category for autograded assessments. This is to allow for cases when hiding autograder results is be desirable, such as for PEs.

image

^New interface for grading submissions showing implementation of isGradingPublished. (Note: this is a display on tanstack, and will look a bit different after the imminent migration to ag-grid under #2893.)

image

^Even though mission has been fully graded, student sees "grading in progress.". Students can only see grades once published.

publishAll, unpublishAll

A publishAll and unpublishAll feature has been added to the admin-only ground control to allow for bulk publishing / unpublishing.

image

^new field added to controls, 'Release Grading', with unpublish and publish all buttons added in the category (in that order)

image

^example prompt when 'publish all' button is clicked - checks are done in the backend to prevent publishing of non-graded work

isAutoPublished

To avoid the inconvenience of publishing for trivial autograded assessments like Paths, a new backend field, 'isAutoPublished', has been added to each type of assessment, controllable within the admin panel. This field permits the autograder to publish on behalf of the avenger one it has fully graded the work. This field can only be set to true for assessments that are not manually graded (frontend and backend guards are present to prevent manually graded assessments from being auto-published, and vice-versa).

image

^new toggle, "is Auto-published", has been introduced.

Revamp derived frontend field

With the introduction of publishing for submissions, multiple backend fields can now be revamped and combined into a single comprehensive 'progress' state - you would have observed this in the screenshots above as the single column, 'progress', for the grading submissions table.

This new field is calculated as such:

isManuallyGraded status is fully graded? is grading published? progress
false ? ? ? autograded
true attempting ? ? attempting
true attempted ? ? attempted
true submitted false ? submitted
true submitted true false graded
true submitted true true published

Multiple backend parameters are considered and combined into a single field, allowing avengers to easily track and filter the submissions they need to grade under a single column.

image

^old control flow - credit to PR description in #2522

image

^old appearance of grading table - notice that 2 columns are used to track each assessment.

Remove avenger need to track autograded work

newAvengerWorkflow

^new control flow for avengers.

All submissions which are part of autograded work has been marked as 'autograded'. Under our current changes, we expect 2 scenarios, both of which do not require the TA to be involved.

  • Autograded, autopublished assessments (e.g. Paths): They should not be unsubmitted or published by the TA. It is unfair since the marking scheme will be automatically revealed - besides, it should be too trivial to be cared about.
  • Autograded, manually published work (e.g. practical exams): They are too serious and should not be able to be unsubmitted or published by the avenger. Only the admins should be able to publish autograding all at once using Ground Control.

Hence, our default filter will hide autograded work and disable actions for autograded work. Avengers will only be able to look through submissions and adjust XP if absolutely needed (see the lack of grading actions available for autograded work above).

Notification message change

image

^Old notifications show "autograded" notification even when not yet published.

image

^New notifications do not show any notification until published.

Miscellaneous (code quality improvements, bugfixes):

Issues

Clone this wiki locally