Improve the stability and reliability of automatic grading

[markkuriekkinen]

From #470:
The grader may fail to send the grading results back to A+ because, for example,

• its disk becomes full during grading
• there is a network error when the grader tries to send HTTP POST to A+ frontend
• the grading container fails to send the results to the MOOC-grader
• MOOC-grader fails to receive the grading results from the container (the container sends an HTTP POST request to the MOOC-grader at the end of the grading)
• In addition, if grader doesn't resolve or it's http queue is full, submission is left in initialized state.
  • To elaborate, when A+ receives the submission, the submission is at first in the initialized state. If A+ fails to send the submission to the MOOC-grader (i.e., MOOC-grader does not respond or responds with an error), then the submission in A+ is stuck in the initialized state.

Submissions shouldn't get stuck so easily in the grading pipeline.

Related to #793.

#470 (comment)

Indeed, I was more paying attention to the proposed solution than problem description when closing the issue. Not sure how to make the recovery fully automatic (assuming that we can never fully get rid of network/server failure situations). Simple timer-based retries might just potentially congest the (under-failure) system unnecessarily -- what should be the length of timeout in that case, anyway? #793 might be good enough if failures are not very common (e.g., alert on course front page in teacher's view, with some sort of "retry all" button).

#470 (comment)

Yeah, it is not a trivial question. However, from a teacher's perspective, the student has submitted and the system loses it. It makes the platform look fragile and unreliable. Retrying the grading wouldn't always have to be immediate if there is a risk of overloading the system. One simple idea is that the system could look for lost/stuck submissions every hour and use the new mass regrade feature on them.

#470 (comment)

Yes. There could also be some sort of intelligence observing if mass-regrade-initiated grading for those submissions actually complete (and if not, maybe cancel mass regrade and try again after another hour). Which reminds me of a couple of additional features that could be added:

• Cancel-button for an ongoing mass regrade
• I guess submitters would appreciate email notification that the earlier postponed grading has completed

#470 (comment)

I'm not sure if I documented it anywhere, but I think I was visioning to solve this in the following way:

• If assessment request fail for an exercise, the exercise is put into maintenance mode or error mode. While in this mode, no assessments are attempted.
• Background task will poll backend of an exercise and if that exercise comes back online, then the mode of the exercise is restored and thus background assessment request can continue (assuming workers query DB for submissions waiting to be assessed).
• Assuming a single backend/domain handles multiple exercises, then there could be one more level for maintenance mode, where a backend services is put on the maintenance mode, which implies all exercises provided by it are in maintenance mode. Thus pollin is done in domain level and not in an exercise level.

That said, if amount of error emails doesn't bother, then running that mass regrade hourly might be an acceptable solution for now