Feature request: test cases without points

[lukasmoldon]

Hi,

as the title suggests, it would be helpful to have the option to create test cases, which have no effect on the final grading and do not have a score. For example, I would like to give the students feedback if their list of group members has the correct format and if it contains enough participants, without giving home assignment points for this sanity check. Visual feedback with colors (red/green) and text (comments, stdout) should be preserved.

Correct me if I am wrong, but at the current stage it is only possible to create test cases with "np.nan" as maximal score. But this implies a NaN score for the final sum of points at the end of the report.

best,
Lukas

[0b11001111]

What happens if you assign a score of 0 to the test?

[lukasmoldon]

What happens if you assign a score of 0 to the test?

ValueError: there is a mismatch the number of failed(1)/passed(1)/partially passed(0)/pending(0) tests and the total number of tests(1)

This is the result of a simple test script with only one test case, which has a maximum score of 0. I think there is no possibility to let such a test case fail or to let it pass. I tested this both for a test case with only pass in the function body and also for a test case containing only return 0 - with the same outcome (error from above). I guess the issue is, that a test case with score=0 can only fail and pass at the same time.

I am using the latest autograde version v0.4.2.

[0b11001111]

@tills98 can you take over here? One idea to face this was to make the score optional, i.e. setting the score to None for a test case explicitly. Be careful when implementing this as it will have implications at several places in the code base!