Skip to content

Commit

Permalink
Clarify status and message in test runner spec (#175)
Browse files Browse the repository at this point in the history
* Clarify top-level `fail` in test runner spec

Consider the situation of:
- The user submits a solution in a compiled language
- The user's solution compiles without error
- The first test passes, but the next test "produces an error" (for
  example, it raises an exception).

In this case, the correct top-level status is `fail`, with one
test-level status of `pass`, and one test-level status of `error`. But
the previous wording could imply that a top-level status of `fail`
is only correct if at least one test has the status `fail`.

* Clarify per-test `message` in test runner spec

The previous wording could imply that a per-test `message` property
can only be present when the test `status` is `fail`.

* Clarify top-level `error` in test runner spec

To me, the previous wording implied that a top-level `status` of `error`
is valid when at least one test "errors". But actually, if at least one
test was "executed", the top-level status should be `pass` or `fail`.

* Try to clarify test-level `error` in test runner spec

* Further clarify test-level `message`

Co-authored-by: Erik Schierboom <erik_schierboom@hotmail.com>

* Describe top-level `error` more consistently

Co-authored-by: Erik Schierboom <erik_schierboom@hotmail.com>

* Update building/tooling/test-runners/interface.md

Co-authored-by: ee7 <45465154+ee7@users.noreply.github.com>

Co-authored-by: Erik Schierboom <erik_schierboom@hotmail.com>
  • Loading branch information
ee7 and ErikSchierboom authored Jul 22, 2021
1 parent 431d03c commit 0011404
Showing 1 changed file with 5 additions and 5 deletions.
10 changes: 5 additions & 5 deletions building/tooling/test-runners/interface.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,16 +40,16 @@ The version of the spec that this file adheres to:
The following overall statuses are valid:

- `pass`: All tests passed
- `fail`: At least one test failed
- `error`: To be used when the tests did not run correctly (e.g. a compile error, a syntax error)
- `fail`: At least one test has the status `fail` or `error`
- `error`: No test was executed correctly (this usually means e.g. a compile error, or a syntax error)

#### Message

> key: `message`, type: `string`, presence: required if `status` = `error`, or when `status` = `fail` and `version` = `1`
> version: 1, 2, 3
Where the status is `error` (the tests fail to execute cleanly), the top level `message` key should be provided. It should provide the occurring error to the user. As it is the only piece of information a user will receive on how to debug their issue, it must be as clear as possible. For example, in Ruby, in the case of a syntax error, we provide the error and stack trace. In compiled languages, the compilation error should be provided. The top level `message` value is not limited in length.
Where the status is `error` (no test was executed correctly), the top level `message` key should be provided. It should provide the occurring error to the user. As it is the only piece of information a user will receive on how to debug their issue, it must be as clear as possible. For example, in Ruby, in the case of a syntax error, we provide the error and stack trace. In compiled languages, the compilation error should be provided. The top level `message` value is not limited in length.

When the status is not `error`, either set the value to `null` or omit the key entirely.

Expand Down Expand Up @@ -119,15 +119,15 @@ The following per-test statuses are valid:

- `pass`: The test passed
- `fail`: The test failed
- `error`: The test errored
- `error`: The test errored - that is, it did not return a value

#### Message

> key: `message`, type: `string`, presence: optional
> version: 2, 3
The per-test `message` key is used to return the results of a failed test. It should be as human-readable as possible. Whatever is written here will be displayed to the student when their test fails. If there is no error message, either set the value to `null` or omit the key entirely. It is also permissible to output test suite output here. The `message` value is not limited in length.
The per-test `message` key is used to return the results of a test with a `status` of `fail` or `error`. It should be as human-readable as possible. Whatever is written here will be displayed to the student when their test does not pass. If there is no test failure message or error message, either set the value to `null` or omit the key entirely. It is also permissible to output test suite output here. The `message` value is not limited in length.

#### Output

Expand Down

0 comments on commit 0011404

Please sign in to comment.