Skip to content

Clarify status and message in test runner spec #175

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Jul 22, 2021
Merged

Clarify status and message in test runner spec #175

Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
2af18a6
Clarify top-level `fail` in test runner spec
ee7 Jul 7, 2021
8b5174a
Clarify per-test `message` in test runner spec
ee7 Jul 7, 2021
920131c
Clarify top-level `error` in test runner spec
ee7 Jul 7, 2021
718393f
Try to clarify test-level `error` in test runner spec
ee7 Jul 7, 2021
5daab45
Further clarify test-level `message`
ee7 Jul 22, 2021
e8cb9c5
Describe top-level `error` more consistently
ee7 Jul 22, 2021
60e3eda
Update building/tooling/test-runners/interface.md
ErikSchierboom Jul 22, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions building/tooling/test-runners/interface.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,16 +40,16 @@ The version of the spec that this file adheres to:
The following overall statuses are valid:

- `pass`: All tests passed
- `fail`: At least one test failed
- `error`: To be used when the tests did not run correctly (e.g. a compile error, a syntax error)
- `fail`: At least one test has the status `fail` or `error`
- `error`: No test was executed correctly (this usually means e.g. a compile error, or a syntax error)

#### Message

> key: `message`, type: `string`, presence: required if `status` = `error`, or when `status` = `fail` and `version` = `1`

> version: 1, 2, 3

Where the status is `error` (the tests fail to execute cleanly), the top level `message` key should be provided. It should provide the occurring error to the user. As it is the only piece of information a user will receive on how to debug their issue, it must be as clear as possible. For example, in Ruby, in the case of a syntax error, we provide the error and stack trace. In compiled languages, the compilation error should be provided. The top level `message` value is not limited in length.
Where the status is `error` (no test was executed correctly), the top level `message` key should be provided. It should provide the occurring error to the user. As it is the only piece of information a user will receive on how to debug their issue, it must be as clear as possible. For example, in Ruby, in the case of a syntax error, we provide the error and stack trace. In compiled languages, the compilation error should be provided. The top level `message` value is not limited in length.

When the status is not `error`, either set the value to `null` or omit the key entirely.

Expand Down Expand Up @@ -119,15 +119,15 @@ The following per-test statuses are valid:

- `pass`: The test passed
- `fail`: The test failed
- `error`: The test errored
- `error`: The test errored - that is, it did not return a value
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not happy with this, but I think we can do better than just "The test errored".

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah this is good enough I feel


#### Message

> key: `message`, type: `string`, presence: optional

> version: 2, 3

The per-test `message` key is used to return the results of a failed test. It should be as human-readable as possible. Whatever is written here will be displayed to the student when their test fails. If there is no error message, either set the value to `null` or omit the key entirely. It is also permissible to output test suite output here. The `message` value is not limited in length.
The per-test `message` key is used to return the results of a test with a `status` of `fail` or `error`. It should be as human-readable as possible. Whatever is written here will be displayed to the student when their test does not pass. If there is no test failure message or error message, either set the value to `null` or omit the key entirely. It is also permissible to output test suite output here. The `message` value is not limited in length.

#### Output

Expand Down