Skip to content

Commit

Permalink
Specify that tests must be output in order (#10)
Browse files Browse the repository at this point in the history
* Specify that tests must be run in order

In the Research Site, the responsibly of ordering the tests lay with the website. It used the `config.json` file as its basis for ordering. Now that file has been deprecated, the responsibility for ordering moves to the individual test-runners.

* Update anatomy/track-tooling/test-runners/interface.md

Co-authored-by: Jeremy Walker <jez.walker@gmail.com>

* Update anatomy/track-tooling/test-runners/interface.md

Co-authored-by: Erik Schierboom <erik_schierboom@hotmail.com>

Co-authored-by: Erik Schierboom <erik_schierboom@hotmail.com>
  • Loading branch information
iHiD and ErikSchierboom authored Jan 6, 2021
1 parent 46a02f0 commit 44ab0d5
Showing 1 changed file with 11 additions and 0 deletions.
11 changes: 11 additions & 0 deletions anatomy/track-tooling/test-runners/interface.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,17 @@ Where the status is `error` (the tests fail to execute cleanly), the top level `

When the status is not `error`, either set the value to `null` or omit the key entirely.

#### Tests

> key: `tests`
This is an array of the test results, specified in the "Per-test" section below.

The tests **MUST** be returned in the order they are specified in the tests file.
For languages that execute tests in a random order, this may mean re-ordering the results in line with the order specified in the tests file.

The rationale for this is that only the first failure is shown to students and therefore it is important that the correct failure is shown. Because tests are generally ordered in the tests file in a TDD way, and because for Practice Exercises the students see the tests file in the editor, aligning the results with the tests file is critical.

### Per-test

#### Name
Expand Down

0 comments on commit 44ab0d5

Please sign in to comment.