From 44ab0d50068561f212bc61c87ec066627a38058e Mon Sep 17 00:00:00 2001 From: Jeremy Walker Date: Wed, 6 Jan 2021 12:53:23 +0000 Subject: [PATCH] Specify that tests must be output in order (#10) * Specify that tests must be run in order In the Research Site, the responsibly of ordering the tests lay with the website. It used the `config.json` file as its basis for ordering. Now that file has been deprecated, the responsibility for ordering moves to the individual test-runners. * Update anatomy/track-tooling/test-runners/interface.md Co-authored-by: Jeremy Walker * Update anatomy/track-tooling/test-runners/interface.md Co-authored-by: Erik Schierboom Co-authored-by: Erik Schierboom --- anatomy/track-tooling/test-runners/interface.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/anatomy/track-tooling/test-runners/interface.md b/anatomy/track-tooling/test-runners/interface.md index aa5c69fd..dfceaf55 100644 --- a/anatomy/track-tooling/test-runners/interface.md +++ b/anatomy/track-tooling/test-runners/interface.md @@ -53,6 +53,17 @@ Where the status is `error` (the tests fail to execute cleanly), the top level ` When the status is not `error`, either set the value to `null` or omit the key entirely. +#### Tests + +> key: `tests` + +This is an array of the test results, specified in the "Per-test" section below. + +The tests **MUST** be returned in the order they are specified in the tests file. +For languages that execute tests in a random order, this may mean re-ordering the results in line with the order specified in the tests file. + +The rationale for this is that only the first failure is shown to students and therefore it is important that the correct failure is shown. Because tests are generally ordered in the tests file in a TDD way, and because for Practice Exercises the students see the tests file in the editor, aligning the results with the tests file is critical. + ### Per-test #### Name