Specify that tests must be output in order #10
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In the Research Site, the responsibly of ordering the tests lay with the website. It used the `config.json` file as its basis for ordering. Now that file has been deprecated, the responsibility for ordering moves to the individual test-runners.
|
If your track only supports randomly-ordered test runs, use a stable seed (for example 42), if you want a "quick fix", before spending time to re-order the output. My proposal to re-order the output is literally doing that. When you have the results of the tests, order them based on the entry in the test file. You can use AST for that, but you can also use test-matches (and order by index). |
ErikSchierboom
approved these changes
Jan 5, 2021
iHiD
commented
Jan 5, 2021
|
I would rename this PR to output in order, not run. edit: haha that edit was literal. nice yoda grammar |
iHiD
changed the title
Co-authored-by: Jeremy Walker <jez.walker@gmail.com>
ErikSchierboom
approved these changes
Jan 6, 2021
Co-authored-by: Erik Schierboom <erik_schierboom@hotmail.com>
5 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In the Research Site, the responsibly of ordering the tests lay with the website. It used the
config.jsonfile as its basis for ordering. Now that file has been deprecated, the responsibility for ordering moves to the individual test-runners.I can't see a situation where a test-runner can't achieve this by reordering the tests based on grepping the file, but if anyone finds one, we can work out how to deal with it then.