Make random order stable and platform-independent #974
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
With this change, using the --order=random option with a particular seed will produce the same order on every Ruby interpreter. Previously, the random seed would produce consistent results on a given interpreter, but this could not always be reproduced on another version of Ruby. This occurred because Array#shuffle is implemented differently in different interpreters: Ruby 1.9.3 and JRuby 1.7.x share one implementation, whilst Ruby 2.x has a different one.
In addition to predictability, another useful property of random ordering is stability, i.e. if a given seed results in the order BCDA, then skipping a step will not affect the relative order of the other steps: if C is skipped, the order should be BDA. This is a necessary precondition for bisection, should we wish to add that in the future. As randomisation takes place before filtering we get stability for free. This change just documents it and preserves it from future regression.
|
Looks good @threedaymonk, thanks! |
|
@danascheider Do not forget to update the History.md, when you merge a PR (this PR, and issue #971 that it closes, needs to be mentioned there now). |
|
Sorry, will do! |
|
I often forget too. It's a hassle but I think it's well worth it. I just wish there was some automatic way of doing it... |
|
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This makes two important changes to random ordering.
--order=randomoption with a particular seed will now produce the same order on every Ruby interpreter.Previously, the random seed would produce consistent results on a given interpreter, but this could not always be reproduced on another version of Ruby. This occurred because
Array#shuffleis implemented differently in different interpreters: Ruby 1.9.3 and JRuby 1.7.x share one implementation, whilst Ruby 2.x has a different one.Stability of ordering is a necessary precondition for using bisection to resolve an ordering dependency (as RSpec can do), should we wish to add that in the future.
Ordering is now implemented by sorting on the 256-bit SHA2 hash of the string representation of the seed plus the index of the scenario. As randomisation takes place before non-matching steps are discarded, stability is maintained.
Fixes #971.