Use scores to evalute test results #4
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This hopefully makes things a bit mroe readable
Note, this removes the functionality to distinguish priority threshold failures.
It only handles unweighted scores, and does not return any information for error messages.
This is a bit of a big commit that implements all of the score based testing. The expected properties, unexpected properties, and priorityThresh are all scored now, instead of simply pass fail. It does quite a bit of fiddling with the values passed around, to make things slightly cleaner. Things it doesn't implement: * cleaning up setting up the test cases * actually passing configurable weights to test cases (though the scoring methods can handle it) TODO: * improve error handling quite a bit * add a few more test cases * add more comments
This message isn't currently used, but it could be nice to display some day.
This was needed only because some code elsewhere was not doing its job.
Previously, it was all in eval_test and was taking up quite a bit of space there.
Reduce, while sharing a name with the method on Array, is a little confusing not in context.
These conditions were only needed because code elsewhere wasn't properly sanitising the test case in one place.
This context properly adjusts locations, priorityThresh, and weights for every individual test case.
This creates a helper function setDefaultWeights that fully handles setting all default weights in one place, allowing some default setting in other functions to be removed.
This prevents having to pass a large number of arguments all the time
Otherwise, missing sub-objects or arrays can cause an error.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As described in the fuzzy tests story, this PR adds a ton of functionality to enable individual tests to be evaluated not just on a pass fail, but by a numeric score that can be influenced by specifying weights for individual values.
In addition there is a TON of refactoring that was very needed to make this change easy. There are now lots of individual files with very focused responsiblity, lots of unit tests, and lots of comments.
Fixes #2