Change TestRegression class test methods to fix victim flakiness #303
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When using the
pytest-flakefinder
plugin with the TestRegression module, it fails. To reproduce, run:pytest --flake-finder --flake-runs=2 -rP pingouin/tests/test_regression.py::TestRegression
For context, the flakefinder plugin runs the tests multiple times in a different order each time. This helps us check how robust the tests are. In this case, the tests failed because the two test modules inside
TestRegression
,test_logistic_regression
andtest_mediation_analysis
were victim tests (defined as per iFixFlakies). The first test,test_linear_regression
, was polluting thedf
variable.More specifically, at least for the
test_logistic_regression
test, theYbin
column was getting populated with a single2
value, when it is supposed to contain only0
s and1
s (as per mediation.csv). Re-initializingdf
variable in each test seems to fix this problem. The entire module runs fine if the tests are run in the current order as they are written in, but fails if run in any other order.Since the tests access the same data, I thought of using pytest namespaces and pytest fixtures to make the code changes less repetitive but that would involve a lot of variable name changes, so this is just a quick and reliable fix for you to consider.