Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change TestRegression class test methods to fix victim flakiness #303

Merged
merged 1 commit into from
Oct 2, 2022

Conversation

blazyy
Copy link
Contributor

@blazyy blazyy commented Sep 30, 2022

When using the pytest-flakefinder plugin with the TestRegression module, it fails. To reproduce, run:

pytest --flake-finder --flake-runs=2 -rP pingouin/tests/test_regression.py::TestRegression

For context, the flakefinder plugin runs the tests multiple times in a different order each time. This helps us check how robust the tests are. In this case, the tests failed because the two test modules inside TestRegression, test_logistic_regression and test_mediation_analysis were victim tests (defined as per iFixFlakies). The first test, test_linear_regression, was polluting the df variable.

More specifically, at least for the test_logistic_regression test, the Ybin column was getting populated with a single 2 value, when it is supposed to contain only 0s and 1s (as per mediation.csv). Re-initializing df variable in each test seems to fix this problem. The entire module runs fine if the tests are run in the current order as they are written in, but fails if run in any other order.

Since the tests access the same data, I thought of using pytest namespaces and pytest fixtures to make the code changes less repetitive but that would involve a lot of variable name changes, so this is just a quick and reliable fix for you to consider.

@codecov
Copy link

codecov bot commented Sep 30, 2022

Codecov Report

Base: 98.71% // Head: 98.70% // Decreases project coverage by -0.00% ⚠️

Coverage data is based on head (ec6c167) compared to base (c66b685).
Patch has no changes to coverable lines.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #303      +/-   ##
==========================================
- Coverage   98.71%   98.70%   -0.01%     
==========================================
  Files          19       18       -1     
  Lines        3335     3312      -23     
  Branches      537      537              
==========================================
- Hits         3292     3269      -23     
  Misses         25       25              
  Partials       18       18              
Impacted Files Coverage Δ
pingouin/__init__.py

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@raphaelvallat
Copy link
Owner

Hi @blazyy,

Thanks for this, good catch! There is a unit test failure on Py3.8-windows, which is unrelated to this PR so I'll approve anyway.

@raphaelvallat raphaelvallat merged commit bc79523 into raphaelvallat:master Oct 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docs/testing:book: Documentation and unit testing
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants