Seed flaky test TestSamplePPC.test_normal_scalar
#6220
Conversation
This is related to the issue Flaky test: test_normal_scalar pymc-devs#6211 pymc-devs#6211 * Remove call to function with unused output * Add assert to test function with list input * Add comment to refer to the issue
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## main #6220 +/- ##
==========================================
+ Coverage 93.58% 93.61% +0.02%
==========================================
Files 101 101
Lines 22136 22120 -16
==========================================
- Hits 20716 20707 -9
+ Misses 1420 1413 -7
|
|
Apparently, that call was covering some additional lines but no assert was there. I will understand what it was supposed to test and add an assert accordingly |
Not sure what you mean. For me this looks all good already. |
|
Relevant comment: #6210 (comment) Looks like the |
|
Found the problem, we need to pass explicitly the |
This really makes the samples fixed and reproducible. There are functions called by sample_posterior_predictive that ignore the global random seed if one is not provided explicitely.
| idata_ppc = pm.sample_posterior_predictive( | ||
| trace, var_names=["a"], random_seed=random_state | ||
| ) |
There was a problem hiding this comment.
Only this test requires a seed, the others are not checking anything that depends on the draws. Although passing a seed doesn't hurt, it makes reading the git history less straightforward (unless it was a commit about adding random_seed to all tests, and not just this one in particular).
There was a problem hiding this comment.
I see, removed the seed from all the others.
ricardoV94
changed the title
TestSamplePPC.test_normal_scalar
|
Thanks @mattiadg! |
This is related to the issue
What is this PR about?
The issue Flaky test: test_normal_scalar #6211 reported a Flaky test in test_normal_scalar https://github.com/pymc-devs/pymc/blob/main/pymc/tests/test_sampling.py#L617. I could not reproduce it yet, but I found other minor issues in the same test: calls to the tested function that were not calling any assert. I added an assert after the line with an input different from the other calls, and removed the other line that looks redundant. Under the suggestion of @michaelosthege, I added a comment asking to report further failing of this test under the same issue
Checklist
Major / Breaking Changes
Bugfixes / New features
Docs / Maintenance
TestSamplePPC.test_normal_scalar