Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

linearGaussian_snre_test fails #1090

Closed
bkmi opened this issue Mar 22, 2024 · 3 comments · Fixed by #1107
Closed

linearGaussian_snre_test fails #1090

bkmi opened this issue Mar 22, 2024 · 3 comments · Fixed by #1107
Assignees
Labels
bug Something isn't working

Comments

@bkmi
Copy link
Contributor

bkmi commented Mar 22, 2024

Describe the bug
The following tests fail (stochastically) since the training data produces one Inf for each of them.

test_api_snre_multiple_trials_and_rounds_map[SNRE_B-1]
test_api_snre_multiple_trials_and_rounds_map[SNRE_C-1]

To Reproduce
Please add a minimal code example that reproduces the problem:

  1. Name Python version and SBI version: Python 3.8.18 git commit 105008efbacf320758e43a5db5bee4e3a0f9de36
  2. Minimal code example pytest tests/linearGaussian_snre_test.py -k test_api_snre_multiple_trials_and_rounds_map
  3. [Optional]: error message

Expected behavior
should not create an Inf

Additional context
I am working on #966 , but it shouldn't affect this test.

@bkmi bkmi added the bug Something isn't working label Mar 22, 2024
@bkmi bkmi self-assigned this Mar 22, 2024
@bkmi bkmi changed the title linearGaussian_snre_test fails on test_api_snre_multiple_trials_and_rounds_map[SNRE_B-1] and test_api_snre_multiple_trials_and_rounds_map[SNRE_C-1] failing linearGaussian_snre_test fails on test_api_snre_multiple_trials_and_rounds_map[SNRE_B-1] and test_api_snre_multiple_trials_and_rounds_map[SNRE_C-1] Mar 22, 2024
@bkmi bkmi changed the title linearGaussian_snre_test fails on test_api_snre_multiple_trials_and_rounds_map[SNRE_B-1] and test_api_snre_multiple_trials_and_rounds_map[SNRE_C-1] linearGaussian_snre_test fails Mar 22, 2024
@bkmi bkmi mentioned this issue Mar 22, 2024
8 tasks
@bkmi bkmi removed their assignment Mar 22, 2024
@famura
Copy link
Contributor

famura commented Mar 22, 2024

I just ran pytest tests/linearGaussian_snre_test.py -k test_api_snre_multiple_trials_and_rounds_map on the current main (commit b81420b). The first 2 tries passed, on the 3rd one I got. This error seems to be non-deterministic.

FAILED tests/linearGaussian_snre_test.py::test_api_snre_multiple_trials_and_rounds_map[SNRE_B-1] - ValueError: Found 0 NaN simulations and 2 Inf simulations.SNRE does not allow invalid simulations. Replace the invalid values with an unreasonably low or hig...

@michaeldeistler I think that this is related to what I approached you about, even though the tests in our PyMC PR have been successful afterward. However, now I think we just got lucky 😅

@manuelgloeckler
Copy link
Contributor

manuelgloeckler commented Mar 24, 2024

Just happened on the main branch :o Rerunning the workflow fixed it, so yeah seems to be stochastic.

See: https://github.com/sbi-dev/sbi/actions/runs/8410614947/attempts/1

Seems to be related to Inf/Nan, probably in multi-round SNRE. Likely related to #1060

@famura
Copy link
Contributor

famura commented Mar 25, 2024

Check out my PR #1107 @manuelgloeckler @bkmi. I think I am about to fix this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants