Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Random assertions can fail randomly (despite seeding) #533

Open
penelopeysm opened this issue Oct 10, 2024 · 3 comments · May be fixed by #539
Open

Random assertions can fail randomly (despite seeding) #533

penelopeysm opened this issue Oct 10, 2024 · 3 comments · May be fixed by #539

Comments

@penelopeysm
Copy link
Member

penelopeysm commented Oct 10, 2024

e.g. this block in the VI tutorial:

```{julia}
#| echo: false
let
v, m = (mean(rand(q, 2000); dims=2)...,)
# On Turing version 0.14, this atol could be 0.01.
@assert isapprox(v, 1.022; atol=0.1) "Mean of s (VI posterior, 1000 samples): $v"
@assert isapprox(m, -0.027; atol=0.03) "Mean of m (VI posterior, 1000 samples): $m"
end
```

Somehow, these assertions failed in one CI run: https://github.com/TuringLang/docs/actions/runs/11281015562/job/31375300973

Here's another failure of the same code block: https://github.com/TuringLang/docs/actions/runs/11281201489/job/31376048754

Here's the Gaussian mixture model tutorial failing: https://github.com/TuringLang/docs/actions/runs/11283217753/job/31382221263

Not sure how this happened as the random seed was already set at the top of the file. It seems that some function isn't respecting it.

@penelopeysm
Copy link
Member Author

Turns out Random.seed! only affects calls within the same code cell. Unsure why.

@penelopeysm
Copy link
Member Author

penelopeysm commented Oct 10, 2024

Reported upstream: PumasAI/QuartoNotebookRunner.jl#192

@penelopeysm penelopeysm changed the title VI tutorial can fail randomly Random assertions can fail randomly Oct 10, 2024
@penelopeysm penelopeysm changed the title Random assertions can fail randomly Random assertions can fail randomly (despite seeding) Oct 10, 2024
@penelopeysm
Copy link
Member Author

Upstream issue fixed, so the remaining task is to re-enable asserts and check that they don't fail.

@penelopeysm penelopeysm linked a pull request Oct 17, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant