Output seed to stdout for tests #1645
Comments
larseggert
added a commit
to larseggert/neqo
that referenced
this issue
Oct 10, 2024
Into the directory given in the `DUMP_SIMULATION_SEEDS` environment variable. Also, export them as artifacts from the CI runs. Fixes mozilla#1645
github-merge-queue bot
pushed a commit
that referenced
this issue
Oct 11, 2024
* test: Dump simulator RNG seeds Into the directory given in the `DUMP_SIMULATION_SEEDS` environment variable. Also, export them as artifacts from the CI runs. Fixes #1645 * mkdir * Print leading zeros * TypoPrint leading zeros * Fixes * Conditional * Separate out sanitizer runs * Per sanitizer * Update test-fixture/src/sim/rng.rs Co-authored-by: Martin Thomson <mt@lowentropy.net> Signed-off-by: Lars Eggert <lars@eggert.org> * Update .github/workflows/check.yml Co-authored-by: Martin Thomson <mt@lowentropy.net> Signed-off-by: Lars Eggert <lars@eggert.org> * Update test-fixture/src/sim/mod.rs Co-authored-by: Martin Thomson <mt@lowentropy.net> Signed-off-by: Lars Eggert <lars@eggert.org> * Prefix seed with test name * Merge --------- Signed-off-by: Lars Eggert <lars@eggert.org> Co-authored-by: Martin Thomson <mt@lowentropy.net>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This is harder than it sounds.
The "simulator" we use in some of our randomized tests uses a random seed. The idea is that if a random test fails, we could take that seed and reproduce the exact failure conditions again to debug them. Originally, this seed was printed to stdout, which would be captured by the test runner, then later validated. Unfortunately, when I used the same code for benchmarking, stdout was not captured. I had to suppress this message.
Ideally, if we are benchmarking, we would suppress this output, but in regular test runs we wouldn't. It is possible that we could test
debug_assertionsfor this on that basis that benchmarks run in release mode and tests run in debug mode, but I'm not 100% confident that this is the right way to do that.The text was updated successfully, but these errors were encountered: