You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We currently run the submodules in the tests/ directory in separate jobs:
benchmarks
runtime
stability
Each of those jobs has its own test matrix, so we actually wind up having a large number of sub-jobs for every workflow run.
I'm not convinced that is serving us well today:
Each job is parallelized using pytest-xdist with ten workers. That makes a lot of sense, since the vast majority of the work is done by coiled clusters, and the CI runner is mostly sitting around waiting for results. However, we aren't getting the full value out of pytest-xdist because we are running it with loadscope, so each test module is run as a single unit. This basically means we don't have enough modules to actually saturate the xdist workers.
The different jobs have different test matrices. In particular, "stability" has the largest matrix, testing a few combinations that are not tested in others. I think this is because, at the time of writing, "stability" was the fastest submodule to run, so we could afford the extra matrices (is this right @jrbourbeau?). But since Integration tests for spilling #229 the stability tests are quite slow. @hendrikmakait may have ideas to speed it up a bit, but I still am not convinced it's a good idea to have different test matrices for different submodules.
There is a lot of code duplication in the workflow yaml for each submodule, which adds cognitive overhead.
I propose we consolidate the above jobs into a single "test" job, which is itself parameterized over a consistent test matrix. I doubt that would actually change the runtime too much, and might even speed it up, as the xdist workers would be better able to balance work. It would also simplify maintenance going forward.
The text was updated successfully, but these errors were encountered:
We currently run the submodules in the
tests/
directory in separate jobs:Each of those jobs has its own test matrix, so we actually wind up having a large number of sub-jobs for every workflow run.
I'm not convinced that is serving us well today:
pytest-xdist
with ten workers. That makes a lot of sense, since the vast majority of the work is done by coiled clusters, and the CI runner is mostly sitting around waiting for results. However, we aren't getting the full value out ofpytest-xdist
because we are running it withloadscope
, so each test module is run as a single unit. This basically means we don't have enough modules to actually saturate the xdist workers."stability"
has the largest matrix, testing a few combinations that are not tested in others. I think this is because, at the time of writing,"stability"
was the fastest submodule to run, so we could afford the extra matrices (is this right @jrbourbeau?). But since Integration tests for spilling #229 the stability tests are quite slow. @hendrikmakait may have ideas to speed it up a bit, but I still am not convinced it's a good idea to have different test matrices for different submodules.I propose we consolidate the above jobs into a single "test" job, which is itself parameterized over a consistent test matrix. I doubt that would actually change the runtime too much, and might even speed it up, as the xdist workers would be better able to balance work. It would also simplify maintenance going forward.
The text was updated successfully, but these errors were encountered: