Rebalance test allocation across jobs

[ricardoV94]

Now that #5777 is completed many tests changed place. We should check if the jobs need to be rebalanced to have similar runtimes

[ricardoV94]

CC @michaelosthege

[ricardoV94]

@Armavica is this something you would be interested in? It boils down to checking the last ~3 successful jobs and checking the total time each one took to try and redistribute tests from long jobs to short ones

[Armavica]

Sure, I will do that.

[Armavica]

Ubuntu and Windows can be rebalanced, but test_sampling.py is especially slow on Mac, and this will be the slowest runner after we rebalance Ubuntu and Windows.

Looking a bit closer at the individual test durations in test_sampling.py, it looks like no particular test is hanging everything up: everything is just about twice as slow on Mac than on Ubuntu. Should I try splitting up this file with pytest -k between Mac runners? Otherwise, we won't gain much from rebalancing Windows and Ubuntu.

[ricardoV94]

@Armavica Yeah we've noticed the Mac machines are quite slow. There were plans to split the contents of sampling.py in two files, so that might help: #6141

Otherwise we can move those tests to another platform

[Armavica]

Ok thanks, I will try to find a solution.

Otherwise we can move those tests to another platform

They are actually being tested on all three platforms (and a fourth time on windows with float32).

[Armavica]

To try and see through the variability of the GitHub runners, I put together a hierarchical model and analyzed the last 9 runs. It is not perfect but it seems consistent with what I usually observe. This will help to rebalance the tests.

Two things to note:

• The macos runners are indeed about 2-2.5x as slow as the Windows or Linux ones
• Besides the OS, there remains indeed a large variability at the instance level, of about two natural orders of magnitude (~x7).