Refactor test subsets in CI workflows

[michaelosthege]

Initially I wanted to re-use test subsets with anchors (#4517), but GitHub actions don't support YAML anchors :(

The Windows workflow was merged into the pytest.yml and the pytest job renamed to ubuntu.

Previously the Windows jobs ran with bash.exe as the shell, but this caused lots of problems related to maximum path lenghts. One option might have been (I didn't try) to change git options, but I decided to switch the shell to the standard Windows cmd. To deal with multiline run options I changed | to >- and &&. The environment variable is now resolved by %TEST_SUBSET% instead of $TEST_SUBSET.

The scipts\check_all_tests_are_covered.py (which currently just asserts that tests don't run multiple times, but only warns about tests that are ignored) was refactored to analyze the workflows based on their YAML job matrix.
This way it now prints a table of how often the tests are run with different OSes and floatX settings:

(pm3-dev) E:\Source\Repos\pymc3>python scripts\check_all_tests_are_covered.py
INFO:scripts\check_all_tests_are_covered.py:Found 42 tests in total.
INFO:scripts\check_all_tests_are_covered.py:Number of test runs (❌=0, ✅=once)
floatX                                            float32                                                float64
os                                           macos-latest ubuntu-18.04 ubuntu-latest windows-latest macos-latest ubuntu-18.04 ubuntu-latest windows-latest
testfile
pymc3/tests/test_aesaraf.py                             ❌            ✅             ❌              ❌            ❌            ✅             ❌              ❌
pymc3/tests/test_data_container.py                      ❌            ✅             ❌              ❌            ❌            ✅             ❌              ❌
pymc3/tests/test_dist_math.py                           ❌            ✅             ❌              ❌            ❌            ✅             ❌              ❌
pymc3/tests/test_distributions.py                       ❌            ✅             ❌              ✅            ✅            ✅             ✅              ✅
pymc3/tests/test_distributions_random.py                ❌            ✅             ❌              ✅            ✅            ✅             ✅              ✅
pymc3/tests/test_distributions_timeseries.py            ❌            ❌             ❌              ❌            ❌            ❌             ❌              ❌
pymc3/tests/test_examples.py                            ❌            ✅             ❌              ❌            ❌            ✅             ❌              ❌
pymc3/tests/test_gp.py                                  ❌            ✅             ❌              ✅            ❌            ✅             ❌              ✅
pymc3/tests/test_hmc.py                                 ❌            ✅             ❌              ❌            ❌            ✅             ❌              ❌
pymc3/tests/test_idata_conversion.py                    ❌            ✅             ❌              ❌            ❌            ✅             ❌              ❌
pymc3/tests/test_logp.py                                ❌            ✅             ❌              ❌            ❌            ✅             ❌              ❌
pymc3/tests/test_math.py                                ❌            ✅             ❌              ❌            ❌            ✅             ❌              ❌
pymc3/tests/test_minibatches.py                         ❌            ✅             ❌              ❌            ❌            ✅             ❌              ❌
pymc3/tests/test_missing.py                             ❌            ✅             ❌              ❌            ❌            ✅             ❌              ❌
pymc3/tests/test_mixture.py                             ❌            ❌             ❌              ❌            ❌            ❌             ❌              ❌
pymc3/tests/test_model.py                               ❌            ✅             ❌              ✅            ❌            ✅             ❌              ✅
pymc3/tests/test_model_func.py                          ❌            ✅             ❌              ✅            ❌            ✅             ❌              ✅
pymc3/tests/test_model_graph.py                         ❌            ❌             ❌              ❌            ❌            ❌             ❌              ❌
pymc3/tests/test_modelcontext.py                        ❌            ✅             ❌              ✅            ❌            ✅             ❌              ✅
pymc3/tests/test_ndarray_backend.py                     ❌            ✅             ❌              ❌            ❌            ✅             ❌              ❌
pymc3/tests/test_ode.py                                 ❌            ✅             ❌              ✅            ❌            ✅             ❌              ✅
pymc3/tests/test_parallel_sampling.py                   ❌            ❌             ❌              ❌            ❌            ❌             ❌              ❌
pymc3/tests/test_pickling.py                            ❌            ✅             ❌              ✅            ❌            ✅             ❌              ✅
pymc3/tests/test_plots.py                               ❌            ✅             ❌              ❌            ❌            ✅             ❌              ❌
pymc3/tests/test_posdef_sym.py                          ❌            ✅             ❌              ❌            ❌            ✅             ❌              ❌
pymc3/tests/test_posteriors.py                          ❌            ✅             ❌              ❌            ❌            ✅             ❌              ❌
pymc3/tests/test_profile.py                             ❌            ❌             ❌              ❌            ❌            ❌             ❌              ❌
pymc3/tests/test_quadpotential.py                       ❌            ✅             ❌              ❌            ❌            ✅             ❌              ❌
pymc3/tests/test_sampling.py                            ❌            ✅             ❌              ✅            ✅            ✅             ✅              ✅
pymc3/tests/test_sampling_jax.py                        ❌            ❌             ❌              ❌            ❌            ❌             ✅              ❌
pymc3/tests/test_shape_handling.py                      ❌            ✅             ❌              ❌            ❌            ✅             ❌              ❌
pymc3/tests/test_shared.py                              ❌            ✅             ❌              ✅            ❌            ✅             ❌              ✅
pymc3/tests/test_smc.py                                 ❌            ✅             ❌              ❌            ❌            ✅             ❌              ❌
pymc3/tests/test_starting.py                            ❌            ✅             ❌              ❌            ❌            ✅             ❌              ❌
pymc3/tests/test_step.py                                ❌            ✅             ❌              ❌            ❌            ✅             ❌              ❌
pymc3/tests/test_tracetab.py                            ❌            ✅             ❌              ❌            ❌            ✅             ❌              ❌
pymc3/tests/test_transforms.py                          ❌            ✅             ❌              ❌            ❌            ✅             ❌              ❌
pymc3/tests/test_tuning.py                              ❌            ❌             ❌              ❌            ❌            ❌             ❌              ❌
pymc3/tests/test_types.py                               ❌            ❌             ❌              ❌            ❌            ❌             ❌              ❌
pymc3/tests/test_updates.py                             ❌            ✅             ❌              ❌            ❌            ✅             ❌              ❌
pymc3/tests/test_util.py                                ❌            ✅             ❌              ❌            ❌            ✅             ❌              ❌
pymc3/tests/test_variational_inference.py               ❌            ❌             ❌              ❌            ❌            ❌             ❌              ❌
WARNING:scripts\check_all_tests_are_covered.py:8 tests are completely ignored:
{'pymc3/tests/test_mixture.py', 'pymc3/tests/test_tuning.py', 'pymc3/tests/test_model_graph.py', 'pymc3/tests/test_variational_inference.py', 'pymc3/tests/test_profile.py', 'pymc3/tests/test_distributions_timeseries.py', 'pymc3/tests/test_parallel_sampling.py', 'pymc3/tests/test_types.py'}

[michaelosthege]

Note: I can't reproduce the latest failures of the CI job by running this tests case individually.
I have a feeling that too many things are being added to the PATH..

Yeah, I can reproduce the error locally only by running the entire testfile, but not when running the tests individually.

[MarcoGorelli]

no objection here, but pandas is a pretty heavy dependency for a pre-commit hook. I'll have a look at this later, perhaps it can be done without it

[michaelosthege]

It certainly can be done with a bunch of dictionaries, but then the summarization doesn't work with one-liners and we'd have to do our own table printing. Not sure if that's worth the effort?

[twiecki]

Thanks @michaelosthege!