Speeding up CI

[JukkaL]

CI is currently pretty slow. In particular, if there are several PRs in quick succession, we'll run out of CI capacity. A reasonable medium-term goal might to get CI builds finish in 10 minutes in typical cases. I'm not sure if that is realistic, however.

Here are some ideas about how to make things better:

• Drop appveyor (it seems redundant now) [done]
• Run selected slow tests only after a PR has been merged
  • Alternatively, only run them once per PR iteration, and run on all platforms and Python versions after merge
• If a PR only touches mypyc, don't run non-compiled mypy tests
• Remove some slow tests from CI if they don't bring much value
• Run more tests in incremental mode
• If a PR only touches docs, don't run all tests
• Test fewer platform / Python version combinations (possibly only in PR builds)
• Speed up individual tests somehow
• Somehow get more CI capacity

Can anybody suggest more things that could help, or which of the above things would be worth pursuing first?

[srittau]

Two more ideas:

• Run expensive tests once per day for master only.
• Get more capacity by migrating some tests to GitHub Actions.

[JukkaL]

Run expensive tests once per day for master only.

I like this idea!

Get more capacity by migrating some tests to GitHub Actions.

If I remember correctly, Travis has similar (at least parallel) capacity to GitHub Actions. If that's correct, having them both set up might give us more total capacity. My knowledge may not be up to date, however.

[emmatyping]

I think we can pretty easily reduce the amount of time tests take by taking advantage of travis and github actions together. Travis offers something like 10 concurrent jobs (which is about what we are using), while actions offers 20 (only 5 for MacOS, but that should be more than enough). If we split the test suite to run in different jobs we could probably cut the time from ~30 minutes down to ~10.

With Github actions we can set up a cron task to run the expensive tests daily.

Also we could move the mypyc tests to github actions in the mypyc repo (and add Windows tests via actions!).

[JukkaL]

I think we can pretty easily reduce the amount of time tests take by taking advantage of travis and github actions together

A good idea!

If we split the test suite to run in different jobs we could probably cut the time from ~30 minutes down to ~10.

The mypyc compiled tests take some time to compile mypy before running tests, so increasing parallelism will give less than linear gains there. However, it should be sufficient to run the mypyc compiled tests in only two configurations perhaps (e.g. the oldest supported Python and the most recent). We can run mypyc tests that don't involve compiling mypy in more configurations.

With Github actions we can set up a cron task to run the expensive tests daily.

Yeah, this could be a good compromise between finding issues and speeding up CI.

Also we could move the mypyc tests to github actions in the mypyc repo (and add Windows tests via actions!).

I wonder if we could still trigger these for all mypy commits, and post the results to mypy PRs? Otherwise there would be a fairly significant usability hit.

[JukkaL]

I've been thinking about this some and came up with additional ideas. Here are more concrete ideas about speeding PR builds in particular. On master we could still run a more complete battery of tests. Some additional tests could be run daily.

Use fewer configurations

We currently run tests in 7 configurations. Just having 3 configurations might be good enough to catch almost all issues. For example:

1. Oldest supported Python version on Linux, compiled
2. Newest supported Python version on macOS, interpreted
3. Newest supported Python version on 64-bit Windows, compiled

We could split each of these into 2-3 separate jobs for more parallelism.

I think that this mostly covers all the biggest factors: Python version continuum (we use the oldest and the latest version), different platforms (all three major platforms) and interpreted/compiled.

Run expensive tests in only one configuration

Some tests are expensive and don't always give a lot of signal, so only run them in one configuration on the newest supported Python (e.g. macOS interpreted). Examples of such expensive tests:

• Command-line tests
• Mypyc tests
• Mypy daemon tests

These could be run in a separate job (or jobs).

Mypy self check will catch most uses of features not supported on older Python version, so only using the latest version should be mostly fine (except for mypyc, but see below for more ideas).

Trigger some tests only for certain changes

These tests are pretty slow and can probably be skipped for most PRs:

• mypy/test/teststubtest.py
• mypy/test/testpep561.py
• mypy/test/teststubgen.py

We could run these only if a file matching some patterns is modified. For example, we'd run stubtest tests if mypy/stubtest.py is modified but not otherwise (for PRs).

Additional rules

These rules could be used to optimize things some more:

• If a PR only touches documentation, we can skip most tests.
• For a typeshed change we may want to run additional tests (on various platforms).
• If a change directly touches the mypy daemon implementation, run the relevant tests on all platforms.
• Run mypyc tests in more configurations if something under mypyc/ is changed. Also run mypyc tests on macOS.

Iterate

We can initially reduce the number of tests pretty drastically. Once we have failures that we fail to catch before a PR is merged, we can refine the rules and run some additional tests.

[JukkaL]

We can perhaps do even better. We don't run the full suite on macOS right now, so we can skip the macOS platform. By default, we'd run tests in two configurations: on Linux (compiled) and Windows (interpreted).

[hauntsaninja]

We could try something like https://github.com/mark-adams/pytest-test-groups or https://github.com/AdamGleave/pytest-shard to easily make use of more concurrent jobs

I think the first two suggestions in #8783 (comment) should be easy and don't have much risk.

[hauntsaninja]

Recent changes: #10922 cut 7m off CI (as well as most non-incremental runs!), moving to Github Actions also sped us up. We've acquired some more cmdline tests, could probably combine some of them.