ci: parellelize tests in CI

[bolinocroustibat]

• Add install step cache fallback

• Parallelize the execution of tests among 4 CircleCI executors. This results in a gain of an average approx. ~40% on the total CI execution time (around 60% specifically on the tests job)

  • Note 1: For this, we need to deactivate the maximum coverage rule for test success, as each executor reach a different coverage. The gain makes it worth to deactivate this feature, as anyway we can monitor the coverage, and we can activate the minimum coverage threshold when running locally if necessary.
  • Note 2: it has also been tested with 2 executors, 3 executors and 10 executors, but 4 executors seems like a good balance for time gains

Before:

After:

[magopian]

Nice speedup, well done!

I guess the appropriate number of threads depends on the number of CPUs/cores that are available in a circleCI run?

Would using pytest-xdist make sense to "automate" the parallelization of tests instead of dealing with the executors "manually"?

[bolinocroustibat]

I guess the appropriate number of threads depends on the number of CPUs/cores that are available in a circleCI run?

No, I think each CircleCI executor is a separate execution environment, either a Docker container or a virtual machine. (https://circleci.com/docs/executor-intro/)
I have trouble to find documentation about the maximum number of executors we can use in free tier, but it seems to be at least 10 since I've been trying with 10 with no issue.

Would using pytest-xdist make sense to "automate" the parallelization of tests instead of dealing with the executors "manually"?

Yes, that would also good so that it would run in parallel also locally, but that would mean some tests and code refactoring. This way is easier for now. Also I wonder if CircleCI executors have several CPUs/cores?