Better primer tests (faster, less duplication, more diverse)

[Pierre-Sassoulas]

Current problem

Right now the primer tests take more than double the time of the next slower tests. Also the CI configuration is duplicated between primer for stdlib and primer for external tests. The choice of repository to check is also arbitrary.

Desired solution

• Update primer tests to own file, separate batches and concurrent runs #5425 > Burst the primer-external CI job into multiple jobs to cut down on CI run time
• Update primer tests to own file, separate batches and concurrent runs #5425 > Remove the duplication in the GitHub action configuration and move primer to its own workflow file
• Clarify the criteria for choosing a repository and update the documentation
• Remove repositories that are redundant or less useful considering what we already have
• Fixing dependency to a specific commit to be able to cache repository
• Caching of repository for primers
• Remove duplication in primer jobs by using a job template
• Update primer tests to own file, separate batches and concurrent runs #5425 > Only run when something changes in pylint internals or primer testutils
• Improve error message on crashes and fatal errors within the primer logs
• Check and see if CI workflow file can be shortened/improved

Additional context

#5173

[DanielNoord]

@Pierre-Sassoulas Just something I thought of:
I'm not sure if we get unlimited amount of CI minutes from Github, but if we need to be conservative with them we could consider not running the external primer on all three versions.

[Pierre-Sassoulas]

We definitely need to cache everything we can too. (Setting the hash of the repository we lint would help).

[cdce8p]

I'm not sure if we get unlimited amount of CI minutes from Github

@DanielNoord AFAIK public repos are only limited in the number of concurrent job runners, not the actual time.

• https://docs.github.com/en/billing/managing-billing-for-github-actions/about-billing-for-github-actions#about-billing-for-github-actions
• https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration#usage-limits

[cdce8p]

One suggestion for something I saw recently. If the primer tests are moved to a new workflow, you could also add concurrency control. I.e. it doesn't make sense to continue with primer tests if a new commit has been pushed in the meantime.
https://docs.github.com/en/actions/learn-github-actions/workflow-syntax-for-github-actions#concurrency

This is how Home Assistant does it: Home-Assistant -> ci.yaml, although it probably isn't necessary to cancel all other jobs as they are quite fast and even partial feedback could be useful.

[Pierre-Sassoulas]

Huge first step was done in #5425 by @DanielNoord we still can do:

• Fixing dependency to a specific commit to be able to cache repository
• Caching of repository for primers
• Remove duplication in primer jobs by using a job template