Introduce Turbo cache in CI

[danielbate]

In #1649 we introduced a Turbo cache in CI, however it was prone to flakiness. As the docs build is dependent on an autogenerated file from providers, if providers hits the cache but docs misses, the job will fail.

Adding a build cache would improve the time of the second job in our test CI, so we should look to introduce it. However something that is more robust than what was added in #1649, #1694 and #1721.

[arboleya]

FWIW, it seems that the error happens when re-running a failed workflow.

We should stress test this case if we re-add the work reverted by #1752.

---

The failing workflow is something like this:

Job 1

perform env specific tests in parallel

1. Looks for a cache entry using the commit SHA - does not find one
2. Performs install / build
3. providers.prebuild generates the operations file
4. docs.build generates docs, referencing the providers.operations file
5. Performs an environment specific test and uploads results

Job 2

1. Looks for a cache entry using the SHA - finds one and fills dist directories, as well as any autogenerated files (providers.operations, docs/api).
2. Performs install and build - hits cache on build
3. providers.prebuild and docs.build not executed
4. Downloads test results, merges and uploads.

---

So if we have inconsistencies between cache hits/misses with the GH cache, and the turbo package caches - it will fail.

Most commonly it seems to be providers hitting the cache but docs missing it.

[danielbate]

FWIW, it seems that the error happens when re-running a failed workflow.

Ah okay this is good information. A similar implementation to work reverted in #1752 may suffice then, as long as it accounts for the failed stage. Unsure why at the moment though, I'd expect the cache entries to still be present and it would still run using the same SHA.

[danielbate]

Closing this for now. This has been investigated by the above PRs and someone is welcome to have another go however the real value for our CI and better use of time will come from #2493 and #1755.