Proof of concept: support remote caching to Github Actions cache

[huonw]

This PR does a 'spike' to validate using a remote blob store as a (fine-grained) remote cache. It isn't at all ready for landing or detailed review (lots of debugging code, poorly factored, hard-coded config, plus #[allow(warnings)] in a bunch of places), but I think it validates one key thing: it's not too hard to do non-REAPI remote caching.

Specifically, this PR adds support for caching to/from Github Actions' built-in cache, ala https://github.com/actions/cache but fine-grained and integrated into pants. This is inspired by the support for using GHA as a fine-grained cache in both docker buildx and, recently, sccache.

This work relates to #11149, as it isn't specific to Github Actions, beyond the details of actually pushing bytes to it. That is, this PR also proves out the possibility of using other blob stores such as other CI caches or AWS's S3 or GCP's GCS. The core moving parts are two traits, which can be slotted into the lowest level of remote cache processing, and then the rest of the system seems to work with minimal changes:

// in `fs/store/`:
#[async_trait]
pub trait ByteStoreProvider: Sync + Send + 'static {
  async fn store_bytes(&self, digest: Digest, bytes: ByteSource) -> Result<(), String>;
  async fn load_bytes(&self, digest: Digest) -> Result<Option<Bytes>, String>;

  /// List any missing digests.
  ///
  /// None = not supported.
  async fn list_missing_digests(&self, _digests: &mut (dyn Iterator<Item = Digest> + Send)) -> Result<Option<HashSet<Digest>>, String> {
    Ok(None)
  }

  fn chunk_size_bytes(&self) -> usize;
}

// in `process_execution/`:
#[async_trait]
pub trait RemoteCacheProvider: Sync + Send + 'static {
  async fn update_action_result(&self, action_digest: Digest, action_result: ActionResult) -> Result<(), String>;
  async fn get_action_result(&self, action_digest: Digest, context: &Context) -> Result<Option<ActionResult>, String>;
}

I'm imagining, once this style of infrastructure is here, it's then easy enough to add support for other options when people want them, covering the options supported by the various build systems listed in #11149 (comment). (Assuming the table in that PR is still up-to-date, pants would have the easiest/cheapest remote caching via CI-focused caching like this proof-of-concept.)

Example

I've run this as an example in https://github.com/huonw/pants-cache-github-actions-poc, with https://github.com/huonw/pants-cache-github-actions-poc/actions/runs/3717717056/jobs/6305350011 being a specific example (logs saved for reference). I set up 30 Python test files, that each take 10s. When cached, the job runs much faster, and reports:

✓ //test_01.py:tests succeeded in 10.32s (cached remotely).
✓ //test_02.py:tests succeeded in 10.34s (cached remotely).
...
✓ //test_30.py:tests succeeded in 10.34s (cached remotely).

(That step seems to take a while (33s), but is running with MODE=debug for faster pants_from_sources Rust compilation. In release mode, it's more like 10-15s.)

Getting this ready to land

I've thought through a loose plan for next steps, assuming there's high-level agreement about landing this at all:

1. preparatory refactoring to make this easier to land, particularly a little bit of teasing out of the REAPI types/calls, and adjusting the load_bytes_with function
2. working out how to do the configuration required for more general caching
3. actually landing the GHA support, with additional controls like concurrency limits (this final support would likely look quite different to the code in this PR)

Questions

1. Before I spend more effort on this, is work towards this likely to be accepted?
2. Any thoughts on how to configure this?
   1. I've been assuming a remote_backend = "..." global option with values like reapi and gha. Maybe the remoting config should move to its own subsystem somehow, but maybe that doesn't have to block this work?
   2. Maybe the URL/token provided by multiple environment variables: for the URL, allow reusing remote_store_address/PANTS_REMOTE_STORE_ADDRESS but also support the GHA name ACTIONS_CACHE_URL? For the token, somehow generalise remote_oauth_bearer_token_path.
   3. What about erroring out for irrelevant options?

Notes

• This sort of "local" remote caching in CI may be able to augment truly remote caching, by providing a 'closer' cache that saves on network bandwidth. For instance, when reading a blob, first check the CI cache, before going out to the REAPI server (and when writing, write to both).
• This would make https://www.pantsbuild.org/docs/using-pants-in-ci cleaner and more compelling.
• The split to be generic over different "providers" seems like it would make testing the remote caching code easier, most tests can use an in-memory cache that implements the right traits.
• This doesn't put any effort towards making the remote cache backend pluggable, meaning additional backends will need to be upstream in pants' Rust code. Pluggability seems like a much bigger effort.
• Specifics of caching to GHA:
  • It's... kinda slow, but that's fine (since there's local speculation too). For example, the build/logs linked above have histograms on, and show remote_cache_get_action_result_time_micros ranging from 74ms - 1.1s (p50: 610ms), and remote_store_read_blob_time_micros ranging from 33ms - 561ms (p50: 249ms).
  • It's scoped per-branch, with 'feature' branches being able to read from caches from the default branch, but not write, meaning a PR workflow would still have to build new targets at least twice: once in the PR branch (available for reuse for later builds of that PR), and then once in the default branch after merge (available for reuse by all builds).
  • I'm not sure GHA can effectively handle cache_content_behavior = "validate", as, AFAIK, there's no bulk query available. It'd have to be a bunch of individual queries, but maybe that's okay?
  • This sort of caching only works for CI builds, and doesn't provide any benefit for local dev.
  • It seems like the environment variables required to find the cache are only provided to actions, rather than shell run steps, which forces the use of github-script to reexport them: https://github.com/huonw/pants-cache-github-actions-poc/blob/2e7ccfbfe/.github/workflows/ci.yml#L41-L46. This would be able to be packaged up into a init-pants, and/or automatically provided by if there's "invoke pants" action too.

[chris-stetter]

Would love this feature 🤟

[huonw]

This has been broken up into a whole bunch of pre-work PRs (see all the mentions above), which have culminated in the rather simpler #19831. Closing favour of that.