Add support for OpenDAL-backed remote stores (file://, initially)

[huonw]

This is the final major step that unlocks #11149:

• adding new byte store and action cache providers, implemented using https://opendal.apache.org
• plumbing through to be able to be use them via remote_store_address = "..."
• (and implementing an "example" one, that's backed by the local file system)

This doesn't yet resolve that issue, as it doesn't implement any sort of useful "simple" truly-remote cache, but it sets up most of the infrastructure.

User facing change

This only adds support for one additional store for now: experimental:file:///path/to/dir, which writes the store to /path/to/dir.

This is store is enough to validate that this functionality (mostly) works, and even seems like it would be useful for:

• debugging/testing remote cache behaviour (e.g. can set up reduced reproducers for remote caching bugs using normal file manipulation commands)
• potentially for real, for using an NFS mount as a remote cache, without forcing all of Pants' caches to have the NFS overhead.

The user facing functionality here is configured entirely through remote_store_address: that option now supports URIs that have schemes other than grpc or grpcs. To start with, just file. It also explicitly supports these schemes being experimental, communicated through the experimental: scheme-prefix. This is to, hopefully, allow landing the feature earlier and with less risk.

For example, running pants test :: in the follow example writes various blobs to subdirectories of /tmp/remote-cache-example (e.g. action-cache/e3/2f/e32f034f... and byte-store/50/ce/50ce4a68...), and a second pants --no-pantsd --no-local-cache test :: command is able to service it from cache, reporting //:slow-test succeeded in 5.01s (cached remotely)

# pants.toml
[GLOBAL]
pants_version = "2.17.0"

backend_packages = ["pants.backend.shell"]

remote_store_address = "experimental:file:///tmp/remote-cache-example"
remote_cache_read = true
remote_cache_write = true

# BUILD
experimental_test_shell_command(name="slow-test", command="sleep 5", tools=["sleep"])

Details of this change

There's three main bits of work:

1. add the new generic OpenDAL byte store and action cache provider (in the two pairs of base_opendal.rs + tests files)
2. hook up the specific FS one (in store/src/remote.rs and remote/src/remote_cache.rs)
3. adjust the options parsing to support them, especially the experimental: scheme handling (in global_options.py)

None of these are huge in isolation, but they add up to a fair chunk of code. I think each of those files can be reviewed somewhat in isolation, in that order.

Why OpenDAL?

It's used by sccache for similar remote-caching purposes (https://github.com/search?q=repo%3Amozilla%2Fsccache%20opendal&type=code), and supports a whole lot of services:

• blob stores (e.g. S3, GCS)
• key-value caches (e.g. Redis)
• CI caches (e.g. GitHub Actions)
• weirder ones (IPFS, Google Drive)

Pants doesn't/shouldn't support all of them, but definitely blob stores and CI caches seem exciting!

Next steps

This only adds support for caching "remotely" to the local file system but I'll add support for GitHub Actions as follow up work (#19831), and maybe S3 too. Hopefully as part of doing that I'll work through any authentication (etc.) issues and so it becomes easier for others to add the backends they need too.

I've suggested we use the URI scheme for deciding on the service, rather than an explicit remote_store_service = "reapi" option, or similar. In future, some services may use the same scheme, e.g. I imagine several services might conventionally use http:// or https://. My current thinking is to solve this similar to pip's VCS requirements (vcs+transport://...) or sqlalchemy database URLs (dialect+driver://...), e.g. remote_store_address = "s3+http://..." or remote_store_address = "http+s3://...", and then the provider would strip off the s3 part as appropriate.

There's also a whole lot of TODOs that I don't think are critical to landing this code, and I can chip away at once it's in: by landing this sooner, I'm hoping we can start having the user-facing parts of #11149 being tested, validated and enjoyed sooner rather than later. I've already done a loooot of pre-factoring (see all the PRs referencing #11149), so it'd be nice to switch to having the feature in and just do some normal re-factoring. 😅

TODOs mostly tracked in #19902

[huonw]

I've refreshed the GHA cache backend #17840 PR to #19831, which seems to indicate that this is going in the right direction: that PR is very small!

[stuhood]

Thanks @huonw ! Sorry that I didn't get to this this week: will review by EOD Monday.

[huonw]

All good, no rush. It's a big one!

[stuhood]

Thanks a lot!

[stuhood]

So, on the contrary: I think that it would be good to separate the grpc based provider into a separate crate from the opendal provider, so that they can be compiled independently / in-parallel.

The process_execution crate had previously gotten unwieldy due to having grpc, docker, and nailgun clients all living in it, and I think that the store crate is likely to end up in the same position with another client implementation added to it (particular one as potentially-hefty as opendal).

[huonw]

Yeah, potentially a crate for with the traits and then separate providers is better.

This source comment is a bit ambiguous, sorry, but it's more referencing that it'd be nice to have the store and action cache providers for each service together in one crate, not necessarily that the gRPC/REAPI and OpenDAL ones should be grouped too.

[huonw]

Getting into a bit of a distracting discussion, but separate crates for each provider might end up with 4 new crates, which is probably fine/manageable?

graph BT
   store("fs/store (existing)")
   remote("process_execution/remote (existing)")
   traits("remote_providers/traits (new)")
   grpc("remote_providers/reapi (new)")
   opendal("remote_providers/opendal (new)")
   selector("remote_providers (new)")
   grpc --> traits
   opendal --> traits
   selector --> grpc
   selector --> opendal
   store --> selector
   remote --> selector

Loading

Where the "top-level" remote_providers centralises things like the functions for choosing a provider based on options, and, potentially more of the validation code that's currently in global_options.py.

[stuhood]

That layout looks good, yea.

[stuhood]

There is a lot of potential cleanup that can happen here, yea.

Ideally we could expose a struct all the way through to Python (with a very light Python wrapper around the inner type), such that it could directly be constructed from global options. Currently that's not the case -- in particular: see ExecutionOptions, which is copied into positional args to the scheduler.

[huonw]

I've filed #19902 tracking the refactoring for me.