Remote cache for pip

[kvet]

Hi, I am trying to setup bazel remote cache. It works out of the box for build and test rules. But I didn't manage to get it working with pip. And because packages installation is quite slow, I don't see noticeable benefits of remote cache.

Here is my setup:

# WORKSPACE
pip_parse(
    name = "pypi",
    python_interpreter_target = interpreter310,
    requirements_lock = "//reqs/python:lockfile.default.txt",
)
load("@pypi//:requirements.bzl", "install_deps")
install_deps(
    python_interpreter_target = interpreter310,
)

# BUILD
load("@pypi//:requirements.bzl", "requirement")
py_library(
    ...
    deps = [requirement("protobuf")]
)

I am running local bazel-remote:

bazel-remote --dir=$HOME/.bazel-remote --max_size 1000 --experimental_remote_asset_api

And here is content of .bazelrc:

build --remote_cache=grpc://localhost:9092
build --experimental_remote_downloader=grpc://localhost:9092
build --noexperimental_remote_downloader_local_fallback
build --remote_upload_local_results=true
build --remote_download_minimal
build --experimental_guard_against_concurrent_changes
build --experimental_remote_cache_async
build --experimental_repository_cache_hardlinks
common --noenable_bzlmod

Am I doing something wrong? Or doesn't it work at all? Haven't found any information on web...

[arrdem]

Hey @kvet, no you aren't doing anything wrong.

Bazel has two "classes" of rules as you may be aware -- normal build rules which go from determinate input files and configuration to determinate output files, and "repository rules" which are allowed to go from configuration ex nihilo to indeterminate files by performing arbitrary code execution. This is the basis for how all the Bazel rulesets implement 3rdparty dependencies.

Entries in the remote cache are identified by fingerprints reflecting the fingerprints of all input files transitively among other things. Since workspace rules -- that say run pip install -- could lay down arbitrary files, Bazel can't construct identifiers for build products which depend on those external workspaces without doing the leg work to materialize the external workspaces first. Which as you note can be quite slow.

Turning on the downloader also doesn't (yet) help because rules_python today lets pip do the downloading. I believe there's some experimental work aimed at trying to change that. Even with the downloader on I don't believe that Bazel is able to skip materializing the external workspaces, although maybe it could some day.

Operationally this means it's critical to use stateful CI runners rather than stateless ones so that the results of the pip install can get reused across multiple builds. In general Bazel should be understood as a build database that wants to be operated like a database rather than as a build tool which you spin up and shut down. Ensuring that your requirements are fully resolved to prebuilt wheels (--only-binary to pip) so you can pip download rather than pip install can also help because it makes materializing your external dependencies faster.

But in general I think it's safe to say Python under Bazel gets less mileage out of remote caching than say C or Go compilation since for the most part rules_python binaries are just copying around (or creating symlinks to) unmodified source files defined by repository rules which happen before the remote cache can be brought to bear.

[kvet]

Thank you so much for the detailed reply!

I am still trying to comprehend how to properly organize caching with GitHub Actions and bazel.

Right now I am just caching the content of ~/.cache/bazel with actions/cache. It works somehow. Cache sizes are enormous (like hundreds of MBs) but it gets the job done unless it breaks. I can't say for sure what is happening, but the cache gets "spoiled" after a some period of time and making more harm than good. I am rotating caches every week to prevent these breakages . It works quite well considering the amount of effort I put in. But it have a tendency to break even more often. And it requires a manual cache rotation from me or resetting cache more often which eliminates the profit of cache.

Can you please guide me how to cache pip packages in a proper way?