Use docker cache mounts for apt, pip and cargo

[mjpieters]

The cache mounts are cached using standard github actions cache when building in the CI pipeline.

Note that the build stage no longer contains the whole source tree, these are instead mounted into the build container when building to avoid invalidating cached build container layers.

[zanieb]

Is this just for Docker image build performance?

[zanieb]

See also, previous discussion at #3372

[mjpieters]

Is this just for Docker image build performance?

Yes, both locally and CI workflows. I had completely missed the other PR 🤦 so thanks for the reference! I'll review the discussion and see if this needs closing or updating or if I change the merge target to that PR.

[zanieb]

I think that one might have been a little more aggressive about refactoring. I'm more liable to just accept cache mounts. The big caveat there is probably captured by #3372 (comment) with the notable update that we're now using Depot runners in our org! So... we could set up Depot for Docker builds.

[mjpieters]

Yeah, the author of that pr clearly has a lot of rust-in-docker build knowledge, more than I have.

Bottom line is that this PR makes an incremental change with the aim of improving build time here in the GH pipeline. If you are moving the build to a different platform however then there is not much value in merging this one.

[samypr100]

Note, I would see #3372 for context as to why some of these changes haven't been done.

(edit: woops, hadn't seen this was already mentioned #11106 (comment))

[mjpieters]

I've now refactored the cache mounts to be simpler and (hopefully) more effective in the Github pipeline

All in-docker build tool caches now live in a single /buildkit-cache directory. This includes the global caches for cargo, rustup, pip, zig and cargo-zigbuild. This is keyed just on the Dockerfile hash, because these can trivially be re-used even if the Cargo.lock file changes.

The /root/target cache mount is no longer cached or restored in the GitHub pipeline, but it is still hugely helpful when building the docker container locally as it massively speeds up incremental builds in that case. Propagating the cache to the Github action cache was not effective however because when you have the exact same Cargo.lock, Cargo.toml and crates file tree then Github has also cached that specific Docker layer already, meaning that the whole build step is cached.

[mjpieters]

I forced a push and this does make the build faster, by about 90-120 seconds. Compare the times for the latest release:

• release:

  • amd64 build: 12m 38s
  • arm64 build: 13m 57s

• this PR:

  • amd64 build: 11m 4s (Δ 1m 34s faster)
  • arm64 build: 11m 50s (Δ 2m 7s faster)

I've since pushed another small update to add a rustup self update to use the cached version of rustup when available, instead of re-downloading and having to silence rustup from complaining about the cached cargo and rust toolchain.