Avoid substituting top-level attributes if that's not needed (CI mode)

[domenkozar]

When only checking if everything builds, substituting derivations for top-level attributes is a waste of time.

The problem

When nix-build is ran in a clean container it will substitute all derivations even if there's nothing to build to confirm everything builds.

[edolstra]

Hm, I don't understand what you mean. What does CI mode mean? And downloading top-level attributes?

[domenkozar]

@edolstra I've updated the description to use Nix terminology.

[zimbatm]

In the context of GitHub Actions for example, the CI runner is always starting with an empty local cache.

In most cases, all we want is build things that are not in the cache, and then upload the successful ones. And skip all of them that are already built.

With the current implementation a "noop" nix-build is dominated by downloads from the cache that then get immediately discarded after the run.

[edolstra]

Ideally this would be addressed by a FUSE-based store that only downloads store paths when they are actually accessed (something like 5219b5b). This would not only prevent fetching top-level paths but also dependencies that are never accessed.

[zimbatm]

Would the FUSE-based store be able to distinguish what store paths are not in the cache and only realize those entries?

[edolstra]

That would be the idea, but the FUSE FS in 5219b5b doesn't have any support for building yet. Maybe something for the Rust rewrite of libstore...

[zimbatm]

Wouldn't that make the builds run in the background if they are triggered by file access?

The FUSE FS seems interesting to have for other use-cases, like on big monorepos where there is a large shell.nix and the user only uses a subset of it during development.

In this case, I think it would be more straight-forward to filter out the top-level derivations by querying the cache and then just realize those that are left-over. What do you think?

[Mic92]

I have implemented a FUSE and looked into its performance caveats when writing cntr also see the paper for in-depth performance analysis. The overheads are significant if the whole nix store would be served through a nix. Data might be cached twice if twice in the page cache, metadata lookups require a lot of context switching. My recommendation would be to use something like autofs to delay downloading of dependencies and replace them with bind mounts instead when the path is actually accessed. However I don't think that this solution would solve unused dependencies in a build in practice. We have support hooks in nixpkgs that would trigger stat() systemcalls when iterating over build inputs and therefore would mount them all.

[Mic92]

Another practical example for this feature: cachix/cachix-action#40 (comment)

In a nutshell this makes it easier to embed nix in a CI that is not nix-aware.

[domenkozar]

There's now https://github.com/Mic92/nix-build-uncached - although having an implementation in Nix itself would really be ideal.

[stale]

I marked this as stale due to inactivity. → More info

[Mic92]

still relevant.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/github-actions-and-or-with-hercules-ci/13537/2

[stale]

I marked this as stale due to inactivity. → More info

[endgame]

Still desirable.

[arianvp]

I was able to get this working in CI:

store="${{ vars.CACHE_BUCKET }}&secret-key=$(realpath ./cache-secret-key)"
# check if the build is already in the cache and else build and copy it
if ! nix path-info --eval-store auto --store "$store" .#nixosConfigurations.web-push.config.system.build.toplevel; then
  nix copy --eval-store auto --to "$store" .#nixosConfigurations.web-push.config.system.build.toplevel
fi