cudaPackages.setupCudaHook: propagate deps and the hook

[SomeoneSerge]

Description of changes

This is an attempt to fix the build failures in the cuda variants of torch/opencv4 cmake consumers. Basically, this lets one avoid ad hoc changes like this: https://github.com/NixOS/nixpkgs/pull/269639/files#diff-dcb82f7bc26e70a69ecb21e6801c8f2c32dbd91847e2b2db9b4fbb986f412abc.

The approach is to propagate the build dependencies (which we used to do), but to do so in a separate output, so as to avoid the cuda fat leaking into python withPackages environments

In addition to propagating libraries like cuda_cudart we also have to propagate the setupCudaHook because somebody's got to set -DCUDAToolkit_ROOT &c

I tested this on cctag which had failed at find_package(opencv) in #269639

I also realized we need to remove the hook's dependency on cudart and especially on the compiler: when we consume a cross-compiled opencv4 in a native build, the propagated compiler is going to be useless, because we couldn't have known about the native build when compiling opencv4...

I haven't yet figured out what happens if a package has both buildInputs = [ opencv4 ] and nativeBuildInputs = [ cuda_nvcc ]. Both are going to propagate the hook, so the hook might run twice, and then it might add the env hooks, preConfigureHooks, and postFixupHooks twice. I need some sort of #pragma once

@NixOS/cuda-maintainers

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

Priorities

Add a 👍 reaction to pull requests you find important.

[SomeoneSerge]

I maybe broke torch

[SomeoneSerge]

Here's evidence that propagatedHostTargetDeps (propagatedBuildInputs) work and HostHost doesn't: https://gist.github.com/SomeoneSerge/7d0e633743175bee6470b03281fe74e1

Same goes for BuildHost instead of BuildBuild when propagating the hook to nvcc's users. Whatever the reason, a transitive dependency is assigned the offsets of (prevHost + relHost, prevHost + relTarget) rather than (prevHost + relHost, prevTarget + relTarget):

nixpkgs/pkgs/stdenv/generic/setup.sh

Lines 552 to 560 in 82fa717

	function mapOffset() {
	local -r inputOffset="$1"
	local -n outputOffset="$2"
	if (( inputOffset <= 0 )); then
	outputOffset=$((inputOffset + hostOffset))
	else
	outputOffset=$((inputOffset - 1 + targetOffset))
	fi
	}

The relevant message in the cross-compilation matrix: https://matrix.to/#/%23cross-compiling%3Anixos.org/%248lsLV15zqjt5JdopUy-3aD3o8ym1NdITs18bZRZXMc8?via=someonex.net&via=matrix.org&via=catgirl.cloud&via=nixos.dev (yet to receive replies)

[SomeoneSerge]

I suspect that most of this PR wouldn't have been required with #102613

[SomeoneSerge]

The builds verified so far:

❯ nix build -f ./. --arg config '{ cudaSupport = true; cudaCapabilities = [ "8.6" ]; allowUnfree = true; }' -L python3Packages.{torch,torchaudio,torchvision,opencv4} cctag --print-out-paths  --keep-going
/nix/store/csa6zg49ih3zzfnh28ins5sy0xsa5iiz-python3.11-torch-2.1.1
/nix/store/4sdd2281hbz4cp0f24zinsyq6ma2n83i-python3.11-torchaudio-2.1.1
/nix/store/jr9z2jn008xl6vd17n1y5papl8m2spaj-python3.11-torchvision-0.16.1
/nix/store/v1p716yhdx99jldx7s4vmmrsrl0s6xpj-opencv-4.7.0
/nix/store/i4c5j077qqi25kz8y946ggj7axifz715-cctag-1.0.3

IIRC openvino is broken on master, the error is something like:

In file included from /build/source/src/inference/include/openvino/runtime/intel_gpu/ocl/ocl.hpp:18,
                 from /build/source/docs/snippets/gpu/context_sharing.cpp:2:
/build/source/src/inference/include/openvino/runtime/intel_gpu/ocl/ocl_wrapper.hpp:47:14: fatal error: CL/cl2.hpp: No such file or directory
   47 | #    include <CL/cl2.hpp>

[SomeoneSerge]

The closure for python3.withPackages (ps: with ps; [ torch torchaudio torchvision ]) is now 8G for the cudaCapabilities = [ "8.6" ] variant: https://gist.github.com/SomeoneSerge/4672d16693d03ae5808fa1e81b4544cc

One odd thing I observe is that nvcc is part of the closure, and that means that setupCudaHook is, and so is gcc11 because the hook links it directly...

Also it seems like the cxxdev trick may not have worked as intended because if I add torch.cxxdev to that list I still get 8G? The cxxdev hack did work because the cxxdev output isn't present in the gisted list

• NVCC is part of the closure because of openai-triton
• libcublas-static part of the closure because of opencv4, which is part of the closure because of ffmpeg, which is part of the closure because of torchaudio

[SomeoneSerge]

Back to ~6G:

❯ nix build --impure --expr 'with (import ./. { config = { cudaSupport = true; cudaCapabilities = [ "8.6" ]; allowUnfree = true; }; }); python3.withPackages (ps: with ps; [ torch torchvision torchaudio torch.cxxdev ])' --print-out-paths | xargs nix path-info --closure-size --human-readable
/nix/store/mf3cy3la3d8ywa0yjpgf4bf20g8hp7cd-python3-3.11.6-env     6.1G

[github-actions]

Backport failed for staging-23.11, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin staging-23.11
git worktree add -d .worktree/backport-271078-to-staging-23.11 origin/staging-23.11
cd .worktree/backport-271078-to-staging-23.11
git switch --create backport-271078-to-staging-23.11
git cherry-pick -x be9c779deba0e898802dd341a1ba9c04c4e9abe8 ada3991349beb5880e3994f25c65a0cf68941b83 45698380295187b35f3872542b71efc2223f8201 55af9329429a30ce81f7ad01da95406e1d62f785 71c248ec1309381136bf74339d453a58b400b2a9 3ececb9efafd80058525571d77d881767de6f5b8 44611c4a6d16b0eeb1488e9557b6a11e45193a46 2df7ccfa1498f5038b15acd50bc9277ad768dcbf