glibc: NVCC aarch64 intrinsics are unsupported

[ConnorBaker]

Context

The GLIBC 2.38 update introduces intrinsics for aarch64-linux in math.h. NVCC (NVIDIA's CUDA Compiler) is now unable to compile any CUDA file for aarch64-linux because it does not support these intrinsics: https://forums.developer.nvidia.com/t/nvcc-fails-to-build-with-arm-neon-instructions-cpp-vs-cu/248355/2. As a result, PyTorch, Magma, Jax, and any other CUDA-enabled package now fail to build on aarch64-linux.

An alternative approach might be to patch the GLIBC used by NVCC. This approach is not taken for several reasons:

• Users with config.cudaSupport = true would need to bootstrap a second copy of Nixpkgs with a glibc containing these changes.
• Manually patching NVCC, CMake, and the include paths of a package like Magma was not enough: CMake's compiler identification routine fails to build its trivial CUDA file because it picks the unpatched GLIBC.
• Assuming a manual patch against NVCC and or the build systems does work, every CUDA-enabled package must incorporate the relevant changes to ensure the unpatched GLIBC does not make its way in.
• This patch follows the precedent set by upstream by guarding against inclusion of these intrinsics when an unsupported compiler is detected. We ensure that only NVCC is affected by this patch.

Description of changes

GLIBC 2.37 -> 2.38 broke CUDA compilation on ARM because it introduced ARM intrinsics in math.h. Here are the ones which are causing the errors: https://github.com/bminor/glibc/blob/36f2487f13e3540be9ee0fb51876b1da72176d3f/sysdeps/aarch64/fpu/bits/math-vector.h#L28-L36.

NVCC declares itself to be the same compiler as its host compiler. This causes inclusion of unsupported aarch64-linux intrinsics.

This patch follows the precedent set by upstream by guarding against inclusion of such intrinsics when an unsupported compiler (like NVCC) is detected.

Note: upstream does not intend to fix this. Instead, they place the onus (perhaps rightly) on downstream compilers/pre-processors (like NVCC), with the reasoning that they should not claim to implement a version of a compiler that they do not.

The patch is applied only for aarch64-linux.

References

• GLIBC bug report about this issue: https://sourceware.org/bugzilla/show_bug.cgi?id=30909
• GLIBC patch: https://patchwork.sourceware.org/project/glibc/patch/20230927141839.57421-1-simon.chopin@canonical.com/
  • Note that this patch was dropped and will not move forward
  • Upstream believes compilers should not pretend to implement recent GCC versions if they do not

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 23.11 Release Notes (or backporting 23.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

[SomeoneSerge]

LGTM from the cudaPackages perspective, and I understand you've already verified the patch is sufficient to build cudaPackages.saxpy

Waiting for someone familiar with nixpkgs' glibc to sign off

[jonringer]

@ofborg build cudaPackages.saxpy

[Ma27]

it is important that this patch is applied within Nixpkgs, rather than waiting for upstream to accept and apply such a fix.

This is partially correct only: for each release there's an active release branch where patches from their main branch are backported to, we mainly need to update the patchset here.

to accept and apply such a fix.

Please file it upstream anyways. I'd very much prefer to only update our patchlevel rather than maintaining our own patches that nobody else uses.

This patch follows the precedent set by upstream by guarding against inclusion of these intrinsics when an unsupported compiler is detected. We ensure that only NVCC is affected by this patch.

Agreed 👍
The approach seems OK to me.

Is there any particular reason though that this is a draft?

[jonringer]

@ofborg build python3Minimal

[jonringer]

Seems to be a noop in non-aarch64, non-linux, non-cuda builds.

I'll defer to @Ma27 for final merge, but I'm fine with this.

[ConnorBaker]

Is there any particular reason though that this is a draft?

I had some trouble rebasing this change from master to staging and wanted to avoid mass-pinging a bunch of people until I got it right. Moving it out of draft now!

[Ma27]

The change itself looks good and I agree with the approach. However, I'd like to see this patch being filed against upstream glibc before this gets merged here.

[ConnorBaker]

@Ma27 this is a blocker for a number of tasks I'm working on -- is it okay if we do them in parallel?

Here's the list of tasks to get them to consider the patch:

• Email the source ware administrators to have them create an account on Bugzilla for me.
• Set up account and post bug report of observed breakage with NVCC.
  • Added a comment to https://sourceware.org/bugzilla/show_bug.cgi?id=30909#c2
• Email the patch which references the bug report.
  • Upstream doesn't intend to fix this.

EDIT: Updated with status of tasks and note that upstream doesn't intend to fix this.

[ConnorBaker]

@Ma27 @SomeoneSerge @jonringer after some more research:

• GLIBC bug report about this issue: https://sourceware.org/bugzilla/show_bug.cgi?id=30909
• GLIBC patch: https://patchwork.sourceware.org/project/glibc/patch/20230927141839.57421-1-simon.chopin@canonical.com/

it's clear that upstream believes compilers should not pretend to implement recent GCC versions if they do not, and does not intend to fix this issue. That's principled and I applaud them for it; however, the fact remains that there's no easy way to patch NVCC because it is a closed source project and relied on heavily in the High Performance Computing (HPC) space.

With your blessing, I'd like to merge this to unblock a number of different issues and restore functionality to the CUDA packages Nixpkgs provides.

I don't share the same concern as the GLIBC maintainers do about bitrot or forgotten fixes because we maintain our patches alongside the project, rather than embedding them in the source code. If and when support is added and new versions of NVCC are released for all prior versions of CUDA with a fix, we could revisit this patch.

Is this agreeable?

Upstream will not issue such a patch and will leave all released versions of NVCC broken

[Ma27]

Just read the ML thraed about the other patch suggested and I agree with your assesment.. Because of that I'm inclined to give an OK to this. There's one thing we should talk about before however given that this patch will probably live a little longer in nixpkgs: are you open to reviewing changes that possibly affect this patch? This includes at least glibc version upgrades and changes to the patch itself.

AFAIU I won't be able to do that without nvidia hardware, so I can't do that, correct? If so, please add a note about you maintaining the patch above the inclusion in common.nix.

[SomeoneSerge]

Is this agreeable?

I think this is right. We still probably have to patch some downstream apps (torch, opencv?) on master, because otherwise your modules+jetson support PR is blocked for a while

are you open to reviewing changes that possibly affect this patch

I'm up for reviewing such changes as well

AFAIU I won't be able to do that without nvidia hardware, so I can't do that, correct

I'll just note that one doesn't need nvidia hardware to run the builds

[ConnorBaker]

... are you open to reviewing changes that possibly affect this patch? This includes at least glibc version upgrades and changes to the patch itself.

I am!

AFAIU I won't be able to do that without nvidia hardware, so I can't do that, correct?

Nah, it's a build failure and the build only requires the CPU. You'd run into this so long as config.cudaSupport = true and you're building for (or on) aarch64-linux.

If so, please add a note about you maintaining the patch above the inclusion in common.nix.

Will do!

[ConnorBaker]

Added myself as a maintainer, merging. Thank you for your help and guidance, @Ma27!