Precompiling `Reactant` errors in GPU-related symbol

[Todorbsc]

CC @mofeing

I got this error while trying to update the version of Reactant.jl to a branch:

pkg> add Reactant#ss/split-invoke-call-rewrite
    Updating git-repo `https://github.com/EnzymeAD/Reactant.jl.git`
   Resolving package versions...
    Updating `<project-path>/Project.toml`
  [3c362404] ~ Reactant v0.2.18 `https://github.com/EnzymeAD/Reactant.jl.git#main` ⇒ v0.2.18 `https://github.com/EnzymeAD/Reactant.jl.git#ss/split-invoke-call-rewrite`
    Updating `<project-path>/Manifest.toml`
  [3c362404] ~ Reactant v0.2.18 `https://github.com/EnzymeAD/Reactant.jl.git#main` ⇒ v0.2.18 `https://github.com/EnzymeAD/Reactant.jl.git#ss/split-invoke-call-rewrite`
Precompiling project...
  ✗ Reactant
  ✗ Reactant → ReactantStatisticsExt
  ✗ Reactant → ReactantPythonCallExt
  ✗ Reactant → ReactantSpecialFunctionsExt
  ✗ Reactant → ReactantAbstractFFTsExt
  ✗ Reactant → ReactantOffsetArraysExt
  ✗ Reactant → ReactantYaoBlocksExt
  ✗ Tenet → TenetReactantExt
  0 dependencies successfully precompiled in 32 seconds. 382 already precompiled.
  8 dependencies errored.
  For a report of the errors see `julia> err`. To retry use `pkg> precompile`

julia> err
PkgPrecompileError: The following 8 direct dependencies failed to precompile:

ReactantSpecialFunctionsExt 

Failed to precompile ReactantSpecialFunctionsExt [5981d66b-43ae-5ef9-a655-fb5bed72485b] to "~/.julia/compiled/v1.11/ReactantSpecialFunctionsExt/jl_29LleC".
ERROR: LoadError: InitError: could not load library "~/.julia/artifacts/bf5c832056d192639a75739fc1016591bc1c64c5/lib/libReactantExtra.so"
~/.julia/artifacts/bf5c832056d192639a75739fc1016591bc1c64c5/lib/libReactantExtra.so: undefined symbol: cuStreamBeginCaptureToGraph
Stacktrace:
  [1] dlopen(s::String, flags::UInt32; throw_error::Bool)
    @ Base.Libc.Libdl ./libdl.jl:120
  [2] dlopen(s::String, flags::UInt32)
    @ Base.Libc.Libdl ./libdl.jl:119
  [3] macro expansion
    @ ~/.julia/packages/JLLWrappers/GfYNv/src/products/library_generators.jl:63 [inlined]
  [4] __init__()
    @ Reactant_jll ~/.julia/packages/Reactant_jll/k7Xgx/src/wrappers/x86_64-linux-gnu-cxx11-gpu+cuda-mode+opt.jl:30
  [5] run_module_init(mod::Module, i::Int64)
    @ Base ./loading.jl:1336
  [6] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:1324
  [7] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any}, ignore_native::Nothing; register::Bool)
    @ Base ./loading.jl:1213
  [8] _include_from_serialized (repeats 2 times)
    @ ./loading.jl:1169 [inlined]
  [9] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128, stalecheck::Bool; reasons::Dict{String, Int64}, DEPOT_PATH::Vector{String})
...

It can be temporarily solved by adding this to LocalPreferences.toml:

[Reactant_jll]
gpu = "none"

[Todorbsc]

It is worthmentioning that I got this error in the cluster but not in my local PC (Ubuntu 22.04). Also, it gives the same error for Reactant#main branch.

[mofeing]

@wsmoses @giordano this cluster doesn't have NVIDIA GPUs in it. i don't know why it's trying to download the CUDA version of it.

[giordano]

this cluster doesn't have NVIDIA GPUs in it

That's funny, because the only way to end up with gpu=cuda that I can think of is to have a loadable libcuda.so.1/libcuda.so in the dynamic loader search path. That, or there was a preference file already forcing gpu=cuda, or the environment variable REACTANT_GPU was set to cuda, but hopefully you'd be aware if you did any of that.

[avik-pal]

~/.julia/artifacts/bf5c832056d192639a75739fc1016591bc1c64c5/lib/libReactantExtra.so: undefined symbol: cuStreamBeginCaptureToGraph

I encountered this yesterday when CUDA_Runtime_jll was set to 12.4 with Reactant. It doesn't seem to be an issue with 12.6. This was on a server with nvidia gpus, but preferences need to be manually set since compute nodes don't have internet

[wsmoses]

@giordano maybe we should add an extra test to the JLL that the libcuda version is at least a certain version (currently the cuda library version support is baked into the jll). Of course we can build more jll's for different versions

Reactant.jl/deps/ReactantExtra/.bazelrc

Line 21 in a5fb8cb

build:cuda --repo_env=HERMETIC_CUDA_VERSION="12.6.2"

[giordano]

I wonder if it's picking libcuda.so.1 from CUDA_Driver

[giordano]

Ook, I can reproduce the issue with just Reactant_jll@0.0.39. Let's see what's going on...

[giordano]

Ah, I do have libcuda.so.1 on this machine somehow

julia> using Libdl

julia> dlpath(dlopen("libcuda.so.1"))
"/lib/x86_64-linux-gnu/libcuda.so.1"

but I don't think there's a GPU here 😄 @Todorbsc @Wu-Chenyang Is it possible you're in a similar situation?

Edit: full reproducer:

julia> rm(joinpath(Base.DEPOT_PATH[1], "compiled", "v$(VERSION.major).$(VERSION.minor)", "Reactant_jll"); recursive=true, force=true)

julia> ENV["JULIA_DEBUG"] = "Reactant_jll"
"Reactant_jll"

julia> using Pkg; Pkg.activate(; temp=true); Pkg.add("Reactant_jll")
  Activating new project at `/tmp/jl_cNaAWd`
   Resolving package versions...
    Updating `/tmp/jl_cNaAWd/Project.toml`
  [0192cb87] + Reactant_jll v0.0.39+0
    Updating `/tmp/jl_cNaAWd/Manifest.toml`
  [692b3bcd] + JLLWrappers v1.7.0
  [21216c6a] + Preferences v1.4.3
  [4ee394cb] + CUDA_Driver_jll v0.11.0+0
  [0192cb87] + Reactant_jll v0.0.39+0
  [0dad84c5] + ArgTools v1.1.2
  [56f22d72] + Artifacts v1.11.0
  [2a0f44e3] + Base64 v1.11.0
  [ade2ca70] + Dates v1.11.0
  [f43a241f] + Downloads v1.6.0
  [7b1f6079] + FileWatching v1.11.0
  [4af54fe1] + LazyArtifacts v1.11.0
  [b27032c2] + LibCURL v0.6.4
  [76f85450] + LibGit2 v1.11.0
  [8f399da3] + Libdl v1.11.0
  [56ddb016] + Logging v1.11.0
  [d6f4376e] + Markdown v1.11.0
  [ca575930] + NetworkOptions v1.2.0
  [44cfe95a] + Pkg v1.11.0
  [de0858da] + Printf v1.11.0
  [9a3f8284] + Random v1.11.0
  [ea8e919c] + SHA v0.7.0
  [fa267f1f] + TOML v1.0.3
  [a4e569a6] + Tar v1.10.0
  [cf7118a7] + UUIDs v1.11.0
  [4ec0a83e] + Unicode v1.11.0
  [deac9b47] + LibCURL_jll v8.6.0+0
  [e37daf67] + LibGit2_jll v1.7.2+0
  [29816b5a] + LibSSH2_jll v1.11.0+1
  [c8ffd9c3] + MbedTLS_jll v2.28.6+0
  [14a3606d] + MozillaCACerts_jll v2023.12.12
  [83775a58] + Zlib_jll v1.2.13+1
  [8e850ede] + nghttp2_jll v1.59.0+0
  [3f19e933] + p7zip_jll v17.4.0+2
Precompiling project...
  1 dependency successfully precompiled in 2 seconds. 27 already precompiled.
  1 dependency had output during precompilation:
┌ Reactant_jll
│  ┌ Debug: Adding include dependency on /lib/x86_64-linux-gnu/libcuda.so.1
│  └ @ Reactant_jll ~/.julia/packages/Reactant_jll/k7Xgx/.pkg/platform_augmentation.jl:58

[mofeing]

Ah, I do have libcuda.so.1 on this machine somehow

ah funk, we do too.

julia> using Libdl

julia> dlpath(dlopen("libcuda.so.1"))
"/lib64/libcuda.so.1"

but no idea why neither.

[giordano]

So, one thing I may have changed in practice in JuliaPackaging/Yggdrasil@06cc185 (#10190) is that the GPU mode basically became opt-out instead of opt-in. I'm happy to change it to "if there's no preference already set assume the user doesn't want to use GPU", but I believe at the moment this is working as (I?) intended: automatically choose GPU=cuda if libcuda.so.1 is there.

[wsmoses]

Maybe we can update it where if we're compiling for cuda 12.6 (as is presently), we check if a relevant cuda 12 symbol exists (so then cuda 11 won't cause errors)

[wsmoses]

It doesn't matter if you have a GPU or not (reactant would just startup warning no GPU found). However the issue here is that we link against symbols in the gpu jll, which if not found, will err like above.

[mofeing]

yeah, opt-out is nice. the problem is that we have to enforce CUDA 12.6.

one question: if the minimum CUDA version is not found in the system but the user install it in the environment by installing CUDA.jl (or CUDA_jll directly), would it detect correctly and download the CUDA version of Reactant_jll?