gcc 9.3.0 migration

[beckermr]

to do

• do put libgfortran SONAME in the package name for linux
  • impl: REF add soname to libgfortran package name ctng-compilers-feedstock#45
  • activation: REF use soname for libgfortran ctng-compiler-activation-feedstock#38
• add target platform to build hashes
  • can do globally: Include target_platform in hash conda/conda-build#4065
  • waiting on conda build release but we can put the patch in our feedstock if we want
  • rebuild osx gfortran compilers to include target_platform in the hashes (ENH move to latest upstream sha gfortran_impl_osx-64-feedstock#32)
• do gfortran migration with bot because of SONAME change
• bump pinnings

[jakirkham]

cc @raydouglass @kkraus14 @mike-wendt (for vis)

[kkraus14]

Thanks @jakirkham. We may need to handle pinning the compiler in the nvcc feedstock to prevent gcc-9 from being pulled when nvcc 10.2 or lower is used.

[jakirkham]

So I saw PR ( conda-forge/openmpi-feedstock#67 ) recently, which is moving to GCC 9 for a package that also builds CUDA components. Admittedly OpenMPI's usage of CUDA is pretty minimal, but AIUI we probably shouldn't be doing that. Can you please confirm Keith?

If not, maybe we should discuss how we can exclude these packages from the GCC 9 migration to avoid creating packages that are broken. Not sure how we do that, but want to raise the concern.

[beckermr]

We are building both gcc7 and gcc9 on linux-64. So we can mark the gcc9 ones broken and put in proper skips in the recipe. We could also zip the cuda keys with the other compiler keys, but this will be more complicated.

[kkraus14]

So I saw PR ( conda-forge/openmpi-feedstock#67 ) recently, which is moving to GCC 9 for a package that also builds CUDA components. Admittedly OpenMPI's usage of CUDA is pretty minimal, but AIUI we probably shouldn't be doing that. Can you please confirm Keith?

The CUDA compiler, nvcc, depends on the host compiler, gcc / g++ in our case. CUDA 11.0 is the first version of CUDA that supports gcc <=9.x. CUDA 10.0 supports gcc <=7.x, CUDA 10.1 and 10.2 support gcc <=8.x (https://gist.github.com/ax3l/9489132).

Note that if a package is only using the CUDA host APIs but not actually compiling any CUDA code then using gcc 9.x should be safe regardless.

[beckermr]

Note that if a package is only using the CUDA host APIs but not actually compiling any CUDA code then using gcc 9.x should be safe regardless.

Thanks @kkraus14! I think given this we should put the constraint directly in openmpi if it is needed. Our cuda universe is not so big right now that we can't handle this more selectively.

[beckermr]

I made an issue here: conda-forge/openmpi-feedstock#68 - We can discuss more there.

[kkraus14]

Thanks @kkraus14! I think given this we should put the constraint directly in openmpi if it is needed. Our cuda universe is not so big right now that we can't handle this more selectively.

I think this is an argument for us to split the nvcc_linux-64 package into multiple packages. Right now it's a package that handles the symlinking for both the compiler as well as the headers. If we had a separate package for just the headers that the nvcc_linux-64 package could depend on then libraries like arrow-cpp that use CUDA host APIs but don't actually have any CUDA code (.cu files) could depend on just the header package.

[jakirkham]

Maybe this could be handled as part of a cudatoolkit package? 😉

[kkraus14]

Maybe this could be handled as part of a cudatoolkit package? 😉

The headers, compiler, and driver library are not allowed to be redistributed per the CUDA EULA. Right now we actually don't symlink the headers in any way, we just symlink the compiler and the CUDA driver library and the compiler handles finding the necessary headers. For libraries like arrow-cpp, they've likely already special cased finding the necessary CUDA headers on the system and don't require our nvcc package regardless, but it's currently the way to get to a CUDA based docker image.

[jakirkham]

Right not saying we package the headers or the driver library. Just saying we set the appropriate environment variables as we do currently. Maybe with an activation script?

[jakirkham]

So when an incompatible GCC and CUDA Version are used, would there be compiler errors or is it possible for it to silently succeed (even if what it produced is problematic)?

[jakirkham]

FWIW here's a use case where we do need to match GNU compilers to CUDA compilers ( conda-forge/tomopy-feedstock#39 ). Tried proposing a way to combine them correctly ( conda-forge/tomopy-feedstock#39 (comment) ), but would not at all be surprised if that is wrong and needs changes. Though hopefully that gets us started thinking about how we should address this use case.

[beckermr]

This one is done!