Add new activation script and/or run dependency?

[leofang]

Continuing the discussion from AnacondaRecipes/cudatoolkit-feedstock#5 (comment) and cupy/cupy#3222 (comment):

1. It'd be much better if nvcc can exist in a conda env where CuPy is installed
2. If not, we should set CUDA_PATH for CuPy to look up where CUDA is in the conda env.

[leofang]

For my first proposal, I'm worrying that we need to resolve conda-forge/nvcc-feedstock#25 (comment) first. Right, @jakirkham?

[jakirkham]

If we think that is needed, I suppose we could just add them here independently of how that issue is resolved.

That said, it seems like 2 is the lighter weight of these proposals.

[leofang]

John, I feel I could be wrong. Am I right that the CUDA headers aren't included in either nvcc or cudatoolkit? And why aren't they included?

[jakirkham]

They are not. Why do you ask?

My guess is they are not redistributable according to the license, but am not sure.

@kkraus14, do you know?

FWIW the headers we use during the build come from the Docker image.

[kkraus14]

The headers are not permitted for redistribution per the EULA: https://docs.nvidia.com/cuda/eula/index.html#attachment-a

[leofang]

I see, thanks for confirming, John & Keith!

I am asking because this CuPy test is failing with CF's nvcc_linux-64, gcc_linux-64, gxx_linux-64, cupy=7.3.0, cudatoolkit=10.0:

$ pytest cupy_tests/core_tests/test_raw.py

(...omit output...)

>                   raise cex
E                   cupy.cuda.compiler.CompileException: `nvcc` command returns non-zero exit status. 
E                   command: ['/home/leofang/miniconda3/envs/CF_cupy_test2/bin/nvcc', '-gencode=arch=compute_70,code=sm_70', '--ptx', '-I/home/leofang/miniconda3/envs/CF_cupy_test2/lib/python3.7/site-packages/cupy/core/include', '-I /home/leofang/miniconda3/envs/CF_cupy_test2/lib/python3.7/site-packages/cupy/core/include/cupy/_cuda/cuda-10.0', '-I /usr/local/cuda/include', '-ftz=true', '/tmp/tmp6r4ykd15/preprocess.cu']
E                   return-code: 1
E                   stdout/stderr: 
E                   cc1plus: fatal error: cuda_runtime.h: No such file or directory
E                   compilation terminated.

../../miniconda3/envs/CF_cupy_test2/lib/python3.7/site-packages/cupy/cuda/compiler.py:198: CompileException
======================================================================= short test summary info =======================================================================
FAILED cupy_tests/core_tests/test_raw.py::TestRaw_param_0_{backend='nvrtc'}::test_dynamical_parallelism - RuntimeError: Parameterized test failed.
FAILED cupy_tests/core_tests/test_raw.py::TestRaw_param_1_{backend='nvcc'}::test_basic - cupy.cuda.compiler.CompileException: `nvcc` command returns non-zero exit s...
FAILED cupy_tests/core_tests/test_raw.py::TestRaw_param_1_{backend='nvcc'}::test_compiler_flag - cupy.cuda.compiler.CompileException: `nvcc` command returns non-zer...
FAILED cupy_tests/core_tests/test_raw.py::TestRaw_param_1_{backend='nvcc'}::test_cuDoubleComplex - cupy.cuda.compiler.CompileException: `nvcc` command returns non-z...
FAILED cupy_tests/core_tests/test_raw.py::TestRaw_param_1_{backend='nvcc'}::test_cuFloatComplex - cupy.cuda.compiler.CompileException: `nvcc` command returns non-ze...
FAILED cupy_tests/core_tests/test_raw.py::TestRaw_param_1_{backend='nvcc'}::test_dynamical_parallelism - cupy.cuda.compiler.CompileException: `nvcc` command returns...
FAILED cupy_tests/core_tests/test_raw.py::TestRaw_param_1_{backend='nvcc'}::test_dynamical_parallelism_compile_failure - cupy.cuda.compiler.CompileException: `nvcc`...
FAILED cupy_tests/core_tests/test_raw.py::TestRaw_param_1_{backend='nvcc'}::test_get_function_failure - cupy.cuda.compiler.CompileException: `nvcc` command returns ...
FAILED cupy_tests/core_tests/test_raw.py::TestRaw_param_1_{backend='nvcc'}::test_invalid_compiler_flag - cupy.cuda.compiler.CompileException: `nvcc` command returns...
FAILED cupy_tests/core_tests/test_raw.py::TestRaw_param_1_{backend='nvcc'}::test_kernel_attributes - cupy.cuda.compiler.CompileException: `nvcc` command returns non...
FAILED cupy_tests/core_tests/test_raw.py::TestRaw_param_1_{backend='nvcc'}::test_module - cupy.cuda.compiler.CompileException: `nvcc` command returns non-zero exit ...
FAILED cupy_tests/core_tests/test_raw.py::TestRaw_param_1_{backend='nvcc'}::test_module_both_code_and_path - cupy.cuda.compiler.CompileException: `nvcc` command ret...
FAILED cupy_tests/core_tests/test_raw.py::TestRaw_param_1_{backend='nvcc'}::test_module_load_failure - cupy.cuda.compiler.CompileException: `nvcc` command returns n...
FAILED cupy_tests/core_tests/test_raw.py::TestRaw_param_1_{backend='nvcc'}::test_module_neither_code_nor_path - cupy.cuda.compiler.CompileException: `nvcc` command ...
=================================================================== 14 failed, 12 passed in 11.07s ====================================================================

What these tests do is to use RawModule and RawKernel to compile some CUDA codes with both nvrtc and nvcc backends. The former works just fine (except for the one that needs cudadevrt, which is not yet available in cudatoolkit), but CF's nvcc fails all tests due to the absence of CUDA headers.

(To reproduce this, you need to mask your CUDA_PATH and /usr/local/cuda, pretending you don't have a system-wide CUDA.)

@jakirkham It seems patching get_cuda_path() as you suggested in cupy/cupy#3222 (comment) would not help revolve this, because the CUDA headers are simply unavailable. We likely need to implement a detection scheme to tell if we're using CF cupy, and simply ban the usage of the nvcc backend in RawModule and RawKernel.

However, your sys.prefix solution as well as the two of mine are still valid proposals for improving runtime discovery, which is needed to tell nvrtc where cudadevrt is, for example.

[jakirkham]

Interesting I feel like I'm learning a lot about how CuPy works internally 😄

FWIW we do add the headers to the compiler flags. Though it sounds like you are saying CuPy doesn't use these, is that correct? 😉

[leofang]

Haha, sorry for making you learn, John 😅 btw it's not that internal. The backend thing is documented. They didn't advertise it, though.

I meant CuPy would likely find the CUDA path and add -I$CUDA_HOME/include when invoking the compiler, but CUDA headers are not in $CUDA_HOME/include or anywhere in a conda env. (See the error message above: it says cuda_runtime.h not found.)