[installation errors] fatal error: nccl.h: No such file or directory

[qianyuzqy]

Hello, thanks for providing such a great work. However, I cannot install tutel successfully during
the compilation. I have exported the lib path of nccl_2.7.8-1-cuda10.1/include/nccl.h into LD_LIBRARY_PATH. But the error logs seem that it still cannot find the NCCL path.
Do you have any idea on how to solve this error? Thanks!

running install_lib
running build_py
running build_ext
building 'tutel_custom_kernel' extension
Emitting ninja build file /mnt/lustre/chengguangliang/zhouqianyu/segdgformer/cvpr_2023/tutel-main/build/temp.linux-x86_64-3.7/build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/1] /mnt/cache/share/gcc/gcc-7.3.0/bin/g++ -MMD -MF /mnt/lustre/chengguangliang/zhouqianyu/tutel-main/build/temp.linux-x86_64-3.7/
tutel/custom/custom_kernel.o.d -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/mnt/lustre/chengguangliang/miniconda3/lib/python3
.7/site-packages/torch/include -I/mnt/lustre/chengguangliang/miniconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/mnt/lustre/che
ngguangliang/miniconda3/lib/python3.7/site-packages/torch/include/TH -I/mnt/lustre/chengguangliang/miniconda3/lib/python3.7/site-packages/torch/include/TH
C -I/mnt/lustre/share/cuda-10.1/include -I/mnt/lustre/chengguangliang/miniconda3/include/python3.7m -c -c /mnt/lustre/chengguangliang/zhouqianyu/tutel-main/tutel/custom/custom_kernel.cpp -o /mnt/lustre/chengguangliang/zhouqianyu/tutel-main/build/temp.linux-x86_64-
3.7/tutel/custom/custom_kernel.o -Wno-sign-compare -Wno-unused-but-set-variable -Wno-terminate -Wno-unused-function -Wno-strict-aliasing -DUSE_GPU -DUSE_N
CCL -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENS
ION_NAME=tutel_custom_kernel -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
FAILED: /mnt/lustre/chengguangliang/zhouqianyu/tutel-main/build/temp.linux-x86_64-3.7/tutel/custom/custom_kernel.o
/mnt/cache/share/gcc/gcc-7.3.0/bin/g++ -MMD -MF /mnt/lustre/chengguangliang/zhouqianyu/tutel-main/build/temp.linux-x86_64-3.7/tutel/
custom/custom_kernel.o.d -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/mnt/lustre/chengguangliang/miniconda3/lib/python3.7/sit
e-packages/torch/include -I/mnt/lustre/chengguangliang/miniconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/mnt/lustre/chengguan
gliang/miniconda3/lib/python3.7/site-packages/torch/include/TH -I/mnt/lustre/chengguangliang/miniconda3/lib/python3.7/site-packages/torch/include/THC -I/m
nt/lustre/share/cuda-10.1/include -I/mnt/lustre/chengguangliang/miniconda3/include/python3.7m -c -c /mnt/lustre/chengguangliang/zhouqianyu/tutel-main/tutel/custom/custom_kernel.cpp -o /mnt/lustre/chengguangliang/zhouqianyu/tutel-main/build/temp.linux-x86_64-3.7/tu
tel/custom/custom_kernel.o -Wno-sign-compare -Wno-unused-but-set-variable -Wno-terminate -Wno-unused-function -Wno-strict-aliasing -DUSE_GPU -DUSE_NCCL -D
TORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NA
ME=tutel_custom_kernel -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
/mnt/lustre/chengguangliang/zhouqianyu/tutel-main/tutel/custom/custom_kernel.cpp:19:10: fatal error: nccl.h: No such file or directory
#include <nccl.h>
^~~~~~~~
compilation terminated.
ninja: build stopped: subcommand failed.
Try installing without NCCL extension..

My machine details are as follows:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Apr_24_19:10:27_PDT_2019
Cuda compilation tools, release 10.1, V10.1.168

I used pytorch1.8.1 with cuda 10.1. I wonder whether tutal can be installed with cuda10.1?

I used the following commands for the installation:

export PATH=/mnt/lustre/share/gcc/gcc-5.4/bin/:$PATH
export LD_LIBRARY_PATH=/mnt/lustre/share/polaris/dep/nccl_2.7.8-1-cuda10.1/include/nccl.h:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/mnt/lustre/share/gcc/gmp-4.3.2/lib:/mnt/lustre/share/gcc/mpfr-2.4.2/lib:/mnt/lustre/share/gcc/mpc-0.8.1/lib:$LD_LIBRARY_PATH
export TORCH_CUDA_ARCH_LIST='3.5;5.0+PTX;6.0;7.0'

python setup.py install --user

[ghostplant]

This is an improper environment configuration not recognized by pytorch. Can you make a copy of nccl.h to /usr/include, and a copy of libnccl.so to /usr/lib/x86_64-linux-gnu? (Softlink is also fine.)