Install problem: Cannot compile on platform 'posix' with 'nvidia' compiler

[andyfcx]

I am installing implicit through pip install git+https://github.com/benfred/implicit.git@f33d2e7d753f3ab4da0901485bd68e47dba7b9eb

And I got this error during installation

error: don't know how to compile C/C++ code on platform 'posix' with 'nvidia' compiler
----------------------------------------
ERROR: Failed building wheel for implicit
Failed to build implicit
ERROR: Could not build wheels for implicit, which is required to install pyproject.toml-based projects

I am using tensorflow/tensorflow:2.3.0-gpu as my docker base image, image digest: 9a52554e85ae666767c28f9af70e56caa7e6d75325951ab021fbd4bb1447a340

This error never happened before, I am not sure what happened to the compiler or which compiler should be used?

Please help me, any help or advise are welcomed.

[tRosenflanz]

Note this is reproducible locally with the latest implicit too AND didn't happen before. The machine doesn't have a GPU which is what is probably causing the error BUT this shouldn't be an issue since nvcc works fine without it (tested by compiling https://gist.github.com/f0k/0d6431e3faa60bffc788f8b4daa029b1)
Tensorflow itself and other dependencies install fine too

FROM tensorflow/tensorflow:2.3.0-gpu

ENV PIP_IGNORE_INSTALLED 0
ENV PYTHONUNBUFFERED 1

ENV PATH ${HOME}/.local/bin:${PATH}
ENV LD_LIBRARY_PATH /usr/local/cuda-10.1/lib64

RUN set -ex && \
    apt-get update && \
    apt-get install -y git && \
    apt-get install -y python3.7 && \
    apt-get install -y nvidia-cuda-toolkit && \
    apt-get install  -y python3.7-dev && \
    apt-get install -y python3-pip && \
    apt-get install -y nano && \
    apt-get install -y iproute2 && \
    apt-get install -y xvfb && \
    DEBIAN_FRONTEND=noninteractive apt-get install -y wkhtmltopdf && \
    rm -f /usr/bin/python3 /usr/bin/python && \
    ln -s /usr/bin/python3.7 /usr/bin/python && \
    ln -s /usr/bin/python3.7 /usr/bin/python3 && \
    python3.7 -m pip install --upgrade pip setuptools

RUN pip install implicit

[tRosenflanz]

This seems relevant considering the setup scripts borrow heavily from cupy:
cupy/cupy#5476

[benfred]

@tRosenflanz nice find with the cupy issue!

does the fix that cupy put in place ( https://github.com/cupy/cupy/pull/5482/files ) for this problem solve the issue here ?

[tRosenflanz]

No, inserting it into cuda_setup.py didn't help

[tRosenflanz]

I tested on an ec2 instance with a gpu and it fails as well. So this isn't a cpu isolated issue

[andyfcx]

@tRosenflanz @benfred
Found this is a setuptool problem, setuptool recent release date is 2021/12/21.
We can successfully install implicit with setuptools==58.5.3
Testing other versions

[andyfcx]

The max success version number of setuptools is 59.8.0

[tRosenflanz]

Tested more and this only works for the specific fixed version
pip install git+https://github.com/benfred/implicit.git@f33d2e7d753f3ab4da0901485bd68e47dba7b9eb
But the most recent version still fails even with rollback of setuptools

[benfred]

Can you try with adding the cupy fix to setup.py - right before the setuptools imports? I think adding to cuda_setup.py is too late, and this needs added before the setuptools imports

Applying this patch to the dockerfile supplied by @tRosenflanz above fixes this error:

diff --git a/setup.py b/setup.py
index fd42db5..eef201b 100644
--- a/setup.py
+++ b/setup.py
@@ -5,6 +5,11 @@ import os.path
 import platform
 import sys
 
+try:
+    import numpy.distutils
+except ImportError:
+    pass
+
 from setuptools import Extension, find_packages, setup
 
 from cuda_setup import CUDA, build_ext

however , it still doesn't compile all the way - since we need cuda 11.0 with the latest code to compile, and the nvcc on that tensorflow image you're using is only 10.1 , and it fails with complaining about trying to pass a --extended-lambda flag to nvcc which doesn't exist with nvcc 10.1

[tRosenflanz]

That totally makes sense. I missed the upgrade to CUDA 11 since we have been using the fixed version from above. Glad that the monkeypatching is actually helpful though

[benfred]

I've managed to replicate this issue on my dev machine, just by upgrading to the latest setuptools. I've submitted a fix here that enables the build again: #500 .

Sorry about the lack of notice on dropping support for cuda < 11.0 =(. It wasn't obvious to me that the new GPU inference code wouldn't compile on older versions of CUDA until I hit issues with the GPU packages in conda-forge. Btw, I'm making some breaking API changes #481 that you should probably be aware of.

The good news is that the GPU inference code is substantially faster - and with upgrades in the the apirefactor branch will get even faster still by adding a batch mode to GPU inference. As an example, to generate all the similar items for the 'lastfm' example used to take around 40 minutes on my deskop, and is now taking around 8 seconds.