cupy v11.1.0

[regro-cf-autotick-bot]

It is very likely that the current package version for this feedstock is out of date.

Checklist before merging this PR:

• Dependencies have been updated if changed: see upstream
• Tests have passed
• Updated license if changed and license_file is packaged

Information about this PR:

1. Feel free to push to the bot's branch to update this PR if needed.
2. The bot will almost always only open one PR per version.
3. The bot will stop issuing PRs if more than 3 version bump PRs generated by the bot are open. If you don't want to package a particular version please close the PR.
4. If you want these PRs to be merged automatically, make an issue with @conda-forge-admin,please add bot automerge in the title and merge the resulting PR. This command will add our bot automerge feature to your feedstock.
5. If this PR was opened in error or needs to be updated please add the bot-rerun label to this PR. The bot will close this PR and schedule another one. If you do not have permissions to add this label, you can use the phrase @conda-forge-admin, please rerun bot in a PR comment to have the conda-forge-admin add it for you.

Dependency Analysis

We couldn't run dependency analysis due to an internal error in the bot. :( Help is very welcome!

_{This PR was created by the regro-cf-autotick-bot. The regro-cf-autotick-bot is a service to automatically track the dependency graph, migrate packages, and propose package version updates for conda-forge. Feel free to drop us a line if there are any issues! This PR was generated by https://github.com/regro/autotick-bot/actions/runs/2968944341, please use this URL for debugging.}

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[leofang]

Hi @kmaehashi @emcastillo Any chance the errors in the Win + CUDA 11.2 CIs look familiar to you?

[kmaehashi]

The first error seems here: https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=561424&view=logs&j=84cf05f7-5267-5f38-137b-63208e75d05b&t=7c51bcc0-b6e0-5e4f-1968-7bb191e490bd&l=2192

I haven't seen this before, but maybe Visual Studio version used in the build bumped recently?
https://stackoverflow.com/a/67736902/8860262

[jakirkham]

Potentially related ( conda-forge/vc-feedstock#48 )

[leofang]

@jakirkham I think we can try downgrading the vs version, but how to do it in meta.yaml?

[jakirkham]

Maybe this is helpful?

[leofang]

Ahh ok so during rerendering the bot changed the compiler from vc2017 to vc2019. I didn't notice it as the diff was hidden. Let's roll it back and give it a shot.

There's still one problem remaining, though: Why does CUDA 11.2 hate vc2019, but not 11.1 and earlier (including 10.2)?

[leofang]

It seems the real culprit is conda-forge/conda-forge-pinning-feedstock#3167, not the vc2019 update (in the vc-feedstock).

[leofang]

@kmaehashi @jakirkham thank you for your help, I was clueless! I see two potential solutions:

1. Roll back to vc2017 (as mentioned above)
2. Try to compile with CUDA 11.3+

Any preference which we should try first?

[h-vetinari]

There's still one problem remaining, though: Why does CUDA 11.2 hate vc2019, but not 11.1 and earlier (including 10.2)?

I've used vs2019 + CUDA 11.2 for a long time in the faiss feedstock.

Based on the traceback it indeed looks like it might have something to do with the update of the redistributable (although I'd have said that the chance for that should have been really low). If you want, you can check if using vs2015_runtime<14.29.30339 changes anything.

Also interesting: the traceback contains 14.29.30133 (presumably from the windows-2019 image), while conda-forge/vc-feedstock#48 uses 14.29.30139.

  "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX64\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -DCUPY_NO_NVTX=1 -D_FORCE_INLINES=1 -DCUPY_CUB_VERSION_CODE=101000 -DCUPY_JITIFY_VERSION_CODE=-1 "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.2\include\cub" -ID:\bld\cupy_1662012999200\work\cupy/_core/include "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.2\include" -ID:\bld\cupy_1662012999200\_h_env\include -ID:\bld\cupy_1662012999200\_h_env\Include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22000.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22000.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22000.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22000.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22000.0\cppwinrt" -ID:\bld\cupy_1662012999200\_h_env\Library\include "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.2\include" -ID:\bld\cupy_1662012999200\_h_env\Library\include /d1trimfile:D:\bld\cupy_1662012999200\work /EHsc /Tpcupy\_core\dlpack.cpp /Fobuild\temp.win-amd64-cpython-37\Release\cupy\_core\dlpack.obj
  dlpack.cpp
  C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\include\xutility(1260): error: expected a "("
            detected during instantiation of "void std::_Adl_verify_range(const _Iter &, const _Sentinel &) [with _Iter=const char *, _Sentinel=const char *]"
  C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\include\xlocale(1971): here

CC @isuruf

[leofang]

Does the Windows image also have a copy of the compiler toolchain?

[h-vetinari]

Does the Windows image also have a copy of the compiler toolchain?

Seems so. That path definitely looks like it's not from conda.

[leofang]

@conda-forge-admin, please rerender

[github-actions]

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you, but it looks like there was nothing to do.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/cupy-feedstock/actions/runs/3003678551.

[leofang]

@conda-forge-admin, please rerender

[leofang]

@kmaehashi I have some clues, please help me verify my theory.

CuPy relies on distutils, which records the host compiler information that is used to build Python. Conda-forge's Python was built with vs2017, and for some reason the version 14.29.30133 was used (search its build log) instead of the ones distributed on conda-forge, thus we're hitting this issue.

If my theory is correct, rolling back to vs2017 (as done in commit 464c701) would fix it.

[kmaehashi]

Rolling back system-wide compiler to vs2017 sounds good to me.

To my understanding, setuptools/distutils discovers Visual Studio only from system-wide installation unless DISTUTILS_USE_SDK is set. If that env var is set, setuptools does nothing so Visual Studio env vars need to be set by the calling process (usually via vcvarsall.bat batch file which is a part of Visual Studio.)

https://github.com/pypa/setuptools/blob/ba3995e5705a22e13bb5d2231ac22c77e4417747/setuptools/msvc.py#L194

[leofang]

Thanks for pointer, @kmaehashi! I see it's documented here: https://setuptools.pypa.io/en/latest/deprecated/distutils/apiref.html?highlight=DISTUTILS_USE_SDK#module-distutils.msvccompiler.

To be clear we should not use system compiler on conda-forge. However, in this case it's leaking from python-feedstock, leaving us no choice but to follow because

Typically, extension modules need to be compiled with the same compiler that was used to compile Python.

I will open an issue in python-feedstock to track this (though I doubt it's fixable there), but for now we'll have to live with it.

[h-vetinari]

Typically, extension modules need to be compiled with the same compiler that was used to compile Python.

That's a general recommendation because most people have no idea/capabilities to track the impact the compiler has on the ABI of certain packages. But VS2015-VS2022 is compatible, and conda-forge is very careful about these things.

To be clear we should not use system compiler on conda-forge. However, in this case it's leaking from python-feedstock, leaving us no choice but to follow because

I don't understand what you mean by system compilers. Conda-forge does its own setup for the Microsoft compilers, and you should use that (e.g. vs2017 or vs2019; if you add that in the conda_build_config.yaml and rerender, it'll get picked up). We only cannot redistribute it due to license reasons.

[leofang]

I don't understand what you mean by system compilers.

@h-vetinari Please see #179 (comment) as a response to your finding in #179 (comment). Pay attention to the version number 14.29.30133. This is the "leak".

and you should use that (e.g. vs2017 or vs2019; if you add that in the conda_build_config.yaml and rerender, it'll get picked up).

Yup it is done in the current PR.

[h-vetinari]

@h-vetinari Please see #179 (comment) as a response to your finding in #179 (comment). Pay attention to the version number 14.29.30133. This is the "leak".

There are different components involved here, the compiler and the redistributable (and so on). Both should be compatible with each other. Microsoft has kept compatibility in the whole 14.x line, so I think it's a stretch for me to imagine that 14.29.30133 and 14.29.30139 are somehow incompatible. It's probably a symptom of something else.

Perhaps @isuruf knows what's happening...

[leofang]

Yeah, and thanks for the compatibility note, I wasn't aware of the msvc support range. Anyway, we're good with vs2017. We can discuss the remaining puzzles

• How does 14.29.30133 appear?
• Why only CUDA 11.2 was impacted?

either here or elsewhere.

[leofang]

I will open an issue in python-feedstock to track this

conda-forge/python-feedstock#580