WIP: Add GPU builds #21

h-vetinari · 2021-04-10T13:10:17Z

Found out that upstream supports GPUs, let's try building for it.

~~There are a bunch of relevant changes that haven't seen a release yet (I've asked when that might be), but for the moment, I'm building from master.~~

Edit: new version release 🥳

conda-forge-linter · 2021-04-10T13:10:22Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

mcg1969 · 2021-04-10T15:49:45Z

I'm not sufficiently connected to the core devs anymore to weigh in intelligently on this. Perhaps @bodono can weigh in?

h-vetinari · 2021-04-10T23:14:22Z

Interestingly, this only fails for cuda 10.2 on windows, but works otherwise. I'd be completely fine in skipping that failing combination.

@conda-forge/nvcc it's a bit hard to tell for me if building for all CUDA versions is really required here, but the dependence on cudart makes me guess so?

In any case, feedback is welcome! :)

h-vetinari · 2021-04-10T23:16:55Z

Seems my enthusiasm might have been premature - the test suite is too old to be run with pytest, and everything just gets skipped... 🤦

EDIT: wasn't an issue after all. :)

recipe/meta.yaml

recipe/build-pkg.bat

kkraus14 · 2021-04-12T00:57:04Z

@conda-forge/nvcc it's a bit hard to tell for me if building for all CUDA versions is really required here, but the dependence on cudart makes me guess so?

In CUDA 11.0+ the soname on libcudart is the same, so if no newer features are used then in theory you can build with the latest CUDA 11.x version and run on all 11.x versions of libcudart (at least on the linux side, not sure about the windows side).

h-vetinari · 2021-04-12T21:19:50Z

@xhochy @isuruf
I'd be glad if you could have a quick look at this PR. There are some build errors on win + cuda 10.2 (which remain also with vs2019), but I'd be fine to skip that and just build win+cuda>=11.

Obviously those changes are still outstanding (revert vs2019, skip win+cuda10.2), but wanted to leave it as is right now for ease of reviewing.

xhochy · 2021-04-15T09:08:06Z

@h-vetinari Should I look at this PR in general or with regards to win/GPU? For the latter, I have no experience/expertise.

h-vetinari · 2021-04-15T09:36:05Z

Should I look at this PR in general or with regards to win/GPU?

General review would already be very nice, there's no win-specific GPU parts here anyway. :)

xhochy · 2021-04-15T09:57:03Z

recipe/meta.yaml

-    - export BLAS_LAPACK_LIBS="lapack:cblas:blas"   # [unix]
-    - {{ PYTHON }} -m pip install . -vv
-  skip: true  # [py==27 and (win or (not x86_64))]
+  number: {{ number }}


Keep this as a plain number, otherwise the bot will not update it anymore.

xhochy · 2021-04-15T09:57:36Z

recipe/meta.yaml

+    script: build-pkg.bat  # [win]
+    build:
+      string: "py{{ CONDA_PY }}h{{ PKG_HASH }}_{{ number }}_{{ scs_proc_type }}"                                                  # [cuda_compiler_version == "None"]
+      string: "py{{ CONDA_PY }}cuda{{ cuda_compiler_version|replace(".", "") }}h{{ PKG_HASH }}_{{ number }}_{{ scs_proc_type }}"  # [cuda_compiler_version != "None"]


Suggested change

string: "py{{ CONDA_PY }}cuda{{ cuda_compiler_version|replace(".", "") }}h{{ PKG_HASH }}_{{ number }}_{{ scs_proc_type }}" # [cuda_compiler_version != "None"]

string: "py{{ CONDA_PY }}cuda{{ cuda_compiler_version|replace(".", "") }}h{{ PKG_HASH }}_{{ PKG_BUILDNUM }}_{{ scs_proc_type }}" # [cuda_compiler_version != "None"]

xhochy · 2021-04-15T09:57:42Z

recipe/meta.yaml

+    script: build-pkg.sh   # [not win]
+    script: build-pkg.bat  # [win]
+    build:
+      string: "py{{ CONDA_PY }}h{{ PKG_HASH }}_{{ number }}_{{ scs_proc_type }}"                                                  # [cuda_compiler_version == "None"]


Suggested change

string: "py{{ CONDA_PY }}h{{ PKG_HASH }}_{{ number }}_{{ scs_proc_type }}" # [cuda_compiler_version == "None"]

string: "py{{ CONDA_PY }}h{{ PKG_HASH }}_{{ PKG_BUILDNUM }}_{{ scs_proc_type }}" # [cuda_compiler_version == "None"]

recipe/meta.yaml

h-vetinari · 2021-04-21T22:44:10Z

Thanks for the review @xhochy; I'm sorry this slipped under the radar in a very busy week. Will try to get back to it soon.

h-vetinari

Thanks again for the review @xhochy, didn't know the PKG_BUILDNUM trick.

recipe/meta.yaml

h-vetinari · 2021-05-07T09:42:14Z

@conda-forge/help-pypy @conda-forge/nvcc
Quick question, are there any CUDA-enabled pypy packages in CF already? Is the expectation that this just works? CI runs through, but is obviously not testing the GPU paths.

jaimergp · 2021-05-07T10:55:35Z

OpenMM has some!

h-vetinari · 2021-08-18T10:19:26Z

@conda-forge-admin, please rerender

conda-forge-linter · 2021-08-18T10:21:17Z

Hi! This is the friendly automated conda-forge-linting service.

I was trying to look for recipes to lint for you, but it appears we have a merge conflict.
Please try to merge or rebase with the base branch to resolve this conflict.

Please ping the 'conda-forge/core' team (using the @ notation in a comment) if you believe this is a bug.

conda-forge-linter · 2021-08-18T10:52:51Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

h-vetinari · 2021-08-18T10:52:59Z

@conda-forge-admin, please rerender

h-vetinari · 2021-11-16T22:47:56Z

@conda-forge-admin, please rerender

…nda-forge-pinning 2022.01.14.19.19.28

h-vetinari requested review from djsutherland and mcg1969 as code owners April 10, 2021 13:10

h-vetinari marked this pull request as draft April 10, 2021 13:10

h-vetinari force-pushed the gpu branch 5 times, most recently from 1ed5ac8 to deec5d9 Compare April 10, 2021 18:22

This was referenced Apr 10, 2021

"undefined symbol: cusparseDcsrmv" error when building from source with GPU support in conda environment bodono/scs-python#23

Closed

CUDA paths for Windows bodono/scs-python#27

Merged

kkraus14 reviewed Apr 12, 2021

View reviewed changes

recipe/meta.yaml Outdated Show resolved Hide resolved

kkraus14 reviewed Apr 12, 2021

View reviewed changes

recipe/build-pkg.bat Outdated Show resolved Hide resolved

h-vetinari mentioned this pull request Apr 12, 2021

Strategy of solving many SOCPs with GPU bodono/scs-python#31

Open

h-vetinari force-pushed the gpu branch from 047de92 to 0f22391 Compare April 12, 2021 11:48

h-vetinari requested review from isuruf and xhochy April 12, 2021 21:17

xhochy reviewed Apr 15, 2021

View reviewed changes

h-vetinari force-pushed the gpu branch from 47b9fcf to 9de915e Compare May 7, 2021 09:25

h-vetinari commented May 7, 2021

View reviewed changes

recipe/meta.yaml Show resolved Hide resolved

This was referenced Jun 5, 2021

New test failures with 2.1.4 bodono/scs-python#43

Open

scs c library #18

Open

h-vetinari force-pushed the gpu branch 3 times, most recently from d3f53b8 to 992a6b7 Compare June 6, 2021 22:00

h-vetinari force-pushed the gpu branch from f742043 to 885848d Compare August 18, 2021 10:52

h-vetinari force-pushed the gpu branch from 46e67c2 to f261e03 Compare October 24, 2021 02:14

h-vetinari mentioned this pull request Oct 24, 2021

Leverage cuda-python for GPU detection bodono/scs-python#52

Open

h-vetinari force-pushed the gpu branch from a72a5c7 to 9a18e05 Compare November 12, 2021 11:13

conda-forge deleted a comment from conda-forge-linter Nov 12, 2021

h-vetinari mentioned this pull request Nov 12, 2021

Saving artefacts on azure runs into PermissionError on unix conda-forge/conda-smithy#1542

Open

h-vetinari added 8 commits January 15, 2022 09:05

add gpu builds; turn into multi-output recipe

558cc94

add scipy as host-dep

a1f1908

manual addition of GPU migrators

a8e1e66

use setup.py for gpu builds directly

7486bb3

skip CUDA 10.2 on windows

b15fe00

delete pyproject.toml for osx-arm

857dea3

DEBUG: store build artifacts on azure

c33bea8

MNT: Re-rendered with conda-build 3.21.7, conda-smithy 3.16.1, and co…

5db874a

…nda-forge-pinning 2022.01.14.19.19.28

h-vetinari force-pushed the gpu branch from 910c4d8 to 5db874a Compare January 14, 2022 22:06

h-vetinari added 2 commits January 15, 2022 09:18

remove pyproject.toml unconditionally

ac3e20f

debug: skip test suite for GPU packages to get installable artefacts

c6560df

h-vetinari closed this Feb 4, 2022

h-vetinari reopened this Feb 4, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Add GPU builds #21

WIP: Add GPU builds #21

h-vetinari commented Apr 10, 2021 •

edited

Loading

conda-forge-linter commented Apr 10, 2021

mcg1969 commented Apr 10, 2021

h-vetinari commented Apr 10, 2021

h-vetinari commented Apr 10, 2021 •

edited

Loading

kkraus14 commented Apr 12, 2021

h-vetinari commented Apr 12, 2021

xhochy commented Apr 15, 2021

h-vetinari commented Apr 15, 2021

xhochy Apr 15, 2021

xhochy Apr 15, 2021

xhochy Apr 15, 2021

h-vetinari commented Apr 21, 2021

h-vetinari left a comment

h-vetinari commented May 7, 2021

jaimergp commented May 7, 2021

h-vetinari commented Aug 18, 2021

conda-forge-linter commented Aug 18, 2021

conda-forge-linter commented Aug 18, 2021

h-vetinari commented Aug 18, 2021

h-vetinari commented Nov 16, 2021

	string: "py{{ CONDA_PY }}cuda{{ cuda_compiler_version\|replace(".", "") }}h{{ PKG_HASH }}_{{ number }}_{{ scs_proc_type }}" # [cuda_compiler_version != "None"]
	string: "py{{ CONDA_PY }}cuda{{ cuda_compiler_version\|replace(".", "") }}h{{ PKG_HASH }}_{{ PKG_BUILDNUM }}_{{ scs_proc_type }}" # [cuda_compiler_version != "None"]

	string: "py{{ CONDA_PY }}h{{ PKG_HASH }}_{{ number }}_{{ scs_proc_type }}" # [cuda_compiler_version == "None"]
	string: "py{{ CONDA_PY }}h{{ PKG_HASH }}_{{ PKG_BUILDNUM }}_{{ scs_proc_type }}" # [cuda_compiler_version == "None"]

WIP: Add GPU builds #21

Are you sure you want to change the base?

WIP: Add GPU builds #21

Conversation

h-vetinari commented Apr 10, 2021 • edited Loading

conda-forge-linter commented Apr 10, 2021

mcg1969 commented Apr 10, 2021

h-vetinari commented Apr 10, 2021

h-vetinari commented Apr 10, 2021 • edited Loading

kkraus14 commented Apr 12, 2021

h-vetinari commented Apr 12, 2021

xhochy commented Apr 15, 2021

h-vetinari commented Apr 15, 2021

xhochy Apr 15, 2021

Choose a reason for hiding this comment

xhochy Apr 15, 2021

Choose a reason for hiding this comment

xhochy Apr 15, 2021

Choose a reason for hiding this comment

h-vetinari commented Apr 21, 2021

h-vetinari left a comment

Choose a reason for hiding this comment

h-vetinari commented May 7, 2021

jaimergp commented May 7, 2021

h-vetinari commented Aug 18, 2021

conda-forge-linter commented Aug 18, 2021

conda-forge-linter commented Aug 18, 2021

h-vetinari commented Aug 18, 2021

h-vetinari commented Nov 16, 2021

h-vetinari commented Apr 10, 2021 •

edited

Loading

h-vetinari commented Apr 10, 2021 •

edited

Loading