Add CUDA build #101

pearu · 2020-03-25T11:15:27Z

This PR requires conda-forge/arrow-cpp-feedstock#125

Checklist

Used a fork of the feedstock to propose changes
Bumped the build number (if the version is unchanged)
Reset the build number to 0 (if the version changed)
Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
Ensured the license file is being packaged.

conda-forge-linter · 2020-03-25T11:15:34Z

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipe) and found some lint.

Here's what I've got...

For recipe:

Failed to even lint the recipe, probably because of a conda-smithy bug 😢. This likely indicates a problem in your meta.yaml, though. To get a traceback to help figure out what's going on, install conda-smithy and run conda smithy recipe-lint . from the recipe directory.

conda-forge-linter · 2020-03-25T11:15:36Z

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipe) and found some lint.

Here's what I've got...

For recipe:

The recipe must have some tests.

conda-forge-linter · 2020-03-25T13:00:06Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

…onda-forge-pinning 2020.04.17.01.00.32

…nda-forge-pinning 2020.04.27.17.45.41

…ilds

pearu · 2020-05-15T16:44:46Z

recipe/meta.yaml

 host:
 # directly pin boost-cpp as we also seem to directly include boost symbols
 # in the Python modules.
- - arrow-cpp {{ version }}
+ - arrow-cpp ={{ version }}=*{{ build_ext }}


@h-vetinari , arrow-cpp-proc in run_constraint ought to handle the matching with build_ext.

I know, but the same information is in the build-string of arrow-cpp, making it wholly unnecessary to depend on the version (and build string) of arrow-cpp-proc. If you don't like it, then feel free to revert - I consider it cleaner/simpler.

This notation is probably the preferred one for conda.

Suggested change

- arrow-cpp ={{ version }}=*{{ build_ext }}

- arrow-cpp {{ version }} *{{ build_ext }}

@pearu The run_constraint will only be applied if arrow-cpp-proc is installed. Currently it seems to me that when you don't have arrow-cpp-proc installed, you could install the cpu version of arrow-cpp while installing the CUDA version of pyarrow.

This notation is probably the preferred one for conda.

Personally I dislike significant whitespace for this - I think it's very hard to read.

Currently it seems to me that when you don't have arrow-cpp-proc installed, you could install the cpu version of arrow-cpp while installing the CUDA version of pyarrow.

That's IMO not possible because we're depending on the specific build version of arrow-cpp already.

This notation is probably the preferred one for conda.

Personally I dislike significant whitespace for this - I think it's very hard to read.

I didn't make the standard but I like to keep it :(

Currently it seems to me that when you don't have arrow-cpp-proc installed, you could install the cpu version of arrow-cpp while installing the CUDA version of pyarrow.

That's IMO not possible because we're depending on the specific build version of arrow-cpp already.

The comment was meant in regard to not having arrow-cpp ={{ version }}=*{{ build_ext }} in run. From my understanding, if that would be omitted, the mentioned situation could happen (or I'm missing another safeguard?)

The comment was meant in regard to not having arrow-cpp ={{ version }}=*{{ build_ext }} in run. From my understanding, if that would be omitted, the mentioned situation could happen (or I'm missing another safeguard?)

No you're right, but not having any requirement on arrow-cpp in run broke the build, as both @pearu and I had to find out. So I don't think it can be omitted anyway, and then why shouldn't we also add the build_ext?

I didn't make the standard but I like to keep it :(

It's not a hill I'm gonna die on, but that's IMO a bad reason to write hard-to-read recipes (and it's not like the variants are so different as to be unrecognisable).

pearu · 2020-05-15T19:43:56Z

Nothing I touched should have affected the osx/gandiva builds, but there's an error
import: 'pyarrow.gandiva'

CUDA enabled arrow-cpp 0.16 disables gandiva for ppc64le and aarch64, see https://github.com/conda-forge/arrow-cpp-feedstock/blob/0.16.x/recipe/build-arrow.sh#L11

CUDA enabled arrow-cpp 0.17, however, will enable gandiva support, see conda-forge/arrow-cpp-feedstock#137 .

However, for this PR, that is arrow-cpp 0.16 based, the gandiva support must be disabled.

pearu · 2020-05-15T20:46:35Z

Same error on linux64 (but not on aarch/ppc):

import: 'pyarrow.gandiva'
...
ImportError: libre2.so.6: cannot open shared object file: No such file or directory

The latest re2 package (v 2020.05) provides libre2.so.7 but arrow-cpp 0.16 was built against re2 package v 2020.04 that provides libre2.so.6.

pearu · 2020-05-15T23:24:38Z

This PR is blocked by conda-forge/conda-forge-ci-setup-feedstock#93 .

xhochy · 2020-05-16T08:35:01Z

Same error on linux64 (but not on aarch/ppc):
import: 'pyarrow.gandiva'
...
ImportError: libre2.so.6: cannot open shared object file: No such file or directory
The latest re2 package (v 2020.05) provides libre2.so.7 but arrow-cpp 0.16 was built against re2 package v 2020.04 that provides libre2.so.6.

Made a hotfix PR: conda-forge/admin-requests#53

recipe/meta.yaml

h-vetinari · 2020-05-18T18:20:09Z

@conda-forge-admin, please restart ci

h-vetinari · 2020-05-18T18:42:57Z

@pearu @xhochy
The GPU builds are green now 🥳
Just waiting for aarch/ppc to finish.

recipe/meta.yaml

pearu · 2020-05-20T05:35:56Z

@h-vetinari @xhochy @isuruf , are there any issues left that prevent landing this PR?

xhochy · 2020-05-20T05:46:52Z

recipe/meta.yaml

@@ -12,7 +15,11 @@ source:
 sha256: {{ checksum }}

 build:
- number: 2
+ number: {{ number }}


Suggested change

number: {{ number }}

number: 3

xhochy · 2020-05-20T05:46:58Z

recipe/meta.yaml

@@ -1,6 +1,9 @@
 {% set version = "0.16.0" %}
 {% set filename = "apache-arrow-" + version + ".tar.gz" %}
 {% set checksum = "261992de4029a1593195ff4000501503bd403146471b3168bd2cc414ad0fb7f5" %}
+{% set number = "3" %}


Suggested change

{% set number = "3" %}

h-vetinari · 2020-05-20T05:49:39Z

recipe/meta.yaml

@@ -12,7 +15,11 @@ source:
 sha256: {{ checksum }}

 build:
- number: 2
+ number: {{ number }}
+ string: "py{{ CONDA_PY }}h{{ PKG_HASH }}_{{ number }}_{{ build_ext }}"


@xhochy
The number is also necessary for the build string. While I'm the first to be against useless templating, I think having {{ number }} makes a lot of sense in this recipe.

xhochy · 2020-05-20T05:50:12Z

Ignore the suggestions, if CI passes again, we can merge.

github-actions · 2020-05-20T06:34:29Z

Hi! This is the friendly conda-forge automerge bot!

I considered the following status checks when analyzing this PR:

linter: passed
travis: passed
azure: passed

Thus the PR was passing and merged! Have a great day!

pearu requested review from cpcloud, jreback, kou, kszucs, pitrou, robertnishihara, siddharthteotia, wesm and xhochy as code owners March 25, 2020 11:15

pearu added 4 commits April 17, 2020 11:45

Enable tests

be243d6

rebase, use arrow-cpp-proc, add pearu to recipe-maintainers

424860d

rebase (2)

3929c18

MNT: Re-rendered with conda-build 3.18.11, conda-smithy 3.6.16, and c…

f785cba

…onda-forge-pinning 2020.04.17.01.00.32

pearu force-pushed the pearu/enable-cuda branch from 536e20f to f785cba Compare April 17, 2020 08:58

pearu requested a review from pcmoritz as a code owner April 17, 2020 08:58

xhochy changed the base branch from master to 0.16.x April 27, 2020 12:31

pearu added 11 commits April 28, 2020 12:23

Update

662f537

MNT: Re-rendered with conda-build 3.18.11, conda-smithy 3.7.0, and co…

d1409fa

…nda-forge-pinning 2020.04.27.17.45.41

Use build_ext on arrow-cpp

f854758

Use build_ext on arrow-cpp (2nd try)

90418f5

Fix typo.

b1c330e

Use build_ext on arrow-cpp, also in run.

e4f7f6a

Add python, cudatoolkit deps. Allow newer numpy than 1.16.

f781817

Move python to build for win. Disable plasma test for cuda enabled bu…

414f256

…ilds

Disable plasma and cuda import tests for cuda enabled builds.

780e457

Try python instead of %PYTHON% in bld.bat

c20e410

Restore %PYTHON%

ed8d21c

pearu commented May 15, 2020

View reviewed changes

pearu mentioned this pull request May 15, 2020

Add cuda support for master conda-forge/arrow-cpp-feedstock#137

Merged

5 tasks

Fix re2 issue. Don't run pyarrow unittests.

7c1174e

pearu mentioned this pull request May 15, 2020

pyarrow should be an output of arrow-cpp #107

Closed

jorisvandenbossche mentioned this pull request May 16, 2020

Gandiva module not working / wrong version of re2 conda-forge/arrow-cpp-feedstock#138

Closed

h-vetinari mentioned this pull request May 16, 2020

Avoid setting CFLAGS, CPPFLAGS, and CXXFLAGS conda-forge/cupy-feedstock#59

Merged

5 tasks

h-vetinari reviewed May 16, 2020

View reviewed changes

recipe/meta.yaml Outdated Show resolved Hide resolved

Update recipe/meta.yaml

c8f8ec4

conda-forge-linter closed this May 18, 2020

conda-forge-linter reopened this May 18, 2020

pearu commented May 18, 2020

View reviewed changes

recipe/meta.yaml Outdated Show resolved Hide resolved

remove re2; align/minimize some selectors

be4c0c8

h-vetinari force-pushed the pearu/enable-cuda branch from 6d51813 to be4c0c8 Compare May 18, 2020 19:59

xhochy closed this May 20, 2020

xhochy reopened this May 20, 2020

xhochy reviewed May 20, 2020

View reviewed changes

h-vetinari reviewed May 20, 2020

View reviewed changes

xhochy added the automerge Merge the PR when CI passes label May 20, 2020

github-actions bot merged commit 5d1ee02 into conda-forge:0.16.x May 20, 2020

h-vetinari mentioned this pull request Sep 8, 2022

Join as maintainer to take care of migrations conda-forge/arrow-cpp-feedstock#839

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add CUDA build #101

Add CUDA build #101

pearu commented Mar 25, 2020 •

edited

Loading

conda-forge-linter commented Mar 25, 2020

conda-forge-linter commented Mar 25, 2020

conda-forge-linter commented Mar 25, 2020

pearu May 15, 2020

h-vetinari May 15, 2020

xhochy May 19, 2020

xhochy May 19, 2020

h-vetinari May 19, 2020

xhochy May 19, 2020

h-vetinari May 19, 2020

pearu commented May 15, 2020

pearu commented May 15, 2020

pearu commented May 15, 2020

xhochy commented May 16, 2020

h-vetinari commented May 18, 2020

h-vetinari commented May 18, 2020

pearu commented May 20, 2020

xhochy May 20, 2020

xhochy May 20, 2020

h-vetinari May 20, 2020

xhochy commented May 20, 2020

github-actions bot commented May 20, 2020

	- arrow-cpp ={{ version }}=*{{ build_ext }}
	- arrow-cpp {{ version }} *{{ build_ext }}

Add CUDA build #101

Add CUDA build #101

Conversation

pearu commented Mar 25, 2020 • edited Loading

conda-forge-linter commented Mar 25, 2020

conda-forge-linter commented Mar 25, 2020

conda-forge-linter commented Mar 25, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pearu commented May 15, 2020

pearu commented May 15, 2020

pearu commented May 15, 2020

xhochy commented May 16, 2020

h-vetinari commented May 18, 2020

h-vetinari commented May 18, 2020

pearu commented May 20, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xhochy commented May 20, 2020

github-actions bot commented May 20, 2020

pearu commented Mar 25, 2020 •

edited

Loading