GH-37621: [Packaging][Conda] Sync conda recipes with feedstocks #37624

h-vetinari · 2023-09-08T08:12:40Z

Syncing after the release of 13.0.0 + a couple of migrations (state as of conda-forge/arrow-cpp-feedstock#1168 & conda-forge/r-arrow-feedstock#68)

Relevant updates:

we're not building twice for different protobuf versions anymore
new abseil version (fixes [Python][FlightRPC] Tests segfault on OSX in conda-forge #36908)
we've finally upgraded the aws-sdk to 1.11
the default R versions (on unix) are now 4.2 & 4.3.

Also some further hardening of the activation scripts & clean-ups for dependencies & test skips.

Closes: Sync conda-forge recipe #37621

github-actions · 2023-09-08T08:13:07Z

⚠️ GitHub issue #37621 has been automatically assigned in GitHub to PR creator.

h-vetinari · 2023-09-08T08:14:10Z

@github-actions crossbow submit -g conda

github-actions · 2023-09-08T08:17:08Z

Revision: c04c157

Submitted crossbow builds: ursacomputing/crossbow @ actions-630421cd79

Task	Status
conda-clean
conda-linux-aarch64-cpu-py3
conda-linux-aarch64-cpu-r42
conda-linux-aarch64-cpu-r43
conda-linux-aarch64-cuda-py3
conda-linux-ppc64le-cpu-py3
conda-linux-ppc64le-cuda-py3
conda-linux-x64-cpu-py3
conda-linux-x64-cpu-r42
conda-linux-x64-cpu-r43
conda-linux-x64-cuda-py3
conda-osx-arm64-cpu-py3
conda-osx-arm64-cpu-r42
conda-osx-arm64-cpu-r43
conda-osx-x64-cpu-py3
conda-osx-x64-cpu-r42
conda-osx-x64-cpu-r43
conda-win-x64-cpu-py3
conda-win-x64-cpu-r41
conda-win-x64-cuda-py3

dev/tasks/conda-recipes/.ci_support/linux_ppc64le_cuda_compiler_version11.2.yaml

h-vetinari · 2023-09-08T09:36:01Z

Something is not working out with the CUDA cross-compilation here:

[82/139] Linking CXX shared module release/_cuda.cpython-310-powerpc64le-linux-gnu.so
FAILED: release/_cuda.cpython-310-powerpc64le-linux-gnu.so 
: && $BUILD_PREFIX/bin/powerpc64le-conda-linux-gnu-c++ -fPIC -Wno-noexcept-type  -Wall -fno-semantic-interposition -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -mcpu=power8 -mtune=power8 -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O3 -pipe -isystem $PREFIX/include -fdebug-prefix-map=/build/apache-arrow_1694161537567/work=/usr/local/src/conda/pyarrow-14.0.0.dev124 -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix -isystem /usr/local/cuda/targets/ppc64le-linux/include -fdiagnostics-color=always  -fno-omit-frame-pointer -Wno-unused-variable -Wno-maybe-uninitialized -O3 -DNDEBUG -O2 -ftree-vectorize  -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--allow-shlib-undefined -Wl,-rpath,$PREFIX/lib -Wl,-rpath-link,$PREFIX/lib -L$PREFIX/lib -shared  -o release/_cuda.cpython-310-powerpc64le-linux-gnu.so CMakeFiles/_cuda.dir/_cuda.cpp.o  -Wl,-rpath,"\$ORIGIN"  release/libarrow_python.so  $PREFIX/lib/libarrow_cuda.so.1400.0.0  $PREFIX/lib/libarrow_dataset.so.1400.0.0  $PREFIX/lib/libarrow_acero.so.1400.0.0  $PREFIX/lib/libparquet.so.1400.0.0  $PREFIX/lib/libarrow.so.1400.0.0  /usr/local/cuda-11.2/targets/x86_64-linux/lib/stubs/libcuda.so && :
$BUILD_PREFIX/powerpc64le-conda-linux-gnu/bin/ld: /usr/local/cuda-11.2/targets/x86_64-linux/lib/stubs/libcuda.so: error adding symbols: file in wrong format
collect2: error: ld returned 1 exit status

It does work in conda-forge, and CUDA on aarch/ppc is a bit more of a niche setup, so I guess we could also drop it here. For completeness, I looked at the diff for the build scripts, but I couldn't determine something that would touch upon where the CUDA_HOME would point -- the error is pretty clearly that we're not pointing to the ppc version of libcuda.so:

$BUILD_PREFIX/powerpc64le-conda-linux-gnu/bin/ld: /usr/local/cuda-11.2/targets/x86_64-linux/lib/stubs/libcuda.so: error adding symbols: file in wrong format

h-vetinari · 2023-09-08T13:24:03Z

Is it intended that gandiva now has a run-time dependence on libllvm?

    import pyarrow.gandiva
ImportError: libLLVM-15.so: cannot open shared object file: No such file or directory

Adding the respective host-dependence (+ respective run-export) is not hard, I'm just double-checking that this is intentional.

CC @pitrou @kou @raulcd @jorisvandenbossche

PS. This is a rare case where windows looks better than unix, only one test failure:

=========================== short test summary info ===========================
FAILED tests/parquet/test_basic.py::test_fastparquet_cross_compatibility - As...
= 1 failed, 7154 passed, 355 skipped, 12 deselected, 22 xfailed, 2 xpassed, 8 warnings in 302.94s (0:05:02) =

kou · 2023-09-08T21:49:20Z

Yes. It's caused by #37412.
We use libLLVM.so by default on non-Windows. If you don't like it you can add -DARROW_LLVM_USE_SHARED=OFF to use static LLVM libraries.
(I think that libLLVM.so is better to reduce duplication.)

h-vetinari · 2023-09-12T23:10:33Z

It does work in conda-forge, and CUDA on aarch/ppc is a bit more of a niche setup, so I guess we could also drop it here.

I started debugging this PR on conda-forge infrastructure in conda-forge/arrow-cpp-feedstock#1170, and it turns out that the problem hits us there as well. I'm not sure what changed since 13.0, but it seems something is now overriding (or not respecting) our CUDA_HOME, which is necessary to make cross-compilation work with CUDA...

kou · 2023-09-13T00:29:41Z

Could you try -DCUDAToolkit_ROOT=${CUDA_HOME} instead of -DCUDA_TOOLKIT_ROOT_DIR=${CUDA_HOME}?

CUDA_TOOLKIT_ROOT_DIR is for old FindCUDA: https://cmake.org/cmake/help/latest/module/FindCUDA.html
We use FindCUDAToolkit for new CMake: https://cmake.org/cmake/help/latest/module/FindCUDAToolkit.html

h-vetinari · 2023-09-13T00:44:46Z

Thanks for the quick response, that sounds like a very promising candidate! Rebased & retriggered conda-forge/arrow-cpp-feedstock#1170. Will sync back to this PR if passing.

python/pyarrow/tests/test_fs.py

h-vetinari · 2023-09-17T22:27:10Z

dev/tasks/conda-recipes/arrow-cpp/meta.yaml

+        # currently broken
+        {% set tests_to_skip = tests_to_skip + " or test_fastparquet_cross_compatibility" %}


Haven't raised an issue for this yet, but this is consistently failing. Not sure what's the difference between our CI and the one here.

Do you know a CI build log that shows the error for this? (the last ones will have this already skipped)
(to have an idea what is going on here)

_____________________ test_fastparquet_cross_compatibility _____________________ tempdir = PosixPath('/tmp/pytest-of-conda/pytest-0/test_fastparquet_cross_compati0') @pytest.mark.pandas @pytest.mark.fastparquet @pytest.mark.filterwarnings("ignore:RangeIndex:FutureWarning") @pytest.mark.filterwarnings("ignore:tostring:DeprecationWarning:fastparquet") def test_fastparquet_cross_compatibility(tempdir): fp = pytest.importorskip('fastparquet') df = pd.DataFrame( { "a": list("abc"), "b": list(range(1, 4)), "c": np.arange(4.0, 7.0, dtype="float64"), "d": [True, False, True], "e": pd.date_range("20130101", periods=3), "f": pd.Categorical(["a", "b", "a"]), # fastparquet writes list as BYTE_ARRAY JSON, so no roundtrip # "g": [[1, 2], None, [1, 2, 3]], } ) table = pa.table(df) # Arrow -> fastparquet file_arrow = str(tempdir / "cross_compat_arrow.parquet") pq.write_table(table, file_arrow, compression=None) fp_file = fp.ParquetFile(file_arrow) df_fp = fp_file.to_pandas() > tm.assert_frame_equal(df, df_fp) pyarrow/tests/parquet/test_basic.py:741: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ testing.pyx:55: in pandas._libs.testing.assert_almost_equal ??? _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ??? E AssertionError: DataFrame.iloc[:, 3] (column name="d") are different E E DataFrame.iloc[:, 3] (column name="d") values are different (66.66667 %) E [index]: [0, 1, 2] E [left]: [True, False, True] E [right]: [False, False, False] E At positional index 0, first diff: True != False testing.pyx:173: AssertionError

Searching for fastparquet in our code base, I assume that we actually don't have any CI build that includes it ... So we are just not running that test anywhere, whoops.

Now, the cross-compatibility is also tested in the pandas test suite, so maybe it's not too important to have it here as well. Anyway, opened an issue at #37853

Although fastparquet reading a column of boolens wrongly for a pyarrow-written file seems a quite serious issue ..
Now, similar data is used in the pandas test suite, where this is passing.

Thanks for raising the issue. I agree it does looks potentially serious, but I hadn't looked too closely because it was just one test. The first thing would be to add testing here in CI. I think this PR is fine as is (I try to keep the test skips to an absolute minimum and remove them whenever possible; that then gets picked up by the next recipe sync anyway)

I took a closer look at it (could reproduce this locally), and so actually it was good to do so, as this is a problem in fastparquet reading parquet files generated by latest pyarrow, see details at #37853 (comment)

But yes, so skipping for now is perfectly fine.

dev/tasks/conda-recipes/arrow-cpp/meta.yaml

h-vetinari · 2023-09-17T22:31:26Z

@github-actions crossbow submit -g conda

github-actions · 2023-09-17T22:34:33Z

Revision: d7af614

Submitted crossbow builds: ursacomputing/crossbow @ actions-20144be8d4

Task	Status
conda-clean
conda-linux-aarch64-cpu-py3
conda-linux-aarch64-cpu-r42
conda-linux-aarch64-cpu-r43
conda-linux-aarch64-cuda-py3
conda-linux-ppc64le-cpu-py3
conda-linux-ppc64le-cuda-py3
conda-linux-x64-cpu-py3
conda-linux-x64-cpu-r42
conda-linux-x64-cpu-r43
conda-linux-x64-cuda-py3
conda-osx-arm64-cpu-py3
conda-osx-arm64-cpu-r42
conda-osx-arm64-cpu-r43
conda-osx-x64-cpu-py3
conda-osx-x64-cpu-r42
conda-osx-x64-cpu-r43
conda-win-x64-cpu-py3
conda-win-x64-cpu-r41
conda-win-x64-cuda-py3

github-actions · 2023-09-21T10:41:45Z

Revision: 53a4b70

Submitted crossbow builds: ursacomputing/crossbow @ actions-80a2017be8

Task	Status
conda-clean
conda-linux-aarch64-cpu-py3
conda-linux-aarch64-cpu-r42
conda-linux-aarch64-cpu-r43
conda-linux-aarch64-cuda-py3
conda-linux-ppc64le-cpu-py3
conda-linux-ppc64le-cuda-py3
conda-linux-x64-cpu-py3
conda-linux-x64-cpu-r42
conda-linux-x64-cpu-r43
conda-linux-x64-cuda-py3
conda-osx-arm64-cpu-py3
conda-osx-arm64-cpu-r42
conda-osx-arm64-cpu-r43
conda-osx-x64-cpu-py3
conda-osx-x64-cpu-r42
conda-osx-x64-cpu-r43
conda-win-x64-cpu-py3
conda-win-x64-cpu-r41
conda-win-x64-cuda-py3

h-vetinari · 2023-09-25T02:48:01Z

Failures:

linux-64

Failed during artefact upload -- irrelevant

linux-aarch64

Connection lost to agent -- irrelevant

osx-64

=================================== FAILURES ===================================
_____________ test_debug_memory_pool_disabled[system_memory_pool] ______________

pool_factory = <built-in function system_memory_pool>

    @pytest.mark.parametrize('pool_factory', supported_factories())
    def test_debug_memory_pool_disabled(pool_factory):
        res = run_debug_memory_pool(pool_factory.__name__, "")
        # The subprocess either returned successfully or was killed by a signal
        # (due to writing out of bounds), depending on the underlying allocator.
        if os.name == "posix":
            assert res.returncode <= 0
        else:
            res.check_returncode()
>       assert res.stderr == ""
E       AssertionError: assert 'libc++abi: P...ion called!\n' == ''
E         + libc++abi: Pure virtual function called!

pyarrow/tests/test_memory.py:255: AssertionError
----------------------------- Captured stderr call -----------------------------
libc++abi: Pure virtual function called!

I can skip the respective test on osx if that's what people prefer. Otherwise this PR should be ready.

dev/tasks/conda-recipes/arrow-cpp/meta.yaml

h-vetinari · 2023-09-28T10:54:27Z

Anything else to do here?

jorisvandenbossche · 2023-09-28T11:14:14Z

I don't think so, thanks for the ping, and for all the work here!

conbench-apache-arrow · 2023-09-28T14:26:53Z

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit c9674bc.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.

…apache#37624) Syncing after the release of 13.0.0 + a couple of migrations (state as of conda-forge/arrow-cpp-feedstock#1168 & conda-forge/r-arrow-feedstock#68) Relevant updates: * we're not building twice for different protobuf versions anymore * new abseil version (fixes apache#36908) * we've finally upgraded the aws-sdk to 1.11 * the default R versions (on unix) are now 4.2 & 4.3. Also some further hardening of the activation scripts & clean-ups for dependencies & test skips. * Closes: apache#37621 Lead-authored-by: H. Vetinari <h.vetinari@gmx.com> Co-authored-by: h-vetinari <h.vetinari@gmx.com> Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

h-vetinari added 3 commits September 8, 2023 19:11

update arrow-cpp recipe

a19b8a7

update arrow-cpp ci_support files

af4c65d

update r-arrow's ci_support files

c04c157

h-vetinari requested review from assignUser, kou and raulcd as code owners September 8, 2023 08:12

github-actions bot added the awaiting review Awaiting review label Sep 8, 2023

raulcd reviewed Sep 8, 2023

View reviewed changes

dev/tasks/conda-recipes/.ci_support/linux_ppc64le_cuda_compiler_version11.2.yaml Show resolved Hide resolved

github-actions bot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Sep 8, 2023

h-vetinari mentioned this pull request Sep 8, 2023

WIP: arrow 14 conda-forge/arrow-cpp-feedstock#1170

Closed

h-vetinari mentioned this pull request Sep 13, 2023

[Python][FlightRPC] Tests segfault on OSX in conda-forge #36908

Closed

h-vetinari added 3 commits September 18, 2023 09:18

fixture teardown should not fail test

f0b2133

sync recipe

3584b5d

update ci_support files (new abseil)

d7af614

github-actions bot added Component: Python awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Sep 17, 2023

h-vetinari commented Sep 17, 2023

View reviewed changes

h-vetinari commented Sep 25, 2023

View reviewed changes

dev/tasks/conda-recipes/arrow-cpp/meta.yaml Outdated Show resolved Hide resolved

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Sep 25, 2023

Remove test skip that's not necessary anymore.

7c0b94f

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Sep 25, 2023

h-vetinari commented Sep 25, 2023

View reviewed changes

dev/tasks/conda-recipes/arrow-cpp/meta.yaml Outdated Show resolved Hide resolved

Add test skip for risky test on osx

68bf65c

jorisvandenbossche mentioned this pull request Sep 25, 2023

[Python][CI] Tests involving fastparquet are never run #37853

Open

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Sep 25, 2023

jorisvandenbossche approved these changes Sep 28, 2023

View reviewed changes

github-actions bot added awaiting merge Awaiting merge and removed awaiting changes Awaiting changes labels Sep 28, 2023

jorisvandenbossche merged commit c9674bc into apache:main Sep 28, 2023
8 checks passed

jorisvandenbossche removed the awaiting merge Awaiting merge label Sep 28, 2023

h-vetinari deleted the conda_sync branch September 28, 2023 11:58

jorisvandenbossche mentioned this pull request May 17, 2024

Fix run exports to restrict the minor version conda-forge/arrow-cpp-feedstock#1409

Merged

5 tasks

This was referenced Jun 11, 2024

[CI][Packaging] Re-synchronize conda_recipes with arrow-cpp-feedstock #42111

Open

GH-42111: [CI][Packaging] Re-synchronize conda_recipes with arrow-cpp-feedstock #42114

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-37621: [Packaging][Conda] Sync conda recipes with feedstocks #37624

GH-37621: [Packaging][Conda] Sync conda recipes with feedstocks #37624

h-vetinari commented Sep 8, 2023 •

edited

Loading

github-actions bot commented Sep 8, 2023

h-vetinari commented Sep 8, 2023

github-actions bot commented Sep 8, 2023

h-vetinari commented Sep 8, 2023

h-vetinari commented Sep 8, 2023

kou commented Sep 8, 2023

h-vetinari commented Sep 12, 2023

kou commented Sep 13, 2023

h-vetinari commented Sep 13, 2023

h-vetinari Sep 17, 2023

jorisvandenbossche Sep 21, 2023

h-vetinari Sep 21, 2023

jorisvandenbossche Sep 25, 2023

jorisvandenbossche Sep 25, 2023 •

edited

Loading

h-vetinari Sep 25, 2023

jorisvandenbossche Sep 25, 2023

h-vetinari commented Sep 17, 2023

github-actions bot commented Sep 17, 2023

github-actions bot commented Sep 21, 2023

h-vetinari commented Sep 25, 2023

h-vetinari commented Sep 28, 2023

jorisvandenbossche commented Sep 28, 2023

conbench-apache-arrow bot commented Sep 28, 2023

		# currently broken
		{% set tests_to_skip = tests_to_skip + " or test_fastparquet_cross_compatibility" %}

GH-37621: [Packaging][Conda] Sync conda recipes with feedstocks #37624

GH-37621: [Packaging][Conda] Sync conda recipes with feedstocks #37624

Conversation

h-vetinari commented Sep 8, 2023 • edited Loading

github-actions bot commented Sep 8, 2023

h-vetinari commented Sep 8, 2023

github-actions bot commented Sep 8, 2023

h-vetinari commented Sep 8, 2023

h-vetinari commented Sep 8, 2023

kou commented Sep 8, 2023

h-vetinari commented Sep 12, 2023

kou commented Sep 13, 2023

h-vetinari commented Sep 13, 2023

h-vetinari Sep 17, 2023

Choose a reason for hiding this comment

jorisvandenbossche Sep 21, 2023

Choose a reason for hiding this comment

h-vetinari Sep 21, 2023

Choose a reason for hiding this comment

jorisvandenbossche Sep 25, 2023

Choose a reason for hiding this comment

jorisvandenbossche Sep 25, 2023 • edited Loading

Choose a reason for hiding this comment

h-vetinari Sep 25, 2023

Choose a reason for hiding this comment

jorisvandenbossche Sep 25, 2023

Choose a reason for hiding this comment

h-vetinari commented Sep 17, 2023

github-actions bot commented Sep 17, 2023

github-actions bot commented Sep 21, 2023

h-vetinari commented Sep 25, 2023

linux-64

linux-aarch64

osx-64

h-vetinari commented Sep 28, 2023

jorisvandenbossche commented Sep 28, 2023

conbench-apache-arrow bot commented Sep 28, 2023

h-vetinari commented Sep 8, 2023 •

edited

Loading

jorisvandenbossche Sep 25, 2023 •

edited

Loading