GH-14993: [CI][Conda] Fix missing RECIPE_ROOT variable now expected by conda build #15014

h-vetinari · 2022-12-18T02:58:21Z

Closes: [CI][Conda] conda-linux-aarch64-* are failing #14993

github-actions · 2022-12-18T02:58:40Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

In the case of old issues on JIRA the title also supports:

ARROW-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:

github-actions · 2022-12-18T02:59:26Z

Closes: [CI][Conda] conda-linux-aarch64-* are failing #14993

github-actions · 2022-12-18T02:59:28Z

⚠️ GitHub issue #14993 has been automatically assigned in GitHub to PR creator.

h-vetinari · 2022-12-18T03:00:40Z

@github-actions crossbow submit -g conda

github-actions · 2022-12-18T03:04:23Z

Revision: f3b77cc

Submitted crossbow builds: ursacomputing/crossbow @ actions-ccb0a7fe84

Task	Status
conda-clean
conda-linux-aarch64-cpu-py3
conda-linux-aarch64-cpu-r41
conda-linux-aarch64-cpu-r42
conda-linux-ppc64le-cpu-py3
conda-linux-x64-cpu-py3
conda-linux-x64-cpu-r41
conda-linux-x64-cpu-r42
conda-linux-x64-cuda-py3
conda-osx-arm64-cpu-py3
conda-osx-arm64-cpu-r41
conda-osx-arm64-cpu-r42
conda-osx-x64-cpu-py3
conda-osx-x64-cpu-r41
conda-osx-x64-cpu-r42
conda-win-x64-cpu-py3
conda-win-x64-cpu-r41
conda-win-x64-cuda-py3

h-vetinari · 2022-12-18T06:45:12Z

So, the R-builds for aarch are still failing with:

> library('arrow')
Error: package or namespace load failed for ‘arrow’ in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '[...]lib/R/library/arrow/libs/arrow.so':
  [...]/lib/R/library/arrow/libs/arrow.so: undefined symbol: _ZTIN5arrow17RecordBatchReaderE
Execution halted

I don't understand why this is looking for arrow.so instead of libarrow.so - the recipe hasn't changed.

CC @xhochy

h-vetinari · 2022-12-18T08:01:28Z

I don't understand why this is looking for arrow.so instead of libarrow.so - the recipe hasn't changed.

Though on the other hand, the build.sh for r-arrow is calling straight into the top-level r/-folder:

${R} CMD INSTALL --build r/. ${R_ARGS}

so something broke there in the last couple of days. For example, dca8c07 changed r/configure quite a bit, which could explain why now the libs are being built differently or not linked correctly anymore.

CC @thisisnic

xhochy · 2022-12-18T16:31:45Z

I don't understand why this is looking for arrow.so instead of libarrow.so - the recipe hasn't changed.

This is OK, arrow.so is the R module. libarrow.so is the pure C++ shared library.

xhochy · 2022-12-18T16:33:27Z

@h-vetinari This looks problematic:

2022-12-18T05:51:13.5618933Z *** Found local C++ source: '../cpp'
2022-12-18T05:51:13.5619835Z *** Building libarrow from source
2022-12-18T05:51:13.5620314Z     For build options and troubleshooting, see the install guide:
2022-12-18T05:51:13.5621188Z     https://arrow.apache.org/docs/r/articles/install.html
2022-12-18T05:51:13.8434741Z **** cmake
2022-12-18T05:51:18.7191251Z **** arrow  
2022-12-18T05:51:20.8624297Z **** Error building Arrow C++. Re-run with ARROW_R_DEV=true for debug information. 
2022-12-18T05:51:21.5582869Z PKG_CFLAGS=  -std=c++17
2022-12-18T05:51:21.5589264Z PKG_LIBS=

xhochy · 2022-12-18T16:40:30Z

@github-actions crossbow submit conda-linux-aarch64-cpu-r41

github-actions · 2022-12-18T16:43:42Z

Revision: 6efeeec

Submitted crossbow builds: ursacomputing/crossbow @ actions-8a6b714265

Task	Status
conda-linux-aarch64-cpu-r41

kou · 2022-12-19T03:01:41Z

For example, dca8c07 changed r/configure quite a bit, which could explain why now the libs are being built differently or not linked correctly anymore.

Oh, the change was by me. I'll look into this.

kou · 2022-12-19T03:31:17Z

@github-actions crossbow submit conda-linux-aarch64-cpu-r41

github-actions · 2022-12-19T03:33:56Z

Revision: e04c9dc

Submitted crossbow builds: ursacomputing/crossbow @ actions-297ce68126

Task	Status
conda-linux-aarch64-cpu-r41

kou · 2022-12-19T03:36:09Z

@github-actions crossbow submit conda-linux-aarch64-cpu-r41

github-actions · 2022-12-19T03:38:33Z

Revision: b26892d

Submitted crossbow builds: ursacomputing/crossbow @ actions-5a2723bac4

Task	Status
conda-linux-aarch64-cpu-r41

For example, Apache Arrow C++ is installed in system directory.

kou · 2022-12-19T06:42:59Z

@github-actions crossbow submit conda-linux-aarch64-cpu-r41

github-actions · 2022-12-19T06:45:25Z

Revision: 3b9c72d

Submitted crossbow builds: ursacomputing/crossbow @ actions-5e596172d1

Task	Status
conda-linux-aarch64-cpu-r41

h-vetinari · 2022-12-19T09:39:29Z

Success! 🥳

Well done @kou & @xhochy

xhochy · 2022-12-19T10:20:38Z

dev/tasks/conda-recipes/r-arrow/build.sh


 if [[ "${target_platform}" == osx-* ]]; then
    # See https://conda-forge.org/docs/maintainer/knowledge_base.html#newer-c-features-with-old-sdk
    export ARROW_R_CXXFLAGS="${ARROW_R_CXXFLAGS} -D_LIBCPP_DISABLE_AVAILABILITY"
 fi

-export PKG_CONFIG_PATH="${PREFIX}/lib/pkgconfig${PKG_CONFIG_PATH:+:${PKG_CONFIG_PATH}}"
+export PKG_CONFIG_PATH="${BUILD_PREFIX}/lib/pkgconfig${PKG_CONFIG_PATH:+:${PKG_CONFIG_PATH}}"


This doesn't look right. BUILD_PREFIX is the build system, not host. It has been failing in the cross-compiling case and this change makes me suspicious whether we then link to the wrong libraries.

Ah, sorry. This is not related to fix the R problem.
We may be able remove this line entirely. I'll try it.

github-actions · 2022-12-19T20:57:39Z

⚠️ GitHub issue #14993 has been automatically assigned in GitHub to PR creator.

kou · 2022-12-19T21:01:58Z

@github-actions crossbow submit conda-linux--cpu-r -g r

github-actions · 2022-12-19T21:07:07Z

Revision: f018c90

Submitted crossbow builds: ursacomputing/crossbow @ actions-5001db49ca

Task	Status
conda-linux-aarch64-cpu-r41
conda-linux-aarch64-cpu-r42
conda-linux-x64-cpu-r41
conda-linux-x64-cpu-r42
conda-osx-arm64-cpu-r41
conda-osx-arm64-cpu-r42
conda-osx-x64-cpu-r41
conda-osx-x64-cpu-r42
conda-win-x64-cpu-r41
homebrew-r-autobrew
homebrew-r-brew
r-binary-packages
test-fedora-r-clang-sanitizer
test-r-arrow-backwards-compatibility
test-r-depsource-bundled
test-r-depsource-system
test-r-dev-duckdb
test-r-devdocs
test-r-gcc-11
test-r-gcc-12
test-r-install-local
test-r-install-local-minsizerel
test-r-library-r-base-latest
test-r-linux-as-cran
test-r-linux-rchk
test-r-linux-valgrind
test-r-minimal-build
test-r-offline-maximal
test-r-offline-minimal
test-r-rhub-debian-gcc-devel-lto-latest
test-r-rhub-debian-gcc-release-custom-ccache
test-r-rhub-ubuntu-gcc-release-latest
test-r-rstudio-r-base-4.1-opensuse153
test-r-rstudio-r-base-4.2-centos7-devtoolset-8
test-r-rstudio-r-base-4.2-focal
test-r-ubuntu-22.04
test-r-versions
test-ubuntu-18.04-r-sanitizer

h-vetinari · 2022-12-19T23:27:49Z

crossbow submit conda-linux-_-cpu-r_ -g r

is _ the placeholder here? And what's the difference between before -g and after?

Extrapolating from crossbow submit -g conda, I would have naïvely expected to be able to run all conda-r builds as

crossbow submit - g conda-*-r

kou · 2022-12-20T00:07:10Z

is _ the placeholder here?

_ isn't a placeholder. The real crossbow command what I used is @github-actions crossbow submit conda-linux-*-cpu-r* -g r. * is a glob pattern. If the command is processed as a Markdown text, *...* is replaced with <em>...</em>. This is the trick. :-)

And what's the difference between before -g and after?

Without -g: Find tasks from tasks: in dev/tasks/tasks.yml

With -g: Find groups from groups: in dev/tasks/tasks.yml`. Found groups are expanded to tasks.

Extrapolating from crossbow submit -g conda, I would have naïvely expected to be able to run all conda-r builds as

crossbow submit - g conda-*-r

We can use @github-actions crossbow submit conda-*-r for them.

h-vetinari · 2022-12-20T02:34:19Z

@kou, I think this would be good to merge (as all the conda builds now pass)? I have other changes lined up, but those would be better in a separate PR IMO.

kou · 2022-12-20T03:05:51Z

OK. I merge this.

ursabot · 2022-12-20T10:01:43Z

Benchmark runs are scheduled for baseline = 23a7a15 and contender = 4e9b65a. 4e9b65a is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Finished ⬇️1.42% ⬆️0.07%] test-mac-arm
[Finished ⬇️0.0% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.14% ⬆️0.03%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 4e9b65a4 ec2-t3-xlarge-us-east-2
[Finished] 4e9b65a4 test-mac-arm
[Finished] 4e9b65a4 ursa-i9-9960x
[Finished] 4e9b65a4 ursa-thinkcentre-m75q
[Finished] 23a7a159 ec2-t3-xlarge-us-east-2
[Finished] 23a7a159 test-mac-arm
[Finished] 23a7a159 ursa-i9-9960x
[Finished] 23a7a159 ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

fix missing RECIPE_ROOT variable now expected by conda build

f3b77cc

h-vetinari changed the title ~~fix missing RECIPE_ROOT variable now expected by conda build~~ GH-14993: fix missing RECIPE_ROOT variable now expected by conda build Dec 18, 2022

h-vetinari mentioned this pull request Dec 18, 2022

[CI][Conda] conda-linux-aarch64-* are failing #14993

Closed

Don't build libarrow from source

6efeeec

Debug

e04c9dc

github-actions bot added the Component: R label Dec 19, 2022

Use BUILD_PREFIX

b26892d

"pkg-config --cflags" may be empty

3b9c72d

For example, Apache Arrow C++ is installed in system directory.

This was referenced Dec 19, 2022

BUG: segmentation faults in the presence of sparse optional dependency (within conda builds) #15018

Closed

ARROW-17838: [Python] Unify CMakeLists.txt in python/ #14925

Merged

xhochy reviewed Dec 19, 2022

View reviewed changes

kou changed the title ~~GH-14993: fix missing RECIPE_ROOT variable now expected by conda build~~ GH-14993: [CI][Conda] Fix missing RECIPE_ROOT variable now expected by conda build Dec 19, 2022

Remove needless codes

f018c90

kou merged commit 4e9b65a into apache:master Dec 20, 2022

h-vetinari deleted the conda_ci branch December 20, 2022 03:26

h-vetinari mentioned this pull request Dec 20, 2022

GH-15045: [CI][conda] don't build pyarrow in r-jobs #15046

Draft

h-vetinari mentioned this pull request Jun 12, 2024

GH-42111: [CI][Packaging] Re-synchronize conda_recipes with arrow-cpp-feedstock #42114

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-14993: [CI][Conda] Fix missing RECIPE_ROOT variable now expected by conda build #15014

GH-14993: [CI][Conda] Fix missing RECIPE_ROOT variable now expected by conda build #15014

h-vetinari commented Dec 18, 2022 •

edited by github-actions bot

Loading

github-actions bot commented Dec 18, 2022

github-actions bot commented Dec 18, 2022

github-actions bot commented Dec 18, 2022

h-vetinari commented Dec 18, 2022

github-actions bot commented Dec 18, 2022

h-vetinari commented Dec 18, 2022 •

edited

Loading

h-vetinari commented Dec 18, 2022

xhochy commented Dec 18, 2022

xhochy commented Dec 18, 2022

xhochy commented Dec 18, 2022

github-actions bot commented Dec 18, 2022

kou commented Dec 19, 2022

kou commented Dec 19, 2022

github-actions bot commented Dec 19, 2022

kou commented Dec 19, 2022

github-actions bot commented Dec 19, 2022

kou commented Dec 19, 2022

github-actions bot commented Dec 19, 2022

h-vetinari commented Dec 19, 2022

xhochy Dec 19, 2022

kou Dec 19, 2022

github-actions bot commented Dec 19, 2022

kou commented Dec 19, 2022

github-actions bot commented Dec 19, 2022

h-vetinari commented Dec 19, 2022

kou commented Dec 20, 2022

h-vetinari commented Dec 20, 2022

kou commented Dec 20, 2022

ursabot commented Dec 20, 2022

GH-14993: [CI][Conda] Fix missing RECIPE_ROOT variable now expected by conda build #15014

GH-14993: [CI][Conda] Fix missing RECIPE_ROOT variable now expected by conda build #15014

Conversation

h-vetinari commented Dec 18, 2022 • edited by github-actions bot Loading

github-actions bot commented Dec 18, 2022

github-actions bot commented Dec 18, 2022

github-actions bot commented Dec 18, 2022

h-vetinari commented Dec 18, 2022

github-actions bot commented Dec 18, 2022

h-vetinari commented Dec 18, 2022 • edited Loading

h-vetinari commented Dec 18, 2022

xhochy commented Dec 18, 2022

xhochy commented Dec 18, 2022

xhochy commented Dec 18, 2022

github-actions bot commented Dec 18, 2022

kou commented Dec 19, 2022

kou commented Dec 19, 2022

github-actions bot commented Dec 19, 2022

kou commented Dec 19, 2022

github-actions bot commented Dec 19, 2022

kou commented Dec 19, 2022

github-actions bot commented Dec 19, 2022

h-vetinari commented Dec 19, 2022

xhochy Dec 19, 2022

Choose a reason for hiding this comment

kou Dec 19, 2022

Choose a reason for hiding this comment

github-actions bot commented Dec 19, 2022

kou commented Dec 19, 2022

github-actions bot commented Dec 19, 2022

h-vetinari commented Dec 19, 2022

kou commented Dec 20, 2022

h-vetinari commented Dec 20, 2022

kou commented Dec 20, 2022

ursabot commented Dec 20, 2022

h-vetinari commented Dec 18, 2022 •

edited by github-actions bot

Loading

h-vetinari commented Dec 18, 2022 •

edited

Loading