BUG: segmentation faults in the presence of sparse optional dependency (within conda builds)

[h-vetinari]

Describe the bug, including details regarding any error messages, version, and platform.

#14832 switched on the test suite within the conda-builds, which itself synced with conda-forge/arrow-cpp-feedstock#875

In the process of doing so an optional dependency had to be removed

arrow/dev/tasks/conda-recipes/arrow-cpp/meta.yaml

Lines 353 to 354 in 2c768a1

	# causes segfaults
	# - sparse

because it lead to segmentation faults in the gandiva tests. The only additional dependencies that were pulled in were

+    libllvm11:           11.1.0-he0ac6c6_5           conda-forge
+    llvmlite:            0.39.1-py310h58363a5_1      conda-forge
+    numba:               0.56.4-py310ha5257ce_0      conda-forge
+    sparse:              0.13.0-pyhd8ed1ab_0         conda-forge

which seems to point towards an interaction with the llvm headers being in use in gandiva AFAIU. Perhaps it's not depending on only the headers after all?

To fix this issue, the line quoted above should be uncommented, and the conda tests still pass.

Component(s)

C++, C++ - Gandiva, Continuous Integration, Python

[kou]

Gandiva use LLVM headers and LLVM static libraries (not shared libraries).
Is it related?

[h-vetinari]

Gandiva use LLVM headers and LLVM static libraries (not shared libraries).
Is it related?

My guess is yes because of how sparse brings in libllvm11 through numba. But I have no idea what's causing the segfault, theoretically those things should be isolated. In particular, libllvm11 only brings in strongly versioned artefacts.

[kou]

Can we try the following to confirm whether LLVM version difference (14 for arrow-cpp and 11 for numba) is related or not?

diff --git a/dev/tasks/conda-recipes/arrow-cpp/meta.yaml b/dev/tasks/conda-recipes/arrow-cpp/meta.yaml
index 1c3478582b..6e2ef6bad0 100644
--- a/dev/tasks/conda-recipes/arrow-cpp/meta.yaml
+++ b/dev/tasks/conda-recipes/arrow-cpp/meta.yaml
@@ -4,7 +4,7 @@
 {% set build_ext_version = ARROW_VERSION %}
 {% set build_ext = "cuda" if cuda_enabled else "cpu" %}
 {% set proc_build_number = "0" %}
-{% set llvm_version = "14" %}
+{% set llvm_version = "11" %}
 
 package:
   name: apache-arrow

[h-vetinari]

I'm waiting for #15014 to be finished/merged, then I'll tackle some of these issues.

[h-vetinari]

We finished this in the meantime. The problem was somehow related to libllvm15, for more details see conda-forge/pyarrow-feedstock#121 and links therein.

arrow 👉 libllvm 👇	v14	v15	v16	v17
v14	✅	-	-	-
v15	💥	💥	-	-
v16	✅	✅	✅	✅
v17	-	✅	-	-