Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GH-37410: [C++][Gandiva] Add support for using LLVM shared library #37412

Merged
merged 2 commits into from
Sep 4, 2023

Conversation

kou
Copy link
Member

@kou kou commented Aug 28, 2023

Rationale for this change

Gandiva always links LLVM statically. But we can use LLVM shared library instead to reduce libgandiva.so size.

What changes are included in this PR?

Add ARROW_LLVM_USE_SHARED like other dependencies.

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes.

This has a backward incompatible change. LLVM shared library not static libraries is used by default. If you want to use LLVM static libraries, you need to specify -DARROW_LLVM_USE_SHARED=OFF or -DARROW_DEPENDENCY_USE_SHARED=OFF explicitly.

@github-actions
Copy link

⚠️ GitHub issue #37410 has been automatically assigned in GitHub to PR creator.

@kou
Copy link
Member Author

kou commented Aug 28, 2023

@github-actions crossbow submit java-jars -g wheel

@github-actions
Copy link

Revision: 5117655

Submitted crossbow builds: ursacomputing/crossbow @ actions-63e615699a

Task Status
java-jars Github Actions
wheel-clean Github Actions
wheel-macos-big-sur-cp310-arm64 Github Actions
wheel-macos-big-sur-cp311-arm64 Github Actions
wheel-macos-big-sur-cp38-arm64 Github Actions
wheel-macos-big-sur-cp39-arm64 Github Actions
wheel-macos-mojave-cp310-amd64 Github Actions
wheel-macos-mojave-cp311-amd64 Github Actions
wheel-macos-mojave-cp38-amd64 Github Actions
wheel-macos-mojave-cp39-amd64 Github Actions
wheel-manylinux-2-28-cp310-amd64 Github Actions
wheel-manylinux-2-28-cp310-arm64 Github Actions
wheel-manylinux-2-28-cp311-amd64 Github Actions
wheel-manylinux-2-28-cp311-arm64 Github Actions
wheel-manylinux-2-28-cp38-amd64 Github Actions
wheel-manylinux-2-28-cp38-arm64 Github Actions
wheel-manylinux-2-28-cp39-amd64 Github Actions
wheel-manylinux-2-28-cp39-arm64 Github Actions
wheel-manylinux-2014-cp310-amd64 Github Actions
wheel-manylinux-2014-cp310-arm64 Github Actions
wheel-manylinux-2014-cp311-amd64 Github Actions
wheel-manylinux-2014-cp311-arm64 Github Actions
wheel-manylinux-2014-cp38-amd64 Github Actions
wheel-manylinux-2014-cp38-arm64 Github Actions
wheel-manylinux-2014-cp39-amd64 Github Actions
wheel-manylinux-2014-cp39-arm64 Github Actions
wheel-windows-cp310-amd64 Github Actions
wheel-windows-cp311-amd64 Github Actions
wheel-windows-cp38-amd64 Github Actions
wheel-windows-cp39-amd64 Github Actions

@kou kou force-pushed the cpp-gandiva-llvm-shared branch from f42ae12 to 780357c Compare August 29, 2023 00:44
@kou
Copy link
Member Author

kou commented Aug 29, 2023

@github-actions crossbow submit -g cpp

@github-actions
Copy link

Revision: 780357c

Submitted crossbow builds: ursacomputing/crossbow @ actions-3d4f488389

Task Status
test-alpine-linux-cpp Github Actions
test-build-cpp-fuzz Github Actions
test-conda-cpp Github Actions
test-conda-cpp-valgrind Azure
test-cuda-cpp Github Actions
test-debian-11-cpp-amd64 Github Actions
test-debian-11-cpp-i386 Github Actions
test-fedora-35-cpp Github Actions
test-ubuntu-20.04-cpp Github Actions
test-ubuntu-20.04-cpp-bundled Github Actions
test-ubuntu-20.04-cpp-minimal-with-formats Github Actions
test-ubuntu-20.04-cpp-thread-sanitizer Github Actions
test-ubuntu-22.04-cpp Github Actions
test-ubuntu-22.04-cpp-20 Github Actions
test-ubuntu-22.04-cpp-no-threading Github Actions

@kou
Copy link
Member Author

kou commented Aug 29, 2023

I'll merge this in this week if nobody objects this.

@js8544
Copy link
Collaborator

js8544 commented Aug 29, 2023

+1. Thanks!

if(MSVC)
# It seems that conda's llvmdev for Windows doesn't provide shared
# library.
set(ARROW_LLVM_USE_SHARED_DEFAULT OFF)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I don't think we should change this for conda. Instead, conda users on Windows will have to explicitly pass -DARROW_LLVM_USE_SHARED=OFF.

Copy link
Member Author

@kou kou Sep 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that LLVM itself doesn't support shared library with MSVC: https://github.com/llvm/llvm-project/blob/main/llvm/tools/llvm-shlib/CMakeLists.txt#L14-L16

@github-actions github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Sep 1, 2023
@github-actions github-actions bot added awaiting change review Awaiting change review awaiting changes Awaiting changes and removed awaiting changes Awaiting changes awaiting change review Awaiting change review labels Sep 1, 2023
@kou
Copy link
Member Author

kou commented Sep 4, 2023

@github-actions crossbow submit -g linux

@github-actions
Copy link

github-actions bot commented Sep 4, 2023

Revision: 52caff9

Submitted crossbow builds: ursacomputing/crossbow @ actions-e8c648a76a

Task Status
almalinux-8-amd64 Github Actions
almalinux-8-arm64 Github Actions
almalinux-9-amd64 Github Actions
almalinux-9-arm64 Github Actions
amazon-linux-2023-amd64 Github Actions
amazon-linux-2023-arm64 Github Actions
centos-7-amd64 Github Actions
centos-8-stream-amd64 Github Actions
centos-8-stream-arm64 Github Actions
centos-9-stream-amd64 Github Actions
centos-9-stream-arm64 Github Actions
debian-bookworm-amd64 Github Actions
debian-bookworm-arm64 Github Actions
debian-bullseye-amd64 Github Actions
debian-bullseye-arm64 Github Actions
debian-trixie-amd64 Github Actions
debian-trixie-arm64 Github Actions
ubuntu-focal-amd64 Github Actions
ubuntu-focal-arm64 Github Actions
ubuntu-jammy-amd64 Github Actions
ubuntu-jammy-arm64 Github Actions
ubuntu-lunar-amd64 Github Actions
ubuntu-lunar-arm64 Github Actions

@kou
Copy link
Member Author

kou commented Sep 4, 2023

+1

@kou kou merged commit 3ad2d0c into apache:main Sep 4, 2023
30 checks passed
@kou kou deleted the cpp-gandiva-llvm-shared branch September 4, 2023 02:41
@kou kou removed the awaiting changes Awaiting changes label Sep 4, 2023
@conbench-apache-arrow
Copy link

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 3ad2d0c.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.

loicalleyne pushed a commit to loicalleyne/arrow that referenced this pull request Nov 13, 2023
…ary (apache#37412)

### Rationale for this change

Gandiva always links LLVM statically. But we can use LLVM shared library instead to reduce `libgandiva.so` size.

### What changes are included in this PR?

Add `ARROW_LLVM_USE_SHARED` like other dependencies.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes.

This has a backward incompatible change. LLVM shared library not static libraries is used by default. If you want to use LLVM static libraries, you need to specify `-DARROW_LLVM_USE_SHARED=OFF` or `-DARROW_DEPENDENCY_USE_SHARED=OFF` explicitly.
* Closes: apache#37410

Lead-authored-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
dgreiss pushed a commit to dgreiss/arrow that referenced this pull request Feb 19, 2024
…ary (apache#37412)

### Rationale for this change

Gandiva always links LLVM statically. But we can use LLVM shared library instead to reduce `libgandiva.so` size.

### What changes are included in this PR?

Add `ARROW_LLVM_USE_SHARED` like other dependencies.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes.

This has a backward incompatible change. LLVM shared library not static libraries is used by default. If you want to use LLVM static libraries, you need to specify `-DARROW_LLVM_USE_SHARED=OFF` or `-DARROW_DEPENDENCY_USE_SHARED=OFF` explicitly.
* Closes: apache#37410

Lead-authored-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants