GH-47632: [CI][C++] Add a CI job for JNI on Linux #47746

kou · 2025-10-08T00:04:51Z

Rationale for this change

This is for preventing to break Apache Arrow Java JNI use case on Linux.

What changes are included in this PR?

Add a CI job that uses build options for JNI use case
Install more packages in manylinux image that is also used by JNI build

Are these changes tested?

Yes.

Are there any user-facing changes?

No.

GitHub Issue: [CI][C++] Add a CI job that uses the same build options for JNI on Linux #47632

github-actions · 2025-10-08T00:05:17Z

⚠️ GitHub issue #47632 has been automatically assigned in GitHub to PR creator.

kou · 2025-10-08T00:05:31Z

ci/docker/python-wheel-manylinux.dockerfile

@@ -89,7 +89,9 @@ RUN --mount=type=secret,id=github_repository_owner \
        --x-install-root=${VCPKG_ROOT}/installed \
        --x-manifest-root=/arrow/ci/vcpkg \
        --x-feature=azure \
+        --x-feature=dev \


This is for ARROW_BUILD_TESTS=ON.

Do we really need to enable this for Python wheels?

Yes, to reuse this image for build configuration for JNI.

I think that reusing this image reduces maintenance cost but you don't want to increase this image size, right?

I'll create a new image for build configuration for JNI.

My main concern is that it makes vcpkg build times even larger when the cache isn't fresh or there is no cache.

kou · 2025-10-08T00:07:14Z

ci/scripts/cpp_build.sh

+elif [ -n "${CMAKE_PRESET}" ]; then
+  cmake \
+    --preset="${CMAKE_PRESET}" \
+    ${ARROW_CMAKE_ARGS} \
+    ${source_dir}


We want to provide preset for JNI build to share build options in apache/arrow and apache/arrow-java.

Why not pass --preset to the existing cmake call below?

If we use it, all variables by --preset are overridden because it specify most variables explicitly. Explicit variables are used instead of variables set by --preset.

kou · 2025-10-08T00:08:14Z

ci/scripts/cpp_build.sh

-  if [ -x "$(command -v sudo)" ]; then
-    SUDO=sudo
-  else
+  if [ "$(id --user)" -eq 0 ]; then


We don't need to use sudo with the root user.

manylinux image doesn't provide /usr/bin/sudo but /opt/rh/gcc-toolset-12/root/usr/bin/sudo exists.

kou · 2025-10-08T00:08:53Z

ci/scripts/cpp_test.sh

+if ! type storage-testbench >/dev/null 2>&1; then
+  exclude_tests+=("arrow-gcsfs-test")
+fi
+if ! type minio >/dev/null 2>&1; then
+  exclude_tests+=("arrow-s3fs-test")
+fi


They don't exist in manylinux image.

kou · 2025-10-08T00:09:42Z

ci/scripts/cpp_test.sh

+if [ "${ARROW_USE_MESON:-OFF}" = "OFF" ] && [ "${ARROW_EMSCRIPTEN:-OFF}" = "OFF" ]; then
+  CMAKE_PREFIX_PATH="${CMAKE_INSTALL_PREFIX:-${ARROW_HOME}}"
+  if [ -n "${VCPKG_ROOT}" ] && [ -n "${VCPKG_TARGET_TRIPLET}" ]; then
+    CMAKE_PREFIX_PATH+=";${VCPKG_ROOT}/installed/${VCPKG_TARGET_TRIPLET}"
+  fi
+  cmake \
+    -S ${source_dir}/examples/minimal_build \
+    -B ${build_dir}/examples/minimal_build \
+    -DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}"
+  cmake --build ${build_dir}/examples/minimal_build
+  pushd ${source_dir}/examples/minimal_build
+  ${build_dir}/examples/minimal_build/arrow-example
+  popd
+fi


This is a test for our CMake packages. If find_package(Arrow) doesn't work, this will detect it.

JNI build uses find_package(Arrow).

kou · 2025-10-08T00:12:51Z

ci/vcpkg/vcpkg.json

@@ -19,6 +19,7 @@
    "re2",
    "snappy",
    "utf8proc",
+    "xsimd",


We can add this again because we have #47221 and vcpkg provides xsimd 13.2.0 since #47627.

kou · 2025-10-08T00:26:24Z

cpp/src/arrow/testing/process.cc

@@ -176,7 +176,7 @@ class Process::Impl {
      for (const auto& kv : process::environment::current()) {
        env[kv.key()] = process::environment::value(kv.value());
      }
-      env["PATH"] = process::environment::value(current_exe.parent_path());
+      env["PATH"] = process::environment::value(current_exe.parent_path().string());


Recent Boost or old g++ (12.2.1) don't accept std::filesystem::path:

/arrow/cpp/src/arrow/testing/process.cc: In member function ‘arrow::Status arrow::util::Process::Impl::SetExecutable(const std::string&)’: /arrow/cpp/src/arrow/testing/process.cc:179:74: error: call of overloaded ‘value(std::filesystem::__cxx11::path)’ is ambiguous 179 | env["PATH"] = process::environment::value(current_exe.parent_path()); | ^ In file included from /arrow/cpp/src/arrow/testing/process.cc:44: /opt/vcpkg/installed/amd64-linux-static-release/include/boost/process/v2/environment.hpp:727:14: note: candidate: ‘boost::process::v2::environment::value::value(boost::process::v2::environment::value_view)’ 727 | explicit value(value_view kv) : value_(kv.c_str()) {} | ^~~~~ /opt/vcpkg/installed/amd64-linux-static-release/include/boost/process/v2/environment.hpp:723:5: note: candidate: ‘boost::process::v2::environment::value::value(string_type&&)’ 723 | value( string_type&& source ) : value_(std::move(source)) {} | ^~~~~ /opt/vcpkg/installed/amd64-linux-static-release/include/boost/process/v2/environment.hpp:722:5: note: candidate: ‘boost::process::v2::environment::value::value(const string_type&)’ 722 | value( const string_type& source ) : value_(source) {} | ^~~~~

We need to convert std::filesystem::path to std::string explicitly.

kou · 2025-10-09T10:47:17Z

.github/workflows/cpp_extra.yml


 permissions:
  contents: read
+  packages: write


This is for using GitHub Packages as vcpkg cache.

Can you add a comment about this?

Done and moved to steps.jni-linux.permissions from the top-level permissions to reduce scope.

kou · 2025-10-10T08:55:59Z

This is ready.

This is a blocker of new Apache Arrow Java release. So this is needed for 22.0.0 C++ release.

kou · 2025-10-11T12:20:45Z

@raulcd Can we merge this for 22.0.0?

raulcd · 2025-10-12T14:50:49Z

@raulcd Can we merge this for 22.0.0?

Yes, I'll cherry-pick it on the maintenance branch once we merge it.

kou · 2025-10-12T14:57:18Z

Thanks. Let's merge this.

conbench-apache-arrow · 2025-10-12T22:20:08Z

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 0c8ce5f.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 2 possible false positives for unstable benchmarks that are known to sometimes produce them.

### Rationale for this change This is for preventing to break Apache Arrow Java JNI use case on Linux. ### What changes are included in this PR? * Add a CI job that uses build options for JNI use case * Install more packages in manylinux image that is also used by JNI build ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * GitHub Issue: #47632 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

kou · 2025-10-13T07:33:05Z

docker compose push failed: #47795

This is also a blocker of 22.0.0.

raulcd · 2025-10-13T07:38:54Z

This is also a blocker of 22.0.0.

Thanks for pointing that out!

### Rationale for this change This is for preventing to break Apache Arrow Java JNI use case on Linux. ### What changes are included in this PR? * Add a CI job that uses build options for JNI use case * Install more packages in manylinux image that is also used by JNI build ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * GitHub Issue: apache#47632 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

pitrou · 2025-10-15T13:58:30Z

ci/scripts/cpp_test.sh

+# This is for testing find_package(Arrow).
+#
+# Note that this is not a perfect solution. We should improve this
+# later.


It seems that this new check is breaking some C++ builds (CUDA, Thread Sanitizer). @kou

Oh, sorry. I missed these failures: #47840

kou requested review from assignUser, jonkeane and raulcd as code owners October 8, 2025 00:04

github-actions bot added Component: C++ awaiting committer review Awaiting committer review labels Oct 8, 2025

kou commented Oct 8, 2025

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Oct 8, 2025

kou added 5 commits October 8, 2025 09:41

apacheGH-47632: [CI][C++] Add a CI job for JNI on Linux

d17eec8

Don't build example with ASAN

33febe5

Fix lint

6388be4

Debug

b7eaa97

Rebase

651e4f2

kou force-pushed the ci-java-jni-linux branch from 21f73d5 to 651e4f2 Compare October 8, 2025 00:42

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Oct 8, 2025

kou added 10 commits October 8, 2025 10:03

Fix inverted condition

2e2408f

Adjust path

e19f5dd

Increase timeout

6cd6fe5

Set PATH

1999452

Use cache

2186d47

Fix syntax

2ec5f60

Write

5748fa3

Debug

dcbcbf7

Use VC

b8ec827

Fix vcpkg arch

d7e89a0

kou commented Oct 9, 2025

View reviewed changes

Fix naming style

9656683

github-actions bot removed the awaiting change review Awaiting change review label Oct 9, 2025

kou added 3 commits October 10, 2025 10:32

Reduce scope

48dac5d

Install gdb

d8fa54f

Skip

177d0c7

github-actions bot added Component: Gandiva awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Oct 10, 2025

Use separated image

3f30e63

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Oct 10, 2025

Fix arch

0321c0c

kou mentioned this pull request Oct 10, 2025

GH-898: Upgrade to Apache POM 35 and identify fixes needed to have CI happy apache/arrow-java#865

Merged

Push

45df0cb

kou merged commit 0c8ce5f into apache:main Oct 12, 2025
48 checks passed

kou deleted the ci-java-jni-linux branch October 12, 2025 14:57

kou removed the awaiting change review Awaiting change review label Oct 12, 2025

kou mentioned this pull request Oct 12, 2025

[CI][C++] Add a CI job that uses the same build options for JNI on Linux #47632

Closed

pitrou reviewed Oct 15, 2025

View reviewed changes

github-actions bot added the awaiting committer review Awaiting committer review label Oct 15, 2025

kou mentioned this pull request Oct 17, 2025

[CI][C++] Example build is filing in CUDA job #47840

Closed

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Oct 17, 2025

GH-47632: [CI][C++] Add a CI job for JNI on Linux #47746

GH-47632: [CI][C++] Add a CI job for JNI on Linux #47746

Uh oh!

Conversation

kou commented Oct 8, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions bot commented Oct 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kou commented Oct 10, 2025

Uh oh!

kou commented Oct 11, 2025

Uh oh!

raulcd commented Oct 12, 2025

Uh oh!

kou commented Oct 12, 2025

Uh oh!

Uh oh!

conbench-apache-arrow bot commented Oct 12, 2025

Uh oh!

kou commented Oct 13, 2025

Uh oh!

raulcd commented Oct 13, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kou commented Oct 8, 2025 •

edited by github-actions bot

Loading