Move arrow-pyarrow tests that require pyarrow to be installed into arrow-pyarrow-testing crate
#7742
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
alamb
changed the title
pyarrow to be installed into arrow-pyarrow-testing crate
Contributor
Author
|
Here is an example of these tests running in CI passing: https://github.com/apache/arrow-rs/actions/runs/15824854202/job/44602338829?pr=7742#step:9:93 |
alamb
commented
Jun 23, 2025
| source venv/bin/activate | ||
| cargo test -p arrow-pyarrow | ||
| - name: Run tests | ||
| cd arrow-pyarrow-testing |
Contributor
Author
There was a problem hiding this comment.
this is the key change -- move the tests to a different module
| //! pip3 install --break-system-packages pyarrow | ||
| //! ``` | ||
| use arrow_array::builder::{BinaryViewBuilder, StringViewBuilder}; |
Contributor
Author
There was a problem hiding this comment.
the tests are unmodified -- I just added some comments about how to run the tests locally
alamb
added a commit
that referenced
this pull request
Jun 23, 2025
…yarrow` and `parquet-variant` (#7745) Note this targets a release branch, not main I have a different proposed fix for `main`: - #7742 I will also make a fix for parquet-variant test failures # Which issue does this PR close? - Related to #7394 - Related to #7736 - Related to #7746 # Rationale for this change `cargo test --all` requires python and pyarrow installed, where it did not in previous versions of arrow due to breaking out `arrow-pyarrow` into its own crate in - #7694 Also the new `parquet-variant` crate tests fail as part of the verification script too -- see ttps://github.com//issues/7746 In order to get a script that can automatically verify a release candidate, let's ignore this new module for now. More details - #7736 - #7742 Note that the arrow-pyarrow tests do run as part of CI and succeed on this branch # What changes are included in this PR? 1. Exclude running the `arrow-pyarrow` and `parquet-variant` tests as part of `verify-release-acndidate` # Testing I verified locally that `./dev/release/verify-release-candidate.sh 55.2.0 1` passes with this script # Are there any user-facing changes? No this is a development process only change
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR refactors the testing setup for the pyarrow integration by moving tests that require the pyarrow package into a separate crate, thereby decoupling them from the main workspace tests.
- Relocates pyarrow-dependent tests to the arrow-pyarrow-testing crate
- Updates Cargo.toml and GitHub workflow files to reflect the new testing layout
- Adds detailed documentation regarding the environment setup for these tests
Reviewed Changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| arrow-pyarrow-testing/tests/pyarrow.rs | Adds documentation for tests that require pyarrow to be installed |
| arrow-pyarrow-testing/src/lib.rs | Provides a minimal library for the test crate |
| arrow-pyarrow-testing/Cargo.toml | Defines the test crate and its dependencies |
| Cargo.toml | Updates workspace exclude list to remove arrow-pyarrow-testing from the main tests |
| .github/workflows/rust.yml | Adjusts test commands to run workspace tests without excluding arrow-pyarrow |
| .github/workflows/integration.yml | Adds steps to run tests in arrow-pyarrow-testing and integration testing |
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Contributor
Author
|
@mbrobbel I wonder if you might have time to review this PR? |
mbrobbel
approved these changes
Jul 7, 2025
Contributor
Author
|
Thank you @mbrobbel 🙏 |
berkaysynnada
pushed a commit
to synnada-ai/arrow-rs
that referenced
this pull request
Jul 17, 2025
…yarrow` and `parquet-variant` (apache#7745) Note this targets a release branch, not main I have a different proposed fix for `main`: - apache#7742 I will also make a fix for parquet-variant test failures # Which issue does this PR close? - Related to apache#7394 - Related to apache#7736 - Related to apache#7746 # Rationale for this change `cargo test --all` requires python and pyarrow installed, where it did not in previous versions of arrow due to breaking out `arrow-pyarrow` into its own crate in - apache#7694 Also the new `parquet-variant` crate tests fail as part of the verification script too -- see ttps://github.com/apache/issues/7746 In order to get a script that can automatically verify a release candidate, let's ignore this new module for now. More details - apache#7736 - apache#7742 Note that the arrow-pyarrow tests do run as part of CI and succeed on this branch # What changes are included in this PR? 1. Exclude running the `arrow-pyarrow` and `parquet-variant` tests as part of `verify-release-acndidate` # Testing I verified locally that `./dev/release/verify-release-candidate.sh 55.2.0 1` passes with this script # Are there any user-facing changes? No this is a development process only change
berkaysynnada
pushed a commit
to synnada-ai/arrow-rs
that referenced
this pull request
Aug 26, 2025
…yarrow` and `parquet-variant` (apache#7745) Note this targets a release branch, not main I have a different proposed fix for `main`: - apache#7742 I will also make a fix for parquet-variant test failures # Which issue does this PR close? - Related to apache#7394 - Related to apache#7736 - Related to apache#7746 # Rationale for this change `cargo test --all` requires python and pyarrow installed, where it did not in previous versions of arrow due to breaking out `arrow-pyarrow` into its own crate in - apache#7694 Also the new `parquet-variant` crate tests fail as part of the verification script too -- see ttps://github.com/apache/issues/7746 In order to get a script that can automatically verify a release candidate, let's ignore this new module for now. More details - apache#7736 - apache#7742 Note that the arrow-pyarrow tests do run as part of CI and succeed on this branch # What changes are included in this PR? 1. Exclude running the `arrow-pyarrow` and `parquet-variant` tests as part of `verify-release-acndidate` # Testing I verified locally that `./dev/release/verify-release-candidate.sh 55.2.0 1` passes with this script # Are there any user-facing changes? No this is a development process only change
berkaysynnada
pushed a commit
to synnada-ai/arrow-rs
that referenced
this pull request
Aug 26, 2025
…yarrow` and `parquet-variant` (apache#7745) Note this targets a release branch, not main I have a different proposed fix for `main`: - apache#7742 I will also make a fix for parquet-variant test failures # Which issue does this PR close? - Related to apache#7394 - Related to apache#7736 - Related to apache#7746 # Rationale for this change `cargo test --all` requires python and pyarrow installed, where it did not in previous versions of arrow due to breaking out `arrow-pyarrow` into its own crate in - apache#7694 Also the new `parquet-variant` crate tests fail as part of the verification script too -- see ttps://github.com/apache/issues/7746 In order to get a script that can automatically verify a release candidate, let's ignore this new module for now. More details - apache#7736 - apache#7742 Note that the arrow-pyarrow tests do run as part of CI and succeed on this branch # What changes are included in this PR? 1. Exclude running the `arrow-pyarrow` and `parquet-variant` tests as part of `verify-release-acndidate` # Testing I verified locally that `./dev/release/verify-release-candidate.sh 55.2.0 1` passes with this script # Are there any user-facing changes? No this is a development process only change
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
55.2.0(June 2025) #7394test_to_pyarrowtests fail during release verification #7736Rationale for this change
At its core, if someone isn't using / modifying the pyarrow integration for arrow-rs they shouldn't have to install / configure python to get the tests working in
arrow-rsRunning
cargo test --workspacenow also runs tests that require python to be setup and thepyarrowmodule to be installed. This is problematic because:The second item was very confusing for me while I tried to debug what going on as I ket getting an error about pyarrow not being installed, even though it was installed in my
venv:What changes are included in this PR?
arrow-pyarrow-testing, which is not part of the workspace and thus not run withcargo test --allcargo test --exclude arrow-pyarrowFrequently Asked Questions
Why not add
--exclude arrow-pyarrowtoverify_release_candidate.sh?While the minimal fix would be to add
--exclude arrow-pyarrowto verify_release_candidate.sh this requires all users of arrow to remember to add--exclude arrow-pyarrowto their tests even if they don't care about pythonWhy not in
pyarrow-arrow-integration-testing?I did not put this test in
pyarrow-arrow-integration-testingbecause that module doesn't compile for me with the stock python installSomehow python needs to be installed with the ability to make dynamic libraries that I haven't figured out and don't really want to. It seems maybe related to https://pyo3.rs/v0.18.1/getting_started#python (thanks to @Xuanwo for the pointer in PyO3/pyo3#2136 / apache/opendal#1675)
Are there any user-facing changes?
If there are user-facing changes then we may require documentation to be updated before approving the PR.
If there are any breaking changes to public APIs, please call them out.