[Data][CI] Add fix_block.py back to CI #57841
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: You-Cheng Lin <106612301+owenowenisme@users.noreply.github.com>
bveeramani
reviewed
Oct 21, 2025
Member
bveeramani
left a comment
bveeramani
left a comment
There was a problem hiding this comment.
@owenowenisme I don't think ChunkedArray supports zero_copy_only for PyArrow 9, and I think self._column can be either an Array or ChunkedArray: https://arrow.apache.org/docs/9.0/python/generated/pyarrow.ChunkedArray.html#pyarrow.ChunkedArray.to_numpy.
Also, how does zero_copy_only=False fix the test failure, or is that change unrelated?
Member
Author
|
Changed to not pass zero_copy_only when the instance is a ChunkedArray on PyArrow versions prior to 13.0.0. |
bveeramani
approved these changes
Oct 23, 2025
|
|
||
| def to_numpy(self, zero_copy_only: bool = False) -> np.ndarray: | ||
| # NOTE: Pyarrow < 13.0.0 does not support ``zero_copy_only`` | ||
| if get_pyarrow_version() < _MIN_PYARROW_VERSION_TO_NUMPY_ZERO_COPY_ONLY: |
Member
There was a problem hiding this comment.
Nit: Might be worth renaming _MIN_PYARROW_VERSION_TO_NUMPY_ZERO_COPY_ONLY to clarify that it only applies to ChunkedArray, but I guess the name would get pretty verbose.
Will defer to you
Member
Author
There was a problem hiding this comment.
_MIN_PYARROW_VERSION_TO_NUMPY_ZERO_COPY_ONLY is single usage at this time, I think we can leave it as it is for now.
xinyuangui2
pushed a commit
to xinyuangui2/ray
that referenced
this pull request
Oct 27, 2025
## Description https://arrow.apache.org/docs/python/generated/pyarrow.Array.html#pyarrow.Array.to_numpy <img width="772" height="270" alt="Screenshot 2025-10-18 at 3 14 36 PM" src="https://github.com/user-attachments/assets/d9cbf986-4271-41e6-9c4c-96201d32d1c6" /> `zero_copy_only` is actually default to True, so we should explicit pass False, for pyarrow version < 13.0.0 https://github.com/ray-project/ray/blob/1e38c9408caa92c675f0aa3e8bb60409c2d9159f/python/ray/data/_internal/arrow_block.py#L540-L546 ## Related issues Closes ray-project#57819 ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc. --------- Signed-off-by: You-Cheng Lin (Owen) <mses010108@gmail.com> Signed-off-by: You-Cheng Lin <youchenglin@youchenglin-L3DPGF50JG.local> Signed-off-by: You-Cheng Lin <106612301+owenowenisme@users.noreply.github.com> Signed-off-by: You-Cheng Lin <mses010108@gmail.com> Co-authored-by: You-Cheng Lin <youchenglin@youchenglin-L3DPGF50JG.local> Signed-off-by: xgui <xgui@anyscale.com>
landscapepainter
pushed a commit
to landscapepainter/ray
that referenced
this pull request
Nov 17, 2025
## Description https://arrow.apache.org/docs/python/generated/pyarrow.Array.html#pyarrow.Array.to_numpy <img width="772" height="270" alt="Screenshot 2025-10-18 at 3 14 36 PM" src="https://github.com/user-attachments/assets/d9cbf986-4271-41e6-9c4c-96201d32d1c6" /> `zero_copy_only` is actually default to True, so we should explicit pass False, for pyarrow version < 13.0.0 https://github.com/ray-project/ray/blob/1e38c9408caa92c675f0aa3e8bb60409c2d9159f/python/ray/data/_internal/arrow_block.py#L540-L546 ## Related issues Closes ray-project#57819 ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc. --------- Signed-off-by: You-Cheng Lin (Owen) <mses010108@gmail.com> Signed-off-by: You-Cheng Lin <youchenglin@youchenglin-L3DPGF50JG.local> Signed-off-by: You-Cheng Lin <106612301+owenowenisme@users.noreply.github.com> Signed-off-by: You-Cheng Lin <mses010108@gmail.com> Co-authored-by: You-Cheng Lin <youchenglin@youchenglin-L3DPGF50JG.local>
Aydin-ab
pushed a commit
to Aydin-ab/ray-aydin
that referenced
this pull request
Nov 19, 2025
## Description https://arrow.apache.org/docs/python/generated/pyarrow.Array.html#pyarrow.Array.to_numpy <img width="772" height="270" alt="Screenshot 2025-10-18 at 3 14 36 PM" src="https://github.com/user-attachments/assets/d9cbf986-4271-41e6-9c4c-96201d32d1c6" /> `zero_copy_only` is actually default to True, so we should explicit pass False, for pyarrow version < 13.0.0 https://github.com/ray-project/ray/blob/1e38c9408caa92c675f0aa3e8bb60409c2d9159f/python/ray/data/_internal/arrow_block.py#L540-L546 ## Related issues Closes ray-project#57819 ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc. --------- Signed-off-by: You-Cheng Lin (Owen) <mses010108@gmail.com> Signed-off-by: You-Cheng Lin <youchenglin@youchenglin-L3DPGF50JG.local> Signed-off-by: You-Cheng Lin <106612301+owenowenisme@users.noreply.github.com> Signed-off-by: You-Cheng Lin <mses010108@gmail.com> Co-authored-by: You-Cheng Lin <youchenglin@youchenglin-L3DPGF50JG.local> Signed-off-by: Aydin Abiar <aydin@anyscale.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
https://arrow.apache.org/docs/python/generated/pyarrow.Array.html#pyarrow.Array.to_numpy
zero_copy_onlyis actually default to True, so we should explicit pass False, for pyarrow version < 13.0.0ray/python/ray/data/_internal/arrow_block.py
Lines 540 to 546 in 1e38c94
Related issues
Closes #57819
Additional information