Skip to content

Commit b6e8467

Browse files
owenowenismeYou-Cheng Lin
andauthored
[Data][CI] Add fix_block.py back to CI (#57841)
## Description https://arrow.apache.org/docs/python/generated/pyarrow.Array.html#pyarrow.Array.to_numpy <img width="772" height="270" alt="Screenshot 2025-10-18 at 3 14 36 PM" src="https://github.com/user-attachments/assets/d9cbf986-4271-41e6-9c4c-96201d32d1c6" /> `zero_copy_only` is actually default to True, so we should explicit pass False, for pyarrow version < 13.0.0 https://github.com/ray-project/ray/blob/1e38c9408caa92c675f0aa3e8bb60409c2d9159f/python/ray/data/_internal/arrow_block.py#L540-L546 ## Related issues Closes #57819 ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc. --------- Signed-off-by: You-Cheng Lin (Owen) <mses010108@gmail.com> Signed-off-by: You-Cheng Lin <youchenglin@youchenglin-L3DPGF50JG.local> Signed-off-by: You-Cheng Lin <106612301+owenowenisme@users.noreply.github.com> Signed-off-by: You-Cheng Lin <mses010108@gmail.com> Co-authored-by: You-Cheng Lin <youchenglin@youchenglin-L3DPGF50JG.local>
1 parent 0b5b80d commit b6e8467

File tree

3 files changed

+12
-3
lines changed

3 files changed

+12
-3
lines changed

python/ray/data/_internal/arrow_block.py

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -572,9 +572,13 @@ def to_pylist(self) -> List[Any]:
572572
return self._column.to_pylist()
573573

574574
def to_numpy(self, zero_copy_only: bool = False) -> np.ndarray:
575-
# NOTE: Pyarrow < 13.0.0 does not support ``zero_copy_only``
576575
if get_pyarrow_version() < _MIN_PYARROW_VERSION_TO_NUMPY_ZERO_COPY_ONLY:
577-
return self._column.to_numpy()
576+
if isinstance(
577+
self._column, pyarrow.ChunkedArray
578+
): # NOTE: ChunkedArray in Pyarrow < 13.0.0 does not support ``zero_copy_only``
579+
return self._column.to_numpy()
580+
else:
581+
return self._column.to_numpy(zero_copy_only=zero_copy_only)
578582

579583
return self._column.to_numpy(zero_copy_only=zero_copy_only)
580584

python/ray/data/tests/test_block.py

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -156,3 +156,9 @@ def test_find_partitions_duplicates():
156156
assert partitions[1].to_pydict() == {"value": []} # [1,2)
157157
assert partitions[2].to_pydict() == {"value": [2, 2, 2, 2, 2]} # [2,3)
158158
assert partitions[3].to_pydict() == {"value": []} # >=3
159+
160+
161+
if __name__ == "__main__":
162+
import sys
163+
164+
sys.exit(pytest.main(["-v", __file__]))

semgrep.yml

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,6 @@ rules:
3535
exclude:
3636
# FIXME: These tests weren't run in CI, and now they're failing.
3737
- "python/ray/data/tests/test_arrow_serialization.py"
38-
- "python/ray/data/tests/test_block.py"
3938
- "python/ray/data/tests/test_hash_shuffle.py"
4039
languages:
4140
- python

0 commit comments

Comments
 (0)