Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[train][tune] Safely check if the storage filesystem is pyarrow.fs.S3FileSystem #48216

Merged
merged 4 commits into from
Oct 30, 2024
Merged

[train][tune] Safely check if the storage filesystem is pyarrow.fs.S3FileSystem #48216

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e6264b3
Updated storage to check type_name instead of instance
ccoulombe Oct 23, 2024
e5f04ce
Updated _pyarrow_fs_copy_files to check for S3 support in Arrow via i…
ccoulombe Oct 25, 2024
7c1b83a
Satisfy linter, do not sort pyarrow imports
ccoulombe Oct 25, 2024
e7721e5
Merge branch 'master' into patch-1
ccoulombe Oct 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion python/ray/train/_internal/storage.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,12 @@
"pyarrow is a required dependency of Ray Train and Ray Tune. "
"Please install with: `pip install pyarrow`"
) from e

try:
# check if Arrow has S3 support
from pyarrow.fs import S3FileSystem
except ImportError:
S3FileSystem = None
# isort: on

import fnmatch
Expand Down Expand Up @@ -98,7 +104,7 @@ def find(self, path, maxdepth=None, withdirs=False, detail=False, **kwargs):
def _pyarrow_fs_copy_files(
source, destination, source_filesystem=None, destination_filesystem=None, **kwargs
):
if isinstance(destination_filesystem, pyarrow.fs.S3FileSystem):
if S3FileSystem and isinstance(destination_filesystem, pyarrow.fs.S3FileSystem):
# Workaround multi-threading issue with pyarrow. Note that use_threads=True
# is safe for download, just not for uploads, see:
# https://github.com/apache/arrow/issues/32372
Expand Down