Skip to content

Commit adb6847

Browse files
committed
Add exception handling for URI download failures
- Added try-except block in load_uri_bytes function to handle invalid URIs - Failed downloads now return None instead of crashing the pipeline - Added warning log message for failed URI downloads
1 parent 50ffca4 commit adb6847

File tree

1 file changed

+8
-2
lines changed

1 file changed

+8
-2
lines changed

python/ray/data/_internal/planner/plan_download_op.py

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -189,8 +189,14 @@ def download_bytes_threaded(
189189
def load_uri_bytes(uri_path_iterator):
190190
"""Function that takes an iterator of URI paths and yields downloaded bytes for each."""
191191
for uri_path in uri_path_iterator:
192-
with fs.open_input_file(uri_path) as f:
193-
yield f.read()
192+
try:
193+
with fs.open_input_file(uri_path) as f:
194+
yield f.read()
195+
except Exception as e:
196+
logger.warning(
197+
f"Failed to download URI '{uri_path}' from column with error: {e}"
198+
)
199+
yield None
194200

195201
# Use make_async_gen to download URI bytes concurrently
196202
# This preserves the order of results to match the input URIs

0 commit comments

Comments
 (0)