Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow path operations on UPath instances without import #199

Closed
ap-- opened this issue Feb 21, 2024 · 1 comment · Fixed by #200
Closed

Allow path operations on UPath instances without import #199

ap-- opened this issue Feb 21, 2024 · 1 comment · Fixed by #200
Assignees
Labels
enhancement 🚀 New feature or request

Comments

@ap--
Copy link
Collaborator

ap-- commented Feb 21, 2024

This would allow to implement a "PurePath" equivalent for path manipulation, that would not require the specific dependencies to be installed.

For known_implementations we can vendor the two required methods (_strip_protocol and _get_kwargs_from_urls) for each protocol.
That allows for example, to do all kinds of path operations on s3 without s3fs to be installed.

@ap-- ap-- added the enhancement 🚀 New feature or request label Feb 21, 2024
@ap-- ap-- self-assigned this Feb 21, 2024
@bolkedebruin
Copy link

Responding here for apache/airflow#37524

I think only operations that do something on the fs should invoke the backend. String operations like is_relative_to ,stripping protocols or just str(upath) should not. This is equivalent how Pathlib handles it currently.

In Airflow's case the DAG processor would need to have present and instantiate every backend that often does additional eager loading. That is something we really don't want as this can add significant compute time for simple operations.

For now we derive ObjectStoragePath from CloudPath for strict object storage behavior (buckets, keys, etc). So also file would behave the same way (I'm not sure if it does now with upath > 0.2.1). We might additional ones if we can make them behave the same way so users have a uniform and predictable experience.

Summarizing: I think upath should not forward anything to the backend if _protocol_dispatch is False for non filesystem operations.

@ap-- ap-- closed this as completed in #200 Mar 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement 🚀 New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants