Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pin fsspec to use default expand_path #1681

Merged
merged 5 commits into from
Sep 21, 2023
Merged

Pin fsspec to use default expand_path #1681

merged 5 commits into from
Sep 21, 2023

Conversation

mariosasko
Copy link
Contributor

... to avoid having to update HfFileSystem.expand_path when new changes are made to the default implementation (e.g., the latest fsspec release (2023.9.0) added a maxdepth param to the signature (PR))

@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented Sep 19, 2023

The documentation is not available anymore as the PR was closed or merged.

@codecov
Copy link

codecov bot commented Sep 19, 2023

Codecov Report

Patch coverage: 8.33% and project coverage change: +0.18% 🎉

Comparison is base (e3061b4) 81.77% compared to head (4b2b403) 81.95%.

❗ Current head 4b2b403 differs from pull request most recent head cd04527. Consider uploading reports for the commit cd04527 to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1681      +/-   ##
==========================================
+ Coverage   81.77%   81.95%   +0.18%     
==========================================
  Files          62       62              
  Lines        7121     7123       +2     
==========================================
+ Hits         5823     5838      +15     
+ Misses       1298     1285      -13     
Files Changed Coverage Δ
src/huggingface_hub/hf_file_system.py 85.77% <8.33%> (-5.79%) ⬇️

... and 9 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@mariosasko mariosasko requested a review from Wauplin September 19, 2023 18:42
Comment on lines 379 to 383
if version.parse(fsspec.__version__) < version.parse("2023.5.0"):
# The default implementation in fsspec<2023.5.0 does not allow passing custom kwargs (e.g., we use these kwargs to propagate the `revision`)
def expand_path(
self, path: Union[str, List[str]], recursive: bool = False, maxdepth: Optional[int] = None, **kwargs
) -> List[str]:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens then on recent versions (fsspec>=2023.5.0)? It defaults back to the expand_path implemented in fsspec directly?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, expand_path allows passing **kwargs in versions >=2023.5.0 (thanks to fsspec/filesystem_spec@834115f), so there is no need to override it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I see, so the implementation is just a copy of the official fsspec one.
In your opinion, could we pin fsspec>=2023.5.0 in our dependencies to avoid having to support older versions?
Otherwise if you feel strongly against it, PR looks good to me. I'd just add a comment linking to https://github.com/fsspec/filesystem_spec/blob/4f0eb488f26299592df5cc6494a5ebd8cb8cca4f/fsspec/spec.py#L1139 to say that it's a duplicate (at first I understood that we has a custom implementation here, in addition of the kwargs).

Copy link
Contributor Author

@mariosasko mariosasko Sep 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pinning the minimal version sounds good to me :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For instance, Colab builds the env with version 2023.6.0, so this should have little impact on the user experience.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good to know! Thanks for making the change.

@mariosasko mariosasko changed the title Use default expand_path in newer fsspec versions Pin fsspec to use default expand_path Sep 21, 2023
Copy link
Contributor

@Wauplin Wauplin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for iterating on the PR @mariosasko ! Removing code always feels good 😄
CI is failing on an unrelated test so we are good to merge 🎉

@Wauplin Wauplin merged commit 6c2ebcc into main Sep 21, 2023
15 of 16 checks passed
@Wauplin Wauplin deleted the use-default-expand_path branch September 21, 2023 15:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants