Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Broken tests due to upgrade of fsspec #956

Closed
ejguan opened this issue Jan 20, 2023 · 2 comments
Closed

Broken tests due to upgrade of fsspec #956

ejguan opened this issue Jan 20, 2023 · 2 comments

Comments

@ejguan
Copy link
Contributor

ejguan commented Jan 20, 2023

🐛 Describe the bug

After fsspec is updated 2023.1.0 from 2022.11.0 , this test becomes broken due to AttributeError: 'dict' object has no attribute 'startswith'

Example failed workflow: https://github.com/pytorch/data/actions/runs/3969171216/jobs/6803298832

Versions

main branch
@NivekT
Copy link
Contributor

NivekT commented Jan 20, 2023

I have identified the issue. In the old version, the default argument is details=False for method ls, thus it returned a list of str. In the new version, the default is details=True, which returns list of dict.

Source code:
https://github.com/fsspec/filesystem_spec/blob/2023.1.0/fsspec/implementations/memory.py#L34

I will open a PR to fix soon.

@ejguan
Copy link
Contributor Author

ejguan commented Jan 20, 2023

Thank you!

@NivekT NivekT mentioned this issue Jan 20, 2023
Closed
SvenDS9 pushed a commit to SvenDS9/PytorchData that referenced this issue Jan 24, 2023
@NivekT @SvenDS9
Fixing fsspec DataPipe to be compatible with new version of fsspec (p…
08599ad 
…ytorch#957)

Summary:
Pull Request resolved: pytorch#957

Fixes pytorch#956

Context:

In the old version of `fsspec`, the default argument is `detail=False` for method `ls` for the filesystem `memory`, thus it returned a list of `str`. In the new version, the default is `detail=True`, which returns list of `Dict`.

Source code:
https://github.com/fsspec/filesystem_spec/blob/2023.1.0/fsspec/implementations/memory.py#L34

Test Plan: Imported from OSS

Reviewed By: ejguan

Differential Revision: D42646692

Pulled By: NivekT

fbshipit-source-id: 41258a5a16487303a7e6895c484794c1f1d66e39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@NivekT @ejguan