Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[v3] remote store support (s3, gcs, azure, http) #1757

Closed
3 tasks
jhamman opened this issue Apr 5, 2024 · 7 comments · Fixed by #1785
Closed
3 tasks

[v3] remote store support (s3, gcs, azure, http) #1757

jhamman opened this issue Apr 5, 2024 · 7 comments · Fixed by #1785
Assignees
Labels
V3 Affects the v3 branch
Milestone

Comments

@jhamman
Copy link
Member

jhamman commented Apr 5, 2024

The v3 branch currently has an initial (but broken) implementation of a remote store. This issue tracks getting that operational so we car start using v3 against s3, gcs, azure, http, etc.

TODOs:

  • rename to FsspecStore
  • limit store to only fsspec implementation that are async-friendly
  • decide on approach to testing

@martindurant - any chance you can lend a hand on this one? I optimistically put you in as the assignee but let us know if this is not something you can engage on.

Blocked by #1755

@jhamman jhamman added the V3 Affects the v3 branch label Apr 5, 2024
@jhamman jhamman added this to the 3.0.0.alpha milestone Apr 5, 2024
@jhamman jhamman moved this to Todo in Zarr-Python - 3.0 Apr 5, 2024
@martindurant
Copy link
Member

Hi, I'm watching. I can help, but I don't have a huge number of hours, of course.

I notice the "blocked by" - and there has been no progress on that issue yet,

@d-v-b
Copy link
Contributor

d-v-b commented Apr 10, 2024

Personally I don't think we need to wait for a final store API before implementing remote storage.
We already implement local storage for v3, so we should just make a remote API that matches it.

Any friction generated in that process can go towards shaping the final API :)

@martindurant
Copy link
Member

Is the use of universal_pathlib desired/required?

@d-v-b
Copy link
Contributor

d-v-b commented Apr 11, 2024

@martindurant universal_pathlib was carried over from zarrita; maybe @normanrz can weigh in on the pros and cons of keeping it in.

@normanrz
Copy link
Contributor

I think upath is quite useful to compose paths. It would be great if the FsspecStore would support opening from a UPath object. Maybe it should be an optional dependency, though.

@martindurant
Copy link
Member

OK, I can have it optional. Honestly I doubt we'll see it much, since typical invocation would be like zarr.open("s3://bucket/data.zarr") (which already works).

@normanrz
Copy link
Contributor

UPath becomes interesting if you store some storage_options like this and use it in different places:

path = UPath("s3://bucket", key='some', secret='credentials')
z1 = zarr.open(path / "data.zarr")
z2 = zarr.open(path / "other.zarr")

@jhamman jhamman moved this from Todo to In Progress in Zarr-Python - 3.0 May 7, 2024
@github-project-automation github-project-automation bot moved this from In Progress to Done in Zarr-Python - 3.0 Jun 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
V3 Affects the v3 branch
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

4 participants