Create Zarr in parallel by appending to an array which size and indices are not known: valid use case? #2140
Unanswered
NicolasM-Forsk
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hello
I'd like to create a compact zarr containing an item list which is not determined by advance.
The zarr contains an (item_id, y, x) shape, chunked by item_id (ex 8), resulting in 8 raster items being stored in 1 file of the item_data array.
I want to append raster item 8 by 8 from different celery workers with compression enabled, and I see that item_id/.zarray and item_id/0 are updated each time.
I wonder if I can lock only these files, and if this is a valid use case for Zarr V2 or Zarr V3.
Beta Was this translation helpful? Give feedback.
All reactions