Create a Base store class for Zarr Store (update) #789
Conversation
add assert statements to test_capabilities
|
The |
|
Yes, I just started looking into it. I can reproduce it locally after upgrading to NumPy 1.21. It does not occur for NumPy 1.20.3 |
|
I opened numpy/numpy#19325 regarding this segfault |
Codecov Report
@@ Coverage Diff @@
## master #789 +/- ##
=======================================
Coverage 99.94% 99.94%
=======================================
Files 31 32 +1
Lines 11133 11175 +42
=======================================
+ Hits 11127 11169 +42
Misses 6 6
|
|
cc @martindurant @rabernat @jhamman @shoyer @shikharsg (in case any of you have thoughts here 🙂 this will affect how other store implementations work 😉) |
|
Also cc @TomAugspurger 🙂 |
|
Looking ahead to adding v3 spec support: In terms of class members for a
The |
|
Green now except for the removed build (==1.16.4) which can be ignored. |
|
Clarification: if we open an array or group, and pass a dict-like as before, it gets wrapped to become a Store, right? |
ensures open_array, etc can work when the user supplies a dict
want to test the non-Store code path in _ensure_store
There was a problem hiding this comment.
As mentioned on call, this is looking good. Thanks, @grlee77. Probably the most important remaining step is to get the release blurb so that we can all agree on the degree of breakage and have a comprehensive list of what (if anything) will no longer take dicts. Additionally some documentation may be necessary around the "pass by reference" semantics.
Propose then to target this for a 2.11
|
Solved the (minor) merge conflict, @grlee77. From my side, I'd say let's try to get this into a RC as early as tomorrow. (Or someone else can do it this evening. I'm going to 🛏️ ) Up to you if you think we should also try to get metadata handling in. (Release blurb would still be appreciated.) |
I think we may as well also merge the metadata handling for the RC. Let's discuss in the community call. One remaining doubt about the approach here is the presence of A future StoreV3 class could then inherit from BaseStore instead of StoreV2, so that it would not have to inherit additional methods that are not part of the spec. That said, at the moment many high-level functions use |
BaseStore does not have the listdir or rmdir methods cleaned up some type declerations, making sure mypy passes
Otherwise the save_array doc example fails to write to a ZipStore
| @@ -133,7 +133,7 @@ def rmdir(store: StoreLike, path: Path = None): | |||
| this will be called, otherwise will fall back to implementation via the | |||
| `Store` interface.""" | |||
| path = normalize_storage_path(path) | |||
| if hasattr(store, "rmdir"): | |||
| if hasattr(store, "rmdir") and store.is_erasable(): # type: ignore | |||
There was a problem hiding this comment.
Not a blocker for getting this into a 2.11 alpha build but is the hasattr not now redundant?
There was a problem hiding this comment.
Not quite yet. There are calls to rmdir from _init_array_metadata (via init_array which still allows a MutableMapping for the store). This is why I put store as StoreLike in the type annotation, but then I had to add "# type: ignore" on this line or mypy complained that MutableMappings do not have an is_erasable attribute.
Use a base Store class instead of just MutableMapping
I am resuming the work @Carreau started in #612 here. This is mostly just a rebase of that PR with a small new commit in 8136713. Matthias and I met earlier today and we decided that starting with this PR should make it easier to add zarr-v3 support without trying to implement an entirely independent set of classes. The goal is to reduce code duplication between v2 and v3 where possible (None of the changes in this PR are specific to v3 yet)
Note that I updated the version string text to
2.9.0in a couple of places, but that may need to be updated again depending on when this is merged.TODO: