Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move metadata handling to a class (updated) #839

Merged
merged 51 commits into from
Nov 4, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
9d6d07d
fix conflicts
grlee77 Jun 23, 2021
1e5ed1a
cleanup naming
Carreau Feb 27, 2021
6ca9631
zip move
Carreau Feb 27, 2021
b3db0f1
fix erasability test
Carreau Feb 27, 2021
1db3be2
test for warning
Carreau Feb 27, 2021
8d28678
please flake
Carreau Mar 10, 2021
b5240e4
remove uncovered lines
Carreau Mar 10, 2021
b7aa2d8
remove uncovered lines in tests
Carreau Mar 10, 2021
30afcbb
pragma no cover for exceptional case
Carreau Mar 10, 2021
8136713
minor docstring fixes
grlee77 Jun 23, 2021
2a95219
Merge branch 'master' into base-store-v2
grlee77 Jun 23, 2021
e38a968
pep8 fix
grlee77 Jun 23, 2021
eb2a6da
avoid NumPy 1.21.0 due to https://github.com/numpy/numpy/issues/19325
grlee77 Jun 24, 2021
3c7e90e
Merge remote-tracking branch 'upstream/master' into base-store-v2
grlee77 Jun 25, 2021
e2c57a9
Merge remote-tracking branch 'upstream/master' into base-store-v2
grlee77 Sep 10, 2021
6796f79
move Store class and some helper functions to zarr._storage.store
grlee77 Sep 10, 2021
d0a1b00
BUG: ABSStore should inherit from Store
grlee77 Sep 10, 2021
3d39d8c
pep8 fix
grlee77 Sep 10, 2021
c2d6159
Merge branch 'master' into base-store-v2
joshmoore Sep 22, 2021
5f7c7f5
TST: make CustomMapping a subclass of Store
grlee77 Sep 22, 2021
7e1e92a
Merge remote-tracking branch 'upstream/master' into base-store-v2
grlee77 Sep 22, 2021
b72be38
Merge branch 'base-store-v2' of github.com:grlee77/zarr-python into b…
grlee77 Sep 22, 2021
52deac0
update version mentioned in Store docstring
grlee77 Sep 22, 2021
a7cc4db
update version mentioned in warning message
grlee77 Sep 22, 2021
c59f3be
use Store._ensure_store in Attributes class
grlee77 Sep 22, 2021
a9a98a9
TST: add Attributes test case ensuring store gets coerced to a Store
grlee77 Sep 22, 2021
0a6d923
use Store._ensure_store in normalize_store_arg
grlee77 Sep 22, 2021
a40ed54
TST: make sure high level creation functions also work when passed a …
grlee77 Sep 22, 2021
3fa6f51
TST: add test case with group initialized from dict
grlee77 Sep 22, 2021
0f34ad2
TST: add test case with Array initialized from dict
grlee77 Sep 22, 2021
52fe871
change CustomMapping back to type object, not Store
grlee77 Sep 22, 2021
cb1c5f0
pep8 fixes
grlee77 Sep 24, 2021
6ce0981
update/fix new hierarchy test case to complete code coverage
grlee77 Sep 24, 2021
bb2e8f9
Merge remote-tracking branch 'origin/master' into pr-789
joshmoore Oct 19, 2021
82f7376
create a BaseStore parent for Store
grlee77 Oct 20, 2021
3481ea4
Convert functions in zarr.meta to class methods
grlee77 Sep 24, 2021
9ad4ef8
Add a _metadata_class attribute to the Store class
grlee77 Sep 24, 2021
4cd0416
Use _metadata_class attribute in test_storage.py
grlee77 Sep 24, 2021
8c385f6
remove unused imports
grlee77 Sep 24, 2021
fce042c
remove unnecessary hasattr checks
grlee77 Sep 24, 2021
9ed426f
test migrate1to2 with store=dict() and store=KVStore(dict())
grlee77 Sep 24, 2021
c6b5d52
pep8: remove unused imports
grlee77 Sep 24, 2021
c171c2e
misc
grlee77 Oct 20, 2021
26a9cef
s
grlee77 Oct 21, 2021
87dc567
remove unused FLOAT_FILLS
grlee77 Oct 21, 2021
ca4db80
mypy fixes
grlee77 Oct 21, 2021
15d4d78
sync metadata methods with current master branch
grlee77 Oct 21, 2021
c7c2624
flake8
grlee77 Oct 21, 2021
9ba6e73
Merge remote-tracking branch 'upstream/master' into metadata-classes-v2
grlee77 Oct 21, 2021
4588c01
restore is_erasable check to rmdir function
grlee77 Oct 21, 2021
fff210a
Merge remote-tracking branch 'upstream/master' into metadata-classes-v2
grlee77 Oct 21, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions zarr/_storage/store.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
from collections.abc import MutableMapping
from typing import Any, List, Optional, Union

from zarr.meta import Metadata2
from zarr.util import normalize_storage_path

# v2 store keys
Expand Down Expand Up @@ -32,6 +33,8 @@ class BaseStore(MutableMapping):
_writeable = True
_erasable = True
_listable = True
_store_version = 2
_metadata_class = Metadata2

def is_readable(self):
return self._readable
Expand Down Expand Up @@ -114,6 +117,7 @@ class Store(BaseStore):
.. added: 2.11.0

"""

def listdir(self, path: str = "") -> List[str]:
path = normalize_storage_path(path)
return _listdir_from_keys(self, path)
Expand Down
3 changes: 1 addition & 2 deletions zarr/attrs.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
from collections.abc import MutableMapping

from zarr.meta import parse_metadata
from zarr._storage.store import Store
from zarr.util import json_dumps

Expand Down Expand Up @@ -40,7 +39,7 @@ def _get_nosync(self):
except KeyError:
d = dict()
else:
d = parse_metadata(data)
d = self.store._metadata_class.parse_metadata(data)
return d

def asdict(self):
Expand Down
5 changes: 2 additions & 3 deletions zarr/core.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,6 @@
is_scalar,
pop_fields,
)
from zarr.meta import decode_array_metadata, encode_array_metadata
from zarr.storage import array_meta_key, attrs_key, getsize, listdir, BaseStore
from zarr.util import (
all_equal,
Expand Down Expand Up @@ -210,7 +209,7 @@ def _load_metadata_nosync(self):
else:

# decode and store metadata as instance members
meta = decode_array_metadata(meta_bytes)
meta = self._store._metadata_class.decode_array_metadata(meta_bytes)
self._meta = meta
self._shape = meta['shape']
self._chunks = meta['chunks']
Expand Down Expand Up @@ -267,7 +266,7 @@ def _flush_metadata_nosync(self):
compressor=compressor_config, fill_value=self._fill_value,
order=self._order, filters=filters_config)
mkey = self._key_prefix + array_meta_key
self._store[mkey] = encode_array_metadata(meta)
self._store[mkey] = self._store._metadata_class.encode_array_metadata(meta)

@property
def store(self):
Expand Down
4 changes: 1 addition & 3 deletions zarr/hierarchy.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@
GroupNotFoundError,
ReadOnlyError,
)
from zarr.meta import decode_group_metadata
from zarr.storage import (
BaseStore,
MemoryStore,
Expand Down Expand Up @@ -134,8 +133,7 @@ def __init__(self, store, path=None, read_only=False, chunk_store=None,
except KeyError:
raise GroupNotFoundError(path)
else:
meta = decode_group_metadata(meta_bytes)
self._meta = meta
self._meta = self._store._metadata_class.decode_group_metadata(meta_bytes)

# setup attributes
akey = self._key_prefix + attrs_key
Expand Down
Loading