Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add N5 Support #309

Merged
merged 76 commits into from
Feb 26, 2019
Merged
Show file tree
Hide file tree
Changes from 48 commits
Commits
Show all changes
76 commits
Select commit Hold shift + click to select a range
92eab8c
Add N5Store for paths ending in '.n5'
funkey Oct 13, 2018
8f1d721
Add N5 chunk headers via N5ChunkWrapper as a codec
funkey Oct 14, 2018
f21ae8e
Add compressor support to N5ChunkWrapper
funkey Oct 14, 2018
c74c79f
Invert axis order for dimensions and blockSize N5 attributes
funkey Oct 14, 2018
38675fe
Convert raw chunks to big endian in N5ChunkWrapper
funkey Oct 14, 2018
89b95dd
Consolidate compressor conversion according to N5 spec 1.0.0
funkey Oct 14, 2018
89ca5c9
Always change byte-order to big endian in N5ChunkWrapper
funkey Oct 14, 2018
deb8c2d
Cleanup unused code fragments in n5.py
funkey Oct 14, 2018
1e84925
Transparently inject N5ChunkWrapper in array meta data
funkey Oct 14, 2018
e12e32b
Map array attributes fill_value, order, and filters
funkey Oct 16, 2018
0e416f4
Use numpy's tobytes for raw encoding in N5ChunkWrapper
funkey Oct 16, 2018
cba3e56
Add gzip compressor support to N5Storage
funkey Oct 16, 2018
719bf5b
Support N5's partial chunks at dataset boundaries
funkey Oct 16, 2018
59cb72e
Add unit test stub for N5Store
funkey Oct 16, 2018
1534e0c
Comply with zarr meta key names in N5Store
funkey Oct 18, 2018
a4939e7
Overwrite __delitem__ and listdir in N5Store
funkey Oct 18, 2018
ae95a38
Allow fill_value==None in N5Store
funkey Oct 18, 2018
a6a6c15
Allow compressor_config to be None in N5Store
funkey Oct 18, 2018
cdffeb7
Support older np.byteswap API
funkey Oct 18, 2018
2557661
Expose N5Store in zarr top-level package
funkey Oct 18, 2018
f04eaf7
Refactor attribute conversion between zarr and N5
funkey Oct 18, 2018
f471e7d
Delegate __getitem__ to parent class for unrecognized keys
funkey Oct 18, 2018
3abf68c
Fix runtime errors in N5Store
funkey Oct 18, 2018
d22aabb
Invert coords only for chunk keys in N5Store
funkey Oct 18, 2018
dd07cc2
Add no cover pragmas to N5Store
funkey Oct 18, 2018
2a17ae2
Use _load_n5_attrs where possible in N5Store
funkey Oct 18, 2018
08b11a7
Remove conditional that is always true in N5Store
funkey Oct 18, 2018
ca4c516
Add tests for N5Store
funkey Oct 18, 2018
82c7bc3
Add N5Store entry to release notes
funkey Oct 18, 2018
eb5c17c
Ensure str type for json.loads in N5Store
funkey Oct 20, 2018
9ddf849
Fix reading of partial N5 chunks
funkey Oct 21, 2018
50b6555
Merge 'zarr-developers/master' into 'funkey/master'
jakirkham Nov 30, 2018
9fe09ce
Bump the Numcodecs requirement to 0.6.1
jakirkham Nov 30, 2018
b8c40b8
Enable testing of GZip as well
jakirkham Nov 30, 2018
1b04af4
Change `_ensure_str` to `ensure_str`
jakirkham Nov 30, 2018
a6ca2e7
Switch from `buffer_copy` to `ndarray_copy`
jakirkham Nov 30, 2018
fb662af
Merge 'zarr-developers/master' into 'funkey/master'
jakirkham Dec 17, 2018
d0a1c93
Raise more informative TypeError for unsupported dtypes in N5
funkey Dec 19, 2018
e3546e0
Skip test_structured_array_* for N5
funkey Dec 19, 2018
4510da5
Fix flake8 errors for N5 store and related tests
funkey Dec 19, 2018
28761ce
Use `struct.pack` to convert integers to `bytes`
jakirkham Dec 20, 2018
7a7b83c
Use `struct.unpack` to convert `bytes` to integers
jakirkham Dec 20, 2018
fa71958
Optionally use LZMA if it is available
jakirkham Dec 20, 2018
325a3db
Ensure `str` before using `json.loads`
jakirkham Dec 20, 2018
dd2100a
Ensure JSON output is the same for Python 2/3
jakirkham Dec 20, 2018
cef6ea5
Fix remaining coveralls tests
funkey Jan 10, 2019
6ccde35
Merge 'zarr-developers/master' into 'funkey/master'
jakirkham Jan 16, 2019
7aa2a5d
Add N5 paragraph to tutorial
funkey Feb 4, 2019
11104e5
Fix typo in zarr/tests/test_storage.py
alimanfoo Feb 4, 2019
e8db9b8
Merge 'zarr-developers/master' into 'funkey/master'
jakirkham Feb 12, 2019
1586481
Move N5 containers entry to 2.3.0
jakirkham Feb 12, 2019
4bb046f
Add author info to release entry
jakirkham Feb 12, 2019
3bb9c24
Add experimental notes to N5 support
jakirkham Feb 12, 2019
8bac1db
Drop `mode` from `N5Store`
jakirkham Feb 13, 2019
6abc758
Test N5 without filters and with
jakirkham Feb 13, 2019
8b213c4
Test that using Fortran order with N5 errors
jakirkham Feb 13, 2019
a6d6480
Test structured array cases raise TypeError on N5
jakirkham Feb 14, 2019
6bb94d0
Test object arrays raise with N5
jakirkham Feb 14, 2019
6aaf2eb
Add API docs for N5
jakirkham Feb 14, 2019
7713d70
Drop unneeded call to `close`
jakirkham Feb 14, 2019
09d5351
Merge 'zarr-developers/master' into 'funkey/master'
jakirkham Feb 16, 2019
a4eaba4
Drop storage tests integrated into array tests
jakirkham Feb 16, 2019
0efe4bd
Drop unused imports (reported by flake8)
jakirkham Feb 16, 2019
c537eef
Convert user-facing assertions to errors
jakirkham Feb 23, 2019
5168d47
Use non-deprecated `.warning(...)` instead
jakirkham Feb 23, 2019
9f45993
Use warnings instead of logging
jakirkham Feb 23, 2019
86f5f82
Drop unused `logging` import
jakirkham Feb 23, 2019
8548f30
Check N5Store RuntimeWarnings for some compressors
jakirkham Feb 23, 2019
ca17069
Limit `RuntimeWarning` check to array creation
jakirkham Feb 23, 2019
87c9864
Test that N5Store raises for certain attributes
jakirkham Feb 23, 2019
c711246
Test `fill_values` with N5Store-backed arrays
jakirkham Feb 23, 2019
b908b72
Convert Blosc blocksize check back to an assert
jakirkham Feb 23, 2019
149f3a4
Handle loading and storing XZ data for N5
jakirkham Feb 24, 2019
ba1298b
Test XZ support and update LZMA for N5Store arrays
jakirkham Feb 24, 2019
ed467e8
Remove LZ4 support for N5
jakirkham Feb 24, 2019
08009f5
Merge 'zarr-developers/master' into 'funkey/master'
jakirkham Feb 24, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/release.rst
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,10 @@ Enhancements
the existing :class:`zarr.storage.DirectoryStore` class but nests chunk files
for multidimensional arrays into sub-directories. :issue:`155`, :issue:`177`.

* **New storage class for N5 containers**. The :class:`zarr.n5.N5Store` has been
added, which uses :class:`zarr.storage.NestedDirectoryStore` to support
reading and writing from and to N5 containers.
jakirkham marked this conversation as resolved.
Show resolved Hide resolved

* **New tree() method for printing hierarchies**. The ``Group`` class has a new
:func:`zarr.hierarchy.Group.tree` method which enables a tree representation of
a group hierarchy to be printed. Also provides an interactive tree
Expand Down
15 changes: 15 additions & 0 deletions docs/tutorial.rst
Original file line number Diff line number Diff line change
Expand Up @@ -729,6 +729,21 @@ group (requires `lmdb <http://lmdb.readthedocs.io/>`_ to be installed)::
>>> z[:] = 42
>>> store.close()

For compatibility with the `N5<https://github.com/saalfeldlab/n5`_ data format,
Zarr also provides an N5 backend. Similar to the zip storage class, an
:class:`zarr.n5.N5Store` can be instantiated directly::
jakirkham marked this conversation as resolved.
Show resolved Hide resolved

>>> store = zarr.N5Store('data/example.n5', mode='w')
jakirkham marked this conversation as resolved.
Show resolved Hide resolved
>>> root = zarr.group(store=store)
>>> z = root.zeros('foo/bar', shape=(1000, 1000), chunks=(100, 100), dtype='i4')
>>> z[:] = 42
>>> store.close()
jakirkham marked this conversation as resolved.
Show resolved Hide resolved

For convenience, the N5 backend will automatically be chosen when the filename
ends with `.n5`::

>>> root = zarr.open('data/example.n5', mode='w')

jakirkham marked this conversation as resolved.
Show resolved Hide resolved
Distributed/cloud storage
~~~~~~~~~~~~~~~~~~~~~~~~~

Expand Down
1 change: 1 addition & 0 deletions zarr/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,5 +14,6 @@
from zarr.convenience import (open, save, save_array, save_group, load, copy_store,
copy, copy_all, tree, consolidate_metadata,
open_consolidated)
from zarr.n5 import N5Store
from zarr.errors import CopyError, MetadataError, PermissionError
from zarr.version import version as __version__
3 changes: 3 additions & 0 deletions zarr/creation.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
from zarr.core import Array
from zarr.storage import (DirectoryStore, init_array, contains_array, contains_group,
default_compressor, normalize_storage_path, ZipStore)
from zarr.n5 import N5Store
from numcodecs.registry import codec_registry
from zarr.errors import err_contains_array, err_contains_group, err_array_not_found

Expand Down Expand Up @@ -132,6 +133,8 @@ def normalize_store_arg(store, clobber=False, default=dict):
if store.endswith('.zip'):
mode = 'w' if clobber else 'a'
return ZipStore(store, mode=mode)
elif store.endswith('.n5'):
return N5Store(store)
else:
return DirectoryStore(store)
else:
Expand Down
Loading