Suppressing ZipFile duplication warning

[jakirkham]

Python's ZipFile allows writing duplicate files and does this by default. When it writes a duplicate file, it raises a UserWarning. This occurs for each file and is a bit noisy. As there doesn't seem to be a standard way of solving this, would recommend that we simply suppress this warning. Combining this with a resolution to issue ( https://github.com/alimanfoo/zarr/issues/128 ), would ensure that deduplication already occurs so the warning is no longer relevant.

Edit: Added link to Python bug after the fact.

[jakirkham]

Proposed fix in PR ( https://github.com/alimanfoo/zarr/pull/130 ).

[alimanfoo]

FWIW I think this needs some consideration. If duplicate files are being written into a zip file, and this is happening often, then it is likely that something rather sub-optimal is happening. In the pathological case, a user could be storing many multiples of the actual data for an array without realising, then wonder why the zip file is so large.

Writing directly to a zip file is really only efficient if the array or arrays being stored in the zip store are written only once, and write operations can be perfectly aligned with chunk boundaries, in which case no duplicate chunk files will ever get created. This can be achieved if an array is created with zarr.array(data=data, ...) or with something like z = zarr.zeros(...); z[:] = data, as in either case zarr internally aligns the write operations with chunk boundaries.

If the use case requires that data are written and then overwritten, and/or that write operations cannot be aligned with chunk boundaries, then a better approach is probably to initially store the data using DirectoryStore. Then when all writing has finished, the containing directory can be stored into a zip file using the standard command line zip utility. The resulting zip file can then be read directly by zarr without having to unpack. This should be at least as efficient in general as deduplicating a zip file on close by copying to a new zip file.

[jakirkham]

It's a fair point honestly. Though I really do like the idea of operating on a single file that includes all of the array data. Zip is nice as it is easy to inspect. Though I don't feel wedded to it given the issues I'm already experiencing by playing with it. Is there some other reasonable storage type that we could add to Zarr that wouldn't have these limitations?

[alimanfoo]

I don't know of anything better. Tar is worse apparently as it doesn't support random access. It would be possible (if a little twisted) to use an HDF5 file, however you'd lose the ability to do multi-threaded reads (which seem to work on a zip store surprisingly). cc @mrocklin.

[mrocklin]

When I looked into this a long while ago I found that yes, there are other single-file compression formats out there that support random access, but none seemed common place. Generally speaking writing variable sized byte blocks into a single file is a hard problem.

Another alternative would be an embedded key-value database. Zict has a MutableMapping for LMDB. https://github.com/dask/zict/blob/master/zict/lmdb.py This would be a single directory rather than a single file, but balances large writes and many small writes well.

[alimanfoo]

Maybe shelve is an option? It supports the MutableMapping interface so you could probably just use a Shelf as a store without needing to write any new code...

[alimanfoo]

Ha, @mrocklin you get much kudos for advocating the MutableMapping interface...

Python 3.5.3 | packaged by conda-forge | (default, Jan 23 2017, 19:01:48) 
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import zarr
>>> zarr.__version__
'2.1.4'
>>> import numpy as np
>>> import shelve
>>> store = shelve.open('shelf')
>>> z = zarr.array(data=np.arange(1000), store=store, chunks=100)
>>> np.all(z[:] == np.arange(1000))
True
>>> sorted(store)
['.zarray', '.zattrs', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9']

[mrocklin]

Hooray standard interfaces!

[alimanfoo]

Looks like shelve supports multi-threaded reads...

In [1]: import zarr

In [2]: import shelve

In [4]: store = shelve.open('shelf')

In [5]: z = zarr.zeros(10000000000, dtype='i4', chunks=1000000, store=store, overwrite
   ...: =True)

In [7]: %time z[:] = 42
CPU times: user 18.1 s, sys: 808 ms, total: 18.9 s
Wall time: 5.55 s

In [8]: !ls -lh shelf*
-rw-r--r-- 1 aliman aliman   25 Feb 24 23:29 shelf.bak
-rw-r--r-- 1 aliman aliman 199M Feb 24 23:29 shelf.dat
-rw-r--r-- 1 aliman aliman 259K Feb 24 23:29 shelf.dir

In [9]: z
Out[9]: 
Array((10000000000,), int32, chunks=(1000000,), order=C)
  nbytes: 37.3G; initialized: 10000/10000
  compressor: Blosc(cname='lz4', clevel=5, shuffle=1)
  store: DbfilenameShelf

In [10]: import dask.array as da

In [11]: d = da.from_array(z, chunks=z.chunks)

In [14]: %time d.mean().compute()
CPU times: user 1min 13s, sys: 1.25 s, total: 1min 14s
Wall time: 12.6 s
Out[14]: 42.0

[alimanfoo]

A BerkeleyDB hash table would probably be another option.

[jakirkham]

The fact that Zarr is using a MutableMapping seems like a very useful thing. Not that I have looked into this at all, but I wonder if there are any Key-Value Stores that would work well here.

[mrocklin]

See note above about LMDB, for which there is a MutableMapping in zict

…

On Fri, Feb 24, 2017 at 6:38 PM, jakirkham ***@***.***> wrote: The fact that Zarr is using a MutableMapping seems like a very useful thing. Not that I have looked into this at all, but I wonder if there are any Key-Value Stores that would work well here. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/alimanfoo/zarr/issues/129#issuecomment-282434914>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszFkoGcuXmnALrIAYG1CLCzI38CG9ks5rf2njgaJpZM4MLhEG> .

[alimanfoo]

Yes, any key-value store should be an option.

[jakirkham]

Thanks for the feedback. I'll give this some more thought.