Creating ZipStore with file-like object

[oeway]

The current ZipStore implementation assumes the input is always a path which becomes a bit limiting when dealing with in-memory file or virtual file object (e.g. in Pyodide in the browser). The virtual file object support is crucial for making zarr library useful in the browser environment, since the emscripten file system itself are rather limited at the moment.

It would be nice if we can allow passing a file-like object. I did a test with a small modification in the init function and it seems to be working nicely in Pyodide/JupyterLite:

from threading import Lock, RLock
import zipfile

class InMemoryZipStore(zarr.ZipStore):
    def __init__(self, path, compression=zipfile.ZIP_STORED, allowZip64=True, mode='a',
                 dimension_separator=None):

        # store properties
        self.path = None # TODO: This need to be handled properly for os.PathLike or file-like object
        self.compression = compression
        self.allowZip64 = allowZip64
        self.mode = mode
        self._dimension_separator = dimension_separator

        # Current understanding is that zipfile module in stdlib is not thread-safe,
        # and so locking is required for both read and write. However, this has not
        # been investigated in detail, perhaps no lock is needed if mode='r'.
        self.mutex = RLock()

        # open zip file
        self.zf = zipfile.ZipFile(path, mode=mode, compression=compression,
                                  allowZip64=allowZip64)

I will maintain this piece of code somewhere for now, but it would be great if we can support this from upstream and eventually have it in pyodide.

[joshmoore]

Hey @oeway. Thanks for the suggestion! In your mind is what we need the addition of an optional argument for path? Seems like the only possible downside of that would be if someone was expecting the exception from os.path.abspath(path) to detect None.

[oeway]

Hey @oeway. Thanks for the suggestion! In your mind is what we need the addition of an optional argument for path? Seems like the only possible downside of that would be if someone was expecting the exception from os.path.abspath(path) to detect None.

How about we rename the first argument path to file_or_path? Then we try to infer the path based on the types:

if isinstance(file_or_path, (str, os.PathLike)):
    self.path = file_or_path
else: # file-like object
    if hasattr(file_or_path, 'name'):
        # Normal file object has the name property which contains the path
        self.path = file_or_path.name
    else:
        self.path = "" # the default is empty? or None?
 
self.zf = zipfile.ZipFile(file_or_path, mode=mode, compression=compression,
                                 allowZip64=allowZip64)

[joshmoore]

I wouldn't suggest renaming since existing code could be using ZipStore(path=). If permitting a FileLike for something named "path" is too confusing, then perhaps we add a static initializer ZipStore.from_file(file, *args, **kwargs) (alternative names: wrap(), ...)

[oeway]

I wouldn't suggest renaming since existing code could be using ZipStore(path=). If permitting a FileLike for something named "path" is too confusing, then perhaps we add a static initializer ZipStore.from_file(file, *args, **kwargs) (alternative names: wrap(), ...)

Good point! ZipStore.from_file sounds good to me!

[jakirkham]

Think it would be ok to just accept ZipFile instances in path. We can just clarify the docs on this point.

We have similar flexibility in Dask.

[jakirkham]

As a side note we might consider using position only and keyword only arguments in Zarr 3 to avoid these issues around renaming arguments.

[joshmoore]

Issue created, @jakirkham.

Think it would be ok to just accept ZipFile instances in path.

So the logic would be:

• is_zip_file()
• is_file_like()
• is_path_or_string_like()

Or would you just pass a ZipFile, @oeway ?

[oeway]

Hi, for my case it's a fake file object making http requests to a file with range header, so not a zipfile.

[jakirkham]

Thanks Josh! 😄

Intriguing would be curious to look at the object if you don't mind sharing 🙂

Does it support one of the file ABCs? Or could it? If so, then it would be pretty easy to check if it isinstance of one of those and use it

[oeway]

Intriguing would be curious to look at the object if you don't mind sharing 🙂

Does it support one of the file ABCs? Or could it? If so, then it would be pretty easy to check if it isinstance of one of those and use it

Sure, here it is: https://github.com/imjoy-team/imjoy-rpc/blob/af739ec829d984da35bc5b87b93aa1a553944fe3/python/imjoy_rpc/utils.py#L672-L842

It's a class inherit from io.IOBase, I guess fsspec also has something similar.

[jakirkham]

cc @martindurant (in case this is of interest especially given our ZIP discussion recently)

[martindurant]

I would point out that fsspec supports a zarr-compatible key-value store over any fs it can instantiate, including ZIP and in-memory. You can pass these directly to zarr.

# this bit just for example
data = fsspec.filesystem("http").cat("https://github.com/fsspec/filesystem_spec/archive/refs/heads/master.zip")
fsspec.filesystem("memory").pipe("mem.zip", data)

# make storage mapper like this
store = fsspec.get_mapper("zip::memory://mem.zip")

[martindurant]

Obviously the thing above won't work in pyidide, since the HTTP part uses aiohttp - I should have said. But you can replace this part or wait until I write HTTP-for-pyodide (maybe next week?!).

The separate issue I was talking with @jakirkham about, is that accessing this way uses a file - so access to zarr chunks would be always serial. With kerchunk, we can index the ZIP (i.e., translate the existing index embedded in the file) and attain concurrent access to zarr chunks. This is pretty simple, but not yet done, but should work in pyodide/pyscript too.

[TomasPuverle]

I was just looking at the same problem. Creating from another ZipFile, or file (e.g. io.BytesIO) would be super useful.

I was actually looking at potentially contributing this but there are a number of places in ZipFile where the .zf is recreated from self.path again. That may be an issue if en external file (instead of a path) is provided, no?