How to configure zarr to store/compress arrays deterministically?

[jjyyxx]

I recently switched from simple np.savez to zarr for more flexibility with generating a big dataset. The only thing currently missing is that the compressed array is not deterministic given same input. This leads to certain difficulty in our version maintenance. So, where does the randomness come from? Is it possible to configure zarr to store/compress arrays deterministically, like setting a seed?

[rabernat]

The only thing currently missing is that the compressed array is not deterministic given same input.

Can you explain what you mean by "not deterministic"? Most of the popular zarr compressors are lossless compressors. Can you share the details of your configuration (what dtype, compressor and parameters) and perhaps a reproducible example.

[jjyyxx]

Unless it is platform- or hardware-dependent, larger arrays seem to cause non-determinism.

import zarr

def test_determinism_once(size: int):
    store1 = zarr.MemoryStore()
    store2 = zarr.MemoryStore()

    arr1 = zarr.create(store=store1, shape=size, chunks=False)
    arr1[:] = 2

    arr2 = zarr.create(store=store2, shape=size, chunks=False)
    arr2[:] = 2

    return dict(store1) == dict(store2)

def test_determinism(size: int, n: int = 1000):
    percentage = sum(test_determinism_once(size) for _ in range(n)) / n
    print(f'{percentage:>6.1%} of {n} runs are deterministic for size {size}')

test_determinism(1000)
test_determinism(262143)
test_determinism(262144)  # 2**18
test_determinism(1000000)

prints

100.0% of 1000 runs are deterministic for size 1000
100.0% of 1000 runs are deterministic for size 262143
 53.6% of 1000 runs are deterministic for size 262144
  0.0% of 1000 runs are deterministic for size 1000000

Guess such non-determinism may be multi-threading related.

[rabernat]

Thanks for this great example!

It appears to be compressor dependent. The default compressor is Blosc(cname='lz4', clevel=5, shuffle=SHUFFLE, blocksize=0)

If I try with a different compressor, I get 100% determinism.

from numcodecs import Zstd
compressor = Zstd(level=1)
arr1 = zarr.create(store=store1, shape=size, chunks=False, compressor=compressor)
arr2 = zarr.create(store=store2, shape=size, chunks=False, compressor=compressor)

(You'll also note that zstd is much slower!)

It could be that the lz4 compression algorithm itself is non-deterministic or involves some sort of seed. I would do some research on that. It's not specifically a zarr issue I don't think. Zarr is just sending the bytes to numcodecs.

edit: also possibly related to blosc multithreading.

[jjyyxx]

A bit of context: I was originally using np.savez_compressed to store my dataset, but it uses ZIP_DEFLATED which is extremely slow for large arrays and does not support chunks. So, I switched to zarr to get over 10 times speed up. The size on disk is 20% larger compared to ZIP_DEFLATED, but I'm OK with that. (So, switching back to zstd is not an option to me)

[rabernat]

Try configuring blosc to only use a single thread.

[ktpolanski]

I just ran into something similar and found this discussion very helpful. I initially spotted the nondeterminism in AnnData's zarr storage of a small data frame, and similarly noticed it escalating as I tried a larger input later. I can confirm that running blosc with a single thread led to everything behaving deterministically.

I set up the single threaded blosc like so, just in case there's a better way to do this:

from numcodecs import Blosc
import numcodecs.blosc
numcodecs.blosc.set_nthreads(1)
compressor = Blosc()