Is it possible to read a compressed Zarr store?

[bmcandr]

Is it possible to read a compressed Zarr store at this time?

---

I am attempting to read a simple Zarr store created by the xarray Python library like so:

ds = xarray.open_dataset('simple_xy.nc')
ds.to_zarr(<path>, consolidated=True)

The simple_xy.nc dataset being the file generated by the example F90 program simple_xy_wr.f90.

Dumping the Zarr store produces incorrect data values:

$ ncdump file://zlib.zarr#mode=zarr
netcdf simple_xy_xarray {
dimensions:
	x = 6 ;
	y = 12 ;
variables:
	int data(x, y) ;
data:

 data =
  70320386, 288, 288, 112, 20, 88, 16793343, 84148994, 151521030, 218893066,
    286265102, 353637138,
  421009174, 488381210, 555753246, 623125282, 690497318, 757869354,
    825241390, 892613426, 959985462, 1027357498, 1094729534, 1162101570,
  18246, 83886080, 5290752, 0, 0, 0, 0, 0, 0, 0, 0, 0,
  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ;
}

Reading the data with a modified version of simple_xy_rd.f90 produces the same values as above rather than the expected output:

$ ncdump simple_xy.nc
netcdf simple_xy {
dimensions:
	x = 6 ;
	y = 12 ;
variables:
	int data(x, y) ;
data:

 data =
  0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11,
  12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23,
  24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35,
  36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47,
  48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59,
  60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71 ;
}

Dumping a Zarr store created with xarray without compression produces the expected output so the issue appears to be due to the compression of the Zarr store. xarray/zarr seems to use the lz4 codec by default and I see there is a recently opened issue discussing support for this codec.

[DennisHeimbigner]

There is a rather circuitous path. But I need some info.
Is the data compressed using LZ4 or using Blosc wiht subcompressor LZ4?

[bmcandr]

Looks like Blosc with LZ4 subcompressor:

$ cat simple_xy_xarray.zarr/data/.zarray
{
    "chunks": [
        6,
        12
    ],
    "compressor": {
        "blocksize": 0,
        "clevel": 5,
        "cname": "lz4",
        "id": "blosc",
        "shuffle": 1
    },
    "dtype": "<i4",
    "fill_value": null,
    "filters": null,
    "order": "C",
    "shape": [
        6,
        12
    ],
    "zarr_format": 2
}

[DennisHeimbigner]

Ok. If you install c-blosc, then
the current master should allow you to read it.

[bmcandr]

Thanks, I'll give it a shot!

[edwardhartnett]

@DennisHeimbigner really? Did you add BLOSC and LZ4 to master?

[bmcandr]

Thanks for asking that question @edwardhartnett. Scouring the docs and GitHub discussions led me to believe it was not yet possible, prompting this discussion post. If it is possible, is there a guide or example to c-blosc with netcdf-fortran available? I have installed c-blosc, but I'm not sure how to use it with netcdf-fortran. Do I need to use it as an hdf5 plugin?

I appreciate any guidance you might be able to provide!

[DennisHeimbigner]

Sort of; they are not offically supported, but do work.
Actually, the LZ4 support is just that provided by c-blosc.
There is no e.g. nc_def_var_lz4, for example.

[DennisHeimbigner]

Let me make sure I understand.
You have a zarr dataset called simple_xy_array.zarr
that somehow was created using blosc+lz4, correct?
Your goal is get data from that zarr dataset. Correct?

[bmcandr]

You have a zarr dataset called simple_xy_array.zarr that somehow was created using blosc+lz4, correct?

Correct. The example Zarr dataset can be created by opening the file simple_xy.nc generated by this example program from the netcdf example collection with xarray and then writing as a Zarr store with default options:

ds = xarray.open_dataset('simple_xy.nc')
ds.to_zarr('simple_xy_array.zarr')

Your goal is get data from that zarr dataset. Correct?

Correct. I would like to read the data in with a Fortran program. I am exploring this capability because we are interested in reading data into our modeling framework that is contained in a Zarr store hosted on S3 that was created with a similar workflow.

[bmcandr]

@DennisHeimbigner, I am actually working with the NASA POWER team and they said they've shared a sample of their Zarr store with you which I see in the nczarr_test subdirectory of the netcdf-c repo (ref_power_901_constants.zip). Is there a test program in that repo that I demonstrates the process for reading their stores? I'm having trouble figuring out which test reads that file.

[bmcandr]

Update: I was able to successfully install the NetCDF-C library with Blosc support (via c-blosc) after changing line 1072 from IF(!MSVC) to IF(NOT MSVC) in CMakeLists.txt. All tests, including nczarr_test_run_specific_filters pass.

BUT, I am still unable to read the compressed Zarr store. ncdump, for example, hangs and produces no output or error message when attempting to read such a store.

I have tried this on both Mac OSX 11.6.4 and SUSE Linux Enterprise Server 12.3 (the latter required a change to lines 29-31 in cmake/modules/FindBlosc.cmake to look for the Blosc library directory in lib64 rather than just lib).

[DennisHeimbigner]

I do use ref_power_901_constants.zip as one of our test cases.
The relevant test is netcdf-d/nczarr_tests/run_interop.sh

[DennisHeimbigner]

I realize I am being imprecise. I am referring to using the netcdf-c main branch.

[bmcandr]

Thanks @DennisHeimbigner! Running that test helped me realize that I had not set $HDF5_PLUGIN_PATH. I can successfully read the a Blosc+LZ4 compressed Zarr store now. 🎉

I can open a pull request to fix the issue in the netcdf-c CMakeLists.txt that I described above if you'd like.

[DennisHeimbigner]

Please do. Although I cannot find it in the current main branch, so it may have
been fixed as a side effect of some other PR.