From a5cae51efce2d7b04859a34a19f16308b34423b0 Mon Sep 17 00:00:00 2001 From: Dennis Heimbigner Date: Wed, 23 Mar 2022 19:28:06 -0600 Subject: [PATCH 01/17] Turn on the xarray convention for NCZarr format re: https://github.com/pydata/xarray/issues/6374 As a result of a discussion about Xarray (see above issue), I decided to turn on the xarray convention for NCZarr datasets where possible so that xarray can read a larger set of nczarr generated datasets. This causes the following changes: * If the user wants to generate a pure zarr file, then the mode "zarr" must be explicitly used; it is no longer the case that "mode=xarray" or mode="noxarray" implies "mode=zarr". * It is still the case that "mode=noxarray" will turn off the XArray convention. The following conditions will cause ''_ARRAY_DIMENSIONS'' to not be written. * The variable is not in the root group, * Any dimension referenced by the variable is not in the root group. --- .github/workflows/run_tests_ubuntu.yml | 2 +- RELEASE_NOTES.md | 1 + docs/nczarr.md | 23 ++++++++------ libdispatch/dinfermodel.c | 11 +++---- libnczarr/zarr.c | 5 ++- libnczarr/zsync.c | 43 +++++++++++++++++--------- nc_test4/tst_broken_files.c | 5 +-- nczarr_test/ref_newformatpure.cdl | 3 +- nczarr_test/run_purezarr.sh | 6 ++-- 9 files changed, 59 insertions(+), 40 deletions(-) diff --git a/.github/workflows/run_tests_ubuntu.yml b/.github/workflows/run_tests_ubuntu.yml index 723b6a2415..eb410325a7 100644 --- a/.github/workflows/run_tests_ubuntu.yml +++ b/.github/workflows/run_tests_ubuntu.yml @@ -4,7 +4,7 @@ name: Run Ubuntu/Linux netCDF Tests -on: [ pull_request ] +on: [pull_request] jobs: diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 6195b414c3..0072a3e6e9 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -7,6 +7,7 @@ This file contains a high-level description of this package's evolution. Release ## 4.8.2 - TBD +* [Enhancement] Turn on the XArray convention for NCZarr files by default. See [Github #????](https://github.com/Unidata/netcdf-c/pull/????). * [Enhancement] Improve filter support. More specifically (1) add nc_inq_filter_avail to check if a filter is available, (2) add the notion of standard filters, (3) cleanup szip support to fix interaction with NCZarr. See [Github #2245](https://github.com/Unidata/netcdf-c/pull/2245). * [Bug Fix] Require that the type of the variable in nc_def_var_filter is not variable length. See [Github #/2231](https://github.com/Unidata/netcdf-c/pull/2231). * [File Change] Apply HDF5 v1.8 format compatibility when writing to previous files, as well as when creating new files. The superblock version remains at 2 for newly created files. Full backward read/write compatibility for netCDF-4 is maintained in all cases. See [Github #2176](https://github.com/Unidata/netcdf-c/issues/2176). diff --git a/docs/nczarr.md b/docs/nczarr.md index af10387c60..7071db7c78 100644 --- a/docs/nczarr.md +++ b/docs/nczarr.md @@ -136,16 +136,18 @@ Note that It should be the case that zipping a _file_ format directory tree will produce a file readable by the _zip_ storage format, and vice-versa. -By default, _mode=zarr_ also supports the XArray _\_ARRAY\_DIMENSIONS_ convention. The _noxarray_ mode tells the library to disable the XArray support. +By default, the XArray convention is supported and used for +both NCZarr files and pure Zarr files. This +means that every variable in the root group whose named dimensions +are also in the root group will have an attribute called +*\_ARRAY\_DIMENSIONS* that stores those dimension names. +The _noxarray_ mode tells the library to disable the XArray support. The netcdf-c library is capable of inferring additional mode flags based on the flags it finds. Currently we have the following inferences. - -- _xarray_ => _zarr_ -- _noxarray_ => _zarr_ - _zarr_ => _nczarr_ -So for example: ````...#mode=noxarray,zip```` is equivalent to this. -````...#mode=nczarr,zarr,noxarray,zip +So for example: ````...#mode=zarr,zip```` is equivalent to this. +````...#mode=nczarr,zarr,zip ````