ZStandard support in the core library; what will it look like?

[WardF]

Creating this discussion thread to follow on from #2173.

[edwardhartnett]

In configure/CMake we need to check for zstandard library. If --enable-zstandard is used, and the library is not found, that's a configure error. Here's the CCR code, which turns on zstandard by default, but lets users build without it with --disable-zstd:

# Does the user want Zstandard?
AC_MSG_CHECKING([whether Zstandard filter library should be built and installed])
AC_ARG_ENABLE([zstd],
              [AS_HELP_STRING([--disable-zstd],
                              [Disable the build and install of Zstandard filter library.])])
test "x$enable_zstd" = xno || enable_zstd=yes
AC_MSG_RESULT($enable_zstd)
AM_CONDITIONAL(BUILD_ZSTD, [test "x$enable_zstd" = xyes])
if test "x$enable_zstd" = xyes; then
   AC_DEFINE([BUILD_ZSTD], 1, [If true, build with zstd filter.])
fi
AC_SUBST([BUILD_ZSTD], [$enable_zstd])

The HDF5 filter code is shipped with netCDF, compiled, and installed in the HDF5 filter directory (which can be overridden at configure). A copy of the code (one C file) can be found here:
https://github.com/ccr/ccr/blob/master/hdf5_plugins/ZSTANDARD/src/H5Zzstandard.c

We add a new set of functions to the dispatch table. Here's what works in CCR:

    int nc_def_var_zstandard(int ncid, int varid, int level);
    int nc_inq_var_zstandard(int ncid, int varid, int *zstandardp, int *levelp);

These have to be used before enddef, after def_var.

Similar functions as wrappers in F77. In F90 additional argument in def_var, and two in inq_var.

Here's the implementation from CCR. This is working code that was recently extensively tested at NOAA.

@DennisHeimbigner may wish to implement this in the Zarr layer as well.

/**
 * Turn on Zstandard compression for a variable.
 *
 * @param ncid File ID.
 * @param varid Variable ID.
 * @param level From -131072 to 22 (depends on Zstandard version). 
 * when compressing. Regular compression levels are from 1 up to 19.
 * Use levels >= 20, labeled `--ultra`, cautiously: they require more memory. 
 * Negative compression levels that extend the range of speed vs. ratio preferences.
 * The lower the level, the faster the speed (at the cost of compression).
 *
 * @return 0 for success, error code otherwise.
 * @author Charlie Zender
 */
int
nc_def_var_zstandard(int ncid, int varid, int level)
{
    unsigned int cd_value = level;
    int ret;

    /* Level must be between -131072 and 22 on Zstandard v. 1.4.5 (~202009)
       Earlier versions have fewer levels (especially fewer negative levels) */
    if (level < -131072 || level > 22)
        return NC_EINVAL;

    if (!H5Zfilter_avail(ZSTANDARD_ID))
    {
        printf ("Zstandard filter not available.\n");
        return NC_EFILTER;
    }

    /* Set up the Zstandard filter for this var. */
    if ((ret = nc_def_var_filter(ncid, varid, ZSTANDARD_ID, 1, &cd_value)))
        return ret;

    return 0;
}

/**
 * Learn whether Zstandard compression is on for a variable, and, if so,
 * the level setting.
 *
 * @param ncid File ID.
 * @param varid Variable ID.
 * @param zstandardp Pointer that gets a 0 if Zstandard is not in use for this
 * var, and a 1 if it is. Ignored if NULL.
 * @param levelp Pointer that gets the level setting (from -131072 to 22), if
 * Zstandard is in use. Ignored if NULL.
 *
 * @return 0 for success, error code otherwise.
 * @author Charlie Zender
 */
int
nc_inq_var_zstandard(int ncid, int varid, int *zstandardp, int *levelp)
{
    int zstandard = 0; /* Is Zstandard in use? */
    unsigned int level;
    size_t nparams;
    int ret;

#ifdef HAVE_MULTIFILTERS
    {
	size_t nfilters;
	unsigned int *filterids;
	int f;
	
	/* Get filter information. */
	if ((ret = nc_inq_var_filter_ids(ncid, varid, &nfilters, NULL)))
	    return ret;
	
	/* If there are no filters, we're done. */
	if (nfilters == 0)
	{
	    if (zstandardp)
		*zstandardp = 0;
	    return 0;
	}

	/* Allocate storage for filter IDs. */
	if (!(filterids = malloc(nfilters * sizeof(unsigned int))))
	    return NC_ENOMEM;

	/* Get the filter IDs. */
	if ((ret = nc_inq_var_filter_ids(ncid, varid, &nfilters, filterids)))
	    return ret;
    
	/* Check each filter to see if it is Zstandard. */
	for (f = 0; f < nfilters; f++)
	{
	    if (filterids[f] == ZSTANDARD_ID)
		zstandard++;

	    /* If Zstandard is in use, check parameter. */
	    if (zstandard)
	    {
	    
		if ((ret = nc_inq_var_filter_info(ncid, varid, filterids[f], &nparams, &level)))
		    return ret;

		/* For Zstandard, there is one parameter. */
		if (nparams != 1)
		    return NC_EFILTER;

		/* Tell the caller, if they want to know. */
		if (levelp)
		    *levelp = (int)level;

		/* Exit loop to report parameters (neglect remaining filters) */
		break;

	    }
	}

	/* Free resources. */
	free(filterids);

	if (zstandardp)
	    *zstandardp = zstandard;
    }
#else
    {
	unsigned int id;
	
	/* Get filter information. */
	ret = nc_inq_var_filter(ncid, varid, &id, &nparams, &level);
	if (ret == NC_ENOFILTER)
	{
	    if (zstandardp)
		*zstandardp = 0;
	    return 0;
	}
	else if (ret)
	    return ret;
	
	/* Is Zstandard in use? */
	if (id == ZSTANDARD_ID)
	    zstandard++;
	
	/* Does caller want to know if Zstandard is in use? */
	if (zstandardp)
	    *zstandardp = zstandard;
	
	/* If Zstandard is in use, check parameter. */
	if (zstandard)
	{
	    /* For Zstandard, there is one parameter. */
	    if (nparams != 1)
		return NC_EFILTER;
	    
	    /* Tell the caller, if they want to know. */
	    if (levelp)
		*levelp = level;
	}
    }
#endif /* HAVE_MULTIFILTERS */
    return 0;
}

And BTW I see that this is all Charlie's code! Thanks @czender !

[czender]

First, this has been a team effort. I inherited this code from @edwardhartnett 's LZ4 and Bzip2 filters and rewrote it for Zstandard (when I wrote the CCR Zstandard HDF5 filter wrapper that descends from @dmh's Bzip2 example). Then @edwardhartnett rewrote this to account for multifilters. Thanks to all the contributors along the way.

Second, I agree that Zstandard is here to stay. It's an IETF standard (like DEFLATE), and is now used within the Linux kernel. So it ain't disappearing anytime soon. The only way I know how to guarantee that Windows users will have it is by getting it through Anaconda, though there are probably multiple alternatives.

Third, it's really encouraging to see Unidata promoting this change. I was at a workshop yesterday where all the Grib folks at ECMWF and CMIP6 folks from UKMO and GFDL were emphasizing the need for both transparent access (like with netCDF/zlib) and a fast decoder (like Zstandard) for exascale computing. For multiple compressors, the BLOSC metacompressor seems like the way to go, though perhaps too complex/ambitious. For adding just one compressor to succeed zlib, Zstandard seems the way to go.

[DennisHeimbigner]

Once we start down this road, there is no reason to stop at zstandard. I plan to also add blosc
as well. Note that if we add a dispatch function equivalent to H5Zfilter_avail to the dispatch table,
then nc_def_var_zstandard can be implemented without adding it to the dispatch table because
it can use H5Zfilter_avail + nc_def_var_filter to implement nc_def_var_zstandard.

[dopplershift]

Blosc needs to have the same consideration as zstandard, so I would caution to go a little more slowly than "I plan to also add blosc as well". The technical burden for zstandard were light due to the availability of a native Java implementation. Last we discussed, I believe the Java implementation of Blosc relies on JNI, which means cross-platform availability is an open question.

[edhartnett]

I agree that blosc is more complex and needs a second discussion. That's why I proposed zstandard - it's a good fit for netCDF and will help a lot, right now. It's the low-hanging fruit. Once zstandard is in place, we can discuss blosc - as Ryan points out, that may be more complex and need more work, for a smooth integration. Or that may not seem like a good idea, once we have examined it. WRT the nc_def_var_zstd()/nc_inq_var_zstd() functions, I believe they are still necessary, for two reasons: 1 - putting together the correct filter arguments is complex and something most science programmers will get wrong before they get right. 2 - the F77/F90 APIs really benefit from having simple C functions to wrap. So I think we want a nc_def_var_zstandard()/nc_inq_var_zstandard() function. It does not have to be in the dispatch table, since the filter functions in the dispatch table already can be used. But it should be in the API and the documentation.

…

On Fri, Feb 11, 2022 at 11:38 AM Ryan May ***@***.***> wrote: Blosc needs to have the same consideration as zstandard, so I would caution to go a little more slowly than "I plan to also add blosc as well". The technical burden for zstandard were light due to the availability of a native Java implementation. Last we discussed, I believe the Java implementation of Blosc relies on JNI, which means cross-platform availability is an open question. — Reply to this email directly, view it on GitHub <#2214 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCSXXF44NHC4FVPSUHT7ELU2VJUFANCNFSM5NQICZ6A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[DennisHeimbigner]

I guess I was not clear; I agree about having nc_def/inq_var_zstandard (or should it be called zstd to match
NumCodecs?).
I already have blosc implemented for netcdf-c. So perhaps
we need to decide if always having a Java version is the limiting factor.
For example, is there a native Java SZIP implementation?

[dopplershift]

The lack of szip support in netcdf-java isn't an argument for making things worse. 😉

[dopplershift]

If netcdf-java could write files that netcdf-c was unable to read, that would be bad and I'd imagine many people here would have strong opinions regarding that.

I'm always surprised that I have to sell the fact that netcdf-c consciously making files that netcdf-java is unable to read is a bad thing.

[haileyajohnson]

I think blosc is going to be a necessary headache for the java library, the feedback from the Zarr community is generally that the Zarr support doesn't amount to much without blosc.

[edhartnett]

OK, quick work! The Java requirement has never been the case previously, and indeed netcdf-java can read many formats that netcdf-c cannot. ;-) However, the most important thing, IMO, is to get some advanced compression into netcdf-c/netcdf-fortran. If zstandard has achieved consensus, let's go for that. If blosc needs more discussion, let's not hold up zstandard - let's get it out there to the users, and work out the question of what to do with blosc.

…

On Fri, Feb 11, 2022 at 1:18 PM Dennis Heimbigner ***@***.***> wrote: I guess I was not clear; I agree about having nc_def/inq_var_zstandard (or should it be called zstd to match NumCodecs?). I already have blosc implemented for netcdf-c. So perhaps we need to decide if always having a Java version is the limiting factor. For example, is there a native Java SZIP implementation? — Reply to this email directly, view it on GitHub <#2214 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCSXXH3UYIYPY26VLOEAADU2VVILANCNFSM5NQICZ6A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

[dopplershift]

netcdf-java can read many formats that netcdf-c cannot.

That is fundamentally different than making a particular compression part of the netcdf files that are distributed widely.

[edhartnett]

I certainly understand that blosc raises some serious issues, which is why the current proposal focuses on zstandard. Once zstandard is in, I would be happy to help with blosc in any way, if and when it is considered suitable. There are some Java questions that no doubt need to be sorted out. It may be that blosc also has an easy Java solution, as zstandard does.

…

On Fri, Feb 11, 2022 at 2:16 PM haileyajohnson ***@***.***> wrote: I think blosc is going to be a necessary headache for the java library, the feedback from the Zarr community is generally that the Zarr support doesn't amount to much without blosc. — Reply to this email directly, view it on GitHub <#2214 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCSXXCY4OGFMZTOVDZ3YWLU2V4DTANCNFSM5NQICZ6A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

[edhartnett]

Lack of development effort on netcdf-java should not hold back C development, which is where the data producers are. And it never has - when netcdf-4 was first developed, netcdf-java could not read those files. It took John some time to implement that code, but that was after netcdf-4 already had proved itself useful and data producers were switching to it. I am happy to hear of efforts to bring zstandard to netcdf-java, and that is a huge step forward. Let's not get hung up in arguments about possible future additions - let's instead move forward on the consensus we have achieved around zstandard, and get that out to the users. The inclusion of zstandard will give us time to further examine blosc, including how netcdf-java may handle it.

…

On Fri, Feb 11, 2022 at 2:09 PM Ryan May ***@***.***> wrote: If netcdf-java could write files that netcdf-c was unable to read, that would be bad and I'd imagine many people here would have strong opinions regarding that. I'm always surprised that I have to sell the fact that netcdf-c consciously making files that netcdf-java is unable to read is a bad thing. — Reply to this email directly, view it on GitHub <#2214 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCSXXHVFIP6GBQCXWFXNH3U2V3KPANCNFSM5NQICZ6A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

[dopplershift]

Lack of development effort on netcdf-java should not hold back C
development, which is where the data producers are.

That's a pretty miopic view. What good does it do data producers to have a feature if a significant portion of their data consumers can't handle the data they're producing?

[DennisHeimbigner]

Ed is correct in that you (Ryan) are adding a new constraint to what filters we provide.
We now have three separate constraints on what filters to include: HDF5, Zarr (NumCodecs) and Java.

[dopplershift]

@edhartnett @DennisHeimbigner I appreciate the historical context, which indeed is inline with how I understand things have worked in the past.

Honestly, the push back here is fascinating, since all I said was: "Blosc needs to have the same consideration as zstandard" and "netcdf-c consciously making files that netcdf-java is unable to read is a bad thing." I would love to know what about those statements is controversial. Neither they, nor anything else, says "We can't do blosc".

netCDF is more than the C library. It is a data model and a set of formats for encoding that data model, and an ecosystem of tools around that data. Unidata has and continues to develop a wide portfolio of tools serving that ecosystem: MetPy/Siphon, netCDF C and Java implementations, THREDDS Data Server, and IDV. It is our duty, in supporting the ENTIRE COMMUNITY of atmospheric science data users that we do our best to keep this portfolio cohesive and minimize confusion. The C and Java implementations of netCDF that Unidata currently has are in relatively good parity, so it's important that we are careful and intentional when making additions. Having Unidata's own C netCDF implementation produce files that cannot be handled by the Java implementation yields data that are inaccessible to the IDV as well as the THREDDS server (which then impacts the Python space), not to mention numerous community tools, would be bad, and should be avoided as much as possible.

I recognize this is a departure from how things have operated in the past.

TLDR; All I'm saying is we need to think carefully and have a plan for what to do about the world beyond netcdf's C implemenation. The fact that me advocating for that yields pushback is...disconcerting? Confounding?

[DennisHeimbigner]

Ed, On a different note.
Do we need an option to disable zstandard?
Is there really any use case for this?

[edhartnett]

If you mean, do we need a --disable-zstandard/--enable-zstandard option with configure, we don't really. Instead, we could require zstandard, the same way zlib is required.

…

On Fri, Feb 11, 2022 at 3:41 PM Dennis Heimbigner ***@***.***> wrote: Ed, On a different note. Do we need an option to disable zstandard? Is there really any use case for this? — Reply to this email directly, view it on GitHub <#2214 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCSXXCCPSDO5MWE3NFLESDU2WGCZANCNFSM5NQICZ6A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

[WardF]

We can talk about this at the meeting next Thursday, but I would say yes,it would be best practice to allow our users the option to disable features if they have a particular reason we have not anticipated.

[DennisHeimbigner]

So once we know that libzstd is installed, we need to have the HDF5/NCZarr wrapper for it.
It would be nice if we could test for its existence somehow. Any ideas about how we do this?
Or do we assume that the netcdf-c installation will provide it?

[WardF]

Regarding this support, I think it will be a good idea to allow users to disable it via a --disable-zstd option. I think having it enabled by default will be fine.

There are several reasons for making this optional that I can think of off the top of my head.

• libzstd is very common but there's no guarantee that it's present in extant environments, and would pass the burden on to our users; given that some of the users don't have admin access in their environments and rely upon University IT staff, this is a potential temporary roadblock. This is exacerbated on Windows, which lacks a unified package management system.
• libzstd is distributed under a GPLv2 license. This differs from the BSD 3-clause license NetCDF is distributed under. This is already going to be an issue when figuring out what to do about the pre-compiled binary installation bundles which we distribute for Windows, but it makes me wonder what the impact would be for a Mathworks, or an ESRI, or any other company distributing software bundled with netCDF. At the very least, it would present an unnecessary burden on them in order to adopt the latest version of libnetcdf.
• Finally, "Who Knows?". There could be any number of un-anticipated reasons that somebody would want to disable libzstd support in their particular libnetcdf instance, and it seems like it will be easy enough for us to provide them that option.

[DennisHeimbigner]

The implementation I have created automatically disables zstandard if libzstd is not found.
So the use case we need is that someone wants to disable zstandard even though libzstd
is available. Not sure I can see an example for this specific case.

[edhartnett]

Ryan, perhaps a good way forward would be to focus on what you can do to help netcdf-java keep up. Do you have a plan to incorporate zstandard compression in netcdf-java? How long will it take? (Or has this already been done?) It should be no more work in Java than it was in C/Fortran - it took me a few days work at most. What's the progress on netcdf-java with zstandard?

[haileyajohnson]

Ryan, perhaps a good way forward would be to focus on what you can do to
help netcdf-java keep up.

I think Ryan and I would both agree with that, but focusing on that is not as trivial as just a few days work. The biggest issue is that there's just not enough bandwidth to jump on this right away, and before I can work on zstandard, there needs to be a place in the library to put it, which depends on this PR getting fixed up and merged, which I haven't had time to revisit in a bit.

The second issue is that we really do want to make sure the java library can keep up with the C library moving forward with regards to compression and filters, which in the big picture means reconsidering how and where we implement filters in the java library, both natively and otherwise. So while I do plan on implement zstandard in netCDF-Java, it will be the first new filter to be added since effort went in to refactoring filters, so I really want to put forethought into it and make sure it's done "right".

All that boils down to, there will be zstandard support in the netCDF-Java library in the near future, but it won't be immediate. That's been communicated to the C group, and they're planning accordingly.

[dopplershift]

I don't have much to add to what @haileyajohnson explained so well, except to note that part of:

focus on what you can do to help netcdf-java keep up.

means ensuring decisions made for netCDF consider the technical cost for ALL supported implementations. The goal that motivated the current discussion was "get users faster and better compression" not "add zstandard". Hypothetically, if there was another solution that gave 90% of zstandard performance benchmarks but was much easier to support in all implementations, I'd be advocating for that.

[edhartnett]

WRT enable/disable options. 1 - Dennis, it is not good to automatically detect the zstandard library and add support for zstandard if the library is found, and not if not found. This will hide the absence of zstandard from the builder of the software. We need the builder to take responsibility one way or another. 2 - Should it be always enabled, as zlib is? That would be useful just as it's very useful to know that zlib is available for all netCDF builds. I think this is the best choice. If we don't find the zstandard library, we stop the build and demand that the user install it. This is what we do for zlib - we require it in all cases. There is no --disable-zlib option. 3 - If it cannot be always required, it should be enabled by default. This would mean that if the zstandard library is not present, the build would stop. The user must either install it, or rebuild with --disable-zstandard. This way, the builder takes responsibility for building without zstandard, and knows that. 4 - The least best option, IMO, is to have zstandard disabled by default, as szip is. In this case, most users will not enable it, leading to all kinds of support questions when those users try to open a file compressed with zstandard. Since zstandard is so widely available, it makes sense to assume it can always be used.

[edhartnett]

NOAA will probably be switching to zstandard sometime this year, in UFS R&D work. In the course of time, this will become operational code. So zstandard data files are coming, on their own schedule. (This can be done with current releases of netCDF, HDF5, and CCR. See my AGU extended abstract for details: https://www.researchgate.net/publication/357001251_Quantization_and_Next-Generation_Zlib_Compression_for_Fully_Backward-Compatible_Faster_and_More_Effective_Data_Compression_in_NetCDF_Files?_sg%5B0%5D=WV1a3JRFVw4ijMCIZ_ArqVNDxV5a2BIyoF-JzFvPdibjT8risKO_vIvnjcgpR1YTPM4RDkhUklSyZPJoRGdJmCp2RWn4XvhvutOl0Qm7.paT902AXrqzvj6m7H-Tgii52biW3w6__E2OxZbCACTSIV-6ZbfR2-YNsTkfnJJBIdxskEae7MiiSxy_Wbuc2mA ) As Ryan points out, if something better comes along, like BLOSC, that may also be used. Once again, immediate operational requirements will drive decisions. Ryan, I'm happy to hear you will be advocating for even better compression, because that opportunity is probably coming. However, *this* discussion is about adding zstandard, not those future, perhaps even better, compression libraries. Perhaps someone should start a separate discussion about BLOSC.

[edhartnett]

Dennis, from your remarks about the dispatch table, I gather that the same filter commands that turn on (for example) zstandard for netCDF/HDF5 files will also turn it on for Zarr? If so, very neat. That means that yes, the filter available call needs to be in the dispatch table (maybe nc_inq_filter_avail()?). That's a great idea to avoid having to change the dispatch table, but still being able to add convenience functions like nc_def_var_zstandard()/nc_inq_var_zstandard(), which will work for both netCDF/HDF5 files and Zarr. Clever!

[DennisHeimbigner]

I do not like the idea of making zstandard required in order to deploy netcdf-c.

[edwardhartnett]

I have started separate discussions for LZ4 and BLOSC, so that this discussion can remain focused on Zstandard.

[edwardhartnett]

Possibly this discussion should be closed, since support for zstandard was included in the 4.9.0 release...

[dopplershift]

Discussions can't be closed, only locked if necessary.