Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update HDF5 format compatibility #951

Closed
Dave-Allured opened this issue May 3, 2018 · 26 comments · Fixed by #2176
Closed

Update HDF5 format compatibility #951

Dave-Allured opened this issue May 3, 2018 · 26 comments · Fixed by #2176

Comments

@Dave-Allured
Copy link
Contributor

This is a proposal to update netCDF4/HDF5 file format controls within netCDF-C. The recent HDF5 release 1.10.2 in March 2018 provides new options for fine grained control of the HDF5 internal file format. HDF5 documentation hints at possible performance gains and bug avoidance.

The best format version for universal compatibility is the one created by the latest HDF5 1.8.* library version, currently 1.8.20. This is the traditional netCDF4 format that was made by all previous netCDF4 library versions, until HDF5 1.10.0 was released in March 2016. This is the so-called "superblock 2" format.

Objectives

  1. Maintain full backward read and write format compatibility. Files created through this proposed change should be readable and writable by older software builds, specifically older versions of netCDF and HDF5 libraries.

  2. When possible, avoid "earliest" HDF5 file format selection, loosely known as "superblock 0" format. This variation is unavoidable when using HDF5 library versions 1.10.0 and 1.10.1, due to lack of fine grained controls. However, this format variation is avoidable with both older (1.8.*) and newer (1.10.2 and later) HDF5 library versions.

History of HDF5 format control

Current and previous HDF5 versions provide file format control through this function, which takes arguments "low" and "high":

https://portal.hdfgroup.org/display/HDF5/H5P_SET_LIBVER_BOUNDS

All netCDF-C library versions 4.0 through 4.4.0, linked with HDF5 1.8.*, created the traditional "V1.8" or "superblock 2" format using these settings:

low  = H5F_LIBVER_LATEST
high = H5F_LIBVER_LATEST

With the advent of HDF5 1.10.0, more recent netCDF-C versions 4.4.1 through 4.6.1 used the following settings to preserve backward format compatibility. See #250 for the original discussion. This created "superblock 0" files:

low  = H5F_LIBVER_EARLIEST
high = H5F_LIBVER_LATEST

Proposed format control for new netCDF-C releases

Select format controls based on the detected HDF5 library version within the current netcdf-C library build.

All HDF5 1.8.* versions: Use the traditional settings to select "V1.8" file format:

low  = H5F_LIBVER_LATEST
high = H5F_LIBVER_LATEST

HDF5 1.10.0 and 1.10.1: These versions lack the new fine grained control. Select "earliest" as the only available backward compatible setting. This will create "superplock 0" files, and avoid making incompatible "superblock 3" files:

low  = H5F_LIBVER_EARLIEST
high = H5F_LIBVER_LATEST

HDF5 1.10.2 (new release) and later: Select the traditional "V1.8" or "superblock 2" format with this newly available configuration:

low  = H5F_LIBVER_V18
high = H5F_LIBVER_LATEST

Note: "high = H5F_LIBVER_LATEST" is retained in all cases. This setting allows the possible development of new HDF5 features such as SWMR, without additional complication of the format control code.

@edhartnett
Copy link
Contributor

This makes sense.

@Dave-Allured
Copy link
Contributor Author

For anyone interested, here is more detailed documentation on HDF5 format controls and the internal implementation. This came from the HDF help desk. The version on the HDF5 website is currently out of date.

RFC-bounds.v4.2018-jan-19.docx

@DennisHeimbigner
Copy link
Collaborator

I have recollection that when we used the original 1.10 mechanism,
which was feature based, we discovered that we never actually generated
any superblock versions other than 0 (zero).
Ward- do you remember this?

@WardF
Copy link
Member

WardF commented May 4, 2018

I do recall that that was the case, but that was with the original 1.10 version (as you mention). I do not recall if this was expected behavior or a bug. We should re-investigate.

@Dave-Allured
Copy link
Contributor Author

Dave-Allured commented May 4, 2018

Release announcement for HDF5 1.10.2, 2016 March 30:
https://forum.hdfgroup.org/t/release-of-hdf5-1-10-2-newsletter-160/4247

Full release notes:
https://portal.hdfgroup.org/display/support/HDF5+1.10.2

@Dave-Allured
Copy link
Contributor Author

Dave-Allured commented May 4, 2018

HDF5 format compatibility was documented in the first netCDF-C release announcement following HDF5-1.10.0. This was netCDF-C 4.4.1, release date 2016 June 28:
https://www.unidata.ucar.edu/blogs/news/entry/netcdf-4-4-1

@Dave-Allured
Copy link
Contributor Author

Format version control is an HDF5 file access property, not a file creation property. This means that version settings are transient, not permanently embedded somehow in the file. In particular the superblock number is not a reliable indicator of internal HDF5 object versions, if a file was ever updated after initial creation.

https://portal.hdfgroup.org/display/HDF5/H5P_SET_LIBVER_BOUNDS

For this proposal to work as expected, format settings must be selected every time a file is re-opened for writing, not just during initial creation. Otherwise you can get a non-optimal mix of object versions within the same file. That is probably safe, but non-optimal.

In the current netcdf-C release 4.6.1, H5Pset_libver_bounds is called only during create, and not when a file is re-opened. Please add to this proposal, format control should be selected in both create and read/write modes. Apply the same three-way algorithm above, for both of these open modes.

@DennisHeimbigner
Copy link
Collaborator

This raises an interesting question. Can the superblock version number change
over time. You seem to be saying it can, which surprises me. Have you run a test
case to verify this?

@Dave-Allured
Copy link
Contributor Author

@DennisHeimbigner, my understanding is that the superblock is written one time on file creation, and never again rewritten during normal read or write operations. There are a few dynamic fields within the superblock that get updated, such as end of file pointer and file lock status. But the superblock version number and some of the other fields appear to be permanent.

@Dave-Allured
Copy link
Contributor Author

I am withdrawing this proposal. I will submit a new proposal to include an optional control for 1.10 format forward compatibility.

@Dave-Allured
Copy link
Contributor Author

Re-opening. This part, update to v1.8 format compatibility, is good as it stands now. It would be better to get this part done first, then consider a 1.10 compatibility option later. 1.10 compatibility will be problematic.

@Dave-Allured Dave-Allured reopened this Jan 6, 2020
@brtnfld
Copy link
Contributor

brtnfld commented Jan 7, 2021

@Dave-Allured @WardF @DennisHeimbigner

Was there anymore work done for this, i.e. a new proposal submitted? I need to introduce a mechanism for enabling the latest HDF5 version as an option. Currently, the latest is always set to 1.8:

#ifdef HAVE_H5PSET_LIBVER_BOUNDS
#if H5_VERSION_GE(1,10,2)
    if (H5Pset_libver_bounds(fapl_id, H5F_LIBVER_EARLIEST, H5F_LIBVER_V18) < 0)
#else
        if (H5Pset_libver_bounds(fapl_id, H5F_LIBVER_EARLIEST,
                                 H5F_LIBVER_LATEST) < 0)
#endif
            BAIL(NC_EHDFERR);
#endif

This will not work with the DAOS VOL connector we have developed, which is only able to produce 1.12 compatible versions. I can make it, if NetCDF is creating a file using the DAOS connector, to automatically use the latest HDF5 version. However, if the DAOS container is converted to native HDF5, it will not be readable by HDF5 1.8 or 1.10.

@WardF
Copy link
Member

WardF commented Jan 7, 2021

I need to review this issue so I may be repeating some things addressed above as I think out loud below; feel free to tell me that this is a non-issue (I'd welcome such news, frankly :)).

The issue we'll need to consider internally, before accepting this change, would be the impact resulting when files created this way are distributed. From a practical perspective, we want to avoid a situation where netCDF-4 files are being created and distributed (say, through the Unidata THREDDS server) which cannot be read, failing in a non-transparent way, on systems which are using an older libhdf5 install. The last time I tested this (which was a while ago and will bear re-testing, but it was right after the 1.10.0 release, which caught us off guard), libhdf5 1.8.x-based libnetcdf installs simply reported a generic HDF5 file error.

So as I return to speed on this issue, we will need to particularly focus on the potential impact of broad compatibility, both to avoid creating and distributing files which are indistinguishable from corrupted files, and to avoid the administrative/support overhead on the already-stretched Unidata staff and their support time.

Having stated that I'll start digging in to this, and will also ask @lesserwhirls and @dopplershift for comment.

@Dave-Allured
Copy link
Contributor Author

@brtnfield, no, I never submitted a new proposal. What I outlined above should still be good for maximal forward and backward compatibility.

I think there is actually a mistake in the current implementation (4.7.4). This line which you quoted above:

   if (H5Pset_libver_bounds(fapl_id, H5F_LIBVER_EARLIEST, H5F_LIBVER_V18) < 0)

Should actually be:

   if (H5Pset_libver_bounds(fapl_id, H5F_LIBVER_V18, H5F_LIBVER_LATEST) < 0)

This would correspond with the third part of my suggestion above under "Proposed format control", for HDF5 1.10.2 and above.

There may be a reason for the current implementation, but I do not understand it. This current implementation, combined with newer HDF5 versions 1.10.2 and above, will create only netcdf-4 files with "v1.6 compatibility", represented as superblock zero. This suggested one-line change should create the desired netcdf-4 files with "v1.8 compatibility", represented as superblock 2.

Also, if the third argument is changed to the suggested H5F_LIBVER_LATEST, then advanced HDF5 features will be unblocked. This might achieve compatibility with your DAOS VOL connector, without needing further changes. Would you mind patching netcdf 4.7.4 with this one-line change, and see if this by itself solves your compatibility needs?

@edwardhartnett
Copy link
Contributor

I think @Dave-Allured has a great suggestion above. Can you submit a PR with this one-line change?

There is not a great reason for the current implementation. I think Dave's makes more sense.

@gsjaardema
Copy link
Contributor

The CGNS library has the following:

    /* Compatibility with V1.8 */
    H5Pset_libver_bounds(g_propfileopen,
#if H5_VERSION_GE(1,10,3)
          H5F_LIBVER_V18, H5F_LIBVER_V18);
#else
          H5F_LIBVER_LATEST, H5F_LIBVER_LATEST);
#endif

If compatability with hdf5-1.8.X is desired and the hdf5 library is 1.10.3 or newer, then use H5F_LIBVER_V18 for both arguments. If the library is older or we don't need compatibility with hdf5-1.8.X then use H5F_LIBVER_LATEST for both arguments.

We also are looking at using VOL capabilities...

@brtnfld
Copy link
Contributor

brtnfld commented Jan 8, 2021

To be clear, the current behavior by specifying H5F_LIBVER_EARLIEST, H5F_LIBVER_V18 is the superblock will be v0 (v1.6) if no v1.8 format features are being used. If v1.8 format features are being used, which is possible with NetCDF, then the superblock will be v2.

The proposal is to use H5F_LIBVER_V18, H5F_LIBVER_LATEST, which will create, by default, v2 superblock. If a newer format feature is used, which requires a higher format (SWMR, VDS, new references), the lowest (H5F_LIBVER_V18) will be set to V110. The superblock will be v3.

CGNS users had an issue with readers built with hdf5 1.8 not reading the CGNS files (we had used H5Pset_file_space_strategy, a format upgrade), so CGNS is currently enforcing 1.8 compatibilities. This policy will need to be changed because it would be beneficial to use some of the new HDF5 features.

NetCDF users might have the same issue with readers not reading the netCDF files with this change. It should be fine unless NetCDF adds an API that always enforces a v3 superblock (like CGNS did); otherwise, it should, for the most part, always create a v2 superblock.

A good summary of the libverbounds behavior can be found at:

https://www.hdfgroup.org/2018/04/why-should-i-care-about-the-hdf5-1-10-2-release/rfc-setting-bounds-for-object-creation-in-hdf5-1-10-0-update/

brtnfld added a commit to brtnfld/netcdf-c that referenced this issue Jan 8, 2021
… than HDF5 1.8, which

may require to have a superblock greater than v2.

Ref: Update HDF5 format compatibility Unidata#951
brtnfld added a commit to brtnfld/netcdf-c that referenced this issue Jan 8, 2021
… than HDF5 1.8, which

may require to have a superblock greater than v2.

Ref: Update HDF5 format compatibility Unidata#951
@Dave-Allured
Copy link
Contributor Author

@gsjaardema et al, I will go into the CGNS case because it is a good example for netcdf. The SECOND function arg (low =) is the important one for this part of the discussion. You said:

    /* Compatibility with V1.8 */
    H5Pset_libver_bounds(g_propfileopen,
#if H5_VERSION_GE(1,10,3)
          H5F_LIBVER_V18, H5F_LIBVER_V18);
#else
          H5F_LIBVER_LATEST, H5F_LIBVER_LATEST);
#endif

My understanding is that this will result in v1.10 (superblock 3) files when used with HDF5 library versions 1.10.0, 1.10.1, and 1.10.2, in other words files that are NOT v1.8 compatible, therefore NOT universally readable. This is because the second arg "low = H5F_LIBVER_LATEST" takes effect for these particular cases, therefore v1.10 format is engaged.

If the CGNS project does not care about possible use of older HDF5 versions, then just mandate HDF5 1.10.3 or later, and there is no problem. If they DO care, as I do for netcdf, then I suggest a change along the lines of my original proposal above. At minimum, change the LOW arg in the ELSE clause as follows.

#if H5_VERSION_GE(1,10,3)
          H5F_LIBVER_V18, H5F_LIBVER_V18);
#else
          H5F_LIBVER_EARLIEST, H5F_LIBVER_LATEST);
#endif

This will result in v1.6 (superblock 0) files, which are universally readable, when used with older HDF5 library versions, specifically all 1.8.X versions, also 1.10.0 through 1.10.2.

Now I am no fan of unnecessary downgrading, so I suggest this more fine grained solution would be better. I think I will also invert the conditionals so that the code will progress sensibly from earliest to latest HDF5 versions. This corresponds to my original proposal above, except for the final "high = " arg which relates only to advanced features, to be discussed later. Notice also the tweak for lib version 1.10.2.

#if H5_VERSION_LT(1,10,0)
          H5F_LIBVER_LATEST, H5F_LIBVER_LATEST);     /* all HDF5 1.8 lib versions */
#else
#if H5_VERSION_LE(1,10,1)
          H5F_LIBVER_EARLIEST, H5F_LIBVER_LATEST);    /* lib versions 1.10.0, 1.10.1 */
#else
          H5F_LIBVER_V18, H5F_LIBVER_V18);      /* lib versions 1.10.2 and higher */
#endif
#endif

This will result in the optimal v1.8 (superblock 2) files with ALL HDF5 library versions, older and newer, EXCEPT for library versions 1.10.0 and 1.10.1, which will create v1.6 (superblock 0) files. Both of these format versions are universally readable. Reminder, this middle exception occurs only because of a compatibility option that is missing in only those two library versions.

This part of the discussion deals only with basic file compatibility across library versions, and does not address advanced features. Do you agree with this analysis so far?

@DennisHeimbigner
Copy link
Collaborator

DennisHeimbigner commented Jan 8, 2021

I think I need some clarification vis-a-vis netcdf backward compatibility.
At the moment, we are implicitly committed to superblock <= 2 so that netcdf
combined with hdf5 1.8.21 (our minimal recommended version) can read all
currently created files. This change appears to replace this commitment with one
in which netcdf can only read certain netcdf files (with superblock > 2) if the
netcdf library is built using some version of HDF5 with version greater than 1.8.21.
Is this correct?

@Dave-Allured
Copy link
Contributor Author

@DennisHeimbigner, no, not correct. For normal netcdf operations, the governing of file format during create and write operations is the SECOND function arg, known as "low =" in HDF5 documentation.

When linked with all 1.8.X HDF5 lib versions, "low = H5F_LIBVER_LATEST" results in v1.8 (superblock 2) files.

When linked with lib versions 1.10.0 and 1.10.1, "low = H5F_LIBVER_EARLIEST" results in v1.6 (superblock 0) files. For optimal results, use "low = H5F_LIBVER_EARLIEST" only when required to link with one of these now-obsolescent lib versions.

When linked with lib versions 1.10.2 and greater, "low = H5F_LIBVER_V18" results in v1.8 (superblock 2) files.

All three together, via these conditionals, should always result in the superblock <= 2 condition that you want for normal netcdf creates and writes.

The third arg "high =" governs format only when some discrete operation invokes an advanced HDF5 feature, such as SWMR. My limited understanding of the netcdf core library is that such advanced features are never invoked in normal operations. This is something that would only be done, and very explicitly, in support of a new capability, such as the DAOS VOL connector discussed above.

All this is subject to oversight and testing, to be sure. I have not lately run any advanced test cases for this.

@brtnfld
Copy link
Contributor

brtnfld commented Jan 8, 2021

... if the netcdf library is built using some version of HDF5 with version greater than 1.8.21.

The minimum superblock value for earliest is not dependent on what version of hdf5 is used; it is possible that a user can build netCDF with HDF5 1.12 and have a file with superblock v0. It only depends on what APIs are called. For example, if NetCDF calls H5Pset_attr_creation_order, that would force the superblock to be v2.

Currently, if NetCDF had the upper bound to be lastest, it will never have a superblock > v2 since it does not use any feature requiring greater than that, independent of the HDF5 version. The commitment to <=2 is still valid with these changes; it will only be an issue in the future if NetCDF starts to use v3 dependent APIs. It is possible for a user to not have version > 2 assuming they don't use any new NetCDF feature that needs v3 superblock.

@brtnfld
Copy link
Contributor

brtnfld commented Jan 8, 2021

@Dave-Allured I guess I don't understand why low is set to H5F_LIBVER_EARLIEST for some cases and not for others; why do you want to allow for v0 with a certain version of HDF5 and not others? That would change the current behavior of what NetCDF allows now.

I could see doing that if Netcdf finds that v2 should always be used because it performs better when used with a certain version of the HDF5 library.

@Dave-Allured
Copy link
Contributor Author

@brtnfld, the short answer is that the function H5Pset_libver_bounds and its enum parameters have changed several times over the history of HDF5. You can find traces of this in the function doc page and the release notes. There are a lot of gory details if you dive into the full history.

The proposed three-way conditional represents the most appropriate method to coax a "universally readable" file format (superblock <= 2), at each of three different stages in the API evolution.

H5F_LIBVER_EARLIEST is necessary only with HDF5 library versions 1.10.0 and 1.10.1, because unfortunately v1.8 format compatibility was neglected in those versions. This meant that netcdf had to resort to a downgrade from v1.8 to v1.6 compatility (superblock 0) following the 1.10.0 HDF5 release in 2016 April. See #250 for history.

The desired symbol H5F_LIBVER_V18 was added in HDF5 library version 1.10.2. This remains the optimal choice for the "low =" parameter through the remainder of 1.10.X, and into 1.12.X library versions.

Unfortunately I never got around to following up on restoring low = H5F_LIBVER_V18, which is why netcdf has been stuck on v1.6 compatibility (superblock 0), until now.

brtnfld added a commit to brtnfld/netcdf-c that referenced this issue Jan 11, 2021
than HDF5 1.8. This will allow for HDF5 features (VDS, SWMR, new references, etc...)
which may require to have a superblock greater than v2.

See for discussion Ref: Update HDF5 format compatibility Unidata#951
@edwardhartnett
Copy link
Contributor

Are we done here? Or is there more work to be done?

@brtnfld
Copy link
Contributor

brtnfld commented Jan 1, 2022

I'm done with this issue.

@WardF WardF closed this as completed Jan 3, 2022
Dave-Allured added a commit to Dave-Allured/netcdf-c that referenced this issue Jan 8, 2022
This PR selects the best HDF5 format compatibility options when re-opening an existing netCDF-4 file for writing, such as appending, or adding new groups or variables.  This change selects the optimal HDF5 v1.8 compatibility when possible, and otherwise falls back to the adequate v1.6 compatibility.

Format compatibility is a transient property of the HDF5 library, rather than baked in at file creation time.  Therefore, format compatibility options must be re-selected every time a netCDF-4 file is re-opened for writing.

This builds on the similar update for initial file creation, PR Unidata#1931, by @brtnfld, released in netcdf-c version 4.8.1.

In particular, this PR moves the compatibility controls into a single central location,  a new common function that is shared by both create and open functions.

Fore more details, see issue Unidata#951, and documentation at the top of libhdf5/hdf5set_format_compatibility.c.
Dave-Allured added a commit to Dave-Allured/netcdf-c that referenced this issue Jan 8, 2022
This commit selects the best HDF5 format compatibility options when re-opening an existing netCDF-4 file for writing, such as appending, or adding new groups or variables.

The general objective is to make netCDF-4 files that can be read and written by all previous library  versions.  Optimal HDF5 v1.8 compatibility is selected whenever possible.  Otherwise this falls back to the adequate v1.6 compatibility.

Format compatibility is a transient property of the HDF5 library, rather than baked in at file creation time.  Therefore, compatibility options must be re-selected every time a netCDF-4 file is re-opened for writing.

This builds on the previous update for initial file creation, PR Unidata#1931, by @brtnfld, released in netcdf-c version 4.8.1.

In particular, this commit moves compatibility controls into a single central location, a new common function that is shared by both create and open functions.

For more details, see issue Unidata#951, also documentation at the top of libhdf5/hdf5set_format_compatibility.c.

This commit also makes several corrections and cleanups to previous comments about the use of related property lists.
@Dave-Allured
Copy link
Contributor Author

Ignore that first commit. I wish I understood how to back out mistakes more gracefully than this. The PR is good.

Dave-Allured added a commit to Dave-Allured/netcdf-c that referenced this issue Jan 21, 2022
* Add current release note for Unidata#2176.
* Add missing release note for Unidata#951 in last release 4.8.1.
* Improve related old note for release 4.4.1.  Better consistency with recent updates.
Dave-Allured added a commit to Dave-Allured/netcdf-c that referenced this issue Feb 1, 2022
* Add current release note for Unidata#2176.
* Add missing note for Unidata#951 in release 4.8.0.
* Improve related old note for release 4.4.1.  Better consistency with recent updates.
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this issue Jun 3, 2023
Release Notes       {#RELEASE_NOTES}
=============

\brief Release notes file for the netcdf-c package.

This file contains a high-level description of this package's
evolution. Releases are in reverse chronological order (most recent
first). Note that, as of netcdf 4.2, the `netcdf-c++` and
`netcdf-fortran` libraries have been separated into their own
libraries.

## 4.9.3 - TBD

## 4.9.2 - March 14, 2023

This is the maintenance release which adds support for HDF5 version
1.14.0, in addition to a handful of other changes and bugfixes.

* Fix 'make distcheck' error in run_interop.sh. See [Github
  #????](https://github.com/Unidata/netcdf-c/pull/????).
* Update `nc-config` to remove inclusion from automatically-detected
  `nf-config` and `ncxx-config` files, as the wrong files could be
  included in the output.  This is in support of [GitHub
  #2274](Unidata/netcdf-c#2274).

* Update H5FDhttp.[ch] to work with HDF5 version 1.13.2 and later. See
  [Github #2635](Unidata/netcdf-c#2635).

* [Bug Fix] Update DAP code to enable CURLOPT_ACCEPT_ENCODING by
  default. See [Github
  #2630](Unidata/netcdf-c#2630).

* [Bug Fix] Fix byterange failures for certain URLs. See [Github
  #2649](Unidata/netcdf-c#2649).

* [Bug Fix] Fix 'make distcheck' error in run_interop.sh. See [Github
  #2631](Unidata/netcdf-c#2631).

* [Enhancement] Update `nc-config` to remove inclusion from
  automatically-detected `nf-config` and `ncxx-config` files, as the
  wrong files could be included in the output.  This is in support of
  [GitHub #2274](Unidata/netcdf-c#2274).

* [Enhancement] Update H5FDhttp.[ch] to work with HDF5 version
  1.14.0. See [Github
  #2615](Unidata/netcdf-c#2615).

## 4.9.1 - February 2, 2023

## Known Issues

* A test in the `main` branch of `netcdf-cxx4` is broken by this rc; this will
  bear further investigation, but not being treated as a roadblock for the
  release candidate.

* The new document, `netcdf-c/docs/filter_quickstart.md` is in
  rough-draft form.

* Race conditions exist in some of the tests when run concurrently with large
  numbers of processors

## What's Changed from v4.9.0 (automatically generated)

* Fix nc_def_var_fletcher32 operation by \@DennisHeimbigner in
  Unidata/netcdf-c#2403

* Merge relevant info updates back into `main` by \@WardF in
  Unidata/netcdf-c#2387

* Add manual GitHub actions triggers for the tests. by \@WardF in
  Unidata/netcdf-c#2404

* Use env variable USERPROFILE instead of HOME for windows and mingw. by
  \@DennisHeimbigner in Unidata/netcdf-c#2405

* Make public a limited API for programmatic access to internal .rc tables by
  \@DennisHeimbigner in Unidata/netcdf-c#2408

* Fix typo in CMakeLists.txt by \@georgthegreat in
  Unidata/netcdf-c#2412

* Fix choice of HOME dir by \@DennisHeimbigner in
  Unidata/netcdf-c#2416

* Check for libxml2 development files by \@WardF in
  Unidata/netcdf-c#2417

* Updating Doxyfile.in with doxygen-1.8.17, turned on WARN_AS_ERROR, added
  doxygen build to CI run by \@edwardhartnett in
  Unidata/netcdf-c#2377

* updated release notes by \@edwardhartnett in
  Unidata/netcdf-c#2392

* increase read block size from 1 KB to 4 MB by \@wkliao in
  Unidata/netcdf-c#2319

* fixed RELEASE_NOTES.md by \@edwardhartnett in
  Unidata/netcdf-c#2423

* Fix pnetcdf tests in cmake by \@WardF in
  Unidata/netcdf-c#2437

* Updated CMakeLists to avoid corner case cmake error by \@WardF in
  Unidata/netcdf-c#2438

* Add `--disable-quantize` to configure by \@WardF in
  Unidata/netcdf-c#2439

* Fix the way CMake handles -DPLUGIN_INSTALL_DIR by \@DennisHeimbigner in
  Unidata/netcdf-c#2430

* fix and test quantize mode for NC_CLASSIC_MODEL by \@edwardhartnett in
  Unidata/netcdf-c#2445

* Guard _declspec(dllexport) in support of #2446 by \@WardF in
  Unidata/netcdf-c#2460

* Ensure that netcdf_json.h does not interfere with ncjson. by
  \@DennisHeimbigner in Unidata/netcdf-c#2448

* Prevent cmake writing to source dir by \@magnusuMET in
  Unidata/netcdf-c#2463

* more quantize testing and adding pre-processor constant NC_MAX_FILENAME to
  nc_tests.h by \@edwardhartnett in
  Unidata/netcdf-c#2457

* Provide a default enum const when fill value does not match any enum
  constant by \@DennisHeimbigner in
  Unidata/netcdf-c#2462

* Fix support for reading arrays of HDF5 fixed size strings by
  \@DennisHeimbigner in Unidata/netcdf-c#2466

* fix musl build by \@magnusuMET in
  Unidata/netcdf-c#1701

* Fix AWS SDK linking errors by \@dzenanz in
  Unidata/netcdf-c#2470

* Address jump-misses-init issue. by \@WardF in
  Unidata/netcdf-c#2488

* Remove stray merge conflict markers by \@WardF in
  Unidata/netcdf-c#2493

* Add support for Zarr string type to NCZarr by \@DennisHeimbigner in
  Unidata/netcdf-c#2492

* Fix some problems with PR 2492 by \@DennisHeimbigner in
  Unidata/netcdf-c#2497

* Fix some bugs in the blosc filter wrapper by \@DennisHeimbigner in
  Unidata/netcdf-c#2461

* Add option to control accessing external servers by \@DennisHeimbigner in
  Unidata/netcdf-c#2491

* Changed attribute case in documentation by \@WardF in
  Unidata/netcdf-c#2482

* Adding all-error-codes.md back in to distribution documentation. by \@WardF in
  Unidata/netcdf-c#2501

* Update hdf5 version in github actions. by \@WardF in
  Unidata/netcdf-c#2504

* Minor update to doxygen function documentation by \@gsjaardema in
  Unidata/netcdf-c#2451

* Fix some addtional errors in NCZarr by \@DennisHeimbigner in
  Unidata/netcdf-c#2503

* Cleanup szip handling some more by \@DennisHeimbigner in
  Unidata/netcdf-c#2421

* Check for zstd development headers in autotools by \@WardF in
  Unidata/netcdf-c#2507

* Add new options to nc-config by \@WardF in
  Unidata/netcdf-c#2509

* Cleanup built test sources in nczarr_test by \@DennisHeimbigner in
  Unidata/netcdf-c#2508

* Fix inconsistency in netcdf_meta.h by \@WardF in
  Unidata/netcdf-c#2512

* Small fix in nc-config.in by \@WardF in
  Unidata/netcdf-c#2513

* For loop initial declarations are only allowed in C99 mode by \@gsjaardema in
  Unidata/netcdf-c#2517

* Fix some dependencies in tst_nccopy3 by \@WardF in
  Unidata/netcdf-c#2518

* Update plugins/Makefile.am by \@WardF in
  Unidata/netcdf-c#2519

* Fix prereqs in ncdump/tst_nccopy4 in order to avoid race conditions. by
  \@WardF in Unidata/netcdf-c#2520

* Move construction of VERSION file to end of the build by \@DennisHeimbigner in
  Unidata/netcdf-c#2527

* Add draft filter quickstart guide by \@WardF in
  Unidata/netcdf-c#2531

* Turn off extraneous debug output by \@DennisHeimbigner in
  Unidata/netcdf-c#2537

* typo fix by \@wkliao in Unidata/netcdf-c#2538

* replace 4194304 with READ_BLOCK_SIZE by \@wkliao in
  Unidata/netcdf-c#2539

* Rename variable to avoid function name conflict by \@ibaned in
  Unidata/netcdf-c#2550

* Add Cygwin CI and stop installing unwanted plugins by \@DWesl in
  Unidata/netcdf-c#2529

* Merge subset of v4.9.1 files back into main development branch by \@WardF in
  Unidata/netcdf-c#2530

* Add a Filter quickstart guide document by \@WardF in
  Unidata/netcdf-c#2524

* Fix race condition in ncdump (and other) tests. by \@DennisHeimbigner in
  Unidata/netcdf-c#2552

* Make dap4 reference dap instead of hard-wired to be disabled. by \@WardF in
  Unidata/netcdf-c#2553

* Suppress nczarr_test/tst_unknown filter test by \@DennisHeimbigner in
  Unidata/netcdf-c#2557

* Add fenceposting for HAVE_DECL_ISINF and HAVE_DECL_ISNAN by \@WardF in
  Unidata/netcdf-c#2559

* Add an old static file. by \@WardF in
  Unidata/netcdf-c#2575

* Fix infinite loop in file inferencing by \@DennisHeimbigner in
  Unidata/netcdf-c#2574

* Merge Wellspring back into development branch by \@WardF in
  Unidata/netcdf-c#2560

* Allow ncdump -t to handle variable length string attributes by \@srherbener in
  Unidata/netcdf-c#2584

* Fix an issue I introduced with make distcheck by \@WardF in
  Unidata/netcdf-c#2590

* make UDF0 not require NC_NETCDF4 by \@jedwards4b in
  Unidata/netcdf-c#2586

* Expose user-facing documentation related to byterange DAP functionality.  by
  \@WardF in Unidata/netcdf-c#2596

* Fix Memory Leak by \@DennisHeimbigner in
  Unidata/netcdf-c#2598

* CI: Change autotools CI build to out-of-tree build. by \@DWesl in
  Unidata/netcdf-c#2577

* Update github action configuration scripts. by \@WardF in
  Unidata/netcdf-c#2600

* Update the filter quickstart guide.  by \@WardF in
  Unidata/netcdf-c#2602

* Fix symbol export on Windows by \@WardF in
  Unidata/netcdf-c#2604

## New Contributors
* \@georgthegreat made their first contribution in Unidata/netcdf-c#2412
* \@dzenanz made their first contribution in Unidata/netcdf-c#2470
* \@DWesl made their first contribution in Unidata/netcdf-c#2529
* \@srherbener made their first contribution in Unidata/netcdf-c#2584
* \@jedwards4b made their first contribution in Unidata/netcdf-c#2586

**Full Changelog**: Unidata/netcdf-c@v4.9.0...v4.9.1

### 4.9.1 - Release Candidate 2 - November 21, 2022

#### Known Issues

* A test in the `main` branch of `netcdf-cxx4` is broken by this rc;
  this will bear further investigation, but not being treated as a
  roadblock for the release candidate.

* The new document, `netcdf-c/docs/filter_quickstart.md` is in rough-draft form.

#### Changes

* [Bug Fix] Fix a race condition when testing missing filters. See
  [Github #2557](Unidata/netcdf-c#2557).

* [Bug Fix] Fix some race conditions due to use of a common file in multiple
  shell scripts . See [Github
  #2552](Unidata/netcdf-c#2552).


### 4.9.1 - Release Candidate 1 - October 24, 2022

* [Enhancement][Documentation] Add Plugins Quick Start Guide.  See
  [GitHub #2524](Unidata/netcdf-c#2524) for
  more information.

* [Enhancement] Add new entries in `netcdf_meta.h`, `NC_HAS_BLOSC` and
  `NC_HAS_BZ2`. See [Github
  #2511](Unidata/netcdf-c#2511) and [Github
  #2512](Unidata/netcdf-c#2512) for more
  information.

* [Enhancement] Add new options to `nc-config`: `--has-multifilters`,
  `--has-stdfilters`, `--has-quantize`, `--plugindir`.  See [Github
  #2509](Unidata/netcdf-c#2509) for more
  information.

* [Bug Fix] Fix some errors detected in PR 2497. [PR
  #2497](Unidata/netcdf-c#2497) . See [Github
  #2503](Unidata/netcdf-c#2503).

* [Bug Fix] Split the remote tests into two parts: one for the
  remotetest server and one for all other external servers. Also add a
  configure option to enable the latter set. See [Github
  #2491](Unidata/netcdf-c#2491).

* [Bug Fix] Fix blosc plugin errors. See [Github
  #2461](Unidata/netcdf-c#2461).

* [Bug Fix] Fix support for reading arrays of HDF5 fixed size
  strings. See [Github
  #2466](Unidata/netcdf-c#2466).

* [Bug Fix] Fix some errors detected in [PR
  #2492](Unidata/netcdf-c#2492) . See [Github
  #2497](Unidata/netcdf-c#2497).

* [Enhancement] Add support for Zarr (fixed length) string type in
  nczarr. See [Github
  #2492](Unidata/netcdf-c#2492).

* [Bug Fix] Split the remote tests into two parts: one for the
  remotetest server and one for all other external servers. Also add a
  configure option to enable the latter set. See [Github
  #2491](Unidata/netcdf-c#2491).

* [Bug Fix] Fix support for reading arrays of HDF5 fixed size
  strings. See [Github
  #2462](Unidata/netcdf-c#2466).

* [Bug Fix] Provide a default enum const when fill value does not
  match any enum constant for the value zero. See [Github
  #2462](Unidata/netcdf-c#2462).

* [Bug Fix] Fix the json submodule symbol conflicts between libnetcdf
  and the plugin specific netcdf_json.h. See [Github
  #2448](Unidata/netcdf-c#2448).

* [Bug Fix] Fix quantize with CLASSIC_MODEL files. See [Github
  #2405](Unidata/netcdf-c#2445).

* [Enhancement] Add `--disable-quantize` option to `configure`.

* [Bug Fix] Fix CMakeLists.txt to handle all acceptable boolean values
  for -DPLUGIN_INSTALL_DIR. See [Github
  #2430](Unidata/netcdf-c#2430).

* [Bug Fix] Fix tst_vars3.c to use the proper szip flag. See [Github
  #2421](Unidata/netcdf-c#2421).

* [Enhancement] Provide a simple API to allow user access to the
  internal .rc file table: supports get/set/overwrite of entries of
  the form "key=value". See [Github
  #2408](Unidata/netcdf-c#2408).

* [Bug Fix] Use env variable USERPROFILE instead of HOME for windows
  and mingw. See [Github
  #2405](Unidata/netcdf-c#2405).

* [Bug Fix] Fix the nc_def_var_fletcher32 code in hdf5 to properly
  test value of the fletcher32 argument. See [Github
  #2403](Unidata/netcdf-c#2403).

## 4.9.0 - June 10, 2022

* [Enhancement] Add quantize functions nc_def_var_quantize() and
  nc_inq_var_quantize() to enable lossy compression. See [Github
  #1548](Unidata/netcdf-c#1548).

* [Enhancement] Add zstandard compression functions nc_def_var_zstandard() and
  nc_inq_var_zstandard(). See [Github
  #2173](Unidata/netcdf-c#2173).

* [Enhancement] Have netCDF-4 logging output one file per processor when used
  with parallel I/O. See [Github
  #1762](Unidata/netcdf-c#1762).

* [Enhancement] Improve filter installation process to avoid use of an extra
  shell script. See [Github
  #2348](Unidata/netcdf-c#2348).

* [Bug Fix] Get "make distcheck" to work See [Github
  #2343](Unidata/netcdf-c#2343).

* [Enhancement] Allow the read/write of JSON-valued Zarr attributes to allow for
  domain specific info such as used by GDAL/Zarr. See [Github
  #2278](Unidata/netcdf-c#2278).

* [Enhancement] Turn on the XArray convention for NCZarr files by
  default. WARNING, this means that the mode should explicitly specify "nczarr"
  or "zarr" even if "xarray" or "noxarray" is specified. See [Github
  #2257](Unidata/netcdf-c#2257).

* [Enhancement] Update the documentation to match the current filter
  capabilities See [Github
  #2249](Unidata/netcdf-c#2249).

* [Enhancement] Update the documentation to match the current filter
  capabilities. See [Github
  #2249](Unidata/netcdf-c#2249).

* [Enhancement] Support installation of pre-built standard filters into
  user-specified location. See [Github
  #2318](Unidata/netcdf-c#2318).

* [Enhancement] Improve filter support. More specifically (1) add
  nc_inq_filter_avail to check if a filter is available, (2) add the notion of
  standard filters, (3) cleanup szip support to fix interaction with NCZarr. See
  [Github #2245](Unidata/netcdf-c#2245).

* [Enhancement] Switch to tinyxml2 as the default xml parser implementation. See
  [Github #2170](Unidata/netcdf-c#2170).

* [Bug Fix] Require that the type of the variable in nc_def_var_filter is not
  variable length. See [Github
  #/2231](Unidata/netcdf-c#2231).

* [File Change] Apply HDF5 v1.8 format compatibility when writing to previous
  files, as well as when creating new files.  The superblock version remains at
  2 for newly created files.  Full backward read/write compatibility for
  netCDF-4 is maintained in all cases.  See [Github
  #2176](Unidata/netcdf-c#2176).

* [Enhancement] Add ability to set dataset alignment for netcdf-4/HDF5
  files. See [Github #2206](Unidata/netcdf-c#2206).

* [Bug Fix] Improve UTF8 support on windows so that it can use utf8
  natively. See [Github #2222](Unidata/netcdf-c#2222).

* [Enhancement] Add complete bitgroom support to NCZarr. See [Github
  #2197](Unidata/netcdf-c#2197).

* [Bug Fix] Clean up the handling of deeply nested VLEN types. Marks
  nc_free_vlen() and nc_free_string as deprecated in favor of
  ncaux_reclaim_data(). See [Github
  #2179](Unidata/netcdf-c#2179).

* [Bug Fix] Make sure that netcdf.h accurately defines the flags in the
  open/create mode flags. See [Github
  #2183](Unidata/netcdf-c#2183).

* [Enhancement] Improve support for msys2+mingw platform. See [Github
  #2171](Unidata/netcdf-c#2171).

* [Bug Fix] Clean up the various inter-test dependencies in ncdump for
  CMake. See [Github #2168](Unidata/netcdf-c#2168).

* [Bug Fix] Fix use of non-aws appliances. See [Github
  #2152](Unidata/netcdf-c#2152).

* [Enhancement] Added options to suppress the new behavior from [Github
  #2135](Unidata/netcdf-c#2135).  The options for
  `cmake` and `configure` are, respectively `-DENABLE_LIBXML2` and
  `--(enable/disable)-libxml2`. Both of these options defaul to 'on/enabled'.
  When disabled, the bundled `ezxml` XML interpreter is used regardless of
  whether `libxml2` is present on the system.

* [Enhancement] Support optional use of libxml2, otherwise default to ezxml. See
  [Github #2135](Unidata/netcdf-c#2135) -- H/T to
  [Egbert Eich](https://github.com/e4t).

* [Bug Fix] Fix several os related errors. See [Github
  #2138](Unidata/netcdf-c#2138).

* [Enhancement] Support byte-range reading of netcdf-3 files stored in private
  buckets in S3. See [Github
  #2134](Unidata/netcdf-c#2134)

* [Enhancement] Support Amazon S3 access for NCZarr. Also support use of the
  existing Amazon SDK credentials system. See [Github
  #2114](Unidata/netcdf-c#2114)

* [Bug Fix] Fix string allocation error in H5FDhttp.c. See [Github
  #2127](Unidata/netcdf-c#2127).

* [Bug Fix] Apply patches for ezxml and for selected oss-fuzz detected
  errors. See [Github #2125](Unidata/netcdf-c#2125).

* [Bug Fix] Ensure that internal Fortran APIs are always defined. See [Github
  #2098](Unidata/netcdf-c#2098).

* [Enhancement] Support filters for NCZarr. See [Github
  #2101](Unidata/netcdf-c#2101)

* [Bug Fix] Make PR 2075 long file name be idempotent. See [Github
  #2094](Unidata/netcdf-c#2094).


## 4.8.1 - August 18, 2021

* [Bug Fix] Fix multiple bugs in libnczarr. See [Github
  #2066](Unidata/netcdf-c#2066).

* [Enhancement] Support windows network paths (e.g. \\svc\...). See [Github
  #2065](Unidata/netcdf-c#2065).

* [Enhancement] Convert to a new representation of the NCZarr meta-data
  extensions: version 2. Read-only backward compatibility is provided. See
  [Github #2032](Unidata/netcdf-c#2032).

* [Bug Fix] Fix dimension_separator bug in libnczarr. See [Github
  #2035](Unidata/netcdf-c#2035).

* [Bug Fix] Fix bugs in libdap4. See [Github
  #2005](Unidata/netcdf-c#2005).

* [Bug Fix] Store NCZarr fillvalue as a singleton instead of a 1-element
  array. See [Github #2017](Unidata/netcdf-c#2017).

* [Bug Fixes] The netcdf-c library was incorrectly determining the scope of
  dimension; similar to the type scope problem. See [Github
  #2012](Unidata/netcdf-c#2012) for more information.

* [Bug Fix] Re-enable DAP2 authorization testing. See [Github
  #2011](Unidata/netcdf-c#2011).

* [Bug Fix] Fix bug with windows version of mkstemp that causes failure to
  create more than 26 temp files. See [Github
  #1998](Unidata/netcdf-c#1998).

* [Bug Fix] Fix ncdump bug when printing VLENs with basetype char. See [Github
  #1986](Unidata/netcdf-c#1986).

* [Bug Fixes] The netcdf-c library was incorrectly determining the scope of
  types referred to by nc_inq_type_equal. See [Github
  #1959](Unidata/netcdf-c#1959) for more information.

* [Bug Fix] Fix bug in use of XGetopt when building under Mingw. See [Github
  #2009](Unidata/netcdf-c#2009).

* [Enhancement] Improve the error reporting when attempting to use a filter for
  which no implementation can be found in HDF5_PLUGIN_PATH. See [Github
  #2000](Unidata/netcdf-c#2000) for more information.

* [Bug Fix] Fix `make distcheck` issue in `nczarr_test/` directory. See [Github
  #2007](Unidata/netcdf-c#2007).

* [Bug Fix] Fix bug in NCclosedir in dpathmgr.c. See [Github
  #2003](Unidata/netcdf-c#2003).

* [Bug Fix] Fix bug in ncdump that assumes that there is a relationship between
  the total number of dimensions and the max dimension id. See [Github
  #2004](Unidata/netcdf-c#2004).

* [Bug Fix] Fix bug in JSON processing of strings with embedded quotes. See
  [Github #1993](Unidata/netcdf-c#1993).

* [Enhancement] Add support for the new "dimension_separator" enhancement to
  Zarr v2. See [Github #1990](Unidata/netcdf-c#1990) for
  more information.

* [Bug Fix] Fix hack for handling failure of shell programs to properly handle
  escape characters. See [Github
  #1989](Unidata/netcdf-c#1989).

* [Bug Fix] Allow some primitive type names to be used as identifiers depending
  on the file format. See [Github
  #1984](Unidata/netcdf-c#1984).

* [Enhancement] Add support for reading/writing pure Zarr storage format that
  supports the XArray _ARRAY_DIMENSIONS attribute. See [Github
  #1952](Unidata/netcdf-c#1952) for more information.

* [Update] Updated version of bzip2 used in filter testing/functionality, in
  support of [Github #1969](Unidata/netcdf-c#1969).

* [Bug Fix] Corrected HDF5 version detection logic as described in [Github
  #1962](Unidata/netcdf-c#1962).

## 4.8.0 - March 30, 2021

* [Enhancement] Bump the NC_DISPATCH_VERSION from 2 to 3, and as a side effect,
  unify the definition of NC_DISPATCH_VERSION so it only needs to be defined in
  CMakeLists.txt and configure.ac. See [Github
  #1945](Unidata/netcdf-c#1945) for more information.

* [Enhancement] Provide better cross platform path name management. This
  converts paths for various platforms (e.g. Windows, MSYS, etc.) so that they
  are in the proper format for the executing platform. See [Github
  #1958](Unidata/netcdf-c#1958) for more information.

* [Bug Fixes] The nccopy program was treating -d0 as turning deflation on rather
  than interpreting it as "turn off deflation". See [Github
  #1944](Unidata/netcdf-c#1944) for more information.

* [Enhancement] Add support for storing NCZarr data in zip files. See [Github
  #1942](Unidata/netcdf-c#1942) for more information.

* [Bug Fixes] Make fillmismatch the default for DAP2 and DAP4; too many servers
  ignore this requirement.

* [Bug Fixes] Fix some memory leaks in NCZarr, fix a bug with long strides in
  NCZarr. See [Github #1913](Unidata/netcdf-c#1913) for
  more information.

* [Enhancement] Add some optimizations to NCZarr, dosome cleanup of code cruft,
  add some NCZarr test cases, add a performance test to NCZarr. See [Github
  #1908](Unidata/netcdf-c#1908) for more information.

* [Bug Fix] Implement a better chunk cache system for NCZarr. The cache now uses
  extendible hashing plus a linked list for provide a combination of
  expandibility, fast access, and LRU behavior. See [Github
  #1887](Unidata/netcdf-c#1887) for more information.

* [Enhancement] Provide .rc fields for S3 authentication: HTTP.S3.ACCESSID and
  HTTP.S3.SECRETKEY.

* [Enhancement] Give the client control over what parts of a DAP2 URL are URL
  encoded (i.e. %xx). This is to support the different decoding rules that
  servers apply to incoming URLS. See [Github
  #1884](Unidata/netcdf-c#1884) for more information.

* [Bug Fix] Fix incorrect time offsets from `ncdump -t`, in some cases when the
  time `units` attribute contains both a **non-zero** time-of-day, and a time
  zone suffix containing the letter "T", such as "UTC".  See [Github
  #1866](Unidata/netcdf-c#1866) for more information.

* [Bug Fix] Cleanup the NCZarr S3 build options. See [Github
  #1869](Unidata/netcdf-c#1869) for more information.

* [Bug Fix] Support aligned access for selected ARM processors.  See [Github
  #1871](Unidata/netcdf-c#1871) for more information.

* [Documentation] Migrated the documents in the NUG/ directory to the dedicated
  NUG repository found at https://github.com/Unidata/netcdf

* [Bug Fix] Revert the internal filter code to simplify it. From the user's
  point of view, the only visible change should be that (1) the functions that
  convert text to filter specs have had their signature reverted and renamed and
  have been moved to netcdf_aux.h, and (2) Some filter API functions now return
  NC_ENOFILTER when inquiry is made about some filter. Internally, the dispatch
  table has been modified to get rid of the complex structures.

* [Bug Fix] If the HDF5 byte-range Virtual File Driver is available )HDf5 1.10.6
  or later) then use it because it has better performance than the one currently
  built into the netcdf library.

* [Bug Fix] Fixed byte-range support with cURL > 7.69. See
  [Unidata/netcdf-c#1798].

* [Enhancement] Added new test for using compression with parallel I/O:
  nc_test4/tst_h_par_compress.c. See
  [Unidata/netcdf-c#1784].

* [Bug Fix] Don't return error for extra calls to nc_redef() for netCDF/HDF5
  files, unless classic model is in use. See
  [Unidata/netcdf-c#1779].

* [Enhancement] Added new parallel I/O benchmark program to mimic NOAA UFS data
  writes, built when --enable-benchmarks is in configure. See
  [Unidata/netcdf-c#1777].

* [Bug Fix] Now allow szip to be used on variables with unlimited dimension
  [Unidata/netcdf-c#1774].

* [Enhancement] Add support for cloud storage using a variant of the Zarr
  storage format. Warning: this feature is highly experimental and is subject to
  rapid evolution
  [https://www.unidata.ucar.edu/blogs/developer/en/entry/overview-of-zarr-support-in].

* [Bug Fix] Fix nccopy to properly set default chunking parameters when not
  otherwise specified. This can significantly improve performance in selected
  cases. Note that if seeing slow performance with nccopy, then, as a
  work-around, specifically set the chunking
  parameters. [Unidata/netcdf-c#1763].

* [Bug Fix] Fix some protocol bugs/differences between the netcdf-c library and
  the OPeNDAP Hyrax server. Also cleanup checksum handling
  [https://github.com/Unidata/netcdf-c/issues/1712].* [Bug Fix] IMPORTANT: Ncgen
  was not properly handling large data sections. The problem manifests as
  incorrect ordering of data in the created file. Aside from examining the file
  with ncdump, the error can be detected by running ncgen with the -lc flag (to
  produce a C file). Examine the file to see if any variable is written in pieces
  as opposed to a single call to nc_put_vara. If multiple calls to nc_put_vara are
  used to write a variable, then it is probable that the data order is
  incorrect. Such multiple writes can occur for large variables and especially
  when one of the dimensions is unlimited.

* [Bug Fix] Add necessary __declspec declarations to allow compilation of netcdf
  library without causing errors or (_declspec related) warnings
  [Unidata/netcdf-c#1725].

* [Enhancement] When a filter is applied twice with different
parameters, then the second set is used for writing the dataset
[Unidata/netcdf-c#1713].

* [Bug Fix] Now larger cache settings are used for sequential HDF5 file creates/opens on parallel I/O capable builds; see [Github #1716](Unidata/netcdf-c#1716) for more information.

* [Bug Fix] Add functions to libdispatch/dnotnc4.c to support
   dispatch table operations that should work for any dispatch
   table, even if they do not do anything; functions such as
   nc_inq_var_filter [Unidata/netcdf-c#1693].

* [Bug Fix] Fixed a scalar annotation error when scalar == 0; see [Github
  #1707](Unidata/netcdf-c#1707) for more information.

* [Bug Fix] Use proper CURLOPT values for VERIFYHOST and VERIFYPEER; the
  semantics for VERIFYHOST in particular changed. Documented in NUG/DAP2.md. See
  [Unidata/netcdf-c#1684].

* [Bug Fix][cmake] Correct an issue with parallel filter test logic in
  CMake-based builds.

* [Bug Fix] Now allow nc_inq_var_deflate()/nc_inq_var_szip() to be called for
  all formats, not just HDF5. Non-HDF5 files return NC_NOERR and report no
  compression in use. This reverts behavior that was changed in the 4.7.4
  release. See [Unidata/netcdf-c#1691].

* [Bug Fix] Compiling on a big-endian machine exposes some missing forward
  delcarations in dfilter.c.

* [File Change] Change from HDF5 v1.6 format compatibility, back to v1.8
  compatibility, for newly created files.  The superblock changes from version 0
  back to version 2.  An exception is when using libhdf5 deprecated versions
  1.10.0 and 1.10.1, which can only create v1.6 compatible format.  Full
  backward read/write compatibility for netCDF-4 is maintained in all cases.
  See [Github #951](Unidata/netcdf-c#951).

## 4.7.4 - March 27, 2020

* [Windows] Bumped packaged HDF5 to 1.10.6, HDF4 to 4.2.14, and libcurl to
  7.60.0.

* [Enhancement] Support has been added for HDF5-1.12.0.  See
  [Unidata/netcdf-c#1528].

* [Bug Fix] Correct behavior for the command line utilities when directly
  accessing a directory using utf8 characters. See
  [Github #1669] (Unidata/netcdf-c#1669),
  [Github #1668] (Unidata/netcdf-c#1668) and
  [Github #1666] (Unidata/netcdf-c#1666) for more information.

* [Bug Fix] Attempts to set filters or chunked storage on scalar vars will now
  return NC_EINVAL. Scalar vars cannot be chunked, and only chunked vars can
  have filters. Previously the library ignored these attempts, and always
  storing scalars as contiguous storage. See
  [Unidata/netcdf-c#1644].

* [Enhancement] Support has been added for multiple filters per variable.  See
  [Unidata/netcdf-c#1584].

* [Enhancement] Now nc_inq_var_szip retuns 0 for parameter values if szip is not
  in use for var. See [Unidata/netcdf-c#1618].

* [Enhancement] Now allow parallel I/O with filters, for HDF5-1.10.3 and
  later. See [Unidata/netcdf-c#1473].

* [Enhancement] Increased default size of cache buffer to 16 MB, from 4
  MB. Increased number of slots to 4133. See
  [Unidata/netcdf-c#1541].

* [Enhancement] Allow zlib compression to be used with parallel I/O writes, if
  HDF5 version is 1.10.3 or greater. See
  [Unidata/netcdf-c#1580].

* [Enhancement] Restore use of szip compression when writing data (including
  writing in parallel if HDF5 version is 1.10.3 or greater). See
  [Unidata/netcdf-c#1546].

* [Enhancement] Enable use of compact storage option for small vars in
  netCDF/HDF5 files. See [Unidata/netcdf-c#1570].

* [Enhancement] Updated benchmarking program bm_file.c to better handle very
  large files. See [Unidata/netcdf-c#1555].

* [Enhancement] Added version number to dispatch table, and now check version
  with nc_def_user_format(). See
  [Unidata/netcdf-c#1599].

* [Bug Fix] Fixed user setting of MPI launcher for parallel I/O HDF5 test in
  h5_test. See [Unidata/netcdf-c#1626].

* [Bug Fix] Fixed problem of growing memory when netCDF-4 files were opened and
  closed. See [Unidata/netcdf-c#1575 and
  Unidata/netcdf-c#1571].

* [Enhancement] Increased size of maximum allowed name in HDF4 files to
  NC_MAX_NAME. See [Unidata/netcdf-c#1631].

## 4.7.3 - November 20, 2019

* [Bug Fix]Fixed an issue where installs from tarballs will not properly compile
  in parallel environments.

* [Bug Fix] Library was modified so that rewriting the same attribute happens
  without deleting the attribute, to avoid a limit on how many times this may be
  done in HDF5. This fix was thought to be in 4.6.2 but was not. See
  [Unidata/netcdf-c#350].

* [Enhancement] Add a dispatch version number to netcdf_meta.h and
  libnetcdf.settings, in case we decide to change dispatch table in future. See
  [Unidata/netcdf-c#1469].

* [Bug Fix] Now testing that endianness can only be set on atomic ints and
  floats. See [Unidata/netcdf-c#1479].

* [Bug Fix] Fix for subtle error involving var and unlimited dim of the same
  name, but unrelated, in netCDF-4. See
  [Unidata/netcdf-c#1496].

* [Enhancement] Update for attribute documentation. See
  [Unidata/netcdf-c#1512].

* [Bug Fix][Enhancement] Corrected assignment of anonymous (a.k.a. phony)
  dimensions in an HDF5 file. Now when a dataset uses multiple dimensions of the
  same size, netcdf assumes they are different dimensions. See [GitHub
  #1484](Unidata/netcdf-c#1484) for more information.

## 4.7.2 - October 22, 2019

* [Bug Fix][Enhancement] Various bug fixes and enhancements.

* [Bug Fix][Enhancement] Corrected an issue where protected memory was being
  written to with some pointer slight-of-hand.  This has been in the code for a
  while, but appears to be caught by the compiler on OSX, under circumstances
  yet to be completely nailed down.  See
  [GitHub #1486] (Unidata/netcdf-c#1486) for more information.

* [Enhancement] [Parallel IO] Added support for parallel functions in MSVC. See
  [Github #1492](Unidata/netcdf-c#1492) for more
  information.

* [Enhancement] Added a function for changing the ncid of an open file.  This
  function should only be used if you know what you are doing, and is meant to
  be used primarily with PIO integration. See
  [GitHub #1483] (Unidata/netcdf-c#1483) and
  [GitHub #1487] (Unidata/netcdf-c#1487) for more information.

## 4.7.1 - August 27, 2019

* [Enhancement] Added unit_test directory, which contains unit tests for the
  libdispatch and libsrc4 code (and any other directories that want to put unit
  tests there). Use --disable-unit-tests to run without unit tests (ex. for code
  coverage analysis). See
  [GitHub #1458] (Unidata/netcdf-c#1458).

* [Bug Fix] Remove obsolete _CRAYMPP and LOCKNUMREC macros from code. Also
  brought documentation up to date in man page. These macros were used in
  ancient times, before modern parallel I/O systems were developed. Programmers
  interested in parallel I/O should see nc_open_par() and nc_create_par(). See
  [GitHub #1459](Unidata/netcdf-c#1459).

* [Enhancement] Remove obsolete and deprecated functions nc_set_base_pe() and
  nc_inq_base_pe() from the dispatch table. (Both functions are still supported
  in the library, this is an internal change only.) See [GitHub
  #1468](Unidata/netcdf-c#1468).

* [Bug Fix] Reverted nccopy behavior so that if no -c parameters are given, then
  any default chunking is left to the netcdf-c library to decide. See [GitHub
  #1436](Unidata/netcdf-c#1436).

## 4.7.0 - April 29, 2019

* [Enhancement] Updated behavior of `pkgconfig` and `nc-config` to allow the use
  of the `--static` flags, e.g. `nc-config --libs --static`, which will show
  information for linking against `libnetcdf` statically. See
  [Github #1360] (Unidata/netcdf-c#1360) and
  [Github #1257] (Unidata/netcdf-c#1257) for more information.

* [Enhancement] Provide byte-range reading of remote datasets. This allows
  read-only access to, for example, Amazon S3 objects and also Thredds Server
  datasets via the HTTPService access method.  See
  [GitHub #1251] (Unidata/netcdf-c#1251).

* Update the license from the home-brewed NetCDF license to the standard
  3-Clause BSD License.  This change does not result in any new restrictions; it
  is merely the adoption of a standard, well-known and well-understood license
  in place of the historic NetCDF license written at Unidata.  This is part of a
  broader push by Unidata to adopt modern, standardized licensing.

## 4.6.3 - February 28, 2019

* [Bug Fix] Correctly generated `netcdf.pc` generated either by `configure` or
  `cmake`.  If linking against a static netcdf, you would need to pass the
  `--static` argument to `pkg-config` in order to list all of the downstream
  dependencies.  See
  [Github #1324](Unidata/netcdf-c#1324) for more information.

* Now always write hidden coordinates attribute, which allows faster file opens
  when present. See
  [Github #1262](Unidata/netcdf-c#1262) for more information.

* Some fixes for rename, including fix for renumbering of varids after a rename
  (#1307), renaming var to dim without coordinate var. See
  [Github #1297] (Unidata/netcdf-c#1297).

* Fix of NULL parameter causing segfaults in put_vars functions. See
   [Github #1265] (Unidata/netcdf-c#1265) for more information.

* Fix of --enable-benchmark benchmark tests
   [Github #1211] (Unidata/netcdf-c#1211)

* Update the license from the home-brewed NetCDF license to the standard
  3-Clause BSD License.  This change does not result in any new restrictions; it
  is merely the adoption of a standard, well-known and well-understood license
  in place of the historic NetCDF license written at Unidata.  This is part of a
  broader push by Unidata to adopt modern, standardized licensing.

* [BugFix] Corrected DAP-related issues on big-endian machines. See
  [Github #1321] (Unidata/netcdf-c#1321),
  [Github #1302] (Unidata/netcdf-c#1302) for more information.

* [BugFix][Enhancement] Various and sundry bugfixes and performance
  enhancements, thanks to \@edhartnett, \@gsjarrdema, \@t-b, \@wkliao, and all
  of our other contributors.

* [Enhancement] Extended `nccopy -F` syntax to support multiple variables with a
  single invocation. See
  [Github #1311](Unidata/netcdf-c#1311) for more information.

* [BugFix] Corrected an issue where DAP2 was incorrectly converting signed
  bytes, resulting in an erroneous error message under some circumstances. See
  [GitHub #1317] (Unidata/netcdf-c#1317) for more
  information.  See
  [Github #1319] (Unidata/netcdf-c#1319) for related information.

* [BugFix][Enhancement] Modified `nccopy` so that `_NCProperties` is not copied
  over verbatim but is instead generated based on the version of `libnetcdf`
  used when copying the file.  Additionally, `_NCProperties` are displayed
  if/when associated with a netcdf3 file, now. See
  [GitHub#803] (Unidata/netcdf-c#803) for more information.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants