Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow parallel writes to use zlib #1581

Closed
wants to merge 6 commits into from
Closed

Allow parallel writes to use zlib #1581

wants to merge 6 commits into from

Conversation

edwardhartnett
Copy link
Contributor

Fixes #1580

@WardF WardF self-assigned this Dec 19, 2019
@WardF WardF added this to the 4.7.4 milestone Dec 19, 2019
@WardF
Copy link
Member

WardF commented Dec 19, 2019

Is there a minimum version of HDF5 required to support this? Or is it a feature that just has been available in parallel HDF5 but not enabled in netCDF?

@gsjaardema
Copy link
Contributor

I know that 1.10.3 and 1.10.4 had bugs that caused "Too many communicators" failures that was fixed in 1.10.5.

@gsjaardema
Copy link
Contributor

Looks like parallel compression output was released in 1.10.2:

In the past, HDF5 parallel applications could read compressed data, but couldn’t create and write data using compression and other filters like Fletcher32 for raw data checksum. This restriction was removed in HDF5 1.10.2. 

@WardF
Copy link
Member

WardF commented Dec 19, 2019

@gsjaardema thanks! @edwardhartnett I think we need to find a way to fencepost this so that it doesn't break builds for people still using older versions of libhdf5.

@edwardhartnett
Copy link
Contributor Author

Yes, I should have made it clear that this PR is not quite ready to merge. I just wanted to try it on the CI.

I would like to:

  • Add more tests.
  • Better handle cases of older HDF5
  • See if this also applies to fletcher32 (for completeness - I don't think many people use fletcher32.)

@edwardhartnett
Copy link
Contributor Author

I'm just going to take this down while I work on it a bit more. I will put up a complete PR in a bit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Allow parallel I/O writes to use zlib
3 participants