Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix checking for HDF5 max dims, no longer re-create atts if not needed, confirm behavior for HDF5 cyclical files, allow user to set mpiexec #1119

Merged

Conversation

edhartnett
Copy link
Contributor

@edhartnett edhartnett commented Aug 21, 2018

Fixes #320.
Fixes #322.
Fixes #350.
Fixes #433.
Fixes #1099.
Part of #857.
Fixes #302.

This PR continues the work of preparing for lazy vars, and I also cleared up some old issues along the way.

In this PR:

  • Change constant for NC_FILL_INT in netcdf.h to int type (instead of long int).
  • Check for max HDF5 number of dims (32) in nc_def_var.
  • If user re-writes an att of the same type and size, just reuse the att, instead of deleting and re-creating.
  • Removed some unneeded base_pe functions in netcdf4 layer.
  • Tests for all these changes.
  • Documentation fixes along the way.
  • Allows user to set mpiexec program for parallel tests.

@edhartnett edhartnett changed the title Ejh loop cleanup 2 Fix checking for HDF5 max dims, no longer re-create atts if not needed, confirm behavior for HDF5 cyclical files Aug 21, 2018
@edhartnett edhartnett closed this Aug 21, 2018
@edhartnett edhartnett reopened this Aug 28, 2018
@edhartnett
Copy link
Contributor Author

edhartnett commented Aug 28, 2018

@WardF this also includes the changes in #1116, which I have taken down.

@edhartnett edhartnett changed the title Fix checking for HDF5 max dims, no longer re-create atts if not needed, confirm behavior for HDF5 cyclical files Fix checking for HDF5 max dims, no longer re-create atts if not needed, confirm behavior for HDF5 cyclical files, allow user to set mpiexec Aug 28, 2018
@edhartnett
Copy link
Contributor Author

@WardF can we get this merged soon please? I think you want this in the next release. There are some important bug fixes and the removal of an unneeded file lookup for every netcdf call, which will improve performance for power-users with thousands of files open.

@WardF WardF added this to the 4.7.0 milestone Sep 5, 2018
@WardF WardF self-assigned this Sep 5, 2018
@WardF
Copy link
Member

WardF commented Sep 6, 2018

Double checking something on Windows, but looks good will merge shortly.

@WardF WardF merged commit 17f8eb1 into Unidata:master Sep 6, 2018
@edhartnett
Copy link
Contributor Author

Thanks!

@edhartnett edhartnett deleted the ejh_loop_cleanup_2 branch September 6, 2018 17:52
ckhroulev added a commit to ckhroulev/netcdf-c that referenced this pull request Nov 13, 2019
1) We have to use H5Tequal() to compare HDF5 type IDs.
2) When checking if we can re-use an NC_CHAR attribute it is enough to
   compare data types (H5Tequal() takes care of the size comparison).
3) This commit adds missing code (reuse_att was set but not used).

Now an attribute in a NetCDF-4 file can be modified as many times as
necessary, as long its type and length remain the same.

Modifications changing either type or length of an attribute require
deleting and re-creating an attribute which increments the attribute
order creation index. Once this index reaches 65535 all attribute
modifications (for a particular group or variable) will fail.

For reference:

Issue 350 title: NetCDF-4 limits the number of times an attribute can
be modified

Pull request 1119 title: Fix checking for HDF5 max dims, no longer
re-create atts if not needed, confirm behavior for HDF5 cyclical
files, allow user to set mpiexec
@edhartnett
Copy link
Contributor Author

OK, you are quite right. Sorry about that, not sure what happened to that fix and test, but there was a lot going on in that PR - too much I guess. ;-)

@WardF can you reopen this issue? I will take a look...

WardF added a commit that referenced this pull request Nov 15, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment