NetCDF-4 limits the number of times an attribute can be modified

[ckhroulev]

Summary

Attributes of a variable or a group in a NetCDF-4 file cannot be modified more than 2^16 times. (This includes special attributes such as _NCProperties.)

This refers to the total number of modifications of all attributes of a variable.

Here

• Creating an attribute counts as a modification.
• A "modification" is the number of nc_put_att_... calls followed by synchronizing the dataset to disk. For example, three nc_put_att_int(...) calls followed by one nc_sync(...) is one modification. (nc_enddef and nc_close calls also cause synchronization and have the same effect.)

In particular:

• a variable may not have more than 2^16 attributes (including ones used by the NetCDF library),
• if a variable has 2^16 attributes then their values cannot be changed (i.e. this will cause a "crash").

This issue was first reported by Heiko Klein in March of 2015.

It would be great if this behavior could be changed to match the NetCDF-3 library. If this is not feasible please consider documenting it so that it does not surprise others the way it surprised me.

Thanks!

Details

NetCDF-4 enables attribute creation order tracking using the HDF5 call H5Pset_attr_creation_order (see the NetCDF manual).

With creation order tracking turned "on" the HDF5 library limits the number of times an attribute can be created. This HDF5 "feature" is present in both branches (1.8.x and 1.10) and is not documented. See the code for the place where the limit is enforced and the definition of the limit.

In NetCDF-4 attribute modification is performed in three steps:

1. If an attribute exists, delete it.
2. Re-create an attribute.
3. Write the new value.

So, each attribute modification increments the attribute order creation index and the program crashes once it reaches 65535.

Why does this matter?

1. This is a change in behavior compared to NetCDF-3.
2. This behavior is not documented.
3. It is important to be able to update metadata.
4. The number of attribute modifications (the "attribute creation order index") is saved in the file. This means that a user may acquire a file that has metadata that cannot be updated. (This can happen even if this user's workflow is perfectly normal.) It is easy to modify the code below to produce a file like that.

Reproducing the issue

This code mimics the use case in which a file is modified using several tools, showing that the attribute creation order index is stored in a file.

#include <stdio.h>              /* printf */
#include <stdlib.h>             /* exit */

#include <netcdf.h>

static const char *filename = "netcdf_bug_report.nc";

static void check(const char *func, int ierr, int i) {
  if (ierr != NC_NOERR) {
    printf("NetCDF error: %s failed with %s at iteration %d\n", func, nc_strerror(ierr), i);
    exit(ierr);
  }
}

int main(int argc, char **argv) {
  int ierr = NC_NOERR, ncid = 0, i = 0;

  /* Create a NetCDF-4 file. */
  ierr = nc_create(filename, NC_NETCDF4, &ncid); check("nc_create", ierr, i);
  ierr = nc_close(ncid); check("nc_close", ierr, i);

  /* Modify an attribute 2^16 times. */
  for (i = 0; i < 65536; ++i) {
    ierr = nc_open(filename, NC_WRITE, &ncid); check("nc_open", ierr, i);
    ierr = nc_put_att_int(ncid, NC_GLOBAL, "attribute", NC_INT, 1, &i); check("nc_put_att_int", ierr, i);
    ierr = nc_close(ncid); check("nc_close", ierr, i);
  }

  return 0;
}

Expected result

This code should create a NetCDF-4 file with one global attribute with the value of 65535.

Observed result

~/netcdf4-bug $ ./bug_report_1
HDF5-DIAG: Error detected in HDF5 (1.8.18) thread 0:
  #000: H5Adeprec.c line 190 in H5Acreate1(): unable to create attribute
    major: Attribute
    minor: Unable to initialize object
  #001: H5Aint.c line 280 in H5A_create(): unable to create attribute in object header
    major: Attribute
    minor: Unable to insert object
  #002: H5Oattribute.c line 310 in H5O_attr_create(): attribute creation index can't be incremented
    major: Attribute
    minor: Unable to increment reference count
NetCDF error: nc_close failed with NetCDF: Can't open HDF5 attribute at iteration 65534

Alternative test code

This code avoids closing and re-opening the file. (This is significantly faster.)

#include <stdio.h>              /* printf */
#include <stdlib.h>             /* exit */

#include <netcdf.h>

static const char *filename = "netcdf_bug_report.nc";

static void check(const char *func, int ierr, int i) {
  if (ierr != NC_NOERR) {
    printf("NetCDF error: %s failed with %s at iteration %d\n", func, nc_strerror(ierr), i);
    exit(ierr);
  }
}

int main(int argc, char **argv) {
  int ierr = NC_NOERR, ncid = 0, i = 0;

  /* Create a NetCDF-4 file. */
  ierr = nc_create(filename, NC_NETCDF4, &ncid); check("nc_create", ierr, i);
  ierr = nc_enddef(ncid); check("nc_enddef", ierr, i);

  /* Modify an attribute 2^16 times. */
  for (i = 0; i < 65536; ++i) {
    ierr = nc_redef(ncid); check("nc_redef", ierr, i);
    ierr = nc_put_att_int(ncid, NC_GLOBAL, "attribute", NC_INT, 1, &i); check("nc_put_att_int", ierr, i);
    ierr = nc_enddef(ncid); check("nc_enddef", ierr, i);
  }

  ierr = nc_close(ncid); check("nc_close", ierr, i);

  return 0;
}

Output

HDF5-DIAG: Error detected in HDF5 (1.8.18) thread 0:
  #000: H5Adeprec.c line 190 in H5Acreate1(): unable to create attribute
    major: Attribute
    minor: Unable to initialize object
  #001: H5Aint.c line 280 in H5A_create(): unable to create attribute in object header
    major: Attribute
    minor: Unable to insert object
  #002: H5Oattribute.c line 310 in H5O_attr_create(): attribute creation index can't be incremented
    major: Attribute
    minor: Unable to increment reference count
NetCDF error: nc_enddef failed with NetCDF: Can't open HDF5 attribute at iteration 65534

Environment Information

• What platform are you using? (please provide specific distribution/version in summary)
  • Linux
  • Windows
  • OSX
  • Other
  • NA
• 32 and/or 64 bit?
  • 32-bit
  • 64-bit
• What build system are you using?
  • autotools (configure)
  • cmake
• Can you provide a sample netCDF file or C code to recreate the issue?
  • Yes (please attach to this issue, thank you!)
  • No
  • Not at this time

[ckhroulev]

Code needed to reproduce the issue, as attachments: bug_report_1.c.txt, bug_report_2.c.txt.

(Unfortunately GitHub does not seem to support attachments with the .c filename extension.)

[Dave-Allured]

Can Netcdf-4 attribute modification be simply changed so that the attribute is NOT deleted and re-created, if a pre-existing attribute is fully compatible with the modification? I think this would mean that the new data type and number of elements would agree with the previous attribute.

This change would greatly decrease the impacts of this problem within my data management group.

[DennisHeimbigner]

I am starting a conversation with the HDF group to see if they can provide
some insight into this.

[DennisHeimbigner]

I was thinking about this, and I wonder if you are not, in a sense,
misusing the attribute mechanism. In a semantic sense, one
thinks of attributes as relatively stable.
Have you considered converting these heavily modified attributes to be
true variables. If you make them variables, then you can modify them
as much as you want.

[Dave-Allured]

I have never thought of attributes in the sense of relatively stable. I think attributes are the best place for storing descriptive metadata. Storage systems should easily support overwrites, regardless of the frequency.

I have one application where I store actual_min and actual_max at every time step and every level for a massive netcdf file. That requires 2-D indexing. THAT is an appropriate situation for using a variable for metadata.

I will ask my team about switching from attributes to variables to save actual_range on a per-file basis.

[ckhroulev]

Here's one more reason to allow changing attributes an arbitrary number of times: Appendix A: Attribute Conventions of the NetCDF Manual:

history

A global attribute for an audit trail. This is a character array with a line for each invocation of a program that has modified the dataset. Well-behaved generic netCDF applications should append a line containing: date, time of day, user name, program name and command arguments.

[DennisHeimbigner]

Your use case: using attribute to store min/max properties about a var is a good one.
I can see that for a var whose content is changing a lot, the attributes would also have
to change.
I guess we will have to see that the hdf developers say about this.
I can imagine also that re-creating attributes is the problem (as opposed to changing
it value). Assuming hdf5 supports this (I have not looked), we can probably modify
the netcdf-c library to avoid deleting and then re-creating the attribute and instead
just change its value.

[DennisHeimbigner]

I received this from the hdf group.

We discussed this issue in the developer's meeting. The creation index
is stored in a 16-bit value. Code would either have to be written to
get around the issue, or the creation index size would have to be increased
(which would require a file format change). Either way, we would need funding
to change this.
HOWEVER, the developer suggested overwriting the existing attribute, which
would not increment the creation index. Could you try that?
It should be possible to modify netcdf-c to just re-write the value if the name and type
stay the same. I will need to experiment to see if the size can alter or not.

[edhartnett]

I cannot address the HDF5 part of the problem, but I have refactored the code so that if an attribute exists, and is the same type and length, then it is reused instead of deleted/recreated.

I have added this to tst_atts.c and it now works fine:

      /* Modify an attribute 2^16 times. This test suggested by
       * Constantine Khrulev. */
      for (i = 0; i < 65536; ++i)
      {
         if (nc_open(FILE_NAME, NC_WRITE, &ncid)) ERR;
         if (nc_put_att_int(ncid, NC_GLOBAL, "attribute", NC_INT, 1, &i)) ERR;
         if (nc_close(ncid)) ERR;
      }

So hopefully this will resolve the issue in almost all actual cases. I assume that not many people are also changing the type and the length of the attribute when they change its value.

This is in my branch ejh_loop_cleanup and will be presented as a PR in a while, when its turn comes.

[Dave-Allured]

Thanks Ed, for getting this done. I agree, this should greatly reduce incidents of this confounding problem in the wild.

[ckhroulev]

Unfortunately this issue is not fixed by #1119: the issue is present in the current master branch (I tested 108e938). Test cases included above still fail and I could not find the test mentioned by @edhartnett in #350 (comment) anywhere in the repository.

Please see #1521 for an improved fix.

[edhartnett]

Rewriting the att never made it somehow. I am working on a new PR now...