growing memory in netCDF library?

[edwardhartnett]

@epourmal reported that netCDF has some memory problem, where memory use grows over time.

I don't want to forget to look into this important issue. Elena, can you provide any more info?

[gsjaardema]

This is described in HDF5 Forum at https://forum.hdfgroup.org/t/investigating-memory-growth-with-hdf5-netcdf-4/6187

[gsjaardema]

Down toward the bottom is the following result:

close and reopen dataset instead of keeping them all open – This had a large affect. The peak for the “smaller” model was dropped from 114 MiB down to 31 MiB (83 MiB decrease). In this case, the execution time increased by about 1.5X. However, on a larger model, the peak dropped from 172 MiB down to 99 MiB (73 MiB decrease) and the execution time decreased by about 10%.

This seems to be a good option to investigate with the NetCDF 4 developers. My current hack is just to close and immediately reopen right after the H5Dwrite call which probably isn’t the best implementation, but does show the different behavior.

I haven’t tried any of the metadata cache options yet.

[gsjaardema]

Here is a patch showing the hack referenced in previous comment.
dataset-close-open.patch.txt

[epourmal]

Hi Ed,

I believe I pointed you to HDFFV-10915. Can you see the issue? Please let me know if you still cannot see JIRA issues and do not have access to our repo.

[edwardhartnett]

Unfortunately I do not have a good test for this. My memory checking during tests is not showing the problem.

Does anyone have a simple C program that can demonstrate the memory growth?

[mathomp4]

@edwardhartnett We might have encountered something similar with netCDF-Fortran in GEOS (so probably in netCDF-C). We have a bit of a reproducer, but I need to make it a bit more portable. Give a bit of time and I'll try and get it to you.

ETA: Actually, I was out of the loop. It turns out we thought it was netCDF causing most of our issues, but through diligent Valgrind runs, we found a that other libraries were the dominant cause. But by the time I was informed of that, I did make an open/close tester with netCDF-Fortran and found that there does seem to be a memory leak, but not a major one. After opening/closing a file 100000 times, where the main loop is:

   do i = 1, 100000
      call check( nf90_open(FILE_NAME, NF90_NOWRITE, ncid) )
      call check( nf90_close(ncid) )
   end do

about 400 MB seems to be used up:

 Initial Committed_AS: 13478.5469 MB; Percentage:  42.204
 Final   Committed_AS: 13915.7227 MB; Percentage:  43.573

I'm pretty certain this is not due to the rest of the program. There are two calls to read /proc/meminfo but if I run the job without the open/close loop, I get no memory difference start and end.

[gsjaardema]

OK, sorry for the delay, but I finally had a chance to take a better look at this and come up with a reproducer. This isn't fixed by #1634

There should be two files. close.txt and open.txt. Both of these are from valgrind --tool=massif' and show the memory use during the running of my modified tst_vars.c. As a summary, the max memory use inopen.txt` which is from the current NetCDF master pull (de4c487) is 10.8 MByte.

The max memory use in close.txt which is from a modified version is 2.6 MByte. The change between the two is that I close the HDF5 dataset between writes. In my sample code, I close and then immediately reopen the dataset, but a better fix would probably close and then defer the reopen until the next write. The patch is also attached.

In a more complicated example using one of my applications, the corresponding difference is 101.4 MByte vs 16.6 MByte.

The difference is related to chunking since if I don't enable chunking (via the shuffle filter), the memory use between the two options is the same.

The disadvantage of this change is that it does slow down the execution for small runs; for large runs, it seems to be faster (this is mentioned in previous comments above)
open.txt
close.txt
close_dataset.patch.txt

[gsjaardema]

@epourmal

[gsjaardema]

And the forgotten reproducer...
tst_vars.c.txt

[edwardhartnett]

@gsjaardema are you still seeing this in current master? I thought I had fixed it...

[gsjaardema]

Yes, it is a different issue than what you fixed.