Possible memory leak when opening a netCDF-4 file

[planetMDX]

When running:

import Foundation
import SwiftNetCDF

setenv("HDF5_USE_FILE_LOCKING", "FALSE", 1)
while true {
    if let file = try NetCDF.open(path: "/home/plane/Downloads/test_echam_spectral-deflated.nc", allowUpdate: false) {
    }
}

the memory consumption keeps growing. In this case it hit 346 MiB after 10 minutes, when I shut it down.

Note

I tested this with GOTEX.C130-example.nc , smith_sandwell_topo_v8_2.nc, sresa1b_ncar_ccsm3-example.nc, test_echam_spectral-deflated.nc and test_echam_spectral.nc from https://www.unidata.ucar.edu/software/netcdf/examples/files.html and only test_echam_spectral-deflated.nc showed this behaviour.

[patrick-zippenfenig]

Strange. There is not much code involved that could leak. It basically wraps netcdf open/close and that's it.

Could you try to run it with Xcode memory leak profiler to see where the leak occurs? I suspect the leak will be inside the NetCDF/HDF5 library.

[planetMDX]

These are the stacktraces of allocations over a 4 minute window which didn't get deallocated in the following 4 minutes, and they do occur in the NetCDF/HDF5 libraries. The growing memory use doesn't get classified as a leak as it seems to still be reachable, but not enough get deallocated.

The relative distribution of allocations mostly matches the the distribution which I profiled on Linux (by which I mean the source of the leak seems to be the same on MacOS and Linux).

[patrick-zippenfenig]

As expected everything is happening inside the C NetCDF4/HDF5 library. This Swift wrapper is not causing this leak.

HDF5 is using a cache when reading compressed chunks. Maybe there is a way to clear it (or disable it for testing). You can also open NetCDF4 files directly with HDF5 to get better control. I once started to implement a native HDF5 Swift implementation, but stopped at decoding compressed chunks. The binary HDF5 format is however relatively easy to understand.

[patrick-zippenfenig]

Any news? Otherwise please consider closing this ticket, as I assume the NetCDF4/HDF5 library is causing the leak

[planetMDX]

I have opened a issue at Unidata/netcdf-c#2589

[patrick-zippenfenig]

Good work on catching the leak in the HDF library. Much appreciated!