memory leak in MOM6

[junwang-noaa]

Description

The high resolution C384 coupled test failed due to the memory leak issue. The log files showed that MOM6 has memory leak issue.

20210819 210631.993 INFO PET312 Leaving MOM update_ocean_model: - MemInfo: VmPeak: 2112852 kB
...
20210819 211322.476 INFO PET312 Leaving MOM update_ocean_model: - MemInfo: VmPeak: 2112852 kB
20210819 211405.081 INFO PET312 Leaving MOM update_ocean_model: - MemInfo: VmPeak: 2611544 kB
...

To Reproduce:

Give explicit steps to reproduce the behavior.

1. Check out ufs-weather-model develop branch and turn on the memory profile check in tests/parm/nems.configure.cpld.IN by:
@@ -29,7 +29,7 @@ OCN_petlist_bounds:             @[ocn_petlist_bounds]
 OCN_attributes::
   Verbosity = 0
   DumpFields = false
-  ProfileMemory = false
+  ProfileMemory = true

2. run coupled test cpld_bmark_wave_v16_p7b.
3. Check the MOM PET files for memory info

Additional context

Some related discussion in issue #746:

I checked Jessica's run directory, just to confirm that the memory increase
is reduced, it has a ~2% memory increase just after 14 days, then memory
stays unchanged, just like the previous run without CA. MOM6 memory
increases from 3660532 kB to 4217332 kB, the increases only happen when
time steps are multiple of 12 (12,24,36, 60, 228...)

@junwang-noaa I tested the latest 3 commit of MOM6 in ufs, all of them have memory leak issue. Below is from Marshall Ward:

**I have started doing more aggressive memory checking, and recently
fixed many of them, but we know of a few that are not yet fixed.

Nearly all of the leaks are because we do not properly call the
MOM_end_*() functions during the finalization, so do not normally
affect the model during the run.

We are planning to enable valgrind testing once we've fixed all the
known leaks, but this is on hold until we finish up some other
projects.**

[DeniseWorthen]

See associated Disscusion #779

[arunchawla-NOAA]

@junwang-noaa @DeniseWorthen @JessicaMeixner-NOAA @jiandewang is there an update on this ? Is there development happening on this at the GFDL end ?

[jiandewang]

Marshall is aware of this and it's on his to do list. In the discussion #799 it can be seen clearly that the minor memory leak is directly related to the using FMS module to write model output which Marshall believes it's the main cause.

[DeniseWorthen]

@jiandewang Have you heard anything recently about this issue? Earlier Marshall mentions that it mostly is an issue w/ the MOM_end_*() functions. Later you mention FMS and model output as the main culprit. I'm a little confused which thing is suspected as the cause of the memory leak.

[marshallward]

It's been a long time since I looked at this. I did not find any significant loss of memory in MOM6, other than memory which was not deallocated at cleanup (i.e. MOM_end_*). Realistically, this is not going to have much impact on any simulations.

There were some memory holes coming from FMS, but these were in the FMS1 I/O and may have been fixed in the FMS2 I/O. Even then, I think we're talking something like O(10M) per rank, which is not huge.

However, this was from our benchmark tests, and may not reflect production runs UFS runs. There could be some untested components in MOM6 which have poor memory usage.

[DeniseWorthen]

@marshallward Thanks for the update. We're trying to clean up/close old issues which led me to ask the status.

@jiandewang Looking at the discussion #779, maybe it could you repeat the test you did previously and report the current status? At that point we can make a decision whether to close the issue.

[jiandewang]

@marshallward Thanks for the update. We're trying to clean up/close old issues which led me to ask the status.

@jiandewang Looking at the discussion #779, maybe it could you repeat the test you did previously and report the current status? At that point we can make a decision whether to close the issue.

@DeniseWorthen sure I will repeat our previous test (sorry for the delayed response as I was out of town last week)

[jiandewang]

@DeniseWorthen I checkout the ufs-weather-model code (Jan 23, 2023 commit, hash # 70de7ef) and repeated what I did before by running C96 coupled with 1x1 ocean and ice for 10 days. Overall I don't see memory leak issue. This figure is from two randomly selected PE memory vmpeak values:

this one is for all PE (percentage of vmpeak value wrt. it's 2nd step value)

I think we can close this issue.

[junwang-noaa]

@jiandewang Thanks for the results. I will close the issue.