GcMemoryInfo returns incorrect values in .NET 8 on macOS 14 Sonoma

[mrahl]

Description

We build a library that uses the ratio between GCMemoryInfo.MemoryLoadBytes/GCMemoryInfo.TotalAvailableMemoryBytes returned form the GC.GetGCMemoryInfo API to monitor memory consumption an compact a cache when consumption runs above thresholds. We now have a customer report that the cache compact is constantly triggered and it turn out it is because this ratio is seemingly always at 99% when running in .NET 8 on Mac OS 14. Running on .NET 7 on the same machine returns the actual value (corresponding to what is seen in system tools). The value is also correct on Windows and in Linux containers in .NET 8.

Reproduction Steps

Note: I don't actually have Mac to reproduce this on, so this is a bit hypothetical:

1. Create an app that calls the GC.GetGCMemoryInfo API and outputs/logs the GCMemoryInfo.MemoryLoadBytes/GCMemoryInfo.TotalAvailableMemoryBytes ratio.
2. Add a forced GC.Collect() call before, to make sure the values are updated.
3. Run the app on MacOS 14.

Expected behavior

The output memory load ratio is roughly the same as can be seen in the OS tools.

Actual behavior

The output memory load is always basically 100% memory utilization, regardless of the actual memory pressure as can be seen in OS tools.

Note: This only happens on Mac OS X 14, on Windows and Linux (container) the output is the expected.

Regression?

Yes, in .NET 7 the returned value is correct.

Known Workarounds

No response

Configuration

.NET 8 RTM
Mac OS 14
ASP.NET Core application

Other information

No response

[ghost]

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

We build a library that uses the ratio between GCMemoryInfo.MemoryLoadBytes/GCMemoryInfo.TotalAvailableMemoryBytes returned form the GC.GetGCMemoryInfo API to monitor memory consumption an compact a cache when consumption runs above thresholds. We now have a customer report that the cache compact is constantly triggered and it turn out it is because this ratio is seemingly always at 99% when running in .NET 8 on Mac OS 14. Running on .NET 7 on the same machine returns the actual value (corresponding to what is seen in system tools). The value is also correct on Windows and in Linux containers in .NET 8.

Reproduction Steps

Note: I don't actually have Mac to reproduce this on, so this is a bit hypothetical:

1. Create an app that calls the GC.GetGCMemoryInfo API and outputs/logs the GCMemoryInfo.MemoryLoadBytes/GCMemoryInfo.TotalAvailableMemoryBytes ratio.
2. Add a forced GC.Collect() call before, to make sure the values are updated.
3. Run the app on MacOS 14.

Expected behavior

The output memory load ratio is roughly the same as can be seen in the OS tools.

Actual behavior

The output memory load is always basically 100% memory utilization, regardless of the actual memory pressure as can be seen in OS tools.

Note: This only happens on Mac OS X 14, on Windows and Linux (container) the output is the expected.

Regression?

Yes, in .NET 7 the returned value is correct.

Known Workarounds

No response

Configuration

.NET 8 RTM
Mac OS 14
ASP.NET Core application

Other information

No response

Author:	mrahl
Assignees:	-
Labels:	area-GC-coreclr
Milestone:	-

[OttoG]

This issue is very easy to reproduce. Here is a Program.cs file for a simple console app written according to @mrahl’s reproduction steps:

var format = new System.Globalization.NumberFormatInfo()
{
    NumberGroupSeparator = ","
};
GC.Collect();
var info = GC.GetGCMemoryInfo();
Console.WriteLine("Framework version: " + Environment.Version);
Console.WriteLine("MemoryLoadBytes: " + info.MemoryLoadBytes.ToString("N0", format));
Console.WriteLine("TotalAvailableMemoryBytes: " + info.TotalAvailableMemoryBytes.ToString("N0", format));

And a memory-info-issue.csproj file to put in the same folder:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net8.0</TargetFramework>
    <RootNamespace>memory_info_issue</RootNamespace>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>
</Project>

The output will of course depend on the machine where the code is run. Here is the output of dotnet run on an ARM M2 Mac with macOS Sonoma 14.1.1 and 24 GB of installed memory and a handful of apps running:

Framework version: 8.0.0
MemoryLoadBytes: 25,512,105,738
TotalAvailableMemoryBytes: 25,769,803,776

If net8.0 on line 4 of the csproj file is changed to net7.0, the output in the very same context is instead:

Framework version: 7.0.14
MemoryLoadBytes: 17,781,164,605
TotalAvailableMemoryBytes: 25,769,803,776

In both instances, the macOS Activity Monitor reported around 18 GB of “Memory Used” while the code was run.

[OttoG]

I can add the .NET 8.0 output of the issue reproduction code under a Linux virtual machine – Ubuntu 22.04.3 LTS (GNU/Linux 5.15.0-88-generic aarch64) – with 4 GB of memory (a server setup with only some basic services running):

Framework version: 8.0.0
MemoryLoadBytes: 533,193,523
TotalAvailableMemoryBytes: 4,101,488,640

And under Windows 11 (Arm64) in a virtual machine – also with 4 GB of memory:

Framework version: 8.0.0
MemoryLoadBytes: 3,474,404,720
TotalAvailableMemoryBytes: 4,289,388,544

[Maoni0]

@janvorli mentioned to me that the difference between 7 and 8 is in 8 we are doing "available memory - free" whereas in 7 we are doing "active + inactive + wired".

I'm wondering if the discrepancy comes from the compressed memory. would you mind showing the output of vm_stat?

[OttoG]

@Maoni0 thank you for your comment. Here is the output of the issue reproduction code and vm_stat (macOS Sonoma 14.2.1, M2, 24 GB RAM):

% dotnet run
Framework version: 8.0.0
MemoryLoadBytes: 25,512,105,738
TotalAvailableMemoryBytes: 25,769,803,776
% vm_stat
Mach Virtual Memory Statistics: (page size of 16384 bytes)
Pages free:                               19099.
Pages active:                            374368.
Pages inactive:                          373569.
Pages speculative:                          174.
Pages throttled:                              0.
Pages wired down:                        179640.
Pages purgeable:                          10066.
"Translation faults":                2766402827.
Pages copy-on-write:                  174174584.
Pages zero filled:                   1201795858.
Pages reactivated:                    153602038.
Pages purged:                         115606586.
File-backed pages:                       227356.
Anonymous pages:                         520755.
Pages stored in compressor:             2912073.
Pages occupied by compressor:            576560.
Decompressions:                       188999306.
Compressions:                         239359971.
Pageins:                              101231079.
Pageouts:                               1199101.
Swapins:                                1934303.
Swapouts:                               3082680.

The Activity Monitor shows these figures:

[Maoni0]

thanks so much @OttoG! yeah so it certainly looks like "Pages occupied by compressor" is part of the physical memory. @mrahl, is it possible that you could also run vm_stat and see if the compressed memory is the culprit (ie, it basically just takes up whatever that would otherwise be free memory. if so obviously we should not include it)?

[OttoG]

I found this discussion that has some information on the topic:

https://apple.stackexchange.com/questions/423717/

It perhaps raises more questions than it answers, but it seems to suggest that the most useful value might actually be total installed memory multiplied by the memory pressure as reported by the memory_pressure command.

It also gives a formula for “Memory Used” from Activity Monitor that unfortunately underestimates the number by about a gigabyte in my case.

Here is the output of a simplified version of the script from the answer (only using values from vm_stat). At the same time on the same machine, the memory_pressure command reports 63% free (i.e., a memory pressure of 37%). A screen dump from Activity Monitor below. (From a few seconds later. Some numbers fluctuate quite quickly by several hundred megabytes, but the Memory Used number only changes by very small amounts from one second to the next.)

% dotnet run
Framework version: 8.0.0
MemoryLoadBytes: 25,512,105,738
TotalAvailableMemoryBytes: 25,769,803,776
% vm_stat | awk -F '[:.]' '{ s[$1] = $2; } END {
gsub(/[^0-9]/, "", s["Mach Virtual Memory Statistics"]);
gb = s["Mach Virtual Memory Statistics"] / (1024 * 1024 * 1024);
print "Numbers from vm_stat in GB";
app_mem = s["Anonymous pages"] - s["Pages purgeable"];
print "App Memory", app_mem * gb;
print "Wired Memory", s["Pages wired down"] * gb;
print "Compressed", s["Pages occupied by compressor"] * gb;
print "Memory Used", (app_mem + s["Pages wired down"] + s["Pages occupied by compressor"]) * gb;
print "Cached Files", (s["File-backed pages"] + s["Pages purgeable"]) * gb;
print "Total", (app_mem + s["Pages wired down"] + s["Pages occupied by compressor"] + s["File-backed pages"] + s["Pages purgeable"] + s["Pages free"]) * gb;
}'
Numbers from vm_stat in GB
App Memory 9.03206
Wired Memory 2.8353
Compressed 5.54578
Memory Used 17.4131
Cached Files 5.59967
Total 23.2451

[Maoni0]

the memory_pressure command reports 63% free (i.e., a memory pressure of 37%)

hmm, it actually reports 63% free? in your case based on the calculation from vm_stat, it's more like 63% in use (17.4 / 24 = 72.5%). I dunno how memory_pressure does its calculation. perhaps it breaks down app memory by active and inactive and only use the active part?

[mrahl]

@Maoni0, @OttoG is also the original source for the report to me about our library. I don't actually have a Mac so I cannot repro myself. @OttoG thanks for providing more info here.

[OttoG]

Hi @Maoni0 it seems to be quite opaque how the memory pressure percentage is calculated. In the discussion I quoted earlier, it was simply stated that (emphasis and comments in brackets are mine):

Memory Pressure is just a number which provides a simple way of indicating memory load. It can be calculated from vm_stat results [it seems very difficult to find out how to do that, if at all possible] or just taken from memory_pressure [easy enough]. We should not worry too much about its physical interpretation - it just an indicative number.

At https://support.apple.com/en-gb/guide/activity-monitor/actmntr1004/mac it is simply stated very loosely by Apple that: Memory pressure is determined by the amount of free memory, swap rate, wired memory and file cached memory.

Still, there seems to be wide agreement in various discussions online that it is a relevant number and that it is the best indicator of memory pressure that Apple has come up with.

This is emphasized by the fact that the memory_pressure command is the means that Apple provides to developers to simulate high memory pressure. See the man page quoted here: https://stackoverflow.com/a/54541689/2416627 The same percentage values that are reported in the Activity Monitor and by the command itself can be used to specify the memory load to simulate, which indicates that the number is relevant to the task.

There are two easy ways of retrieving this number (as a memory free percentage, i.e., 100% minus the actual memory pressure percentage) from the command line:

% memory_pressure | grep System-wide
# Sample output:
# System-wide memory free percentage: 68%
% sysctl kern.memorystatus_level
# Sample output:
# kern.memorystatus_level: 68

These results consistently match what I see in the memory pressure chart of the Activity Monitor as I launch or quit apps.

I also wrote a simple C program (memory-pressure-test.c) that fetches the memory free percentage using system calls. The results consistently match the numbers from the shell commands above.

#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/sysctl.h>

/*
# Test of the kern.memorystatus_level system call -- the output is expected to match the output of the following two shell commands:
memory_pressure | grep System-wide
sysctl kern.memorystatus_level
# Build with the following command (Xcode command-line tools must be installed):
clang memory-pressure-test.c -o memory-pressure-test
# Run with:
./memory-pressure-test
*/

int main() {
    const char* mem_free_name = "kern.memorystatus_level";
    int i;
    uint32_t mem_free = 0;
    size_t mem_free_length = sizeof(uint32_t);
    int* mib; 
    size_t mib_length = 0;
    int64_t mem_size, mem_load_bytes;
    size_t mem_size_length = sizeof(int64_t);
    /*
    Simple method -- use sysctlbyname() and supply the name in cleartext
    */
    sysctlbyname(mem_free_name, &mem_free, &mem_free_length, NULL, 0);
    printf("Memory free (%s) fetched using sysctlbyname(): %u%%\n", mem_free_name, mem_free);
    /*
    More efficient method if the value will be checked repeatedly -- three times faster according to the SYSCTL(3) man page:
    -- Use sysctlnametomib() to check the length of the MIB (Management Information Base) array needed to specify the call
    -- Allocate memory for the MIB array
    -- Use sysctlnametomib() to look up and fetch the MIB array
    -- Use sysctl() to fetch the memory free percentage
    */
    mem_free = 0;
    sysctlnametomib(mem_free_name, NULL, &mib_length);
    mib = calloc(mib_length, sizeof(int));
    sysctlnametomib(mem_free_name, mib, &mib_length);
    sysctl(mib, mib_length, &mem_free, &mem_free_length, NULL, 0);
    printf("Memory free (%s, MIB array ", mem_free_name);
    for (i = 0; i < mib_length; i++) {
        printf("%s%d", i ? ", " : "", mib[i]);
    }
    printf(") fetched using sysctlnametomib() and sysctl(): %u%%\n", mem_free);
    /*
    Output formats that are more relevant to the definition of GCMemoryInfo.MemoryLoadBytes in .NET 8
    */
    printf("Memory pressure: %u%%\n", 100 - mem_free);
    sysctlbyname("hw.memsize", &mem_size, &mem_size_length, NULL, 0);
    mem_load_bytes = mem_size * (100 - mem_free) / 100;
    printf("Memory load bytes: %lld (%.2lf GB)\n", mem_load_bytes, (double)mem_load_bytes / (1024 * 1024 * 1024));
    return 0;
}

Sample output of the program (again, on a machine with 24 GB RAM):

Memory free (kern.memorystatus_level) fetched using sysctlbyname(): 68%
Memory free (kern.memorystatus_level, MIB array 1, 238) fetched using sysctlnametomib() and sysctl(): 68%
Memory pressure: 32%
Memory load bytes: 8246337208 (7.68 GB)

In my view, after now having done some research, the most useful definition of GCMemoryInfo.MemoryLoadBytes on macOS would be 100 minus the result of kern.memorystatus_level, multiplied by the total amount of memory (GCMemoryInfo.TotalAvailableMemoryBytes) and divided by 100.

[Maoni0]

yep, that sounds reasonable and thanks so much for doing the research, @OttoG!

we should use kern.memorystatus_level to calculate our loaded byte. CC @janvorli.