Change in GC behavior in 8.0.17

[KeterSCP]

Description

Starting from 8.0.17 runtime version, GC does not return unmanaged memory to the OS on forced full GC.

Suspected change is #115023
Related: #108081

Reproduction Steps

1. Make sure you have both 8.0.16 and 8.0.17 runtimes installed, 8.0.410 and 8.0.411 respective versions of SDK

2. Program.csproj:

<Project Sdk="Microsoft.NET.Sdk">
    <PropertyGroup>
        <OutputType>Exe</OutputType>
        <TargetFramework>net8.0</TargetFramework>
        <RollForward>Disable</RollForward>
        <ImplicitUsings>enable</ImplicitUsings>
        <Nullable>enable</Nullable>
        <ServerGarbageCollection>true</ServerGarbageCollection>
    </PropertyGroup>
</Project>

3. Program.cs:

using System.Diagnostics;
using System.Runtime.CompilerServices;

Console.WriteLine(Environment.Version);

for (int i = 0; i < 10; i++)
{
    _ = Allocate();
}

Console.WriteLine($"Private memory before GC: {GetPrivateMemoryMb()} MB");
TriggerGc(); // No matter how many times you call this, process private memory stays the same on 8.0.17
Console.WriteLine($"Private memory after GC: {GetPrivateMemoryMb()} MB");

[MethodImpl(MethodImplOptions.NoInlining)]
int Allocate()
{
    var data = new byte[1_000_000_000];
    return data.Length;
}

void TriggerGc()
{
    GC.Collect(generation: 2, GCCollectionMode.Forced, blocking: true, compacting: true);
    GC.WaitForPendingFinalizers();
    Thread.Sleep(1000);
}

double GetPrivateMemoryMb()
{
    using var process = Process.GetCurrentProcess();
    var mb = process.PrivateMemorySize64 / 1024.0 / 1024.0;
    return Math.Round(mb, 2);
}

4. Run dotnet clean -c Release && dotnet run -c Release -p:RuntimeFrameworkVersion='8.0.16'

5. Run dotnet clean -c Release && dotnet run -c Release -p:RuntimeFrameworkVersion='8.0.17'

Expected behavior

Behavior should not change between patch versions of runtime, and preferably, GC should return memory to the OS on forced compacting GC.

Actual behavior

8.0.16
Private memory before GC: 9614.54 MB
Private memory after GC: 60.44 MB

8.0.17
Private memory before GC: 9614.56 MB
Private memory after GC: 9615.96 MB

Regression?

Yes. Regression from 8.0.16

Known Workarounds

• Use 8.0.16 runtime version

Configuration

OS: Windows 11
Architecture: x64

Other information

No response

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

[mangod9]

Thanks for reporting this issue @KeterSCP, we have heard about it through other channels recently. Hence: https://github.com/dotnet/runtime/tree/undo_distribute_regions.

[KeterSCP]

@mangod9 not sure if it is related, but while experimenting with allocation values inside memory-constrained environment (docker container), I managed to get a set of different outcomes when allocating the same specific amount of bytes in a loop:

Source code

using System.Diagnostics;
using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;

var iterationAllocationSize = int.Parse(args[0].Replace("_", ""));

const int iterations = 20;

Console.WriteLine($"{Environment.Version} {RuntimeInformation.ProcessArchitecture}");

Console.WriteLine($"Expected total allocation size: {iterationAllocationSize * iterations / 1024.0 / 1024.0} MB");
Console.WriteLine($"Private memory before Allocate: {GetPrivateMemoryMb()} MB");

for (int i = 0; i < iterations; i++)
{
  _ = Allocate();
}

Console.WriteLine($"Private memory before GC: {GetPrivateMemoryMb()} MB");
TriggerGc();
Console.WriteLine($"Private memory after GC: {GetPrivateMemoryMb()} MB");

[MethodImpl(MethodImplOptions.NoInlining)]
int Allocate()
{
  var data = new byte[iterationAllocationSize];
  return data.Length;
}

void TriggerGc()
{
  GC.Collect(generation: 2, GCCollectionMode.Forced, blocking: true, compacting: true);
  GC.WaitForPendingFinalizers();
  Thread.Sleep(1000);
}

double GetPrivateMemoryMb()
{
  using var process = Process.GetCurrentProcess();
  var mb = process.PrivateMemorySize64 / 1024.0 / 1024.0;
  return Math.Round(mb, 2);
}

• .NET 8.0.16 outcomes:

1. Allocations succeed, process exits with 0 code.
2. Allocations fail with OOM.
3. Allocations succeed, process hangs indefinitely.

• .NET 8.0.17 hanged 10/10 times.

I managed to get dumps via dotnet-dump both for 8.0.16 and 8.0.17 hanged processes if you would like to analyze it (dumps do not contain sensitive information):

frozen_8_0_16.zip

frozen_8_0_17.zip

[mangod9]

Does the issue only occur with DynamicAdaptationMode enabled?

[KeterSCP]

Yes, without DATAS freeze does not occur. And 8.0.16 fails with OOM somewhere near 33.5 MB/iteration, while 8.0.17 OOMs at ~156 MB/iteration.

I also checked .NET 9.0.6 and it behaves the same as 8.0.17, failing near 156 MB/iteration, but AFAIK .NET 9 has DATAS enabled by default, so the issue with freeze seems to be fixed in .NET 9

[mangod9]

yeah there have been a few reliability fixes for DATAS in 9, which we hadn't back ported to 8 since its not the default. If this is a blocker we can look into it, but since more tuning has gone into 9 as well, prefer to not enable DATAS in 8.

[KeterSCP]

Completely not a blocker - just something that came up as a side thing during experiment while investigating memory consumption of our arm64 kubernetes pods

[Maoni0]

hi, I just got back from vacation today and took a look at this.

what happened was in #105521, the calls to region_free_list::age_free_regions was refactored into gc_heap::age_free_regions which also included a bug fix that's to age the huge free regions on the global huge free list if we are aging all regions -

    if (age_all_region_kinds)
    {
        global_free_huge_regions.age_free_regions();
    }

this was done a long time ago and during the recent 8.0 port of #109431 (#115023, which went into the 8.0 release you got) we missed the few lines of changes in #105521 that changed the calls from region_free_list::age_free_regions to gc_heap::age_free_regions so the huge free regions were not properly aged, and therefore not put on the decommit list to be decommmitted. all we need to do to make the 8.0 port work is to get the few lines of changes in #105521 that did that.

8.0port_fix.patch

[KeterSCP]

Hi @Maoni0! Hope you had a great time vacationing.
Thanks for the analysis. Yeah, it seems very reasonable to keep optimization from #109431 alongside with fix of the decommit problem.