SslSessionsCache growth on Linux with MTLS using many client certificates

[RobinsonWM]

Description

If an application acting as a client creates many mutually-authenticated TLS connections to a server, with each connection using a different client certificate, the application's resident memory grows linearly.

If I run the same code on Windows and Linux, I only see the memory growth on Linux.

This cache appears to grow without bound:

runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslSessionsCache.cs

Line 16 in dbc7d8c

private static readonly ConcurrentDictionary<SslCredKey, SafeCredentialReference> s_cachedCreds =

There are several IDisposables involved in generating a certificate and making an HTTP request, and I believe I am disposing all of them.

Reproduction Steps

https://gist.github.com/RobinsonWM/cae7244762173127d85f9b842cc3c1ba

The code in this gist reproduces the issue reliably for me. I was running it by starting a container, docker run mcr.microsoft.com/dotnet/sdk:6.0, copying in the code, and then running dotnet run.

The code starts an HTTPS server and then makes requests against it very quickly, each from a new HttpClient with a unique certificate. The test app periodically logs out the result of GC.GetTotalMemory(true) and SslSessionsCache.s_cachedCreds.Count, showing that both increase linearly on Linux but not on Windows.

This test application does muddy up the water a bit since it's running both the client and the server in the same process. I did that to make it easy to reproduce this issue. When I originally encountered this issue, it was in an application that was running only the client; the server was a separate process not written in .NET.

Expected behavior

Over time, if load is steady, the application's resident memory usage is also steady.

Actual behavior

The application works fine for a long time, but its resident memory usage grows linearly.

Regression?

I don't know if this is a regression

Known Workarounds

The same workaround for #65563 works for this issue, except for this issue it would need to be run on a timer.

private static void ClearCache()
{
    var sslAssembly = Assembly.GetAssembly(typeof(SslStream));
    var sslSessionCacheClass = sslAssembly.GetType("System.Net.Security.SslSessionsCache");
    var cachedCredsInfo = sslSessionCacheClass.GetField("s_cachedCreds", BindingFlags.NonPublic | BindingFlags.Static);
    var cachedCreds = cachedCredsInfo.GetValue(null);
    cachedCreds.GetType().GetMethod("Clear", BindingFlags.Public | BindingFlags.Instance).Invoke(cachedCreds, null);
}

Configuration

.NET 6.0 on Linux x64, with the Docker image mcr.microsoft.com/dotnet/runtime-deps:6.0
I have observed this issue on Linux x64. The same code run on Windows x64 works and does not exhibit the problem. I believe this issue is specific to Linux.

Other information

I realize this use case may sound contrived. I ran into this issue while developing a load-generation application used as part of a performance test. In order to simulate multiple users when generating load, I needed my HTTPS requests to use certificates generated on-the-fly, one for each user.

[ghost]

Tagging subscribers to this area: @dotnet/ncl, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

If an application acting as a client creates many mutually-authenticated TLS connections to a server, with each connection using a different client certificate, the application's resident memory grows linearly.

If I run the same code on Windows and Linux, I only see the memory growth on Linux.

This cache appears to grow without bound:

runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslSessionsCache.cs

Line 16 in dbc7d8c

private static readonly ConcurrentDictionary<SslCredKey, SafeCredentialReference> s_cachedCreds =

There are several IDisposables involved in generating a certificate and making an HTTP request, and I believe I am disposing all of them.

Reproduction Steps

https://gist.github.com/RobinsonWM/cae7244762173127d85f9b842cc3c1ba

The code in this gist reproduces the issue reliably for me. I was running it by starting a container, docker run mcr.microsoft.com/dotnet/sdk:6.0, copying in the code, and then running dotnet run.

The code starts an HTTPS server and then makes requests against it very quickly, each from a new HttpClient with a unique certificate. The test app periodically logs out the result of GC.GetTotalMemory(true) and SslSessionsCache.s_cachedCreds.Count, showing that both increase linearly on Linux but not on Windows.

This test application does muddy up the water a bit since it's running both the client and the server in the same process. I did that to make it easy to reproduce this issue. When I originally encountered this issue, it was in an application that was running only the client; the server was a separate process not written in .NET.

Expected behavior

Over time, if load is steady, the application's resident memory usage is also steady.

Actual behavior

The application works fine for a long time, but its resident memory usage grows linearly.

Regression?

I don't know if this is a regression

Known Workarounds

The same workaround for #65563 works for this issue, except for this issue it would need to be run on a timer.

private static void ClearCache()
{
    var sslAssembly = Assembly.GetAssembly(typeof(SslStream));
    var sslSessionCacheClass = sslAssembly.GetType("System.Net.Security.SslSessionsCache");
    var cachedCredsInfo = sslSessionCacheClass.GetField("s_cachedCreds", BindingFlags.NonPublic | BindingFlags.Static);
    var cachedCreds = cachedCredsInfo.GetValue(null);
    cachedCreds.GetType().GetMethod("Clear", BindingFlags.Public | BindingFlags.Instance).Invoke(cachedCreds, null);
}

Configuration

.NET 6.0 on Linux x64, with the Docker image mcr.microsoft.com/dotnet/runtime-deps:6.0
I have observed this issue on Linux x64. The same code run on Windows x64 works and does not exhibit the problem. I believe this issue is specific to Linux.

Other information

I realize this use case may sound contrived. I ran into this issue while developing a load-generation application used as part of a performance test. In order to simulate multiple users when generating load, I needed my HTTPS requests to use certificates generated on-the-fly, one for each user.

Author:	RobinsonWM
Assignees:	-
Labels:	area-System.Net.Security
Milestone:	-

[wfurt]

There should be cap on the SslSessionsCache but I don't think we have any tests to verify that. That part changed a bit in 7.0 with TLS resume and it will change even more with #69527. I think we should look into it for 7.0

[karelz]

Triage: We should confirm if we grow indefinitely.

[wfurt]

Can you please try it with 7.0 preview build @RobinsonWM? I did little bit of digging and it seems like this comes from client side in 6.0. I don't see the growth with 7.0 but it would be nice to confirm it with your setup.

[ghost]

This issue has been marked needs-author-action and may be missing some important information.

[karelz]

Triage: We do not have 7.0 repro - it seems to be solved there. Closing as Fixed in 7.0.