Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NetworkChange.NetworkAddressChanged event leaks threads on Linux #63788

Closed
vdjuric opened this issue Jan 14, 2022 · 18 comments · Fixed by #63963
Closed

NetworkChange.NetworkAddressChanged event leaks threads on Linux #63788

vdjuric opened this issue Jan 14, 2022 · 18 comments · Fixed by #63963
Labels
area-System.Net bug os-linux Linux OS (any supported distro)
Milestone

Comments

@vdjuric
Copy link
Contributor

vdjuric commented Jan 14, 2022

Description

When you enable HTTP/3 support on Linux and repeatedly get HTTP/2 or HTTP3 web page, ".NET Network Address Change" threads will start to accumulate.

It seems network address change event is triggered from https://source.dot.net/#System.Net.Http/System/Net/Http/SocketsHttpHandler/HttpConnectionPoolManager.cs with call _poolManager.StartMonitoringNetworkChanges();

NetworkAddressChange.Unix.cs will start creating threads with name ".NET Network Address Change":
https://source.dot.net/#System.Net.NetworkInformation/System/Net/NetworkInformation/NetworkAddressChange.Unix.cs

Reproduction Steps


.NET 6 console project, source code example:

using System.Diagnostics;

static async Task ThreadDiagnostics()
{
    var pid = Process.GetCurrentProcess().Id;
    var psi = new ProcessStartInfo("ps", $"-T -p {pid}");
    psi.RedirectStandardOutput = true;

    using var process = Process.Start(psi);
    if (process == null)
    {
        throw new Exception("Could not create process");
    }

    int threadCountNetworkAddressChange = 0;
    var output = await process.StandardOutput.ReadToEndAsync();
    using (var sr = new StringReader(output))
    {
        while (true)
        {
            var line = await sr.ReadLineAsync();
            if (line == null)
            {
                break;
            }

            if (line.IndexOf(".NET Network Ad") > 0)
            {
                threadCountNetworkAddressChange++;
            }

            //NOTE:
            //we are searching for threads containing ".NET Network Ad"

            //new Thread(s => LoopReadSocket((int)s!))
            //{
            //    IsBackground = true,
            //    Name = ".NET Network Address Change"
            //}.UnsafeStart(newSocket);
            //
            //https://source.dot.net/#System.Net.NetworkInformation/System/Net/NetworkInformation/NetworkAddressChange.Unix.cs
        }
    }

    //Console.WriteLine(output);
    Console.WriteLine($"{DateTime.Now} '.NET Network Address Change' threads: {threadCountNetworkAddressChange}");
}

for (var i = 0; i < 60; i++)
{
    try
    {
        using (var client = new HttpClient())
        {
            client.DefaultVersionPolicy = HttpVersionPolicy.RequestVersionExact;
            using var req = new HttpRequestMessage(HttpMethod.Get, "https://cloudflare-quic.com/"); //HTTP/2 or HTTP/3 enabled web server
            req.Version = System.Net.HttpVersion.Version20;

            using var res = await client.SendAsync(req);
            res.EnsureSuccessStatusCode();
        }

        await ThreadDiagnostics();
    }
    catch (Exception ex)
    {
        Console.WriteLine($"Unhandled exception: {ex}");
    }

    await Task.Delay(1000);
}

Dockerfile with installed libmsquic and required tools (procps, curl)

FROM mcr.microsoft.com/dotnet/runtime:6.0-bullseye-slim AS base
WORKDIR /app

#install some tools
RUN apt-get update && apt-get install -y procps curl

#install libmsquic
#https://docs.microsoft.com/en-us/dotnet/core/extensions/httpclient-http3
RUN curl https://packages.microsoft.com/debian/11/prod/pool/main/libm/libmsquic/libmsquic_1.9.0_amd64.deb --output libmsquic.deb
RUN dpkg -i libmsquic.deb

FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build
WORKDIR /src
COPY ["HttpClientMemoryLeak/HttpClientMemoryLeak.csproj", "HttpClientMemoryLeak/"]
RUN dotnet restore "HttpClientMemoryLeak/HttpClientMemoryLeak.csproj"
COPY . .
WORKDIR "/src/HttpClientMemoryLeak"
RUN dotnet build "HttpClientMemoryLeak.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "HttpClientMemoryLeak.csproj" -c Release -o /app/publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "HttpClientMemoryLeak.dll"]

Expected behavior

".NET Network Address Change" should not accumulate, especially if HTTP connections are not pooled.

Check https://source.dot.net/#System.Net.Http/System/Net/Http/SocketsHttpHandler/HttpConnectionPoolManager.cs

bool avoidStoringConnections =
                settings._maxConnectionsPerServer == int.MaxValue &&
                (settings._pooledConnectionIdleTimeout == TimeSpan.Zero ||
                 settings._pooledConnectionLifetime == TimeSpan.Zero);

Actual behavior

".NET Network Address Change" threads will start to accumulate and consume resources / memory.

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

No response

@dotnet-issue-labeler dotnet-issue-labeler bot added area-System.Net.Http untriaged New issue has not been triaged by the area owner labels Jan 14, 2022
@ghost
Copy link

ghost commented Jan 14, 2022

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

Description

When you enable HTTP/3 support on Linux and repeatedly get HTTP/2 or HTTP3 web page, ".NET Network Address Change" threads will start to accumulate.

It seems network address change event is triggered from https://source.dot.net/#System.Net.Http/System/Net/Http/SocketsHttpHandler/HttpConnectionPoolManager.cs with call _poolManager.StartMonitoringNetworkChanges();

NetworkAddressChange.Unix.cs will start creating threads with name ".NET Network Address Change":
https://source.dot.net/#System.Net.NetworkInformation/System/Net/NetworkInformation/NetworkAddressChange.Unix.cs

Reproduction Steps


.NET 6 console project, source code example:

`using System.Diagnostics;

static async Task ThreadDiagnostics()
{
var pid = Process.GetCurrentProcess().Id;
var psi = new ProcessStartInfo("ps", $"-T -p {pid}");
psi.RedirectStandardOutput = true;

using var process = Process.Start(psi);
if (process == null)
{
    throw new Exception("Could not create process");
}

int threadCountNetworkAddressChange = 0;
var output = await process.StandardOutput.ReadToEndAsync();
using (var sr = new StringReader(output))
{
    while (true)
    {
        var line = await sr.ReadLineAsync();
        if (line == null)
        {
            break;
        }

        if (line.IndexOf(".NET Network Ad") > 0)
        {
            threadCountNetworkAddressChange++;
        }

        //NOTE:
        //we are searching for threads containing ".NET Network Ad"

        //new Thread(s => LoopReadSocket((int)s!))
        //{
        //    IsBackground = true,
        //    Name = ".NET Network Address Change"
        //}.UnsafeStart(newSocket);
        //
        //https://source.dot.net/#System.Net.NetworkInformation/System/Net/NetworkInformation/NetworkAddressChange.Unix.cs
    }
}

//Console.WriteLine(output);
Console.WriteLine($"{DateTime.Now} '.NET Network Address Change' threads: {threadCountNetworkAddressChange}");

}

for (var i = 0; i < 60; i++)
{
try
{
using (var client = new HttpClient())
{
client.DefaultVersionPolicy = HttpVersionPolicy.RequestVersionExact;
using var req = new HttpRequestMessage(HttpMethod.Get, "https://cloudflare-quic.com/"); //HTTP/2 or HTTP/3 enabled web server
req.Version = System.Net.HttpVersion.Version20;

        using var res = await client.SendAsync(req);
        res.EnsureSuccessStatusCode();
    }

    await ThreadDiagnostics();
}
catch (Exception ex)
{
    Console.WriteLine($"Unhandled exception: {ex}");
}

await Task.Delay(1000);

}
`


Dockerfile with installed libmsquic and required tools (procps, curl)

#See https://aka.ms/containerfastmode to understand how Visual Studio uses this Dockerfile to build your images for faster debugging.

FROM mcr.microsoft.com/dotnet/runtime:6.0-bullseye-slim AS base
WORKDIR /app

#install some tools
RUN apt-get update && apt-get install -y procps curl

#install libmsquic
#https://docs.microsoft.com/en-us/dotnet/core/extensions/httpclient-http3
RUN curl https://packages.microsoft.com/debian/11/prod/pool/main/libm/libmsquic/libmsquic_1.9.0_amd64.deb --output libmsquic.deb
RUN dpkg -i libmsquic.deb

FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build
WORKDIR /src
COPY ["HttpClientMemoryLeak/HttpClientMemoryLeak.csproj", "HttpClientMemoryLeak/"]
RUN dotnet restore "HttpClientMemoryLeak/HttpClientMemoryLeak.csproj"
COPY . .
WORKDIR "/src/HttpClientMemoryLeak"
RUN dotnet build "HttpClientMemoryLeak.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "HttpClientMemoryLeak.csproj" -c Release -o /app/publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "HttpClientMemoryLeak.dll"]

Expected behavior

".NET Network Address Change" should not accumulate, especially if HTTP connections are not pooled.

Check https://source.dot.net/#System.Net.Http/System/Net/Http/SocketsHttpHandler/HttpConnectionPoolManager.cs

bool avoidStoringConnections =
                settings._maxConnectionsPerServer == int.MaxValue &&
                (settings._pooledConnectionIdleTimeout == TimeSpan.Zero ||
                 settings._pooledConnectionLifetime == TimeSpan.Zero);

Actual behavior

".NET Network Address Change" threads will start to accumulate and consume resources / memory.

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

No response

Author: vdjuric
Assignees: -
Labels:

area-System.Net.Http, untriaged

Milestone: -

@ManickaP ManickaP self-assigned this Jan 14, 2022
@vdjuric
Copy link
Contributor Author

vdjuric commented Jan 14, 2022

Just small update: HTTP/3 must be enabled in .csproj by adding following entry

<ItemGroup>
	<RuntimeHostConfigurationOption Include="System.Net.SocketsHttpHandler.Http3Support" Value="true" />
</ItemGroup>

@ManickaP
Copy link
Member

I just re-ran the repro. It reproduces, but it doesn't seem to be specific to HTTP/3 but rather a leak in NetworkChange.NetworkAddressChanged event.

This also reproduces with just this, instead of HttpClient usage:

NetworkAddressChangedEventHandler networkChangedDelegate;
networkChangedDelegate = delegate
{
    Console.WriteLine("called");
};
NetworkChange.NetworkAddressChanged += networkChangedDelegate;
NetworkChange.NetworkAddressChanged -= networkChangedDelegate;

Seems like bug in NetworkChange.

@ManickaP ManickaP changed the title Linux memory leak from HttpConnectionPoolManager with enabled HTTP/3 NetworkChange.NetworkAddressChanged event leaks threads on Linux Jan 14, 2022
@ghost
Copy link

ghost commented Jan 14, 2022

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

Description

When you enable HTTP/3 support on Linux and repeatedly get HTTP/2 or HTTP3 web page, ".NET Network Address Change" threads will start to accumulate.

It seems network address change event is triggered from https://source.dot.net/#System.Net.Http/System/Net/Http/SocketsHttpHandler/HttpConnectionPoolManager.cs with call _poolManager.StartMonitoringNetworkChanges();

NetworkAddressChange.Unix.cs will start creating threads with name ".NET Network Address Change":
https://source.dot.net/#System.Net.NetworkInformation/System/Net/NetworkInformation/NetworkAddressChange.Unix.cs

Reproduction Steps


.NET 6 console project, source code example:

using System.Diagnostics;

static async Task ThreadDiagnostics()
{
    var pid = Process.GetCurrentProcess().Id;
    var psi = new ProcessStartInfo("ps", $"-T -p {pid}");
    psi.RedirectStandardOutput = true;

    using var process = Process.Start(psi);
    if (process == null)
    {
        throw new Exception("Could not create process");
    }

    int threadCountNetworkAddressChange = 0;
    var output = await process.StandardOutput.ReadToEndAsync();
    using (var sr = new StringReader(output))
    {
        while (true)
        {
            var line = await sr.ReadLineAsync();
            if (line == null)
            {
                break;
            }

            if (line.IndexOf(".NET Network Ad") > 0)
            {
                threadCountNetworkAddressChange++;
            }

            //NOTE:
            //we are searching for threads containing ".NET Network Ad"

            //new Thread(s => LoopReadSocket((int)s!))
            //{
            //    IsBackground = true,
            //    Name = ".NET Network Address Change"
            //}.UnsafeStart(newSocket);
            //
            //https://source.dot.net/#System.Net.NetworkInformation/System/Net/NetworkInformation/NetworkAddressChange.Unix.cs
        }
    }

    //Console.WriteLine(output);
    Console.WriteLine($"{DateTime.Now} '.NET Network Address Change' threads: {threadCountNetworkAddressChange}");
}

for (var i = 0; i < 60; i++)
{
    try
    {
        using (var client = new HttpClient())
        {
            client.DefaultVersionPolicy = HttpVersionPolicy.RequestVersionExact;
            using var req = new HttpRequestMessage(HttpMethod.Get, "https://cloudflare-quic.com/"); //HTTP/2 or HTTP/3 enabled web server
            req.Version = System.Net.HttpVersion.Version20;

            using var res = await client.SendAsync(req);
            res.EnsureSuccessStatusCode();
        }

        await ThreadDiagnostics();
    }
    catch (Exception ex)
    {
        Console.WriteLine($"Unhandled exception: {ex}");
    }

    await Task.Delay(1000);
}

Dockerfile with installed libmsquic and required tools (procps, curl)

FROM mcr.microsoft.com/dotnet/runtime:6.0-bullseye-slim AS base
WORKDIR /app

#install some tools
RUN apt-get update && apt-get install -y procps curl

#install libmsquic
#https://docs.microsoft.com/en-us/dotnet/core/extensions/httpclient-http3
RUN curl https://packages.microsoft.com/debian/11/prod/pool/main/libm/libmsquic/libmsquic_1.9.0_amd64.deb --output libmsquic.deb
RUN dpkg -i libmsquic.deb

FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build
WORKDIR /src
COPY ["HttpClientMemoryLeak/HttpClientMemoryLeak.csproj", "HttpClientMemoryLeak/"]
RUN dotnet restore "HttpClientMemoryLeak/HttpClientMemoryLeak.csproj"
COPY . .
WORKDIR "/src/HttpClientMemoryLeak"
RUN dotnet build "HttpClientMemoryLeak.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "HttpClientMemoryLeak.csproj" -c Release -o /app/publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "HttpClientMemoryLeak.dll"]

Expected behavior

".NET Network Address Change" should not accumulate, especially if HTTP connections are not pooled.

Check https://source.dot.net/#System.Net.Http/System/Net/Http/SocketsHttpHandler/HttpConnectionPoolManager.cs

bool avoidStoringConnections =
                settings._maxConnectionsPerServer == int.MaxValue &&
                (settings._pooledConnectionIdleTimeout == TimeSpan.Zero ||
                 settings._pooledConnectionLifetime == TimeSpan.Zero);

Actual behavior

".NET Network Address Change" threads will start to accumulate and consume resources / memory.

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

No response

Author: vdjuric
Assignees: ManickaP
Labels:

area-System.Net, untriaged

Milestone: -

@ManickaP ManickaP removed their assignment Jan 14, 2022
@stephentoub
Copy link
Member

This also reproduces with just this, instead of HttpClient usage:

Are you saying if you put that in a loop it creates lots of threads that never go away?

@vdjuric
Copy link
Contributor Author

vdjuric commented Jan 14, 2022

With my test: yes

@stephentoub
Copy link
Member

With my test: yes

I was particularly wondering about @ManickaP's.

@vdjuric
Copy link
Contributor Author

vdjuric commented Jan 14, 2022

Following Harmony patch seems to workaround this issue :S

var harmony = new HarmonyLib.Harmony("MemoryLeakFix");

var t = typeof(System.Net.NetworkInformation.NetworkChange);

{
	var method = t.GetMethod("CreateSocket", System.Reflection.BindingFlags.Static | System.Reflection.BindingFlags.NonPublic);
	var methodNew = typeof(MyClass).GetMethod("Transpiler", System.Reflection.BindingFlags.Static | System.Reflection.BindingFlags.NonPublic);
	harmony.Patch(method, transpiler: new HarmonyLib.HarmonyMethod(methodNew));

}

{
	var method = t.GetMethod("CloseSocket", System.Reflection.BindingFlags.Static | System.Reflection.BindingFlags.NonPublic);
	var methodNew = typeof(MyClass).GetMethod("Transpiler", System.Reflection.BindingFlags.Static | System.Reflection.BindingFlags.NonPublic);
	harmony.Patch(method, transpiler: new HarmonyLib.HarmonyMethod(methodNew));
}

private static IEnumerable<HarmonyLib.CodeInstruction> Transpiler(IEnumerable<HarmonyLib.CodeInstruction> instructions)
{
	return new[] { new HarmonyLib.CodeInstruction(System.Reflection.Emit.OpCodes.Ret) };
}

@stephentoub
Copy link
Member

stephentoub commented Jan 14, 2022

It does look like there's a race condition between the worker thread and CreateSocket. Should be easy to fix.

@vdjuric
Copy link
Contributor Author

vdjuric commented Jan 14, 2022

If I try ManickaP example, I get "'.NET Network Address Change' threads: 1"

If I try HttpClient example:

for (var i = 0; i < 10; i++)
{
    try
    {
        using (var client = new HttpClient())
        {
            client.DefaultVersionPolicy = HttpVersionPolicy.RequestVersionExact;
            using var req = new HttpRequestMessage(HttpMethod.Get, "https://cloudflare-quic.com/"); //HTTP/2 or HTTP/3 enabled web server
            req.Version = System.Net.HttpVersion.Version20;

            using var res = await client.SendAsync(req);
            res.EnsureSuccessStatusCode();
        }

        await ThreadDiagnostics();
    }
    catch (Exception ex)
    {
        Console.WriteLine($"Unhandled exception: {ex}");
    }
    await Task.Delay(1000);
}

When loop is finished I get '.NET Network Address Change' threads: 10 (also measured few minutes after loop ended)

@vdjuric
Copy link
Contributor Author

vdjuric commented Jan 14, 2022

Sorry, it is returning now '.NET Network Address Change' threads: 9

for (var i = 0; i < 10; i++)
{
    NetworkAddressChangedEventHandler networkChangedDelegate;
    networkChangedDelegate = delegate
    {
        Console.WriteLine("called");
    };
    NetworkChange.NetworkAddressChanged += networkChangedDelegate;
    NetworkChange.NetworkAddressChanged -= networkChangedDelegate;
}

while (true)
{
    await ThreadDiagnostics();
    await Task.Delay(1000);
}

@ManickaP
Copy link
Member

With my example I got growing number of threads, even though slowly than with HttpClient. I did run it from command line though, natively on Linux. When I run it with debugger attached, it reports 1 thread (which still should be 0 after removing the event listener).

This seems like a race. I also dug a bit around the code and the thread should finish on its own once the s_socket is nulled:

private static unsafe void LoopReadSocket(int socket)
{
while (socket == s_socket)
{
Interop.Sys.ReadEvents(socket, &ProcessEvent);
}
}

The only think I can think of is that the subsequent call of:

public static unsafe partial Error CreateNetworkChangeListenerSocket(int* socket);

returns the same fd for the new socket, thus s_socket gets set to the same number even though it's a different run and different registration...

Based on Linux docs: https://man7.org/linux/man-pages/man2/socket.2.html I assume that getting the same fd is highly likely:

The file descriptor returned by a successful call will be the lowest-numbered file descriptor not currently open for the process.

@stephentoub
Copy link
Member

That was my guess as well.

@MihaZupan
Copy link
Member

Yep, it's always getting the same socket id

@karelz
Copy link
Member

karelz commented Jan 18, 2022

Triage: Used in HTTP/3, we should fix it in 7.0.

@karelz karelz added this to the 7.0.0 milestone Jan 18, 2022
@karelz karelz added bug os-linux Linux OS (any supported distro) and removed untriaged New issue has not been triaged by the area owner labels Jan 18, 2022
@vdjuric
Copy link
Contributor Author

vdjuric commented Jan 18, 2022

Possible solution: adding "s_socketCreateContext" counter which changes each time CreateSocket() method is called. Method LoopReadSocket then accepts LoopReadSocketArgs args instead of int socket:

private static volatile int s_socketCreateContext;

private class LoopReadSocketArgs
{
	public LoopReadSocketArgs(int socketCreateContext, int socket)
	{
		SocketCreateContext = socketCreateContext;
		Socket = socket;
	}

	public int SocketCreateContext { get; }
	public int Socket { get; }
}

private static unsafe void CreateSocket()
{
	Debug.Assert(s_socket == 0, "s_socket != 0, must close existing socket before opening another.");
	int newSocket;
	Interop.Error result = Interop.Sys.CreateNetworkChangeListenerSocket(&newSocket);
	if (result != Interop.Error.SUCCESS)
	{
		string message = Interop.Sys.GetLastErrorInfo().GetErrorMessage();
		throw new NetworkInformationException(message);
	}

	s_socket = newSocket;
	var socketCreateContext = System.Threading.Interlocked.Increment(ref s_socketCreateContext);
	
	new Thread(s => LoopReadSocket((LoopReadSocketArgs)s!))
	{
		IsBackground = true,
		Name = ".NET Network Address Change"
	}.UnsafeStart(new LoopReadSocketArgs(socketCreateContext, newSocket));
}

private static unsafe void LoopReadSocket(LoopReadSocketArgs args)
{
	while (args.Socket == s_socket && args.SocketCreateContext == s_socketCreateContext)
	{
		Interop.Sys.ReadEvents(args.Socket, &ProcessEvent);
	}
}

@ManickaP
Copy link
Member

@vdjuric can you put up a PR for what you suggest?

@vdjuric
Copy link
Contributor Author

vdjuric commented Jan 18, 2022

Not perfect solution, but I think it should solve thread leaking. I will prepare PR.

vdjuric added a commit to vdjuric/runtime-NetworkChange-unix-thread-leak that referenced this issue Jan 18, 2022
@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label Jan 19, 2022
ManickaP added a commit that referenced this issue Jan 27, 2022
…t leaks threads on Linux (#63963)

* Trying to fix issue #63788: "NetworkChange.NetworkAddressChanged event leaks threads on Linux"

* Adding socket receive timeout to native method + functional test

* Update src/native/libs/System.Native/pal_networkchange.c

ManickaP replaced tabs with spaces

Co-authored-by: Marie Píchová <11718369+ManickaP@users.noreply.github.com>

* Using setsockopt after socket was successfully set, checking setsockopt result.

* Adding support for conditional network address changed functional test

* Adding support for conditional network address changed functional test

* Using reference checking method to determine if socket is changed as suggested by user @tmds

* Test method update

* Test method update due to some failing tests

* Removing yield return from test

* Reverting back to yield and adding ToArray() when testing if ps is available

* Socket wrapper is now nullable

* Update src/libraries/System.Net.NetworkInformation/tests/FunctionalTests/NetworkAddressChangedTests.cs

Co-authored-by: vdjuric <vladimir@LEGION21.localdomain>
Co-authored-by: Marie Píchová <11718369+ManickaP@users.noreply.github.com>
@ghost ghost removed the in-pr There is an active PR which will close this issue when it is merged label Jan 27, 2022
@ghost ghost locked as resolved and limited conversation to collaborators Feb 26, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-System.Net bug os-linux Linux OS (any supported distro)
Projects
None yet
5 participants