[Bug]: System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer.

[atrauzzi]

Testcontainers version

4.1.0

Using the latest Testcontainers version?

Yes

Host OS

Linux (Fedora, latest)

Host arch

amd64

.NET version

9.0.101

Podman version

podman 5.3.1

What happened?

I updated from dotnet 8 to 9 in a project I've been using TestContainers in to manage some containers for running benchmarks. Up until the upgrade it's been fine and stable.

Something however has caused it to start throwing exceptions when I try and use multiple containers within a single invocation.

Even more strange, sometimes this issue doesn't happen and sometimes it does. It's very intermittent, but I'd say it's consistently observable within 1-3 consecutive attempts.

Please note, I don't start and stop a new container as part of the benchmarks, so overall my use of TestContainers is fairly tame.

Relevant log output

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer.
 ---> System.Net.Sockets.SocketException (104): Connection reset by peer
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
   at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
--- End of stack trace from previous location ---
   at Docker.DotNet.MultiplexedStream.ReadOutputAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken) in /_/src/Docker.DotNet/MultiplexedStream.cs:line 81
   at Docker.DotNet.MultiplexedStream.CopyOutputToAsync(Stream stdin, Stream stdout, Stream stderr, CancellationToken cancellationToken) in /_/src/Docker.DotNet/MultiplexedStream.cs:line 188
   at Docker.DotNet.MultiplexedStream.ReadOutputToEndAsync(CancellationToken cancellationToken) in /_/src/Docker.DotNet/MultiplexedStream.cs:line 133
   at DotNet.Testcontainers.Clients.DockerContainerOperations.ExecAsync(String id, IList`1 command, CancellationToken ct) in /_/src/Testcontainers/Clients/DockerContainerOperations.cs:line 156
   at DotNet.Testcontainers.Configurations.UntilUnixCommandIsCompleted.UntilAsync(IContainer container) in /_/src/Testcontainers/Configurations/WaitStrategies/UntilUnixCommandIsCompleted.cs:line 22
   at DotNet.Testcontainers.Containers.DockerContainer.CheckReadinessAsync(WaitStrategy waitStrategy, CancellationToken ct) in /_/src/Testcontainers/Containers/DockerContainer.cs:line 542
   at DotNet.Testcontainers.Configurations.WaitStrategy.<>c__DisplayClass24_0.<<WaitUntilAsync>g__UntilAsync|0>d.MoveNext() in /_/src/Testcontainers/Configurations/WaitStrategies/WaitStrategy.cs:line 184
--- End of stack trace from previous location ---
   at DotNet.Testcontainers.Configurations.WaitStrategy.WaitUntilAsync(Func`1 wait, TimeSpan interval, TimeSpan timeout, Int32 retries, CancellationToken ct) in /_/src/Testcontainers/Configurations/WaitStrategies/WaitStrategy.cs:line 213
   at DotNet.Testcontainers.Containers.DockerContainer.CheckReadinessAsync(IEnumerable`1 waitStrategies, CancellationToken ct) in /_/src/Testcontainers/Containers/DockerContainer.cs:line 560
   at DotNet.Testcontainers.Containers.DockerContainer.UnsafeStartAsync(CancellationToken ct) in /_/src/Testcontainers/Containers/DockerContainer.cs:line 480
   at DotNet.Testcontainers.Containers.DockerContainer.StartAsync(CancellationToken ct) in /_/src/Testcontainers/Containers/DockerContainer.cs:line 283
   at Atrauzzi.Presentation.OrmsAreFine.Scenario.SimpleJoin.BenchmarkSimpleJoin.<Setup>b__4_0() in /home/atrauzzi/Development/atrauzzi/orms-are-fine/Scenario/SimpleJoin/BenchmarkSimpleJoin.cs:line 27
   at Atrauzzi.Presentation.OrmsAreFine.Scenario.SimpleJoin.BenchmarkSimpleJoin.Setup() in /home/atrauzzi/Development/atrauzzi/orms-are-fine/Scenario/SimpleJoin/BenchmarkSimpleJoin.cs:line 25
   at BenchmarkDotNet.Engines.EngineFactory.CreateReadyToRun(EngineParameters engineParameters)
   at BenchmarkDotNet.Autogenerated.Runnable_1.Run(IHost host, String benchmarkName) in /home/atrauzzi/Development/atrauzzi/orms-are-fine/Cli/bin/Debug/net9.0/c929bf0f-a20b-4f69-8b2b-1fdf82d93e41/c929bf0f-a20b-4f69-8b2b-1fdf82d93e41.notcs:line 368
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)
   --- End of inner exception stack trace ---
   at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)
   at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(String[] args) in /home/atrauzzi/Development/atrauzzi/orms-are-fine/Cli/bin/Debug/net9.0/c929bf0f-a20b-4f69-8b2b-1fdf82d93e41/c929bf0f-a20b-4f69-8b2b-1fdf82d93e41.notcs:line 57

Additional information

I'm running with podman in rootless mode. My env vars are set to use /var/run/user/1000/podman/podman.sock.

public static class MsSqlServer
{
    public static MsSqlContainer CreateMsSqlServer(string? name = null) => new MsSqlBuilder()
        // see: https://github.com/microsoft/mssql-docker/issues/868
        .WithImage("mcr.microsoft.com/mssql/server:2022-CU12-ubuntu-22.04")
        .WithName($"orms-are-fine-mssql{name switch
        {
            null => "",
            _ => $"-{name}",
        }}")
        .WithEnvironment("ACCEPT_EULA", "Y")
        .WithEnvironment("MSSQL_SA_PASSWORD", "Zxczxc123!")
        .WithEnvironment("MSSQL_PID", "Developer")
        .WithPassword("Zxczxc123!")
        .WithExposedPort(1433)
        .WithWaitStrategy(Wait.ForUnixContainer().UntilPortIsAvailable(1433))
        // note: Neither of these make a difference.
        // .WithReuse(true)
        // .WithCleanUp(true)
        .Build();
}

Again, probably worth repeating that this problem only surfaced after updating to dotnet 9.

[HofmeisterAn]

Thank you for creating this issue. It seems that other Podman users are experiencing the same problem. Please refer to the initial discussion: #1325. I have created an issue in the upstream repository, Docker.DotNet, let's track the progress here: testcontainers/Docker.DotNet#11.

Again, probably worth repeating that this problem only surfaced after updating to dotnet 9.

That's an interesting observation. Until now, I thought the issue occurred only when upgrading the NuGet version.

[atrauzzi]

Might be that too, I encountered this whole doing a whole project catch up.