Skip to content
This repository has been archived by the owner on Dec 18, 2018. It is now read-only.

.NET Core 2.1: UvException: Error -107 ENOTCONN socket is not connected #2920

Closed
ZOXEXIVO opened this issue Sep 12, 2018 · 12 comments
Closed

.NET Core 2.1: UvException: Error -107 ENOTCONN socket is not connected #2920

ZOXEXIVO opened this issue Sep 12, 2018 · 12 comments
Milestone
Discussions

Comments

@ZOXEXIVO
Copy link

ZOXEXIVO commented Sep 12, 2018 •
edited
Loading

Last week see see this error in our logs:

Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.Networking.UvException: Error -107 ENOTCONN socket is not connected
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.Networking.LibuvFunctions.ThrowError(Int32 statusCode)
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.Networking.UvTcpHandle.GetPeerIPEndPoint()
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.LibuvConnection..ctor(ListenerContext context, UvStreamHandle socket)
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.ListenerSecondary.ReadStartCallback(UvStreamHandle handle, Int32 status)

It should be fixed with changing loglevel in v2.1 release #2045, #2583, #2658, but we see it now

We use this packages:

<PackageReference Include="Microsoft.AspNetCore" Version="2.1.2" />
<PackageReference Include="Microsoft.AspNetCore.Mvc" Version="2.1.1" />
<PackageReference Include="Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv" Version="2.1.2" />

and build app with docker image
microsoft/dotnet:2.1-sdk
and run it with
microsoft/dotnet:2.1-aspnetcore-runtime

Last build date from container - 23 August 2018
@muratg
Copy link
Contributor

muratg commented Sep 13, 2018

Did you recently upgrade to 2.1 by any chance? I wonder if you may have an older container running still. Because like you said, it should have been fixed already.

Can you run dotnet info inside your container and paste the output? (You can docker exec -it --rm YOUR_CONTAINER_NAME bash to get a shell to it. If you have it running under Kubernetes, you can kubectl exec --it YOUR_CONTAINER NAME instead.)

@ZOXEXIVO
Copy link
Author

ZOXEXIVO commented Sep 13, 2018 •
edited
Loading

dotnet info in running container

Host (useful for support):
  Version: 2.1.3
  Commit:  124038c13e

.NET Core SDKs installed:
  No SDKs were found.

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.3 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.3 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.3 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

logs looks like this

image

@muratg
Copy link
Contributor

muratg commented Sep 13, 2018

That looks good. I am not sure why you would still get these logs.

Any chance you recently update to 2.1 and you had the logs from a previous deployment?

@ZOXEXIVO
Copy link
Author

As i said - last container build and deploy - 23 august 2018.
This error appears twice a week with 100-200 errors.

@muratg muratg added the investigate Investigation item label Sep 19, 2018
@halter73
Copy link
Member

halter73 commented Sep 20, 2018 •
edited
Loading

@OXEXIVO I'm certain that either something wasn't updated to 2.1.3 or that the logs occurred prior to the upgrade taking effect. Starting in 2.1.3, there is nothing that logs "ListenerSecondary.OnConnection" anymore.

The "ListenerSecondary.OnConnection" and "Listener.OnConnection" logs should be replaced with debug level logs containing Connection id "(null)" reset.

Can you provide us with the output of lsof -p <PID> where the <PID> is that of your dotnet exec process?

@ZOXEXIVO
Copy link
Author 

MMAND   PID USER   FD      TYPE DEVICE SIZE/OFF      NODE NAME
dotnet  23594 root  cwd       DIR   0,45    12288        11 /app
dotnet  23594 root  rtd       DIR   0,45     4096         2 /
dotnet  23594 root  txt       REG   0,45   105872        31 /usr/share/dotnet/dotnet
dotnet  23594 root  mem       REG   0,45                132 /app/Microsoft.CodeAnalysis.CSharp.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                178 /app/RabbitMQ.Client.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                 70 /app/EasyNetQ.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                192 /app/runtimes/unix/lib/netstandard2.0/System.Data.SqlClient.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                371 /app/runtimes/linux-x64/native/libuv.so (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                499 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2 (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                498 /usr/lib/x86_64-linux-gnu/libssl.so.1.0.2 (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                383 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.3/System.Security.Cryptography.Native.OpenSsl.so (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                176 /app/Newtonsoft.Json.Bson.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                212 /app/WOF.Infrastructure.Logging.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                 89 /app/Microsoft.AspNetCore.Diagnostics.Abstractions.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                170 /app/Microsoft.IdentityModel.Tokens.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                 68 /app/Devart.Data.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                 67 /app/Devart.Data.Oracle.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                497 /lib/x86_64-linux-gnu/libresolv-2.24.so (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                495 /lib/x86_64-linux-gnu/libnss_dns-2.24.so (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                493 /lib/x86_64-linux-gnu/libnss_files-2.24.so (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                 97 /app/Microsoft.AspNetCore.Http.Extensions.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                367 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.3/mscorlib.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                 72 /app/IdentityModel.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                 95 /app/Microsoft.AspNetCore.Html.Abstractions.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                 66 /app/Dapper.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                130 /app/Microsoft.AspNetCore.WebUtilities.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                167 /app/Microsoft.IdentityModel.Logging.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                150 /app/Microsoft.Extensions.FileProviders.Composite.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                101 /app/Microsoft.AspNetCore.JsonPatch.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                154 /app/Microsoft.Extensions.Localization.Abstractions.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                129 /app/Microsoft.AspNetCore.StaticFiles.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                173 /app/MongoDB.Driver.Core.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                172 /app/MongoDB.Bson.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                174 /app/MongoDB.Driver.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                196 /app/System.Runtime.CompilerServices.Unsafe.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                171 /app/Microsoft.Net.Http.Headers.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                 84 /app/Microsoft.AspNetCore.Connections.Abstractions.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                193 /app/System.IO.Pipelines.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                137 /app/Microsoft.Extensions.Caching.Memory.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                136 /app/Microsoft.Extensions.Caching.Abstractions.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                110 /app/Microsoft.AspNetCore.Mvc.Razor.Extensions.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                133 /app/Microsoft.CodeAnalysis.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                416 /usr/lib/x86_64-linux-gnu/libicudata.so.57.1 (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                116 /app/Microsoft.AspNetCore.Razor.Language.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                134 /app/Microsoft.CodeAnalysis.Razor.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                 75 /app/Microsoft.AspNetCore.Antiforgery.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                120 /app/Microsoft.AspNetCore.Routing.Abstractions.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                 82 /app/Microsoft.AspNetCore.Authorization.Policy.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                 83 /app/Microsoft.AspNetCore.Authorization.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                426 /app/WOF.Authentication.Web.Views.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                148 /app/Microsoft.Extensions.DependencyModel.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                121 /app/Microsoft.AspNetCore.Routing.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                103 /app/Microsoft.AspNetCore.Mvc.Abstractions.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                106 /app/Microsoft.AspNetCore.Mvc.Cors.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                108 /app/Microsoft.AspNetCore.Mvc.Formatters.Json.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                107 /app/Microsoft.AspNetCore.Mvc.DataAnnotations.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                113 /app/Microsoft.AspNetCore.Mvc.TagHelpers.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                112 /app/Microsoft.AspNetCore.Mvc.RazorPages.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                118 /app/Microsoft.AspNetCore.Razor.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                111 /app/Microsoft.AspNetCore.Mvc.Razor.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                114 /app/Microsoft.AspNetCore.Mvc.ViewFeatures.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                104 /app/Microsoft.AspNetCore.Mvc.ApiExplorer.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                 85 /app/Microsoft.AspNetCore.Cors.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                207 /app/System.Text.Encodings.Web.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                 87 /app/Microsoft.AspNetCore.DataProtection.Abstractions.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                 86 /app/Microsoft.AspNetCore.Cryptography.Internal.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                 78 /app/Microsoft.AspNetCore.Authentication.Core.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                 81 /app/Microsoft.AspNetCore.Authentication.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                 77 /app/Microsoft.AspNetCore.Authentication.Cookies.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                166 /app/Microsoft.Extensions.WebEncoders.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                 76 /app/Microsoft.AspNetCore.Authentication.Abstractions.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                211 /app/WOF.Bus.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                194 /app/System.IdentityModel.Tokens.Jwt.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                208 /app/WOF.Authentication.Core.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                 73 /app/IdentityServer4.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                115 /app/Microsoft.AspNetCore.Mvc.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                209 /app/WOF.Authentication.Data.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                 88 /app/Microsoft.AspNetCore.DataProtection.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                183 /app/Serilog.Sinks.File.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                185 /app/Serilog.Sinks.RollingFile.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                181 /app/Serilog.Sinks.Async.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                139 /app/Microsoft.Extensions.Configuration.Binder.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                187 /app/Serilog.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                179 /app/Serilog.Extensions.Logging.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                 91 /app/Microsoft.AspNetCore.HostFiltering.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                163 /app/Microsoft.Extensions.Options.ConfigurationExtensions.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                159 /app/Microsoft.Extensions.Logging.Debug.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                158 /app/Microsoft.Extensions.Logging.Console.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                157 /app/Microsoft.Extensions.Logging.Configuration.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                 93 /app/Microsoft.AspNetCore.Hosting.Server.Abstractions.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                127 /app/Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                177 /app/Newtonsoft.Json.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                105 /app/Microsoft.AspNetCore.Mvc.Core.dll (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                376 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.3/System.Native.so (stat: No such file or directory)
dotnet  23594 root  mem       REG   0,45                418 /usr/lib/x86_64-linux-gnu/libicui18n.so.57.1 (stat: No such file ...

@ZOXEXIVO
Copy link
Author 

/usr/share/dotnet# dotnet --info

Host (useful for support):
  Version: 2.1.3
  Commit:  124038c13e

.NET Core SDKs installed:
  No SDKs were found.

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.3 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.3 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.3 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

@halter73
Copy link
Member

@ZOXEXIVO And you're sure you're still seeing new "Listener(Secondary).OnConnection" logs? Those are logged by Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.dll, and your lsof output doesn't show that assembly as loaded. Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.dll, which is loaded, doesn't create those logs.

@ZOXEXIVO
Copy link
Author

ZOXEXIVO commented Sep 26, 2018 •
edited
Loading

@halter73 Last error date - 12 September, and we not changing anything on server (before/after 09/12)

@ZOXEXIVO
Copy link
Author

Catched today
image

@halter73
Copy link
Member

@ZOXEXIVO You should look at the dependences for the process logging that error. That error is simply not logged by in anywhere in Kestrel 2.1.3. Moreover, that error isn't logged by any version of the Sockets transport.

@muratg muratg removed the investigate Investigation item label Oct 5, 2018
@muratg muratg added this to the Discussions milestone Oct 5, 2018
@ZOXEXIVO
Copy link
Author

ZOXEXIVO commented Nov 11, 2018 •
edited
Loading

After fix IO.Pipelines CPU spikes we migrate to Sockets transport.
This error appears, when linux oom killer kill some container on host and other containers goining to swap with full freezing

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
Discussions
Development

No branches or pull requests
3 participants
@halter73 @muratg @ZOXEXIVO