Unix socket not cleaned up on exit #14134
Comments
|
We hit the same thing in our tests. It does feel strange that we don't explicitly create it yet we're responsible for deleting it. We do the same in our tests 😄 https://github.com/aspnet/AspNetCore/blob/master/src/Servers/Kestrel/test/FunctionalTests/UnixDomainSocketsTests.cs#L113 (maybe that should have been a sign). This seems like something we should fix in 3.1 cc @anurse |
|
Is it possible to mention this issue in the readme of ASP.NET Core 3.0, or perhaps just in the Kestrel docs? Anyone using it with UNIX sockets will hit this issue. |
|
We should, at least, provide an option to control deletion behavior. If we're worried about the breaking change of always deleting, we can make the option off by default. |
|
This is how I'm working around it for now: public static void Main(string[] args)
{
var host = CreateWebHostBuilder(args).Build();
DeleteOldSocketIfExists(host);
host.Run();
}
private static void DeleteOldSocketIfExists(IWebHost host)
{
// Delete UNIX pipe if it exists at startup (eg. previous process crashed before cleaning it up)
// Workaround for https://github.com/aspnet/AspNetCore/issues/14134
var addressFeature = host.ServerFeatures.Get<IServerAddressesFeature>();
var url = BindingAddress.Parse(addressFeature.Addresses.First());
if (url.IsUnixPipe && File.Exists(url.UnixPipePath))
{
Console.WriteLine("UNIX pipe {0} already existed, deleting it.", url.UnixPipePath);
File.Delete(url.UnixPipePath);
}
} |
|
While @Daniel15 provided a workaround, it may not help if using Generic Host in ASP.NET Core 3.0. |
|
BTW I advice to cleanup on startup rather than on shutdown because your app might crash and than socket will remain there anyway. This can be done in systemd hooks. |
Do you have an example of this? |
|
You could cleanup the socket like this: |
|
Triage decision: The workaround is good enough, and we won't be making a product change here. |
|
@shirhatti That's unfortunate because this is very clearly a bug. An app should clean up its allocated resources when they're no longer required. This would be similar to not fixing a memory leak because a workaround exists. |
|
Yea I agree this shouldn’t be closed. @Daniel15 can you submit a PR here? |
|
@davidfowl My free time is fairly limited these days but I can try to submit a PR if nobody else gets around to that. |
|
EDITED: |
|
@sherlock1982 maybe, I'm wondering if we can create a delete on close file to handle to handle that scenario. |
|
Triage: We're trying to see how many folks are being hit by this issue. |
|
I wonder if this should be a BCL issue. I have some data about how this should be done and maybe it should be possible open an existing file |
|
https://troydhanson.github.io/network/Unix_domain_sockets.html
|
|
Then this interaction with systemd interesting: |
|
@davidfowl That page you linked (https://troydhanson.github.io/network/Unix_domain_sockets.html) is interesting. Sounds like the old socket should be cleaned up on start, not on exit. That'd also handle the case where the app doesn't exit cleanly (eg if you Might be worth seeing what other frameworks do? |
|
@tmds Do you have experience/opinions on whether the server should cleanup the socket that's passed in? |
BrennanConroy
added
affected-very-few
|
Thanks for the workaround @Daniel15! But this may not quite work for others reading. I needed to make one little tweak.
Grabbing the path to the socket from public class Program
{
private const string ServerSocket = "/tmp/kestrel.sock";
public static void Main(string[] args)
{
var host = CreateHostBuilder(args).Build();
DeleteOldSocketIfExists(host);
host.Run();
}
private static void DeleteOldSocketIfExists(IWebHost host)
{
if (File.Exists(ServerSocket))
{
Console.WriteLine("UNIX pipe {0} already existed, deleting it.", ServerSocket);
File.Delete(ServerSocket);
}
}
public static IWebHostBuilder CreateHostBuilder(string[] args) =>
WebHost.CreateDefaultBuilder(args)
.UseKestrel(options => {options.ListenUnixSocket(ServerSocket);})
.UseStartup<Startup>();
}``` |
It is appropriate to delete the file on shutdown. When you delete on start, the benefit is the app can restart when it crashed. |
I wrote an issue proposing the |
|
Another issue that I hit is that the socket permissions are always -rwxr-x-r-x and owned by the user and group under which Kestrel runs upon socket creation. That's an issue when hosting multiple web apps running under different user accounts, and having the reverse proxy server running under totally different account and trying to read each web app socket. See how messy it gets setting the correct permissions on each app socket, when restarting any of the web apps?! |
How about when you create the socket just giving unique name (Example: /var/run/websock/mysitedomain.sock)? That's how you can avoid deleting another apps socket by mistake. |
|
.NET 6 will remove the unix socket file when the |
|
Sweet, we can close this! Thanks @tmds ! |
ghost
locked as resolved and limited conversation to collaborators
amcasey
added
area-networking
Describe the bug
I hit a similar issue in 2015 (aspnet/KestrelHttpServer#419) and I think it's still lingering today. When I shutdown the systemd service for an ASP.NET site, it won't restart again as the UNIX socket is not deleted on shutdown. I can work around the issue by deleting the socket in Program.cs, but Kestrel should be doing this itself on shutdown.
To Reproduce
Steps to reproduce the behavior:
/etc/systemd/system/test-site.service:It works fine.
It fails to start:
Expected behavior
Should restart cleanly
Additional context
dotnet --infoon my computer:On the server, it's using a self-contained deployment so I don't think there's a
dotnet ---infocommand I can run, but it's using3.0.0-preview9-19423-09The text was updated successfully, but these errors were encountered: