UNIX sockets not cleaned up on exit

[Daniel15]

I'm using Supervisord to run Kestrel for my site. This is my Supervisor config:

[program:dan]
command=/var/www/.dnx/runtimes/dnx-mono.1.0.0-rc1-final/bin/dnx --appbase /var/www/dan.cx/live/site/approot/packages/Daniel15.Web/1.0.0/root Microsoft.Dnx.ApplicationHost --configuration Release web-live
environment=ASPNET_ENV=Production
user=www-data
directory=/var/www/dan.cx/live/site/

Command in project.json:

"web-live": "Microsoft.AspNet.Server.Kestrel --server.urls http://unix:/run/dan-live.sock",

This is working fine, but when I stop Kestrel (supervisorctl stop dan, "dan" being the name of the site in the Supervisor config), it does not clean up the UNIX socket (/run/dan-live.sock). The next time I try to start it, I get an "address already in use" error:

Microsoft.AspNet.Server.Kestrel.Networking.UvException: Error -98 EADDRINUSE address already in use
  at Microsoft.AspNet.Server.Kestrel.Networking.Libuv.Check (Int32 statusCode) <0x41d3a2a0 + 0x00063> in <filename unknown>:0
  at Microsoft.AspNet.Server.Kestrel.Networking.Libuv.pipe_bind (Microsoft.AspNet.Server.Kestrel.Networking.UvPipeHandle handle, System.String name) <0x41d3e0e0 + 0x00053> in <filename unknown>:0
  at Microsoft.AspNet.Server.Kestrel.Networking.UvPipeHandle.Bind (System.String name) <0x41d3e0a0 + 0x00027> in <filename unknown>:0
  at Microsoft.AspNet.Server.Kestrel.Http.PipeListener.CreateListenSocket () <0x41d3d9c0 + 0x00087> in <filename unknown>:0
  at Microsoft.AspNet.Server.Kestrel.Http.Listener+<>c__DisplayClass5_0.<StartAsync>b__0 (System.Object _) <0x41d3d8c0 + 0x00030> in <filename unknown>:0

I need to manually rm /run/dan-live.sock before I can start Kestrel again. I think Kestrel simply needs to delete the socket on exit.

[Daniel15]

As a guess, PipeListener and PipeListenerPrimary should delete the socket in their Dispose implementation.

[cesarblum]

Supervisor sends SIGTERM by default to stop a process. If you set stopsignal to INT, that should gracefully stop kestrel and the UNIX socket file will be deleted. Could you try that and get back to us with the results?

[Daniel15]

@CesarBS - unfortunately it looks like that didn't work. This is my full config:

[program:dan]
command=/var/www/.dnx/runtimes/dnx-mono.1.0.0-rc1-final/bin/dnx --appbase /var/www/dan.cx/live/site/approot/packages/Daniel15.Web/1.0.0/root Microsoft.Dnx.ApplicationHost --configuration Release web-live
environment=ASPNET_ENV=Production
user=www-data
directory=/var/www/dan.cx/live/site/
stopsignal=INT

Here's what I was doing:

root@dan:~# supervisorctl status dan
dan                              RUNNING    pid 16159, uptime 0:00:27

root@dan:~# ls -lah /run/dan-live.sock
srwxr-xr-x 1 www-data www-data 0 Dec  8 19:55 /run/dan-live.sock

root@dan:~# supervisorctl stop dan
dan: stopped

root@dan:~# ls -lah /run/dan-live.sock
srwxr-xr-x 1 www-data www-data 0 Dec  8 19:55 /run/dan-live.sock

SIGTERM should still clean up the sockets, right? I didn't know much about the different signals, but I found this useful post on Quora that explains them. Emphasis mine:

SIGTERM is the termination signal. The default behavior is to terminate the process, but it also can be caught or ignored. The intention is to kill the process, gracefully or not, but to first allow it a chance to cleanup.

[cesarblum]

My suggestion was actually an educated guess. I didn't really run Kestrel behind Supervisor - what I actually did was to check the behavior difference between killing kestrel with SIGTERM and SIGINT and I noticed that the socket lingered around with SIGTERM but not with SIGINT.

Just to make sure: did you restart Supervisor after setting stopsignal to INT?

[Daniel15]

Yeah, I restarted Supervisor.

[cesarblum]

When I try running an app with supervisor, I get

kestrel: ERROR (abnormal termination)

Have you ever run into that?

[Daniel15]

Try checking the output. supervisorctl tail appname for stdout and supervisorctl tail appname stderr for stderr, or look at the log files for the process (they're in /var/log/supervisord/ on my system)

[cesarblum]

Can your app run on CoreCLR? I'm now able to repro your issue, but I noticed it only happens on Mono. If I use the CoreCLR DNX the socket is gone once Supervisor stops the process.

[Daniel15]

Unfortunately I can't use CoreCLR yet as I'm using some third-party libraries that don't support it and also some stuff in the System.Drawing namespace.

[cesarblum]

Can you run your app manually, kill it with SIGINT and check if the socket file is still there after that? I'd like to check that you're seeing the same results as I'm seeing. When I kill the process manually, the socket is deleted.

[Daniel15]

Interesting... It does appear to correctly clean up the socket when I run the app manually and use SIGINT. Is Supervisor doing something differently?

[cesarblum]

It's what I'm trying to figure out. So far I haven't found anything weird in their code. I suspected Python's os.kill could be doing something funny but I tested it and the socket is cleaned. I'll keep investigating.

[cesarblum]

@Daniel15 After a little more investigation, this continues to be a mystery. It looks very much like a Mono issue. Can you open an issue on Mono for this?

Meanwhile I suggest you work around this issue by setting your Supervisor command to a script that first deletes the socket file and then launches your app.

@muratg Since this looks like a Mono issue, I'll go ahead and close this.

[shirhatti]

I tried to repro this outside of Supervisor
I was using rc1 coreclr bits.

SIGTERM does not cleanup the socket file but SIGINT does.

[cesarblum]

This is a known issue that we had with DNX and will continue to have with dotnet. It's not really up to Kestrel to do it, but in any case we're unable to handle SIGTERM. @pakrym should have more info.

[Daniel15]

I suggest you work around this issue by setting your Supervisor command to a script that first deletes the socket file and then launches your app

Now that ASP.NET Core websites are just .NET executables, it was easy enough to modify my Main method to delete the socket if it exists on startup:

public static void Main(string[] args)
{
    var config = new ConfigurationBuilder()
        .AddCommandLine(args)
        .Build();

    var host = new WebHostBuilder()
        .UseKestrel()
        .UseConfiguration(config)
        .UseContentRoot(Directory.GetCurrentDirectory())
        .UseIISIntegration()
        .UseStartup<Startup>()
        .Build();

    // Delete UNIX pipe if it exists at startup (eg. previous process crashed before cleaning it up)
    var addressFeature = host.ServerFeatures.Get<IServerAddressesFeature>();
    var url = ServerAddress.FromUrl(addressFeature.Addresses.First());
    if (url.IsUnixPipe && File.Exists(url.UnixPipePath))
    {
        Console.WriteLine("UNIX pipe {0} already existed, deleting it.", url.UnixPipePath);
        File.Delete(url.UnixPipePath);
    }

    host.Run();
}