Shutting the server down causes a hang

[davidfowl]

I ran the sample on osx using k run and then hit enter to quit the app after making a single request, it hangs forever

[davidfowl]

Seems it's not OSX specific

[chriseldredge]

Just ran the sample on OS X (commit de6c32d) and it shut down cleanly when I hit enter.

[loudej]

I've seen this too, it doesn't repro all the time. I believe it has something to do with the code that's closing all of the open libuv handles.

[chriseldredge]

Ah, yes after some more test runs I've gotten these 2 stack traces, but still no hangs for me.

Unhandled Exception:
System.Exception: Error -16 EBUSY resource busy or locked
  at Microsoft.AspNet.Server.Kestrel.Networking.Libuv.Check (Int32 statusCode) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.Server.Kestrel.Networking.Libuv.loop_close (Microsoft.AspNet.Server.Kestrel.Networking.UvLoopHandle handle) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.Server.Kestrel.Networking.UvLoopHandle.ReleaseHandle () [0x00000] in <filename unknown>:0
  at System.Runtime.InteropServices.SafeHandle.Finalize () [0x00000] in <filename unknown>:0

Unhandled Exception:
System.NullReferenceException: Object reference not set to an instance of an object
  at Microsoft.AspNet.Server.Kestrel.Networking.UvHandle.ReleaseHandle () [0x00000] in <filename unknown>:0
  at System.Runtime.InteropServices.SafeHandle.Finalize () [0x00000] in <filename unknown>:0

[natemcmaster]

I can consistently reproduce this by sending SIGINT (Ctrl+C in the terminal - which causes the terminal thread to freeze) and then hitting the port with another HTTP request.

Command:
k Microsoft.AspNet.Hosting --server Kestrel --server.urls http://localhost:5001

[simonmurdock]

I'm experiencing the exact same behaviour as reported by @natemcmaster. Ctrl+C results in a non responsive terminal, where as enter results in an immediate exit.

After a few tries I got this different error which may be helpful:

Unhandled Exception:
   System.Exception: Error -16 EBUSY resource busy or locked
  at Microsoft.AspNet.Server.Kestrel.Networking.Libuv.Check (Int32 statusCode) [0x00000] in <filename unknown>:0 
  at Microsoft.AspNet.Server.Kestrel.Networking.Libuv.loop_close (Microsoft.AspNet.Server.Kestrel.Networking.UvLoopHandle handle) [0x00000] in <filename unknown>:0 
  at Microsoft.AspNet.Server.Kestrel.Networking.UvLoopHandle.ReleaseHandle () [0x00000] in <filename unknown>:0 
  at System.Runtime.InteropServices.SafeHandle.Finalize () [0x00000] in <filename unknown>:0

[borgdylan]

This also manifests itself on linux. Solving this would allow me to move to kestrel from Nowin.

[lodejard]

Yeah, that's really irritating.

[borgdylan]

is this libuv specific, as in it is handling ctrl-c and messing stuff up?

[lodejard]

Not really - it's in the hosting program.main... I think it's either a problem somewhere in how it's asking libuv to close connections before returning, or how it's getting ctrl-c notifications on different platforms.

[borgdylan]

its a SIGINT on Linux/MAC OSX. I do not know what Windows reads it as

[chriseldredge]

I don't know how the coming CLR on POSIX will affect signal handling, but in the mean time one potential solution is to use Mono.Posix to handle SIGINT on Linux and OS X. The startup code would have to probe for OS type and maybe Runtime, but there are already places in aspnet 5 that have to do this.

[suhasj]

+1 for beta2 package of Kestrel on Ubuntu 14.0.4. Hitting Enter does not exist the server and seems to hag forever. Need to do Ctrl + C

[Tragetaschen]

I have Console.WriteLine'd this down:
The Join when disposing does not return or timeout.
Inside ThreadStart, everything runs as expected, but the last _loop.Dispose triggers a scary exception, that's swallowed by the generic handler:

Exception: GCHandle value belongs to a different domain
  at System.Runtime.InteropServices.GCHandle.op_Explicit (IntPtr value) [0x00000] in <filename unknown>:0
  at System.Runtime.InteropServices.GCHandle.FromIntPtr (IntPtr value) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.Server.Kestrel.Networking.UvMemory.DestroyMemory (IntPtr memory) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.Server.Kestrel.Networking.UvLoopHandle.ReleaseHandle () [0x00000] in <filename unknown>:0
  at System.Runtime.InteropServices.SafeHandle.Close () [0x00000] in <filename unknown>:0
  at System.Runtime.InteropServices.SafeHandle.Dispose (Boolean disposing) [0x00000] in <filename unknown>:0
  at System.Runtime.InteropServices.SafeHandle.Dispose () [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.Server.Kestrel.KestrelThread.ThreadStart (System.Object parameter) [0x00000] in <filename unknown>:0

If you put a last Console.WriteLine at the end of ThreadStart, this gets executed. This means the thread's function runs to an end and the Join should return. I can only speculate about the reason, but given the exception, I would guess memory / stack corruption and you cannot trust the runtime anymore.

[davidfowl]

I send a PR for this, see #47. Can somebody test it out and let me know how it works out? It's working well for me.

[suhasj]

@davidfowl I can take a quick look

[Tragetaschen]

Sadly, I don't see a difference.

My setup:

• amd64 (Gentoo) or ARM (Yocto)
• Mono 3.10.0
• libuv 1.0.2
• Mostly beta1

I have to revise my analysis a bit, though.

The reason for the hang is that the KestrelThread's Stop rethrows the Exception and this bubbles up through all Dispose layers (Listener, KestrelEngine, ServerFactory, HostingEngine) until Hosting's Program.cs, where the ShutdownRequested's cancellation token event handler does not set the ManualResetEvent to actually unblock the program.

[Tragetaschen]

I found the places where the memory gets overwritten:

During loop_init
During loop_close

I have stored the value of CreateMemory's GCHandle.Alloc in a local variable and compare that to what is used in DestroyMemory. Before loop_close, the value is 0, afterwards, it's -1 (since I compiled without NDEBUG).

I meant to ask since the first time I looked at Kestrel: Why is the whole P/Invoke handling implemented so complicated? All libuv's loop needs is a pointer to a piece of memory that's large enough (loop_size()). Why not just save the IntPtr of AllocCoTaskMem of the proper size in a SafeHandle's handle and be done with it?

[davidfowl]

@Tragetaschen I was testing my change on ubuntu and it still hung during shutdown. I think there was a change in libuv though since it works on windows and OSx with the bundled version.

As for the pinvoke handling you can ask @loudej 😄

[Tragetaschen]

The Windows source code does not have the memsets and thus does not overwrite the struct's (user-)data right at the beginning of the memory. As for OSx: shrug

[Tragetaschen]

#60 fixes this