HyperFastCgi ThreadPool'ed NativeListener does not work with the new Mono 4.2.1 release

[derFunk]

I'm running HyperFastCGI in an isolated Docker container based on Debian Jessie, and just switched Mono from version 4.0 to 4.2.1.102.

With 4.2.1 the ASP.NET application does not come up / does not get initialized. Requests end in a timeout.

Nothing else except updating to Mono 4.2.1.102 has been changed.

HyperFastCGI is built from the latest commit in trunk (9f8a4df).

I don't have any more informations right now, but will continue researching on this.

/Edit: It's the NativeListener which is not working. When switching to the ManagedFastCgiListener the ASP.NET app gets called and responds.

[derFunk]

I was able to isolate the issue to the ThreadPool.

This config is not working with Mono 4.2.1:

<listener type="HyperFastCgi.Listeners.NativeListener">
    <apphost-transport type="HyperFastCgi.Transports.NativeTransport">
            <multithreading>ThreadPool</multithreading>
    </apphost-transport>
    <protocol>InterNetwork</protocol>
    <address>127.0.0.1</address>
    <port>9000</port>
</listener>

And this is working:

<listener type="HyperFastCgi.Listeners.NativeListener">
    <apphost-transport type="HyperFastCgi.Transports.NativeTransport">
            <multithreading>Single</multithreading>
    </apphost-transport>
    <protocol>InterNetwork</protocol>
    <address>127.0.0.1</address>
    <port>9000</port>
</listener>

Maybe because of this:

• http://www.mono-project.com/docs/about-mono/releases/4.2.1/#microsofts-threadpool

Microsoft’s ThreadPool
We have replaced Mono’s implementation of the ThreadPool with Microsoft’s implementation. It has many interesting properties in terms of how threads are started up and terminated, which was too aggressive previously.
The new code is also simpler to maintain.

Unfortunately I see nothing in the logs which could give more hints. That's the log when ThreadPool is set:

/usr/bin/hyperfastcgi4 /config=/hyperfastcgi.conf /applications=/:/usr/aspnet/ /verbose /loglevels=All /logfile=/var/log/nginx/fastcgi.log /keepalive=false
[2015-12-08 13:48:17Z] Debug HyperFastCgi
[2015-12-08 13:48:18Z] Debug Register native transport
host-list.c:38: register_host(): (null): -1 :/:/usr/aspnet/ host=0x7f8856462fc8 pinned_host=0x7f8856462fc8 domain=0x1e76480
[2015-12-08 13:48:18Z] Debug Configured host in domain d95d475a, id=1
libev.c:461: Listen(): libevent version: 2.0.21-stable
libev.c:475: Listen(): libevent is using epoll for events.
libev.c:317: cmd_error(): Remote host disconnected from fd 14.

The disconnect happens after timeout.

[xplicit]

I need to look at it. I did not test HFC on mono 4.2 yet, because mono 4.2 looks quite unstable for me, it crashes to core dump on some basic operations (working with monodevelop, running and debugging ASP.NET applicaitons via XSP). Maybe this is just my configuration, but I haven't seen such crashes on mono 4.0 or mono 3.12)
I think you are right, this is because they did breaking change with threadpool, possible thread does not start to execute the code, but I need to look deeper to get the cause of such behavior.

[xplicit]

Could you check if you change setting Threadpool to Task is there timeout or not?

[derFunk]

Unfortunately, Task is not working as well - no response and timeout.

[derFunk]

Mono 4.2.1 is indeed unstable (with HFC). When using Singlethreading, the first couple of requests work, but then it suddenly stops and no responses are sent back, resulting in a timeout.

We've for now rolled back to Mono 4.0.

[xplicit]

Could you check the latest commit from master branch? Did it resolve the issue for you?

[derFunk]

Looks good so far, thanks!

I've updated our dev config to use Mono 4.2.1 again together with the latest commit (a1d73e8).

Let's see if this is production ready ;)

Are there any explicit load/stress tests (except brute force stress testing) that you would recommend for testing HCI?

[xplicit]

Because the threadpool was changed in the mono I would try to test creating large number of threads and look it is stable. This can be emulated by executing long-running responses. (For example make 500 simultaneous calls of ASP.NET page, which is rendered about minute and look on the thread creation/returning to the thread pool).

[derFunk]

I've continued my tests and opened two issues here:
#47
#48

[derFunk]

@xplicit I got more details now, but unfortunately no resilient test case: It seems that at least #48 is related to a large number of long running threads, with /keepalive=false, like you described above. This seems not to be stable making HFC crash.