Nancy: What is Nancy.Hosting.Event2.dll? + Add discussion summary to README

[phillip-haydon]

What is this host?

https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/nancy/lib/Nancy.Hosting.Event2.dll

The benchmarks for any .NET stuff are pretty terrible so I'm trying to figure out what this junk is and see if we can get some better benchmarks in.

Is there any source code?

Also why does this exist? https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/nancy/src/Global.asax.cs

[bhauer]

Your second question appears to be the same as #879, which in turn I believe is related to #325.

However, the Event2.dll is not something I have any knowledge of. Let me see if I can loop in past contributors @kppullin and @MalcolmEvershed for their opinions.

[kppullin]

Event2.dll

I believe this is based on https://github.com/kekekeks/evhttp-sharp . @kekekeks should be able to fill in the details. IIRC either the pull request or the evhttp repo had sample results showing the net effect.

---

Global.asax thread configuration

I based much of the Nancy and ServiceStack configurations on the then-existing ASP.NET setup, which also set these values in a similar manner. To keep things consistent I carried the idea over and also because they provide better end results (based on previous production tuning at work and testing via the FrameworkBenchmarks suite).

There are two issues with the default values (which also vary by framework version, but a test against 4.5 just showed 1 min thread per core, and 2048 max per core [though the max might be a hard cap regardless of # of cores]).

Min

On the min value side, once the number of threads allocated to the threadpool exceeds the min value, subsequent threads are spun up at the rate of 1 every 500ms, up to the max limit.

Let's consider that the 256 concurrent request benchmark. Based on the default settings it will take 64 seconds to allocate 256 threads on the server side (256 requests / 2 cores * .5 seconds). This is certainly non-optimal for the worse case latency metric (whether we should optimize for this aspect is a separate topic). It affects the other metrics as well, but to a lesser extent as the average settles out over time.

Max

On the max value side, as the number of threads allocated grows, we first start running into diminishing returns and, shortly thereafter, negative returns - particularly for the non-IO dependent benchmarks. So setting a reasonable max value helps prevent run away thread allocations.

Conclusion

After a bit of testing the values in the config provided the "best" overall result. However, this was on a couple medium EC2 nodes. Other settings are likely more optimal for other configurations, which is difficult to determine without having access to such configurations.

Such is the crux of performance tuning : )! Perhaps it makes sense to revert to the defaults due to this setting being 'non-standard' (though I'd argue should be a best practice).

For more details check the links referenced here: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/aspnet/src/Application.cs#L67

---

With all that, I'd love to see some more love put in to all of the .NET tests, both on Windows and Linux, and also with the new ASP.NET vNext bits! I had a baby 5 months ago and am just starting to find time to breathe, but not enough yet to help contribute at the moment :)

Let me know if you have further questions and I'll help the best I'm able.

[kekekeks]

The main problems with .NET serverside running on Mono are:

• slow slow cross-AppDomain calls
• crappy implementation of ASP.NET stack. It has several bottlenecks and non-optimal code. There is https://github.com/xplicit/HyperFastCgi that has some workarounds, but its still slow.
• non-optimal System.Net.Sockets (it follows Windows model that is designed for IOCP with thread pool waiting for events, while for Linux it's better to have several worker threads using epoll and handle events in the same thread that got it from the kernel)

While to trying to get Nancy up and running, I found out that self-hosting of it works way faster than hosting it using mono-xsp. Then I've created a wrapper around libevent2's built-in HTTP-server that can handle twice more requests per second.

ASP.NET vNext will eliminate only the second problem. For Mono they use NOwin, which is still limited by Mono's network subsystem. It can be eliminated by creating Mono.Sockets on top of libuv or something like that (See http://lists.ximian.com/pipermail/mono-devel-list/2014-May/041484.html )

[phillip-haydon]

Right now none of the Nancy tests run.

The Windows one has such bad results despite testing I did last year which (without async) is twice as fast as MVC/WebAPI, and ServiceStack is faster again. Yet the results from this Framework Benchmark show super poor results.

While the results shown for Mono pale in comparison to tests I did with fastcgi-mono-server4 and nginx. However I was unable to get async working with this method.

[kekekeks]

AFAIK, Mono ASP.NET stack implementation doesn't support async/await pattern yet.

[phillip-haydon]

So I'm told, but it does work if you push the requests from nginx to a self host which I think is described on the Nancy wiki.

[kekekeks]

Self host uses HttpListener and own request management code, so async/await works fine. It also works fine with evhttp-sharp for the same reason. However, its useless for benchmarks on Mono, since neither Mysql.Data.dll and Npgsql.dll support *Async methods. Such functionality is scheduled for Npgsql, but isn't done yet.

[phillip-haydon]

True, but EF 6 supports async and MySQL. So can async not be done with that?

Also you could just wrap the call yourself for the purpose of the benchmark test, even if it's run as a separate test. I have no Mono benchmarks to know if there is any gain but it would be interesting to see if there is.

[kekekeks]

DbCommand has default implementations of *Async methods that run synchronously, for example https://github.com/mono/mono/blob/master/mcs/class/System.Data/System.Data.Common/DbCommand.cs#L172

[jchannon]

So thats a mono issue? FYI : Dapper supports async which is what is being used in these tests but until mono make them truely async then i guess theres no gain

[kekekeks]

It's MySql/Npgsql issue, Mono just provides default implementation.

[jchannon]

Ah I see because MySql/Npgsql need to inherit that abstract class

[hamiltont]

@phillip-haydon Can we close this?

For everyone else involved here ( @jchannon , @kekekeks @kppullin ) -- I would to see a pull request that moves a summary of this discussion into the README for the affected frameworks so it's easy to find for anyone interested