kestrel memory consumption is very high, it eats all the available memory in the machine !!

[akramayasrah]

In the production, I'm trying to use Kestrel as the only hosting web server for my asp.net core application without using IIS as a reverse proxy, but i have a problem with this as the consumed memory by my asp.net core application process is keep growing until it eats all the available memory in the machine!!
However, this issue is not exist when i use IIS

Why is this happening? and why it is fixed when i use IIS?

[blowdart]

IIS encapsulates certain types of requests that can cause this. This is why we say don't expose kestrel on untrusted networks. While we're working on Kestrel to mitigate this, that work is not complete yet. Your only fix is to use a reverse proxy for now.

[akramayasrah]

Thanks. but could you please share with me what those kind of requests that might cause such issue?

[blowdart]

No, I can't because it's hard to tell which one of the conditions you hit, and other people that ignore the always use a proxy are going to be at risk of the same problem.

[davidfowl]

@akramayasrah If you can capture a memory dump that would be great. As @blowdart says, you shouldn't be using kestrel on the internet until we have the proper mitigations in place but it would be good to know what your memory profile looks like to see if there are just general problems with the server.

[akramayasrah]

My memory dump is too huge, but let me see if i can share it with you. However, if i have to use iis with kestrel, then i have always to carry the overhead of IIS (http.sys....etc) and in this case i will not take the advantage of kestrel performance & high throughput and kestrel will not add a real value for me.

To prove this, I ran a simple benchmark tool to compare the performance of kestrel alone versus kestrel with iis and the results were as below:

• With kestrel alone: I got around 400000 r/s with 1.5 ms latency (which is fair enough)
• With kestrel & iis : I got just 32000 r/s with 14 ms latency (which is bad)

Please advice.

[halter73]

@akramayasrah If you can host your memory dump on skydrive/dropbox or something like that, that would be very helpful. It's not unusual for dumps to be many GBs, especially when there's some sort of memory leak.

If that's not possible, creating a gist with the output of !dumpheap -stat from windbg's sos extension, might point us in the right direction.

[akramayasrah]

OK i have shut down my asp.net app because by time it eats all my server memory, however let me run it again and once it start to consume big memory I will create a dump file for you.

Now, can anyone please advice on my performance comparison between (kestrel alone) vs (kestrel with iis as a proxy)?

And how I should proceed with my deployment if (kestrel alone) will lead to a memory leak and (kestrel with iis) is performing very bad comparing to kestrel alone?!!

[akramayasrah]

@halter73 you can download the memory dump from the below link:
https://www.dropbox.com/s/gzhmmrauudfv7fw/AspNetCore.DMP?dl=0

[halter73]

@akramayasrah Your download link now 404s, can you double check it to make sure it works?

It isn't too surprising that running Kestrel behind IIS would be slower in microbenchmarks. When the application be benchmarked isn't doing anything expensive, the overhead of the proxy can be relatively large.

I can't really explain the high memory usage of Kestrel alone without more details. The memory dump will probably be a good start.

[akramayasrah]

@halter73 i have shared the dump file with you on google docs (an invitation email has been sent to you)

I can understand that deploying behind a proxy would be slower a little bit, but in our case the overhead of IIS is very large (from 400000 r/s to 32000 r/s), is it expected that IIS will drop the performance to this limit?!

How to take the advantage of kestrel performance if i have to run it always behind iis? in other words, what is the added value for me in term of performance if i want to migrate from asp.net to asp.net core?

[halter73]

@akramayasrah Looking at your dump, I noticed that you have over 100,000 active HTTP connections established to the server. I looked over a small sample these connections, and all of the ones I looked at were in the "RequestPending" state meaning the connection is idle.

There are a couple of issues at play here. The first, most obvious is that Kestrel does not timeout idle keep alive connections. Right now, we rely on the reverse proxy like IIS or nginx to do that instead. @CesarBS is currently working on #1066 which will add a configurable keep-alive timeout that will kill these idle connections. Once this is merged, using the nightlies from https://dotnet.myget.org/gallery/aspnetcore-dev should bring memory consumption in line with what you see behind IIS.

The other issue is that while Kestrel has been very optimized to allocate extremely little per request, there is still relatively low hanging fruit in terms of lowering memory utilization per idle connection. E.g.: #1005, #699.

We plan to address the issues I mentioned above by the 1.1 release, but in the meantime you can try to work around this by limiting the number of idle connections at any given time. Ideally this would be done by the server with the idle timeout that is being introduced by #1066. But if you have some control over the clients (say it's being accessed by a mobile app or library you control), you can close the HTTP connection from the client when you are done making requests for a period time.

If you don't control the clients (e.g. browsers), you can have your app on the server set the Connection: close header on its response. The downside of this is that any subsequent requests made by the client receiving the Connection: close response will have to be made over a new connection which obviously is less efficient than using an existing connection.

[akramayasrah]

@halter73 when it is expected to release 1.1 release? are we talking about a few weeks? few months? or it is a long term release?

And after 1.1 release, can we run kestrel on production without IIS?

[akramayasrah]

@halter73 I appreciate if you answer my above questions

[halter73]

@akramayasrah I'm not sure that we've publicly announced any timelines for a 1.1 release more specific than Fall 2016. Not saying we haven't, just that I'm not sure. I believe that it will be closer to a few months than a few weeks.

As far as supporting kestrel in production without IIS or nginx after 1.1 goes, you'd have to ask @blowdart. We are currently working on hardening Kestrel so it can be used as an edge server at some point. The keep-alive timeouts are part of that work.

[blowdart]

It's unlikely our hosting recommendations will change in 1.1.

[akramayasrah]

@blowdart can we say that the high memory consumption issue willl be fixed in 1.1 release?

[blowdart]

Stephen said yes in #1053 (comment) - that precise issue will get resolved.

[muratg]

Identified issues are addressed.