CPU Usage on 2.1 constantly builds

[ByronAP]

I have increasing CPU which seems to be from HTTP. It was not noticeable until I went to production which looked great after deployment, but a few hours later everything was redlined due to ever-increasing CPU demand as shown below. Unfortunately, this is a show stopper and is forcing me to rollback.

Runtime: core 2.1 Linux x64
Containers: YES Docker
OS: UBUNTU & CENTOS

Reported by others here: microsoft/ApplicationInsights-aspnetcore#690

[stephentoub]

I have increasing CPU which seems to be from HTTP.

You profiled and it showed it was from System.Net.Http? If so, can you provide more details? What in System.Net.Http did you see causing the issue here?

[ByronAP]

My initial profile looked good but looking at how it builds up over time I need to leave it running for a lot longer. When not extensively using HTTP, services stayed in spec (and I recall some issues with HTTP from RC) hence the assumption it is HTTP related. I will profile again tonight and see what I get (hopefully it shows on Windows as well).

[ByronAP]

so far no luck on Windows :-(

[karelz]

Did you try to collect and analyze a trace from the time the CPU is high?

[ByronAP]

I would if I could, unfortunately, It requires me to setup a new testbed which I do not have time for right away, I'm still rolling back changes :-(

[karelz]

Understood. Do you think you will have time later? I am afraid that with the information available there is not much we can do :(

[ByronAP]

When later will be is unknown at this time but I will definitely be investigating and keeping thread updated. I have to get my system back to a nominal state before I can really spend any more time on it. Which so far I am 24 hours almost straight on this and about to have a heart attack lol << no ones fault but my own

[emsfeld]

Cannot provide any more details than the OP but wanted to affirm that I had the exact same issue on a very similar setup as OP (Ubuntu + Docker + Core 2.1). I rolled back also. I also suspected the System.Net.Http being the culprit as I believe I read somewhere that you moved away from using curl. However, I experienced this client side (using HttpClient) rather than server side.

[karelz]

@emsfeld do you have a chance to provide more information? Sadly, this is non-actionable from our side. There's plenty of customers on 2.1 who don't hit the problem, so it is specific to the app pattern. Without knowing the pattern or having some logs/traces, there is nothing we can do :(.

[karelz]

@emsfeld can you try to downgrade or disable AppInsights? - see microsoft/ApplicationInsights-aspnetcore#690 (comment) for possible culprit.

[emsfeld]

@karelz I am neither using AppInsights nor AspNetCore. I basically just repeatedly pull the source from websites using HttpClient from System.Net.Http. Maybe I should have reported this as a separate issue but (possibly wrongly) assumed it is related. I will try to create a short reproducible test case.

[karelz]

@emsfeld yeah, let's file a separate issue for investigation to keep things simple - we can merge them later if they turn out to be the same root-cause.
If you can give us some repro we can look at locally, that would be amazing!

[ByronAP]

my use case is the same, I make lots of http requests but do not use asp .net core or app insights. So far what I think is happening is that the new http system is not releasing all of the resources properly. I am at the create method as hot and also showing some zombies, next run I will hopefully be able to see deeper into that.

[ByronAP]

also for those of you who hit this, before rolling back try going back to curl via SetSwitch ala AppContext.SetSwitch("System.Net.Http.UseSocketsHttpHandler", false);

[stephentoub]

I make lots of http requests

The same site or to many different sites? e.g. are you closer to a) accessing the same web service over and over, or b) implementing a web crawler?

[ByronAP]

(b) many connections using proxies, however anomalies showed up for my redis connections which uses only 1 server and no proxy. However, the redis thing may have been a side effect of the main issue (my current belief).

[ByronAP]

@emsfeld are you using proxies?

[emsfeld]

@ByronAP No, no proxies.

[ByronAP]

It seems in my case the issue was caused by a bad library (stackexchange.redis) and not the framework YAY

Closing this unless someone else can provide details on an actual corefx related issue that warrants a reopen.

[danmoseley]

Thanks for the update @ByronAP . do you have a stackexchange.redist bug link to paste in?