admission control: improve idle CPU usage

[RaduBerinde]

I have an empty local roachprod cluster running with no data and no load and each cockroach process takes about 8-10% of a CPU core. Others have observed that even with a SQL-only tenant process, we still see 5-8%.

This seems excessive to me. We should look at some profiles and see if there is any low hanging fruit for improving on this.

[ajwerner]

Is this golang/go#27707?

[RaduBerinde]

The main culprit was determined to be the util/goschedstats loop. #66894 reduces the frequency of this significantly on 21.1. Hopefully in 21.2 we can switch to using the new /sched/latencies:seconds in runtime/metrics and collect that as needed (not with a fixed rate).

[petermattis]

The main culprit was determined to be the util/goschedstats loop.

Heh. The finger points back at yourself.

[RaduBerinde]

Yeah.. We tested that it has very little observable effect on workloads, but we didn't think that idle usage would be so different.

[RaduBerinde]

#66894 fixed the problem on 21.1. This bug tracks fixing the problem on master, in a way that doesn't negatively impact the new admission control code.

The new metric in go 1.17 likely will not help because the admission control code relies on frequent updates, and the histograms are likely going to make things even slower.

One approach is to use a much reduced update frequency until and unless admission control is actually queueing things.

[andy-kimball]

Note that this issue is a hard blocker for the Serverless launch. We'd like to launch on a beta version of 21.2 cut during the stabilization period, so we need to address this soon.

[blathers-crl]

Hi @RaduBerinde, please add branch-* labels to identify which branch(es) this release-blocker affects.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}