proposal: runtime/pprof: add SetMaxDumpGoroutine to limit the number of goroutines dumped in a profile

[doujiang24]

We are using pprof for online profiling and we found goroutine dumping may take too much time when there are many goroutines.

In this case, it will take ~300ms when there are 100k goroutines on my side.
https://gist.github.com/doujiang24/9d066bce3a2bdd0f1b9fe1ef49699e4e

It's a too long time since there are almost 10k-100k goroutines in our system.

I think the easier way is to introduce a new API SetMaxDumpGoroutineNum.
I have implemented it in this PR: #50771

But, the SetMaxDumpGoroutineNum API introduced a new global variable.
This may not be a good idea.

Any feedback would be greatly appreciated. Thank you!

[aclements]

Could you explain your use case more? Usually "online profiling" is done using a CPU profile, not a goroutine profile.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[rhysh]

Goroutine profiles stop the world, iterate over the entire list of live goroutines, and then restart the world. That whole-app pause can be hundreds of milliseconds when there are hundreds of thousands of goroutines.

@doujiang24 , is the problem for your apps the length of time that the app cannot do any other useful work? If so, it sounds similar to what's described in #33250.

Or is it the length of time that specific goroutine calling runtime/pprof.Lookup("goroutine").WriteTo needs to wait before it gets results?

---

@aclements , I've found regular collection of goroutine profiles to be one of the best tools (combination of effectiveness and ease-of-use) for tracking down problems of "why did my app get slow", including ones like the net/http deadlock in issue 32388. When I encountered that one, it affected a single service instance and self-resolved after about 15 minutes, so we were only able to understand it thanks to regular and automated collection of goroutine profiles. See also fgprof. Though @doujiang24 likely has other uses.

[doujiang24]

Could you explain your use case more? Usually "online profiling" is done using a CPU profile, not a goroutine profile.

@aclements Sorry for the delay.
We want to know what are the goroutines doing or waiting for when there are too many goroutines.
thus, we may get a chance to reduce the number of goroutines.
Sampling is good enough for us since we don't need to know every goroutine.

[doujiang24]

@rhysh Yeah, it's similar to #33250.
We use go for building a network proxy platform(MOSN), it's mostly used as a service mesh data plane.
So, latency is very important for us.

[doujiang24]

I came out with a new idea about the API.
We can introduce the WithMaxGoroutine instead of SetMaxDumpGoroutine to avoid introducing a global variable.

usage:

pprof.Lookup("heap").WithMaxGoroutine(1024).WriteTo(...)

[rsc]

The big question is whether the stop-the-world is the problem versus the size of the profile. If the former, then writing less data won't take appreciably less time.

[doujiang24]

The big question is whether the stop-the-world is the problem versus the size of the profile. If the former, then writing less data won't take appreciably less time.

@rsc Thanks for your point. I did more testing and I'm sharing the test results.

less data works for the previous case that takes ~300ms total (https://gist.github.com/doujiang24/9d066bce3a2bdd0f1b9fe1ef49699e4e)

Also, I have got more testing results by adding more log in pprof.go.
It shows 10x faster by writing less data.

100k goroutines, without the max goroutine limitation, means the standard behavior.

1. fetch profile(runtime_goroutineProfileWithLabels, including the STW) takes ~130ms.
2. the total pprof goroutine dump takes ~300ms (including printCountProfile).

100k goroutines, with the max 1k goroutine limitation, by using the PR #50771

1. fetch profile(runtime_goroutineProfileWithLabelsAndLimit) takes ~15ms.
2. the total pprof goroutine dump takes ~18ms.

Maybe STW may take more time in some other cases, but I think this shows less data will really take appreciably less time.

[rsc]

PR 50771 seems to be doing a random sample of the goroutines. Is that helpful in your use case, not to see all of them?
Then there's the question of the API.

/cc @aclements @mknyszek @prattmic for thoughts

[aclements]

I suspect we could eliminate the STW here at the cost of losing "snapshot consistency" across the goroutine stacks.

Just looking over the current implementation, we could certainly improve the performance of this, though I doubt we can squeeze 10x out of it short of substantially redesigning how traceback works (e.g., using frame pointers).

[mknyszek]

I think it's still not clear to me where exactly the problem lies for your use-case, @doujiang24. Is the 300 ms delay between when you request the profile and when you get it the problem, or is the 130 ms (or however long the world is actually stopped) the problem, because it's a global latency hit.

If it's the latter, one thing we could do instead of limiting the number of goroutines is relax the consistency of the goroutine dump. Depending on how strict we want to be on backwards compatibility with this, this might not require a new API. If the goal is statistical goroutine sampling anyway, then I think that's fine. If a major use-case is getting a consistent snapshot that always 100% makes sense, then either this isn't really an option or it needs a knob.

I imagine that the global latency impact of this approach would be the same latency impact as stack scanning by the GC, which is definitely much, much less than 130 ms even for 100,000 goroutines. However, this would not help the end-to-end latency of the operation; it would still take 300 ms for that many goroutines, limiting how often you can sample.

[prattmic]

Here are some of my initial thoughts, I haven't thought too deeply about this.

Currently, the goroutine profile is a complete instantaneous (one-off) view of what every goroutine is doing. Since it involves a stop-the-world (STW), we even get a consistent view of where every goroutine is at a single point in time [1] [2].

On the other hand, this proposal effectively turns this profile into an instantaneous (one-off) sampled view of what goroutines are doing, since we don't capture every goroutine. It would still maintain the consistent view due to STW.

My question for brainstorming is whether this is the best interface for use-cases of this profile? Is sampling OK, or do we typically need all goroutines? Is a consistent snapshot time needed? If so, maybe this API is best.

But I could also imagine perhaps we could provide a continuous sampling goroutine profile. Rather than a single instantaneous profile point, we sample goroutine state over time, eventually capturing all of them but never all at the same time. Would such a profile be useful, I'm not sure.

[1] As opposed to collecting traces while goroutines are running, which would result in each goroutine getting traced at a slightly different time of execution. In this case it would be possible to observe what appear to be "impossible" states between 2 goroutines.

[2] Modulo that some small bits of code not not preemptible and thus cannot be observed in the profile, but I don't think this matters much.