runtime/metrics: add goroutine state counts, total goroutines created, total threads

[deft-code]

Update, Jun 7 2023: runtime/metrics now exists, but there are a few metrics in the draft CL here that aren't yet exposed. See #15490 (comment).

---

MemStats provides a way to monitor allocation and garbage collection.

We need a similar facility to monitor the Scheduler.

Briefly:

• Total goroutines create
• Current number of of goroutines
• Total number of goroutines scheduled
• Current number of goroutines scheduled
• Total thread starts
• Current number of threads.
• Metrics on the delay between a goroutine being ready and running on a proc.

[minux]

You can get all of these statistics by parsing the tracing output from runtime/trace.

[aclements]

@minux, while true, runtime/trace seems like a pretty high overhead way to collect what amounts to a fairly small amount of information. It's certainly low overhead for what it does, but what it does is much more than what's needed here. The metrics @deft-code wants are primarily intended for continuous monitoring (based on offline conversations), so it needs to be cheap.

[aclements]

Here are the notes on the desired metrics I had from our meeting a while ago:

Ring buffer of sampled duration between entering and exiting runnable state

• With some probability, when a goroutine enters runnable, tag it, and when it exits runnable, add the runnable duration to a ring
• Consumers can do what they want with these samples, including just averaging them, or building distributions.

Four global stats

• Current number of goroutines
• Total number of goroutines ever created
• Current number of runnable goroutines
• Total number of runnable-to-running transitions

Maybe current number of running goroutines

[bradfitz]

Assigning to @aclements to decide what we're willing to support long-term.

[adg]

Ping @aclements

[rsc]

Still up to @aclements.

[rsc]

Ping @aclements. Can you look at this during the release candidate quiet?

[aclements]

@deft-code, do the specific stats I suggested in #15490 (comment) address your needs?

[aclements]

Sorry, I'd lost track of the fact that there was a concrete proposal doc for this: https://github.com/deft-code/proposal/blob/master/design/15490-schedstats.md

@deft-code, could you mail a CL to add this to the go-proposal repository and, once submitted, edit your first post to link to it? Thanks.

[deft-code]

I'll get on top of it.

[aclements]

Thanks!

[gopherbot]

CL https://golang.org/cl/38180 mentions this issue.

[rsc]

Do we need to keep this issue open, or should we accept it?

[aclements]

There's definitely still work to do on how and what exactly the API should expose, but I think it's pretty clear we need to provide some visibility into the scheduler.

[rsc]

Teams inside Google are patching in CL 38180 and getting some experience with it. If others would like to do the same, please do. We'll probably wait until Go 1.10 to decide to add the API officially. Putting the proposal on hold until then.

[mknyszek]

If someone would like to take a stab at implementing this, please be my guest. Otherwise, I'll get to it next cycle.

[ianlancetaylor]

@mknyszek Thanks. Should we keep this issue open and retarget to runtime/metrics? Or should we open a new proposal?

[mknyszek]

We can keep this issue open and retarget it. I'll update the header and such.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[rsc]

Retitled based on #15490 (comment).
This is about per-state goroutine counts, a count of total goroutines created, and total number of OS threads.

Have all concerns about this proposal been addressed?

[mknyszek]

Yeah, I believe all concerns are addressed.

[rsc]

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

[rsc]

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

[mknyszek]

As for what these metrics should be named, perhaps:

/sched/threads:threads
/sched/goroutines-created:goroutines (cumulative)
/sched/goroutines/waiting:goroutines
/sched/goroutines/runnable:goroutines
/sched/goroutines/running:goroutines
/sched/goroutines/not-in-go:goroutines

The goroutine state metrics come from https://go-review.googlesource.com/c/go/+/38180/9/src/runtime/pstats.go#18. I figure we can reuse most of that implementation.

[prattmic]

/sched/goroutines/not-in-go:goroutines is intended to cover syscalls and cgo, I assume? i.e., it is _Gsyscall?

[gopherbot]

This issue is currently labeled as early-in-cycle for Go 1.22.
That time is now, so a friendly reminder to look at it again.

[gopherbot]

This issue is currently labeled as early-in-cycle for Go 1.23.
That time is now, so a friendly reminder to look at it again.

[ianlancetaylor]

Too late for 1.23.

[gopherbot]

This issue is currently labeled as early-in-cycle for Go 1.24.
That time is now, so a friendly reminder to look at it again.