runtime: add runtime/pprof goroutine labels to tracebacks

[dfinkel]

Proposal Details

Forking off #23458 and https://go.dev/cl/694119.

#23458 includes adding goroutine labels to traces of various kinds. One of them, is the runtime-generated traceback. This has turned into a more contentious subject than expected, so, I'm moving this to its own issue.

As the traceback format is not covered by the go 1 compatibility promise, both @adonovan and @prattmic have mentioned that they don't think adding these goroutine labels to the traceback out put needs a full proposal.

This issue proposes adding these goroutine labels, and provides space for discussion of different/improved formats for go 1.27+.

https://go.dev/cl/694119 has one format and is currently hidden behind GODEBUG=tracebacklabels=1.

The current version of https://go.dev/cl/694119 adds labels:{"foobar": "baz"} to the goroutine blocking-status at the top of the goroutine-level dump with some simple quoting for the key/value pairs. To avoid issues with unprintable, newlines, control characters, and higher unicode, everything outside the simple printable ASCII range is escaped.

The current escaping-scheme came out of discussions with @adonovan about the possibility of breaking tools parsing these dumps.

As an example, there's a goroutine dump from a current version (from a test):

 goroutine 8 [running labels:{"foobar": "baz"}]:
        runtime_test.TestTracebackGoroutineLabelsDisabledGODEBUG.func1(...)
                /home/davidf/src/golang/src/runtime/traceback_test.go:940
        runtime_test.TestTracebackGoroutineLabelsDisabledGODEBUG(0x1106b50406c8)
                /home/davidf/src/golang/src/runtime/traceback_test.go:952 +0x185
        testing.tRunner(0x1106b50406c8, 0x9cf9b0)
                /home/davidf/src/golang/src/testing/testing.go:2032 +0x102
        created by testing.(*T).Run in goroutine 1
                /home/davidf/src/golang/src/testing/testing.go:2102 +0x61a
        

Notably, unlike the tracebacks from the runtime package, the runtime/pprof package's debug=1 dumps have comment characters leading additional information lines. An alternative is to place these labels on another line like those, so it looks something like:

 goroutine 8 [running]:
#  labels:{"foobar": "baz"}
runtime_test.TestTracebackGoroutineLabelsDisabledGODEBUG.func1(...)
        /home/davidf/src/golang/src/runtime/traceback_test.go:940
<snip>

We could also split out the labels to be one per-line with some indenting:

 goroutine 8 [running]:
#  labels:
#   - "foobar": "baz"
#   - "quux": "bat"
runtime_test.TestTracebackGoroutineLabelsDisabledGODEBUG.func1(...)
        /home/davidf/src/golang/src/runtime/traceback_test.go:940
<snip>

My general preference would be to merge the existing https://go.dev/cl/694119 for go 1.26 and adjust for 1.27 (and only then switch the GODEBUG to default to 1)

[gopherbot]

Change https://go.dev/cl/694119 mentions this issue: runtime: print goroutine labels in tracebacks

[dfinkel]

Notably: #75047 proposes adding goroutine labels to tests run via the testing package, and discussing what labels to add.
(a tentative implementation exists as https://go.dev/cl/696595)

[gabyhelp]

Related Issues

• proposal: runtime/pprof: add individual goroutine profile with labels #74964 (closed)
• proposal: testing: add goroutine labels to tests #75047
• runtime: add 'created by goroutine number' to stack traces #38651 (closed)
• proposal: runtime: permit setting goroutine name to appear in stack dumps #35178 (closed)
• runtime/pprof: doesn't dump goroutine labels when stacks collected #63712
• proposal: errors: let GODEBUG=errstacktrace request stack backtraces #63358
• proposal: runtime: add Chain to emit only a call chain from the callee goroutine #69648
• proposal: runtime: (optionally) print M backtrace on crash #67929 (closed)

Related Code Changes

• runtime: print goroutine labels in tracebacks
• runtime/pprof: plumb labels for goroutine profiles

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[prattmic]

Just brainstorming some ideas:

It might be nice if we didn't require quotes unless there was some special character to escape.

I think there is a somewhat natural fit of "extra" metadata at the bottom of the stack trace for the "created by" line and register dumps (if killed from the signal handler). So I could imagine something like:

panic: boo

goroutine 6 [running]:
main.main.func1()
        /tmp/extract_go/main.go:10 +0x25
sync.(*WaitGroup).Go.func1()
        /usr/local/google/home/mpratt/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.4.linux-amd64/src/sync/waitgroup.go:239 +0x4a
created by sync.(*WaitGroup).Go in goroutine 1
        /usr/local/google/home/mpratt/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.4.linux-amd64/src/sync/waitgroup.go:237 +0x73
labels
        foo: bar
        baz: bat

The main downside I see here is that stack traces can be quite long and the most interesting part is usually the top, so the labels may be off-screen from the top of the stack trace.

The goroutine status ([running]) shows what the goroutine is doing from the Go runtime perspective. The labels arguably show what the goroutine is doing from the application perspective, so those have nice symmetry. (I assume that this was the initial motivation for putting the label in the brackets. To reduce noise, we could perhaps give the labels their own bracketed section alongside the goroutine status. As a dedicated section, it may not need its own label:

panic: boo

goroutine 6 [running] {foo: bar, baz: bat}:
main.main.func1()
        /tmp/extract_go/main.go:10 +0x25
sync.(*WaitGroup).Go.func1()
        /usr/local/google/home/mpratt/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.4.linux-amd64/src/sync/waitgroup.go:239 +0x4a
created by sync.(*WaitGroup).Go in goroutine 1
        /usr/local/google/home/mpratt/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.4.linux-amd64/src/sync/waitgroup.go:237 +0x73

This keeps the labels at the top, but doesn't help with many labels getting unwieldy to read on a single long line.

cc @golang/runtime @adonovan