runtime/pprof: no API to access deep stacktraces with MemProfileRecord and BlockProfileRecord

[korniltsev]

Go version

go version devel go1.23-b788e91bad Tue Jun 11 18:08:44 2024 +0000 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/korniltsev/.cache/go-build'
GOENV='/home/korniltsev/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/korniltsev/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/korniltsev/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/home/korniltsev/github/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/home/korniltsev/github/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='devel go1.23-b788e91bad Tue Jun 11 18:08:44 2024 +0000'
GODEBUG=''
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/korniltsev/.config/go/telemetry'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/home/korniltsev/github/go/src/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1768278824=/tmp/go-build -gno-record-gcc-switches'

What did you do?

I compare stacks from runtime.MemProfile and runtime/pprof.WriteHeapProfile

https://go.dev/play/p/7iUI7DdMI3r?v=gotip

What did you see happen?

I see the MemprofileRecord stacks are truncated at 32 depth and pprof stacks has proper deep stacks.

What did you expect to see?

I expected MemprofileRecord.Stack() to return a slice with correct deep stack.

[gabyhelp]

Similar Issues

• runtime: heap, allocs, block, and mutex profiles truncate stacks at 32 frames #43669
• runtime/pprof: WriteHeapProfile example is not strictly equivalent to testing memprofile #65328
• runtime/pprof: incorrect function names for generics functions #64528
• runtime/pprof: inline frames may not use combined location #37446
• runtime: frame pointer unwinding can fail on system goroutines #63630

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[korniltsev]

runtime.MemProfile and MemprofileRecord are used in [godeltaprof] (https://github.com/grafana/pyroscope-go/tree/main/godeltaprof)

It is probably possible to hack around the limitations by linking into runtime/pprof.memProfileInternal but in the light of #67401 this now looks like less attractive idea.

[felixge]

I worked on the patch for adding deep stack traces. FWIW I initially tried to make deep stack traces available via the public API, see v35 of my CL.

The problem is that there are various go1compat challenges in doing this, so we'll need a proposal that either manages to solve the problem without compat issues, or build consensus for accepting a minor amount of compat issues. Specifically:

1. Are we okay with breaking apps that don't expect Stack() to return more than Stack0 frames?
2. Are we okay if the StackRecord types are no longer properly comparable? (in either the spec sense, or by containing a pointer that makes comparison meaningless)
3. Are we okay with changing the semantics of whether or not the slice returned by Stack() is backed by the Stack0 array in all cases?

[korniltsev]

One of the possible solutions (for godeltaprof) could be to have the same functionality in golang standard library and to deprecate the godeltaprof project alltogether.

#57765
#67942

[znkr]

We have a similar issue in with Google's internal peakheap profile implementation. As the name suggest, the peakheap profile keeps track of high-water-mark heap profiles. I was considering filing a proposal to add peakheap profiles to the runtime, but I didn't have enough time to think it through. Given that I am not the only one with such a problem, I am favoring the idea of exporting the larger stacks via a runtime API now.

Regarding the go1compat challenges: Do we have to change the existing API or could there be a v2 API (e.g. in a new package, say runtime/profile).

[prattmic]

cc @golang/runtime

[rsc]

I believe this is a duplicate of #43669.

[mknyszek]

@rsc I think there's some overlap, but this is really about a specific unresolved part of #43669. Meanwhile, #43669 is now mostly resolved, since as of https://go.dev/cl/572396, deep stacks are available everywhere except the runtime APIs mentioned in this issue. I think maybe we should close #43669 in favor of this issue?

[rsc]

Sure, if you think #43669 is done, then feel free to close it.

Personally, I would say that the API to access the deep stack traces is runtime/pprof, and we just deprecate the old MemProfileRecord, BlockProfileRecord structs.

[korniltsev]

I assume we will have to link into new pprof_memPrifileInternal and co. if we want deep stacktraces? I am not missing anything, right?