cmd/pprof: Local symbolization failed for http.test: open /tmp/go-build348974326/b001/http.test: no such file or directory

[bradfitz]

I was just profiling something like normal and pprof gave me some angry red text about something not working, but it seems to work. I'm not sure what's broken. @hyangah?

bradfitz@gdev:~/go/src$ ./make.bash  
Building Go cmd/dist using /home/bradfitz/go1.4.  
Building Go toolchain1 using /home/bradfitz/go1.4.  
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.  
Building Go toolchain2 using go_bootstrap and Go toolchain1.  
Building Go toolchain3 using go_bootstrap and Go toolchain2.  
Building packages and commands for linux/amd64.  
---  
Installed Go for linux/amd64 in /home/bradfitz/go  
Installed commands in /home/bradfitz/go/bin  


bradfitz@gdev:~/go/src$ cd net/http/  
bradfitz@gdev:~/go/src/net/http$ go test -v -run=NONE -bench=ServeMux -cpuprofile=prof.cpu  
goos: linux  
goarch: amd64  
pkg: net/http  
BenchmarkServeMux-4        10000            113109 ns/op           19907 B/op        540 allocs/op  
PASS  
ok      net/http        1.310s  


bradfitz@gdev:~/go/src/net/http$ go tool pprof prof.cpu  
Local symbolization failed for http.test: open /tmp/go-build348974326/b001/http.test: no such file or directory  
Some binary filenames not available. Symbolization may be incomplete.  
Try setting PPROF_BINARY_PATH to the search path for local binaries.  
File: http.test  
Type: cpu  
Time: Jun 5, 2018 at 8:41pm (UTC)  
Duration: 1.30s, Total samples = 1.17s (89.82%)  
Entering interactive mode (type "help" for commands, "o" for options)  
(pprof) top  
Showing nodes accounting for 630ms, 53.85% of 1170ms total  
Showing top 10 nodes out of 95  
      flat  flat%   sum%        cum   cum%  
     110ms  9.40%  9.40%      270ms 23.08%  runtime.mallocgc  
      80ms  6.84% 16.24%      230ms 19.66%  runtime.concatstrings  
      70ms  5.98% 22.22%     1130ms 96.58%  net/http_test.BenchmarkServeMux  
      70ms  5.98% 28.21%      100ms  8.55%  runtime.mapaccess2_faststr  
      60ms  5.13% 33.33%       60ms  5.13%  runtime.heapBitsSetType  
      60ms  5.13% 38.46%       60ms  5.13%  runtime.memmove  
      60ms  5.13% 43.59%       60ms  5.13%  sync.(*RWMutex).RUnlock  
      40ms  3.42% 47.01%      750ms 64.10%  net/http.(*ServeMux).Handler  
      40ms  3.42% 50.43%       70ms  5.98%  path.Clean  
      40ms  3.42% 53.85%      270ms 23.08%  runtime.concatstring2  
(pprof) ^D

bradfitz@gdev:~/go/src/net/http$ go version  
go version devel +e7ee3b91c4 Tue Jun 5 20:00:37 2018 +0000 linux/amd64  

[hyangah]

I don't think this is a new error but became more noticeable because of the new terminal support (with color). I happened to ask the same question in google/pprof#386 today.

@aalexand @rauls5382 @rsc

The error was from https://github.com/google/pprof/blob/master/internal/symbolizer/symbolizer.go#L343
while processing the mapping data. Currently, runtime/pprof doesn't set has_functions/filenames/line_numbers fields (https://go.googlesource.com/go/+/master/src/runtime/pprof/proto.go#182) even though we do symbolization (for Go symbols). My understanding is that we didn't set them because we don't know what values to set there - especially when non-Go part is mixed in.

Should we set these values based on the collected samples? (if all pcs from the mapping were successfully symbolized, set those fields)

[aalexand]

Setting those fields may prevent locations in the C++ code from being symbolized, there is discussion on b/36645883 which I believe relevant to this.

[aalexand]

Sorry, I missed that you already mentioned what I said. Setting the flags only if all pcs are symbolized sounds reasonable.

[gopherbot]

Change https://golang.org/cl/118275 mentions this issue: runtime/pprof: set HasFunctions of mapping entries

[hyangah]

I sent cl/118275 for review. I am not sure whether it's for 1.11.

[gopherbot]

Change https://golang.org/cl/123779 mentions this issue: runtime/pprof: add a fake mapping when /proc/self/maps is unavailable

[jsha]

Note: If anybody is getting this error and wants to suppress it, you can add -symbolize=remote to your pprof invocation, which will use the remote symbols instead of trying to find them locally.

[hyangah]

@jsha That is only when the remote server (the profile origin) or any symbolization service the pprof can utilize is available.
This bug is marked fixed. If you still experience an issue, please file a bug soon.