Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Panic: Index out of range error when merging profiles #3164

Closed
aleks-p opened this issue Apr 4, 2024 · 1 comment · Fixed by #3188
Closed

Panic: Index out of range error when merging profiles #3164

aleks-p opened this issue Apr 4, 2024 · 1 comment · Fixed by #3188
Assignees

Comments

@aleks-p
Copy link
Contributor

aleks-p commented Apr 4, 2024

Encountered index out of range errors in the read path (ingesters):

func RewriteMappings(p *profilev1.Profile, n []uint32) {
for _, loc := range p.Location {
loc.MappingId = uint64(n[loc.MappingId-1]) + 1
}
}

called from:

m.tmp = slices.GrowLen(m.tmp, len(p.Mapping))
m.mappingTable.Index(m.tmp, p.Mapping)
RewriteMappings(p, m.tmp)

golang.org/x/sync@v0.5.0/errgroup/errgroup.go:72 +0x96
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 93207350
golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75 +0x56
golang.org/x/sync/errgroup.(*Group).Go.func1()
github.com/grafana/pyroscope/pkg/phlaredb/symdb/resolver.go:263 +0x138
github.com/grafana/pyroscope/pkg/phlaredb/symdb.(*Resolver).withSymbols.func1()
github.com/grafana/pyroscope/pkg/phlaredb/symdb/resolver.go:248 +0x178
github.com/grafana/pyroscope/pkg/phlaredb/symdb.(*Resolver).Pprof.func1(0x2258460?, {{0xc18379d420, 0x1, 0x1}, {0xc18379d428, 0x1, 0x1}, {0x0, 0x0, 0x0}})
github.com/grafana/pyroscope/pkg/pprof/merge.go:26
github.com/grafana/pyroscope/pkg/pprof.(*ProfileMerge).Merge(...)
github.com/grafana/pyroscope/pkg/pprof/merge.go:68 +0x62d
github.com/grafana/pyroscope/pkg/pprof.(*ProfileMerge).merge(0xc02ecaa7e0, 0xc0aee65350, 0x1?)
github.com/grafana/pyroscope/pkg/pprof/merge.go:259
github.com/grafana/pyroscope/pkg/pprof.RewriteMappings(...)
goroutine 93207317 [running]:

panic: runtime error: index out of range [0] with length 0

At a glance it looks like we are unable to merge profiles that have no mappings.

@kolesnikovae kolesnikovae self-assigned this Apr 5, 2024
@kolesnikovae
Copy link
Collaborator

Thanks for creating the issue. This is interesting: we assume that at least one mapping must be present. I'll take a look into this.

simonswine added a commit to simonswine/pyroscope that referenced this issue Apr 10, 2024
I have not gotten to the bottom of why the mapping ID is missing when
maxNodes!=0.

This is presenting a workaround until we full understand the issue.

Fixes grafana#3164
simonswine added a commit to simonswine/pyroscope that referenced this issue Apr 10, 2024
```
❯ go test -v ./debug
=== RUN   TestReproducePanic3164
panic: runtime error: index out of range [0] with length 0

goroutine 139 [running]:
github.com/grafana/pyroscope/pkg/pprof.RewriteMappings(...)
        /Users/christian/git/github.com/grafana/pyroscope/pkg/pprof/merge.go:259
github.com/grafana/pyroscope/pkg/pprof.(*ProfileMerge).merge(0x14000656360, 0x140007f36d0, 0x70?)
        /Users/christian/git/github.com/grafana/pyroscope/pkg/pprof/merge.go:68 +0x5b0
github.com/grafana/pyroscope/pkg/pprof.(*ProfileMerge).Merge(...)
        /Users/christian/git/github.com/grafana/pyroscope/pkg/pprof/merge.go:26
github.com/grafana/pyroscope/pkg/phlaredb/symdb.(*Resolver).Pprof.func1(0x14001483040?, {{0x1400077c35c, 0x1, 0x1}, {0x1400077c370, 0x1, 0x1}, {0x0, 0x0, 0x0}})
        /Users/christian/git/github.com/grafana/pyroscope/pkg/phlaredb/symdb/resolver.go:248 +0x164
github.com/grafana/pyroscope/pkg/phlaredb/symdb.(*Resolver).withSymbols.func1()
        /Users/christian/git/github.com/grafana/pyroscope/pkg/phlaredb/symdb/resolver.go:263 +0x110
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /Users/christian/.golang/packages/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75 +0x58
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 103
        /Users/christian/.golang/packages/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:72 +0x98
FAIL    github.com/grafana/pyroscope/debug      4.985s
FAIL
```
simonswine added a commit to simonswine/pyroscope that referenced this issue Apr 10, 2024
```
❯ go test -v ./debug
=== RUN   TestReproducePanic3164
panic: runtime error: index out of range [0] with length 0

goroutine 139 [running]:
github.com/grafana/pyroscope/pkg/pprof.RewriteMappings(...)
        /Users/christian/git/github.com/grafana/pyroscope/pkg/pprof/merge.go:259
github.com/grafana/pyroscope/pkg/pprof.(*ProfileMerge).merge(0x14000656360, 0x140007f36d0, 0x70?)
        /Users/christian/git/github.com/grafana/pyroscope/pkg/pprof/merge.go:68 +0x5b0
github.com/grafana/pyroscope/pkg/pprof.(*ProfileMerge).Merge(...)
        /Users/christian/git/github.com/grafana/pyroscope/pkg/pprof/merge.go:26
github.com/grafana/pyroscope/pkg/phlaredb/symdb.(*Resolver).Pprof.func1(0x14001483040?, {{0x1400077c35c, 0x1, 0x1}, {0x1400077c370, 0x1, 0x1}, {0x0, 0x0, 0x0}})
        /Users/christian/git/github.com/grafana/pyroscope/pkg/phlaredb/symdb/resolver.go:248 +0x164
github.com/grafana/pyroscope/pkg/phlaredb/symdb.(*Resolver).withSymbols.func1()
        /Users/christian/git/github.com/grafana/pyroscope/pkg/phlaredb/symdb/resolver.go:263 +0x110
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /Users/christian/.golang/packages/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75 +0x58
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 103
        /Users/christian/.golang/packages/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:72 +0x98
FAIL    github.com/grafana/pyroscope/debug      4.985s
FAIL
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants