Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

panic on v0.0.76 #562

Closed
Vladikamira opened this issue Mar 16, 2023 · 3 comments · Fixed by #563
Closed

panic on v0.0.76 #562

Vladikamira opened this issue Mar 16, 2023 · 3 comments · Fixed by #563

Comments

@Vladikamira
Copy link
Contributor

after an update to v0.0.76 we're facing such panic quite often, which leads to the Pod crash

{"level":"info","msg":"promxy starting with HTTP...","time":"2023-03-16T12:20:25Z"}
panic: runtime error: index out of range [3] with length 3

goroutine 85351 [running]:
github.com/prometheus/prometheus/promql.findPathRange({0xc002562a80, 0x3, 0x24e9240?}, {0xc0013d2280?, 0x1, 0xc001347d78?})
	/go/src/github.com/jacksontj/promxy/vendor/github.com/prometheus/prometheus/promql/engine_extra.go:18 +0x19b
github.com/prometheus/prometheus/promql.(*Engine).findMinMaxTime.func1({0x38ed608?, 0xc00168ec60?}, {0xc002562a80?, 0x3, 0x3})
	/go/src/github.com/jacksontj/promxy/vendor/github.com/prometheus/prometheus/promql/engine.go:768 +0x11a
github.com/prometheus/prometheus/promql/parser.inspector.Visit(0xc000d88b40, {0x38ed608?, 0xc00168ec60?}, {0xc002562a80?, 0xc001347e70?, 0x4151ba?})
	/go/src/github.com/jacksontj/promxy/vendor/github.com/prometheus/prometheus/promql/parser/ast.go:383 +0x43
github.com/prometheus/prometheus/promql/parser.Walk({0x3900f00, 0xc00011a008}, {0x38e45a0, 0xc000d88b40}, 0xc001f696d0, {0x38ed608, 0xc00168ec60}, {0xc002562a80, 0x3, 0x3}, ...)
	/go/src/github.com/jacksontj/promxy/vendor/github.com/prometheus/prometheus/promql/parser/ast.go:330 +0x209
github.com/prometheus/prometheus/promql/parser.Walk.func1(0x0, {0x38ed608, 0xc00168ec60})
	/go/src/github.com/jacksontj/promxy/vendor/github.com/prometheus/prometheus/promql/parser/ast.go:344 +0x225
created by github.com/prometheus/prometheus/promql/parser.Walk
	/go/src/github.com/jacksontj/promxy/vendor/github.com/prometheus/prometheus/promql/parser/ast.go:342 +0x405

We're having such panic on every Pod in the Deployment, so it would be easy to know if it's fixed.
there are around 400 requests per second on this Promxy instance, fyi

@jacksontj
Copy link
Owner

Thanks for the report, I'm looking into it but unable to trivially reproduce the issue (I'm still looking). If you could provide an example query to reproduce that would significantly speed up a fix, otherwise I'll continue digging in to see if I can find a repro case.

jacksontj added a commit that referenced this issue Mar 16, 2023
jacksontj added a commit that referenced this issue Mar 16, 2023
jacksontj added a commit that referenced this issue Mar 16, 2023
@jacksontj
Copy link
Owner

Alright, I was able to repro the issue and have a fix pending in #563

jacksontj added a commit that referenced this issue Mar 16, 2023
jacksontj added a commit that referenced this issue Mar 16, 2023
@Vladikamira
Copy link
Contributor Author

Thanks! It works 🎉 no panic at all 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants