query: panic on default resolution in subqueries

[xjewer]

Thanos, Prometheus and Golang version used

/ # thanos --version
thanos, version 0.3.2 (branch: HEAD, revision: 4b7320c0e45e3f48a437bd19294f569785bafb02)
  build user:       circleci@4ea007a18ffa
  build date:       20190304-17:16:56
  go version:       go1.10.8

What happened

thanos query panics on subqueries like sum_over_time(rate(instance:container_cpu_time_seconds_total{app="test"}[2m])[60m:]) with default resolution

it works ok with specifying resolution explicitly:
sum_over_time(rate(instance:container_cpu_time_seconds_total{app="test"}[2m])[60m:10m])

What you expected to happen

Returning valid result

Full logs to relevant components

level=error ts=2019-04-10T14:42:00.93989884Z caller=engine.go:690 msg="runtime panic in parser" err="runtime error: integer divide by zero" stacktrace="goroutine 96295832 [running]:\ngithub.com/improbable-eng/thanos/vendor/github.com/prometheus/prometheus/promql.(*evaluator).recover(0xc421f47180, 0xc437edf1f0)\n\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/prometheus/prometheus/promql/engine.go:688 +0xe1\npanic(0xf80c20, 0x18daf40)\n\t/usr/local/go/src/runtime/panic.go:502 +0x229\ngithub.com/improbable-eng/thanos/vendor/github.com/prometheus/prometheus/promql.(*evaluator).eval(0xc421f47180, 0x1202820, 0xc48c93b4d0, 0xc42c184420, 0x0)\n\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/prometheus/prometheus/promql/engine.go:1159 +0x31cc\ngithub.com/improbable-eng/thanos/vendor/github.com/prometheus/prometheus/promql.(*evaluator).evalSubquery(0xc421f47180, 0xc48c93b4d0, 0x111e4c7)\n\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/prometheus/prometheus/promql/engine.go:882 +0x5c\ngithub.com/improbable-eng/thanos/vendor/github.com/prometheus/prometheus/promql.(*evaluator).eval(0xc421f47180, 0x12026e0, 0xc433fb3320, 0xc437edf1f0, 0xc421f47180)\n\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/prometheus/prometheus/promql/engine.go:945 +0x12af\ngithub.com/improbable-eng/thanos/vendor/github.com/prometheus/prometheus/promql.(*evaluator).Eval(0xc421f47180, 0x12026e0, 0xc433fb3320, 0x0, 0x0, 0x0, 0x0)\n\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/prometheus/prometheus/promql/engine.go:699 +0x80\ngithub.com/improbable-eng/thanos/vendor/github.com/prometheus/prometheus/promql.(*Engine).execEvalStmt(0xc420730420, 0x1204e20, 0xc48c93b650, 0xc4222e9ce0, 0xc427d67ef0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/prometheus/prometheus/promql/engine.go:436 +0x487\ngithub.com/improbable-eng/thanos/vendor/github.com/prometheus/prometheus/promql.(*Engine).exec(0xc420730420, 0x1204e20, 0xc48c93b650, 0xc4222e9ce0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/prometheus/prometheus/promql/engine.go:390 +0x5a7\ngithub.com/improbable-eng/thanos/vendor/github.com/prometheus/prometheus/promql.(*query).Exec(0xc4222e9ce0, 0x1204e20, 0xc48c93b230, 0xc421df4090)\n\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/prometheus/prometheus/promql/engine.go:178 +0x94\ngithub.com/improbable-eng/thanos/pkg/query/api.(*API).query(0xc4207faa00, 0xc4289c2c00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\n\t/go/src/github.com/improbable-eng/thanos/pkg/query/api/v1.go:295 +0x4e2\ngithub.com/improbable-eng/thanos/pkg/query/api.(*API).(github.com/improbable-eng/thanos/pkg/query/api.query)-fm(0xc4289c2c00, 0xc4222e9a40, 0x0, 0xc42ce34dd8, 0xc42ce34dd0, 0x11, 0xc423ffb850)\n\t/go/src/github.com/improbable-eng/thanos/pkg/query/api/v1.go:170 +0x34\ngithub.com/improbable-eng/thanos/pkg/query/api.(*API).Register.func1.1(0x1203960, 0xc4222e9a40, 0xc4289c2c00)\n\t/go/src/github.com/improbable-eng/thanos/pkg/query/api/v1.go:157 +0x55\nnet/http.HandlerFunc.ServeHTTP(0xc4207fc5f0, 0x1203960, 0xc4222e9a40, 0xc4289c2c00)\n\t/usr/local/go/src/net/http/server.go:1947 +0x44\ngithub.com/improbable-eng/thanos/vendor/github.com/NYTimes/gziphandler.GzipHandlerWithOpts.func1.1(0x1202360, 0xc422f505c8, 0xc4289c2c00)\n\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/NYTimes/gziphandler/gzip.go:287 +0x1ae\nnet/http.HandlerFunc.ServeHTTP(0xc4203c6630, 0x1202360, 0xc422f505c8, 0xc4289c2c00)\n\t/usr/local/go/src/net/http/server.go:1947 +0x44\ngithub.com/improbable-eng/thanos/pkg/tracing.HTTPMiddleware.func1(0x1202360, 0xc422f505c8, 0xc4289c2b00)\n\t/go/src/github.com/improbable-eng/thanos/pkg/tracing/http.go:37 +0x577\nnet/http.HandlerFunc.ServeHTTP(0xc4207fad20, 0x1202360, 0xc422f505c8, 0xc4289c2b00)\n\t/usr/local/go/src/net/http/server.go:1947 +0x44\nnet/http.(Handler).ServeHTTP-fm(0x1202360, 0xc422f505c8, 0xc4289c2b00)\n\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/prometheus/client_golang/prometheus/http.go:162 +0x4d\ngithub.com/improbable-eng/thanos/vendor/github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x12036e0, 0xc4202eda40, 0xc4289c2b00)\n\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/prometheus/client_golang/prometheus/http.go:289 +0x229\ngithub.com/improbable-eng/thanos/vendor/github.com/prometheus/common/route.(*Router).handle.func1(0x12036e0, 0xc4202eda40, 0xc4269f6100, 0x0, 0x0, 0x0)\n\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/prometheus/common/route/route.go:60 +0x222\ngithub.com/improbable-eng/thanos/vendor/github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc420730480, 0x12036e0, 0xc4202eda40, 0xc4269f6100)\n\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/julienschmidt/httprouter/router.go:299 +0x6c1\ngithub.com/improbable-eng/thanos/vendor/github.com/prometheus/common/route.(*Router).ServeHTTP(0xc420385ce0, 0x12036e0, 0xc4202eda40, 0xc4269f6100)\n\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/prometheus/common/route/route.go:98 +0x4c\nnet/http.(*ServeMux).ServeHTTP(0xc4205dc6f0, 0x12036e0, 0xc4202eda40, 0xc4269f6100)\n\t/usr/local/go/src/net/http/server.go:2340 +0x130\nnet/http.serverHandler.ServeHTTP(0xc4202b2340, 0x12036e0, 0xc4202eda40, 0xc4269f6100)\n\t/usr/local/go/src/net/http/server.go:2697 +0xbc\nnet/http.(*conn).serve(0xc42049d0e0, 0x1204d60, 0xc4216c8340)\n\t/usr/local/go/src/net/http/server.go:1830 +0x651\ncreated by net/http.(*Server).Serve\n\t/usr/local/go/src/net/http/server.go:2798 +0x27b\n"

[bwplotka]

runtime error: integer divide by zero"

[bwplotka]

Does it work on Prometheus directly?

[bwplotka]

Maybe bug in PromQL version, that was fixed upstream but we are on older version? do you mind grep through older Prometheus issues for this?

[xjewer]

thanos doesn't set up default value https://github.com/prometheus/prometheus/blob/master/cmd/prometheus/main.go#L315

[bwplotka]

Awesome find, do you wanna put quick PR?

[xjewer]

will do