Query Frontend: consumes more memory (OOM)

[chalut01]

Thanos Version v0.32.0
Everything is set up the same way, only the version is different.
Before this, I realized that Store are consumes a lot of memory too. #6643
I've decided to rollback all components to version v0.31.0. I'm hoping that this pprof can be useful.

thanos-query-frontend-cpu.pb.gz
thanos-query-frontend-heap.pb.gz

[mhoffm-aiven]

Can you provide a pprof dump and enable active query tracking please?

Edit: i'm a potato and didnt read the initial message properly.

[chalut01]

Fixed in v0.32.2 !!
Thank you everyone for your hard work.

Edit: I'm sorry, it still OOM :(

[chalut01]

Update.

[MichaHoffmann]

Can you enable active query tracking please? Are you using thanos query engine?

[yeya24]

Is vertical sharding enabled?
The only thing I can think about is that we vertical shard more and maybe it caused that heap profile

[chalut01]

My configuration.

thanos query-frontend
--http-address=0.0.0.0:9090
--labels.default-time-range=15m
--labels.max-query-parallelism=50
--labels.max-retries-per-request=2
--labels.response-cache-config-file=/thanos-conf/thanos-query-frontend-redis.yaml
--labels.response-cache-max-freshness=3m
--labels.split-interval=3h
--query-frontend.compress-responses
--query-frontend.downstream-url=http://thanos-query.thanos
--query-frontend.forward-header=Authorization
--query-frontend.log-queries-longer-than=15s
--query-range.max-query-parallelism=50
--query-range.max-retries-per-request=2
--query-frontend.downstream-tripper-config-file=/thanos-conf/thanos-query-frontend-tripper.yaml
--query-range.response-cache-config-file=/thanos-conf/thanos-query-frontend-redis.yaml
--query-range.split-interval=3h
--query-range.partial-response
--query-range.request-downsampled
--log.level=info

[chalut01]

Fixed.
Upgrade to 0.32.5