Store: consumes lots of memory at startup and loop restart cause OOM

[chalut01]

Thanos Version v0.32.0
Everything is set up the same way, only the version is different. Are there others similar to me?

[rgarcia89]

I have experienced exactly the same issue.

Closed my ticket as this one covers the same issue #6644

[pahaeanx]

Same here. Can't get it to start up with 6GB of RAM when it used to run with ~3GB.

[yeya24]

I can only think about #6509 that might change the startup memory usage of store gateway. But ideally this change should improve the mem usage.

Do you enable lazy index header? Can you share the config of store gateway?

[pahaeanx]

I can only think about #6509 that might change the startup memory usage of store gateway. But ideally this change should improve the mem usage.

Do you enable lazy index header? Can you share the config of store gateway?

No, I use a pretty vanilla config I'd say. Failing config (formatted for readability):

thanos store --max-time=-1w --grpc-address=localhost:15000 --http-address=localhost:15001 \ 
--data-dir=/var/lib/thanos-cache/store  --objstore.config-file=/etc/thanos/s3.yml \
--grpc-server-tls-cert=/etc/thanos/thanos.cer --grpc-server-tls-key=/etc/thanos/thanos.key \
--sync-block-duration=30m

This is currently running with 0.30.2 and fails with 0.32.0.

[rgarcia89]

@yeya24 I deploy the thanos store using kube-thanos. It starts with the following args in the manifest. Currently on v0.31.0

        - store
        - --log.level=info
        - --log.format=logfmt
        - --data-dir=/var/thanos/store
        - --grpc-address=0.0.0.0:10901
        - --http-address=0.0.0.0:10902
        - --objstore.config=$(OBJSTORE_CONFIG)
        - --ignore-deletion-marks-delay=24h

[antikilahdjs]

I have the same issue and upgraded to v31, all the other components works perfectly in 0.32 version but the store consume 1tb

[GiedriusS]

Maybe it would be possible for you to take a pprof memory profile during bootup of Thanos Store and share it here? Thanos Store exposes a pprof endpoint on /debug/pprof. 🤔

[bboysoulcn]

the same as you

[chalut01]

@GiedriusS I'm sorry, but I can't actually try out and get pprof on production environment.
However, In my development, It have a small data and everything works well with v0.32.0.

Can someone share a pprof?

[rgarcia89]

Maybe it would be possible for you to take a pprof memory profile during bootup of Thanos Store and share it here? Thanos Store exposes a pprof endpoint on /debug/pprof. 🤔

@GiedriusS I have thanos running as container in kubernetes using kube-thanos. How can I find this endpoint?

[yeya24]

@rgarcia89 Can you port forward one of your store gateway pod using its http port, for example it is 8080.

curl http://localhost:8080/debug/pprof/heap > heap.pprof

You can get the heap profile by running the command above. Make sure to do it when store gateway starts...

[rgarcia89]

@yeya24 here you go - including the heap of running 0.31.0 thanos store vs the heap of a just started 0.32.0 thanos store which then crashed

heap-pprof.zip

[MichaHoffmann]

Oh thats the cuckoo filter completely.

[MichaHoffmann]

Do you have some figures around your cardinality?

[rgarcia89]

@MichaHoffmann anything specific you are looking for? Thanos is currently managing a time series of 1792044 entries

[MichaHoffmann]

This filter should scale ( theoretically ) with the amount of label names you have. But for 128gb it would need like 100s of millions ( again theoretically, maybe there is a bug somewhere )

[rgarcia89]

Good question. I can see everything running smooth with v0.31.0 and also haven't seen any kind of alerts or issues in regards of high cardinality

[MichaHoffmann]

Good question. I can see everything running smooth with v0.31.0 and also haven't seen any kind of alerts or issues in regards of high cardinality

That cuckoo filter was introduced in 0.32.0 so that makes sense

[rgarcia89]

Maybe there is something counting wrong in that filter. Otherwise I am not sure how 128gb can be justified

[saswatamcode]

#6669 should address this

[GiedriusS]

How does it look like with v0.32.1?

[rgarcia89]

Very good. Issue is gone on my clusters

[pahaeanx]

Unfortunately still crashing with 0.32.1. Can't do any more digging today so all I can offer for now is the crash log. But it seems like it OOM crashed again. I can try and scale the VM tomorrow so I can maybe get it to start up.

crash.log.tar.gz

[farodin91]

It looks like store queries around 50% more mem as before, but no escalation.

[MichaHoffmann]

thast not a oom crash!

Aug 28 15:38:40 440019-prod-observer01 thanos[75687]: fatal error: ts=2023-08-28T13:38:40.784103104Z caller=bucket.go:688 level=info msg="loaded new block" elapsed=261.655268ms id=01H8GGHPJEQJ5XWESMV004DW8J
Aug 28 15:38:40 440019-prod-observer01 thanos[75687]: concurrent map iteration and map write

[mateuszdrab]

Works for me now, I can check memory metrics later.

[MichaHoffmann]

#6675

[antikilahdjs]

Works perfectly and thank you guys to work hard to delivery the best to us

[MichaHoffmann]

@chalut01 can we close the issue?

[chalut01]

Fixed in v0.32.2 !!
Thank you everyone for your hard work.