-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Compactor: OOM since 0.32.3, and unsupported type of io.Reader: io.nopCloserWriterTo
#6748
Comments
I am wondering if this is the same error we found in objstore. Could you take a heap profile during compactor start up time? @saswatamcode We need to release a new version for the two fixes anyway I guess |
@yeya24 yes we do! I'm waiting for a few more bug reports before I do it, so that we can tackle all of them in on go in 0.32.4. |
@yeya24 Please let me know if this captures what you're after: |
Same here, v0.32.3 sidecars also started crashing soon after the compactor. |
Thanks for reporting @gebn! Will release v0.32.4 soon. Can you try out a Thanos main image and check if the issue persists? |
To save compiling, are binaries published anywhere? I'm afraid I'm not using containers. The builds don't seem to have (visible) artefacts. |
Sorry @gebn, we only publish nightly images I'm afraid. Would you be able to compile a binary from main and test it out? |
Was running into the same issue, using main-2023-09-26-20d2900 fixes it. |
@Tolsto that's great! Will try to release a patch version by tomorrow! |
v0.32.4 is now available with these fixes: https://github.com/thanos-io/thanos/releases/tag/v0.32.4 |
I think this could be closed. WDYT @gebn ? |
No issues since upgrading! Thanks all |
Thanos, Prometheus and Golang version used:
Object Storage Provider: AWS S3
What happened: OOM kill a few seconds after startup:
What you expected to happen: Compact to run successfully, like it did with v0.32.2.
Full logs to relevant components:
Anything else we need to know: VM has 1Gi memory (t4g.micro), using Debian 12.1.
The text was updated successfully, but these errors were encountered: