-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Existing upload confuses shipper #934
Comments
Dived into this bit more.
|
There is another issue with this -> if you restart sidecar in the middle of process you will have overlap issue as well, but that's another story. |
I still see this on v0.5.0 + prometheus 2.10.0
uploading 300d of old data 💭 happened on another prometheus instance with 40d too... |
@caarlos0 what exactly you want to accomplish? Are you running sidecar with any special flag? |
I'm running with |
This is quite manual feature and single time. To make sure it is safe, it errors out instead of assumming something. The best bet is to try understand the error and mititgate. I don't know you case but by first glance it looks like you have blocks for exactly the same timestamp but diffferent ULID. Are you sure that:
|
yes
it was running at some point, I stopped it but errors continued... |
what happened - I think, is that I started it without the flag, stopped, added the flag and started again maybe it got lost in there? |
Thanos, Prometheus and Golang version used
What happened
Shipper didn't write out
thanos.shipper.json
due to failure to upload a compacted block.What you expected to happen
Shipper should update
thanos.shipper.json
every time it uploads a block, not just when it completes a batch sync.How to reproduce it (as minimally and precisely as possible):
Bucket storage upload is canceled in the middle of a large compacted block upload.
Full logs to relevant components
The text was updated successfully, but these errors were encountered: