-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
objstore: Azure: unusually high number of GetBlobProperties calls with ClientOtherError/404 responses #6412
Comments
i realize the thanos version is quite old, and the i can attempt to setup thanos locally and see if I can reproduce the issue on 0.29.0+ but no guarantees on when I can get back with the results. |
Hello, not using "hierarchical namespace" here. We are not considering this as an error. Also this issue should be moved to https://github.com/thanos-io/objstore |
@bck01215 any comment on whether the Azure storage account has "Hierarchical namespace" enabled? |
That is a difference. I do not have that enabled |
Hi all, @bck01215 : we have the same issue with currently 3.8M+ calls per month on the Azure Storage. "never caused a noticeable impact to my knowledge" Can you look at your storage cost to see if you have the same issue? Best regards, |
@Tiduster Unfortunately, I don't have access to our billing info. We're in the process of migrating to an on prem s3 server. Before this I tried to increase the timeouts in the http configs, but that did not resolve the issue |
After reaching out to the billing team I confirmed the failed requests are affecting our billing. I was also able to confirm the source is coming from
|
It seems like #2565 explored the 404's related to the |
Thank you very much, @bck01215, for verifying your cost figures. We have experienced an exponential cost increase over the past few months on our end. Here's what we've done:
|
Thanos, Prometheus and Golang version used: thanos=0.26.0, quay.io images
Object Storage Provider: Azure
What happened:
ever-increasing number of
GetBlobProperties
call month-over-month, with most of them (95%+) resulting inClientOtherError
/404 responsesWhat you expected to happen:
a fairly lower number of
GetBlobProperties
calls..How to reproduce it (as minimally and precisely as possible):
found partially uploaded block
anddeleted partially uploaded block
Full logs to relevant components:
Anything else we need to know:
the notes from our internal investigation,
{ulid}/chunks/
and{ulid}/
remain even after deletion of all files within{ulid}/
BestEffortCleanAbortedPartialUploads
on the next run due to a missing{ulid}/meta.json
BestEffortCleanAbortedPartialUploads
is unable to delete these directories sincedeleteDirRec
invokes(b *Bucket) Iter
and does not attempt deleting the directory itselfThe text was updated successfully, but these errors were encountered: