Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Thanos with Azure bucket produces many 404 errors #2356

Closed
yanush51 opened this issue Apr 1, 2020 · 10 comments
Closed

Thanos with Azure bucket produces many 404 errors #2356

yanush51 opened this issue Apr 1, 2020 · 10 comments
Labels
stale

Comments

@yanush51
Copy link

yanush51 commented Apr 1, 2020 •
edited
Loading

Thanos, Prometheus and Golang version used: thanos: version 0.11.0 (branch: HEAD, revision: 31cacc4), go version: go1.13.6

Object Storage Provider: AZURE

What happened: Seems to be working but many errors in logs, see below

What you expected to happen: Clear logs

How to reproduce it (as minimally and precisely as possible): Just general template for Azure bucket.

Thanos sidecar options:
sidecar --tsdb.path /var/lib/prometheus/ --objstore.config-file /etc/thanos/bucket.yml

Full logs to relevant components:

Logs

Apr 01 08:27:17 monitoring /usr/local/bin/thanos[61539]: 2020/04/01 08:27:17 ==> REQUEST/RESPONSE (Try=1/3.80412ms, OpTime=3.85952ms) -- REQUEST ERROR HEAD https://*.blob.core.windows.net/*/01E4MP3Q4F9WH2M5JC4AZS09DS/meta.json?timeout=61 Authorization: REDACTED User-Agent: [Thanos Azure-Storage/0.7 (go1.13.6; linux)] X-Ms-Client-Request-Id: [a41db3d0-40e3-48e9-73dd-3e90b9019f14] X-Ms-Version: [2018-11-09] x-ms-date: [Wed, 01 Apr 2020 08:27:17 GMT] -------------------------------------------------------------------------------- RESPONSE Status: 404 The specified blob does not exist. Date: [Wed, 01 Apr 2020 08:27:16 GMT] Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0] X-Ms-Error-Code: [BlobNotFound] X-Ms-Request-Id: [2f44f155-f01e-0022-2eff-07cbd1000000] X-Ms-Version: [2018-11-09] -------------------------------------------------------------------------------- ERROR: -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, /go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.8.0/azblob/zc_storage_error.go:42 ===== RESPONSE ERROR (ServiceCode=BlobNotFound) ===== Description=404 The specified blob does not exist., Details: (none) HEAD https://*.blob.core.windows.net/*/01E4MP3Q4F9WH2M5JC4AZS09DS/meta.json?timeout=61 Authorization: REDACTED User-Agent: [Thanos Azure-Storage/0.7 (go1.13.6; linux)] X-Ms-Client-Request-Id: [a41db3d0-40e3-48e9-73dd-3e90b9019f14] X-Ms-Date: [Wed, 01 Apr 2020 08:27:17 GMT] X-Ms-Version: [2018-11-09] -------------------------------------------------------------------------------- RESPONSE Status: 404 The specified blob does not exist. Date: [Wed, 01 Apr 2020 08:27:16 GMT] Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0] X-Ms-Error-Code: [BlobNotFound] X-Ms-Request-Id: [2f44f155-f01e-0022-2eff-07cbd1000000] X-Ms-Version: [2018-11-09] 

                                                             goroutine 100 [running]:
                                                             github.com/Azure/azure-storage-blob-go/azblob.stack(0xc0009bfdd0, 0xc0009a5800, 0xc0003bbcb0)
                                                                     /go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.8.0/azblob/zc_policy_request_log.go:146 +0x9d
                                                             github.com/Azure/azure-storage-blob-go/azblob.NewRequestLogPolicyFactory.func1.1(0x22bbac0, 0xc0009052c0, 0xc0009a5800, 0x10, 0x1b39ca0, 0x1, 0xc000bfcdc0)
                                                                     /go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.8.0/azblob/zc_policy_request_log.go:96 +0x650
                                                             github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do(0xc000b3e280, 0x22bbac0, 0xc0009052c0, 0xc0009a5800, 0xc000bfcea8, 0xc0007f4940, 0x3f, 0xc0008bcc00)
                                                                     /go/pkg/mod/github.com/!azure/azure-pipeline-go@v0.2.2/pipeline/core.go:43 +0x44
                                                             github.com/Azure/azure-storage-blob-go/azblob.(*SharedKeyCredential).New.func1(0x22bbac0, 0xc0009052c0, 0xc0009a5800, 0x332e320, 0x22bbac0, 0xc0009052c0, 0xc000b03720)
                                                                     /go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.8.0/azblob/zc_credential_shared_key.go:66 +0x2d2
                                                             github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do(0xc0009bfb90, 0x22bbac0, 0xc0009052c0, 0xc0009a5800, 0xc0009052c0, 0xc000b03720, 0xc000000001, 0x0)
                                                                     /go/pkg/mod/github.com/!azure/azure-pipeline-go@v0.2.2/pipeline/core.go:43 +0x44
                                                             github.com/Azure/azure-storage-blob-go/azblob.NewRetryPolicyFactory.func1.1(0x22bba40, 0xc000c37500, 0xc0009a5700, 0x10, 0x1b39ca0, 0x64492d747301, 0xc000bfcc60)
                                                                     /go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.8.0/azblob/zc_policy_retry.go:204 +0x66c
                                                             github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do(0xc000b3e2d0, 0x22bba40, 0xc000c37500, 0xc0009a5700, 0xc000bfcd18, 0x30, 0x28, 0x8)
                                                                     /go/pkg/mod/github.com/!azure/azure-pipeline-go@v0.2.2/pipeline/core.go:43 +0x44
                                                             github.com/Azure/azure-storage-blob-go/azblob.NewUniqueRequestIDPolicyFactory.func1.1(0x22bba40, 0xc000c37500, 0xc0009a5700, 0x10, 0x1b39ca0, 0x40ca01, 0xc000bfcc60)
                                                                     /go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.8.0/azblob/zc_policy_unique_request_id.go:19 +0xae
                                                             github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do(0xc000b6d440, 0x22bba40, 0xc000c37500, 0xc0009a5700, 0xc000bfcd00, 0x2a, 0xc0008bca80, 0xc000522e00)
                                                                     /go/pkg/mod/github.com/!azure/azure-pipeline-go@v0.2.2/pipeline/core.go:43 +0x44
                                                             github.com/Azure/azure-storage-blob-go/azblob.NewTelemetryPolicyFactory.func1.1(0x22bba40, 0xc000c37500, 0xc0009a5700, 0x1, 0x0, 0x1, 0xc0009c0978)
                                                                     /go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.8.0/azblob/zc_policy_telemetry.go:34 +0x15d
                                                             github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do(0xc0009bfbf0, 0x22bba40, 0xc000c37500, 0xc0009a5700, 0xc0009bfbf0, 0xc0009a5700, 0xc000522ed0, 0x40ca58)
                                                                     /go/pkg/mod/github.com/!azure/azure-pipeline-go@v0.2.2/pipeline/core.go:43 +0x44
                                                             github.com/Azure/azure-pipeline-go/pipeline.(*pipeline).Do(0xc000f38e40, 0x22bba40, 0xc000c37500, 0x2270980, 0xc000b11720, 0xc0009a5700, 0x1e, 0xc0007f4880, 0x31, 0x0)
                                                                     /go/pkg/mod/github.com/!azure/azure-pipeline-go@v0.2.2/pipeline/core.go:129 +0x81
                                                             github.com/Azure/azure-storage-blob-go/azblob.blobClient.GetProperties(0xc0008bcab0, 0x5, 0x0, 0x0, 0x0, 0xc0008bcab8, 0x1e, 0xc0007f4880, 0x31, 0x0, ...)
                                                                     /go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.8.0/azblob/zz_generated_blob.go:761 +0x3a5
                                                             github.com/Azure/azure-storage-blob-go/azblob.BlobURL.GetProperties(0xc0008bcab0, 0x5, 0x0, 0x0, 0x0, 0xc0008bcab8, 0x1e, 0xc0007f4880, 0x31, 0x0, ...)
                                                                     /go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.8.0/azblob/url_blob.go:113 +0x11f
                                                             github.com/thanos-io/thanos/pkg/objstore/azure.(*Bucket).Exists(0xc0009ab8c0, 0x22bba40, 0xc000c37500, 0xc0008bc780, 0x24, 0x0, 0x24, 0xc0005235b8)
                                                                     /app/pkg/objstore/azure/azure.go:256 +0x2fb
                                                             github.com/thanos-io/thanos/pkg/objstore.(*metricBucket).Exists(0xc000c6ec60, 0x22bba40, 0xc000c37500, 0xc0008bc780, 0x24, 0x1a, 0x0, 0x0)
                                                                     /app/pkg/objstore/objstore.go:304 +0x9e
                                                             github.com/thanos-io/thanos/pkg/shipper.(*Shipper).Sync.func1(0xc0009ab340, 0x0, 0x0)
                                                                     /app/pkg/shipper/shipper.go:297 +0x42c
                                                             github.com/thanos-io/thanos/pkg/shipper.(*Shipper).iterBlockMetas(0xc000904720, 0xc000523b90, 0xc000523a28, 0xc0002963e8)
                                                                     /app/pkg/shipper/shipper.go:422 +0x1f4
                                                             github.com/thanos-io/thanos/pkg/shipper.(*Shipper).Sync(0xc000904720, 0x22bba40, 0xc000c37500, 0x0, 0x5, 0x1)
                                                                     /app/pkg/shipper/shipper.go:282 +0x35d
                                                             main.runSidecar.func10.2(0xc000523d38, 0xc000523cf0)
                                                                     /app/cmd/thanos/sidecar.go:338 +0x74
                                                             github.com/thanos-io/thanos/pkg/runutil.Repeat(0x6fc23ac00, 0xc000433680, 0xc000523f50, 0x0, 0x0)
                                                                     /app/pkg/runutil/runutil.go:72 +0x91
                                                             main.runSidecar.func10(0x0, 0x0)
                                                                     /app/cmd/thanos/sidecar.go:337 +0x427
                                                             github.com/oklog/run.(*Group).Run.func1(0xc000410ba0, 0xc00015b3b0, 0xc000c6cc40)
                                                                     /go/pkg/mod/github.com/oklog/run@v1.0.0/group.go:38 +0x27
                                                             created by github.com/oklog/run.(*Group).Run
                                                                     /go/pkg/mod/github.com/oklog/run@v1.0.0/group.go:37 +0xbb

@bwplotka
Copy link
Member

bwplotka commented Apr 1, 2020

Thanks.

That does not look like sidecar logs 🤔

@yanush51
Copy link
Author

yanush51 commented Apr 1, 2020

Hi, @bwplotka,
I found similar errors in store
Apr 1 07:00:51 thanos-dev-01 /usr/local/bin/thanos[19138]: 2020/04/01 07:00:51 ==> REQUEST/RESPONSE (Try=1/2.578804ms, OpTime=2.611504ms) -- REQUEST ERROR#012 HEAD https://*.blob.core.windows.net/*/01E4T8MBV2T3BDSG3KJG4GMGYD/index.cache.json?timeout=61#012
But they appears rarely.

@GiedriusS
Copy link
Member

Hi, @bwplotka,
I found similar errors in store
Apr 1 07:00:51 thanos-dev-01 /usr/local/bin/thanos[19138]: 2020/04/01 07:00:51 ==> REQUEST/RESPONSE (Try=1/2.578804ms, OpTime=2.611504ms) -- REQUEST ERROR#012 HEAD https://*.blob.core.windows.net/*/01E4T8MBV2T3BDSG3KJG4GMGYD/index.cache.json?timeout=61#012
But they appears rarely.

Does it coincide with when Thanos Compact deletes blocks from the remote object storage? Care to try the master version to see if this is reproducible?

@yanush51
Copy link
Author

yanush51 commented Apr 2, 2020

Hi @GiedriusS,
I haven't run Thanos Compact yet.
I'll try the latest master.

@yanush51
Copy link
Author

yanush51 commented Apr 2, 2020 •
edited
Loading

Problem persists

$ thanos --version
thanos, version 0.12.0-dev (branch: master, revision: fc754db9075dda3e3278f5aaf528f3b822e2946c)
  build user:       circleci@accce8bf3a6c
  build date:       20200401-15:26:53
  go version:       go1.13.1

@GiedriusS
Copy link
Member

Problem persists

$ thanos --version
thanos, version 0.12.0-dev (branch: master, revision: fc754db9075dda3e3278f5aaf528f3b822e2946c)
  build user:       circleci@accce8bf3a6c
  build date:       20200401-15:26:53
  go version:       go1.13.1

You ran that on both Thanos Store and Thanos Compact, right?

@yanush51
Copy link
Author

yanush51 commented Apr 2, 2020

Yes - the same versions for all components. I don't run Compact for now. Just sidecar, query and store.
I've even dropped container at Azure.

@stale
Copy link

stale bot commented May 2, 2020

Hello 👋 Looks like there was no activity on this issue for last 30 days.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity for next week, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

@stale stale bot added the stale label May 2, 2020
@stale
Copy link

stale bot commented May 9, 2020

Closing for now as promised, let us know if you need this to be reopened! 🤗

@stale stale bot closed this as completed May 9, 2020
@GiedriusS
Copy link
Member

Now 404 errors happen on checking for the deletion marker. Replaced by #2565.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@GiedriusS @bwplotka @yanush51