Thanos Store: missing block.json

[povilasv]

Thanos, Prometheus and Golang version used

What happened

level=info ts=2019-03-26T13:08:12.686269636Z caller=flags.go:75 msg="gossip is disabled"
level=info ts=2019-03-26T13:08:12.686347675Z caller=factory.go:39 msg="loading bucket configuration"
level=warn ts=2019-03-26T13:08:54.466783868Z caller=bucket.go:251 msg="loading block failed" id=01D628X9G41V4JD94P1E1A1M0Z err="new bucket block: load meta: download meta.json: get file: The specified key does not exist."
level=info ts=2019-03-26T13:10:02.97778456Z caller=main.go:256 msg="disabled TLS, key and cert must be set to enable"
level=info ts=2019-03-26T13:10:02.977866938Z caller=store.go:197 msg="starting store node"
level=info ts=2019-03-26T13:10:02.977926563Z caller=store.go:166 msg="Listening for StoreAPI gRPC" address=0.0.0.0:10901
level=info ts=2019-03-26T13:10:02.977971048Z caller=main.go:308 msg="Listening for metrics" address=0.0.0.0:10902
level=warn ts=2019-03-26T13:10:03.004723927Z caller=bucket.go:251 msg="loading block failed" id=01D628X9G41V4JD94P1E1A1M0Z err="new bucket block: load meta: download meta.json: get file: The specified key does not exist."
level=warn ts=2019-03-26T13:13:03.095212672Z caller=bucket.go:251 msg="loading block failed" id=01D628X9G41V4JD94P1E1A1M0Z err="new bucket block: load meta: download meta.json: get file: The specified key does not exist."
level=warn ts=2019-03-26T13:16:03.066464449Z caller=bucket.go:251 msg="loading block failed" id=01D628X9G41V4JD94P1E1A1M0Z err="new bucket block: load meta: download meta.json: get file: The specified key does not exist."
level=warn ts=2019-03-26T13:19:03.069252878Z caller=bucket.go:251 msg="loading block failed" id=01D628X9G41V4JD94P1E1A1M0Z err="new bucket block: load meta: download meta.json: get file: The specified key does not exist."
level=warn ts=2019-03-26T13:22:03.069760477Z caller=bucket.go:251 msg="loading block failed" id=01D628X9G41V4JD94P1E1A1M0Z err="new bucket block: load meta: download meta.json: get file: The specified key does not exist."
level=warn ts=2019-03-26T13:25:03.013610292Z caller=bucket.go:251 msg="loading block failed" id=01D628X9G41V4JD94P1E1A1M0Z err="new bucket block: load meta: download meta.json: get file: The specified key does not exist."
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x57fe89]

goroutine 6745 [running]:
github.com/improbable-eng/thanos/vendor/github.com/prometheus/tsdb/index.(*intersectPostings).Seek(0xc43a20e870, 0x183f7, 0xc43a20e870)
        /go/src/github.com/improbable-eng/thanos/vendor/github.com/prometheus/tsdb/index/postings.go:348 +0x29
github.com/improbable-eng/thanos/vendor/github.com/prometheus/tsdb/index.(*intersectPostings).doNext(0xc43a20e8a0, 0x183f7, 0x1051d40)
        /go/src/github.com/improbable-eng/thanos/vendor/github.com/prometheus/tsdb/index/postings.go:323 +0x47
github.com/improbable-eng/thanos/vendor/github.com/prometheus/tsdb/index.(*intersectPostings).Next(0xc43a20e8a0, 0x57fbac)
        /go/src/github.com/improbable-eng/thanos/vendor/github.com/prometheus/tsdb/index/postings.go:344 +0x60
github.com/improbable-eng/thanos/vendor/github.com/prometheus/tsdb/index.ExpandPostings(0x1206ce0, 0xc43a20e8a0, 0x4, 0x1206ce0, 0xc43a20e8a0, 0xc4746e6a00, 0x2)
        /go/src/github.com/improbable-eng/thanos/vendor/github.com/prometheus/tsdb/index/postings.go:250 +0x57
github.com/improbable-eng/thanos/pkg/store.(*bucketIndexReader).ExpandedPostings(0xc463b83720, 0xc46d1bf7a0, 0x3, 0x3, 0x12062a0, 0x0, 0xc449ab79e0, 0x12062a0, 0xc449ab79e0)
        /go/src/github.com/improbable-eng/thanos/pkg/store/bucket.go:1202 +0x2e3
github.com/improbable-eng/thanos/pkg/store.(*BucketStore).blockSeries(0xc4203dc2d0, 0x1206220, 0xc4522f2680, 0xcd80906c78996901, 0x91d9b69d8a2a3b95, 0xc439c526c0, 0xc463b83720, 0xc45902c120, 0xc46d1bf7a0, 0x3, ...)
        /go/src/github.com/improbable-eng/thanos/pkg/store/bucket.go:484 +0x8c
github.com/improbable-eng/thanos/pkg/store.(*BucketStore).Series.func1(0xc4472920a0, 0x1206220)
        /go/src/github.com/improbable-eng/thanos/pkg/store/bucket.go:706 +0xe6
github.com/improbable-eng/thanos/vendor/github.com/oklog/run.(*Group).Run.func1(0xc46205dda0, 0xc460acc0e0, 0xc42d29f1b0)
        /go/src/github.com/improbable-eng/thanos/vendor/github.com/oklog/run/group.go:38 +0x27
created by github.com/improbable-eng/thanos/vendor/github.com/oklog/run.(*Group).Run
        /go/src/github.com/improbable-eng/thanos/vendor/github.com/oklog/run/group.go:37 +0xa8

What you expected to happen

How to reproduce it (as minimally and precisely as possible):

Full logs to relevant components

Anything else we need to know

[povilasv]

thanos bucket ls -o wide --objstore.config-file=$UBUNTU_HOME/thanos/dev-telecom.yaml

01D61JF27Z1FYA2M2GBPV5ZHH6 -- 2019-03-15 20:00 - 2019-03-15 22:00 Diff: 2h0m0s, Compaction: 1, Downsample: 0, Source: sidecar          
01D61JF27Z465B95CGYVZPAVY4 -- 2019-03-15 20:00 - 2019-03-15 22:00 Diff: 2h0m0s, Compaction: 1, Downsample: 0, Source: sidecar          
01D61SASG20M0YZ0GTH9FD43SN -- 2019-03-15 22:00 - 2019-03-16 00:00 Diff: 2h0m0s, Compaction: 1, Downsample: 0, Source: sidecar          
01D61SASG2TNXMGE175MTDM38W -- 2019-03-15 22:00 - 2019-03-16 00:00 Diff: 2h0m0s, Compaction: 1, Downsample: 0, Source: sidecar          
01D6206GMH5N4GZ0EST5DSQP1Q -- 2019-03-16 00:00 - 2019-03-16 02:00 Diff: 2h0m0s, Compaction: 1, Downsample: 0, Source: sidecar          
01D6206GQZKC4CS6XXHAXD63V4 -- 2019-03-16 00:00 - 2019-03-16 02:00 Diff: 2h0m0s, Compaction: 1, Downsample: 0, Source: sidecar          
01D62727WN0JA17FHMN661NAG3 -- 2019-03-16 02:00 - 2019-03-16 04:00 Diff: 2h0m0s, Compaction: 1, Downsample: 0, Source: sidecar 
01D6272801J0SK0JCX1NV0Y6GW -- 2019-03-16 02:00 - 2019-03-16 04:00 Diff: 2h0m0s, Compaction: 1, Downsample: 0, Source: sidecar
bucket ls command failed: meta.json bkt get for 01D628X9G41V4JD94P1E1A1M0Z: The specified key does not exist.

Thanos sidecar doesn't have any errors. So this might be compaction?

[povilasv]

aws s3 cp --recursive s3://prometheus-storage-dev/01D628X9G41V4JD94P1E1A1M0Z .                           
download: s3://prometheus-storage-dev/01D628X9G41V4JD94P1E1A1M0Z/index to ./index  

that dir only contained index file

[rtkgjacobs]

Seeing this also, but we're in the middle of compaction catching up nearly 2 weeks of backlog - in case that is a edge case, compaction has not completed working on a buckets contents vs other parts of thanos reading it?

[Alexvianet]

Same issue:
Thanos Store Version | 0.3.2

level=warn ts=2019-04-18T09:23:50.02252972Z caller=bucket.go:251 msg="loading block failed" id=01D8QVVW84M8M39J6YXT3SJR8T err="new bucket block: load meta: download meta.json: get file: The specified key does not exist."

[bwplotka]

Partial upload issue (cc @mjd95) - #621 (comment)

So I think it's duplicate.

[mjd95]

Yeah sounds like the same as #621

https://github.com/improbable-eng/thanos/pull/1053/files is one approach that should hopefully fix this.