Skip to content

Commit

Permalink
Make thanos bucket web refresh forever ignoring errors
Browse files Browse the repository at this point in the history
This is unfortunate since ideally the loop should handle errors that can be
retried and bubble up unexpected errors.

The bucket iterator fails with a remote read error while a new block is being
uploaded and over the last couple of days, this program crashed multiple times
with the same error. See logs for example. Ideally this patch should solve it.

```
level=error ts=2019-06-19T09:22:22.810872915Z caller=bucket.go:411 err="meta.json bkt get for 01DDQ6D62GFFHQKYCJ7TCHP307: The specified key does not exist." msg="Failed to downloaded block metadata"
level=error ts=2019-06-19T09:22:22.81573321Z caller=runutil.go:88 msg="function failed. Retrying in next tick" err="meta.json bkt get for 01DDQ6D62GFFHQKYCJ7TCHP307: The specified key does not exist."
level=error ts=2019-06-19T09:23:09.970878178Z caller=main.go:194 msg="running command failed" err="meta.json bkt get for 01DDQ6D62GFFHQKYCJ7TCHP307: The specified key does not exist."
```
  • Loading branch information
jaseemabid committed Jun 19, 2019
1 parent ed0590d commit 566ac2b
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion cmd/thanos/bucket.go
Original file line number Diff line number Diff line change
Expand Up @@ -359,7 +359,7 @@ func refresh(ctx context.Context, logger log.Logger, bucketUI *ui.Bucket, durati

defer runutil.CloseWithLogOnErr(logger, bkt, "bucket client")

return runutil.Repeat(duration, ctx.Done(), func() error {
runutil.Forever(logger, duration, ctx.Done(), func() error {
iterCtx, iterCancel := context.WithTimeout(ctx, 5*time.Minute)
defer iterCancel()

Expand Down

0 comments on commit 566ac2b

Please sign in to comment.