Merge release 0.18

CHANGELOG.md

@@ -27,33 +27,39 @@ We use _breaking :warning:_ to mark changes that are not backward compatible (re
 
 - [#3705](https://github.com/thanos-io/thanos/pull/3705) Store: Fix race condition leading to failing queries or possibly incorrect query results.
 
-## [v0.18.0](https://github.com/thanos-io/thanos/releases) - Release in progress
+## [v0.18.0](https://github.com/thanos-io/thanos/releases/tag/v0.18.0) - 2021.01.27
 
 ### Added
 
-- [#3579](https://github.com/thanos-io/thanos/pull/3579) Cache: Added inmemory cache for caching bucket.
-- [#3469](https://github.com/thanos-io/thanos/pull/3469) StoreAPI: Added `hints` field to `LabelNamesRequest` and `LabelValuesRequest`. Hints in an opaque data structure that can be used to carry additional information from the store and its content is implementation specific.
+- [#3380](https://github.com/thanos-io/thanos/pull/3380) Mixin: Add block deletion panels for compactor dashboards.
+- [#3568](https://github.com/thanos-io/thanos/pull/3568) Store: Optimized inject label stage of index lookup.
+- [#3566](https://github.com/thanos-io/thanos/pull/3566) StoreAPI: Support label matchers in labels API.
+- [#3531](https://github.com/thanos-io/thanos/pull/3531) Store: Optimized common cases for time selecting smaller amount of series by avoiding looking up symbols.
+- [#3469](https://github.com/thanos-io/thanos/pull/3469) StoreAPI: Added `hints` field to `LabelNamesRequest` and `LabelValuesRequest`. Hints are an opaque data structure that can be used to carry additional information from the store and its content is implementation-specific.
 - [#3421](https://github.com/thanos-io/thanos/pull/3421) Tools: Added `thanos tools bucket rewrite` command allowing to delete series from given block.
-- [#3509](https://github.com/thanos-io/thanos/pull/3509) Store: Added touch series limit
-- [#3388](https://github.com/thanos-io/thanos/pull/3378) Tools: Bucket replicator now can specify block IDs to copy.
-- [#3121](https://github.com/thanos-io/thanos/pull/3121) Receive: Added `--receive.hashrings` alternative to `receive.hashrings-file` flag (lower priority). Content of JSON file that contains the hashring configuration.
+- [#3509](https://github.com/thanos-io/thanos/pull/3509) Store: Added a CLI flag to limit the number of series that are touched.
+- [#3444](https://github.com/thanos-io/thanos/pull/3444) Query Frontend: Make POST request to downstream URL for labels and series API endpoints.
+- [#3388](https://github.com/thanos-io/thanos/pull/3388) Tools: Bucket replicator now can specify block IDs to copy.
+- [#3385](https://github.com/thanos-io/thanos/pull/3385) Tools: Bucket prints extra statistics for block index with debug log-level.
+- [#3121](https://github.com/thanos-io/thanos/pull/3121) Receive: Added `--receive.hashrings` alternative to `receive.hashrings-file` flag (lower priority). The flag expects the literal hashring configuration in JSON format.
 
 ### Fixed
 
+- [#3567](https://github.com/thanos-io/thanos/pull/3567) Mixin: Reintroduce `thanos_objstore_bucket_operation_failures_total` alert.
 - [#3527](https://github.com/thanos-io/thanos/pull/3527) Query Frontend: Fix query_range behavior when start/end times are the same
-- [#3560](https://github.com/thanos-io/thanos/pull/3560) query-frontend: Allow separate label cache
-- [#3672](https://github.com/thanos-io/thanos/pull/3672) rule: prevent rule crash from no such host error when using `dnssrv+` or `dnssrvnoa+`.
 - [#3760](https://github.com/thanos-io/thanos/pull/3760) Store: Fix panic caused by a race condition happening on concurrent index-header reader usage and unload, when `--store.enable-index-header-lazy-reader` is enabled.
 - [#3759](https://github.com/thanos-io/thanos/pull/3759) Store: Fix panic caused by a race condition happening on concurrent index-header lazy load and unload, when `--store.enable-index-header-lazy-reader` is enabled.
+- [#3560](https://github.com/thanos-io/thanos/pull/3560) Query Frontend: Allow separate label cache
+- [#3672](https://github.com/thanos-io/thanos/pull/3672) Rule: Prevent crashing due to `no such host error` when using `dnssrv+` or `dnssrvnoa+`.
+- [#3461](https://github.com/thanos-io/thanos/pull/3461) Compact, Shipper, Store: Fixed panic when no external labels are set in block metadata.
 
 ### Changed
 
-- [#3496](https://github.com/thanos-io/thanos/pull/3496) s3: Respect SignatureV2 flag for all credential providers.
+- [#3496](https://github.com/thanos-io/thanos/pull/3496) S3: Respect SignatureV2 flag for all credential providers.
 - [#2732](https://github.com/thanos-io/thanos/pull/2732) Swift: Switched to a new library [ncw/swift](https://github.com/ncw/swift) providing large objects support.
  By default, segments will be uploaded to the same container directory `segments/` if the file is bigger than `1GB`.
  To change the defaults see [the docs](./docs/storage.md#openstack-swift).
-- [#3626](https://github.com/thanos-io/thanos/pull/3626) Failed upload of `meta.json` file doesn't cause block cleanup anymore. This has a potential to generate corrupted blocks under specific conditions. Partial block is left in bucket for later cleanup.
-
+- [#3626](https://github.com/thanos-io/thanos/pull/3626) Shipper: Failed upload of `meta.json` file doesn't cause block cleanup anymore. This has a potential to generate corrupted blocks under specific conditions. Partial block is left in bucket for later cleanup.
 
 ## [v0.17.2](https://github.com/thanos-io/thanos/releases/tag/v0.17.2) - 2020.12.07
 
@@ -66,7 +72,8 @@ We use _breaking :warning:_ to mark changes that are not backward compatible (re
 
 ### Fixed
 
-- [#3480](https://github.com/thanos-io/thanos/pull/3480) Query-frontend: Fixed regression.
+- [#3480](https://github.com/thanos-io/thanos/pull/3480) Query Frontend: Fixed regression.
+- [#3734](https://github.com/thanos-io/thanos/pull/3734) pkg/rules/proxy: fix hotlooping when receiving client errors
 
 ### Changed
 
@@ -278,7 +285,6 @@ sse_config:
 - [#2579](https://github.com/thanos-io/thanos/pull/2579) Store: Experimental caching bucket can now cache metadata as well. Config has changed from #2532.
 - [#2526](https://github.com/thanos-io/thanos/pull/2526) Compact: In case there are no labels left after deduplication via `--deduplication.replica-label`, assign first `replica-label` with value `deduped`.
 - [#2621](https://github.com/thanos-io/thanos/pull/2621) Receive: Added flag to configure forward request timeout. Receive write will complete request as soon as quorum of writes succeeds.
-- [#3380](https://github.com/thanos-io/thanos/pull/3380) mixin: Add block deletion panels for compactor dashboards.
 
 ### Changed
 

VERSION

@@ -1 +1 @@
-0.18.0-dev
+0.18.0

pkg/rules/proxy.go

@@ -108,6 +108,9 @@ func (stream *rulesStream) receive(ctx context.Context) error {
  }
 
  if err != nil {
+ // An error happened in Recv(), hence the underlying stream is aborted
+ // as per https://github.com/grpc/grpc-go/blob/7f2581f910fc21497091c4109b56d310276fc943/stream.go#L117-L125.
+ // We must not continue receiving additional data from it and must return.
  err = errors.Wrapf(err, "receiving rules from rules client %v", stream.client)
 
  if stream.request.PartialResponseStrategy == storepb.PartialResponseStrategy_ABORT {
@@ -118,13 +121,15 @@ func (stream *rulesStream) receive(ctx context.Context) error {
  return errors.Wrapf(err, "sending rules error to server %v", stream.server)
  }
 
- continue
+ // Return no error if response strategy is warning.
+ return nil
  }
 
  if w := rule.GetWarning(); w != "" {
  if err := stream.server.Send(rulespb.NewWarningRulesResponse(errors.New(w))); err != nil {
  return errors.Wrapf(err, "sending rules warning to server %v", stream.server)
  }
+ // Client stream is not aborted, it is ok to receive additional data.
  continue
  }
 

pkg/rules/proxy_test.go

@@ -0,0 +1,253 @@
+// Copyright (c) The Thanos Authors.
+// Licensed under the Apache License 2.0.
+
+package rules
+
+import (
+ "context"
+ "io"
+ "os"
+ "reflect"
+ "testing"
+
+ "github.com/go-kit/kit/log"
+ "github.com/pkg/errors"
+ "github.com/thanos-io/thanos/pkg/rules/rulespb"
+ "github.com/thanos-io/thanos/pkg/store/storepb"
+ "google.golang.org/grpc"
+)
+
+type testRulesClient struct {
+ grpc.ClientStream
+ rulesErr, recvErr error
+ response *rulespb.RulesResponse
+ sentResponse bool
+}
+
+func (t *testRulesClient) String() string {
+ return "test"
+}
+
+func (t *testRulesClient) Recv() (*rulespb.RulesResponse, error) {
+ // A simulation of underlying grpc Recv behavior as per https://github.com/grpc/grpc-go/blob/7f2581f910fc21497091c4109b56d310276fc943/stream.go#L117-L125.
+ if t.recvErr != nil {
+ return nil, t.recvErr
+ }
+
+ if t.sentResponse {
+ return nil, io.EOF
+ }
+ t.sentResponse = true
+
+ return t.response, nil
+}
+
+func (t *testRulesClient) Rules(ctx context.Context, in *rulespb.RulesRequest, opts ...grpc.CallOption) (rulespb.Rules_RulesClient, error) {
+ return t, t.rulesErr
+}
+
+var _ rulespb.RulesClient = &testRulesClient{}
+
+type testRulesServer struct {
+ grpc.ServerStream
+ sendErr error
+ response *rulespb.RulesResponse
+}
+
+func (t *testRulesServer) String() string {
+ return "test"
+}
+
+func (t *testRulesServer) Send(response *rulespb.RulesResponse) error {
+ if t.sendErr != nil {
+ return t.sendErr
+ }
+ t.response = response
+ return nil
+}
+
+func (t *testRulesServer) Context() context.Context {
+ return context.Background()
+}
+
+func TestProxy(t *testing.T) {
+ logger := log.NewLogfmtLogger(os.Stderr)
+
+ for _, tc := range []struct {
+ name string
+ request *rulespb.RulesRequest
+ client rulespb.RulesClient
+ server *testRulesServer
+ wantResponse *rulespb.RulesResponse
+ wantError error
+ }{
+ {
+ name: "rule group proxy success",
+ request: &rulespb.RulesRequest{
+ Type: rulespb.RulesRequest_ALL,
+ PartialResponseStrategy: storepb.PartialResponseStrategy_WARN,
+ },
+ client: &testRulesClient{
+ response: rulespb.NewRuleGroupRulesResponse(&rulespb.RuleGroup{
+ Name: "foo",
+ }),
+ recvErr: nil,
+ },
+ server: &testRulesServer{},
+ wantResponse: rulespb.NewRuleGroupRulesResponse(&rulespb.RuleGroup{
+ Name: "foo",
+ }),
+ },
+ {
+ name: "warning proxy success",
+ request: &rulespb.RulesRequest{
+ Type: rulespb.RulesRequest_ALL,
+ PartialResponseStrategy: storepb.PartialResponseStrategy_WARN,
+ },
+ client: &testRulesClient{
+ response: rulespb.NewWarningRulesResponse(errors.New("warning from client")),
+ recvErr: nil,
+ },
+ server: &testRulesServer{},
+ wantResponse: rulespb.NewWarningRulesResponse(errors.New("warning from client")),
+ },
+ {
+ name: "warn: retreiving rules client failed",
+ request: &rulespb.RulesRequest{
+ Type: rulespb.RulesRequest_ALL,
+ PartialResponseStrategy: storepb.PartialResponseStrategy_WARN,
+ },
+ client: &testRulesClient{
+ response: nil,
+ rulesErr: errors.New("retreiving rules failed"),
+ },
+ server: &testRulesServer{},
+ wantResponse: rulespb.NewWarningRulesResponse(errors.New("fetching rules from rules client test: retreiving rules failed")),
+ },
+ {
+ name: "warn: retreiving rules client failed, forward warning failed",
+ request: &rulespb.RulesRequest{
+ Type: rulespb.RulesRequest_ALL,
+ PartialResponseStrategy: storepb.PartialResponseStrategy_WARN,
+ },
+ client: &testRulesClient{
+ response: nil,
+ rulesErr: errors.New("retreiving rules failed"),
+ },
+ server: &testRulesServer{
+ sendErr: errors.New("forwarding warning response failed"),
+ },
+ wantError: errors.New("forwarding warning response failed"),
+ },
+ {
+ name: "abort: retreiving rules client failed",
+ request: &rulespb.RulesRequest{
+ Type: rulespb.RulesRequest_ALL,
+ PartialResponseStrategy: storepb.PartialResponseStrategy_ABORT,
+ },
+ client: &testRulesClient{
+ response: nil,
+ rulesErr: errors.New("retreiving rules failed"),
+ },
+ server: &testRulesServer{},
+ wantError: errors.New("fetching rules from rules client test: retreiving rules failed"),
+ },
+ {
+ name: "warn: receive failed",
+ request: &rulespb.RulesRequest{
+ Type: rulespb.RulesRequest_ALL,
+ PartialResponseStrategy: storepb.PartialResponseStrategy_WARN,
+ },
+ client: &testRulesClient{
+ response: nil,
+ recvErr: errors.New("503 from Prometheus"),
+ },
+ server: &testRulesServer{},
+ wantResponse: rulespb.NewWarningRulesResponse(errors.New("receiving rules from rules client test: 503 from Prometheus")),
+ },
+ {
+ name: "warn: receive failed, forward warning failed",
+ request: &rulespb.RulesRequest{
+ Type: rulespb.RulesRequest_ALL,
+ PartialResponseStrategy: storepb.PartialResponseStrategy_WARN,
+ },
+ client: &testRulesClient{
+ response: nil,
+ recvErr: errors.New("503 from Prometheus"),
+ },
+ server: &testRulesServer{
+ sendErr: errors.New("forwarding warning response failed"),
+ },
+ wantError: errors.New("sending rules error to server test: forwarding warning response failed"),
+ },
+ {
+ name: "abort: receive failed",
+ request: &rulespb.RulesRequest{
+ Type: rulespb.RulesRequest_ALL,
+ PartialResponseStrategy: storepb.PartialResponseStrategy_ABORT,
+ },
+ client: &testRulesClient{
+ response: nil,
+ recvErr: errors.New("503 from Prometheus"),
+ },
+ server: &testRulesServer{},
+ wantError: errors.New("receiving rules from rules client test: 503 from Prometheus"),
+ },
+ {
+ name: "send failed",
+ request: &rulespb.RulesRequest{
+ Type: rulespb.RulesRequest_ALL,
+ PartialResponseStrategy: storepb.PartialResponseStrategy_WARN,
+ },
+ client: &testRulesClient{
+ response: rulespb.NewRuleGroupRulesResponse(&rulespb.RuleGroup{
+ Name: "foo",
+ }),
+ recvErr: nil,
+ },
+ server: &testRulesServer{
+ sendErr: errors.New("sending message failed"),
+ },
+ wantError: errors.New("rpc error: code = Unknown desc = send rules response: sending message failed"),
+ },
+ {
+ name: "sending warning response failed",
+ request: &rulespb.RulesRequest{
+ Type: rulespb.RulesRequest_ALL,
+ PartialResponseStrategy: storepb.PartialResponseStrategy_WARN,
+ },
+ client: &testRulesClient{
+ response: rulespb.NewWarningRulesResponse(errors.New("warning from client")),
+ recvErr: nil,
+ },
+ server: &testRulesServer{
+ sendErr: errors.New("sending message failed"),
+ },
+ wantError: errors.New("sending rules warning to server test: sending message failed"),
+ },
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ p := NewProxy(logger, func() []rulespb.RulesClient {
+ return []rulespb.RulesClient{tc.client}
+ })
+
+ err := p.Rules(tc.request, tc.server)
+ gotErr := "<nil>"
+ if err != nil {
+ gotErr = err.Error()
+ }
+ wantErr := "<nil>"
+ if tc.wantError != nil {
+ wantErr = tc.wantError.Error()
+ }
+
+ if gotErr != wantErr {
+ t.Errorf("want error %q, got %q", wantErr, gotErr)
+ }
+
+ if !reflect.DeepEqual(tc.wantResponse, tc.server.response) {
+ t.Errorf("want response %v, got %v", tc.wantResponse, tc.server.response)
+ }
+ })
+ }
+}

tutorials/katacoda/thanos/1-globalview/courseBase.sh

@@ -1,4 +1,4 @@
 #!/usr/bin/env bash
 
 docker pull quay.io/prometheus/prometheus:v2.16.0
-docker pull quay.io/thanos/thanos:v0.17.2
+docker pull quay.io/thanos/thanos:v0.18.0

tutorials/katacoda/thanos/1-globalview/step2.md

@@ -10,7 +10,7 @@ component and can be invoked in a single command.
 Let's take a look at all the Thanos commands:
 
 ```
-docker run --rm quay.io/thanos/thanos:v0.17.2 --help
+docker run --rm quay.io/thanos/thanos:v0.18.0 --help
 ```{{execute}}
 
 You should see multiple commands that solves different purposes.
@@ -53,7 +53,7 @@ docker run -d --net=host --rm \
  -v $(pwd)/prometheus0_eu1.yml:/etc/prometheus/prometheus.yml \
  --name prometheus-0-sidecar-eu1 \
  -u root \
- quay.io/thanos/thanos:v0.17.2 \
+ quay.io/thanos/thanos:v0.18.0 \
  sidecar \
  --http-address 0.0.0.0:19090 \
  --grpc-address 0.0.0.0:19190 \
@@ -68,7 +68,7 @@ docker run -d --net=host --rm \
  -v $(pwd)/prometheus0_us1.yml:/etc/prometheus/prometheus.yml \
  --name prometheus-0-sidecar-us1 \
  -u root \
- quay.io/thanos/thanos:v0.17.2 \
+ quay.io/thanos/thanos:v0.18.0 \
  sidecar \
  --http-address 0.0.0.0:19091 \
  --grpc-address 0.0.0.0:19191 \
@@ -81,7 +81,7 @@ docker run -d --net=host --rm \
  -v $(pwd)/prometheus1_us1.yml:/etc/prometheus/prometheus.yml \
  --name prometheus-1-sidecar-us1 \
  -u root \
- quay.io/thanos/thanos:v0.17.2 \
+ quay.io/thanos/thanos:v0.18.0 \
  sidecar \
  --http-address 0.0.0.0:19092 \
  --grpc-address 0.0.0.0:19192 \