Merge pull request #3082 from thanos-io/cut-0.15.0-rc.0

Cut 0.15.0 rc.0

CHANGELOG.md

@@ -7,61 +7,69 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 NOTE: As semantic versioning states all 0.y.z releases can contain breaking changes in API (flags, grpc API, any backward compatibility)
 
-We use *breaking* word for marking changes that are not backward compatible (relates only to v0.y.z releases.)
+We use *breaking :warning:* word for marking changes that are not backward compatible (relates only to v0.y.z releases.)
 
 ## Unreleased
 
-:warning: **WARNING** :warning: Thanos Rule's `/api/v1/rules` endpoint no longer returns the old, deprecated `partial_response_strategy`. The old, deprecated value has been fixed to `WARN` for quite some time. _Please_ use `partialResponseStrategy`.
-
-:warning: **WARNING** :warning: The `sse_encryption` value is now deprecated in favour of `sse_config`. If you used `sse_encryption`, the migration strategy is to set up the following block:
-
-```yaml
-
----
-sse_config:
- type: SSE-S3
-```
+## [v0.15.0-rc.0](https://github.com/thanos-io/thanos/releases/tag/v0.15.0-rc.0) - 2020.08.26
 
+Highlights:
+* Added new Thanos component: [Query Frontend](https://thanos.io/tip/components/query-frontend/) responsible for response caching,
+query scheduling and parallelization (based on Cortex Query Frontend).
+* Added various new, improved UIs to Thanos based on React: Querier' BuildInfo & Flags, Ruler UI, BlockViewer.
+* Optimized Sidecar, Store, Receive, Ruler data retrieval with new TSDB ChunkIterator, capping chunks to 120 samples, fixed various leaks.
+* Fixed sample limit on Store Gateway.
+* Added S3 Server Side Encryption options.
+* Tons of other important fixes!
 
 ### Fixed
 
-- [#2937](https://github.com/thanos-io/thanos/pull/2937) Receive: Fixing auto-configuration of --receive.local-endpoint
-- [#2665](https://github.com/thanos-io/thanos/pull/2665) Swift: fix issue with missing Content-Type HTTP headers.
+- [#2665](https://github.com/thanos-io/thanos/pull/2665) Swift: Fix issue with missing Content-Type HTTP headers.
 - [#2800](https://github.com/thanos-io/thanos/pull/2800) Query: Fix handling of `--web.external-prefix` and `--web.route-prefix`
 - [#2834](https://github.com/thanos-io/thanos/pull/2834) Query: Fix rendered JSON state value for rules and alerts should be in lowercase
 - [#2866](https://github.com/thanos-io/thanos/pull/2866) Receive, Querier: Fixed leaks on receive and querier Store API Series, which were leaking on errors.
+- [#2937](https://github.com/thanos-io/thanos/pull/2937) Receive: Fixing auto-configuration of --receive.local-endpoint
 - [#2895](https://github.com/thanos-io/thanos/pull/2895) Compact: Fix increment of `thanos_compact_downsample_total` metric for downsample of 5m resolution blocks.
 - [#2858](https://github.com/thanos-io/thanos/pull/2858) Store: Fix `--store.grpc.series-sample-limit` implementation. The limit is now applied to the sum of all samples fetched across all queried blocks via a single Series call, instead of applying it individually to each block.
 - [#2936](https://github.com/thanos-io/thanos/pull/2936) Compact: Fix ReplicaLabelRemover panic when replicaLabels are not specified.
 - [#2956](https://github.com/thanos-io/thanos/pull/2956) Store: Fix fetching of chunks bigger than 16000 bytes.
 - [#2970](https://github.com/thanos-io/thanos/pull/2970) Store: Upgrade minio-go/v7 to fix slowness when running on EKS.
-- [#2957](https://github.com/thanos-io/thanos/pull/2957) Rule: now sets all of the relevant fields properly; avoids a panic when `/api/v1/rules` is called and the time zone is _not_ UTC; `rules` field is an empty array now if no rules have been defined in a rule group.
+- [#2957](https://github.com/thanos-io/thanos/pull/2957) Rule: *breaking :warning:* Now sets all of the relevant fields properly; avoids a panic when `/api/v1/rules` is called and the time zone is _not_ UTC; `rules` field is an empty array now if no rules have been defined in a rule group.
+Thanos Rule's `/api/v1/rules` endpoint no longer returns the old, deprecated `partial_response_strategy`. The old, deprecated value has been fixed to `WARN` for quite some time. _Please_ use `partialResponseStrategy`.
 - [#2976](https://github.com/thanos-io/thanos/pull/2976) Query: Better rounding for incoming query timestamps.
 - [#2929](https://github.com/thanos-io/thanos/pull/2929) Mixin: Fix expression for 'unhealthy sidecar' alert and also increase the timeout for 10 minutes.
 - [#3024](https://github.com/thanos-io/thanos/pull/3024) Query: consider group name and file for deduplication
-- [#3064](https://github.com/thanos-io/thanos/pull/3064) s3: Add SSE/SSE-KMS/SSE-C configuration.
+- [#3012](https://github.com/thanos-io/thanos/pull/3012) Ruler,Receiver: Fix TSDB to delete blocks in atomic way.
+- [#3046](https://github.com/thanos-io/thanos/pull/3046) Ruler,Receiver: Fixed framing of StoreAPI response, it was one chunk by one.
+
+```yaml
+sse_config:
+ type: SSE-S3
+```
 
 ### Added
 
-- [#2849](https://github.com/thanos-io/thanos/pull/2849) Query, Ruler : Added request logging for HTTP server side.
-- [#2832](https://github.com/thanos-io/thanos/pull/2832) ui: React: Add runtime and build info page
-- [#2305](https://github.com/thanos-io/thanos/pull/2305) Receive,Sidecar,Ruler: Propagate correct (stricter) MinTime for no-block TSDBs.
+- [#2305](https://github.com/thanos-io/thanos/pull/2305) Receive,Sidecar,Ruler: Propagate correct (stricter) MinTime for TSDBs that have no block.
+- [#2849](https://github.com/thanos-io/thanos/pull/2849) Query, Ruler: Added request logging for HTTP server side.
+- [#2832](https://github.com/thanos-io/thanos/pull/2832) ui React: Add runtime and build info page
 - [#2926](https://github.com/thanos-io/thanos/pull/2926) API: Add new blocks HTTP API to serve blocks metadata. The status endpoints (`/api/v1/status/flags`, `/api/v1/status/runtimeinfo` and `/api/v1/status/buildinfo`) are now available on all components with a HTTP API.
 - [#2892](https://github.com/thanos-io/thanos/pull/2892) Receive: Receiver fails when the initial upload fails.
 - [#2865](https://github.com/thanos-io/thanos/pull/2865) ui: Migrate Thanos Ruler UI to React
 - [#2964](https://github.com/thanos-io/thanos/pull/2964) Query: Add time range parameters to label APIs. Add `start` and `end` fields to Store API `LabelNamesRequest` and `LabelValuesRequest`.
 - [#2996](https://github.com/thanos-io/thanos/pull/2996) Sidecar: Add `reloader_config_apply_errors_total` metric. Add new flags `--reloader.watch-interval`, and `--reloader.retry-interval`.
 - [#2973](https://github.com/thanos-io/thanos/pull/2973) Add Thanos Query Frontend component.
 - [#2980](https://github.com/thanos-io/thanos/pull/2980) Bucket Viewer: Migrate block viewer to React.
+- [#2725](https://github.com/thanos-io/thanos/pull/2725) Add bucket index operation durations: `thanos_bucket_store_cached_series_fetch_duration_seconds` and `thanos_bucket_store_cached_postings_fetch_duration_seconds`.
+- [#2931](https://github.com/thanos-io/thanos/pull/2931) Query: Allow passing a `storeMatch[]` to select matching stores when debugging the querier. See [documentation](https://thanos.io/tip/components/query.md/#store-filtering)
 
 ### Changed
 
 - [#2893](https://github.com/thanos-io/thanos/pull/2893) Store: Rename metric `thanos_bucket_store_cached_postings_compression_time_seconds` to `thanos_bucket_store_cached_postings_compression_time_seconds_total`.
 - [#2915](https://github.com/thanos-io/thanos/pull/2915) Receive,Ruler: Enable TSDB directory locking by default. Add a new flag (`--tsdb.no-lockfile`) to override behavior.
-- [#2902](https://github.com/thanos-io/thanos/pull/2902) ui: React: Separate dedupe and partial response checkboxes per panel.
-- [#2931](https://github.com/thanos-io/thanos/pull/2931) Query: Allow passing a `storeMatch[]` to select matching stores when debugging the querier. See [documentation](https://thanos.io/tip/components/query.md/#store-filtering)
-- [#2991](https://github.com/thanos-io/thanos/pull/2991) store: `operation` label value `getrange` changed to `get_range` for `thanos_store_bucket_cache_operation_requests_total` and `thanos_store_bucket_cache_operation_hits_total` to be consistent with bucket operation metrics.
+- [#2902](https://github.com/thanos-io/thanos/pull/2902) Querier UI:Separate dedupe and partial response checkboxes per panel in new UI.
+- [#2991](https://github.com/thanos-io/thanos/pull/2991) Store: *breaking :warning:* `operation` label value `getrange` changed to `get_range` for `thanos_store_bucket_cache_operation_requests_total` and `thanos_store_bucket_cache_operation_hits_total` to be consistent with bucket operation metrics.
 - [#2876](https://github.com/thanos-io/thanos/pull/2876) Receive,Ruler: Updated TSDB and switched to ChunkIterators instead of sample one, which avoids unnecessary decoding / encoding.
+- [#3064](https://github.com/thanos-io/thanos/pull/3064) s3: *breaking :warning:* Add SSE/SSE-KMS/SSE-C configuration. The S3 `encrypt_sse: true` option is now deprecated in favour of `sse_config`. If you used `encrypt_sse`, the migration strategy is to set up the following block:
 
 ## [v0.14.0](https://github.com/thanos-io/thanos/releases/tag/v0.14.0) - 2020.07.10
 

VERSION

@@ -1 +1 @@
-0.14.0
+0.15.0-rc.0

pkg/objstore/s3/s3.go

@@ -107,7 +107,7 @@ type Bucket struct {
 // parseConfig unmarshals a buffer into a Config with default HTTPConfig values.
 func parseConfig(conf []byte) (Config, error) {
  config := DefaultConfig
- if err := yaml.Unmarshal(conf, &config); err != nil {
+ if err := yaml.UnmarshalStrict(conf, &config); err != nil {
  return Config{}, err
  }
 

pkg/objstore/s3/s3_test.go

@@ -161,7 +161,6 @@ endpoint: "s3-endpoint"
 access_key: "access_key"
 insecure: false
 signature_version2: false
-encrypt_sse: false
 secret_key: "secret_key"
 http_config:
  insecure_skip_verify: false
@@ -176,7 +175,6 @@ endpoint: "s3-endpoint"
 access_key: "access_key"
 insecure: false
 signature_version2: false
-encrypt_sse: false
 secret_key: "secret_key"
 put_user_metadata:
  "X-Amz-Acl": "bucket-owner-full-control"
@@ -195,7 +193,6 @@ endpoint: "s3-endpoint"
 access_key: "access_key"
 insecure: false
 signature_version2: false
-encrypt_sse: false
 secret_key: "secret_key"
 http_config:
  insecure_skip_verify: false
@@ -210,14 +207,29 @@ endpoint: "s3-endpoint"
 access_key: "access_key"
 insecure: false
 signature_version2: false
-encrypt_sse: false
 secret_key: "secret_key"
 part_size: 104857600
 http_config:
  insecure_skip_verify: false
  idle_conn_timeout: 50s`)
-
  cfg2, err := parseConfig(input2)
  testutil.Ok(t, err)
  testutil.Assert(t, cfg2.PartSize == 1024*1024*100, "when part size should be set to 100MiB")
 }
+
+func TestParseConfig_OldSEEncryptionFieldShouldFail(t *testing.T) {
+ input := []byte(`bucket: "bucket-name"
+endpoint: "s3-endpoint"
+access_key: "access_key"
+insecure: false
+signature_version2: false
+encrypt_sse: false
+secret_key: "secret_key"
+see_encryption: true
+put_user_metadata:
+ "X-Amz-Acl": "bucket-owner-full-control"
+http_config:
+ idle_conn_timeout: 0s`)
+ _, err := parseConfig(input)
+ testutil.NotOk(t, err)
+}

pkg/store/tsdb.go

@@ -173,12 +173,14 @@ func (s *TSDBStore) Series(r *storepb.SeriesRequest, srv storepb.Store_SeriesSer
  if frameBytesLeft > 0 && isNext {
  continue
  }
-
  if err := srv.Send(storepb.NewSeriesResponse(&storepb.Series{Labels: seriesLabels.Labels, Chunks: seriesChunks})); err != nil {
  return status.Error(codes.Aborted, err.Error())
  }
- frameBytesLeft = bytesLeftForChunks
- seriesChunks = make([]storepb.AggrChunk, 0, len(seriesChunks))
+
+ if isNext {
+ frameBytesLeft = bytesLeftForChunks
+ seriesChunks = make([]storepb.AggrChunk, 0, len(seriesChunks))
+ }
  }
  if err := chIter.Err(); err != nil {
  return status.Error(codes.Internal, errors.Wrap(err, "chunk iter").Error())

scripts/quickstart.sh

@@ -8,8 +8,8 @@ trap 'kill 0' SIGTERM
 MINIO_ENABLED=${MINIO_ENABLED:-""}
 MINIO_EXECUTABLE=${MINIO_EXECUTABLE:-"minio"}
 MC_EXECUTABLE=${MC_EXECUTABLE:-"mc"}
-PROMETHEUS_EXECUTABLE=${PROMETHEUS_EXECUTABLE:-"./prometheus"}
-THANOS_EXECUTABLE=${THANOS_EXECUTABLE:-"./thanos"}
+PROMETHEUS_EXECUTABLE=${PROMETHEUS_EXECUTABLE:-"prometheus"}
+THANOS_EXECUTABLE=${THANOS_EXECUTABLE:-"thanos"}
 S3_ENDPOINT=""
 
 if [ ! $(command -v "$PROMETHEUS_EXECUTABLE") ]; then
@@ -289,4 +289,6 @@ ${THANOS_EXECUTABLE} rule \
  --grpc-address="0.0.0.0:19998" \
  ${OBJSTORECFG} &
 
+echo "all started; waiting for signal"
+
 wait