Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ksonnet: Reduce querier parallelism to a more sane default value and remove the default setting for storage_backend #2452

Merged
merged 1 commit into from
Jul 30, 2020
Merged

ksonnet: Reduce querier parallelism to a more sane default value and remove the default setting for storage_backend #2452

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 18 additions & 2 deletions docs/sources/operations/upgrade.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,9 @@ On this page we will document any upgrade issues/gotchas/considerations we are a

## Master / Unreleased

Configuration document has been re-orderd a bit and for all the config, corresponding `CLI` flag is
provided.
#### Documentation changes:

Configuration document has been re-orderd a bit and for all the config, corresponding `CLI` flag is provided.

S3 config now supports exapnded config. Example can be found here [s3_expanded_config](../configuration/examples.md#s3-expanded-config)

Expand Down Expand Up @@ -57,6 +58,21 @@ loki_canary_ws_reconnects -> loki_canary_ws_reconnects_total
loki_canary_response_latency -> loki_canary_response_latency_seconds
```

### Ksonnet Changes

In `production/ksonnet/loki/config.libsonnet` the variable `storage_backend` used to have a default value of `'bigtable,gcs'`.
This has been changed to providing no default and will error if not supplied in your environment jsonnet,
here is an example of what you should add to have the same behavior as the default (namespace and cluster should already be defined):

```jsonnet
_config+:: {
namespace: 'loki-dev',
cluster: 'us-central1',
storage_backend: 'gcs,bigtable',
```

Defaulting to `gcs,bigtable` was confusing for anyone using ksonnet with other storage backends as it would manifest itself with obscure bigtable errors.

## 1.5.0

Note: The required upgrade path outlined for version 1.4.0 below is still true for moving to 1.5.0 from any release older than 1.4.0 (e.g. 1.3.0->1.5.0 needs to also look at the 1.4.0 upgrade requirements).
Expand Down
10 changes: 6 additions & 4 deletions production/ksonnet/loki/config.libsonnet
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,10 @@


querier: {
concurrency: 32,
// This value should be set equal to (or less than) the CPU cores of the system the querier runs.
// A higher value will lead to a querier trying to process more requests than there are available
// cores and will result in scheduling delays.
concurrency: 4,
},

queryFrontend: {
Expand All @@ -28,8 +31,7 @@
query_split_factor:: 3,
},

// Default to GCS and Bigtable for chunk and index store
storage_backend: 'bigtable,gcs',
storage_backend: error 'must define storage_backend as a comma separated list of backends in use,\n valid entries: dynamodb,s3,gcs,bigtable,cassandra. Typically this would be two entries, e.g. `gcs,bigtable`',

enabledBackends: [
backend
Expand Down Expand Up @@ -162,7 +164,7 @@
parallelise_shardable_queries: true,
} else {},
querier: {
query_ingesters_within: '2h', // twice the max-chunk age (1h default) for safety buffer
query_ingesters_within: '2h', // twice the max-chunk age (1h default) for safety buffer
},
limits_config: {
enforce_metric_name: false,
Expand Down