Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Evenly spread queriers across available nodes #6415

Merged
merged 6 commits into from
Jun 20, 2022
Merged

Evenly spread queriers across available nodes #6415

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
8a58a4b
Evenly spread queriers across available nodes
salvacorts Jun 17, 2022
b6e86f8
Fix lint issue
salvacorts Jun 17, 2022
96ee6bb
Add entry to the CHANGELOG and the Upgrade Guide
salvacorts Jun 17, 2022
5628d6e
Make topology spread configurable
salvacorts Jun 20, 2022
3dbe0ab
Merge branch 'main' into salvacorts/queriers-nodes-even-spread
salvacorts Jun 20, 2022
1b15fe6
Apply CR feedback
salvacorts Jun 20, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
## Main

* [6415](https://github.com/grafana/loki/pull/6415) **salvacorts** Evenly spread queriers across kubernetes nodes.
* [6410](https://github.com/grafana/loki/pull/6410) **MichelHollands**: Add support for per tenant delete API access enabling.
* [6372](https://github.com/grafana/loki/pull/6372) **splitice**: Add support for numbers in JSON fields.
* [6105](https://github.com/grafana/loki/pull/6105) **rutgerke** Export metrics for the Promtail journal target.
Expand Down
5 changes: 5 additions & 0 deletions docs/sources/upgrading/_index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,11 @@ The output is incredibly verbose as it shows the entire internal config struct u

### Loki

#### Evenly spread queriers across kubernetes nodes

We now evenly spread queriers across the available kubernetes nodes, but allowing more than one querier to be scheduled into the same node.
If you want to keep running up to one querier per node, set `$._config.querier.use_topology_spread` to false.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
If you want to keep running up to one querier per node, set `$._config.querier.use_topology_spread` to false.
If you want to run at most a single querier per node, set `$._config.querier.use_topology_spread` to false.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done


#### Implementation of unwrapped `rate` aggregation changed

The implementation of the `rate()` aggregation function changed back to the previous implemention prior to [#5013](https://github.com/grafana/loki/pulls/5013).
Expand Down
7 changes: 7 additions & 0 deletions production/ksonnet/loki/config.libsonnet
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,13 @@
// A higher value will lead to a querier trying to process more requests than there are available
// cores and will result in scheduling delays.
concurrency: 4,

// If use_topology_spread is true, queriers can run on nodes already running queriers but will be
// spread through the available nodes using a TopologySpreadConstraints with a max skew
dannykopping marked this conversation as resolved.
Show resolved Hide resolved
// of topology_spread_max_skew.
// If use_topology_spread is false, queriers will not be scheduled on nodes already running queriers.
use_topology_spread: true,
topology_spread_max_skew: 1,
},

queryFrontend: {
Expand Down
14 changes: 12 additions & 2 deletions production/ksonnet/loki/querier.libsonnet
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ local k = import 'ksonnet-util/kausal.libsonnet';
]) else {},

local deployment = k.apps.v1.deployment,
local topologySpreadConstraints = k.core.v1.topologySpreadConstraint,

querier_deployment: if !$._config.stateful_queriers then
deployment.new('querier', 3, [$.querier_container]) +
Expand All @@ -35,9 +36,18 @@ local k = import 'ksonnet-util/kausal.libsonnet';
$._config.overrides_configmap_mount_name,
$._config.overrides_configmap_mount_path,
) +
k.util.antiAffinity +
deployment.mixin.spec.strategy.rollingUpdate.withMaxSurge(5) +
deployment.mixin.spec.strategy.rollingUpdate.withMaxUnavailable(1)
deployment.mixin.spec.strategy.rollingUpdate.withMaxUnavailable(1) +
if $._config.querier.use_topology_spread then
deployment.spec.template.spec.withTopologySpreadConstraints(
// Evenly spread queriers among available nodes.
topologySpreadConstraints.labelSelector.withMatchLabels({ name: 'querier' }) +
topologySpreadConstraints.withTopologyKey('kubernetes.io/hostname') +
topologySpreadConstraints.withWhenUnsatisfiable('ScheduleAnyway') +
topologySpreadConstraints.withMaxSkew($._config.querier.topology_spread_max_skew),
)
else
k.util.antiAffinity
else {},

// PVC for queriers when running as statefulsets
Expand Down