Evenly spread queriers across available nodes

[salvacorts]

What this PR does / why we need it:

Currently, queriers run on different nodes. We do this by using the following anti-affinity rule:

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
        matchLabels:
          name: querier
      topologyKey: kubernetes.io/hostname

The original reasoning behind this is to make our systems more resilient to node failures. For example, if we have 5 queriers, all scheduled in the same node, and the node fails, we would not be able to serve queries until the pods get rescheduled and running on another node.

When new queriers are created, they need to be scheduled into different nodes. If there are not enough nodes in the cluster, new nodes need to be provisioned. This has two implications:

• Cost: Each node has an implicit cost on top of the workload we deploy into the node.
• Latency: It takes longer to allocate a new node into the cluster, and then schedule the pod, than to schedule it right away into an already available node.

This PR uses Kubernetes's TopologySpreadConstraints to evenly spread queriers across the available nodes, but allows more than one querier to be scheduled into the same node.

Checklist

• Documentation added
• Tests updated
• Is this an important fix or new feature? Add an entry in the CHANGELOG.md.
• Changes that require user attention or interaction to upgrade are documented in docs/sources/upgrading/_index.md

[grafanabot]

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0.6%

[grafanabot]

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
-        distributor	-0.3%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

[salvacorts]

As suggested by @dannykopping in the design doc, I added an entry to the changelog and the upgrade guide.

[grafanabot]

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
-        distributor	-0.3%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

[dannykopping]

This looks good @salvacorts, but I think we should make it configurable.

If you want to keep running up to one querier per node, you will need to revert the changes for production/ksonnet/loki/querier.libsonnet
made at 6415.

How about we offer both the node affinity & the topology spread via a config value (defaulted to topology spread), and allow configuring the max skew?

This could later be expanded for all components (defaulted to node affinity for now)

[grafanabot]

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0.3%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

[salvacorts]

How about we offer both the node affinity & the topology spread via a config value (defaulted to topology spread), and allow configuring the max skew?

@dannykopping That's a good idea. I just added two new config parameters: querier.use_topology_spread and querier.topology_spread_max_skew. I also updated the upgrade guide to reflect these changes.

[dannykopping]

LGTM! Two minor nits

[dannykopping]

Suggested change

	If you want to keep running up to one querier per node, set `$._config.querier.use_topology_spread` to false.
	If you want to run at most a single querier per node, set `$._config.querier.use_topology_spread` to false.

[salvacorts]

Done

[grafanabot]

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
- querier/queryrange	-0.1%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0.4%
+               loki	0%