Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

removes sharding queue math after global concurrency PR #3356

Merged
merged 1 commit into from
Feb 18, 2021
Merged

removes sharding queue math after global concurrency PR #3356

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 1 addition & 16 deletions production/ksonnet/loki/config.libsonnet
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,14 +42,6 @@
replicas: 2,
shard_factor: 16, // v10 schema shard factor
sharded_queries_enabled: false,
// Queries can technically be sharded an arbitrary number of times. Thus query_split_factor is used
// as a coefficient to multiply the frontend tenant queues by. The idea is that this
// yields a bit of headroom so tenant queues aren't underprovisioned. Therefore the split factor
// should represent the highest reasonable split factor for a query. If too low, a long query
// (i.e. 30d) with a high split factor (i.e. 5) would result in
// (day_splits * shard_factor * split_factor) or 30 * 16 * 5 = 2400 sharded queries, which may be
// more than the max queue size and thus would always error.
query_split_factor:: 3,
},

storage_backend: error 'must define storage_backend as a comma separated list of backends in use,\n valid entries: dynamodb,s3,gcs,bigtable,cassandra. Typically this would be two entries, e.g. `gcs,bigtable`',
Expand Down Expand Up @@ -147,15 +139,8 @@
},
frontend: {
compress_responses: true,
} + if $._config.queryFrontend.sharded_queries_enabled then {
// In process tenant queues on frontends. We divide by the number of frontends;
// 2 in this case in order to apply the global limit in aggregate.
// This is basically base * shard_factor * query_split_factor / num_frontends where
max_outstanding_per_tenant: std.floor(200 * $._config.queryFrontend.shard_factor * $._config.queryFrontend.query_split_factor / $._config.queryFrontend.replicas),
}
else {
max_outstanding_per_tenant: 200,
log_queries_longer_than: '5s',
max_outstanding_per_tenant: 300,
},
frontend_worker: {
frontend_address: 'query-frontend.%s.svc.cluster.local:9095' % $._config.namespace,
Expand Down