Skip to content

Commit

Permalink
Set RemoteReadConcurrencyLimit to fix /api/v1/read
Browse files Browse the repository at this point in the history
Previously this value was unset, defaulting to zero. A zero-value
initialised a channel with no buffer, causing all writes to the buffer
to hang, and therefore all remote_read queries to hang until they
timed-out. This is contradictory to the Prometheus documentation, which
claims that a zero value allows infinite concurrent reads.

Fixes: #350
Signed-off-by: Joe Groocock <jgroocock@cloudflare.com>
  • Loading branch information
frebib authored and jacksontj committed Oct 6, 2020
1 parent 77f24fe commit 9ef7b37
Showing 1 changed file with 4 additions and 0 deletions.
4 changes: 4 additions & 0 deletions cmd/promxy/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,8 @@ type cliOpts struct {
QueryMaxSamples int `long:"query.max-samples" description:"Maximum number of samples a single query can load into memory. Note that queries will fail if they would load more samples than this into memory, so this also limits the number of samples a query can return." default:"50000000"`
QueryLookbackDelta time.Duration `long:"query.lookback-delta" description:"The maximum lookback duration for retrieving metrics during expression evaluations." default:"5m"`

RemoteReadMaxConcurrency int `long:"remote-read.max-concurrency" description:"Maximum number of concurrent remote read calls." default:"10"`

NotificationQueueCapacity int `long:"alertmanager.notification-queue-capacity" description:"The capacity of the queue for pending alert manager notifications." default:"10000"`
AccessLogDestination string `long:"access-log-destination" description:"where to log access logs, options (none, stderr, stdout)" default:"stdout"`
ForOutageTolerance time.Duration `long:"rules.alert.for-outage-tolerance" description:"Max time to tolerate prometheus outage for restoring for state of alert." default:"1h"`
Expand Down Expand Up @@ -317,6 +319,8 @@ func main() {
RuleManager: ruleManager,
Notifier: notifierManager,

RemoteReadConcurrencyLimit: opts.RemoteReadMaxConcurrency,

EnableLifecycle: opts.EnableLifecycle,

Flags: opts.ToFlags(),
Expand Down

0 comments on commit 9ef7b37

Please sign in to comment.