Consul 1.6.3 DDos using consul-template (1.6.2 working fine)

[obourdon]

Please have a look at consul issue #7259

Note that running with -once works perfectly well so I guess the "burst" occurs on refresh

I think the issue/configuration values to solve can be both in consul AND consul-template therefore the post of 2 separate issues

Consul Template version

0.24.1

Configuration

I have reproduced the issue isolated from my complete (crypted and SSL secured) environment using official consul Docker container and get the same behaviour: OK with 1.6.2 KO with 1.6.3 and later.

You can find the code to reproduce it yourself here.

Command

idem

Debug output

None to provide at this step (way too big) but again use provided reproductible test case above

Expected behavior

Either success or some limits to be configured properly to prevent this from happening but tried a lot of combination of consul and consul-template configuration parameters without success

Actual behavior

Lots of EOF are seen within consul-template output and consul client node loses connection to consul server/cluster consul members does not work anymore. However after some time, it seems to self-recover

Steps to reproduce

See README.md of reproductible test case

References

See also consul-template issues and PRs #1279, #1066, #1065, #1107

[eikenb]

I was looking into this and while asking around was pointed to http_max_conns_per_client as the probably source of the problem. I already went into this in the related consul issue. Here's a link to that comment for posterity.

hashicorp/consul#7259 (comment)

[eikenb]

@obourdon see the above (and linked consul comment). I'm adding this to be sure you get messaged about it. Please let me know if this does/doesn't fix your issue. Thanks.

[obourdon]

@eikenb will have a look at this thanks. Sorry for not answering earlier. Will keep you posted

[obourdon]

@eikenb and all please have a look at my latest entry in consul issue #7259

[pierresouchay]

@eikenb Maybe returning HTTP 429 could allow consul-template to detect that behavior and automatically apply rate limit, see hashicorp/consul#7527 and hashicorp/go-connlimit#6

[srstsavage]

Just sanity checking here, based on @pierresouchay's comment above I went looking for some config switch to manually apply some consul agent concurrent request limiting but didn't find any in consul-template -h or in view.go:

https://github.com/hashicorp/consul-template/blob/main/watch/view.go

Other than raising http_max_conns_per_client in consul (which is a bit unfortunate because the ceiling only needs to be raised for the inital startup/first pass burst of a consul-template instance), it seems that the best we can currently do on the consul-template side is to tune -consul-retry-backoff and -consul-retry-max-backoff to limit the max backoff to a second or two.

Currently we're using consul-template to generate nginx proxy config and due to consul's client rate limiting and consul-template's retry backoff we end up with a long delay when starting up a consul-template instance watching ~200 services.

[pierresouchay]

@srstsavage if you have issues with that, you might have a look at consul-templaterb which has heuristics to avoid this: https://github.com/criteo/consul-templaterb/

Also an article about it: https://medium.com/criteo-engineering/template-based-discovery-with-consul-templaterb-8ff88434c457