Prevent xDS tight loop on cfg errors #12195
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Previously the Delta xDS protocol could get into a tight loop where: 1. Consul sends xDS resources it believes are valid 2. Envoy rejects them with a NACK 3. Consul re-generates and sends the resources 4. Loop back to step 2 This tight loop leads to massive amounts of error logs in both Envoy and Consul logs. This commit updates the delta xDS loop to wait for new requests or snapshots after a NACK is received from Envoy. By skipping to the top of the for loop we avoid re-sending the resources that Envoy has already rejected.
kisunji
reviewed
Jan 26, 2022
|
Thanks for the early feedback! Need to see what's going on with these tests as well. |
Nack is tested in TestServer_DeltaAggregatedResources_v3_NackLoop now.
kisunji
reviewed
Jan 28, 2022
kisunji
approved these changes
Jan 28, 2022
freddygv
added a commit
that referenced
this pull request
Jan 28, 2022
Transparent proxies can set up filter chains that allow direct connections to upstream service instances. Services that can be dialed directly are stored in the PassthroughUpstreams map of the proxycfg snapshot. Previously these addresses were not being cleaned up based on new service health data. The list of addresses associated with an upstream service would only ever grow. As services scale up and down, eventually they will have instances assigned to an IP that was previously assigned to a different service. When IP addresses are duplicated across filter chain match rules the listener config will be rejected by Envoy. This commit updates the proxycfg snapshot management so that passthrough addresses can get cleaned up when no longer associated with a given upstream. There is still the possibility of a race condition here where due to timing an address is shared between multiple passthrough upstreams. That concern is mitigated by #12195, but will be further addressed in a follow-up.
rboyer
reviewed
Jan 28, 2022
|
🍒 If backport labels were added before merging, cherry-picking will start automatically. To retroactively trigger a backport after merging, add backport labels and re-run https://circleci.com/gh/hashicorp/consul/580062. |
freddygv
added a commit
that referenced
this pull request
Feb 10, 2022
freddygv
added a commit
that referenced
this pull request
Feb 10, 2022
This was referenced Feb 10, 2022
freddygv
added a commit
that referenced
this pull request
Feb 11, 2022
Transparent proxies can set up filter chains that allow direct connections to upstream service instances. Services that can be dialed directly are stored in the PassthroughUpstreams map of the proxycfg snapshot. Previously these addresses were not being cleaned up based on new service health data. The list of addresses associated with an upstream service would only ever grow. As services scale up and down, eventually they will have instances assigned to an IP that was previously assigned to a different service. When IP addresses are duplicated across filter chain match rules the listener config will be rejected by Envoy. This commit updates the proxycfg snapshot management so that passthrough addresses can get cleaned up when no longer associated with a given upstream. There is still the possibility of a race condition here where due to timing an address is shared between multiple passthrough upstreams. That concern is mitigated by #12195, but will be further addressed in a follow-up.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously the Delta xDS protocol could get into a tight loop where:
This tight loop leads to massive amounts of error logs in both Envoy and
Consul logs.
This commit updates the delta xDS loop to wait for new requests or
snapshots after a NACK is received from Envoy. By skipping to the top of
the for loop we avoid re-sending the resources that Envoy has already
rejected.