Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

consul: prevent re-registration churn by correctly comparing connect sidecar tags #9330

Merged
merged 1 commit into from
Nov 12, 2020

Conversation

shoenig
Copy link
Member

@shoenig shoenig commented Nov 11, 2020

Previously, connect sidecars would be re-registered with consul every cycle
of Nomad's reconciliation loop around Consul service registrations. This is
because part of the comparison used reflect.DeepEqual on []string objects,
which returns false when one object is []string{} and the other is []string{}(nil).

Unfortunately, this was always the case, and every Connect sidecar service
would be re-registered on every iteration, which happens every 30 seconds.

At the very least exacerbates #9307

Copy link
Contributor

@jazzyfresh jazzyfresh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

code LGTM, some circleci tests need to be looked at but Im not sure if those are the flaky ones

return true
}

for i, valueA := range a {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this returns different if the same tags, in a different order.

there's an argument that we should consider the canonical (sorted) form, allowing users to changed the order without re-registering (which may be the case, for example, if the tags are procedurally generated in a way that doesn't guarantee consistent ordering).
there's another argument that says if they change the order of the tags on a job update, we'll reregister the service (with the tags in exactly the order they specified).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consul itself persists tags in the order given, and at least for now I think Nomad should should maintain what the user provides.

@cgbaker
Copy link
Contributor

cgbaker commented Nov 11, 2020

as @jazzyfresh , some of the failing tests look sidecar/tag related; don't know whether they're flaky, fragile, or protective.

…tags

Previously, connect sidecars would be re-registered with consul every cycle
of Nomad's reconciliation loop around Consul service registrations. This is
because part of the comparison used `reflect.DeepEqual` on []string objects,
which returns false when one object is `[]string{}` and the other is `[]string{}(nil)`.

Unforunately, this was always the case, and every Connect sidecar service
would be re-registered on every iteration, which happens every 30 seconds.
@shoenig
Copy link
Member Author

shoenig commented Nov 12, 2020

Yeah sorry the tests were broken because there was a flipped condition. Yay for comprehensive tests!

@shoenig shoenig merged commit 0a90cee into master Nov 12, 2020
@shoenig shoenig deleted the b-connect-registration-bad-tags branch November 12, 2020 00:22
@github-actions
Copy link

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Dec 11, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants