Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ES Client: use ClusterConnectionPool instead of WeightedConnectionPool #134628

Merged

Conversation

pgayvallet
Copy link
Contributor

@pgayvallet pgayvallet commented Jun 17, 2022

Summary

We discovered a bug in @elastic/elasticsearch-js that causes Kibana to not be able to connect to ES nodes for a predetermined amount of requests when using a multi-node configuration and when some, or all, nodes go down then up again: elastic/elasticsearch-js#1714

Until the upstream issue is addressed, we will fall back to using the ClusterConnectionPool implementation instead of WeightedConnectionPool, as this implementation (which was the one used by the version 7.x of the client) doesn't have this issue.

Checklist

Release Note

Fix a bug causing ES nodes going down then up again to be unreachable by Kibana for a given amount of requests, when Kibana is configured to connect to multiple ES node

@pgayvallet pgayvallet added bug Fixes for quality problems that affect the customer experience v8.4.0 v8.3.1 Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc v8.2.4 auto-backport Deprecated - use backport:version if exact versions are needed release_note:fix labels Jun 17, 2022
Comment on lines +38 to +39
// using ClusterConnectionPool until https://github.com/elastic/elasticsearch-js/issues/1714 is addressed
ConnectionPool: ClusterConnectionPool,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't add an integration test here because:

  • Ihmo we shouldn't be testing the @elastic/elasticsearch behavior from Kibana tests
  • The test would be quite long (starting multiple ES nodes at least twice), and by nature, likely flaky

If we really want, I can add one though

@kibana-ci
Copy link
Collaborator

💚 Build Succeeded

Metrics [docs]

✅ unchanged

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@pgayvallet pgayvallet marked this pull request as ready for review June 17, 2022 07:27
@pgayvallet pgayvallet requested a review from a team as a code owner June 17, 2022 07:27
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-core (Team:Core)

Copy link
Contributor

@gsoldevila gsoldevila left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@pgayvallet pgayvallet merged commit 6decdc3 into elastic:main Jun 17, 2022
@kibanamachine
Copy link
Contributor

💔 All backports failed

Status Branch Result
8.3 Backport failed because of merge conflicts
8.2 Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 134628

Questions ?

Please refer to the Backport tool documentation

pgayvallet added a commit to pgayvallet/kibana that referenced this pull request Jun 17, 2022
(cherry picked from commit 6decdc3)

# Conflicts:
#	src/core/server/elasticsearch/client/configure_client.ts
pgayvallet added a commit to pgayvallet/kibana that referenced this pull request Jun 17, 2022
(cherry picked from commit 6decdc3)

# Conflicts:
#	src/core/server/elasticsearch/client/configure_client.ts
pgayvallet added a commit that referenced this pull request Jun 17, 2022
(cherry picked from commit 6decdc3)

# Conflicts:
#	src/core/server/elasticsearch/client/configure_client.ts
pgayvallet added a commit that referenced this pull request Jun 17, 2022
(cherry picked from commit 6decdc3)

# Conflicts:
#	src/core/server/elasticsearch/client/configure_client.ts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
auto-backport Deprecated - use backport:version if exact versions are needed bug Fixes for quality problems that affect the customer experience release_note:fix Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc v8.2.4 v8.3.1 v8.4.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants