Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Remote Cluster State] Allow Remote Cluster state writes to remote use a dedicated threadpool #9816

Closed
linuxpi opened this issue Sep 6, 2023 · 0 comments · Fixed by #10685
Labels
Cluster Manager enhancement Enhancement or improvement to existing feature or request v2.11.0 Issues and PRs related to version 2.11.0

Comments

@linuxpi
Copy link
Collaborator

linuxpi commented Sep 6, 2023

Describe the bug

  • Today Remote cluster state write to remote happens asynchronously. This allows us to parallelize upload of multiple files and reduce overall write time to remote
  • We use the PRIORITY threadpool introduced by the repository-s3 plugin.
  • Remote Translog upload also uses the same threadpool. So in a non dedicated cluster manager setup, the remote cluster state writes would be competing with remote translog writes.
  • This can lead to issues in case translog uploads hogs up the pool and starves remote cluster state writes.

Expected behavior
Remote cluster state write should use a dedicated threadpool to avoid any interruptions due to resource starvation

@linuxpi linuxpi added bug Something isn't working untriaged labels Sep 6, 2023
@shwetathareja shwetathareja added Cluster Manager v2.11.0 Issues and PRs related to version 2.11.0 enhancement Enhancement or improvement to existing feature or request and removed untriaged bug Something isn't working labels Sep 11, 2023
@linuxpi linuxpi changed the title [BUG] [Remote Cluster State] Allow Remote Cluster state writes to remote use a dedicated threadpool [Remote Cluster State] Allow Remote Cluster state writes to remote use a dedicated threadpool Oct 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Cluster Manager enhancement Enhancement or improvement to existing feature or request v2.11.0 Issues and PRs related to version 2.11.0
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants