Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Perform a distribution of aliens across a smaller set of nodes #872

Open
ikopylov opened this issue Apr 6, 2024 · 0 comments
Open

Perform a distribution of aliens across a smaller set of nodes #872

ikopylov opened this issue Apr 6, 2024 · 0 comments

Comments

@ikopylov
Copy link
Member

ikopylov commented Apr 6, 2024

Right now if we have cluster with 100 nodes and 1 node is down, we got 99 partitions all other the cluster for the same period. After alien recovery all these 99 partitions will be moved to the original node. Every new partition reduce performance and consume file descriptors, so this can become a problem.
We can divide all nodes into subsets and distribute the subsets evenly among the nodes. If the target node is down, then a node to store aliens will be selected from the allocated subset. If all nodes in a subset become unavailable (this should happen very rarely), then we can use other subsets. This approach will reduce the number of partitions created in the cluster.
Size of the subset can be configured by user in config file.

Distribution example:
100 nodes, subset size: 10
Node 1: subset 1 (nodes 1-10)
Node 2: subset 2 (nodes 11-20)
...
Node 10: subset 10 (nodes 91-100)
Node 11: subset 1 (nodes 1-10)
...

Related issue #871

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant