Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Auto-expand replicas should clamp to closest value #84788

Closed
DaveCTurner opened this issue Mar 9, 2022 · 2 comments · Fixed by #87505
Closed

Auto-expand replicas should clamp to closest value #84788

DaveCTurner opened this issue Mar 9, 2022 · 2 comments · Fixed by #87505
Assignees
@pxsalehi
Labels
>bug :Distributed Coordination/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination.

Comments

@DaveCTurner
Copy link
Contributor

DaveCTurner commented Mar 9, 2022 •
edited
Loading

Elasticsearch Version

All

Installed Plugins

N/A

Java Version

bundled

OS Version

N/A

Problem Description

Today if an index has an index.auto_expand_replicas setting which cannot be realised in the cluster then the number of replicas is left unchanged by AllocationService#adaptAutoExpandReplicas(). Instead, I think we should change the number of replicas to the closest value within the permitted range.

For instance: if a cluster has a single data node which is marked as shutting down for restart then auto-expand replicas considers this cluster to have zero data nodes, and hence requires -1 replicas. If an index is created with index.auto_expand_replicas: "0-1" then it will not be adjusted to have zero replicas, even though this would be better than leaving it at the default value of 1 replica, and will ultimately be the correct setting.

Steps to Reproduce

  1. Create a cluster with one data node.
  2. Put a RESTART shutdown marker on the data node.
  3. Create an index with index.auto_expand_replicas: "0-1"
  4. Observe that this results in an unassigned replica.

Logs (if relevant)

No response
@DaveCTurner DaveCTurner added >bug :Distributed Coordination/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) labels Mar 9, 2022
@elasticmachine elasticmachine added the Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. label Mar 9, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

@romain-chanu romain-chanu mentioned this issue Mar 10, 2022
Closed
@rudolf rudolf mentioned this issue Apr 12, 2022
Closed
@henningandersen
Copy link
Contributor

For instance: if a cluster has a single data node which is marked as shutting down for restart

Notice that this example is no longer valid since #85277, but the general problem persists.

@pxsalehi pxsalehi self-assigned this Jun 2, 2022
@pxsalehi pxsalehi mentioned this issue Jun 8, 2022
Merged
pxsalehi added a commit to pxsalehi/elasticsearch that referenced this issue Jun 9, 2022
@pxsalehi
Clamp auto-expand replicas to the closest value
1c1b9ea 
Ensure that the number of replicas chosen for an auto-expand-able shard
is within the range of the available data nodes, i.e., excluding those
nodes that cannot be assigned a replica.

Closes elastic#84788
pxsalehi added a commit that referenced this issue Jun 13, 2022
@pxsalehi
Clamp auto-expand replicas to the closest value (#87505)
60efb9d 
Ensure that the number of replicas chosen for an auto-expand-able shard
is within the range of the available data nodes, i.e., excluding those
nodes that cannot be assigned a replica.

Closes #84788
@lukeelmers lukeelmers mentioned this issue Jun 22, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pxsalehi pxsalehi

Labels
>bug :Distributed Coordination/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Clamp auto-expand replicas to the closest value pxsalehi/elasticsearch
4 participants
@pxsalehi @DaveCTurner @elasticmachine @henningandersen