-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Should be able to remove a server from cluster #1471
Comments
I have some questions about different edge cases, as well as what should be supported in this issue. SecurityCan anyone remove a server? Do they have to have admin access? What if authentication is disabled? Quorum StatusCan this only be done if there is a quorum? I'm not sure how we can support this in an unhealthy cluster. Data LossIf the server has shards with an RP of 1, do we still allow them to drop, or do we warn them and ask for something like LeaderCan you drop the server if it is the leader? Or should you have to force it to step down first? Or should we just force it to step down? Data on diskShould we remove the data left on disk? Controlling the ServiceEven if we exit the process, it's likely it will be spun back up by the server depending on how it was configured. Even if we delete all directories, it will spin back up in a default configuration. Is that something we just document? I'm not sure there is much we can do about that. |
Don't remove any data. That should be a distinct step. A user might remove On Thursday, September 24, 2015, Cory LaNou notifications@github.com
|
Also, I think the syntax should have to drop a server by |
@otoolep if you don't remove the data, and it's being controlled by service architecture, it will come right back up and potentially rejoin the cluster. We would need to handle that somehow. |
I'm not sure I fully follow, I guess you mean Raft metadata. What I am On Thu, Sep 24, 2015 at 10:24 AM, Cory LaNou notifications@github.com
|
Yes, drop by ID. Don't delete data off disk. They have to do that by either dropping shards, databases or actually deleting the files themselves. The most common use case for removing a server is removing one from the cluster that has gone down and isn't coming back up. For example if you lose your EC2 instance or DO droplet. However, when removing a server from the cluster it shouldn't matter if it is the leader or not. Removing it from the Raft consensus group should force a leader election. We should generally require a quorum when doing these kinds of operations. However, there are cases in which that won't be possible. For example if you have 3 servers and 2 of them go down and aren't coming back. The other cases where you want control over who participates in Raft is when you want to configure the cluster after its already running so that you get redundancy across hypervisors or racks or AZs. You don't want all your consensus nodes in the same AZ, for instance. As for who should be allowed to remove a server, only cluster admins. We should probably also give our users controls so that cluster admin queries can only be run from the box itself. Although this would be a different feature we should log an issue for. |
This issue is a year old. Any updates? |
This is possible in Implemented via |
Should be able forcibly remove a server from a cluster. Doesn't bother actually moving any data around, should just remove it from the metastore and kill all connections from other servers.
drop server "serverA"
The text was updated successfully, but these errors were encountered: