Should be able to remove a server from cluster

[pauldix]

Should be able forcibly remove a server from a cluster. Doesn't bother actually moving any data around, should just remove it from the metastore and kill all connections from other servers.

drop server "serverA"

[corylanou]

I have some questions about different edge cases, as well as what should be supported in this issue.

Security

Can anyone remove a server? Do they have to have admin access? What if authentication is disabled?

Quorum Status

Can this only be done if there is a quorum? I'm not sure how we can support this in an unhealthy cluster.

Data Loss

If the server has shards with an RP of 1, do we still allow them to drop, or do we warn them and ask for something like DROP SERVER "serverA" WITH FORCE?

Leader

Can you drop the server if it is the leader? Or should you have to force it to step down first? Or should we just force it to step down?

Data on disk

Should we remove the data left on disk?

Controlling the Service

Even if we exit the process, it's likely it will be spun back up by the server depending on how it was configured. Even if we delete all directories, it will spin back up in a default configuration. Is that something we just document? I'm not sure there is much we can do about that.

/cc @pauldix @jwilder

[otoolep]

Don't remove any data. That should be a distinct step. A user might remove
the wrong node and need to quickly put it back.

On Thursday, September 24, 2015, Cory LaNou notifications@github.com
wrote:

I have some questions about different edge cases, as well as what should
be supported in this issue.
Security

Can anyone remove a server? Do they have to have admin access? What if
authentication is disabled?
Quorum Status

Can this only be done if there is a quorum? I'm not sure how we can
support this in an unhealthy cluster.
Data Loss

If the server has shards with an RP of 1, do we still allow them to drop,
or do we warn them and ask for something like DROP SERVER "serverA" WITH
FORCE?
Leader

Can you drop the server if it is the leader? Or should you have to force
it to step down first? Or should we just force it to step down?
Data on disk

Should we remove the data left on disk?
Controlling the Service

Even if we exit the process, it's likely it will be spun back up by the
server depending on how it was configured. Even if we delete all
directories, it will spin back up in a default configuration. Is that
something we just document? I'm not sure there is much we can do about that.

/cc @pauldix https://github.com/pauldix @jwilder
https://github.com/jwilder

—
Reply to this email directly or view it on GitHub
#1471 (comment).

[corylanou]

Also, I think the syntax should have to drop a server by ID, which is what you get when you do a show servers.

[corylanou]

@otoolep if you don't remove the data, and it's being controlled by service architecture, it will come right back up and potentially rejoin the cluster. We would need to handle that somehow.

[otoolep]

I'm not sure I fully follow, I guess you mean Raft metadata. What I am
talking about is don't remove any shard data, actual potentially value
time-series data. Make sense?

On Thu, Sep 24, 2015 at 10:24 AM, Cory LaNou notifications@github.com
wrote:

@otoolep https://github.com/otoolep if you don't remove the data, and
it's being controlled by service architecture, it will come right back up
and potentially rejoin the cluster. We would need to handle that somehow.

—
Reply to this email directly or view it on GitHub
#1471 (comment).

[pauldix]

Yes, drop by ID. Don't delete data off disk. They have to do that by either dropping shards, databases or actually deleting the files themselves.

The most common use case for removing a server is removing one from the cluster that has gone down and isn't coming back up. For example if you lose your EC2 instance or DO droplet.

However, when removing a server from the cluster it shouldn't matter if it is the leader or not. Removing it from the Raft consensus group should force a leader election.

We should generally require a quorum when doing these kinds of operations. However, there are cases in which that won't be possible. For example if you have 3 servers and 2 of them go down and aren't coming back.

The other cases where you want control over who participates in Raft is when you want to configure the cluster after its already running so that you get redundancy across hypervisors or racks or AZs. You don't want all your consensus nodes in the same AZ, for instance.

As for who should be allowed to remove a server, only cluster admins. We should probably also give our users controls so that cluster admin queries can only be run from the box itself. Although this would be a different feature we should log an issue for.

[johnjelinek]

This issue is a year old. Any updates?

[jwilder]

This is possible in 0.10 with drop data server <id> and drop meta server <id> although there are no safeguards in place to prevent losing data if you drop the only server holding shard data.

Implemented via c9d82ad