-
Notifications
You must be signed in to change notification settings - Fork 4.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Tricky to bootstrap cluster after stopped gracefully #750
Comments
This happened for me in 0.4.1 as well. I actually disabled leave-on-terminate for my "server" role Consul agents because of this after the cluster ate itself a few times. |
Why do nodes write null in it when leaving? |
They do not write null, they remove themselves - that's the point of a graceful leave. In 0.4.1, the contents of the file was simply |
Yes
|
@pepov It has to do with the semantics of leave. With a leave you are basically telling Consul "please announce to the cluster your intention to leave, and do not re-join this cluster again unless explicitly told to.". As part of that, the node leaves the raft replication group and broadcasts a gossip message with its intent to leave. When we leave the raft group, the peers are purposely set to |
@armon I understand you, but confused because I thought I'm explicit by setting the start_join in the config. Why the gossip layer is allowed to join together and only the raft peerset is handled differently in this case? Is there a config flag I should use to avoid this, or simply avoid leaving for server nodes? |
@pepov The gossip layer is simpler since joins are idempotent and there is no issues of data safety. The raft layer is much more sensitive, and hence a little more challenging to work with. Typically, you avoid this by never having servers leave. It's rather strange operationally to have servers come and go. |
Maybe I misunderstood your comments. After reading this thread and #454, it seems recovery after planed or unplanned failure was not designed to be covered. This is probably not a good assumption to design a HA system. Virtual instances and bare metal deployment would have all sort of issues. That assumption has no differences between deploying to a single node, it seems to me that this defeats the purpose of building a distributed system... |
When updating to a new version of consul, the servers are leaving, which is expected, correct? I accidentally updated all 3 service simultaneously and ran into this issue and could not get the cluster to bootstrap until I created an atlas account. |
@awheeler If you shutdown the servers gracefully they will leave the cluster and not re-join when you restart. You'll need to |
I ran into this same issue today. Is there no way to gracefully take down an entire Consul cluster and gracefully bring it back up? I am trying to run How do we, for example, reboot an experimental Consul cluster? Process termination seems nasty. |
@theonewolf The experience around this can definitely be improved as Consul was designed for continuous operation. It really dislikes being stopped so the notion of a graceful shutdown of an entire cluster really wasn't planned for. There is handling of a disaster total failure (power loss), but not as something people intend to do. That said, you can just |
I can confirm kill -9 works well for now. I have exercised to kill and recover server or agent under failure scenario (e.g. dc power cut, server crash)
|
@armon I see. I think my/others confusion comes in because of incomplete documentation. I didn't realize ctrl-c took a node out of the cluster in a deep sense. But the design makes sense to me. My confusion came in when I see things like "Graceful shutdown"----> which I took to imply a possibility for a "Graceful startup" and rejoining of the cluster even if all nodes were brought down. So for upgrades and resets we should be sending a different signal to the Consul process? I think this signal, or all of the signals Consul expects and the outcomes of those signals should be very well documented somewhere. Maybe a table listing the signals and Consul states post-signal? |
@theonewolf Partially, I think this was a mistake UX wise. I think that the "deep remove" should not have been the default behavior, but we are here now. Its a damned if you do, damned if you don't situation. The problem with the signals table is that the behavior is configurable. It responds to 3 signals:
|
I see! This does lead to the potential for a healthy lifecycle if the signals are used appropriately. I still think such a table should exist. I'd like to know all the external run-time events that could affect my running Consul processes. Even if an engineer initiated them by accident for example. |
What if I accidentally "gracefully shutdown the whole cluster"? Is there any way to recover? |
It's a bit old thread, but I have confusing results regarding cluster recovery:
i use version 6.3 |
Hi @Dmitry1987 that definitely should not be the case. Can you share a gist with your configuration and some log messages from when this happens? |
Hi @slackpad , i think i figured this out, by adding some parameters that say "do not gracefully leave in any case, hard/soft kill" , don't remember exactly, but problematic configuration BEFORE i found that solution was: { "acl_datacenter": "dc-main", |
I don't remember exactly those 2 directives, but it was 2 settings that say to not delete himself from raft peers file, no matter which kill signal received (or "consul leave"). after that it was working good. |
I'm going to close this out. Here's what we've done related to this:
|
@slackpad could you reference the most recent release with these features? Thanks! |
@theonewolf 0.8.1 has them all :-) They came in 0.7.0, though, so 0.7.5 is fine too if you are on that series. |
@slackpad awesome. We are 0.7.5. |
I've observed that I'm unable to start a gracefully stopped server cluster (using the official 0.5 build). It seems that
peers.json
content will be set tonull
during a graceful leave on the nodes which affects a new bootstrapping process in a bad way. I've managed to start the cluster only by setting the proper value inpeers.json
after the agent has been stopped.Is this a bug or am I missing something?
The text was updated successfully, but these errors were encountered: