-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[0.9.4-nightly] possible "catch up" bug with nodes down in a cluster #3960
Comments
I have noticed the same actually. It appears from logs that a whole slew of connections from the cluster members that were up to the dead node occur, writes occur, and are then closed. However it doesn't seem that the member ever actually gets back into a syncd state. I verified the raft key is set to true on all servers in the cluster by looking at "show servers" |
@jwilder something amiss with hinted handoff? |
@beckettsean @iansmith I was able to repro the issue. Here are the steps
Ran the test twice and each time the node that was brought down is missing points. I'm going to let the system sit for a while and see if it's a timing issue. |
After letting it sit for a while it seems as if the points are still missing. |
Also one thing to note is that the number of missing points is very clean. When I wrote 2 million points the number of points that the third node was reporting was |
It sounds like entire batches are failing to replicate, not just individual points within a batch. @jwilder seems like something is amiss with hinted handoff. What other testing can we do or what details can we provide to help characterize it? |
I'm able to reproduce some issues and am looking into it. |
The meta-store would open but may not have finished loading the raft log. If write requests came in, they could fail or be dropped because of missing shard group info. This change makes the meta store only return after it has found the leader and is really ready. This change also fixed a race in the ClusterRestart test that may be causing it to fail sporadically. Fixes #3677 #3960
The mux listener was handling connections and demux serially. This could cause issues if one handler was slow or blocked. For example, if a node had many hinted handed writes queued for a down node, when the down node was started, it would start handling the write requests (possibly before it had synchronized with the cluster). This would cause the connectiosn to block and also prevent the cluster synchronization from occuring because they both use the same mux listener. Fixes #3960
woo! |
2 days from bug report to fix! Awesome! 🏆 |
I'm testing 0.9.4.1 and I'm not seeing the node that was down during the writes ever recovering. Should I re-open? |
[I'm pretty new to influx, so please close if this is silly.]
I'm running a cluster of 3 nodes, per the instructions in the docs on how to create a cluster, on Ubuntu 14.10:
These are all amazon ec2 instances, connected via a security group. I wanted to see if a writes to working nodes A or C in the cluster while node B was down would eventually get "corrected" when B returns. Can I run a query after B is brought back into the cluster and get the same answers from A, B, or C?
So, I used this script to write 10K points, carefully avoiding one of the nodes because I took it down immediately after database creation (I wasn't sure if db creation would affect anything):
Then a brought the node back up after this script had finished.
In the log of the leader, I saw this when I brought up the node that had been previously down:
which seemed good to me.
Then I used the script below to read back the data I had just written, using all three of the nodes as sources. The two that had been up throughout worked fine. The one that had been down during the writes gave weird values, suggesting that it had about 4700 values out of the expected 10,000.
Perhaps I'm just confused about how this is supposed to work? That seems the most likely explanation.
thanks
The text was updated successfully, but these errors were encountered: