-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Manage MySQL Replication Status States Properly #9853
Conversation
…failed Signed-off-by: Matt Lord <mattalord@gmail.com>
…ationStatus Signed-off-by: Matt Lord <mattalord@gmail.com>
Signed-off-by: Matt Lord <mattalord@gmail.com>
Signed-off-by: Matt Lord <mattalord@gmail.com>
Signed-off-by: Matt Lord <mattalord@gmail.com>
And make function names nicer (removing double context) And add some helper functions Signed-off-by: Matt Lord <mattalord@gmail.com>
Signed-off-by: Matt Lord <mattalord@gmail.com>
Signed-off-by: Matt Lord <mattalord@gmail.com>
Update vtadmin web protobufs Signed-off-by: Matt Lord <mattalord@gmail.com>
@deepthi, @GuptaManan100, and @shlomi-noach: I apologize for the review hassle but unfortunately creating a new PR was the only way forward (see here for details). In the process, I consolidated the changes since the last round of reviews — focused on safely modifying the protobuf to fit the new model — into a single commit: 65226ad Thanks in advance! 🙇♂️ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I feel sad that you had to jump through so many hoops for getting this in 😞
As release-13.0 does not have this: vitessio#9853 Signed-off-by: Matt Lord <mattalord@gmail.com>
…ded (#10123) * Only start SQL thread temporarily to WaitForPosition if needed (#10104) After #9512 we always attempted to start the replication SQL_Thread(s) when waiting for a given position. The problem with this, however, is that if the SQL_Thread is running but the IO_Thread is not then the tablet repair does not try and start replication on a replica tablet. So in certain states such as when initializing a shard, replication may end up in a non-healthy state and never be repaired. This changes the behavior so that: 1. We only attempt to start the SQL_Thread(s) if it's not already running 2. If we explicitly start the SQL_Thread(s) then we also explicitly reset it to what it was (stopped) as we exit the call Because the caller should be/have a TabletManager which has a mutex, this should ensure that the replication manager calls are serialized and because we are resetting the replication state after mutating it, everything should work as it did before #9512 with the exception being that when waiting we ensure that the replica at least has the possibility of catching up. Signed-off-by: Matt Lord <mattalord@gmail.com> * Use older replication status interface As release-13.0 does not have this: #9853 Signed-off-by: Matt Lord <mattalord@gmail.com>
Description
We considered the
Slave_IO_Running
state ofConnecting
as equivalent toRunning
in the replication status results that we get from MySQL. I'm assuming this was done to avoid flapping on low traffic systems due to the-slave_net_timeout
reconnects and avoid doing a tablet repair when replication was healthy (with this PR we can distinguish running from healthy states and take differing actions depending on what we want).After #9308 we properly estimate the replica lag when MySQL is telling us that it does not know, meaning it returns a
NULL
value for theSeconds_Behind_Master
field. But in some cases — e.g. it appears to happen when attempting the first connection to the source — MySQL will report aSeconds_Behind_Master
value of0
(meaning fully caught up and no lag) even when it is not connected to its replication source and has failed to reconnect — meaning that this is not a simple reconnect for any reason (e.g.-slave_net_timeout
), but a reconnect with one or more failures/errors. This PR handles this latter case within Vitess by continuing to treatConnecting
as equivalent toRunning
viaReplicationStatus.Healthy()
— to prevent the noted flapping and errant/unnecessary tablet repairs — unless we had an IO error the last time we tried to reconnect to the replication source.Related Issue(s)
Checklist