Backport: Only start SQL thread temporarily to WaitForPosition if needed #10123
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…sio#10104) After vitessio#9512 we always attempted to start the replication SQL_Thread(s) when waiting for a given position. The problem with this, however, is that if the SQL_Thread is running but the IO_Thread is not then the tablet repair does not try and start replication on a replica tablet. So in certain states such as when initializing a shard, replication may end up in a non-healthy state and never be repaired. This changes the behavior so that: 1. We only attempt to start the SQL_Thread(s) if it's not already running 2. If we explicitly start the SQL_Thread(s) then we also explicitly reset it to what it was (stopped) as we exit the call Because the caller should be/have a TabletManager which has a mutex, this should ensure that the replication manager calls are serialized and because we are resetting the replication state after mutating it, everything should work as it did before vitessio#9512 with the exception being that when waiting we ensure that the replica at least has the possibility of catching up. Signed-off-by: Matt Lord <mattalord@gmail.com>
mattlord
added
Type: Bug
Component: Query Serving
Backport
mattlord
requested review from
deepthi,
harshit-gangal and
systay
as code owners
mattlord
requested review from
GuptaManan100
and removed request for
systay and
harshit-gangal
GuptaManan100
approved these changes
Apr 23, 2022
As release-13.0 does not have this: vitessio#9853 Signed-off-by: Matt Lord <mattalord@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
After #9512 we always attempted to start the replication
SQL_Thread(s) when waiting for a given position. The problem with this, however, is that if theSQL_Threadis running but theIO_Threadis not then the tablet repair does not try and start replication on a replica tablet. So in certain states such as when initializing a shard, replication may end up in a non-healthy state and never be repaired.This changes the behavior so that:
SQL_Thread(s) if it's not already runningSQL_Thread(s)then we also explicitly reset it to what it was (stopped) as we exit the callBecause the caller should be/have a TabletManager which has a mutex, this should ensure that the replication manager calls are serialized and because we are resetting the replication state after mutating it, everything should work as it did before #9512 with the exception being that when waiting we ensure that the replica at least has the possibility of catching up.
Related Issue(s)
Checklist