Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle the EOF/Context canceled error caused by TSO Follower Proxy disabling #4249

Open
JmPotato opened this issue Oct 29, 2021 · 0 comments
Labels
component/client Client logic. component/tso Timestamp Oracle. type/enhancement The issue or PR belongs to an enhancement.

Comments

@JmPotato
Copy link
Member

Feature Request

Describe your feature request related problem

Now we can enable/disable TSO Follower Proxy online, however, when we change it from on to off, there are two corner cases that will occur.

EOF/Context canceled Error

c.tryConnectWithProxy will rebuild the connectionCtxs and cancel the old stream, if we have an ongoing TSO gRPC request, this will cause an EOF/Context canceled error.

Old closed stream chosen

Because we will randomly choose a stream and wait for the incoming TSO request from a channel, so if any connectionCtxs rebuilding happens after the stream choosing, it will make the TSO requests in the future fail due to the chosen stream being closed.

These two cases won't affect the TiDB now because it has the TSO backoff strategy. However, I think it's still worth being fixed.

Describe the feature you'd like

Enabling/Disabling the TSO Follower Proxy online won't affect the TSO request at all. Everything will run smoothly with no error occurring.

@JmPotato JmPotato added type/enhancement The issue or PR belongs to an enhancement. component/client Client logic. component/tso Timestamp Oracle. labels Oct 29, 2021
JmPotato added a commit to JmPotato/pd that referenced this issue Oct 31, 2021
Signed-off-by: JmPotato <ghzpotato@gmail.com>
ti-chi-bot added a commit that referenced this issue Dec 6, 2021
…4251)

* Fix the case where the selected TSO stream was cancelled (ref #4249)

Signed-off-by: JmPotato <ghzpotato@gmail.com>

* Delete a TSO stream if it's canceled

Signed-off-by: JmPotato <ghzpotato@gmail.com>

* Unify the spelling

Signed-off-by: JmPotato <ghzpotato@gmail.com>

* Short the sleep time

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/client Client logic. component/tso Timestamp Oracle. type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Development

No branches or pull requests

1 participant