Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

vttablet should not shutdown on "Too many connections" error #7038

Closed
sougou opened this issue Nov 14, 2020 · 0 comments · Fixed by #7039
Closed

vttablet should not shutdown on "Too many connections" error #7038

sougou opened this issue Nov 14, 2020 · 0 comments · Fixed by #7039
Labels
Component: Query Serving Type: Enhancement Logical improvement (somewhere between a bug and feature)

Comments

@sougou
Copy link
Contributor

sougou commented Nov 14, 2020

If too many queries get killed, mysql can temporarily return a "Too many connections" error until it has accounted for all the killed connections.

In vttablet, this gets treated as a connection error, which causes the tablet server to go into a non-serving state. In this situation, it's better if vttablet continues serving with the outstanding connections instead of shutting down.
@sougou sougou added Component: Query Serving P2 Type: Enhancement Logical improvement (somewhere between a bug and feature) labels Nov 14, 2020
sougou added a commit to planetscale/vitess that referenced this issue Nov 14, 2020
@sougou
vttablet: don't shutdown on too many connections
00ec737 
Fixes vitessio#7038

This change makes IsConnErr treat "Too man connections" as a normal
error, allowing vttablet to continue serving. The same check is also
added to IsMySQLReachable in case it happens to get called through
a different code path.

The error code returned by mysql (2012) is more generic. So, we also
have to check for the specific substring.

It's hard to reliably test this using automated testing. Instead, I
manually created a state where mysql ran out of connections, verified
that vttablet did shut down, and also verified that vttablet continued
to serve after this fix.

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>
@sougou sougou mentioned this issue Nov 14, 2020
Merged
gedgar pushed a commit to planetscale/vitess that referenced this issue Feb 18, 2021
@sougou @gedgar
vttablet: don't shutdown on too many connections
7375935 
Fixes vitessio#7038

This change makes IsConnErr treat "Too man connections" as a normal
error, allowing vttablet to continue serving. The same check is also
added to IsMySQLReachable in case it happens to get called through
a different code path.

The error code returned by mysql (2012) is more generic. So, we also
have to check for the specific substring.

It's hard to reliably test this using automated testing. Instead, I
manually created a state where mysql ran out of connections, verified
that vttablet did shut down, and also verified that vttablet continued
to serve after this fix.

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

Signed-off-by: Gary Edgar <gary@planetscale.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Component: Query Serving Type: Enhancement Logical improvement (somewhere between a bug and feature)
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

vttablet: don't shutdown on too many connections planetscale/vitess
1 participant
@sougou