Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add timeout for mysqld_shutdown #6849

Merged
merged 10 commits into from
Oct 22, 2020

Conversation

ajm188
Copy link
Contributor

@ajm188 ajm188 commented Oct 10, 2020

Resolves #6848

Notable changes:

  • hooks can now be passed contexts via the ExecuteContext method. All hooks are now implemented with exec.CommandContext under the hood instead of exec.Command. Current callers are unaffected, and can opt in to switching over at their discretion.
  • adds a new HookResult.ExitStatus type for hooks that exceed their context deadline/timeout.
  • FakeMysqlDaemon.Start() and FakeMysqlDaemon.Shutdown() are configurable via the FakeMysqlDaemon.StartupTime and FakeMysqlDaemon.ShutdownTime respectively, to artificially cause delay in tests.

…down before aborting

Signed-off-by: Andrew Mason <amason@slack-corp.com>
Signed-off-by: Andrew Mason <amason@slack-corp.com>
Signed-off-by: Andrew Mason <amason@slack-corp.com>
Signed-off-by: Andrew Mason <amason@slack-corp.com>
…sses

Signed-off-by: Andrew Mason <amason@slack-corp.com>
Signed-off-by: Andrew Mason <amason@slack-corp.com>
@@ -182,7 +187,9 @@ func (be *BuiltinBackupEngine) ExecuteBackup(ctx context.Context, params BackupP
params.Logger.Infof("using replication position: %v", replicationPosition)

// shutdown mysqld
err = params.Mysqld.Shutdown(ctx, params.Cnf, true)
shutdownCtx, cancel := context.WithTimeout(ctx, *builtinBackupMysqldDeadline)
err = params.Mysqld.Shutdown(shutdownCtx, params.Cnf, true)
Copy link
Contributor Author

@ajm188 ajm188 Oct 10, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see an easy way to test this. I can't use fakemysqldaemon to create a mysqld that hangs on shutdown, because that package imports this package, which causes circular import errors. 🤔

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solved this with blackbox testing ✅

@ajm188 ajm188 marked this pull request as ready for review October 10, 2020 15:51
Signed-off-by: Andrew Mason <amason@slack-corp.com>
Signed-off-by: Andrew Mason <amason@slack-corp.com>
…behavior

Signed-off-by: Andrew Mason <amason@slack-corp.com>
Copy link
Member

@deepthi deepthi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work! Just the one comment.

@@ -47,6 +48,13 @@ const (
dataDictionaryFile = "mysql.ibd"
)

var (
// BuiltinBackupMysqldDeadline is how long ExecuteBackup should wait for response from mysqld.Shutdown.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typically deadline is used for timestamps and timeout for durations. This looks like a timeout.

Signed-off-by: Andrew Mason <amason@slack-corp.com>
@ajm188 ajm188 requested a review from morgo as a code owner October 19, 2020 21:24
@deepthi deepthi merged commit d3f66bd into vitessio:master Oct 22, 2020
@deepthi
Copy link
Member

deepthi commented Oct 22, 2020

@ajm188 would you please create another PR to merge this into release-8.0?

ajm188 pushed a commit to tinyspeck/vitess that referenced this pull request Oct 22, 2020
…meout

Add timeout for mysqld_shutdown

Signed-off-by: Andrew Mason <amason@slack-corp.com>
deepthi added a commit that referenced this pull request Oct 22, 2020
Backport pull request #6849 (add timeout for mysqld_shutdown) to 8.0
@ajm188 ajm188 deleted the am_mysqld_shutdown_timeout branch January 14, 2021 15:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Wedged mysqld shutdown delays error handling in builtinbackup.ExecuteBackup
2 participants