Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix for vttablet going unhealthy after failed backup #4805

Merged
merged 1 commit into from
Apr 16, 2019

Conversation

deepthi
Copy link
Member

@deepthi deepthi commented Apr 12, 2019

use background context to restart mysqld after backup. fix and add logging.
Fixes #4538

Signed-off-by: deepthi deepthi@planetscale.com

Copy link
Member

@derekperkins derekperkins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM


// Starting from here we won't be able to recover if we get stopped by a cancelled
// context. Thus we use the background context to get through to the finish.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you create a separate ctx := context.Background() here and use it in both places.
Ideally, they say you have to use context.TODO() (because you're not supposed to create background contexts outside of main.go). But I'm not very hung up on that rule.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. I squashed it with the previous commit.

…gging

Signed-off-by: deepthi <deepthi@planetscale.com>
@sougou sougou merged commit c19fd32 into vitessio:master Apr 16, 2019
setassociative added a commit to tinyspeck/vitess that referenced this pull request Apr 26, 2019
It's been busy and exciting in vitess upstream. Some changes that felt worth calling out as they could impact assumptions about behavior:

- vitessio#4832 — this adds a timeout that is 3x the `healthCheckInterval` which at which point the tablet will stop reporting as healthy
- vitessio#4814 — this changes when command line args are used by vttablet
- vitessio#4811 — a planned reparent now sets `super_read_only`
- vitessio#4805 — behavior coming back from a failed backup is different

**Slack changes**  
        @demmer vitessio#4827 Correct suppress logging for begin...commit in autocommit
        @rafael vitessio#4824 adds timeouts for all statements
        @demmer vitessio#4826 adds logging of stack traces as opt-in
        @demmer vitessio#4819 remove begin/commit logs from autocommit txns
        @demmer vitessio#4796 improves support for vtexplain for begin/dml/dml/commit txns

**Non slack changes**  
        vitessio#4839 Improve behavior with reference table routing & vreplication
        vitessio#4833 Support query routing given there could be multiple targets for a table
        vitessio#4832 tablets get new health check behavior (health checks time out)
        vitessio#4837 refresh bug around row streamer handling creds
        vitessio#4830 apply a default value to the db_name attribute in local_metadata
        vitessio#4785 SHOW SCHEMAS aliased to SHOW DATABASES
        vitessio#4829 add experimental support for split clone & vertical split clone via vreplication
        vitessio#4822 do not normalize in order by
        vitessio#4791 Adds vreplication row streaming support
        vitessio#4814 command line flags now used regardless of management mode
        vitessio#4811 vttablet sets super_read_only during planned reparent
        vitessio#4803 xtrabackup testing
        vitessio#4727 support multiple vttablets running against one mysql instance
        vitessio#4746 TopoCat can now produce JSON output
        vitessio#4805 mysqld is now restarted after a failed backup
        vitessio#4685 macos bootstrap bug
        vitessio#4874 ZK opts and java land
        vitessio#4695 Adds support for xtrabackup
        vitessio#4794 fixes build failures
        vitessio#4725 changes mysql8.0 start args
        vitessio#4736 introduce new states to support vreplication
        vitessio#4788 expands orc error

**Docs etc**  
        vitessio#4831 lint
        vitessio#4827 docs
        vitessio#4816 const declaration cleanup
        vitessio#4820 const declaration cleanup
        vitessio#4825 docs
        vitessio#4818 docs
        vitessio#4809 docs
        vitessio#4812 moves consts around
        vitessio#4813 docs
        vitessio#4808 docs
        vitessio#4800 docs
        vitessio#4795 docs
@deepthi deepthi deleted the ds-4538 branch July 6, 2020 19:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Failed backup puts tablet into unhealthy state
3 participants