fix backup failed when pod was auto restarted by k8s #4883
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
[REVIEW NOTIFICATION] This pull request has been approved by:
To complete the pull request process, please ask the reviewers in the list to review by filling The full list of commands accepted by this bot can be found here. Reviewer can indicate their review by submitting an approval review. |
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## master #4883 +/- ##
==========================================
- Coverage 59.43% 52.75% -6.69%
==========================================
Files 226 205 -21
Lines 25694 25836 +142
==========================================
- Hits 15272 13630 -1642
- Misses 8966 10646 +1680
- Partials 1456 1560 +104
|
WizardXiao
changed the title
WizardXiao
changed the title
WizardXiao
changed the title
WangLe1321
reviewed
Feb 20, 2023
fengou1
approved these changes
Feb 20, 2023
|
/test pull-e2e-kind pull-e2e-kind-tikv-scale-simultaneously |
|
/test pull-e2e-kind pull-e2e-kind-across-kubernetes pull-e2e-kind-basic pull-e2e-kind-serial pull-e2e-kind-tngm |
|
/test pull-e2e-kind pull-e2e-kind-across-kubernetes pull-e2e-kind-basic pull-e2e-kind-serial |
|
/test pull-e2e-kind pull-e2e-kind-across-kubernetes |
|
/test pull-e2e-kind-across-kubernetes |
WizardXiao
added a commit
to WizardXiao/tidb-operator
that referenced
this pull request
Mar 10, 2023
* init code for test * just clean before backup data * delete test code * import pingcap/errors * add check version * remove test code * add running status check * add restart condition to clarify logic * fix status update * fix ut
10 tasks
WizardXiao
added a commit
that referenced
this pull request
Mar 11, 2023
* feat: support tiflash backup and restore during volume snapshot (#4812) * feat: calc the backup size from snapshot storage usage (#4819) * fix backup failed when pod was auto restarted by k8s (#4883) * init code for test * just clean before backup data * delete test code * import pingcap/errors * add check version * remove test code * add running status check * add restart condition to clarify logic * fix status update * fix ut * br: ensure pvc names sequential for ebs restore (#4888) * BR: Restart backup when backup job/pod unexpected failed by k8s (#4895) * init code for test * just clean before backup data * delete test code * import pingcap/errors * add check version * remove test code * add running status check * add restart condition to clarify logic * fix status update * fix ut * init code * update crd reference * fix miss update retry count * add retry limit as constant * init runnable code * refine main controller logic * add some note * address some comments * init e2e test code * add e2e env to extend backup time * add e2e env for test * fix complie * just test kill pod * refine logic * use pkill to kill pod * fix reconcile * add kill pod log * add more log * add more log * try kill pod only * wait and kill running backup pod * add wait for pod failed * fix wait pod running * use killall backup to kill pod * use pkill -9 backup * kill pod until pod is failed * add ps to debug * connect commands by semicolon * kill pod by signal 15 * use panic simulate kill pod * test all kill pod test * remove useless log * add original reason of job or pod failure * rename BackupRetryFailed to BackupRetryTheFailed * BR: Auto truncate log backup in backup schedule (#4904) * init schedule log backup code * add run log backup code * update api * refine some nodes * refine cacluate logic * add ut * fix make check * add log backup test * refine code * fix notes * refine function names * fix conflict * fix: add a new check for encryption during the volume snapshot restore (#4914) * br: volume-snapshot may lead to a panic when there is no block change between two snapshot (#4922) * br: refine BackoffRetryPolicy time format (#4925) * refine BackoffRetryPolicy time format * fix some ut --------- Co-authored-by: fengou1 <85682690+fengou1@users.noreply.github.com> Co-authored-by: WangLe1321 <wangle1321@163.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Ref #4805
What is changed and how does it work?
Code changes
Tests
k8s may restart backup pod which is not done.
detail test
drain node test
worker2, drainworker2, backup restart atworker3by k8s.delete pod test
Side effects
Related changes
Release Notes
Please refer to Release Notes Language Style Guide before writing the release note.