Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ebs br: mark backup/restore failed when job failed #5332

Merged
merged 38 commits into from
Nov 23, 2023
Merged

ebs br: mark backup/restore failed when job failed #5332

merged 38 commits into from
Nov 23, 2023

Conversation

WangLe1321
Copy link
Contributor

@WangLe1321 WangLe1321 commented Oct 13, 2023 •
edited
Loading

What problem does this PR solve?

  1. Fix the bug that when resume GC and scheduling normally, but backup failed due to pod failure.
    Closes ebs br: VolumeBackup failed due to resuming GC and scheduling #5331

  2. Fix the bug that restore pod fails due to the container being killed, but restore status doesn't set failed.
    Closes ebs br: When the restore data pod fails, the volumerestore task status is not set to failed #5293

  3. Raise error when deleting EBS snapshot failed except snapshot not found, so that we can block backup deletion when EBS snapshots aren't all be deleted.

  4. Close When checking whether warmup has ended, if there are warmup jobs in other namespaces, it will interfere with the current results. #5405

What is changed and how does it work?

For 1 and 2, watch job status in backup/restore controller. When job is failed but backup/restore status isn't failed, set status failed.

For 3, if encounter error except not found error, return the error.

Code changes

  • Has Go code change
  • Has CI related scripts change

Tests

  • Unit test
  • E2E test
  • Manual test
  • No code

Side effects

  • Breaking backward compatibility
  • Other side effects:

Related changes

  • Need to cherry-pick to the release branch
  • Need to update the documentation

Release Notes

Please refer to Release Notes Language Style Guide before writing the release note.

@ti-chi-bot ti-chi-bot bot requested a review from lichunzhu October 13, 2023 10:02
@ti-chi-bot ti-chi-bot bot added the size/L label Oct 13, 2023
@codecov-commenter
Copy link

codecov-commenter commented Oct 16, 2023 •
edited
Loading

Codecov Report

Merging #5332 (421f44e) into master (4317c34) will increase coverage by 0.05%.
The diff coverage is 36.17%.

Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #5332      +/-   ##
==========================================
+ Coverage   61.61%   61.66%   +0.05%     
==========================================
  Files         228      239      +11     
  Lines       28928    32712    +3784     
==========================================
+ Hits        17823    20172    +2349     
- Misses       9358    10689    +1331     
- Partials     1747     1851     +104
Flag Coverage Δ
e2e 21.80% <24.71%> (?)
unittest 61.58% <37.80%> (-0.04%) ⬇️

@WangLe1321
Copy link
Contributor Author

/run-pull-e2e-kind

@WangLe1321
Copy link
Contributor Author

/run-pull-e2e-kind-across-kubernetes

@BornChanger BornChanger mentioned this pull request Oct 16, 2023
Open
@WangLe1321
Copy link
Contributor Author

run-pull-e2e-kind-br

@csuzhangxc
Copy link
Member

/run-pull-e2e-kind-br

@BornChanger
Copy link
Contributor

/test-pull-e2e-kind-br

@BornChanger
Copy link
Contributor

/run-pull-e2e-kind-br

@ti-chi-bot ti-chi-bot bot added size/XL and removed size/L labels Oct 20, 2023
@ti-chi-bot ti-chi-bot bot added the lgtm label Oct 20, 2023
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Oct 20, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: BornChanger

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot added the approved label Oct 20, 2023
@ti-chi-bot ti-chi-bot bot removed the lgtm label Oct 20, 2023
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Oct 20, 2023

[LGTM Timeline notifier]

Timeline:

  • 2023-10-20 07:55:21.6345874 +0000 UTC m=+1989319.221697545: ☑️ agreed by BornChanger.
  • 2023-10-20 09:26:19.779858153 +0000 UTC m=+1994777.366968299: ✖️🔁 reset by ti-chi-bot[bot].

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Oct 20, 2023

New changes are detected. LGTM label has been removed.

@WangLe1321
Copy link
Contributor Author

/run-pull-e2e-kind-br

@WangLe1321
Copy link
Contributor Author

/run-pull-e2e-kind-br

@WangLe1321
Copy link
Contributor Author

/run-pull-e2e-kind-br

@WangLe1321
Copy link
Contributor Author

/run-pull-e2e-kind-br

@WangLe1321
Copy link
Contributor Author

/run-pull-e2e-kind-br

7 similar comments
@WangLe1321
Copy link
Contributor Author

/run-pull-e2e-kind-br

@WangLe1321
Copy link
Contributor Author

/run-pull-e2e-kind-br

@WangLe1321
Copy link
Contributor Author

/run-pull-e2e-kind-br

@WangLe1321
Copy link
Contributor Author

/run-pull-e2e-kind-br

@WangLe1321
Copy link
Contributor Author

/run-pull-e2e-kind-br

@WangLe1321
Copy link
Contributor Author

/run-pull-e2e-kind-br

@WangLe1321
Copy link
Contributor Author

/run-pull-e2e-kind-br

@csuzhangxc csuzhangxc merged commit afb5531 into pingcap:master Nov 23, 2023
6 of 7 checks passed
@csuzhangxc
Copy link
Member

/cherry-pick release-1.5

@ti-chi-bot
Copy link
Member

@csuzhangxc: new pull request created to branch release-1.5: #5412.

In response to this:

/cherry-pick release-1.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot ti-chi-bot mentioned this pull request Nov 23, 2023
Merged
10 tasks
ti-chi-bot pushed a commit to ti-chi-bot/tidb-operator that referenced this pull request Nov 23, 2023
@WangLe1321 @ti-chi-bot
This is an automated cherry-pick of pingcap#5332
f2c73b1 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
csuzhangxc pushed a commit that referenced this pull request Nov 23, 2023
@ti-chi-bot @WangLe1321
ebs br: mark backup/restore failed when job failed (#5332) (#5412)
8f5e433 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
Co-authored-by: WangLe1321 <wangle1321@163.com>
michaelmdeng pushed a commit to michaelmdeng/tidb-operator that referenced this pull request Nov 29, 2023
@WangLe1321
ebs br: mark backup/restore failed when job failed (pingcap#5332)
373a3cd
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@BornChanger BornChanger BornChanger approved these changes

@lichunzhu lichunzhu Awaiting requested review from lichunzhu

Assignees
No one assigned
Labels
approved area/ebs-br size/XL
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

ebs br: VolumeBackup failed due to resuming GC and scheduling
5 participants
@WangLe1321 @codecov-commenter @csuzhangxc @BornChanger @ti-chi-bot