Stop the running log backup when deleting CR

[RidRisR]

What problem does this PR solve?

In the previous operator version, delete backup CR will do no clean up on running log backup.
Users may mis-operator and find it hard to running task.

Closes #5249

What is changed and how does it work?

1. Add enum for cleanPolicy, make input standardized.
   Now, only Retain;OnFailure;Delete are accepted as cleanPolicy input, and default is Retain.
   (This is actually no change to user behavior.)

2. Add finalizer and clean logic check for on-going log backup
   All log backup will have finalizer now. Operator will check if the log backup "on track". If so, a new task to stop log backup will run.
   The "on track" state is BackupScheduled || BackupPrepare || BackupRunning || BackupPaused || BackupStop. We lack the ability to check whether the log backup should be stopped if it failed. If the task is on fail states, we do not provide auto cleaning.

Code changes

• Has Go code change
• Has CI related scripts change

Tests

• Unit test
• E2E test
• Manual test
• No code

Side effects

• Breaking backward compatibility
• Other side effects:

Related changes

• Need to cherry-pick to the release branch
• Need to update the documentation

Release Notes

Please refer to Release Notes Language Style Guide before writing the release note.

1. Add enum for cleanPolicy, make input standardized.
   Now, only Retain;OnFailure;Delete are accepted as cleanPolicy input, and Retain is default.
   This is actually no change to user behavior.
2. Add finalizer and clean logic check for on-going log backup
    All log backup will have finalizer now. Operator will check if the log backup "on track". If so, a new task to stop log backup will run.
    The "on track" state is `BackupScheduled || BackupPrepare || BackupRunning || BackupPaused`. 
    We lack the ability to check whether the log backup should be stopped if it failed. If the task is on states, we do not provide auto cleaning.

[RidRisR]

/run-pull-e2e-kind-br

[RidRisR]

/run-pull-e2e-kind-br

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 63.07692% with 96 lines in your changes missing coverage. Please review.

Project coverage is 20.78%. Comparing base (9ef26f8) to head (be29feb).
Report is 29 commits behind head on master.

❗ There is a different number of reports uploaded between BASE (9ef26f8) and HEAD (be29feb). Click for more details.

HEAD has 1 upload less than BASE

Flag	BASE (9ef26f8)	HEAD (be29feb)
unittest	1	0

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #5754       +/-   ##
===========================================
- Coverage   61.47%   20.78%   -40.69%     
===========================================
  Files         235      219       -16     
  Lines       30653    30965      +312     
===========================================
- Hits        18843     6436    -12407     
- Misses       9920    23616    +13696     
+ Partials     1890      913      -977     

Flag	Coverage Δ
e2e	20.78% <63.07%> (?)
unittest	?

[RidRisR]

/run-pull-e2e-kind-br

[RidRisR]

/run-pull-e2e-kind-br

[RidRisR]

/run-pull-e2e-kind-br

[RidRisR]

/run-pull-e2e-kind-br

[csuzhangxc]

/run-pull-e2e-kind-br

[RidRisR]

/run-pull-e2e-kind-br

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2024-10-16 07:07:35.665684613 +0000 UTC m=+427652.814594437: ☑️ agreed by WangLe1321.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: csuzhangxc, WangLe1321

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [WangLe1321,csuzhangxc]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[csuzhangxc]

/cherry-pick release-1.6

[csuzhangxc]

/cherry-pick release-1.5

[ti-chi-bot]

@csuzhangxc: new pull request created to branch release-1.6: #5765.

In response to this:

/cherry-pick release-1.6

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[ti-chi-bot]

@csuzhangxc: new pull request created to branch release-1.5: #5766.

In response to this:

/cherry-pick release-1.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.