br: clean volumes when restore volume failed (#5634)

[ti-chi-bot]

This is an automated cherry-pick of #5634

What problem does this PR solve?

Add cleaning volumes function when restore volume failed. Then we can avoid volume leak

Closes #5638

What is changed and how does it work?

Code changes

• Has Go code change
• Has CI related scripts change

Tests

• Unit test
• E2E test
• Manual test
• No code

manual test steps:

1. create a cluster
2. create a volume backup
3. create a restore cluster
4. create a volume restore using the volume backup in step 2. in this step, restore volume is successful and the tikv pods are started
5. edit the restore CR, modify its status from VolumeComplete to Failed to mock restore volume failed scenario
6. edit the tc CR, remove the annotation tidb.pingcap.com/tikv-volumes-ready to block tikv creation, then delete the tikv statefulset to detach the EBS volumes
7. wait for the volumes detached and deleted by tidb-operator

Side effects

• Breaking backward compatibility
• Other side effects:

Related changes

• Need to cherry-pick to the release branch
• Need to update the documentation

Release Notes

Please refer to Release Notes Language Style Guide before writing the release note.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign tennix for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment