Snapshot backup stuck when EBS snapshot failed

[michaelmdeng]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. Create a VolumeBackup (either adhoc or via schedule)
2. If underlying EBS snapshot encounters issue, then data plane backup job for that snapshot will be stuck with log

2024-04-29, 23:06:09.295		
I0430 06:06:09.295599       9 backup.go:302] [2024/04/30 06:06:09.295 +00:00] [INFO] [ebs.go:241] ["check pending snapshots"] [count=3]
...

2. What did you expect to see? (Required)

backup job, Backup, and VolumeBackup should fail. We don't expect these failures to resolve (in fact there is separate status for recoverable errors)

3. What did you see instead (Required)

backup job runs indefinitely until manual cleanup

4. What is your TiDB version? (Required)

v6.5.4

[YuJuncen]

(This may affect 7.1 and 8.1 also, but let's don't add the label until we need, to avoid generating many cherry-picking PRs.)