[Core feature] Add an option to consider deleted cache as cache miss.

[amitani]

Motivation: Why do you think this is important?

Currently it looks like deleted cache leads to error. The only way to safely delete them is to set max-cache-age and delete cache older than that, or rerun pipeline with cache overwrite. This setting is not very flexible, because it is global.

Goal: What should the final outcome look like, ideally?

If Flyte interprets non-exist cache object as cache miss and rerun the tasks, I can use AWS lifecycle policies which can be more flexible (e.g. delete large cache faster).

Describe alternatives you've considered

More tools around cache management and flexible expiration could be added to Flyte. It was raised in the past but it hasn't been added, and it likely requires much bigger change.

#3305
https://discuss.flyte.org/t/12905332/caching-hey-all-i-rewrote-one-of-my-company-s-critical-workf

Propose: Link/Inline OR Additional context

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[welcome]

Thank you for opening your first issue here! 🛠

[pvditt]

@eapolinario we have this #4655 that would add an endpoint to delete cache entries

[amitani]

The PR is more general and covers my use case. Thank you for pointing it out, and I'm closing this.