RdsExportTaskExistenceSensor Run indeterminate if snapshot export failed #46555
Description
Apache Airflow Provider(s)
amazon
Versions of Apache Airflow Providers
apache-airflow-providers-amazon==8.10.0
Apache Airflow version
2.7.3
Operating System
Composer 2.8.5
Deployment
Google Cloud Composer
Deployment details
No response
What happened
A few weeks ago, we encountered a problem while creating an RDS snapshot. Something changed in the validation of the snapshot configuration inputs, and it now checks the s3_prefix name.
How We Discovered It ?
We have several DAGs that create RDS database snapshots and export them to S3. On one fateful day (2025-01-24), we ran into a problem because all these DAGs, which run after hours, were stuck in the RdsExportTaskExistenceSensor (with no timeout defined). Upon checking the AWS console, we saw that the exports had failed. We were using s3_prefix="snapshots/{{ts_nodash}}/" with a trailing slash, and this caused the failure, as shown in the screenshot.
After updating s3_prefix removing the ended backslash, it work as expected.
- s3_prefix="snapshots/{{ts_nodash}}/",
+ s3_prefix="snapshots/{{ts_nodash}}",What you think should happen instead
RdsExportTaskExistenceSensor should raise and exception if an error statuses happens.
How to reproduce
Our DAG uses the following logic
RdsCreateDbSnapshotOperator >> RdsSnapshotExistenceSensor >> RdsStartExportTaskOperator >> RdsExportTaskCompleteSensor >> .....
if s3_prefix in RdsStartExportTaskOperator has trailing backslash it will faill and RdsExportTaskCompleteSensor will not complete and Run indeterminate if timeout is not defined.
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct