SFTPSensor succeeds even if file is absent #51151
Description
Apache Airflow Provider(s)
sftp
Versions of Apache Airflow Providers
$ pip list | grep sftp
apache-airflow-providers-sftp 5.3.0
Apache Airflow version
Version: v2.11.0 Git Version: .release:d9ed7b94d033dc61ee1c05aafb9f1c1f7b82cfb2
Operating System
Debian GNU/Linux 12 (bookworm)
Deployment
Virtualenv installation
Deployment details
$ python3 --version
Python 3.11.2
What happened
We use a SFTPSensor to check if a log file exists on the remote and is therefore ready for download. We switched to Airflow v2.11.0 this week to prepare for the v3 migration. Our workflow, which uses the SFTPSensor, sometimes was behaving abnormal since the update to v2.11.0. We investigated some time in troubleshooting and found that the SFTPSensor only pokes one time and then exits with success whether the file is there or not.
What you think should happen instead
The SFTPSensor should keep poking until the expected file is available.
How to reproduce
Create an SFTPSensor with these parameters and let it listen for a remote file.
task_id='watch_for_log',
sftp_conn_id='my_conn_id',
path='/my_log_file.log',
poke_interval=3
Anything else
We came up with a fix for this, that works in our environment.
In the file providers/sftp/src/airflow/providers/sftp/sensors/sftp.py in line 98 to 104 replace the try-block with this code. The bug here is, that the OSError is already excepted in self.hook.isfile and therefore never excepted in the poke funtion. The self.hook.isfile returns False in case of an error. With this we just have to check if the file exists before passing the self.path to actual_files_to_check.
try:
actual_files_to_check = [self.path] if self.hook.isfile(self.path) else []
except OSError as e:
if e.errno != SFTP_NO_SUCH_FILE:
raise AirflowException from e
actual_files_to_check = []Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct