-
Notifications
You must be signed in to change notification settings - Fork 14.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SFTPOperator throws object of type 'PlainXComArg' has no len() when using with Taskflow API #27328
Comments
Thanks for the bug report @jtommi! I see that you've checked that you're willing to submit a PR, so I have assigned the task to you 😃 |
Value checks in operators should be done in |
@uranusjr Thanks for the initial direction to fixing this. I drafted #27346.
The reason is
The thing is that I don't understand why the code worked with version 2 of the provider package (I actually didn't check with version 4, but I didn't get import errors). |
All the checks should be moved to |
The thing is: either I broke (or didn't find) the code that made Taskflow work in version 4.0 of this package, or my test is incorrect. |
This should also be fixed regardless of the SFTP provider version by #27251 |
Yeah. It's been released in 2.5.0 so just upgrading to 2.5.0 should be enough to test it. |
@potiuk I finally upgraded to 2.5.0 (-python3.10) and the issue persists. Pinning apache-airflow-providers-sftp to 4.0.0 still fixes the issue |
Yep. It looks like this change #26666 broke it (the number indicates it's a bit hell-ish change). the checks are in constructor but should be in the execute() method. |
Woul you like to continue fixing it @jtommi ? And @uranusjr - we both approved the change. I am starting to think that we should attempt to do do some automated prevention of similar issues - this is all too easy (and seemingly obvious) to perform such valiations and conversions in the constructor rather than in execute methods. And apparently it is easy to get past the aproval of both of us, so possibly that's a sign we should summon our CI to prevent such things. Though I am not sure yet how to do it. |
Personally I think the eventual goal would be to ban any logic in |
I like this. Though it will take quite some tmie I am afraid - unless we can do most of it semi-automatically (which I think might be quite possible). |
The SFTP operator had logic in `__init__` that was running checks on passed arguments, but that means that the operator woudl fail when using TaskFlow. Fixes: apache#27328
Created #29069 |
The SFTP operator had logic in `__init__` that was running checks on passed arguments, but that means that the operator woudl fail when using TaskFlow. Fixes: #27328
The SFTP operator had logic in `__init__` that was running checks on passed arguments, but that means that the operator woudl fail when using TaskFlow. Fixes: apache#27328
Apache Airflow Provider(s)
sftp
Versions of Apache Airflow Providers
apache-airflow-providers-sftp==4.1.0
Apache Airflow version
2.4.2 Python 3.10
Operating System
Debian 11 (Official docker image)
Deployment
Docker-Compose
Deployment details
Base image is apache/airflow:2.4.2-python3.10
What happened
When combining Taskflow API and SFTPOperator, it throws an exception that didn't happen with apache-airflow-providers-sftp 4.0.0
What you think should happen instead
The DAG should work as expected
How to reproduce
Anything else
It looks like the offending code was introduced in commit 5f073e3
Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: