Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Airflow 2.x operators (runtime specific components) #3263

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

lresende
Copy link
Member

@lresende lresende commented Dec 29, 2024

What changes were proposed in this pull request?

Support Airflow 2.x operators (runtime specific components)

Fixes #2124

After updating the elyra examples to have Airflow 2.x components, and register that in Elyra, I can see the airflow components and have the properties show up ok on the Pipeline Editor

image

Developer's Certificate of Origin 1.1

   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.

Signed-off-by: Luciano Resende <lresende@apple.com>
@lresende lresende added this to the 4.0.0 milestone Dec 30, 2024
@shalberd
Copy link
Contributor

shalberd commented Dec 30, 2024

Yeah, there are still some changes necessary at airflow_component_parser, though, regarding the properties and values of each airflow operator.
Even though the GUI presence of the operators is ok, when assembling the DAG code from the values and defaults entered, e.g. from Bash operator, the assembled DAG is not correct.

#2124 (comment)

#3208 (comment)

Thank you though for making this starter-PR more complete in the other aspects here. I will see what I can find out about the inner workings of component_parser_airflow in conjunction with the new airflow operator classes, like None vs. Empty String for Bash operator cwd property (empty string is wrong ...) ... I just never had time yet to fully debug.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants