Skip to content

Conversation

@o-nikolas
Copy link
Contributor

Multiple Executor Configuration (aka hybrid executors, AIP-61) has been released for over half a year and it will be almost a year when Airflow 3.0 is released. I think with the release of Airflow 3 we can call it stable, but keep in as experimental in 2.10.X and 2.11.X in case any issues are discovered during those releases.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

Multiple Executor Configuration (aka hybrid executors) has been released
for over half a year and will be almost a year when Airflow 3.0 is
released. I think with the release of Airflow 3 we can call it stable,
but keep in as experimental in 2.10.X and 2.11.X in case any issues are
discovered during those releases.
@o-nikolas o-nikolas requested a review from potiuk as a code owner February 14, 2025 00:41
@boring-cyborg boring-cyborg bot added area:Executors-core LocalExecutor & SequentialExecutor kind:documentation labels Feb 14, 2025
@o-nikolas
Copy link
Contributor Author

Optionally we could mark it as stable in 2.11 if we wanted to give people more time to migrate. Thoughts?

@eladkal
Copy link
Contributor

eladkal commented Feb 14, 2025

I think we better have lazy consensus vote on this one.
Also we need to decide what to do with CeleryKubernetesExecutor and the others od that kind. They qere not deprecated because Hybrid was marked experimintal.

Personally I would prefer to wait with marking it stable. There were reports on this feature not working with helm chart and so far the fix was not released (waiting for chart 1.16)

@o-nikolas
Copy link
Contributor Author

I think we better have lazy consensus vote on this one. Also we need to decide what to do with CeleryKubernetesExecutor and the others od that kind. They qere not deprecated because Hybrid was marked experimintal.

Personally I would prefer to wait with marking it stable. There were reports on this feature not working with helm chart and so far the fix was not released (waiting for chart 1.16)

I agree, I was just typing up a discussion thread for this actually! I will send that out.

I think a course of action that's in the middle is mark this as stable in 2.11 and very publicly mark the old hard coded executors as deprecated in 2.11 and then. Remove them in 3.0

I don't personally think issues with the helm chart stop multi exec config from being stable. That is not related to bugs in the fundamental interface of that feature.

@potiuk
Copy link
Member

potiuk commented Feb 15, 2025

I don't personally think issues with the helm chart stop multi exec config from being stable. That is not related to bugs in the fundamental interface of that feature.

As explained in the devlist message. While it's not a blocker, a call to make it works should be included in a call to make the feature stable.

@o-nikolas
Copy link
Contributor Author

Now that #46944 is merged, and Jed is planning to release a new Helm Chart soon, we should be able to merge this one as well.

This small PR simply maintains consistency with the above mentioned change in 2.10.X

CC: @jedcunningham @jscheffl @amoghrajesh

@o-nikolas o-nikolas merged commit 9be08cf into apache:main Mar 3, 2025
44 checks passed
@o-nikolas o-nikolas deleted the onikolas/mark_multi_exec_config_stable branch March 3, 2025 18:29
Copy link
Contributor

@amoghrajesh amoghrajesh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Late approval but looks good!

shahar1 pushed a commit to shahar1/airflow that referenced this pull request Mar 5, 2025
Multiple Executor Configuration (aka hybrid executors) has been released
for over half a year and will be almost a year when Airflow 3.0 is
released. With the release of Airflow 3 we can call it stable.
nailo2c pushed a commit to nailo2c/airflow that referenced this pull request Apr 4, 2025
Multiple Executor Configuration (aka hybrid executors) has been released
for over half a year and will be almost a year when Airflow 3.0 is
released. With the release of Airflow 3 we can call it stable.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Executors-core LocalExecutor & SequentialExecutor kind:documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants