Nuking SequentialExecutor from the in-tree executors #48528
Conversation
amoghrajesh
requested review from
XD-DENG,
ashb,
gopidesupavan,
hussein-awala,
jedcunningham,
o-nikolas,
pierrejeambrun and
potiuk
as code owners
boring-cyborg
bot
added
area:dev-tools
area:Executors-core
|
Guess this is what's left as of now: |
|
I know the failures here, but unfortunately wont have access to computer till tomorrow evening. I will fix those then.. But its ready for review id say 😀 |
|
Besides the leftovers if you search for "SequentialExecutor" (and also there are some hits when you search for "Sequential Executor" with space!) and some CI fails I think this is looking pretty good. If you lag access to a PC, let me know if I can help. Would be cool to remove before RC0. |
|
Was #44243 resolved? |
I have access now, thanks @jscheffl. Just looking at the CI and missing occurences now. |
Looks like #44243 (comment) is a solution that works. Although i'd want someone to test it again if possible, I know its important but we've had more successes than not in that migration. LocalExec has been tested plenty :) |
@jscheffl it has been handled now 🗡️ |
airflow-core/docs/administration-and-deployment/production-deployment.rst
Show resolved
Hide resolved
jscheffl
left a comment
jscheffl
left a comment
There was a problem hiding this comment.
Tested locally with a couple of DAGs and "small load test" (wow, was able to saturate 80% of my 20 cores) and looks good on my view. Would be cautious making this for a production system but much much better than SequentialExecutor. Cool out-of-the-box experience possible with this!
#protm
airflow-core/docs/administration-and-deployment/logging-monitoring/logging-tasks.rst
Show resolved
Hide resolved
|
Thanks for the reviews folks, merging this one. |
* Nuking SequentialExecutor * some occurences * fixing ci failures -- docs, mypy and tests * comments from jens * fixing tests * fixing occurences - both SequentialExecutor and Sequential Executor * last occurence * comments from niko
* Nuking SequentialExecutor * some occurences * fixing ci failures -- docs, mypy and tests * comments from jens * fixing tests * fixing occurences - both SequentialExecutor and Sequential Executor * last occurence * comments from niko
* Nuking SequentialExecutor * some occurences * fixing ci failures -- docs, mypy and tests * comments from jens * fixing tests * fixing occurences - both SequentialExecutor and Sequential Executor * last occurence * comments from niko
…nificant.rst reading apache#48528, it suggest users to use dag.test() instead of replacing the value as dag.test()
…nificant.rst reading apache#48528, it suggest users to use dag.test() instead of replacing the value as dag.test()
…49048) * docs(newsfragments): check action items that has been done * feat(config_command): add "core.dag_default_view" and "core.dag_orientation" * docs(newsfragments): fix wrong description in newsfragments/48528.significant.rst reading #48528, it suggest users to use dag.test() instead of replacing the value as dag.test() * feat(config_command): add scheduler.create_cron_data_intervals and scheduler.create_delta_data_intervals to config lint * Revert "feat(config_command): add scheduler.create_cron_data_intervals and scheduler.create_delta_data_intervals to config lint" This reverts commit e60bfe3.
5 participants
closes: #48522
Let me do the honors!
Tried generating the image using AI, the spellings is what is hallucinates with! Anyway, worthwhile to add i'd say!
The alternative here is to go and use LocalExecutor -- which is a similar one and it works with sqlite. This was a PR that enabled it #44839
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.