mp_start_method was removed via 11acefa462b6

[mattp-]

Apache Airflow version

main (development)

If "Other Airflow 2 version" selected, which one?

No response

What happened?

looks like 11acefa deleted code that provided functionality for mp_start_method, among other things.

What you think should happen instead?

either remove mp_start_method from config, or re-add implementation.

How to reproduce

n/a

Operating System

all

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[eladkal]

cc @kaxil something we overlooked in the cleanup?

[kaxil]

We swapped Multiprocessing with Task SDK machinery in #44972

11acefa was just a cleanup.

[mattp-]

@kaxil sure, but mp_start_method no longer exists outside of the config. if someone wants to use a different multiprocessing method they no longer can / need to hack something. I discovered this looking at #54510 , where I realized another annoyance, uvicorn also sets spawn in a way that will make it a default if it isn't already set, so there will be more unexpected behavior there if someone is expecting forking.

[kaxil]

Yup, teething issues with Airflow 2 to 3 migration. PRs welcomed

[potiuk]

@kaxil sure, but mp_start_method no longer exists outside of the config. if someone wants to use a different multiprocessing method they no longer can / need to hack something. I discovered this looking at #54510 , where I realized another annoyance, uvicorn also sets spawn in a way that will make it a default if it isn't already set, so there will be more unexpected behavior there if someone is expecting forking.

Yes. Annoyances when we removed things that were never intended to be used are expected. Following what @kaxil wrote - if you feel that your case is also bothering for others and that it is worth turning that into a featire - feel free to a) discuss in devlist, and b) - ideally at the same time - make a PR to implement it. Otherwise - bite the bullet and fight with the annoyance - possibly it;s only you.

I comvert it into a discussion, but if anyone wants to contribute some feature that is generic and useful enough - maybe resurrecting it from the past - opening PR is really best place to start.