Fix setproctitle usage

[jason810496]

related: #52860

Why

When using setproctitle we should skip MacOS ( darwin ) as special case, see benoitc/gunicorn#3021 for more detail.
When refactoring in #52860, I found there are duplicate code to handle setproctitle for MacOS and some usage of setproctitle don't handle the MacOS problem, which might lead to error.

What

• Add a new setproctitle utility in the airflow.utils module to centralize handling of the macOS error, allowing us to avoid the issue from the beginning.
• Add corresponding tests for it

[pierrejeambrun]

Might need a second pair of eyes, but LGTM

[kaxil]

Yeah, like @amoghrajesh mentioned that this is duplicated purposely to not have dependency for it between Core, SDK & Providers

[jason810496]

So I need to wait until the PR for code sharing of "core", "task-sdk" and "provider" described in Dev Mail: Code sharing between Airflow Core and Task SDK - how do we achieve it then keep on this right ?

[kaxil]

So I need to wait until the PR for code sharing of "core", "task-sdk" and "provider" described in Dev Mail: Code sharing between Airflow Core and Task SDK - how do we achieve it then keep on this right ?

The code is too small IMO -- duplication is not always bad :) We can leave this as-is and close this PR

[jason810496]

The code is too small IMO -- duplication is not always bad :) We can leave this as-is and close this PR

I see! I’ll remove this small new module as I was previously a bit stuck trying to avoid duplication.
However, instead of closing this PR, I’ll update it to fix some remaining setproctitle usages that aren’t properly handled on macOS (e.g. in the Celery executor, edge worker, etc.).

[pierrejeambrun]

Just a suggestion, but maybe I missed something.

edit: Just saw other comments, code duplication it is. We can still leave that at the top of each file to avoid duplication within the same file. (which shouldn't bring any cross component dependencies)

[pierrejeambrun]

Looks good. I see the PR is on draft is there anything left you wanted to implement there

[jason810496]

Looks good. I see the PR is on draft is there anything left you wanted to implement there

No, thanks for the reminder, I just want to make sure the CI is green before marking the PR as ready to review.

[jason810496]

Hi @kaxil, this PR still needs your approval to resolve the requested changes. Thanks!

[jason810496]

It seems that this PR still needs @kaxil's approval before it can be merged.
(Due to GitHub restrictions, if any reviewer has requested changes, the PR requires that specific person's approval to unblock and proceed with the merge.)

Stale

[ashb]

Other committers can dismiss reviews. I've just done that, and let me give it a re-review.

stale

[pierrejeambrun]

That looks good to me. I'll let @ashb give the final word on this as requested

[potiuk]

That calls a little for a "shared" distribution to include that code rather than copy it over. But it'a little to small to do it now. But maybe eventually when we expand on our "shared" projects concept we could have a separate "executors base" code an we split out executors to separately installable distributions, we could make the code implemented once. Maybe worth leaving TODO for that ?

Just a suggestion - not a blocker at all.

[ashb]

@potiuk I see it differently, and I'd say this is one of the cases where not using a shared dist and duplicating the code is better for readability -- this sort of case is exactly what I had in mind when I wrote this https://github.com/apache/airflow/tree/main/shared#be-thoughtful-about-what-you-add-under-here

Be Thoughtful about what you add under here

Not every piece of code used in two distributions should be automatically placed in one of the shared libraries, and sometimes "just duplicate it" is the right approach to take. For example, if it's just a 5 or 10 line function and it's used in two places, it might be easier for future developers to understand if the function is in two places.