Set up process for sharing code between different components.

[ashb]

And move airflow.utils.timezone into a shared library as the first example of it working.

In this change we have now setled on an approach using symlinks, but we did
explore other options (see the GH PR for discussion and previous versions,
notably one built upon the vendoring tool)

A lot of the reasoning and mode of operation of this is detailed in shared/README.md in this PR, hence why this description is so short.

Currently various places in TaskSDK and Airflow Core both use these utility functions, and while in this specific case they are small enough that they could just be copied and the duplication wouldn't hurt us long term, this changes shows a way in which we can have a single source of truth, but have it included automatically in built dists.

(For posterity, please see b6ae6a9 for the previous vendoring-based approach)

And thanks to Jarek for the help in making the symlink version work, it's much simpler!

[potiuk]

Plus do we need a pre-commit check to ensure the symlink and projet setup is done correctly?

Yes. We agreed I will write it as follow-up @uranusjr

[potiuk]

A general miss imo. For public docs etc we should use the public API path: airflow.sdk and internally use the canonical imports: from airflow.sdk._shared.timezones import timezone

I think that should be ruff rule. Everything "in" task-sdk should use "_shared" or ("relative" if inside _shared already). For evryhing "using" "task.sdk" it should use the facade. The question is can we somehow forbid (or discourage) importing from `_shared" from outside ?

[potiuk]

Have we discussed backward compatibility (in general)? Like should airflow.utils.timezone should still be accessible by users.

+1 -> See the other comment from @amoghrajesh , but I think task-sdk should have airflow.utils with deprecation redirections to it's _shared code - this is the only distribution that should be accessed by user code I think - i.e. where the deprecations might be needed.

[potiuk]

Looks fantastic! It's great we could combine all the ideas and get something really nice and developer-friendly :) . There are few comments - most of them require more a of discussion, agreement and follow-up than changing the PR.

[ashb]

Cases like this in the providers:

from airflow.utils.timezone import convert_to_utc

I think I will leave as is for now, in this PR and do a single provider-only follow up PR that does the right thing (try sdk.timezone, except import utils.timezone)

[ashb]

+1 -> See the other comment from @amoghrajesh , but I think task-sdk should have airflow.utils with deprecation redirections to it's _shared code - this is the only distribution that should be accessed by user code I think - i.e. where the deprecations might be needed.

Yeah longer term (as in, not in this PR) I agree that is the right place, but we can't do that until we remove everything else form airflow.utils out of the airflow-core dist I don't think?

[potiuk]

Yeah longer term (as in, not in this PR) I agree that is the right place, but we can't do that until we remove everything else form airflow.utils out of the airflow-core dist I don't think?

We could potentially do the legacy namespace pathlib trick in both utils packages - that will nicely work and allow to move stuff between the two over time.

[potiuk]

Yeah longer term (as in, not in this PR) I agree that is the right place, but we can't do that until we remove everything else form airflow.utils out of the airflow-core dist I don't think?

We could potentially do the legacy namespace pathlib trick in both utils packages - that will nicely work and allow to move stuff between the two over time.

Even if temporarily - that would likely work nicely - we could remove it before release.

[ashb]

(╯°□°)╯︵ ┻━┻

[amoghrajesh]

#protm

Lot of good collaboration between various Airflow wizards and impact wise this one unlocks ability for client server separation.

[potiuk]

Let me get the pre-commit now :)