Implement per-provider tests with lowest-direct dependency resolution

[potiuk]

With this change we are running tests of downgrading Airflow
dependencies to "lowest-direct" ones - separately for "core" tests
and for each provider (and run corresponding tests with it). This
should allows us to determine what are the lowest bounds for the
dependencies - for Airflow and for individual providers and continue
doing it while Airflow evolves in the future.

Fixes: #35549

Related: #39100

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[JDarDagran]

Does this PR assume all cross-provider dependencies should be expressed as extras (in terms of python packaging, added in additional-extras for each provider)?
Not saying this is a bad approach, just making sure as e.g. OpenLineage provider exists as extra dep only for DBT provider and probably needs manual work to add it as well (or at least have a check in pre-commit if such extra exists, there's already check for affected providers).
lowest-direct resolution is a great approach though 👍

Edit: While the initial checks cover transitive dependencies, they don't account for potential issues that may arise when using different providers with different versions of OpenLineage. For instance, consider a scenario where:

• In release A.X -> A.Y, a breaking change is introduced in the OpenLineage provider, which also affects the version of openlineage-python support.
• In release B.X -> B.Y, provider B addresses this change by bumping the lowest version in the [openlineage] extra.
• However, the proposed tests only verify that the lowest OpenLineage provider and highest openlineage-python version still work together with provider B.

This leaves a gap in testing, as it doesn't ensure that provider B version B.Y can seamlessly integrate with the older OpenLineage provider A.X, unless we assume that users would always install provider B with the [openlineage] extra.

[potiuk]

Does this PR assume all cross-provider dependencies should be expressed as extras (in terms of python packaging, added in additional-extras for each provider)?

Unfortunately the way it works in Python, those "extra" limits are not enforceable, and It makes little sense to add them (and verify). There is really no good way to speciffy "if you want to install this optional dependency, it has to follow those limits" - there are multiple scenarios where even with those limits set on extra, user will be able to perform a sequence of upgrades to break it, and at any point in their process even pip check will show that everything is fine. So I am not sure we even should attempt to do such a check.

[JDarDagran]

Does this PR assume all cross-provider dependencies should be expressed as extras (in terms of python packaging, added in additional-extras for each provider)?

Unfortunately the way it works in Python, those "extra" limits are not enforceable, and It makes little sense to add them (and verify). There is really no good way to speciffy "if you want to install this optional dependency, it has to follow those limits" - there are multiple scenarios where even with those limits set on extra, user will be able to perform a sequence of upgrades to break it, and at any point in their process even pip check will show that everything is fine. So I am not sure we even should attempt to do such a check.

Fully agreed. Do you still think that would be worth adding different kind of tests too?
I'll stick to OpenLineage as there have been cases for not keeping backwards-compatibility we did not detect. #39530 also brings such and would love to have something that runs checks against this. My initial idea is to maintain a static list of providers that can be downgraded to the latest released version, similar to how Pydantic and SQLAlchemy have special tests for downgrading. This way, if changes are made to a provider that affect compatibility with OpenLineage, tests will fail if the changes are not backwards-compatible.

[potiuk]

Sure - we can think of next steps and once we get this one merged, you could add a similar follow-up PR with those checks :)

[potiuk]

cc: @charliermarsh -> thanks to uv's lowest-direct we finally could automate bumping the lower bounding of direct dependencies for Airflow and it's ~90 providers

Also @notatallshaw -> this is probably as close as it can get to the #39100 you opened and likely will make your apache-airflow[all] installations quite a bit faster. UPDATE: Actually you wanted to put limits for airflow -> provider dependencies, so it's a bit differen (those are limits on airlfow and provider's own dependencies) - but let's see where it can get us.

[potiuk]

And merged :)

[potiuk]

BTW. When it comes to --lowest strategy, we might want to wait to try it when astral-sh/uv#4006 gets merged - then we will see clearer which packages might cause problems there - but I am afraid it will get things "too much" to the floor, and we will be fixing someone else's problems this way.

[simonprydden]

But on a serious note - I really want to hear from Amazon team, because I think the watchtower change might be problematic (can't remember the root cause - so likely we want to fix it before we merge the change).

@potiuk did you hear back from the MWAA team?

The issue is that MWAA pins watchtower==2.0.1 , as per docs

Note
For Apache Airflow v2 and above, Amazon MWAA installs Watchtower version 2.0.1 after performing pip3 install -r requirements.txt, to ensure compatibility with CloudWatch logging is not overridden by other Python library installations.

Trying to find this PR in the amazon providers changelog, to double check which is the max version of the amazon providers that will work with MWAA. I think it's 8.24.0

[potiuk]

@potiuk did you hear back from the MWAA team?

Yes. See #39946 (comment) and maybe @vincbeck or @ferruzzi or @o-nikolas can comment on that :)

[o-nikolas]

@potiuk did you hear back from the MWAA team?

Yes. See #39946 (comment) and maybe @vincbeck or @ferruzzi or @o-nikolas can comment on that :)

I've asked around on the MWAA team and I'll report what I hear back!