Skip to content

Conversation

@potiuk
Copy link
Member

@potiuk potiuk commented Apr 6, 2025

With the recent uv move, we can move some of the optional dependencies to providers - i.e. rather than refer to specific dependenciesm we can refer to the optional dependencies of the providers. This was not easy before - because with pip for development dependencies it would have install the provider itself instead of the dependencies, but with uv workspaces this is perfectly fine as uv will find the provider in the workspace and will install the provider sources and the optional dependencies when given extra is used.

This avoids duplication in the core's dependencies and allows to depend airflow core noot on specific dependencies but on the optional dependencies of provider installed, which is as it should be.

This change also bumps the dependencies of amazon provider that are affected by this change in an attempt to make "resolution too deep" problems.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@potiuk potiuk added the canary When set on PR running from apache repo - behave as canary run label Apr 6, 2025
@potiuk potiuk requested review from eladkal and o-nikolas as code owners April 6, 2025 11:40
@boring-cyborg boring-cyborg bot added area:providers provider:amazon AWS/Amazon - related issues labels Apr 6, 2025
With the recent `uv` move, we can move some of the optional dependencies
to providers - i.e. rather than refer to specific dependenciesm we can
refer to the optional dependencies of the providers. This was not easy
before - because with `pip` for development dependencies it would have
install the provider itself instead of the dependencies, but with uv
workspaces this is perfectly fine as uv will find the provider in
the workspace and will install the provider sources and the optional
dependencies when given extra is used.

This avoids duplication in the core's dependencies and allows to
depend airflow core noot on specific dependencies but on the
optional dependencies of provider installed, which is as it should be.

This change also bumps the dependencies of amazon provider that
are affected by this change in an attempt to make "resolution too
deep" problems.
@potiuk potiuk force-pushed the optional-extras-moved-to-providers branch from ce1b53b to 9d00ad6 Compare April 6, 2025 11:46
@potiuk potiuk merged commit 3b48df7 into main Apr 6, 2025
77 checks passed
Copy link
Contributor

@jscheffl jscheffl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool! Crossing fingers that the resolution too Deep will go away by this!

@potiuk
Copy link
Member Author

potiuk commented Apr 6, 2025

Let's see if it will help for the "resolution too deep". I am in a plain above the Atlantic now ;) so with low network speed no chance to test it locally for image building, but 1) far less number of combos for Amazon, 2) it's likely that having the dependencies on both - extra and different versions of providers made it difficult for pip to figure out the right set of deps.

potiuk added a commit to potiuk/airflow that referenced this pull request Apr 6, 2025
…8843)

With the recent `uv` move, we can move some of the optional dependencies
to providers - i.e. rather than refer to specific dependenciesm we can
refer to the optional dependencies of the providers. This was not easy
before - because with `pip` for development dependencies it would have
install the provider itself instead of the dependencies, but with uv
workspaces this is perfectly fine as uv will find the provider in
the workspace and will install the provider sources and the optional
dependencies when given extra is used.

This avoids duplication in the core's dependencies and allows to
depend airflow core noot on specific dependencies but on the
optional dependencies of provider installed, which is as it should be.

This change also bumps the dependencies of amazon provider that
are affected by this change in an attempt to make "resolution too
deep" problems.
diogotrodrigues pushed a commit to diogotrodrigues/airflow that referenced this pull request Apr 6, 2025
…8843)

With the recent `uv` move, we can move some of the optional dependencies
to providers - i.e. rather than refer to specific dependenciesm we can
refer to the optional dependencies of the providers. This was not easy
before - because with `pip` for development dependencies it would have
install the provider itself instead of the dependencies, but with uv
workspaces this is perfectly fine as uv will find the provider in
the workspace and will install the provider sources and the optional
dependencies when given extra is used.

This avoids duplication in the core's dependencies and allows to
depend airflow core noot on specific dependencies but on the
optional dependencies of provider installed, which is as it should be.

This change also bumps the dependencies of amazon provider that
are affected by this change in an attempt to make "resolution too
deep" problems.
@ashb ashb deleted the optional-extras-moved-to-providers branch April 6, 2025 15:45
simonprydden pushed a commit to simonprydden/airflow that referenced this pull request Apr 8, 2025
…8843)

With the recent `uv` move, we can move some of the optional dependencies
to providers - i.e. rather than refer to specific dependenciesm we can
refer to the optional dependencies of the providers. This was not easy
before - because with `pip` for development dependencies it would have
install the provider itself instead of the dependencies, but with uv
workspaces this is perfectly fine as uv will find the provider in
the workspace and will install the provider sources and the optional
dependencies when given extra is used.

This avoids duplication in the core's dependencies and allows to
depend airflow core noot on specific dependencies but on the
optional dependencies of provider installed, which is as it should be.

This change also bumps the dependencies of amazon provider that
are affected by this change in an attempt to make "resolution too
deep" problems.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers canary When set on PR running from apache repo - behave as canary run provider:amazon AWS/Amazon - related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants