Skip to content

Conversation

@potiuk
Copy link
Member

@potiuk potiuk commented Nov 18, 2025

We have inline project metadata in the prek hooks, and that means that each of the scripts will use their own venv to roon. However Airflow code and libraries does not support Python 3.14 yet (and will not for quite a while - because our dependencies will need time to catch-up. In the meantime we have to limit prek hooks to not install Python 3.14.

Actually in order to have good stability, we are fine to limiting
EVERYONE to use Python 3.10 for prek. There are no real benefits from
using a different Python version (except maybe a little speed but
prek-hooks are generally rather fast anyway) and we can definitely
benefit from better stability and having exactly the same output
for prek hooks by everyone. Uv will handle installation of the
right python version automatically.

Fixes: #56048


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@boring-cyborg boring-cyborg bot added area:dev-tools backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch labels Nov 18, 2025
@potiuk potiuk force-pushed the exclude-python-3-14-for-prek-hooks branch from 9db8f6e to 63c9887 Compare November 18, 2025 16:57
@potiuk potiuk force-pushed the exclude-python-3-14-for-prek-hooks branch from 63c9887 to 2f82106 Compare November 18, 2025 16:58
@potiuk potiuk changed the title Until Airflow supports Python 3.14, all prek hooks should not use it Limit Python prek hooks to use Python 3.10 Nov 18, 2025
@potiuk potiuk force-pushed the exclude-python-3-14-for-prek-hooks branch from 2f82106 to 1deb31b Compare November 18, 2025 16:59
@ashb
Copy link
Member

ashb commented Nov 18, 2025

Can we go for py 3.13 instead for all these?

@potiuk
Copy link
Member Author

potiuk commented Nov 18, 2025

Can we go for py 3.13 instead for all these?

No. Because our Python code base supports 3.10 so we have to go for lowest common denominator.

We have inline project metadata in the prek hooks, and that means
that each of the scripts will use their own venv to roon. However
Airflow code and libraries does not support Python 3.14 yet (and will
not for quite a while - because our dependencies will need time to
catch-up. In the meantime we have to limit prek hooks to not install
Python 3.14.

Actually in order to have good stability, we are fine to limiting
EVERYONE to use Python 3.10 for prek. There are no real benefits from
using a different Python version (except maybe a little speed but
prek-hooks are generally rather fast anyway) and we can definitely
benefit from better stability and having exactly the same output
for prek hooks by everyone. Uv will handle installation of the
right python version automatically.

Fixes: apache#56048
@potiuk potiuk force-pushed the exclude-python-3-14-for-prek-hooks branch from 1deb31b to 2429ad9 Compare November 18, 2025 17:41
@potiuk
Copy link
Member Author

potiuk commented Nov 18, 2025

Can we go for py 3.13 instead for all these?

No. Because our Python code base supports 3.10 so we have to go for lowest common denominator.

For example the whole reason why things started to fail for the user in #56048 (look closely) - is that the code we were using in regular Airflow was using some things that were removed in 3.14.

Now - if we allow people to run their check on 3.13 - and we have some code that does not work on 3.10 - it will work for them but it will not be python 3.10 compliant.

So absolutely safest choice is to have everyone use the same as the lowest denominator for Airflow code - Python 3.10.

Even if Python 3.13 works a bit faster and has better UX for developers (it does) - the "works for me" problem is far bigger than any of the gains.

In this case - optimisation goal is not for this to be "better" - but to be "the same for everyone" (and compatible with the code we write for Airflow).

@potiuk
Copy link
Member Author

potiuk commented Nov 18, 2025

And it has almost (besides a little speed) almost 0 impact for anyone to be perfectly honest, so I am not sure what we would gain from 3.13 support here:

  • those scripts are mostly throwaway and nowadays mostly AI generated - so I don't care about using too many new language features
  • you will not notice it when you run it - Python 3.10 will be automatically downloaded and installed by uv under the hood
  • you can still - if you want - run those scripts manually using higher version of Python if you care for better debugging/UX experieonce - nothing (including IDE support) prevents you from doing so - inline script metadata is largely optional, it will still work fine if you have Python 3.13 venv and run the script there.

So frankly, I do not see practically any benefit of having 3.13 there.. But maybe I missed something?

@ashb
Copy link
Member

ashb commented Nov 18, 2025

Nope, nothing more than "py 3.13 runs 5-50% faster" but I agree with the reasons you listed, it's not worth it. (I was just thinking "most of the scripts are not really tied to what version of python we support, they just do things like read files or check AST etc" but most != all.

Copy link
Contributor

@jscheffl jscheffl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the constructive and not heated discussion. I fully agree to arguments and outcome! LGTM.

@potiuk potiuk merged commit d31e578 into apache:main Nov 18, 2025
53 checks passed
@potiuk potiuk deleted the exclude-python-3-14-for-prek-hooks branch November 18, 2025 22:08
github-actions bot pushed a commit that referenced this pull request Nov 18, 2025
We have inline project metadata in the prek hooks, and that means
that each of the scripts will use their own venv to roon. However
Airflow code and libraries does not support Python 3.14 yet (and will
not for quite a while - because our dependencies will need time to
catch-up. In the meantime we have to limit prek hooks to not install
Python 3.14.

Actually in order to have good stability, we are fine to limiting
EVERYONE to use Python 3.10 for prek. There are no real benefits from
using a different Python version (except maybe a little speed but
prek-hooks are generally rather fast anyway) and we can definitely
benefit from better stability and having exactly the same output
for prek hooks by everyone. Uv will handle installation of the
right python version automatically.
(cherry picked from commit d31e578)

Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Fixes: #56048
@github-actions
Copy link

Backport successfully created: v3-1-test

Status Branch Result
v3-1-test PR Link

potiuk added a commit that referenced this pull request Nov 18, 2025
We have inline project metadata in the prek hooks, and that means
that each of the scripts will use their own venv to roon. However
Airflow code and libraries does not support Python 3.14 yet (and will
not for quite a while - because our dependencies will need time to
catch-up. In the meantime we have to limit prek hooks to not install
Python 3.14.

Actually in order to have good stability, we are fine to limiting
EVERYONE to use Python 3.10 for prek. There are no real benefits from
using a different Python version (except maybe a little speed but
prek-hooks are generally rather fast anyway) and we can definitely
benefit from better stability and having exactly the same output
for prek hooks by everyone. Uv will handle installation of the
right python version automatically.
(cherry picked from commit d31e578)


Fixes: #56048

Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
@potiuk
Copy link
Member Author

potiuk commented Nov 18, 2025

Thanks for the constructive and not heated discussion. I fully agree to arguments and outcome! LGTM.

Heated ? Have you seen heated discussion between us 😄 ?

ephraimbuddy pushed a commit that referenced this pull request Nov 19, 2025
We have inline project metadata in the prek hooks, and that means
that each of the scripts will use their own venv to roon. However
Airflow code and libraries does not support Python 3.14 yet (and will
not for quite a while - because our dependencies will need time to
catch-up. In the meantime we have to limit prek hooks to not install
Python 3.14.

Actually in order to have good stability, we are fine to limiting
EVERYONE to use Python 3.10 for prek. There are no real benefits from
using a different Python version (except maybe a little speed but
prek-hooks are generally rather fast anyway) and we can definitely
benefit from better stability and having exactly the same output
for prek hooks by everyone. Uv will handle installation of the
right python version automatically.
(cherry picked from commit d31e578)


Fixes: #56048

Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
ephraimbuddy pushed a commit that referenced this pull request Nov 19, 2025
We have inline project metadata in the prek hooks, and that means
that each of the scripts will use their own venv to roon. However
Airflow code and libraries does not support Python 3.14 yet (and will
not for quite a while - because our dependencies will need time to
catch-up. In the meantime we have to limit prek hooks to not install
Python 3.14.

Actually in order to have good stability, we are fine to limiting
EVERYONE to use Python 3.10 for prek. There are no real benefits from
using a different Python version (except maybe a little speed but
prek-hooks are generally rather fast anyway) and we can definitely
benefit from better stability and having exactly the same output
for prek hooks by everyone. Uv will handle installation of the
right python version automatically.
(cherry picked from commit d31e578)


Fixes: #56048

Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
ephraimbuddy pushed a commit that referenced this pull request Nov 20, 2025
We have inline project metadata in the prek hooks, and that means
that each of the scripts will use their own venv to roon. However
Airflow code and libraries does not support Python 3.14 yet (and will
not for quite a while - because our dependencies will need time to
catch-up. In the meantime we have to limit prek hooks to not install
Python 3.14.

Actually in order to have good stability, we are fine to limiting
EVERYONE to use Python 3.10 for prek. There are no real benefits from
using a different Python version (except maybe a little speed but
prek-hooks are generally rather fast anyway) and we can definitely
benefit from better stability and having exactly the same output
for prek hooks by everyone. Uv will handle installation of the
right python version automatically.
(cherry picked from commit d31e578)


Fixes: #56048

Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
aaron-wolmutt pushed a commit to aaron-wolmutt/airflow that referenced this pull request Nov 20, 2025
We have inline project metadata in the prek hooks, and that means
that each of the scripts will use their own venv to roon. However
Airflow code and libraries does not support Python 3.14 yet (and will
not for quite a while - because our dependencies will need time to
catch-up. In the meantime we have to limit prek hooks to not install
Python 3.14.

Actually in order to have good stability, we are fine to limiting
EVERYONE to use Python 3.10 for prek. There are no real benefits from
using a different Python version (except maybe a little speed but
prek-hooks are generally rather fast anyway) and we can definitely
benefit from better stability and having exactly the same output
for prek hooks by everyone. Uv will handle installation of the
right python version automatically.

Fixes: apache#56048
potiuk added a commit to potiuk/airflow that referenced this pull request Nov 28, 2025
We have inline project metadata in the prek hooks, and that means
that each of the scripts will use their own venv to roon. However
Airflow code and libraries does not support Python 3.14 yet (and will
not for quite a while - because our dependencies will need time to
catch-up. In the meantime we have to limit prek hooks to not install
Python 3.14.

Actually in order to have good stability, we are fine to limiting
EVERYONE to use Python 3.10 for prek. There are no real benefits from
using a different Python version (except maybe a little speed but
prek-hooks are generally rather fast anyway) and we can definitely
benefit from better stability and having exactly the same output
for prek hooks by everyone. Uv will handle installation of the
right python version automatically.
(cherry picked from commit d31e578)

Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Fixes: apache#56048
ephraimbuddy pushed a commit that referenced this pull request Dec 3, 2025
We have inline project metadata in the prek hooks, and that means
that each of the scripts will use their own venv to roon. However
Airflow code and libraries does not support Python 3.14 yet (and will
not for quite a while - because our dependencies will need time to
catch-up. In the meantime we have to limit prek hooks to not install
Python 3.14.

Actually in order to have good stability, we are fine to limiting
EVERYONE to use Python 3.10 for prek. There are no real benefits from
using a different Python version (except maybe a little speed but
prek-hooks are generally rather fast anyway) and we can definitely
benefit from better stability and having exactly the same output
for prek hooks by everyone. Uv will handle installation of the
right python version automatically.
(cherry picked from commit d31e578)


Fixes: #56048

Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
# under the License.
# /// script
# requires-python = ">=3.10"
# # requires-python = ">=3.10,<3.11"
Copy link
Contributor

@johnslavik johnslavik Dec 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Noticed another one similar to #58439 (comment)

Suggested change
# # requires-python = ">=3.10,<3.11"
# requires-python = ">=3.10,<3.11"

@potiuk I think this wasn't intentional?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #59047

Copilot AI pushed a commit to jason810496/airflow that referenced this pull request Dec 5, 2025
We have inline project metadata in the prek hooks, and that means
that each of the scripts will use their own venv to roon. However
Airflow code and libraries does not support Python 3.14 yet (and will
not for quite a while - because our dependencies will need time to
catch-up. In the meantime we have to limit prek hooks to not install
Python 3.14.

Actually in order to have good stability, we are fine to limiting
EVERYONE to use Python 3.10 for prek. There are no real benefits from
using a different Python version (except maybe a little speed but
prek-hooks are generally rather fast anyway) and we can definitely
benefit from better stability and having exactly the same output
for prek hooks by everyone. Uv will handle installation of the
right python version automatically.

Fixes: apache#56048
itayweb pushed a commit to itayweb/airflow that referenced this pull request Dec 6, 2025
We have inline project metadata in the prek hooks, and that means
that each of the scripts will use their own venv to roon. However
Airflow code and libraries does not support Python 3.14 yet (and will
not for quite a while - because our dependencies will need time to
catch-up. In the meantime we have to limit prek hooks to not install
Python 3.14.

Actually in order to have good stability, we are fine to limiting
EVERYONE to use Python 3.10 for prek. There are no real benefits from
using a different Python version (except maybe a little speed but
prek-hooks are generally rather fast anyway) and we can definitely
benefit from better stability and having exactly the same output
for prek hooks by everyone. Uv will handle installation of the
right python version automatically.

Fixes: apache#56048
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dev-tools backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

prek fails to run insert-license on github codespaces

5 participants