apache-airflow-providers-cncf-kubernetes requires latest kubernetes version

[paramjeet01]

Description

Update apache-airflow-providers-cncf-kubernetes to use kubernetes latest versions

kubernetes	>=21.7.0,<24

Use case/motivation

Kubernetes has released latest versions recently v1.29 but airflow kubernetes is stuck with v1.23 and not compatible with the latest versions.
https://github.com/kubernetes-client/python

Related issues

Since the kubernetes v1.23 has come to end of life and eks is not supporting the version. It would be nice to see upgraded
apache-airflow-providers-cncf-kubernetes

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[raphaelauv]

[gbonazzoli]

@paramjeet01 Well said !!!

I also add that working with apache-airflow-providers-cncf-kubernetes on a cluster kubernetes 1.28.x it seems to have logging capture problems duplicating log lines or missing them at all.

Otherall, the XComs examples in python do not work at all !!!

[raphaelauv]

@gbonazzoli

Otherall, the XComs examples in python do not work at all

it works on my side , could you please open a new issue with a reproducible example , thanks

[paramjeet01]

@raphaelauv , Thanks for the help. The above stated image for future release ?

[gbonazzoli]

The issue is said to be closed but it seems that apache-airflow-providers-cncf-kubernetes 7.14.0 does not think so.

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
apache-airflow-providers-cncf-kubernetes 7.14.0 requires kubernetes<24,>=21.7.0, but you have kubernetes 29.0.0 which is incompatible.

Am I going wrong ?

[raphaelauv]

close cause fix is merged , does not mean it was deploy in a new release

and 7.14.0 is 2 days old , It can't include a fix merged 30 minutes ago

[paramjeet01]

@raphaelauv When are we planning to release the new version ? or is it automated by pipelines and we wait till then ?

[raphaelauv]

https://github.com/apache/airflow/issues?q=is%3Aissue+Status+of+testing+Providers+is%3Aclosed+

[potiuk]

We release our providers roughly every two weeks. Byut if you REALLY want you can easily build your own provider packages we have instructions available to do so - this way you could gest it before and contribute back by verifying if all works as needed.

Do you want links to some instructions there and are you willing to help us with that @paramjeet01 ?

[paramjeet01]

Yeah , I can check that. Please send me the instructions also could this issue 37090 be related to the provider version

[potiuk]

Those are step-by-step instruction release manager does https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#prepare-regular-provider-packages-rc just follow package preparation.

It's ealso explained in detail in the hlelp of breeze commands and the commans are documented here: https://github.com/apache/airflow/blob/main/dev/breeze/doc/09_release_management_tasks.rst#provider-release-commands

[potiuk]

#37090 be related to the provider version

Possibly. Generally there is not a single person who might remember or know all the chages to all the providers and be able to authoritatively answer yes/no. This is an open-source projects with ~ 2800 contributors, so it's likely changes and fixes are implemented that are fixing things that others are unaware.

That's what the release notes are, to see if there were fixes that might be related - and generally upgrading to latest versions of everything is a good idea.

And if you can test it and try it yoursef and verify it - even building the latest packages to try things before they get relased, this is the ultimate answer.