Add "update-providers-next-version" command to manage common provider…

[potiuk]

…s bump

When contributors want to upgrade to the next version of a dependent provider (for example when new feature is added to a common providers and it is already used in another provider) - instead of bumping the minimum version of the dependent provider, contributors should mark it with # use next version comment. This comment will be picked by the new command upgrade-providers-next-version and the bump will happen after provider documentation is prepared for release.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[potiuk]

This is what RM will see:

Now the only remaining part is how we make sure the comments are added.

[kacpermuda]

The more I think about it, the more questions/thoughts I have, but maybe they are all related to each other (I'll use common.compat as example):

1. When this use next version comment is encountered, we use the value from provider.yml of specific provider, make a replacement and we remove this comment. Is there any mechanism that forces the release manager to first bump the versions in provider.yml of all providers and then run this automation? Otherwise, we may end up silently removing this comment BEFORE versions are bumped, therefore not actually bumping the dependency - possibly releasing provider with broken features (it will try to use features from new version of some provider without requiring it in dependencies).
2. What is the incentive for contributors to use this method over explicitly bumping this version in the provider.yml of the dependent provider (e.g. common.compat)? I for sure would like to be certain, that this change is made and not merely left as a comment that may or may not be implemented to avoid potential problem with releasing provider with broken features (see above).
3. I guess it's related to the above, but does this mean that contributors have two ways of bumping this? One can bump the version of common.compat in its provider.yml and use new version in parent provider, or just leave the comment in parent provider and count that it will be seen? May be a bit confusing if there are two ways of achieving the same, and possibly, what if one contributor chooses first and the other chooses the second - we'll probably count on release manager to spot that it has already been bumped and he does not have to do it manually and then run the automation to update other provider.yml files.

[kacpermuda]

Ad. 2 - I mean that I'd like to be sure that the dependency is bumped in parent provider (as it's critical for new features to work), so I'd go and bump the common.compat and then use the updated version as dependency in parent provider. Not sure if I'd trust that this comment will be replaced before releasing unless there is some check blocking it.

[potiuk]

The comment WILL be replaced.. it is now (with this PR) part of the regular release management process. So you can be sure

[potiuk]

The more I think about it, the more questions/thoughts I have, but maybe they are all related to each other (I'll use common.compat as example):

1. When this use next version comment is encountered, we use the value from provider.yml of specific provider, make a replacement and we remove this comment. Is there any mechanism that forces the release manager to first bump the versions in provider.yml

1. Yes. Release process sequence

2. What is the incentive for contributors to use this method over explicitly bumping this version in the provider.yml of the dependent provider (e.g. common.compat)? I for sure would like to be certain, that this change is made and not merely left as a comment that may or may not be implemented to avoid potential problem with releasing provider with broken features (see above).

2. They currently can't do it

3. I guess it's related to the above, but does this mean that contributors have two ways of bumping this?

3. No. currently Contributors CAN'T and SHOULD NoT BUMP versions at all. Only RM can. Look at the comment at the top of project.toml

[potiuk]

Generally with that one ONLY RM should be able to bump provider versions. Zero exceptions

[potiuk]

I am thinking about selective check failing and adding a label If someone tries to modify a cross dependency in project toml. So that RM will have to set that label on their PR.

Thus however still does not fully solve the problem of how to communicate contributors what they should do (i.e. add the right comment) any ideas ?

[kacpermuda]

I am thinking about selective check failing and adding a label If someone tries to modify a cross dependency in project toml. So that RM will have to set that label on their PR.
Thus however still does not fully solve the problem of how to communicate contributors what they should do (i.e. add the right comment) any ideas ?

If this CI check's fail message would point to the doc paragraph about adding use next version comment, would that solve the problem?

[bugraoz93]

Selective checks is a good idea on PRs. Maybe we can think about a prek hook for it 🤔

[potiuk]

If this CI check's fail message would point to the doc paragraph about adding use next version comment, would that solve the problem?

Yes. But again that's part of the problem only - because what happens if the contributes won't even try :) ?

[potiuk]

Selective checks is a good idea on PRs. Maybe we can think about a prek hook for it 🤔

Prek is not best - because we need to see "what's coming in PR" - prek hook are not really good for it - we tried.

Prek naturally works on all files by default or on currently staged changes - and it consider those files as "whole" - and you need to do some extra checks to compare current vs. previous version and it might be heavily skewed anyway - becuase you might not be rebased, and have multiple commits and such - so naturally the scope of prek hook is current commit.

What we are looking for here is "change vs previous commit" - because we only want to flag situation when someone wants to change the dependent version in their PR (i.e. we have to compare vs. target branch - not what happened in a single commit).. Note that it should not flag if "other" parts of pyproject.toml changed -- we should only flag it if "version changed in the pr"

[potiuk]

I will add those precautions later. In the meantime I would love to merge this one and #58390 so that I can try it on rc2 :)

[github-actions]

Backport failed to create: v3-1-test. View the failure log Run details

Status	Branch	Result
❌	v3-1-test

You can attempt to backport this manually by running:

cherry_picker 61a3c55 v3-1-test

This should apply the commit to the v3-1-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

[potiuk]

cc: @kacpermuda @bugraoz93 @jscheffl -> #58412 this is how upgrade will look like in doc-preparation PR. I think it is fairly straightforward. Now we will have to have guardrails to prevent contributors to bump those cross-dependencies.

[potiuk]

cc: @eladkal -> I think that one will nicely solve the problem of bumping versions of dependent providers - only RM will be able to update them in "prepare-documentation" step.

[bugraoz93]

Looks really centralized for providers. I like the approach. Making RM life easier. Decoupling and more automation on structured approach good for our release processes

[amoghrajesh]

@potiuk selective checks / prek hook to ensure nobody bumps the common provider version in pyproject.toml for other providers is a good idea

[kacpermuda]

selective checks / prek hook to ensure nobody bumps the common provider version in pyproject.toml for other providers is a good idea

I think we can probably generalize to all providers and not only common ? Not sure what to in the case when someone wants to bump f.e. sql provider from 1.1 to 1.2 when the latest is already 1.5 - we'd probably have to check the current latest and allow this?

[potiuk]

I think we can probably generalize to all providers and not only common ?

Yes.

Not sure what to in the case when someone wants to bump f.e. sql provider from 1.1 to 1.2 when the latest is already 1.5 - we'd probably have to check the current latest and allow this?

Yes. But this can be done by just assigning the same label that release manager will assign to their PR.

[potiuk]

PR. Here #58430