Resume yandex provider #33574
Resume yandex provider #33574
Conversation
boring-cyborg
bot
added
area:dev-tools
area:providers
kind:documentation
provider:yandex
labels
potiuk
changed the title
|
Oh. Welcome back |
Breeze tests needs to be updated as well. |
|
LGTM from Yandex |
|
hi, Jarek, hi Peter! there are some strange problems in tests, should I close and reopen the Pull Request? from Helm: CeleryExecutor - true - v1.24.15 v1.25.11 v1.26.6 v1.27.3 We are extremely sorry but you've hit the rare case that the credentials you got from GitHub Actions to run are expired, and we cannot do much. You have the following options now:
|
|
1). There is a "build-docs" failure to fix (the links to
The other issues are just "rebuild" needed - but when you fix the docs and push, the build will re-run for all of them |
|
@potiuk build-docs is fixed, please take a look at the second issue |
Previously we have been using provider.yaml file modification as a sign that the docker image needs rebuilding when starting image. However just modification of provider.yaml file is not a sign that the image needs rebuilding. The image needs rebuilding when provider dependencies changed, but there are many more reasons why provider.yaml file changed - especially recently provider.yaml file contains much more information and dependencies are only part of it. Provider.yaml files can also be modified by release manager wnen documentation is prepared, but none of the documentation change is a reason for rebuilding the image. This PR optimize the check for image building introducing two step process: * first we check if provider.yaml files changed * if they did, we regenerate provider dependencies by manully running the pre-commit script * then provider_dependencies.json is used instead of all providers to determine if the image needs rebuilding This has several nice side effects: * the list of files that have been modified displayed to the user is potentially much smaller (no provider.yaml files) * provider_dependencies.json is regenereated automatically when you run any breeze command, which means that you do not have to have pre-commit installed to regenerate it * the notification "image needs rebuilding" will be printed less frequently to the user - only when it is really needed * preparing provider documentation in CI will not trigger image rebuilding (which might occasionally fail in such case especially when we bring back a provider from long suspension like it happened in apache#33574
Interesting one. It actually revealed an interesting bug/edge case/optimization possibility. And it motivated me to implement #33603 - optimization that I wanted to do for some time. What happened in this case - the job that failed was run as part of provider check and in this check we do two things:
The point 1) modified provider.yaml files for all the providers that were updated since last release (including yandex) Then, when 2) was attempted, breeze detected that provider.yaml files changed and attempted to rebuild the CI image - normally it is a "no-op" because we just built the image with the constraints. But yandex provider has no constraints generated, so The optimization in #33603 will solve it in a bit indirect way - when provider.yaml are updated by documentation preparation, breeze will first attempt to regenerate "generated/provider_dependencies.json" and only this file change it will attempt to rebuild the image. Result - slightly faster and more accurate "check providers" job in CI, less frequent prompts for rebuilding the image for regular developers in breeze, the "generated/provider_dependencies.json" will be generated as needed when you just run breeze, without the need to have pre-commit installed. Once #33603 gets merged. you should rebase and I believe the job will succeed. |
* Improve detection of when breeze CI image needs rebuilding Previously we have been using provider.yaml file modification as a sign that the docker image needs rebuilding when starting image. However just modification of provider.yaml file is not a sign that the image needs rebuilding. The image needs rebuilding when provider dependencies changed, but there are many more reasons why provider.yaml file changed - especially recently provider.yaml file contains much more information and dependencies are only part of it. Provider.yaml files can also be modified by release manager wnen documentation is prepared, but none of the documentation change is a reason for rebuilding the image. This PR optimize the check for image building introducing two step process: * first we check if provider.yaml files changed * if they did, we regenerate provider dependencies by manully running the pre-commit script * then provider_dependencies.json is used instead of all providers to determine if the image needs rebuilding This has several nice side effects: * the list of files that have been modified displayed to the user is potentially much smaller (no provider.yaml files) * provider_dependencies.json is regenereated automatically when you run any breeze command, which means that you do not have to have pre-commit installed to regenerate it * the notification "image needs rebuilding" will be printed less frequently to the user - only when it is really needed * preparing provider documentation in CI will not trigger image rebuilding (which might occasionally fail in such case especially when we bring back a provider from long suspension like it happened in #33574 * Update dev/breeze/src/airflow_breeze/commands/developer_commands.py
potiuk
force-pushed
the
resume-yandex-provider
branch
from
e389034
to
d6f6845
Compare
|
OK. Merged my change. Now rebased to see if it helps |
…output. new generated provider_dependencies.json
…thers in suspension
potiuk
force-pushed
the
resume-yandex-provider
branch
from
d6f6845
to
3ca8d58
Compare
|
The ARM image failure is irrelevant. Merging. |
|
AAAND merged ! |
yayy thanks! |
|
@potiuk hello again! just small question - when do you think these changes will be released? |
|
Question to @eladkal but we usually release providers every 2-3 weeks. We also would likely want to release providers before 2.7.1 is out (~ week or so). So generally speaking sooner rather than later |
|
I'll do my best to get to it soon |
* Improve detection of when breeze CI image needs rebuilding Previously we have been using provider.yaml file modification as a sign that the docker image needs rebuilding when starting image. However just modification of provider.yaml file is not a sign that the image needs rebuilding. The image needs rebuilding when provider dependencies changed, but there are many more reasons why provider.yaml file changed - especially recently provider.yaml file contains much more information and dependencies are only part of it. Provider.yaml files can also be modified by release manager wnen documentation is prepared, but none of the documentation change is a reason for rebuilding the image. This PR optimize the check for image building introducing two step process: * first we check if provider.yaml files changed * if they did, we regenerate provider dependencies by manully running the pre-commit script * then provider_dependencies.json is used instead of all providers to determine if the image needs rebuilding This has several nice side effects: * the list of files that have been modified displayed to the user is potentially much smaller (no provider.yaml files) * provider_dependencies.json is regenereated automatically when you run any breeze command, which means that you do not have to have pre-commit installed to regenerate it * the notification "image needs rebuilding" will be printed less frequently to the user - only when it is really needed * preparing provider documentation in CI will not trigger image rebuilding (which might occasionally fail in such case especially when we bring back a provider from long suspension like it happened in #33574 * Update dev/breeze/src/airflow_breeze/commands/developer_commands.py (cherry picked from commit ac0d5b3)
ephraimbuddy
added
the
changelog:skip
Yandex Provider was suspended in #30667
This PR is for resuming Yandex Provider in Airflow releases
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.