Simplifies check whether the CI image should be rebuilt #12181
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Together with just merged #12177 we should finally handle all the cases where Breeze correctly detects when it is faster to build image locally or whether to pull it first. |
potiuk
force-pushed
the
simplify-manifest-check-for-image-rebuild
branch
from
cbdcc80 to
e67bdc9
Compare
|
The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*. |
|
The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*. |
potiuk
force-pushed
the
simplify-manifest-check-for-image-rebuild
branch
2 times, most recently
from
ab94da1 to
1971f27
Compare
|
I would love to get that in and work today on rock-solid image pulling. This is the last part of what I wanted to do with simplifying the detection of "pull vs. build" and with this in place I think I will finally be able to solve some of the remaining unnecessary rebuilts of Breeze. Looking forward to some reviews :) |
potiuk
force-pushed
the
simplify-manifest-check-for-image-rebuild
branch
from
1971f27 to
359fc6c
Compare
feluelle
reviewed
Nov 12, 2020
|
Thanks @feluelle ! Seems I am getting worse with the typos :D |
|
Hey @feluelle. All Good it seems! |
feluelle
approved these changes
Nov 13, 2020
github-actions
bot
added
the
full tests needed
|
The PR needs to run all tests because it modifies core of Airflow! Please rebase it to latest master or ask committer to re-run it! |
potiuk
force-pushed
the
simplify-manifest-check-for-image-rebuild
branch
2 times, most recently
from
da4df98 to
490c469
Compare
|
Hey @feluelle -> i did a few more modifications to make it even clearer to know what's going on and double-checked what happens for migration for those who have the current image/manifest. It should be rather smooth migration now :). |
potiuk
force-pushed
the
simplify-manifest-check-for-image-rebuild
branch
from
490c469 to
41e5860
Compare
Rather than counting changed layers in the image (which was enigmatic, difficult and prone to some magic number) we rely now on random file generated while building the image. We are using the docker image caching mechanism here. The random file will be regenerated only when the previous layer (which is about installling Airflow dependencies for the first time) gets rebuild. And for us this is the indication, that the building the image will take quite some time. This layer should be relatively static - even if setup.py changes the CI image is designed in the way that the first time installation of Airflow dependencies is not invalidated. This should lead to faster and less frequent rebuild for people using Breeze and static checks.
potiuk
force-pushed
the
simplify-manifest-check-for-image-rebuild
branch
from
41e5860 to
2d639d9
Compare
potiuk
added a commit
that referenced
this pull request
Nov 14, 2020
Rather than counting changed layers in the image (which was enigmatic, difficult and prone to some magic number) we rely now on random file generated while building the image. We are using the docker image caching mechanism here. The random file will be regenerated only when the previous layer (which is about installling Airflow dependencies for the first time) gets rebuild. And for us this is the indication, that the building the image will take quite some time. This layer should be relatively static - even if setup.py changes the CI image is designed in the way that the first time installation of Airflow dependencies is not invalidated. This should lead to faster and less frequent rebuild for people using Breeze and static checks. (cherry picked from commit 167b9b9)
potiuk
added
the
type:misc/internal
potiuk
added a commit
that referenced
this pull request
Nov 16, 2020
Rather than counting changed layers in the image (which was enigmatic, difficult and prone to some magic number) we rely now on random file generated while building the image. We are using the docker image caching mechanism here. The random file will be regenerated only when the previous layer (which is about installling Airflow dependencies for the first time) gets rebuild. And for us this is the indication, that the building the image will take quite some time. This layer should be relatively static - even if setup.py changes the CI image is designed in the way that the first time installation of Airflow dependencies is not invalidated. This should lead to faster and less frequent rebuild for people using Breeze and static checks. (cherry picked from commit 167b9b9)
potiuk
added a commit
that referenced
this pull request
Nov 16, 2020
Rather than counting changed layers in the image (which was enigmatic, difficult and prone to some magic number) we rely now on random file generated while building the image. We are using the docker image caching mechanism here. The random file will be regenerated only when the previous layer (which is about installling Airflow dependencies for the first time) gets rebuild. And for us this is the indication, that the building the image will take quite some time. This layer should be relatively static - even if setup.py changes the CI image is designed in the way that the first time installation of Airflow dependencies is not invalidated. This should lead to faster and less frequent rebuild for people using Breeze and static checks. (cherry picked from commit 167b9b9)
kaxil
pushed a commit
that referenced
this pull request
Nov 18, 2020
Rather than counting changed layers in the image (which was enigmatic, difficult and prone to some magic number) we rely now on random file generated while building the image. We are using the docker image caching mechanism here. The random file will be regenerated only when the previous layer (which is about installling Airflow dependencies for the first time) gets rebuild. And for us this is the indication, that the building the image will take quite some time. This layer should be relatively static - even if setup.py changes the CI image is designed in the way that the first time installation of Airflow dependencies is not invalidated. This should lead to faster and less frequent rebuild for people using Breeze and static checks. (cherry picked from commit 167b9b9)
cfei18
pushed a commit
to cfei18/incubator-airflow
that referenced
this pull request
Mar 5, 2021
Rather than counting changed layers in the image (which was enigmatic, difficult and prone to some magic number) we rely now on random file generated while building the image. We are using the docker image caching mechanism here. The random file will be regenerated only when the previous layer (which is about installling Airflow dependencies for the first time) gets rebuild. And for us this is the indication, that the building the image will take quite some time. This layer should be relatively static - even if setup.py changes the CI image is designed in the way that the first time installation of Airflow dependencies is not invalidated. This should lead to faster and less frequent rebuild for people using Breeze and static checks. (cherry picked from commit 167b9b9)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Rather than counting changed layers in the image (which was
enigmatic, difficult and prone to some magic number) we rely now
on random file generated while building the image.
We are using the docker image caching mechanism here. The random
file will be regenerated only when the previous layer (which is
about installling Airflow dependencies for the first time) gets
rebuild. And for us this is the indication, that the building
the image will take quite some time. This layer should be
relatively static - even if setup.py changes the CI image is
designed in the way that the first time installation of Airflow
dependencies is not invalidated.
This should lead to faster and less frequent rebuild for people
using Breeze and static checks.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.