Sync v2-7-stable with v2-7-test to release 2.7.3 #35270
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-authored-by: Hussein Awala <hussein@awala.fr> (cherry picked from commit b33595c)
(cherry picked from commit 6213d56)
* Bump azure-mgmt-containerinstance * Apply review suggestions (cherry picked from commit 9ee14a0)
WTForms 3.1.0 released 10th of October 2023 introduced a breaking change in the way QuerySelectChoices fields are handled. See details in dpgaspar/Flask-AppBuilder#2137 (cherry picked from commit 6b5d369)
(cherry picked from commit 5155e7f)
(cherry picked from commit 27443ec)
) (cherry picked from commit 1e9c807)
Add Open Search Provider --------- Co-authored-by: Hussein Awala <hussein@awala.fr> Co-authored-by: D. Ferruzzi <ferruzzi@amazon.com> (cherry picked from commit 94f1441)
(cherry picked from commit 16afba7)
(cherry picked from commit b7f532a)
PIP recently started backtracking with constraint generation and in order to fix it, more thorough investigation is needed. Source constraints generation works fine, only PyPI one is a problem. We are disabling it now to fix it temporarily. (cherry picked from commit aad573e)
We generate constraints for preview right after images are build, in order to be able to see the output and to diagnose which dependencies have been updated later in the process. So far we were doing it in CI workflow and When constraints generation failed however (for example because pip backtracking takes a log ot time), it faied build image workflow and did not allow tests to complete. This PR extracts constraint generation to CI separate job which does not block tests from running in case constraints generation fails or times out. It also moves out the steps from the composite action which allows to better see which step failed and allows to see the constraints used in each job more easily. (cherry picked from commit a450b81)
The PROD image uses source constraints to get built. The #34990 had moved the constraint generation to CI workflow which works for main builds, but for regular PRs we also need to upload them in the build-image workflow - otherwise PROD image build cannot find them as artifacts. This PR restores only source constraints upload as artifacts (this is very fast) (cherry picked from commit 40b9d2b)
The "test/operators" are one of the longest tests of ours - mainly because of a lot of Python virtualenv operators taking a lot of time to create and tear-down virtualenvs. They contributed a lot to the "Other" category making it one that run for the longest time. In order to improve usage of parallel cores, we separate operator tests to a new "Operators" test typ and let them run in parallel to Other tests - this should help in speeding up the tests overall by around 3-4 minutes on self-hosted runners. (cherry picked from commit 1b4307b)
Currently console.print failes to print output because of extra in error message. (cherry picked from commit 18eb1fc)
) So far, we regenerated the constraints in order to get them updated in constraint branches at the end of the CI build when we knew all the builds succeded. However this was not perfect for two reasons: * there was a race condition that a dependency had been released between the time images were built and tests completed. While it had no impact on "source" constraints (they reflect what is in the image), it could change the PyPI constraints generated or "no-providers" constraints, because they use "current" set of dependencies in PyPI to generate them. * generating constraints (for PyPI and "no-providers") takes time because `pip` has to resolve dependencies again - taking into account what is in the PyPI registry, and this can take a long time (minutes or even it can lead to long `pip` backtracking. We generally cancel running builds when new commit is merged in main, so this could lead to such constraint job being canceled by subsequent merge This PR uses the fact that we have now "preview-constraints" job that uploads constraints as artifacts in CI workflow, and instead of regenerating the constraints, we can download the constraints from uploaded artifact and use it - this is very quick and we can also make a clear dependency between "preview" and "update" constraints jobs - making it clear in case of PIP backtracking that we have problem with constraints, not with the image. This will make it far clearer when we will have problems with constraints and backtracking that this is the real issue we have (allowing such PRs to get merged while constraints backtracking problem is being worked on. (cherry picked from commit f115ec1)
(cherry picked from commit 902b9b3)
(cherry picked from commit 1d08e8b)
(cherry picked from commit 50f6483)
(cherry picked from commit 8d0ad02)
(cherry picked from commit d87ca90)
(cherry picked from commit 551886e)
The CI scripts in `build-image` workflow are taken from main so when you add new (preinstalled) provider to be installed for PROD image, the provider added in the PR in the file is not present in the build-image workflow. Moving the file to airflow/providers is safe because it does not contain any code - just list of packages that breeze command will use when building the PROD image. Related to #34729 (cherry picked from commit eb76276)
The constraint branch checkout needs to happen always when there is an "upgrade-to-newer-dependencies" build, not only wnen there is a "canary" main build, Otherwise when we attempt to update to diff constraints vs. the current ones the diff fails. Only the Commit and Push which are the folowing steps should happen exclusively in the canary run. (cherry picked from commit 570d88b)
(cherry picked from commit f816237)
* Fix CI's docs filter generation * Fix test cases (cherry picked from commit fdc443d)
ephraimbuddy
requested review from
uranusjr,
jhtimmins,
ryanahamilton,
bbovenzi and
XD-DENG
as code owners
boring-cyborg
bot
added
area:API
(cherry picked from commit 3747483)
The sentry_sdk 1.33.0 is broken, it fails to import when greenlet is installed. See: getsentry/sentry-python#2473 Most likely the will quickly release new version so != should be fine. (cherry picked from commit 76847f8)
ephraimbuddy
force-pushed
the
v2-7-test
branch
from
c673cc7 to
596c738
Compare
ephraimbuddy
force-pushed
the
v2-7-test
branch
from
27ab127 to
d612fa2
Compare
ephraimbuddy
commented
Nov 1, 2023
ephraimbuddy
force-pushed
the
v2-7-test
branch
from
d612fa2 to
41178d9
Compare
|
Hey @ephraimbuddy - > As discussed: I cherry-picked few more changes - I marked them with milestone and classified, but have not updated RELEASE_NOTES - I will leave it up to you:
|
(cherry picked from commit 4903b11)
This PR is separated out from #35160 in order to make the big refactor and splitting out our test harness to DB and Non-DB tests far more easy to review. This change contains purely adding db_tests markers to the tests that need them, and as such it requires very litte effort to review. Once this one is merged hoever, the #35160 will become way smaller in terms of number of files to review, which will make it far easier to review using GitHub Interface. (cherry picked from commit 1aa91a4)
This PR marks DB tests as such and allows to split execution of the tests in CI to run the DB tests with the various database while the non-db tests - without the DB in a separate run. In order to do that, the code to select which tests to run has been moved from `entrypoint_ci.sh` bash to breeze's Python code, which is generally much nicer to maintain and common for both "DB" and "non-DB" tests. This will have the nice side effect that it will be easier in the future to manage different test types and contain some specific flaky test types. This change also adds possibility to isolate some of the test types when parallel DB tests are run and adds new test type PythonOperator carved out Operator type. This test is best run in isolation becasue creating and destroing virtualenvs in Docker while running in parallel to other tests is very slow for some reason and leads to flaky tests. Python operator tests are therefore separated out from Operators and treated separately as isolated tests. This will help not only with speed but also with stability of the test suite. (cherry picked from commit a7e76ba)
--------- Co-authored-by: Arthur Volant <arthur.volant@adevinta.com> Co-authored-by: Jarek Potiuk <jarek@potiuk.com> (cherry picked from commit 66871a0)
* Use slightly nicer job name for Non-DB tests in CI To be consistend with DB tests - we show both Python version and set of test types we use for the job. * Update .github/workflows/ci.yml Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com> --------- Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com> (cherry picked from commit 7cbbc75)
* Fix pre-mature evaluation of tasks in mapped task group Getting the relevant upstream indexes of a task instance in a mapped task group should only be done when the task has expanded. If the task has not expanded yet, we should return None so that the task can wait for the upstreams before trying to run. This issue is more noticeable when the trigger rule is ONE_FAILED because then, the task instance is marked as SKIPPED. This commit fixes this issue. closes: #34023 * fixup! Fix pre-mature evaluation of tasks in mapped task group * fixup! fixup! Fix pre-mature evaluation of tasks in mapped task group * fixup! fixup! fixup! Fix pre-mature evaluation of tasks in mapped task group * Fix tests (cherry picked from commit 69938fd)
ephraimbuddy
force-pushed
the
v2-7-test
branch
from
9c88dae to
f124353
Compare
potiuk
approved these changes
Nov 1, 2023
hussein-awala
approved these changes
Nov 2, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Time for
2.7.3rc1!