Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sync v2-10-stable with v2-10-test to release python client v2.10.0 #41610

Merged
merged 18 commits into from
Aug 22, 2024
Merged

Sync v2-10-stable with v2-10-test to release python client v2.10.0 #41610

merged 18 commits into from
Aug 22, 2024

Conversation

utkarsharma2
Copy link
Contributor

No description provided.

potiuk and others added 10 commits August 14, 2024 15:06
@potiuk
Enable pull requests to be run from v*test branches (apache#41474) (a…
b211936 
…pache#41476)

Since we switch from direct push of cherry-picking to open PRs
against v*test branch, we should enable PRs to run for the target
branch.

(cherry picked from commit a9363e6)
@potiuk
Prevent provider lowest-dependency tests to run in non-main branch (a…
5934272 
…pache#41478) (apache#41481)

When running tests in v2-10-test branch, lowest depenency tests
are run for providers - because when calculating separate tests,
the "skip_provider_tests" has not been used to filter them out.

This PR fixes it.

(cherry picked from commit 75da507)
@potiuk
Make PROD image building works in non-main PRs (apache#41480) (apache…
bf0d412 
…#41484)

The PROD image building fails currently in non-main because it
attempts to build source provider packages rather than use them from
PyPi when PR is run against "v-test" branch.

This PR fixes it:

* PROD images in non-main-targetted build will pull providers from
  PyPI rather than build them
* they use PyPI constraints to install the providers
* they use UV - which should speed up building of the images

(cherry picked from commit 4d5f1c4)
@bbovenzi @molcay
Add WebEncoder for trigger page rendering to avoid render failure (ap…
5f8230b 
…ache#41350) (apache#41485)

Co-authored-by: M. Olcay Tercanlı <muhammed_tercanli@epam.com>
@potiuk @howardyoo
Incorrect try number subtraction producing invalid span id for OTEL a…
30cfb73 
…irflow (issue apache#41501) (apache#41502) (apache#41535)

* Fix for issue apache#39336

* removed unnecessary import

(cherry picked from commit dd3c3a7)

Co-authored-by: Howard Yoo <32691630+howardyoo@users.noreply.github.com>
@potiuk
Fix failing pydantic v1 tests (apache#41534) (apache#41541)
fa4ee68 
We need to exclude some versions of Pydantic v1 because it conflicts
with aws provider.

(cherry picked from commit a033c5f)
@potiuk
Fix Non-DB test calculation for main builds (apache#41499) (apache#41543
e36e521 
)

Pytest has a weird behaviour that it will not collect tests
from parent folder when subfolder of it is specified after the
parent folder. This caused some non-db tests from providers folder
have been skipped during main build.

The issue in Pytest 8.2 (used to work before) is tracked at
pytest-dev/pytest#12605

(cherry picked from commit d489826)
@utkarsharma2
Add changelog for airflow python client 2.10.0 (apache#41583) (apache…
3ea764f 
…#41584)

* Add changelog for airflow python client 2.10.0

* Update client version

(cherry picked from commit 317a28e)
@potiuk
Make all test pass in Database Isolation mode (apache#41567)
6441563 
This adds dedicated "DatabaseIsolation" test to airflow v2-10-test
branch..

The DatabaseIsolation test will run all "db-tests" with enabled
DB isolation mode and running `internal-api` component - groups
of tests marked with "skip-if-database-isolation" will be skipped.
@utkarsharma2 @potiuk
Upgrade build and chart dependencies (apache#41570) (apache#41588)
e76dba6 
(cherry picked from commit c88192c)

Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
@boring-cyborg boring-cyborg bot added area:API Airflow's REST/HTTP API area:CLI area:dev-tools area:Executors-core LocalExecutor & SequentialExecutor area:helm-chart Airflow Helm Chart area:production-image Production image improvements and fixes area:Scheduler including HA (high availability) scheduler area:webserver Webserver related Issues labels Aug 20, 2024
@potiuk
Copy link
Member

potiuk commented Aug 21, 2024

Correction:

#41344 and #41414

@potiuk
Copy link
Member

potiuk commented Aug 21, 2024

Also this one: #41626 should fix "latest boto" tests when merged.

@jscheffl
Copy link
Contributor

Hey @utkarsharma2 -> I believe we are getting very close :). There is this one flaky test with mapping and I think it shoudl be fixed by cherry-picking #41414 - this is the only "test" change for database isolation mode that did not make it to v2-10-test branch as far as I can see (cc: @jscheffl - can you maybe take a look as well to double check?)

For the PRs I mentioned I am not sure - they are okay to get to v2-10-test (shall I make PRs for this, lost track about the vast amount of patches we made for isolation mode) but in the last build I only see two problems left caused by boto?

@jscheffl @utkarsharma2
Fix tests/models/test_variable.py for database isolation mode (apache…
055b81d 
…#41414)

* Fix tests/models/test_variable.py for database isolation mode

* Review feedback

(cherry picked from commit 736ebfe)
@utkarsharma2
Copy link
Contributor Author

Hey @utkarsharma2 -> I believe we are getting very close :). There is this one flaky test with mapping and I think it shoudl be fixed by cherry-picking #41414 - this is the only "test" change for database isolation mode that did not make it to v2-10-test branch as far as I can see (cc: @jscheffl - can you maybe take a look as well to double check?)

Yes, we are! Thanks for looking into this @potiuk :)

Correction:
#41344 and #41414

#41344 is already part of the branch. I have cherry-picked #41414

I'm waiting for us to merge #41626 to the main so I can cherry-pick this as well.

@potiuk
Copy link
Member

potiuk commented Aug 21, 2024

I'm waiting for us to merge #41626 to the main so I can cherry-pick this as well.

Merged

@potiuk @utkarsharma2
Make latest botocore tests green (apache#41626)
3dfe19b 
The latest botocore tests are conflicting with a few requirements
and until apache-beam upcoming version is released we need to do
some manual exclusions. Those exclusions should make latest botocore
test green again.

(cherry picked from commit a13ccbb)
@potiuk
Copy link
Member

potiuk commented Aug 21, 2024

Hmm. We still have the flaky D tests/models/test_taskinstance.py::TestTaskInstance::test_check_task_dependencies_for_mapped - and I do recall having it (and fixing) in main... Let me try to dig deeper and find what we are missing here.

@potiuk
Copy link
Member

potiuk commented Aug 21, 2024

cc: @jscheffl -> maybe that one rings a bell for you ?

@potiuk
Copy link
Member

potiuk commented Aug 21, 2024

Anh I know. @utkarsharma2 - this should be fixed by cherry-picking #41389

@potiuk
Copy link
Member

potiuk commented Aug 21, 2024 •
edited
Loading

And #41471 should also make "Database Isolation tests" stable.

@potiuk
Copy link
Member

potiuk commented Aug 21, 2024

Yeah I think the last two @utkarsharma2 #41389 and #41471 should bring the stability back (those are test-only changes).

@potiuk @utkarsharma2
Simpler task retrieval for taskinstance test (apache#41389)
3b09b27 
The test has been updated for DB isolation but the retrieval of
task was not intuitive and it could lead to flaky tests possibly

(cherry picked from commit f25adf1)
@potiuk @utkarsharma2
Skip database isolation case for task mapping taskinstance tests (apa…
15bb0aa 
…che#41471)

Related: apache#41067
(cherry picked from commit 7718bd7)
@utkarsharma2
Copy link
Contributor Author

Yeah I think the last two @utkarsharma2 #41389 and #41471 should bring the stability back (those are test-only changes).

Thanks @potiuk I have cherry-picked them.

@potiuk
Copy link
Member

potiuk commented Aug 21, 2024

OK. I think the remaining ones are the "regular" flakes - that we have no good solution for now - so just re-running them after tests complete should help

@jscheffl
Copy link
Contributor

OK. I think the remaining ones are the "regular" flakes - that we have no good solution for now - so just re-running them after tests complete should help

I double-checked the list of PRs we had for making DB isolation tests working, don't know if implicitly contained but #41450 was also merged on main and seems to be missing in v2-10-test.

@potiuk
Copy link
Member

potiuk commented Aug 21, 2024

Green ! :)

@utkarsharma2 utkarsharma2 merged commit 91f8265 into apache:v2-10-stable Aug 22, 2024
99 checks passed
@utkarsharma2 utkarsharma2 deleted the Sync-v2_10_test branch August 22, 2024 09:00
utkarsharma2 added a commit that referenced this pull request Aug 30, 2024
@utkarsharma2 @potiuk
@bbovenzi @molcay @howardyoo @jscheffl @bugraoz93
Sync v2-10-stable with v2-10-test to release python client v2.10.0 (#…
5314424 
…41610)

* Enable pull requests to be run from v*test branches (#41474) (#41476)

Since we switch from direct push of cherry-picking to open PRs
against v*test branch, we should enable PRs to run for the target
branch.

(cherry picked from commit a9363e6)

* Prevent provider lowest-dependency tests to run in non-main branch (#41478) (#41481)

When running tests in v2-10-test branch, lowest depenency tests
are run for providers - because when calculating separate tests,
the "skip_provider_tests" has not been used to filter them out.

This PR fixes it.

(cherry picked from commit 75da507)

* Make PROD image building works in non-main PRs (#41480) (#41484)

The PROD image building fails currently in non-main because it
attempts to build source provider packages rather than use them from
PyPi when PR is run against "v-test" branch.

This PR fixes it:

* PROD images in non-main-targetted build will pull providers from
  PyPI rather than build them
* they use PyPI constraints to install the providers
* they use UV - which should speed up building of the images

(cherry picked from commit 4d5f1c4)

* Add WebEncoder for trigger page rendering to avoid render failure (#41350) (#41485)

Co-authored-by: M. Olcay Tercanlı <muhammed_tercanli@epam.com>

* Incorrect try number subtraction producing invalid span id for OTEL airflow (issue #41501) (#41502) (#41535)

* Fix for issue #39336

* removed unnecessary import

(cherry picked from commit dd3c3a7)

Co-authored-by: Howard Yoo <32691630+howardyoo@users.noreply.github.com>

* Fix failing pydantic v1 tests (#41534) (#41541)

We need to exclude some versions of Pydantic v1 because it conflicts
with aws provider.

(cherry picked from commit a033c5f)

* Fix Non-DB test calculation for main builds (#41499) (#41543)

Pytest has a weird behaviour that it will not collect tests
from parent folder when subfolder of it is specified after the
parent folder. This caused some non-db tests from providers folder
have been skipped during main build.

The issue in Pytest 8.2 (used to work before) is tracked at
pytest-dev/pytest#12605

(cherry picked from commit d489826)

* Add changelog for airflow python client 2.10.0 (#41583) (#41584)

* Add changelog for airflow python client 2.10.0

* Update client version

(cherry picked from commit 317a28e)

* Make all test pass in Database Isolation mode (#41567)

This adds dedicated "DatabaseIsolation" test to airflow v2-10-test
branch..

The DatabaseIsolation test will run all "db-tests" with enabled
DB isolation mode and running `internal-api` component - groups
of tests marked with "skip-if-database-isolation" will be skipped.

* Upgrade build and chart dependencies (#41570) (#41588)

(cherry picked from commit c88192c)

Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

* Limit watchtower as depenendcy as 3.3.0 breaks moin. (#41612)

(cherry picked from commit 1b602d5)

* Enable running Pull Requests against v2-10-stable branch (#41624)

(cherry picked from commit e306e7f)

* Fix tests/models/test_variable.py for database isolation mode (#41414)

* Fix tests/models/test_variable.py for database isolation mode

* Review feedback

(cherry picked from commit 736ebfe)

* Make latest botocore tests green (#41626)

The latest botocore tests are conflicting with a few requirements
and until apache-beam upcoming version is released we need to do
some manual exclusions. Those exclusions should make latest botocore
test green again.

(cherry picked from commit a13ccbb)

* Simpler task retrieval for taskinstance test (#41389)

The test has been updated for DB isolation but the retrieval of
task was not intuitive and it could lead to flaky tests possibly

(cherry picked from commit f25adf1)

* Skip  database isolation case for task mapping taskinstance tests (#41471)

Related: #41067
(cherry picked from commit 7718bd7)

* Skipping tests for db isolation because similar tests were skipped (#41450)

(cherry picked from commit e94b508)

---------

Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: Brent Bovenzi <brent@astronomer.io>
Co-authored-by: M. Olcay Tercanlı <muhammed_tercanli@epam.com>
Co-authored-by: Howard Yoo <32691630+howardyoo@users.noreply.github.com>
Co-authored-by: Jens Scheffler <95105677+jscheffl@users.noreply.github.com>
Co-authored-by: Bugra Ozturk <bugraoz93@users.noreply.github.com>
utkarsharma2 added a commit that referenced this pull request Aug 30, 2024
@utkarsharma2 @potiuk
@bbovenzi @molcay @howardyoo @jscheffl @bugraoz93
Sync v2-10-stable with v2-10-test to release python client v2.10.0 (#…
b08c862 
…41610)

* Enable pull requests to be run from v*test branches (#41474) (#41476)

Since we switch from direct push of cherry-picking to open PRs
against v*test branch, we should enable PRs to run for the target
branch.

(cherry picked from commit a9363e6)

* Prevent provider lowest-dependency tests to run in non-main branch (#41478) (#41481)

When running tests in v2-10-test branch, lowest depenency tests
are run for providers - because when calculating separate tests,
the "skip_provider_tests" has not been used to filter them out.

This PR fixes it.

(cherry picked from commit 75da507)

* Make PROD image building works in non-main PRs (#41480) (#41484)

The PROD image building fails currently in non-main because it
attempts to build source provider packages rather than use them from
PyPi when PR is run against "v-test" branch.

This PR fixes it:

* PROD images in non-main-targetted build will pull providers from
  PyPI rather than build them
* they use PyPI constraints to install the providers
* they use UV - which should speed up building of the images

(cherry picked from commit 4d5f1c4)

* Add WebEncoder for trigger page rendering to avoid render failure (#41350) (#41485)

Co-authored-by: M. Olcay Tercanlı <muhammed_tercanli@epam.com>

* Incorrect try number subtraction producing invalid span id for OTEL airflow (issue #41501) (#41502) (#41535)

* Fix for issue #39336

* removed unnecessary import

(cherry picked from commit dd3c3a7)

Co-authored-by: Howard Yoo <32691630+howardyoo@users.noreply.github.com>

* Fix failing pydantic v1 tests (#41534) (#41541)

We need to exclude some versions of Pydantic v1 because it conflicts
with aws provider.

(cherry picked from commit a033c5f)

* Fix Non-DB test calculation for main builds (#41499) (#41543)

Pytest has a weird behaviour that it will not collect tests
from parent folder when subfolder of it is specified after the
parent folder. This caused some non-db tests from providers folder
have been skipped during main build.

The issue in Pytest 8.2 (used to work before) is tracked at
pytest-dev/pytest#12605

(cherry picked from commit d489826)

* Add changelog for airflow python client 2.10.0 (#41583) (#41584)

* Add changelog for airflow python client 2.10.0

* Update client version

(cherry picked from commit 317a28e)

* Make all test pass in Database Isolation mode (#41567)

This adds dedicated "DatabaseIsolation" test to airflow v2-10-test
branch..

The DatabaseIsolation test will run all "db-tests" with enabled
DB isolation mode and running `internal-api` component - groups
of tests marked with "skip-if-database-isolation" will be skipped.

* Upgrade build and chart dependencies (#41570) (#41588)

(cherry picked from commit c88192c)

Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

* Limit watchtower as depenendcy as 3.3.0 breaks moin. (#41612)

(cherry picked from commit 1b602d5)

* Enable running Pull Requests against v2-10-stable branch (#41624)

(cherry picked from commit e306e7f)

* Fix tests/models/test_variable.py for database isolation mode (#41414)

* Fix tests/models/test_variable.py for database isolation mode

* Review feedback

(cherry picked from commit 736ebfe)

* Make latest botocore tests green (#41626)

The latest botocore tests are conflicting with a few requirements
and until apache-beam upcoming version is released we need to do
some manual exclusions. Those exclusions should make latest botocore
test green again.

(cherry picked from commit a13ccbb)

* Simpler task retrieval for taskinstance test (#41389)

The test has been updated for DB isolation but the retrieval of
task was not intuitive and it could lead to flaky tests possibly

(cherry picked from commit f25adf1)

* Skip  database isolation case for task mapping taskinstance tests (#41471)

Related: #41067
(cherry picked from commit 7718bd7)

* Skipping tests for db isolation because similar tests were skipped (#41450)

(cherry picked from commit e94b508)

---------

Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: Brent Bovenzi <brent@astronomer.io>
Co-authored-by: M. Olcay Tercanlı <muhammed_tercanli@epam.com>
Co-authored-by: Howard Yoo <32691630+howardyoo@users.noreply.github.com>
Co-authored-by: Jens Scheffler <95105677+jscheffl@users.noreply.github.com>
Co-authored-by: Bugra Ozturk <bugraoz93@users.noreply.github.com>
@utkarsharma2 utkarsharma2 mentioned this pull request Sep 2, 2024
Closed
23 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@phanikumv phanikumv phanikumv approved these changes

@ephraimbuddy ephraimbuddy ephraimbuddy approved these changes

@potiuk potiuk potiuk approved these changes

@ashb ashb Awaiting requested review from ashb ashb is a code owner

@jedcunningham jedcunningham Awaiting requested review from jedcunningham jedcunningham is a code owner

@kaxil kaxil Awaiting requested review from kaxil kaxil is a code owner

@uranusjr uranusjr Awaiting requested review from uranusjr uranusjr is a code owner

@ryanahamilton ryanahamilton Awaiting requested review from ryanahamilton ryanahamilton is a code owner

@bbovenzi bbovenzi Awaiting requested review from bbovenzi bbovenzi is a code owner

@pierrejeambrun pierrejeambrun Awaiting requested review from pierrejeambrun pierrejeambrun is a code owner

@dstandish dstandish Awaiting requested review from dstandish dstandish is a code owner

@hussein-awala hussein-awala Awaiting requested review from hussein-awala hussein-awala is a code owner

@XD-DENG XD-DENG Awaiting requested review from XD-DENG XD-DENG is a code owner

@o-nikolas o-nikolas Awaiting requested review from o-nikolas o-nikolas is a code owner

@eladkal eladkal Awaiting requested review from eladkal eladkal is a code owner

@bolkedebruin bolkedebruin Awaiting requested review from bolkedebruin bolkedebruin is a code owner

Assignees
No one assigned
Labels
area:API Airflow's REST/HTTP API area:CLI area:dev-tools area:Executors-core LocalExecutor & SequentialExecutor area:helm-chart Airflow Helm Chart area:production-image Production image improvements and fixes area:Scheduler including HA (high availability) scheduler area:webserver Webserver related Issues
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@utkarsharma2 @pierrejeambrun @potiuk @jscheffl @ephraimbuddy @phanikumv @bbovenzi @bugraoz93