Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Status of testing Providers that were prepared on February 12, 2024 #37358

Closed
57 of 81 tasks
eladkal opened this issue Feb 12, 2024 · 37 comments
Closed
57 of 81 tasks

Status of testing Providers that were prepared on February 12, 2024 #37358

eladkal opened this issue Feb 12, 2024 · 37 comments
Labels
kind:meta High-level information important to the community testing status Status of testing releases

Comments

@eladkal
Copy link
Contributor

eladkal commented Feb 12, 2024 •
edited
Loading

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

The guidelines on how to test providers can be found in

Verify providers by contributors

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 8.18.0rc1

Provider apache.beam: 5.6.1rc1

Provider apache.drill: 2.6.1rc1

Provider apache.druid: 3.8.1rc1

Provider apache.hive: 7.0.0rc1

Provider apache.livy: 3.7.2rc1

Provider apprise: 1.2.2rc1

Provider celery: 3.6.0rc1

Provider cncf.kubernetes: 8.0.0rc1

Provider common.io: 1.3.0rc1

Provider common.sql: 1.11.0rc2

Provider databricks: 6.2.0rc1

Provider dbt.cloud: 3.6.1rc1

Provider elasticsearch: 5.3.3rc1

Provider google: 10.15.0rc1

Provider hashicorp: 3.6.3rc1

Provider http: 4.9.1rc1

Provider influxdb: 2.4.1rc1

Provider jdbc: 4.2.2rc1

Provider microsoft.mssql: 3.6.1rc1

Provider mongo: 4.0.0rc1

Provider mysql: 5.5.3rc1

Provider oracle: 3.9.2rc1

Provider pagerduty: 3.6.2rc1

Provider postgres: 5.10.1rc1

Provider qdrant: 1.0.0rc1

Provider sftp: 4.9.0rc1

Provider slack: 8.6.1rc1

Provider snowflake: 5.3.1rc1

Provider sqlite: 3.7.1rc1

Provider ssh: 3.10.1rc1

Provider tableau: 4.4.2rc1

Provider trino: 5.6.2rc1

Provider vertica: 3.7.1rc1

Provider weaviate: 1.3.2rc1

Provider yandex: 3.9.0rc1

All users involved in the PRs:
@ellisms @moiseenkov @synsh @Anush008 @infohash @shardbread @raphaelauv @hussein-awala @molcay @pankajastro @selimchergui @dabla @rawwar @SuperBo @ninsbl @V0lantis @kacpermuda @ahzaz @dirrao

Committer

  • I acknowledge that I am a maintainer/committer of the Apache Airflow project.
@eladkal eladkal added kind:meta High-level information important to the community testing status Status of testing releases labels Feb 12, 2024
@moiseenkov
Copy link
Contributor

#36594 works as expected

@moiseenkov
Copy link
Contributor

@potiuk , @eladkal , hi,
Could we please include #37282 into the release?

@eladkal
Copy link
Contributor Author

eladkal commented Feb 12, 2024

@potiuk , @eladkal , hi, Could we please include #37282 into the release?

If PR missed cut off time there is no option to add it. It will have to wait for next release / RC2 if we will have one.

airflow-oss-bot added a commit to astronomer/astronomer-providers that referenced this issue Feb 12, 2024
@airflow-oss-bot
Updating setup.cfg with RC provider packages on apache/airflow#37358
6f8f7f0
airflow-oss-bot added a commit to astronomer/astro-sdk that referenced this issue Feb 12, 2024
@airflow-oss-bot
Updating setup.cfg with RC provider packages on apache/airflow#37358
977c376
@pankajkoti
Copy link
Member

Observing an issue with common.sql: 1.11.0rc2

Posted a question on the PR

@potiuk
Copy link
Member

potiuk commented Feb 12, 2024

Observing an issue with common.sql: 1.11.0rc2

Posted a question on the PR

Nice catch @pankajkoti

@pankajkoti
Copy link
Member

Nice catch @pankajkoti

Actually, it was the (Astronomer) Providers RC automation test suite that reported this failure :)
Thank you @potiuk for providing the fix PR #37359 so quick. Appreciate it 🙏🏽

@infohash
Copy link
Contributor

@vchiapaikeo
Copy link
Contributor

vchiapaikeo commented Feb 12, 2024 •
edited
Loading

#37300 looks good. Also, @Lee-W , thank you for that fix to GKE providers.

However, I have deferrable tasks that should be completing that are not completing due to #37279. Will comment on the PR there. cc @pankajastro @pankajkoti

Link to comment: #37279 (comment)

@pankajastro
Copy link
Member

However, I have deferrable tasks that should be completing that are not completing due to #37279. Will comment on the PR there. cc @pankajastro @pankajkoti

Thanks for testing it. please share the DAG and other details if possible.

@ninsbl
Copy link
Contributor

ninsbl commented Feb 12, 2024

Hmm... I am struggeling to build a recent version of Airflow with celery 3.6.0rc1. The issue seems unrelated to the changes in the provider. I tried to build a container with 2.9.0 but there FAB provider seems to be missing / not installable...

@potiuk
Copy link
Member

potiuk commented Feb 12, 2024

Hmm... I am struggeling to build a recent version of Airflow with celery 3.6.0rc1. The issue seems unrelated to the changes in the provider. I tried to build a container with 2.9.0 but there FAB provider seems to be missing / not installable...

Yes. FAB provider and Airflow 2.9.0 are not yet usable to install. We might want to change that and release FAB provider in the next wave

@potiuk
Copy link
Member

potiuk commented Feb 12, 2024

Added #37362 to make the provider "ready" (we will need to do it when we come closer to 2.9.0 release.

@ellisms
Copy link
Contributor

ellisms commented Feb 12, 2024

#37000 is good.

@molcay
Copy link
Contributor

molcay commented Feb 12, 2024

#36996 and #37192 works as expected

@kacpermuda
Copy link
Contributor

I tested a random sample of changes from #36876 and it works OK. The PR simply changed the warnings.warn to decorator.

@amoghrajesh
Copy link
Contributor

Tried out #37214 using some sample DAGs in breeze environment. Looks good 👍🏽

@rawwar
Copy link
Collaborator

rawwar commented Feb 12, 2024 •
edited
Loading

Tried out #37211

amazon provider - 8.17.0:
image

amazon provider - 8.18.0rc1:
image

Info Log is no longer printed. hence, working as expected.

@V0lantis
Copy link
Contributor

V0lantis commented Feb 12, 2024 via email

@hussein-awala
Copy link
Member

Tested #37110 and #37323 and they both work as expected.

@synsh
Copy link
Contributor

synsh commented Feb 12, 2024

#35379 looks good.

@o-nikolas
Copy link
Contributor

While testing #37137 I encountered an unexpected behaviour with the ECS executor not retrying on failures. I don't think it was caused by this change, it's a separate regression I believe. However, this executor is still experimental so I'm not sure it should block this release.
CC @ferruzzi @syedahsn @shubham22

@potiuk
Copy link
Member

potiuk commented Feb 13, 2024

Checked all my changes. LGTM.

@Lee-W
Copy link
Member

Lee-W commented Feb 13, 2024

#36685 doesn't work as expected. It stucks somehow. Will take a deeper look these days.

#37306 and #37081 works fine without cncf.kubernetes: 8.0.0rc1 Looks like #37279 (comment) is the root cause

@Anush008
Copy link
Contributor

Provider qdrant: 1.0.0rc1
#36805

Tested DAGs. Works.

@eladkal
Copy link
Contributor Author

eladkal commented Feb 13, 2024

Providers amazon, cncf.kubernetes and common.sql will be excluded from RC1.

@SuperBo
Copy link
Contributor

SuperBo commented Feb 13, 2024

Databricks providers, confirm #37025 works when xcom is disabled. However it raises error with xcom enabled when serializing named tuple to json.

@Taragolis
Copy link
Contributor

However it raises error with xcom enabled when serializing named tuple to json.

Serializing named tuple should be add into the core first, e.g. it might be added by #37168

@Taragolis
Copy link
Contributor

Checked all my changes, looking good

@ahzaz
Copy link
Contributor

ahzaz commented Feb 13, 2024

#37014 works as expected.

@pankajkoti
Copy link
Member

#37117 works as expected

@utkarsharma2
Copy link
Contributor

utkarsharma2 commented Feb 14, 2024 via email

@raphaelauv
Copy link
Contributor

Fix hanging KPO on deferrable task with do_xcom_push (#37300): @vchiapaikeo

work 👍

@pankajastro
Copy link
Member

However, I have deferrable tasks that should be completing that are not completing due to #37279. Will comment on the PR there. cc @pankajastro @pankajkoti

Link to comment: #37279 (comment)

Hey @vchiapaikeo would it be possible for you to test #37363 with your use case?

@eladkal
Copy link
Contributor Author

eladkal commented Feb 14, 2024 •
edited
Loading

However, this executor is still experimental so I'm not sure it should block this release.

Since we are having RC2 for amazon provider there is time to fix it if you can raise a PR? @o-nikolas

@ninsbl
Copy link
Contributor

ninsbl commented Feb 15, 2024

#37066 Seems to work fine. The environment variable is acknowledged in the airflow config and long running tasks have been finnishing successfull with Celery 3.6.0rc1.

@Lee-W
Copy link
Member

Lee-W commented Feb 15, 2024

#36685 doesn't work as expected. It stucks somehow. Will take a deeper look these days.

#37306 and #37081 works fine without cncf.kubernetes: 8.0.0rc1 Looks like #37279 (comment) is the root cause

I tested #36685 a few more times today. The issue can no longer be reproduced.

@eladkal
Copy link
Contributor Author

eladkal commented Feb 15, 2024

Thank you everyone. Providers are released.

Providers: amazon, common.sql, cncf.kubernetes were excluded from this release.

I invite everyone to help improve providers for the next release, a list of open issues can be found here.

@eladkal eladkal closed this as completed Feb 15, 2024
@eladkal eladkal mentioned this issue Feb 17, 2024
Closed
33 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind:meta High-level information important to the community testing status Status of testing releases
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests