Status of testing Providers that were prepared on September 28, 2022 #26752

potiuk · 2022-09-28T14:52:13Z

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider alibaba: 2.1.0rc1

Auto tail file logs in Web UI (#26169): @jens-scheffler-bosch

Provider amazon: 6.0.0rc1

Provider apache.drill: 2.2.1rc1

Add common-sql lower bound for common-sql (#25789): @potiuk

Provider apache.druid: 3.2.1rc1

Add common-sql lower bound for common-sql (#25789): @potiuk

Provider apache.hive: 4.0.1rc1

Add common-sql lower bound for common-sql (#25789): @potiuk

Provider apache.pinot: 3.2.1rc1

Provider databricks: 3.3.0rc1

Provider dbt.cloud: 2.2.0rc1

Add DbtCloudListJobsOperator (#26475): @phanikumv

Provider docker: 3.2.0rc1

Provider elasticsearch: 4.2.1rc1

Add common-sql lower bound for common-sql (#25789): @potiuk

Provider exasol: 4.0.1rc1

Add common-sql lower bound for common-sql (#25789): @potiuk

Provider google: 8.4.0rc1

Provider jdbc: 3.2.1rc1

Add common-sql lower bound for common-sql (#25789): @potiuk

Provider jira: 3.1.0rc1

Deprecate jira provider in favor of atlassian.jira provider (#25930): @eladkal

Provider microsoft.azure: 4.3.0rc1

Provider microsoft.mssql: 3.2.1rc1

Add common-sql lower bound for common-sql (#25789): @potiuk

Provider mysql: 3.2.1rc1

Add common-sql lower bound for common-sql (#25789): @potiuk

Provider odbc: 3.1.2rc1

Add common-sql lower bound for common-sql (#25789): @potiuk

Provider oracle: 3.4.0rc1

Provider postgres: 5.2.2rc1

Provider qubole: 3.2.1rc1

Provider sftp: 4.1.0rc1

SFTPOperator - add support for list of file paths (#26666): @pauldalewilliams

Provider slack: 6.0.0rc1

Provider snowflake: 3.3.0rc1

Provider ssh: 3.2.0rc1

SSHHook: Saving new host keys as described in docs (#25979): @doiken

Provider trino: 4.1.0rc1

trino: Support CertificateAuthentication in the trino hook (#26246): @c2zwdjnlcg

Provider vertica: 3.2.1rc1

Add common-sql lower bound for common-sql (#25789): @potiuk

The guidelines on how to test providers can be found in

Verify providers by contributors

Committer

I acknowledge that I am a maintainer/committer of the Apache Airflow project.

The text was updated successfully, but these errors were encountered:

eladkal · 2022-09-28T15:17:47Z

links are wrong? all are RC1 not RC2

potiuk · 2022-09-28T15:47:36Z

Thanks Elad :). Fixing :)

potiuk · 2022-09-28T15:49:02Z

Copy-pasted a wrong command :). All links are fixed now.

patricker · 2022-09-28T18:54:08Z

I've tested #26277, it's good.

I was testing #26285, and there is something off with the way I wrote the value to XCom. The Max Value shows correctly in the text logs, but when it's written to XCom there is some kind of serialization issue, and it throws an error in the web ui when I try to view it.

  File "/usr/local/lib/python3.9/site-packages/airflow/models/xcom.py", line 600, in deserialize_value
    return pickle.loads(result.value)
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/table.py", line 1409, in __getattr__
    value = self._xxx_field_to_index.get(name)
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/table.py", line 1409, in __getattr__
    value = self._xxx_field_to_index.get(name)
  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/table.py", line 1409, in __getattr__
    value = self._xxx_field_to_index.get(name)
  [Previous line repeated 957 more times]
RecursionError: maximum recursion depth exceeded

c2zwdjnlcg · 2022-09-28T18:55:37Z

Tested trino: 4.1.0rc1 and did not hit any issues

potiuk · 2022-09-28T20:11:15Z

I was testing #26285, and there is something off with the way I wrote the value to XCom. The Max Value shows correctly in the text logs, but when it's written to XCom there is some kind of serialization issue, and it throws an error in the web ui when I try to view it.

Hmm. I think we need to take closer look. Can you describe a way you did it? It kinda looks like you have a "table" object serialized somehow rather than primitive.

jscheffl · 2022-09-28T21:42:34Z

I'd like to contribute to the RC tests but for all 4 provider packages touched by my PR I just did code checks for consistency of the logging on API. I have no back-end environment for alibaba, azure, google and amazon in my availability :-( Is somebody besides me able to test?
Test is simple if you have a running environment, can use any existing longer running DAG and check the log view ,as described in PR #26169

patricker · 2022-09-28T22:22:00Z

I was testing #26285, and there is something off with the way I wrote the value to XCom. The Max Value shows correctly in the text logs, but when it's written to XCom there is some kind of serialization issue, and it throws an error in the web ui when I try to view it.

Hmm. I think we need to take closer look. Can you describe a way you did it? It kinda looks like you have a "table" object serialized somehow rather than primitive.

@potiuk In looking at the code again, and the actual Log message, it looks like I accidentally tried to serialize a whole table row instead of the value of the first column of the table row.

Loaded BQ data with max data-landing.api.users_20220928_.updated_at=Row((datetime.datetime(2022, 9, 28, 16, 57, 18, 752695, tzinfo=datetime.timezone.utc),), {'f0_': 0})

This serialized row then causes issues in XCom. I'll see about fixing it.

potiuk · 2022-09-28T22:36:09Z

This serialized row then causes issues in XCom. I'll see about fixing it.

Thanks for flagging/testing/fixing. We can remove google provider and re-release it as rc2 (but let's continue testing it for now by others).

potiuk · 2022-09-28T22:37:43Z

I'd like to contribute to the RC tests but for all 4 provider packages touched by my PR I just did code checks for consistency of the logging on API. I have no back-end environment for alibaba, azure, google and amazon in my availability :-( Is somebody besides me able to test? Test is simple if you have a running environment, can use any existing longer running DAG and check the log view ,as described in PR #26169

Yep. If others can test it - it would be great (especially while testing other issues). Thanks for notifying :)

pauldalewilliams · 2022-09-29T02:14:11Z

I tested both #26666 and #26576 - both are good.

patricker · 2022-09-29T03:06:14Z

@potiuk I'll try and get my other two tested tomorrow. Here is the PR with the fix for #26285 :
#26768.

bharanidharan14 · 2022-09-29T11:38:31Z

Tested both Azure Service Bus (Create and Delete) Topic Operators #25436, working fine 👍

nshetty15 · 2022-09-29T16:15:11Z

#26653 - (apache-airflow-providers-docker 3.2.0rc1) Looks good. Thanks!

patricker · 2022-09-29T19:34:10Z

I tested #26096 and #26190. Both tested fine.
I ran into an issue while testing #26096, but which isn't a "bug" with the original PR's implementation. Just a new bug in a similar part of the code. I cut a PR for it, but doesn't have to go into RC2 if it doesn't make it.
#26786

Taragolis · 2022-09-29T21:21:50Z

#26442 - Only could proof that changed code persists. It is hardly possible to make loggers fail on create hook
#26628 - Check that amazon and databricks do not change anything in airflow.providers_manager.ProvidersManager after changes. (related to removed listed hooks)
#25980 - As expected deprecated s3 connection can be use in aws hooks with warnings. No connection in the UI anymore.
#26464 - Working as expected.

Taragolis · 2022-09-29T23:19:55Z

#26169 - @jens-scheffler-bosch @potiuk It work with S3TaskHandler in breeze with some small artifacts: during fetching logs continuously printed *** Falling back to local log

[2022-09-29, 23:02:50 UTC] {subprocess.py:93} INFO - 70
[2022-09-29, 23:02:51 UTC] {subprocess.py:93} INFO - 71
[2022-09-29, 23:02:52 UTC] {subprocess.py:93} INFO - 72
*** Falling back to local log
[2022-09-29, 23:02:53 UTC] {subprocess.py:93} INFO - 73
[2022-09-29, 23:02:54 UTC] {subprocess.py:93} INFO - 74
*** Falling back to local log
[2022-09-29, 23:02:55 UTC] {subprocess.py:93} INFO - 75
[2022-09-29, 23:02:56 UTC] {subprocess.py:93} INFO - 76
*** Falling back to local log
[2022-09-29, 23:02:57 UTC] {subprocess.py:93} INFO - 77
[2022-09-29, 23:02:58 UTC] {subprocess.py:93} INFO - 78
*** Falling back to local log
[2022-09-29, 23:02:59 UTC] {subprocess.py:93} INFO - 79
[2022-09-29, 23:03:00 UTC] {subprocess.py:93} INFO - 80

If something wrong with connection than also spam with this message

airflow/airflow/providers/amazon/aws/log/s3_task_handler.py

Lines 112 to 113 in ce07117

	log = f'*** Failed to verify remote log exists {remote_loc}.\n{error}\n'

alexkruc · 2022-09-30T09:26:38Z

#25675 - verified

Taragolis · 2022-09-30T14:09:02Z

#25852: Try to test in Airflow 2.2.5 and got an issues that it not work correctly with extra in the UI:

Do not allow create connection with empty value for timeout
If field not specified than write empty value to the DB, as result slack_sdk can't build correct URL

Need to check is it affect 2.3, because in 2.4 no problem. No problem with connection if it created via API or provided as URL (env var, secrets backend)

airflow/airflow/providers/slack/hooks/slack.py

Lines 297 to 314 in 4b456d5

    
           return { 
        
               prefixed_extra_field("timeout", cls.conn_type): IntegerField( 
        
                   lazy_gettext("Timeout"), 
        
                   widget=BS3TextFieldWidget(), 
        
                   description="Optional. The maximum number of seconds the client will wait to connect " 
        
                   "and receive a response from Slack API.", 
        
               ), 
        
               prefixed_extra_field("base_url", cls.conn_type): StringField( 
        
                   lazy_gettext('Base URL'), 
        
                   widget=BS3TextFieldWidget(), 
        
                   description="Optional. A string representing the Slack API base URL.", 
        
               ), 
        
               prefixed_extra_field("proxy", cls.conn_type): StringField( 
        
                   lazy_gettext('Proxy'), 
        
                   widget=BS3TextFieldWidget(), 
        
                   description="Optional. Proxy to make the Slack API call.", 
        
               ), 
        
           }

#26648: Same as previous
#26118: Working fine
#26459: Unsafe imports not presented. Fine.

josh-fell · 2022-09-30T19:04:38Z

#25604 looks good!

potiuk · 2022-10-01T13:38:11Z

#25852: Try to test in Airflow 2.2.5 and got an issues that it not work correctly with extra in the UI:

Do not allow create connection with empty value for timeout

If field not specified than write empty value to the DB, as result slack_sdk can't build correct URL

Need to check is it affect 2.3, because in 2.4 no problem. No problem with connection if it created via API or provided as URL (env var, secrets backend)

Ok. I will wait for the check to relase an RC2 and will remove Slack alongside Google from the release and we will have RC2 together.

potiuk · 2022-10-01T13:44:44Z

If something wrong with connection than also spam with this message

@Taragolis I think it's a small thing, they indicate that something is wrong with configuration. We might want to configure it in Breeze to make it less annoying.

potiuk · 2022-10-01T13:49:45Z

I ran into an issue while testing #26096, but which isn't a "bug" with the original PR's implementation. Just a new bug in a similar part of the code. I cut a PR for it, but doesn't have to go into RC2 if it doesn't make it.
#26786

We are going to have RC2 of Google and I just merged #26786 so it will make it :)

potiuk · 2022-10-01T13:53:28Z

Thanks all who tested it ! As usual we have great help from our community.

And thanks a lot for all the "critical" checks and doubts. Keep having them. It helps to make the right decisions for release!

I am releasing all but google and slack. I am at ApacheCon now. And I will look for @Taragolis comments on the connection for Slack to release RC2s

Taragolis · 2022-10-01T19:43:30Z

@Taragolis I think it's a small thing, they indicate that something is wrong with configuration. We might want to configure it in Breeze to make it less annoying.

I also think it a small things. I use breeze for test because part related to actual print a log during task execution in a core and seem like it not added in any current releases yet (milestones is Airflow 2.5.0)

The issue itself how work S3TaskHandler (as well as other Object storage):

During task execution logs received from worker and after task execution write to S3
If logs received from logger it print message that *** Falling back to local log
If also user incorrectly configure AWS credentials it also write *** Failed to verify remote log exists

So.. Auto tail file logs just continuously execute _read which write this message (only in the UI, this information not presented into the actual logs).

Taragolis · 2022-10-01T20:26:24Z

I am releasing all but google and slack. I am at ApacheCon now. And I will look for @Taragolis comments on the connection for Slack to release RC2s

@potiuk Let me check is might only related to specific version of Airflow, like only for 2.2.5
I have an idea about workaround if I sure that it related for all 2.2.x

Also as an option this version of provider might be released after October 11 with min airflow version 2.3.
In this case we do not need to create workaround which we remove in next version of provider

potiuk · 2022-10-01T22:12:45Z

It might still be cool to fix it as the 'last' version working for 2.2.5 :)

hankehly · 2022-10-02T04:03:33Z

Manually tested the following on 2022/10/02, both OK
#26003
#26005

Taragolis · 2022-10-03T09:00:26Z

It might still be cool to fix it as the 'last' version working for 2.2.5 :)

I try to check it again today to determine with which version it actually has a problem (personally I have a problem with create connection in all providers which define additional fields in get_connection_form_widgets on different 2.2.5 environments and do not have in 2.3.4)

And create a workaround which allow to create Slack Connection in the UI

potiuk added the kind:meta High-level information important to the community label Sep 28, 2022

potiuk mentioned this issue Sep 28, 2022

Executing a Docker Decorator task results in Jinja Error #26718

Closed

2 tasks

potiuk added the testing status Status of testing releases label Oct 1, 2022

potiuk closed this as completed Oct 1, 2022

Taragolis mentioned this issue Oct 3, 2022

Fix Slack Connections created in the UI #26845

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Status of testing Providers that were prepared on September 28, 2022 #26752

Status of testing Providers that were prepared on September 28, 2022 #26752

potiuk commented Sep 28, 2022 •

edited

Loading

eladkal commented Sep 28, 2022

potiuk commented Sep 28, 2022

potiuk commented Sep 28, 2022

patricker commented Sep 28, 2022

c2zwdjnlcg commented Sep 28, 2022

potiuk commented Sep 28, 2022

jscheffl commented Sep 28, 2022

patricker commented Sep 28, 2022

potiuk commented Sep 28, 2022

potiuk commented Sep 28, 2022 •

edited

Loading

pauldalewilliams commented Sep 29, 2022

patricker commented Sep 29, 2022 •

edited

Loading

bharanidharan14 commented Sep 29, 2022 •

edited

Loading

nshetty15 commented Sep 29, 2022

patricker commented Sep 29, 2022 •

edited

Loading

Taragolis commented Sep 29, 2022 •

edited

Loading

Taragolis commented Sep 29, 2022

alexkruc commented Sep 30, 2022

Taragolis commented Sep 30, 2022

josh-fell commented Sep 30, 2022

potiuk commented Oct 1, 2022 •

edited

Loading

potiuk commented Oct 1, 2022

potiuk commented Oct 1, 2022

potiuk commented Oct 1, 2022

Taragolis commented Oct 1, 2022

Taragolis commented Oct 1, 2022

potiuk commented Oct 1, 2022

hankehly commented Oct 2, 2022

Taragolis commented Oct 3, 2022

Status of testing Providers that were prepared on September 28, 2022 #26752

Status of testing Providers that were prepared on September 28, 2022 #26752

Comments

potiuk commented Sep 28, 2022 • edited Loading

Body

Provider alibaba: 2.1.0rc1

Provider amazon: 6.0.0rc1

Provider apache.drill: 2.2.1rc1

Provider apache.druid: 3.2.1rc1

Provider apache.hive: 4.0.1rc1

Provider apache.pinot: 3.2.1rc1

Provider databricks: 3.3.0rc1

Provider dbt.cloud: 2.2.0rc1

Provider docker: 3.2.0rc1

Provider elasticsearch: 4.2.1rc1

Provider exasol: 4.0.1rc1

Provider google: 8.4.0rc1

Provider jdbc: 3.2.1rc1

Provider jira: 3.1.0rc1

Provider microsoft.azure: 4.3.0rc1

Provider microsoft.mssql: 3.2.1rc1

Provider mysql: 3.2.1rc1

Provider odbc: 3.1.2rc1

Provider oracle: 3.4.0rc1

Provider postgres: 5.2.2rc1

Provider qubole: 3.2.1rc1

Provider sftp: 4.1.0rc1

Provider slack: 6.0.0rc1

Provider snowflake: 3.3.0rc1

Provider ssh: 3.2.0rc1

Provider trino: 4.1.0rc1

Provider vertica: 3.2.1rc1

Committer

eladkal commented Sep 28, 2022

potiuk commented Sep 28, 2022

potiuk commented Sep 28, 2022

patricker commented Sep 28, 2022

c2zwdjnlcg commented Sep 28, 2022

potiuk commented Sep 28, 2022

jscheffl commented Sep 28, 2022

patricker commented Sep 28, 2022

potiuk commented Sep 28, 2022

potiuk commented Sep 28, 2022 • edited Loading

pauldalewilliams commented Sep 29, 2022

patricker commented Sep 29, 2022 • edited Loading

bharanidharan14 commented Sep 29, 2022 • edited Loading

nshetty15 commented Sep 29, 2022

patricker commented Sep 29, 2022 • edited Loading

Taragolis commented Sep 29, 2022 • edited Loading

Taragolis commented Sep 29, 2022

alexkruc commented Sep 30, 2022

Taragolis commented Sep 30, 2022

josh-fell commented Sep 30, 2022

potiuk commented Oct 1, 2022 • edited Loading

potiuk commented Oct 1, 2022

potiuk commented Oct 1, 2022

potiuk commented Oct 1, 2022

Taragolis commented Oct 1, 2022

Taragolis commented Oct 1, 2022

potiuk commented Oct 1, 2022

hankehly commented Oct 2, 2022

Taragolis commented Oct 3, 2022

potiuk commented Sep 28, 2022 •

edited

Loading

potiuk commented Sep 28, 2022 •

edited

Loading

patricker commented Sep 29, 2022 •

edited

Loading

bharanidharan14 commented Sep 29, 2022 •

edited

Loading

patricker commented Sep 29, 2022 •

edited

Loading

Taragolis commented Sep 29, 2022 •

edited

Loading

potiuk commented Oct 1, 2022 •

edited

Loading