Status of testing Providers that were prepared on January 02, 2023

[eladkal]

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 7.0.0rc2

• Make pandas dependency optional for Amazon Provider (#28505): @potiuk
• AWS Secrets Manager Backend - major update (#27920): @dwreeves
• Add execution role parameter to AddStepsOperator (#28484): @syedahsn
• Add AWS SageMaker operator to register a model's version (#28024): @vandonr-amz
• Add link for EMR Steps Sensor logs (#28180): @syedahsn
• Add Amazon Elastic Container Registry (ECR) Hook (#28279): @Taragolis
• EMR Notebook operator (#28312): @syedahsn
• Create Lambda create operator and sensor (#28241): @vincbeck
• Better support for Boto Waiters (#28236): @ferruzzi
• Amazon Provider Package user agent (#27823): @ferruzzi
• Allow waiter to be configured via EmrServerless Operators (#27784): @syedahsn
• Add operators + sensor for aws sagemaker pipelines (#27786): @vandonr-amz
• Update RdsHook docstrings to match code (#28108): @hankehly
• add some important log in aws athena hook (#27917): @BobDu
• Lambda hook: make runtime and handler optional (#27778): @vincbeck
• Fix EmrAddStepsOperature wait_for_completion parameter is not working (#28052): @2h-kim
• Correctly template Glue Jobs create_job_kwargs arg (#28403): @IAL32
• Fix template rendered bucket_key in S3KeySensor (#28340): @syun64
• Fix Type Error while using DynamoDBToS3Operator (#28158): @snjypl
• AWSGlueJobHook updates job configuration if it exists (#27893): @romibuzi
• Fix GlueCrawlerOperature failure when using tags (#28005): @IAL32
• Fix S3KeySensor documentation (#28297): @IAL32
• Improve docstring for AwsLambdaInvokeFunctionOperator (#28233): @eladkal
• Remove outdated compat imports/code from providers (#28507): @Taragolis
• Amazon Provider Package release 7.0.0 - Add description of breaking changes (#28582): @dwreeves
• [misc] Get rid of pass statement in conditions (#27775): @Taragolis
• [misc] Replace XOR ^ conditions by exactly_one helper in providers (#27858): @Taragolis

Provider apache.hive: 5.1.0rc2

• Move Hive macros to the provider (#28538): @potiuk
• Make pandas dependency optional for Amazon Provider (#28505): @potiuk

Provider atlassian.jira: 2.0.0rc2

• Changing atlassian JIRA SDK to official atlassian-python-api SDK (#27633): @Adityamalik123

Provider cncf.kubernetes: 5.1.0rc2

• [Fixed] "Adding Flink on K8s Operator" (#28512): @xinbinhuang
• Add volume-related nested template fields for KPO (#27719): @raphaelauv
• Allow longer pod names for k8s executor / KPO (#27736): @dstandish
• Use labels instead of pod name for pod log read in k8s exec (#28546): @dstandish
• Patch "checked" when pod not successful (#27845): @dstandish
• Keep pod name for k8s executor under 63 characters (#28237): @dstandish
• Remove outdated compat imports/code from providers (#28507): @Taragolis
• Restructure Docs (#27235): @TohnJhomas

Provider common.sql: 1.3.2rc2

• Fix isort problems introduced by recent isort release (#28434): @potiuk
• Fix template rendering for Common SQL operators (#28202): @jon-evergreen
• Defer to hook setting for split_statements in SQLExecuteQueryOperator (#28635): @dstandish
• Clarify docstrings for updated DbApiHook (#27966): @dstandish

Provider docker: 3.4.0rc2

• add hostname argument to DockerOperator (#27822): @skabbit
• Move min airflow version down for Docker Provider to 2.3.0 (#28648): @potiuk

Provider elasticsearch: 4.3.2rc2

• Support restricted index patterns in Elasticsearch log handler (#23888): @kouk

Provider ftp: 3.3.0rc2

• Add FTPSFileTransmitOperator (#28318): @RachitSharma2001
• Add FTPFileTransmitOperator (#26974): @RachitSharma2001

Provider google: 8.7.0rc2

• Add table_resource to template fields for BigQueryCreateEmptyTableOperator (#28235): @jbandoro
• Add retry param in GCSObjectExistenceSensor (#27943): @pankajastro
• Add preserveAsciiControlCharacters to src_fmt_configs (#27679): @thangnd197
• Add deferrable mode to CloudBuildCreateBuildOperator (#27783): @VladaZakharova
• GCSToBigQueryOperator allows autodetect None and infers schema (#28564): @vchiapaikeo
• Improve memory usage in Dataproc deferrable operators (#28117): @lwyszomi
• Push job_id in xcom for dataproc submit job op (#28639): @pankajastro
• Fix for issue with reading schema fields for JSON files in GCSToBigQueryOperator (#28284): @VladaZakharova
• Fix GCSToBigQueryOperator not respecting schema_obj (#28444): @vchiapaikeo
• Fix GCSToGCSOperator copying list of objects without wildcard (#28111): @moiseenkov
• Fix: re-enable use of parameters in gcs_to_bq which had been disabled (#27961): @mdering
• Set bigquery use_legacy_sql param in job config correctly (#28522): @pankajastro
• Remove pylint messages control instructions (#28555): @Taragolis
• Remove deprecated AIPlatformConsoleLink from google/provider.yaml (#28449): @vchiapaikeo
• Use object instead of array in config.yml for config template (#28417): @dstandish
• [misc] Get rid of pass statement in conditions (#27775): @Taragolis
• Change log level to DEBUG when secret not found for google secret manager (#27856): @lwyszomi
• [misc] Replace XOR ^ conditions by exactly_one helper in providers (#27858): @Taragolis

Provider imap: 3.1.1rc2

• [misc] Get rid of pass statement in conditions (#27775): @Taragolis

Provider microsoft.azure: 5.0.2rc2

• Re-enable azure service bus on ARM as it now builds cleanly (#28442): @potiuk

Provider microsoft.mssql: 3.3.2rc2

• Re-enable pymsql on ARM as it now builds cleanly (#28530): @Taragolis

Provider microsoft.winrm: 3.1.1rc2

• Remove outdated compat imports/code from providers (#28507): @Taragolis

Provider oracle: 3.6.0rc2

• XCOM push ORA error code in OracleStoredProcedure (#27319): @kolfild26

Provider postgres: 5.4.0rc2

• Bring back psycopg2-binary as dependency instead of psycopg (#28316): @potiuk

Provider presto: 4.2.1rc2

• Remove outdated compat imports/code from providers (#28507): @Taragolis

Provider salesforce: 5.3.0rc2

• Add test_connection to SalesforceHook (#27921): @odaneau-astro

Provider sftp: 4.2.1rc2

• Update codespell and fix typos (#28568): @KarshVashi
• [misc] Get rid of pass statement in conditions (#27775): @Taragolis

Provider slack: 7.1.1rc2

• [misc] Replace XOR ^ conditions by exactly_one helper in providers (#27858): @Taragolis

Provider ssh: 3.4.0rc2

• Add .bash extension to SSHOperator (#28617): @stamixthereal
• Add test_connection method for SSHHook (#28184): @xjf729
• SSH task exit code added to XCOM as 'ssh_exit' key (#27370): @kolfild26
• Remove outdated compat imports/code from providers (#28507): @Taragolis
• [misc] Get rid of pass statement in conditions (#27775): @Taragolis

Provider telegram: 3.1.1rc2

• Limit Telegram to < 20.0.0 (#28671): @potiuk

Provider trino: 4.3.1rc2

• Remove outdated compat imports/code from providers (#28507): @Taragolis

The guidelines on how to test providers can be found in

Verify providers by contributors

All users involved in the PRs:
@Taragolis @syedahsn @xinbinhuang @vchiapaikeo @ferruzzi @jon-evergreen @RachitSharma2001 @syun64 @snjypl @pankajastro @thangnd197 @dwreeves @eladkal @KarshVashi @BobDu @hankehly @xjf729 @Adityamalik123
@jbandoro @IAL32 @lwyszomi @moiseenkov @VladaZakharova @stamixthereal @vincbeck @2h-kim @kolfild26 @potiuk @raphaelauv @TohnJhomas @dstandish @vandonr-amz @romibuzi @mdering @skabbit @odaneau-astro
@kouk

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[vandonr-amz]

✅ for me, as our CI ran the example DAGs sucessfully, which include the newly introduced operators/sensors.

[vchiapaikeo]

Reran our test dags and they both looked good:

[hankehly]

#28108 is a document-only change (docs build successfully)

[stamixthereal]

#28617 also looks good on my local, anyway it is a small change

[potiuk]

All Good:

• The Fix isort problems introduced by recent isort release #28434 was just dev change

• I checked all dependency changes in METADATA of wheel packages (Make pandas dependency optional for Amazon Provider #28505, Move min airflow version down for Docker Provider to 2.3.0 #28648, Re-enable azure service bus on ARM as it now builds cleanly #28442, Bring back psycopg2-binary as dependency instead of psycopg #28316 , Limit Telegram to < 20.0.0 #28671)

• Hive provider when installed provides the macros from the provider instead of the ones from Airflow (Move Hive macros to the provider #28538):

I had no hive and connection defined, but it was correctly added and used by JINJA and was clearly taken from hive provider, not airflow core:

  File "/opt/airflow/airflow/providers/apache/hive/macros/hive.py", line 49, in max_partition
    hive_hook = HiveMetastoreHook(metastore_conn_id=metastore_conn_id)
  File "/opt/airflow/airflow/providers/apache/hive/hooks/hive.py", line 485, in __init__
    self.conn = self.get_connection(metastore_conn_id)
  File "/opt/airflow/airflow/hooks/base.py", line 72, in get_connection
    conn = Connection.get_connection_from_secrets(conn_id)
  File "/opt/airflow/airflow/models/connection.py", line 435, in get_connection_from_secrets
    raise AirflowNotFoundException(f"The conn_id `{conn_id}` isn't defined")
airflow.exceptions.AirflowNotFoundException: The conn_id `metastore_default` isn't defined

Also it is visible in Airflow Webserver as Plugin when the new hive provider is installed (missing when installing hive provider 5.0.0):

[Taragolis]

#28555: This only affect CI and it pass

#28507, #27775, #27858: Working fine

#28279: Check on AWS environment, work fine

#28505 - @potiuk I've found that we try to import numpy before pandas and in case of pandas not installed we more more likely get ModuleNotFoundError raised instead of AirflowOptionalProviderFeatureException. numpy is a core dependency for pandas and it is not a dependency of airflow itself. I've check on slim image of airflow 2.3.4, 2.4.3 and 2.5.0. It is not critical - no raise any error during airflow services starts

airflow/airflow/providers/amazon/aws/transfers/sql_to_s3.py

Lines 25 to 32 in 8cb69bb

	import numpy as np
	try:
	import pandas as pd
	except ImportError as e:
	from airflow.exceptions import AirflowOptionalProviderFeatureException
	raise AirflowOptionalProviderFeatureException(e)

  File "/opt/airflow/dags/example_aws.py", line 5, in <module>
    from airflow.providers.amazon.aws.transfers.sql_to_s3 import SqlToS3Operator
  File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/amazon/aws/transfers/sql_to_s3.py", line 25, in <module>
    import numpy as np
ModuleNotFoundError: No module named 'numpy'

[eladkal]

I've found that we try to import numpy before pandas and in case of pandas not installed we more more likely get ModuleNotFoundError raised instead of AirflowOptionalProviderFeatureException

I don't think this is a concern...
This is a dependency of a dependency.
If you use library directly you should make sure you install it and not relay on another package to do that for you.

[potiuk]

I don't think this is a concern...

Agree. It's just "less clear" exception in this case, not a blocker for sure. But I will add a fix for the next version.

[Taragolis]

Agree. It's just "less clear" exception in this case, not a blocker for sure. But I will add a fix for the next version.

Yeah agree can do it later. This kind of import only critical if it optional part of Hooks, not a operators. We could just move it inside try.. except and suggest to install by extra pip install apache-airflow-providers-amazon[pandas]

[romibuzi]

#27893 all good with a integration test:

[pankajastro]

Tested #27943 #28639 #28522 Looks good

[kolfild26]

Both #27319 and #27370 seem to be 🆗

[jbandoro]

Tested #28235 and it looks good, thanks!

[eladkal]

Thank you everyone.
Providers are released thus closing the issue.
I invite everyone to help improve providers for the next release, a list of open bugs can be found here.