Status of testing Providers that were prepared on August 03, 2024

[eladkal]

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

The guidelines on how to test providers can be found in

Verify providers by contributors

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 8.27.0rc2

• Add RedriveExecution support to StepFunctionStartExecutionOperator (#40976): @gopidesupavan
• AIP-62: add hook lineage collection support for S3Hook (#40819): @mobuchowski
• Introduce Amazon Kinesis Analytics V2 (Managed Service for Apache Flink application) (#40765): @gopidesupavan
• Make EMR Container Trigger max attempts retries match the Operator (#41008): @o-nikolas
  Linked issues:
  • Linked Issue #40483: @akomisarek
• Fix RdsStopDbOperator operator in deferrable mode (#41059): @vincbeck
• Fix RedshiftCreateClusterOperator to always specify PubliclyAccessible (#40872): @vincbeck
• Fix AWS Redshift operators and sensors (#41191): @vincbeck
• Fix EmrServerlessStartJobOperator (#41103): @vincbeck
• Update example_redshift and example_redshift_s3_transfers to use RedshiftDataHook instead of RedshiftSQLHook (#40970): @vincbeck
• openlineage: migrate OpenLineage provider to V2 facets. (#39530): @JDarDagran
• [AIP-62] Translate AIP-60 URI to OpenLineage (#40173): @kacpermuda
  Linked issues:
  • Linked Issue #40335: @JDarDagran
  • Linked Issue #38767: @mobuchowski
• Move AWS Managed Service for Apache Flink sensor states to Hook (#40896): @gopidesupavan
• Replace usages of task context logger with the log table (#40867): @dstandish
• Deprecate SageMakerTrainingPrintLogTrigger (#41158): @vincbeck

Provider apache.beam: 5.7.2rc2

• Update default value for job_name in DataflowConfiguration (#40645): @e-galan
  Linked issues:
  • Linked Issue #40515: @turb

Provider apache.drill: 2.7.3rc1

• Clean up remaining getattr connection DbApiHook (#40665): @dabla

Provider apache.druid: 3.10.2rc1

• Clean up remaining getattr connection DbApiHook (#40665): @dabla

Provider apache.impala: 1.4.2rc1

• Clean up remaining getattr connection DbApiHook (#40665): @dabla

Provider apache.pinot: 4.4.2rc1

• Clean up remaining getattr connection DbApiHook (#40665): @dabla

Provider apprise: 1.3.2rc1

• Fix default behaviour for init function in AppriseNotifier (#41054): @romsharon98

Provider celery: 3.7.3rc1

• Increase broker's visibility timeout to 24hrs (#40879): @ephraimbuddy

Provider cncf.kubernetes: 8.3.4rc1

• Pass content of kube/config file to triggerer as a dictionary (#41178): @VladaZakharova
• Fix confusing log message in kubernetes executor (#41035): @RNHTTR
• Fix ApiException handling when adopting completed pods (#41109): @jedcunningham

Provider common.compat: 1.1.0rc1

• AIP-62: add method to common.compat to not force hooks to try/except every time (#40812): @mobuchowski
• openlineage: migrate OpenLineage provider to V2 facets. (#39530): @JDarDagran
• AIP-62: add hook lineage collection support for S3Hook (#40819): @mobuchowski

Provider common.io: 1.4.0rc1

• [AIP-62] Translate AIP-60 URI to OpenLineage (#40173): @kacpermuda
  Linked issues:
  • Linked Issue #40335: @JDarDagran
  • Linked Issue #38767: @mobuchowski
• AIP-62: add file dataset type support into common.io provider (#40817): @mobuchowski
• openlineage: migrate OpenLineage provider to V2 facets. (#39530): @JDarDagran
• AIP-62: add hook lineage collection support for S3Hook (#40819): @mobuchowski

Provider common.sql: 1.15.0rc1

• Create SQLAlchemy engine from connection in DB Hook and added autocommit param to insert_rows method (#40669): @dabla

Provider databricks: 6.8.0rc1

• Add DatabricksWorkflowPlugin (#40724): @pankajkoti
  Linked issues:
  • Linked Issue #40153: @pankajkoti
  • Linked Issue #40714: @jscheffl
  • Linked Issue #40708: @jscheffl
• DatabricksPlugin - Fix dag view redirect URL by using url_for redirect (#41040): @pankajkoti
• Fix named parameters templating in Databricks operators (#40864): @boraberke
  Linked issues:
  • Linked Issue #40788: @vatsrahul1001
  • Linked Issue #40471: @boraberke
• [Databricks Provider] Revert PRs #40864 and #40471 (#41050): @pankajkoti
  Linked issues:
  • Linked Issue #40864: @boraberke
  • Linked Issue #35433: @josh-fell
  • Linked Issue #40471: @boraberke

Provider docker: 3.12.3rc1

• DockerSwarmOperator: Support line breaks in service logs (#40705): @mr1holmes
  Linked issues:
  • Linked Issue #40571: @bladerail

Provider elasticsearch: 5.4.2rc1

• Clean up remaining getattr connection DbApiHook (#40665): @dabla

Provider exasol: 4.5.3rc1

• Clean up remaining getattr connection DbApiHook (#40665): @dabla

Provider ftp: 3.10.1rc1

• openlineage: migrate OpenLineage provider to V2 facets. (#39530): @JDarDagran

Provider google: 10.21.1rc2

• Dynamic import MySqlHook/PostgresHook in CloudSQLDatabaseHook (#41009): @jhongy1994
  Linked issues:
  • Linked Issue #40175: @bourliam
• Make google_analytics_admin system tests self sufficient (#40951): @moiseenkov
• Fix Custom Training Job operators to accept results without a model in def mode (#40685): @e-galan
  Linked issues:
  • Linked Issue #40476: @vignesh-sc
• Fix behavior for reattach_state parameter in BigQueryInsertJobOperator (#40664): @VladaZakharova
• Fix cloudsql-query system tests (#41092): @moiseenkov
• Refactor dataproc system tests (#40720): @VladaZakharova
• openlineage: migrate OpenLineage provider to V2 facets. (#39530): @JDarDagran
• Resolve google cloud sql deprecation in tests (#40834): @dirrao
• Update default value for job_name in DataflowConfiguration (#40645): @e-galan
  Linked issues:
  • Linked Issue #40515: @turb

Provider jdbc: 4.4.0rc1

• Create SQLAlchemy engine from connection in DB Hook and added autocommit param to insert_rows method (#40669): @dabla
• Clean up remaining getattr connection DbApiHook (#40665): @dabla

Provider microsoft.azure: 10.3.0rc1

• Added priority to Azure Container Instances (#40616): @liamalsopp
  Linked issues:
  • Linked Issue #40574: @liamalsopp
• Bump minimum version for azure containerinstance. (#40767): @potiuk
  Linked issues:
  • Linked Issue #40616: @liamalsopp

Provider microsoft.mssql: 3.8.0rc1

• Added support for replace option when using insert_rows with MsSqlHook (#40836): @dabla
• Clean up remaining getattr connection DbApiHook (#40665): @dabla

Provider mysql: 5.6.3rc1

• Clean up remaining getattr connection DbApiHook (#40665): @dabla

Provider odbc: 4.6.3rc1

• Clean up remaining getattr connection DbApiHook (#40665): @dabla

Provider openlineage: 1.10.0rc1

• openlineage: add AirflowRun on COMPLETE/FAIL events (#40996): @mobuchowski
• openlineage: extend custom_run_facets to on_failed and on_complete (#40953): @kacpermuda
  Linked issues:
  • Linked Issue #38982: @anandhimurali
• openlineage: migrate OpenLineage provider to V2 facets. (#39530): @JDarDagran
• openlineage: Add AirflowDagRunFacet to dag runEvents (#40854): @dolfinus
  Linked issues:
  • Linked Issue #40798: @dolfinus
• [AIP-62] Translate AIP-60 URI to OpenLineage (#40173): @kacpermuda
  Linked issues:
  • Linked Issue #40335: @JDarDagran
  • Linked Issue #38767: @mobuchowski
• Ability to add custom facet in OpenLineage events (#38982): @anandhimurali
• AIP-62: add method to common.compat to not force hooks to try/except every time (#40812): @mobuchowski
• openlineage: use airflow provided getters in conf (#40790): @kacpermuda
  Linked issues:
  • Linked Issue #40780: @mobuchowski
  • Linked Issue #40589: @mobuchowski
• openlineage: add config to include 'full' task info based on conf setting (#40589): @mobuchowski
• Add log_url to OpenLineage AirflowRunFacet (#40797): @dolfinus
• openlineage: add deferrable information to task info in airflowRunFacet (#40682): @kacpermuda
  Linked issues:
  • Linked Issue #38264: @kacpermuda
• openlineage: adjust DefaultExtractor's on_failure detection for airflow 2.10 fix (#41094): @kacpermuda
  Linked issues:
  • Linked Issue #41053: @kacpermuda
• openlineage: make value of slots in attrs.define consistent (#40992): @kacpermuda
  Linked issues:
  • Linked Issue #40972: @JDarDagran
• openlineage: Set slots to True for facets used in dag level (#40972): @JDarDagran
• openlineage: fix / add some task attributes in AirflowRunFacet (#40725): @kacpermuda
  Linked issues:
  • Linked Issue #40371: @mobuchowski
• openlineage: replace dt.now with airflow.utils.timezone.utcnow (#40887): @kacpermuda
• openlineage: remove deprecated parentRun facet key (#40681): @kacpermuda

Provider pgvector: 1.2.2rc1

• updating pgvector to !=0.3.0 to avoid recent issue in latest releases(0.3.0) (#40683): @vatsrahul1001

Provider postgres: 5.11.3rc1

• Clean up remaining getattr connection DbApiHook (#40665): @dabla

Provider qdrant: 1.1.2rc1

• Qdrant Provider: Bump Qdrant client to latest (#40831): @Anush008

Provider sftp: 4.10.3rc1

• openlineage: migrate OpenLineage provider to V2 facets. (#39530): @JDarDagran

Provider slack: 8.8.0rc1

• Add the ability to toggle the unfurl behavior for slack notifier (#40694): @armand-sauzay
• Add newly added unfurl args to the docstring of SlackNotifier (#40709): @armand-sauzay

Provider snowflake: 5.6.1rc1

• openlineage: migrate OpenLineage provider to V2 facets. (#39530): @JDarDagran
• Clean up remaining getattr connection DbApiHook (#40665): @dabla

Provider sqlite: 3.8.2rc1

• Clean up remaining getattr connection DbApiHook (#40665): @dabla

Provider ssh: 3.12.0rc1

• Add on kill to ssh (#40377): @MRLab12
  Linked issues:
  • Linked Issue #40343: @potiuk

Provider teradata: 2.5.0rc1

• Implemented Query Band Support for the Teradata provider (#40716): @sc250072
• Clean up remaining getattr connection DbApiHook (#40665): @dabla

Provider ydb: 1.2.0rc1

• ydb provider: add bulk upsert support (#40631): @uzhastik
• Clean up remaining getattr connection DbApiHook (#40665): @dabla

All users involved in the PRs:
@MRLab12 @uzhastik @dstandish @kacpermuda @jedcunningham @mr1holmes @gopidesupavan @moiseenkov @Anush008 @dirrao @o-nikolas @vatsrahul1001 @dabla @romsharon98 @e-galan @JDarDagran @ephraimbuddy

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[jhongy1994]

I checked #41009 (Linked issue #40175) and working as expected :)

[moiseenkov]

#40951, #41092 work as expected

[Anush008]

Qdrant Provider: Bump Qdrant client to latest (#40831): @Anush008

Tested and verified working.

[VladaZakharova]

#40664 and #40720 work as expected

[potiuk]

Checked my changes - both look good !

[pankajkoti]

Tested my changes in the Databricks provider, all look good!

[dolfinus]

#40854 and #40797 are fine

[sc250072]

#40716 Tested and working fine. All system tests are getting success.

[dabla]

Tested following PR with odbc/jdbc/mssql and looking good:

• Clean up remaining getattr connection DbApiHook (#40665)
• Create SQLAlchemy engine from connection in DB Hook and added autocommit param to insert_rows method (#40669)
• Added support for replace option when using insert_rows with MsSqlHook (#40836)

[kacpermuda]

I've tested all OpenLineage provider changes and my PRs with some example DAGs, all seems to work fine.

[gopidesupavan]

Have tested #40976 , #40765 and #40896. All looks good. thank you.

[JDarDagran]

Tested all my changes - all look fine.

[uzhastik]

ydb provider works as expected, checked #40631
thank you!

[mr1holmes]

Tested #40705
Works as expected.

[eladkal]

Thank you everyone. Providers are released.

I invite everyone to help improve providers for the next release, a list of open issues can be found here.