Status of testing Providers that were prepared on March 09, 2025

[eladkal]

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

The guidelines on how to test providers can be found in

Verify providers by contributors

Let us know in the comments, whether the issue is addressed.

These are providers that require testing as there were some substantial changes introduced:

Provider airbyte: 5.0.1rc1

• Airbyte Provider: api_version on on_kill method in AirbyteTriggerSyncOperator (#46833): @dantonbertuol

Provider amazon: 9.5.0rc1

• Add a backup implementation in AWS MwaaHook for calling the MWAA API (#47035): @ramitkataria
• Add AWS SageMaker Unified Studio Workflow Operator (#45726): @agupta01
• Add error statuses check in RdsExportTaskExistenceSensor (#46917): @vargacypher
• Draft Common Message Queue (#46694): @vikramkoka
• add startTime to paginator.paginate when fetching logs in GlueJobHook (#46950): @rawwar
• Add MwaaDagRunSensor to Amazon Provider Package (#46945): @ramitkataria
• Fix aws trigger tests, use get_async_conn for mock object (#47515): @gopidesupavan
• fix: don't use blocking property access for async purposes (#47326): @m1racoli
• Fix and simplify get_permitted_dag_ids in auth manager (#47458): @vincbeck
• Log state for EMR Containers sensor on failure (#47125): @o-nikolas
• Add some typing and require kwargs for auth manager (#47455): @jedcunningham
• Base AWS classes - S3 (#47321): @o-nikolas
  Linked issues:
  • Linked Issue #35278: @Taragolis
• Remove unused methods from auth managers (#47316): @vincbeck
• Move api-server to port 8080 (#47310): @jedcunningham

Provider apache.beam: 6.0.3rc1

• Fix beam pipeline options False value parsing (#47419): @olegkachur-e

Provider apache.druid: 4.1.0rc1

• Add method to retrieve Druid task status URL based on ingestion type (#47238): @kanagaraj-dhanapal-89

Provider apache.pinot: 4.7.0rc1

• Feat: Add username, password to Pinot Connection (#46826): @p-eye

Provider apache.spark: 5.0.1rc1

• spark on kubernetes removes dependency on Spark Exit code (#46817): @bin-lian

Provider celery: 3.10.3rc1

• Scheduler should not crash when executor_config is provided with CE (#47375): @amoghrajesh
  Linked issues:
  • Linked Issue #47349: @atul-astronomer
• Ensure Celery provider still works on Airflow v2 (#47152): @ashb
  Linked issues:
  • Linked Issue #47139: @MathieuB1

Provider common.io: 1.5.1rc1

• Add local scheme as alternative to file for using the ObjectStoragePath (#46670): @dabla

Provider common.sql: 1.24.0rc1

• Add deferred pagination mode to GenericTransfer (#44809): @dabla
• Fix deprecation warnings in common.sql (#47169): @potiuk
  Linked issues:
  • Linked Issue #43747: @dabla

Provider databricks: 7.2.1rc1

• Fix deprecation warnings in common.sql (#47169): @potiuk
  Linked issues:
  • Linked Issue #43747: @dabla

Provider exasol: 4.7.3rc1

• Fix deprecation warnings in common.sql (#47169): @potiuk
  Linked issues:
  • Linked Issue #43747: @dabla

Provider hashicorp: 4.1.0rc1

• Add custom AWS Region support for STS endpoint to hashicorp provider (#47495): @sdaberdaku
  Linked issues:
  • Linked Issue #47470: @sdaberdaku

Provider http: 5.2.1rc1

• Fix HttpSensor to enable returning xcom value (#46848): @fredthomsen

Provider microsoft.azure: 12.2.1rc1

• Fix support for old method signature of pagination_function in MSGraphAsyncOperator (#47101): @dabla

Provider microsoft.mssql: 4.2.1rc1

• Fix MSSQL provider dependency on common-sql (#47188): @AlteredOracle
  Linked issues:
  • Linked Issue #47185: @JCisneros77

Provider mongo: 5.0.2rc1

• Proper limit for pymongo and better comment explaining root cause (#47338): @potiuk
• Restrict pymongo to less than 4.11 for our CI (#47336): @amoghrajesh

Provider mysql: 6.2.0rc1

• Add regex validation for table names in 'bulk_load' (#47254): @amoghrajesh
• Add regex validation for table names in 'bulk_dump' (#47255): @amoghrajesh
• Adding ticks to user provided table names (#47262): @amoghrajesh

Provider openlineage: 2.1.1rc1

• fix: OpenLineage serialization of dataset timetables for Airflow 2.9 (#47150): @kacpermuda
  Linked issues:
  • Linked Issue #43434: @mobuchowski
• chore: Update description of execution_timeout in OpenLineage provider.yaml (#47448): @kacpermuda

Provider pinecone: 2.2.2rc1

• Fix pinecone client package rename after 6.0.0 (#46980): @jscheffl

Provider sftp: 5.1.1rc1

• Re-added close_conn connection on SFTPHook and get_conn should return SFTPClient instead of context managed connection (#47217): @dabla
• Fix SFTP hook's directory transfer to use get_managed_conn instead of get_conn (#47248): @Dawnpool
• Improve SFTP hook's directory transfer to use a single connection for multiple files (#46582): @Dawnpool

Provider snowflake: 6.1.1rc1

• Fix: SnowflakeSqlApiOperator(without deferable flag) incorrectly marks the task as successful even when jobs are still active (#46672): @subbota1
• [OpenLineage] fixed inputs in OL implementation of CopyFromExternalStageToSnowflakeOperator (#47168): @rahul-madaan
  Linked issues:
  • Linked Issue #47019: @rahul-madaan
• Fix deprecation warnings in common.sql (#47169): @potiuk
  Linked issues:
  • Linked Issue #43747: @dabla

Provider ssh: 4.0.1rc1

• Make command for replacing newlines with literals \n in SSHOperatorconnection MacOs/Linux compatible (#47491): @geovalexis

Provider standard: 0.1.1rc1

• Relocate SmoothOperator to standard provider and add tests (#47530): @shahar1
• Add deferred pagination mode to GenericTransfer (#44809): @dabla

Provider teradata: 3.0.2rc1

• Fix deprecation warnings in common.sql (#47169): @potiuk
  Linked issues:
  • Linked Issue #43747: @dabla

Provider trino: 6.1.0rc1

• Add Trino Connection UI Fields Placeholders (#46996): @wei-juncheng
  Linked issues:
  • Linked Issue #45841: @jjavieralonso

Provider vertica: 4.0.1rc1

• Fix deprecation warnings in common.sql (#47169): @potiuk
  Linked issues:
  • Linked Issue #43747: @dabla

Provider ydb: 2.1.1rc1

• Fix deprecation warnings in common.sql (#47169): @potiuk
  Linked issues:
  • Linked Issue #43747: @dabla

All users involved in the PRs:
@rawwar @m1racoli @jedcunningham @kacpermuda @shahar1 @p-eye @vikramkoka @Dawnpool @gopidesupavan @vincbeck @potiuk @fredthomsen @AlteredOracle @subbota19 @dabla @amoghrajesh @agupta01 @geovalexis @kana

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[potiuk]

I checked both - sql deprecatio removal is gone, mongo has better limits. looks good.

[amoghrajesh]

Tested the following issues where I had contributed:

1. Scheduler should not crash when executor_config is provided with CE #47375
2. Restrict pymongo to less than 4.11 for our CI #47336 (was overridden by Proper limit for pymongo and better comment explaining root cause #47338 but thats good too)
3. Add regex validation for table names in 'bulk_load' #47254
4. Add regex validation for table names in 'bulk_dump' #47255
5. Adding ticks to user provided table names #47262

All of them work fine 👍🏽

[gopidesupavan]

Verified #47515 changes are related tests.

[m1racoli]

Successfully tested fix: don't use blocking property access for async purposes #47326. ✅

[olegkachur-e]

#47419 worked as expected.

[Dawnpool]

Tested #47248 and #46582 of the sftp provider. They all work fine👍

[kacpermuda]

Changes to OpenLineage provider work fine.

[dantonbertuol]

#46833 work fine.

[rahul-madaan]

tested #47168 It is working fine.

[subbota19]

successfully tested snowflake provider #46672

[dabla]

Following works:

• Add local scheme as alternative to file for using the ObjectStoragePath (Add local scheme as alternative to file for using the ObjectStoragePath #46670): @dabla
• Fix support for old method signature of pagination_function in MSGraphAsyncOperator (Fix support for old method signature of pagination_function in MSGraphAsyncOperator #47101): @dabla
• Re-added close_conn connection on SFTPHook and get_conn should return SFTPClient instead of context managed connection (Re-added close_conn connection on SFTPHook and get_conn should return SFTPClient instead of context managed connection #47217): @dabla

Add deferred pagination mode to GenericTransfer (#44809): @dabla doesn't work, it crashes with:

[2025-03-13, 13:10:29 UTC] {taskinstance.py:3313} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/airflow/.venv/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 763, in _execute_task
    result = _execute_callable(context=context, **execute_callable_kwargs)
  File "/airflow/.venv/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 734, in _execute_callable
    return ExecutionCallableRunner(
  File "/airflow/.venv/lib/python3.9/site-packages/airflow/utils/operator_helpers.py", line 252, in run
    return self.func(*args, **kwargs)
  File "/airflow/.venv/lib/python3.9/site-packages/airflow/models/baseoperator.py", line 1816, in resume_execution
    return execute_callable(context)
  File "/airflow/.venv/lib/python3.9/site-packages/airflow/providers/common/sql/operators/generic_transfer.py", line 177, in execute_complete
    raise AirflowException(event.get("message"))
airflow.exceptions.AirflowException: get_hook() got an unexpected keyword argument 'hook_params'

Probably the hooks_params isn't available in in BaseHook in Airflow 2.9, so will need to fix that.

I've create a PR for that which fixes this issue, so an rc2 for common-sql provider will probably be needed.

[eladkal]

Add deferred pagination mode to GenericTransfer (#44809): @dabla doesn't work, it crashes with:

we discussed offline and since the problem affect only GenericTransfer and only with pagination mode which is new we will continue with release. This is a large wave trying to remove common-sql may result in removing many other providers as well.
I will followup with ad-hoc release for common-sql after this wave