Remove deprecated code from providers before Airflow 2.11.0 release

[eladkal]

Body

Following [LAZY CONSENSUS] Remove deprecations from providers prior to Airflow 2.11 release this issue will track the progress of the removal status.

How to remove?

1. PRs should be easy to review, You can handle more than 1 provider in the same PR but please use judgment about how many files the PR changes. Please use meaningful commit message.
2. Find all deprecated code in a provider and accommodate for the needed change (normally the code has deprecation warning that indicates what needs to be removed). Make sure you are not removing code that was just deprecated in recent PR and was never released in stable version. Do not bulk remove!
3. Add entry to CHANGELOG.rst with information about what was removed and how to mitigate (normally it's copy/paste with minor adjustments to the deprecation warning). Use the following blockright after the Changelog line:

main
....

.. warning::
  All deprecated classes, parameters and features have been removed from the {provider_name} provider package.
  The following breaking changes were introduced:

  * what was removed and how to mitigate

4. If provider was already handled and introduce new deprecation do not remove again (at least not for now).

List of providers:

• airbyte Remove deprecated code from Airbyte provider #44577
• alibaba Removed deprecated code from Alibaba provider package #44576
• amazon Remove deprecated stuff from Amazon provider package #42450
• apache.beam Removed deprecated code from provider Apach.beam #44700
• apache.cassandra no deprecations
• apache.drill Remove provider deprecations in Apache Drill #44575
• apache.druid Remove provider deprecations in Apache Druid #44765
• apache.flink. no deprecations
• apache.hdfs no deprecations
• apache.hive Remove deprecations from Apache hive Provider #44715
• apache.iceberg no deprecations
• apache.impala no deprecations
• apache.kafka no deprecations
• apache.kylin no deprecations
• apache.livy Remove Provider Deprecations in Apache Livy #44631
• apache.pig no deprecations
• apache.pinot no deprecations
• apache.spark Remove deprecated code from apache spark provider #44567
• apprise Remove Provider Deprecations in Apprise #44764
• arangodb No deprecations
• asana No deprecations
• atlassian.jira Remove Provider Deprecations in Atlassian Jira #44644
• celery Assume deprecations do not affect user and will stay
• cloudant No deprecations
• cncf.kubernetes remove all deprecations cncf.kubernetes #43689
• cohere No deprecations
• common.compat No deprecations
• common.io No deprecations
• common.sql Remove Provider Deprecations in Common SQL #44645
• databricks Remove deprecations from Databricks Provider #44566
• datadog No deprecations
• dbt Remove Provider Deprecations in DBT #44638
• dingding No deprecations
• discord No deprecations
• docker Remove Provider Deprecations in Docker #44583
• edge New provider, this is sooo fresh and cool it does not have deprecations (yet!)
• elasticsearch Remove Provider Deprecations in Elasticsearch #44629
• exasol No deprecations
• fab Remove deprecations from fab provider #44198
• facebook No deprecations
• ftp No deprecations
• github No deprecations
• google excluded from policy
• grpc No deprecations
• hashicorp Removed deprecated code from hashicorp provider #44598
• http Remove Provider Deprecations in HTTP #44542
• imap No deprecations
• influxdb No deprecations
• jdbc Remove Provider Deprecations in JDBC #44662
• jenkins No deprecations
• microsoft.azure Remove Provider Deprecations in Microsoft-AZURE #44763
• microsoft.mssql Remove Provider Deprecations in Microsoft-MSSQL #44762
• microsoft.psrp Remove Provider Deprecations in Microsoft-PSRP #44761
• microsoft.winrm No deprecations
• mongo No deprecations
• mysql Remove Provider Deprecations in MySQL #44665
• neo4j No deprecations
• odbc No deprecations
• openai No deprecations
• openfaas No deprecations
• openlineage Remove Provider Deprecations in OpenLineage #44636
• opensearch No deprecations
• opsgenie No deprecations
• oracle Remove Provider Deprecations in Oracle #44704
• pagerduty Remove deprecated code from Pagerduty provider  #44653
• papermill No deprecations
• pgvector No deprecations
• pinecone No deprecations
• postgres Remove Provider Deprecations in Postgres #44705
• presto No deprecations
• qdrant No deprecations
• redis Remove Provider Deprecations in Redis #44633
• salesforce No deprecations
• samba No deprecations
• segment No deprecations
• sendgrid Remove Provider Deprecations in SendGrid #44637
• sftp Remove deprecations from SFTP Provider #44740
• singularity No deprecations
• slack Remove deprecations from Slack Provider #44693
• smtp No deprecations
• snowflake Remove deprecations from Snowflake Provider #44756
• sqlite Remove Provider Deprecations in Sqlite #44707
• ssh Remove Provider Deprecations in SSH #44544
• standard Remove Provider Deprecations in Standard #44541
• tableau Remove deprecations from Tableau Provider  #44757
• telegram No deprecations
• teradata Remove deprecations from Teradata Provider #44746
• trino Remove Provider Deprecations in Trino  #44717
• vertica Remove Provider Deprecations in Vertica #44748
• weaviate Remove deprecations from Weaviate Provider #44745
• yandex Remove Provider Deprecations in Yandex provider #44754
• ydb No deprecations
• zendesk No deprecations

Maintainers please update the list with the relevant PRs/status

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[Prab-27]

Hello @potiuk I'm a newbie
I would like to confirm whether I should use the v2-10-test branch for this issue since it is related to version 2

[kunaljubce]

@Prab-27 If you're talking about which branch to raise the PR on, that should be main. Refer to the tagged PRs for inspiration.

[kunaljubce]

@eladkal So in the context of this change, are we good to deprecate the Apache Drill provider completely?

airflow/providers/src/airflow/providers/apache/drill/operators/drill.py

Line 33 in 3747c91

class DrillOperator(SQLExecuteQueryOperator):

[ajitg25]

Hello @eladkal @jedcunningham ,
I have raised a PR to remove deprecated code from Airbyte providers.
Please have a look and let me know if anychanges are required.

Thank you for your time!!

[Prab-27]

@eladkal I’m working on the hashicorp provider and will soon raise a PR.

[jscheffl]

@eladkal I just checked for the noted deprecations in Celery - these are purely for executor interface, not DAG author specific - and the change was just added and require Airflow 2.10.4 as minimum. If we add a breaking change, Celery can not be upgraded for Airflow 2.10.3 and below.
Therefore I'd recommend to keep the Celery deprecations for a moment. No harm for the intend to help users to migrate.

--> FYI marked as to keep it. Hope it is OK?

[Prab-27]

@jscheffl - How can I determine the version number like 5.0.0 in CHANGELOG.rst?

I believe the version number in CHANGELOG.rst is derived from the next versions mentioned in provider.yaml .

[eladkal]

@jscheffl - How can I determine the version number like 5.0.0 in CHANGELOG.rst?

I believe the version number in CHANGELOG.rst is derived from the next versions mentioned in provider.yaml .

You don't, see the instructions in step 3. You set main not version. The version number is chosen by release manager during release time.

[jason810496]

Working on

• Elasticsearch
• Jekins
• Apache Livy
• Mongo
• Redis
• SendGrid
• DBT
• Atlassian Jira
• Common SQL

[kunaljubce]

@eladkal @jscheffl Below have no deprecations. Please update the description -

• apache.hive
• apache.flink
• apache.iceberg
• apache.impala
• apache.kafka
• apache.kylin

[potiuk]

Updatiing

[jscheffl]

Updatiing

Updating descriptions in Github is a pain. I did it already ~10 times and it seems if others update as well then sometimes (at least 10 times already) my changes are lost. Also notes down a few multiple times which have no deprecations. Tracking a set of PRs on a bug ticket is something Github is not made for :-(

[kunaljubce]

FYI - Working on Apache Drill and Apache Druid deprecations.

[potiuk]

Updating descriptions in Github is a pain. I did it already ~10 times and it seems if others update as well then sometimes (at least 10 times already) my changes are lost. Also notes down a few multiple times which have no deprecations. Tracking a set of PRs on a bug ticket is something Github is not made for :-(

What I did it in the past is re-generating those issues automatically from time to time ... Another thing-> I did not link/update the PRs in the list just "clicked" the checkmark - it was good enough usually.

[potiuk]

But maybe a good idea is to do (or find?) a small tool that should do such update via API?

say:

breeze check-issue #12345 --item abc --with-pr #27456

😱

That would make it far more usable

[kunaljubce]

@potiuk @jscheffl While we're at it, below also don't have any deprecations -

• apache.pig
• apache.pinot
• apache.cassandra

With the apache.druid PR up, looks like we're done with the Apache series of deprecations :)

[potiuk]

🚀

[potiuk]

Is it done ?????! ??? 🚀

[potiuk]

Shall you do the honors @eladkal and close it ?

[eladkal]

Created a small followup #44807 to make docs better :)