-
Notifications
You must be signed in to change notification settings - Fork 16.3k
Closed
Labels
area:providerskind:featureFeature RequestsFeature Requestskind:metaHigh-level information important to the communityHigh-level information important to the community
Description
We need community help to migrate all provider packages to use the new Task SDK BaseOperator import for Airflow 3.0 compatibility while maintaining backward compatibility with Airflow 2.x.
Background
With Airflow 3.0, the BaseOperator has moved to the Task SDK (airflow.sdk). All provider packages need to be updated to use version-compatible imports to work with both Airflow 2.x and 3.x versions.
Migration Tasks
Use #52366 & #52292 as reference.
Each provider needs these updates:
- Create/update
version_compat.pywith standardized version compatibility pattern - Update imports in all operator, sensor, hook, and link files
- Fix
BaseOperatorLink.persistmethods that causeAttributeErrorwith SDK BaseOperator - Update test patterns to use
dag_maker.run_ti()instead of deprecated patterns
Provider Status Checklist
- airbyte - @kevinhongzl - Provider Migration: Update airbyte provider for Airflow 3.0 compatibility #52418
- alibaba - @kaxil - Replace
models.BaseOperatorto Task SDK one for Alibaba & Airbyte #52335 - amazon - @kaxil - Replace
models.BaseOperatorto Task SDK one for Amazon Provider #52667 - apache/beam - @kgw7401 - Provider Migration: Update beam for Airflow 3.0 compatibility #52607
- apache/cassandra - @Nataneljpwd - Provider Migration: Update cassandra for Airflow 3.0 compatibility #52623
- apache/drill - No base operator
- apache/druid - @sjyangkevin - Provider Migration: Update Apache Druid for Airflow 3.0 compatibility #52498
- apache/flink - @sjyangkevin - Provider Migration: Update Apache Flink for Airflow 3.0 compatibility #52504
- apache/hdfs - @bdsoha - Provider Migration: Replace
models.BaseOperatorto Task SDK for apache/hdfs #52505 - apache/hive - @bdsoha - Provider Migration: Replace
models.BaseOperatorto Task SDK for apache/hive #52453 - apache/iceberg - no base operator
- apache/impala - @bdsoha - Provider Migration: Replace
models.BaseOperatorto Task SDK for apache/impala #52455 - apache/kafka - @sjyangkevin - Provider Migration: Update Apache Kafka for Airflow 3.0 compatibility #52503
- apache/kylin - @fweilun - Provider Migration: Update Apache Kylin for Airflow 3.0 compatibility #52572
- apache/livy - @bdsoha Provider Migration: Replace
models.BaseOperatorto Task SDK for apache/livy #52499 - apache/pig - @kaxil - Replace
models.BaseOperatorto Task SDK one for Apache Pig #52375 - apache/pinot - no base operator
- apache/spark - @kgw7401 - Provider Migration: Update spark for Airflow 3.0 compatibility #52448
- apache/tinkerpop - @ahmadtfarhan - Replace
models.BaseOperatorto Task SDK one for Apache TinkerPop #52400 - apprise - no base operator
- arangodb - @kaxil - Replace
models.BaseOperatorto Task SDK one for Asana & Arangodb #52374 - asana - @kaxil - Replace
models.BaseOperatorto Task SDK one for Asana & Arangodb #52374 - atlassian/jira - @kaxil - Replace
models.BaseOperatorto Task SDK one for Atlassian #52376 - celery - no base operator
- cloudant - no base operator
- cncf/kubernetes - @kaxil - Update Kubernetes provider for Airflow 3.0 compatibility #52664
- cohere - @sjyangkevin - Provider Migration: Update Cohere for Airflow 3.0 compatibility #52379
- common/compat - @kaxil - Replace models.BaseOperator to Task SDK one for Common Providers #52443
- common/io - @kaxil - Replace models.BaseOperator to Task SDK one for Common Providers #52443
- common/messaging - No BaseOperator
- common/sql - @kaxil - Replace models.BaseOperator to Task SDK one for Common Providers #52443
- databricks - @kaxil - Replace
models.BaseOperatorto Task SDK one for DBT & Databricks #52377 - datadog - @kevinhongzl Provider Migration: Replace
BaseSensorOperatorto Task SDK fordatadog#52583 - dbt/cloud - @kaxil - Replace
models.BaseOperatorto Task SDK one for DBT & Databricks #52377 - dingding - @kevinhongzl Provider Migration: Replace
BaseOperatorto Task SDK fordingding#52577 - discord - No BaseOperator
- docker - @hardeybisey - Provider Migration: Update docker for Airflow 3.0 compatibility #52465
- edge3 - not using BaseOperator
- elasticsearch - no base operator
- exasol - no base operator
- fab - no base operator
- facebook - no base operator
- ftp - @sunank200 - Replace models.BaseOperator to Task SDK one for ftp #52338
- git - no base operator
- github - @kevinhongzl - Provider Migration: Update github provider for Airflow 3.0 compatibility #52415
- google - @kaxil - Replace
models.BaseOperatorto Task SDK one for Google Provider #52366 - grpc - @karenbraganz - Update grpc BaseOperator imports for Airflow 3.0 compatibility #52603
- hashicorp - no base operator
- http - @bdsoha @kaxil - Provider Migration: Replace
models.BaseOperatorto Task SDK for apache/http #52506 | Provider Migration: ReplaceBaseOperatorto Task SDK forapache/http#52528 - imap - no base operator
- influxdb - @karenbraganz - Update influxdb BaseOperator imports for Airflow 3.0 compatibility #52602
- jdbc - no base operator
- jenkins - @gopidesupavan - Update Jenkins for Airflow 3.0 compatibility #52510
- keycloak - no base operator
- microsoft/azure - @kyungjunleeme - Provider Migration: Update azure for Airflow 3.0 compatibility #52618
- microsoft/mssql - no base operator
- microsoft/psrp - @sunank200 - Replace airflow.models.baseoperator to airflow.sdk for Microsoft PSRP #52654
- microsoft/winrm - @sunank200 - Replace airflow.models.baseoperator to airflow.sdk for Microsoft Winrm #52657
- mongo - @jroachgolf84 - Replace
models.BaseSensorOperatorto Task SDK one for Mongo #52566 - mysql - @kgw7401 - Provider Migration: Update mysql for Airflow 3.0 compatibility #52500
- neo4j - @kyungjunleeme Provider Migration: Update neo4j for Airflow 3.0 compatibility #52610
- odbc - no base operator
- openai - @kaxil - Provider Migration: Replace
BaseOperatorto Task SDK forOpenAI#52561 - openfaas - no base operator
- openlineage - no base operator
- opensearch - @kyungjunleeme - Provider Migration: Update opensearch for Airflow 3.0 compatibility #52609
- opsgenie - @jroachgolf84 - Replace
models.BaseOperatorto Task SDK one for OpsGenie #52564 - oracle - @sjyangkevin - Provider Migration: Update Oracle for Airflow 3.0 compatibility #52382
- pagerduty - no base operator
- papermill - @kaxil - Provider Migration: Replace
BaseOperatorto Task SDK forPapermill#52565 - pgvector - Does not directly use
BaseOperator - pinecone - @kaxil - Provider Migration: Replace
BaseOperatorto Task SDK forPinecone#52563 - postgres - @kgw7401 - no base operator
- presto - @kyungjunleeme Provider Migration: Update presto for Airflow 3.0 compatibility #52608
- qdrant - @karenbraganz - Update BaseOperator and BaseSensorOperator imports for Airflow 3.0 compatibility in qdrant provider #52600
- redis - @karenbraganz - Update BaseOperator and BaseSensorOperator imports for Airflow 3.0 compatibility in redis provider #52597
- salesforce - @kevinhongzl - Provider Migration: Replace
BaseOperatorto Task SDK forsalesforce#52587 - samba - @kevinhongzl - Provider Migration: Replace
BaseOperatorto Task SDK forsamba#52588 - segment - @gopidesupavan - Provider Migration: Replace BaseOperator to Task SDK for segment #52579
- sendgrid - no base operator
- sftp - @jroachgolf84 - Replace
models.BaseOperatorto Task SDK one for SFTP #52435 - singularity - @kevinhongzl - Provider Migration: Replace
BaseOperatorto Task SDK forsingularity#52590 - slack - @vatsrahul1001 - Replace models.BaseOperator to Task SDK one for Slack Provider #52347
- smtp - @gyli - Provider Migration: Replace models.BaseOperator to Task SDK for smtp #52596
- snowflake - @rawwar - Provider Migration: Update Snowflake provider for Airflow 3.0 compatibility #52629
- sqlite - no base operator
- ssh - @kaxil - Provider Migration: Replace
BaseOperatorto Task SDK forssh#52558 - standard - @kaxil - Replace
models.BaseOperatorto Task SDK one for Standard Provider #52292 - tableau - @jroachgolf84 - Replace
models.BaseOperatorto Task SDK one for Tableau, Telegram, and Teradata #52642 - telegram - @jroachgolf84 - Replace
models.BaseOperatorto Task SDK one for Tableau, Telegram, and Teradata #52642 - teradata - @jroachgolf84 - Replace
models.BaseOperatorto Task SDK one for Tableau, Telegram, and Teradata #52642 - trino - @kyungjunleeme Provider Migration: Update trino for Airflow 3.0 compatibility #52383
- vertica - no base operator
- weaviate - @sjyangkevin Provider Migration: Update Weaviate for Airflow 3.0 compatibility #52381
- yandex - @kevinhongzl - Provider Migration: Update yandex provider for Airflow 3.0 compatibility #52422
- ydb - no use of BaseOperator
- zendesk - no use of BaseOperator
Implementation Guide
If using AI Agents, refer to #52378 (comment)
Step 1: Create version_compat.py
# In providers/{provider_name}/src/airflow/providers/{provider_name}/version_compat.py
from __future__ import annotations
def get_base_airflow_version_tuple() -> tuple[int, int, int]:
from packaging.version import Version
from airflow import __version__
airflow_version = Version(__version__)
return airflow_version.major, airflow_version.minor, airflow_version.micro
AIRFLOW_V_3_0_PLUS = get_base_airflow_version_tuple() >= (3, 0, 0)
if AIRFLOW_V_3_0_PLUS:
from airflow.sdk import BaseOperator
else:
from airflow.models import BaseOperator
__all__ = ["AIRFLOW_V_3_0_PLUS", "BaseOperator"]Step 2: Update imports
Replace:
from airflow.models.baseoperator import BaseOperator
With:
from airflow.providers.{provider_name}.version_compat import BaseOperatorStep 3: Fix BaseOperatorLink.persist methods
Replace:
def persist(context: Context, task_instance: BaseOperator, url: str):
task_instance.xcom_push(key="link_key", value=url)With:
def persist(context: Context, url: str):
context["task_instance"].xcom_push(key="link_key", value=url)Step 4: Update test patterns
Replace operator.run() with dag_maker.run_ti(task_id, dr). Use #52366 & #52292 as reference.
Test your changes:
# Run provider tests
breeze testing providers-tests --test-type "Providers[{provider_name}]"
How to Contribute
- Pick an unchecked provider from the list above
- Comment on this issue to claim it (prevents duplicate work)
- Follow the implementation guide
- Submit a PR with title: Provider Migration: Update {provider_name} for Airflow 3.0 compatibility
- Check off the completed provider in this issue
potiuk, amoghrajesh and kyungjunleemepotiuk, amoghrajesh and kyungjunleemepotiuk and amoghrajesh
Metadata
Metadata
Assignees
Labels
area:providerskind:featureFeature RequestsFeature Requestskind:metaHigh-level information important to the communityHigh-level information important to the community