Skip to content

Conversation

@nailo2c
Copy link
Contributor

@nailo2c nailo2c commented Jun 22, 2025

Closes: #46863

Why

If an HTTP connection contains login/password and the user doesn't set auth_type, the sync path works (the hook auto-defaults to HTTPBasicAuth), but the deferrable path crashes:

def execute_async(self, context: Context) -> None:
self.defer(
trigger=HttpTrigger(
http_conn_id=self.http_conn_id,
auth_type=self.auth_type,

passes None to auth_type, store it in the trigger table.

 id |                    classpath                     |                                                                                                                           kwargs                                                                                                                            |         created_date         | triggerer_id
----+--------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------+--------------
  3 | airflow.providers.http.triggers.http.HttpTrigger | {"__var": {"http_conn_id": "http_default", "method": "GET", "auth_type": null, "endpoint": "get", "headers": {"__var": {}, "__type": "dict"}, "data": {"__var": {}, "__type": "dict"}, "extra_options": {"__var": {}, "__type": "dict"}}, "__type": "dict"} | 2025-06-18 04:01:51.49263+00 |            3
(1 row)

after deserialization, None is passed here, causing 'NoneType' object is not callable error.

if conn.login:
auth = self.auth_type(conn.login, conn.password)

How

  1. Add _resolve_auth_type() to HttpOperator;
    it auto-picks aiohttp.BasicAuth (when the conn has login/password) before deferral.

  2. Add serialize_auth_type(); always store the fully-qualified class path or None.

  3. Add deserialize_auth_type();
    HttpTrigger.__init__() uses it to turn the stored string back into the class object at runtime.

  4. Serialize auth_type when HttpOperator.execute_async() and deserialises it when HttpTrigger.__init__(),
    so internal logic remains unchanged.

What

Test DAG

from airflow import DAG
from airflow.providers.http.operators.http import HttpOperator
import pendulum

with DAG(
    dag_id="http_deferrable_bug_demo",
    start_date=pendulum.datetime(2025, 1, 1, tz="UTC"),
    schedule=None,
    catchup=False,
) as dag:
    HttpOperator(
        task_id="call_httpbin",
        http_conn_id="http_default",
        endpoint="get",
        method="GET",
        deferrable=True,
    )

Test connection

airflow connections add 'http_default' \
  --conn-type http \
  --conn-host 'https://httpbin.org' \
  --conn-login 'user' \
  --conn-password 'pass'

before
截圖 2025-06-22 auth_type is None

after
截圖 2025-06-22 auth_type is not None

the auth_type in the trigger table

 id |                    classpath                     |                                                                                                                                       kwargs                                                                                                                                       |         created_date          | triggerer_id
----+--------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------+--------------
  1 | airflow.providers.http.triggers.http.HttpTrigger | {"__var": {"http_conn_id": "http_default", "method": "GET", "auth_type": "aiohttp.helpers.BasicAuth", "endpoint": "get", "headers": {"__var": {}, "__type": "dict"}, "data": {"__var": {}, "__type": "dict"}, "extra_options": {"__var": {}, "__type": "dict"}}, "__type": "dict"} | 2025-06-19 22:52:32.606647+00 |            2
(1 row)

@potiuk
Copy link
Member

potiuk commented Jun 23, 2025

Great catch and good implementation.

@potiuk potiuk merged commit c22c1ef into apache:main Jun 23, 2025
71 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

HttpOperator doesn't work in deferred mode when using authenticated connection

2 participants