external_python task throws exception in Airflow 3.0.4 when expect_airflow=False

[GB6638]

Apache Airflow version

3.0.5

If "Other Airflow 2 version" selected, which one?

No response

What happened?

When using @task.external_python with a virtual environment that does not have Airflow or Pendulum installed:

In Airflow 2.9.1, setting expect_airflow=False suppresses the warning No package metadata was found for apache-airflow and the task runs normally.

In Airflow 3.0.4, setting expect_airflow=False also suppresses the warning and the task instead throws an exception.

What you think should happen instead?

Setting expect_airflow=False should suppress warnings and also allow the task to run in an external Python environment without Airflow or Pendulum installed, similar to behavior in 2.9.1.

How to reproduce

1. Create a virtual environment without Airflow or Pendulum.

2. Define a task using @task.external_python:

from airflow.decorators import task
from datetime import timedelta

@task.external_python(
task_id="model_train_uk_auction_task",
execution_timeout=timedelta(seconds=1200),
retries=0,
on_failure_callback=task_failure_callback,
python="path_to_external_env_python",
expect_airflow=False,
)
def model_train():
pass

3. Run the DAG.

Operating System

Ubuntu 22.04

Versions of Apache Airflow Providers

Airflow 3.0.4

Python version: 3.10,14

Deployment

Virtualenv installation

Deployment details

No response

Anything else?

Adding the following patch resolves the issue:

def _iter_serializable_context_keys(self):
    yield from self.BASE_SERIALIZABLE_CONTEXT_KEYS
    if self.expect_airflow and self._get_airflow_version_from_target_env():
        yield from self.AIRFLOW_SERIALIZABLE_CONTEXT_KEYS
        yield from self.PENDULUM_SERIALIZABLE_CONTEXT_KEYS
    elif self._is_pendulum_installed_in_target_env():
        yield from self.PENDULUM_SERIALIZABLE_CONTEXT_KEYS

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[kaxil]

PRs welcomed - assigned it you :)