Structlog 25.5 Breaks PythonVirtualEnvOperator

[amordoch]

Apache Airflow Provider(s)

standard

Versions of Apache Airflow Providers

apache-airflow-providers-standard==1.8.0

Apache Airflow version

3.1.0

Operating System

Amazon Linux 2023 (arm64)

Deployment

Official Apache Airflow Helm Chart

Deployment details

OOTB Helm chart on EKS

What happened

Structlog 25.5.0 removed Styles from structlog.dev, which prevents virtualenv tasks from running if they have apache-airflow in their requirements:

INFO - Output: source=airflow.utils.process_utils loc=process_utils.py:192
INFO - Traceback (most recent call last): source=airflow.utils.process_utils loc=process_utils.py:196
INFO -   File "/tmp/venv-callh3vcy6hf/script.py", line 145, in <module> source=airflow.utils.process_utils loc=process_utils.py:196
INFO -     arg_dict = pickle.load(file) source=airflow.utils.process_utils loc=process_utils.py:196
INFO -                ^^^^^^^^^^^^^^^^^ source=airflow.utils.process_utils loc=process_utils.py:196
INFO -   File "/tmp/.airflow-venv-cache/venv-cf3b86b0/lib/python3.12/site-packages/airflow/__init__.py", line 79, in <module> source=airflow.utils.process_utils loc=process_utils.py:196
INFO -     settings.initialize() source=airflow.utils.process_utils loc=process_utils.py:196
INFO -   File "/tmp/.airflow-venv-cache/venv-cf3b86b0/lib/python3.12/site-packages/airflow/settings.py", line 698, in initialize source=airflow.utils.process_utils loc=process_utils.py:196
INFO -     LOGGING_CLASS_PATH = configure_logging() source=airflow.utils.process_utils loc=process_utils.py:196
INFO -                          ^^^^^^^^^^^^^^^^^^^ source=airflow.utils.process_utils loc=process_utils.py:196
INFO -   File "/tmp/.airflow-venv-cache/venv-cf3b86b0/lib/python3.12/site-packages/airflow/logging_config.py", line 88, in configure_logging source=airflow.utils.process_utils loc=process_utils.py:196
INFO -     from airflow._shared.logging import configure_logging, init_log_folder, translate_config_values source=airflow.utils.process_utils loc=process_utils.py:196
INFO -   File "/tmp/.airflow-venv-cache/venv-cf3b86b0/lib/python3.12/site-packages/airflow/_shared/logging/__init__.py", line 27, in <module> source=airflow.utils.process_utils loc=process_utils.py:196
INFO -     from .structlog import configure_logging, init_log_file, init_log_folder source=airflow.utils.process_utils loc=process_utils.py:196
INFO -   File "/tmp/.airflow-venv-cache/venv-cf3b86b0/lib/python3.12/site-packages/airflow/_shared/logging/structlog.py", line 38, in <module> source=airflow.utils.process_utils loc=process_utils.py:196
INFO -     from .percent_formatter import PercentFormatRender source=airflow.utils.process_utils loc=process_utils.py:196
INFO -   File "/tmp/.airflow-venv-cache/venv-cf3b86b0/lib/python3.12/site-packages/airflow/_shared/logging/percent_formatter.py", line 29, in <module> source=airflow.utils.process_utils loc=process_utils.py:196
INFO -     from structlog.dev import ConsoleRenderer, Styles source=airflow.utils.process_utils loc=process_utils.py:196
INFO - ImportError: cannot import name 'Styles' from 'structlog.dev' (/tmp/.airflow-venv-cache/venv-cf3b86b0/lib/python3.12/site-packages/structlog/dev.py) source=airflow.utils.process_utils loc=process_utils.py:196
ERROR - Task failed with exception source=task loc=task_runner.py:994
CalledProcessError: Command '['/tmp/.airflow-venv-cache/venv-cf3b86b0/bin/python', '/tmp/venv-callh3vcy6hf/script.py', '/tmp/venv-callh3vcy6hf/script.in', '/tmp/venv-callh3vcy6hf/script.out', '/tmp/venv-callh3vcy6hf/string_args.txt', '/tmp/venv-callh3vcy6hf/termination.log', '/tmp/venv-callh3vcy6hf/airflow_context.json']' returned non-zero exit status 1.

File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 920 in run

File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1307 in _execute_task

File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/operator.py", line 416 in wrapper

File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/decorator.py", line 252 in execute

File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/operator.py", line 416 in wrapper

File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/standard/operators/python.py", line 491 in execute

File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/operator.py", line 416 in wrapper

File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/standard/operators/python.py", line 216 in execute

File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/standard/operators/python.py", line 888 in execute_callable

File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/standard/operators/python.py", line 578 in _execute_python_callable_in_subprocess

File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/process_utils.py", line 177 in execute_in_subprocess

File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/process_utils.py", line 200 in execute_in_subprocess_with_kwargs

What you think should happen instead

Task SDK's pyproject.toml should specify structlog < 25.5.0, so that new venv's spun up by the virtual env operator don't run into this compatibility issue (until the actual problem is resolved).

How to reproduce

1. Create DAG with virtual env operator task and specify apache-airflow in the requirements
2. Execute DAG and observe this error

Anything else

There is an easy workaround for now: specify structlog<25.5.0 in the task's requirements directly.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[kaxil]

You need to use constraints file for Airflow Installation which pins structlog to 25.4.0.

Our docs explain why: https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html#why-we-need-constraints

Airflow® installation can be tricky because Airflow is both a library and an application.

Libraries usually keep their dependencies open and applications usually pin them, but we should do neither and both at the same time. We decided to keep our dependencies as open as possible (in pyproject.toml) so users can install different versions of libraries if needed. This means that from time to time plain pip install apache-airflow will not work or will produce an unusable Airflow installation.

[amoghrajesh]

The only officially supported way to install Apache Airflow using pip is with constraints, as documented here:
🔗 https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html#why-we-need-constraints

Please make sure to use the constraint files, they ensure consistent and repeatable installations of Airflow and all provider dependencies. Unconstrained installations are not supported and are the most common cause of dependency-related errors like this one.

[amordoch]

@kaxil @amoghrajesh

I'm using a virtualenv task as I need the s3fs extra for the Amazon provider for my tasks. Even if I use only system_site_packages=True, wouldn't it still update structlog as much as it could based on the dependency spec in the toml files?

In any case I think it would make sense to update the phrasing in the Virtualenv operator documentation to recommend using system site packages only. It currently says this (emphasis mine):

For Airflow context variables make sure that you either have access to Airflow through setting system_site_packages to True or add apache-airflow to the requirements argument.

[amordoch]

As you both said, the better fix is this:

import sys

@task.virtualenv(
    requirements=["apache-airflow-providers-amazon[s3fs]"],
    pip_install_options=["-c", f"https://raw.githubusercontent.com/apache/airflow/refs/tags/constraints-3.1.0/constraints-{sys.version_info.major}.{sys.version_info.minor}.txt"],
    system_site_packages=True,
)
def my_task():
    ...

[amoghrajesh]

@amordoch feel free to contribute a PR to make the docs clearer, those are always welcome!