Error with GCS remote logging #51761
Description
Official Helm Chart version
main (development)
Apache Airflow version
3.0.2
Kubernetes Version
v1.32.4-gke.1353003
Helm Chart configuration
values.yaml
airflow:
config:
logging:
colored_console_log: 'True'
remote_logging: 'True'
remote_base_log_folder: 'gs://airflow-cluster-logs/v3'
remote_log_conn_id: 'gcp-automation-logs'
delete_local_logs: 'True'
gcp-automation-logs connection in Airflow
{
"project": "<project-name>",
"key_path": null,
"keyfile_dict": "***",
"credential_config_file": null,
"scope": "https://www.googleapis.com/auth/devstorage.read_write",
"key_secret_name": null,
"key_secret_project_id": null,
"num_retries": 5,
"impersonation_chain": null,
"idp_issuer_url": null,
"client_id": null,
"client_secret": null,
"idp_extra_parameters": null,
"is_anonymous": false
}
Docker Image customizations
Dockerfile
FROM apache/airflow:slim-3.0.2-python3.12
USER root
RUN apt update \
&& ACCEPT_EULA=Y apt upgrade -y \
&& apt install -y --no-install-recommends \
git-all \
neovim \
libglib2.0-0 \
libsm6 \
libxext6 \
libxrender1 \
libopencv-dev \
ffmpeg \
&& apt autoremove -yqq --purge \
&& apt clean \
&& rm -rf /var/lib/apt/lists/*
USER airflow
WORKDIR /opt/airflow
COPY requirements.txt /opt/airflow/requirements.txt
RUN uv tool install ruff
RUN uv pip install --upgrade pip
RUN uv pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" -r requirements.txt
requirements.txt
a2wsgi==1.10.8
aiofiles==24.1.0
aiohappyeyeballs==2.6.1
aiohttp==3.12.12
aiologic==0.14.0
aiosignal==1.3.2
aiosqlite==0.21.0
alembic==1.16.1
amqp==5.3.1
annotated-types==0.7.0
anyio==4.9.0
apache-airflow==3.0.2
apache-airflow-core==3.0.2
apache-airflow-providers-celery==3.11.0
apache-airflow-providers-common-compat==1.7.0
apache-airflow-providers-common-io==1.6.0
apache-airflow-providers-common-sql==1.27.1
apache-airflow-providers-fab==2.2.0
apache-airflow-providers-google==15.1.0
apache-airflow-providers-http==5.3.0
apache-airflow-providers-postgres==6.2.0
apache-airflow-providers-redis==4.1.0
apache-airflow-providers-smtp==2.1.0
apache-airflow-providers-standard==1.2.0
apache-airflow-task-sdk==1.0.2
apispec==6.8.2
argcomplete==3.6.2
asgiref==3.8.1
asyncpg==0.30.0
attrs==25.3.0
babel==2.17.0
backoff==2.2.1
billiard==4.2.1
blinker==1.9.0
cachelib==0.13.0
cachetools==5.5.2
cadwyn==5.3.3
cattrs==25.1.1
celery==5.5.3
certifi==2025.4.26
cffi==1.17.1
chardet==5.2.0
charset-normalizer==3.4.2
click==8.2.1
click-didyoumean==0.3.1
click-plugins==1.1.1
click-repl==0.3.0
clickclick==20.10.2
colorama==0.4.6
colorlog==6.9.0
connexion==2.14.2
cron-descriptor==1.4.5
croniter==6.0.0
cryptography==42.0.8
db-dtypes==1.4.3
decorator==5.2.1
Deprecated==1.2.18
dill==0.3.1.1
dnspython==2.7.0
docstring_parser==0.16
email_validator==2.2.0
fastapi==0.115.12
fastapi-cli==0.0.7
Flask==2.2.5
Flask-AppBuilder==4.6.3
Flask-Babel==2.0.0
Flask-JWT-Extended==4.7.1
Flask-Limiter==3.12
Flask-Login==0.6.3
Flask-Session==0.5.0
Flask-SQLAlchemy==2.5.1
Flask-WTF==1.2.2
flower==2.0.1
frozenlist==1.7.0
fsspec==2025.5.1
gcloud-aio-auth==5.4.2
gcloud-aio-bigquery==7.1.0
gcloud-aio-storage==9.4.0
gcsfs==2025.5.1
google-ads==27.0.0
google-analytics-admin==0.24.0
google-api-core==2.25.0
google-api-python-client==2.172.0
google-auth==2.40.3
google-auth-httplib2==0.2.0
google-auth-oauthlib==1.2.2
google-cloud-aiplatform==1.97.0
google-cloud-alloydb==0.4.6
google-cloud-appengine-logging==1.6.2
google-cloud-audit-log==0.3.2
google-cloud-automl==2.16.4
google-cloud-batch==0.17.36
google-cloud-bigquery==3.34.0
google-cloud-bigquery-datatransfer==3.19.1
google-cloud-bigquery-storage==2.32.0
google-cloud-bigtable==2.31.0
google-cloud-build==3.31.1
google-cloud-compute==1.31.0
google-cloud-container==2.57.0
google-cloud-core==2.4.3
google-cloud-datacatalog==3.27.1
google-cloud-dataflow-client==0.9.0
google-cloud-dataform==0.6.2
google-cloud-dataplex==2.10.2
google-cloud-dataproc==5.20.0
google-cloud-dataproc-metastore==1.18.3
google-cloud-dlp==3.30.0
google-cloud-firestore==2.21.0
google-cloud-kms==3.5.1
google-cloud-language==2.17.1
google-cloud-logging==3.12.1
google-cloud-managedkafka==0.1.11
google-cloud-memcache==1.12.1
google-cloud-monitoring==2.27.1
google-cloud-orchestration-airflow==1.17.5
google-cloud-os-login==2.17.2
google-cloud-pubsub==2.30.0
google-cloud-redis==2.18.1
google-cloud-resource-manager==1.14.2
google-cloud-run==0.10.18
google-cloud-secret-manager==2.24.0
google-cloud-spanner==3.55.0
google-cloud-speech==2.32.0
google-cloud-storage==2.19.0
google-cloud-storage-transfer==1.17.0
google-cloud-tasks==2.19.2
google-cloud-texttospeech==2.27.0
google-cloud-translate==3.20.2
google-cloud-videointelligence==2.16.1
google-cloud-vision==3.10.1
google-cloud-workflows==1.18.1
google-crc32c==1.7.1
google-genai==1.19.0
google-resumable-media==2.7.2
googleapis-common-protos==1.70.0
greenlet==3.2.2
grpc-google-iam-v1==0.14.2
grpc-interceptor==0.15.4
grpcio==1.72.1
grpcio-gcp==0.2.2
grpcio-status==1.62.3
gspread==6.2.1
gunicorn==23.0.0
h11==0.16.0
httpcore==1.0.9
httplib2==0.22.0
httptools==0.6.4
httpx==0.28.1
humanize==4.12.3
idna==3.10
immutabledict==4.2.1
importlib_metadata==8.4.0
inflection==0.5.1
itsdangerous==2.2.0
Jinja2==3.1.6
jmespath==1.0.1
joblib==1.5.1
jsonschema==4.24.0
jsonschema-specifications==2025.4.1
kombu==5.5.4
lazy-object-proxy==1.11.0
libcst==1.8.0
limits==5.2.0
linkify-it-py==2.0.3
lockfile==0.12.2
looker-sdk==25.8.0
Mako==1.3.10
markdown-it-py==3.0.0
MarkupSafe==3.0.2
marshmallow==4.0.0
marshmallow-sqlalchemy==1.4.2
mdurl==0.1.2
methodtools==0.4.7
more-itertools==10.7.0
msgspec==0.19.0
multidict==6.4.4
numpy==1.26.4
oauthlib==3.2.2
opencv-python==4.11.0.86
opentelemetry-api==1.27.0
opentelemetry-exporter-otlp==1.27.0
opentelemetry-exporter-otlp-proto-common==1.27.0
opentelemetry-exporter-otlp-proto-grpc==1.27.0
opentelemetry-exporter-otlp-proto-http==1.27.0
opentelemetry-proto==1.27.0
opentelemetry-sdk==1.27.0
opentelemetry-semantic-conventions==0.48b0
ordered-set==4.1.0
packaging==25.0
pandas==2.1.4
pandas-gbq==0.29.1
pathspec==0.12.1
pendulum==3.1.0
pluggy==1.6.0
prison==0.2.1
prometheus_client==0.22.1
prompt_toolkit==3.0.51
propcache==0.3.2
proto-plus==1.26.1
protobuf==4.25.8
psutil==7.0.0
psycopg2-binary==2.9.10
pyarrow==20.0.0
pyasn1==0.6.1
pyasn1_modules==0.4.1
pycparser==2.22
pydantic==2.11.5
pydantic-extra-types==2.10.5
pydantic_core==2.33.2
pydata-google-auth==1.9.1
Pygments==2.19.1
PyJWT==2.10.1
pyOpenSSL==25.1.0
pyparsing==3.2.3
python-daemon==3.1.2
python-dateutil==2.9.0.post0
python-dotenv==1.1.0
python-multipart==0.0.20
python-slugify==8.0.4
python-telegram-bot==22.1
pytz==2025.2
PyYAML==6.0.2
redis==5.2.1
referencing==0.36.2
requests==2.32.3
requests-oauthlib==2.0.0
requests-toolbelt==1.0.0
retryhttp==1.3.3
rich==13.9.4
rich-argparse==1.7.1
rich-toolkit==0.14.7
rpds-py==0.25.1
rsa==4.9.1
ruamel.yaml==0.18.14
ruamel.yaml.clib==0.2.12
scikit-learn==1.7.0
scipy==1.15.3
sentry-sdk==2.29.1
setproctitle==1.3.6
setuptools==80.8.0
shapely==2.1.1
shellingham==1.5.4
six==1.17.0
sniffio==1.3.1
SQLAlchemy==1.4.54
sqlalchemy-bigquery==1.14.1
SQLAlchemy-JSONField==1.0.2
sqlalchemy-spanner==1.13.0
SQLAlchemy-Utils==0.41.2
sqlparse==0.5.3
starlette==0.46.2
structlog==25.4.0
svcs==25.1.0
tabulate==0.9.0
tenacity==9.1.2
termcolor==3.1.0
text-unidecode==1.3
threadpoolctl==3.6.0
throttler==1.2.2
tornado==6.5.1
tqdm==4.67.1
typer==0.16.0
types-protobuf==6.30.2.20250516
types-requests==2.32.0.20250602
typing-inspection==0.4.1
typing_extensions==4.13.2
tzdata==2025.2
uc-micro-py==1.0.3
universal_pathlib==0.2.6
uritemplate==4.2.0
urllib3==2.4.0
uuid6==2024.7.10
uv==0.7.8
uvicorn==0.34.3
uvloop==0.21.0
validators==0.35.0
vine==5.1.0
watchfiles==1.0.5
wcwidth==0.2.13
websockets==14.2
Werkzeug==2.2.3
wirerope==1.0.0
wrapt==1.17.2
WTForms==3.2.1
yarl==1.20.1
zipp==3.22.0
What happened
When the remote logging is enabled, the airflow-run-airflow-migrations-xxxxx pod fails with the following error:
ERROR! Maximum number of retries (20) reached.
Last check result:
$ airflow db check
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/logging_config.py", line 57, in load_logging_config
logging_config = import_string(logging_class_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/module_loading.py", line 39, in import_string
module = import_module(module_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/importlib/__init__.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 999, in exec_module
File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/config_templates/airflow_local_settings.py", line 187, in <module>
REMOTE_TASK_LOG = GCSRemoteLogIO(
^^^^^^^^^^^^^^^
TypeError: GCSRemoteLogIO.__init__() missing 3 required positional arguments: 'gcp_keyfile_dict', 'scopes', and 'project_id'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 5, in <module>
from airflow.__main__ import main
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/__init__.py", line 79, in <module>
settings.initialize()
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/settings.py", line 610, in initialize
LOGGING_CLASS_PATH = configure_logging()
^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/logging_config.py", line 84, in configure_logging
logging_config, logging_class_path = load_logging_config()
^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/logging_config.py", line 68, in load_logging_config
raise ImportError(
ImportError: Unable to load logging config from airflow.config_templates.airflow_local_settings.DEFAULT_LOGGING_CONFIG due to: TypeError:GCSRemoteLogIO.__init__() missing 3 required positional arguments: 'gcp_keyfile_dict', 'scopes', and 'project_id'
What you think should happen instead
Airflow should start logging to GCS.
How to reproduce
Pull the official helm chart from the main branch and configure values.yaml with the values provided in the "Helm Chart configuration" section.
Anything else
I have tried configurations with remote_log_conn_id, google_key_path, and even using application default credentials (ADC). Each configuration failed with the same error.
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct