Custom logging handler is ignored

[AradOz]

Apache Airflow version

3.1.2

If "Other Airflow 2/3 version" selected, which one?

No response

What happened?

Hi there,
I am using airflow 3.1.2 and I am trying to add an additional custom logging handler to the default logger.
I saw other people complaining about similar problems but I have not found a solution yet.

The goal of the handler is to merge all task logs of a dag-run into a single file:
logs/dag_id=<dag_id>/run_id=<run_id>/dag.log
NOTE: I want to keep the normal logging of tasks

I added it using this piece of code:

from copy import deepcopy
from airflow.config_templates.airflow_local_settings import DEFAULT_LOGGING_CONFIG

LOGGING_CONFIG = deepcopy(DEFAULT_LOGGING_CONFIG)

LOGGING_CONFIG["handlers"]["dag_run_log_aggregator"] = {
    "class": "config.handlers.dag_run_log_aggregator.DAGRunLogAggregatorHandler",
    "formatter": "airflow",
    "base_log_folder": "/opt/airflow/logs",
    "level": "INFO",
}

LOGGING_CONFIG["loggers"]["airflow.task"] = {
    "handlers": ["dag_run_log_aggregator", "task"],
    "level": "INFO",
    "propagate": True,
}

REMOTE_LOGGING = False
REMOTE_TASK_LOG = None

As you can see, it is loaded:

airflow info

Apache Airflow
version                | 3.1.2                                                                  
executor               | CeleryExecutor                                                         
task_logging_handler   | config.handlers.dag_run_log_aggregator.DAGRunLogAggregatorHandler,     
                       | airflow.utils.log.file_task_handler.FileTaskHandler                    
sql_alchemy_conn       | postgresql+psycopg2://airflow:airflow@postgres/airflow                 
dags_folder            | /opt/airflow/dags                                                      
plugins_folder         | /opt/airflow/plugins                                                   
base_log_folder        | /opt/airflow/logs                                                      
remote_base_log_folder | 

But, airflow ignores all.
I tried to check the handlers of the airflow.task logger on runtime:

@task()
def airflow() -> None:
    import logging
    logger = logging.getLogger("airflow.task")
    
    logger.info(logger.hasHandlers())
    logger.info(logger.handlers)

    parent = logger
    while parent:
        print(parent.name, parent.handlers)
        parent = parent.parent if hasattr(parent, "parent") else None

Output was weird:

[2025-11-07 17:09:36] INFO - Filling up the DagBag from /opt/airflow/dags/dag.py
[2025-11-07 17:09:36] INFO - True
[2025-11-07 17:09:36] INFO - []
[2025-11-07 17:09:36] INFO - airflow.task []
[2025-11-07 17:09:36] INFO - airflow []
[2025-11-07 17:09:36] INFO - Done. Returned value was: None
[2025-11-07 17:09:36] INFO - root [<StreamHandler 98 (INFO)>]

I would be grateful for any help about this situation, I tried almost everything according to airflow's logging documentation.

What you think should happen instead?

No response

How to reproduce

Just setup airflow 3.1.2 on docker compose and try to add a custom logging handler

Operating System

Ubuntu 20.04.6 LTS

Versions of Apache Airflow Providers

No response

Deployment

Docker-Compose

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[amoghrajesh]

CC @ashb

[rich7420]

Could I try on this issue?

[amoghrajesh]

Feel free.

[DartVeDroid]

It's broken in 3.0.1, too.

Currently upgrading 2.11.0 -> 3.0.1, and startup logs look weird (see below). Lines starting with <class are made by me to debug what's happening - and to me it seems like:

• Airflow loading configs
• Airflow creating the correct handlers
• Airflow checking configs for deprecations
• Airflow creating some other handlers
• Airflow reporting that there are custom logging configs

And the problem, IMO, is that while console handler's emit() method will be called later, task handler's emit() method will not (no logs to prove that, because I'm using file creation as a debug tool, sorry).

/opt/airflow/.venv/lib/python3.11/site-packages/airflow/configuration.py:2252 FutureWarning: The 'log_filename_template' setting in [logging] has the old default value of 'dag_id={{ ti.dag_id }}/run_id={{ ti.run_id }}/task_id={{ ti.task_id }}/{% if ti.map_index >= 0 %}map_index={{ ti.map_index }}/{% endif %}attempt={{ try_number }}.log'. This value has been changed to 'dag_id={{ ti.dag_id }}/run_id={{ ti.run_id }}/task_id={{ ti.task_id }}/{% if ti.map_index >= 0 %}map_index={{ ti.map_index }}/{% endif %}attempt={{ try_number|default(ti.try_number) }}.log' in the running config, but please update your config.
<class 'logging_config_package.logging_config.ModifiedRedirectStdHandler'>.{'stream': 'sys.stdout'}
<class 'logging_config_package.logging_config.ModifiedElasticsearchTaskHandler'>.{'base_log_folder': '/usr/tmp/logs/', 'end_of_log_mark': 'end_of_log', 'host': 'https://127.0.0.1:9200', 'frontend': 'https://127.0.0.1:5601', 'write_stdout': True, 'write_to_es': False, 'target_index': 'airflow-logs', 'json_format': True, 'json_fields': 'asctime, filename, lineno, levelname, message', 'host_field': 'host', 'offset_field': 'offset'}
/opt/airflow/.venv/lib/python3.11/site-packages/airflow/cli/cli_config.py:591 DeprecationWarning: The web_server_port option in [webserver] has been moved to the port option in [api] - the old setting has been used, but please update your config.
/opt/airflow/.venv/lib/python3.11/site-packages/airflow/cli/cli_config.py:597 DeprecationWarning: The workers option in [webserver] has been moved to the workers option in [api] - the old setting has been used, but please update your config.
/opt/airflow/.venv/lib/python3.11/site-packages/airflow/cli/cli_config.py:609 DeprecationWarning: The web_server_host option in [webserver] has been moved to the host option in [api] - the old setting has been used, but please update your config.
/opt/airflow/.venv/lib/python3.11/site-packages/airflow/cli/cli_config.py:614 DeprecationWarning: The access_logfile option in [webserver] has been moved to the access_logfile option in [api] - the old setting has been used, but please update your config.
/opt/airflow/.venv/lib/python3.11/site-packages/airflow/cli/cli_config.py:629 DeprecationWarning: The web_server_ssl_cert option in [webserver] has been moved to the ssl_cert option in [api] - the old setting has been used, but please update your config.
/opt/airflow/.venv/lib/python3.11/site-packages/airflow/cli/cli_config.py:634 DeprecationWarning: The web_server_ssl_key option in [webserver] has been moved to the ssl_key option in [api] - the old setting has been used, but please update your config.
<class 'logging.StreamHandler'>.{}
<class 'airflow.sdk.log.StdBinaryStreamHandler'>.{'stream': <_io.FileIO name='<stdout>' mode='wb' closefd=False>}
2025-12-04 00:25:19.976152 [info     ] Detected user-defined logging config. Attempting to load logging_config_package.logging_config.LOGGING_CONFIG [airflow.utils.serve_logs]
2025-12-04 00:25:19.976556 [info     ] Successfully imported user-defined logging config. Flask App will serve log from /usr/tmp/logs/ [airflow.utils.serve_logs]
2025-12-04 00:25:19.979329 [warning  ] The dag_dir_list_interval option in [scheduler] has been moved to the refresh_interval option in [dag_processor] - the old setting has been used, but please update your config. [py.warnings] category=DeprecationWarning filename=/opt/airflow/.venv/lib/python3.11/site-packages/airflow/dag_processing/bundles/base.py lineno=257
2025-12-04 00:25:19.980288 [info     ] starting stale bundle cleanup process [airflow.providers.celery.cli.celery_command]
[2025-12-04 00:25:20 +0000] [19] [INFO] Starting gunicorn 23.0.0
[2025-12-04 00:25:20 +0000] [19] [INFO] Listening at: http://[::]:8793 (19)
[2025-12-04 00:25:20 +0000] [19] [INFO] Using worker: sync
[2025-12-04 00:25:20 +0000] [21] [INFO] Booting worker with pid: 21
[2025-12-04 00:25:20 +0000] [22] [INFO] Booting worker with pid: 22
 
 -------------- celery@6f87e09ee399 v5.5.2 (immunity)
--- ***** -----
-- ******* ---- Linux-6.10.11-linuxkit-x86_64-with-glibc2.36 2025-12-04 00:25:20
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app:         airflow.providers.celery.executors.celery_executor:0x7f0aac693910
- ** ---------- .> transport:   redis://airflow:**@redis:6379/0
- ** ---------- .> results:     postgresql://airflow:**@postgres/airflow
- *** --- * --- .> concurrency: 16 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
 -------------- [queues]
                .> default          exchange=default(direct) key=default


[tasks]
  . execute_workload

2025-12-04 00:25:22.899541 [info     ] Connected to redis://airflow:**@redis:6379/0 [celery.worker.consumer.connection]
2025-12-04 00:25:22.902008 [info     ] mingle: searching for neighbors [celery.worker.consumer.mingle]
2025-12-04 00:25:23.911176 [info     ] mingle: all alone              [celery.worker.consumer.mingle]
2025-12-04 00:25:23.933480 [info     ] celery@6f87e09ee399 ready.     [celery.apps.worker]

[potiuk]

Is there a specific reason why you are migrating to 3.0.1 - when:

• in 3.0 - you have probably few hundred fixes (and only fixes) implemented after 3.0.1
• in 3.1 -> we are just voting on 3.1.4 which probably have cumulatively more than 1000 fixes + new features than 3.0.1

Sounds a weird choice, but I am sure you have some reasonable argument why you made that choice.

[DartVeDroid]

@potiuk, we do it piece by piece since 3.0.0 in a relatively free time, that's why the version is so low, and ideally we want to check and fix all the correct deprecation warnings before upgrading to a newer version. But we faced some problems, which, to us, looked similar enough to #49646 to try and up a single patch version.
Now, it didn't fix our problem by itself =D, and we probably would have downgraded back to 3.0.0, if we were not to face another problem - looking like this one.

UPD:
I do know that 3.1.X will all have a "deprecated-but-usable" stance on old configs.

[potiuk]

You can always migrate to 3.0.LATEST - there are only bugfixes there (we are following semver). This is the slow migration. When you are moving to earlier version of major.minor you are for sure having all the errors that have been fixed in the later patchlevel versions of major.minor. And possibly you are taking valuable time of maintainers for things that are already fixed.

Migrating one patchlevel at a time malkes absolutely no sense.

Closing it. Try with latest patchlevels, ideally with latest 3.1 and report again if you stil hagve hte issue.