Python 3.12 breaks backwards compatibility for logging configuration

[egor-tensin]

Bug report

Bug description:

This worked fine on previous versions:

import logging
import logging.config
import logging.handlers
import multiprocessing as mp


def main():
    config = {
        'version': 1,
        'handlers': {
            'sink': {
                'class': 'logging.handlers.QueueHandler',
                'queue': mp.get_context('spawn').Queue(),
            },
        },
        'root': {
            'handlers': ['sink'],
            'level': 'DEBUG',
        },
    }
    logging.config.dictConfig(config)


if __name__ == '__main__':
    main()

With Python 3.12, it drops an error:

Traceback (most recent call last):
  File "/home/egor/.pyenv/versions/3.12-dev/lib/python3.12/logging/config.py", line 581, in configure
    handler = self.configure_handler(handlers[name])
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/egor/.pyenv/versions/3.12-dev/lib/python3.12/logging/config.py", line 786, in configure_handler
    raise ValueError('No handlers specified for a QueueHandler')
ValueError: No handlers specified for a QueueHandler

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/egor/workspace/personal/cimple/../test.py", line 25, in <module>
    main()
  File "/home/egor/workspace/personal/cimple/../test.py", line 21, in main
    logging.config.dictConfig(config)
  File "/home/egor/.pyenv/versions/3.12-dev/lib/python3.12/logging/config.py", line 912, in dictConfig
    dictConfigClass(config).configure()
  File "/home/egor/.pyenv/versions/3.12-dev/lib/python3.12/logging/config.py", line 588, in configure
    raise ValueError('Unable to configure handler '
ValueError: Unable to configure handler 'sink'

More than that, even the example in the Logging Cookbook is broken now: https://docs.python.org/3/howto/logging-cookbook.html#a-more-elaborate-multiprocessing-example (fails with the same error).

Version info:

# python -VV
Python 3.12.0+ (heads/3.12:f108785, Nov  1 2023, 19:47:19) [GCC 13.2.1 20230801]

CPython versions tested on:

3.12

Operating systems tested on:

Linux

Linked PRs

• gh-111615: Fix regression in QueueHandler configuration. #111638
• [3.12] gh-111615: Fix regression in QueueHandler configuration. (GH-111638) #113507

[vsajip]

Thanks for the report. This is due to a fix for gh-93162 , which allows proper configuration of a QueueListener and QueueHandler together. I'm sorry about the breakage - it appears not enough consideration was given to configuring a Queuehandler without configuring a QueueListener. This was of course the default case before the change went in, and the cookbook example needs to be updated. How are you configuring the QueueListener in your environment?

[egor-tensin]

I'm using QueueListener/QueueHandler w/ multiprocessing to centralize logging in the parent process logger, which I think is common advise on the net. The complete setup is:

@contextmanager
def configure_logging_in_child(queue):
    config = {
        'version': 1,
        'handlers': {
            'sink': {
                'class': 'logging.handlers.QueueHandler',
                'queue': queue,
            },
        },
        'root': {
            'handlers': ['sink'],
            'level': 'DEBUG',
        },
    }
    logging.config.dictConfig(config)
    try:
        yield
    except Exception as e:
        logging.exception(e)

def child_main(log_queue):
    with configure_logging_in_child(log_queue):
        # Do some actual work.
        ...

def main():
    ...
    ctx = multiprocessing.get_context('spawn')
    queue = ctx.Queue()
    listener = logging.handlers.QueueListener(queue, logging.getLogger())
    listener.start()
    # There's actually a list of child processes.
    args = (queue,)
    child = ctx.Process(target=child_main, args=args)
    child.start()
    child.join()
    ...

To be honest, I don't really understand how the queue & listener can be configured in one call, since I configure them separately in the parent & child processes.

[vsajip]

To be honest, I don't really understand how the queue & listener can be configured in one call

Well, they can be if you're using threads rather than processes.

[egor-tensin]

Well, they can be if you're using threads rather than processes.

I understand. I cannot use Python threads though, since I need actual parallel execution (for a number of child processes).

I use a similar pattern in a couple of other projects, where I also must use multiprocessing, since the child process needs to drop permissions, execute as a different user, etc.

[vsajip]

Well, the change in the linked PR should resolve this, but it will have to wait for the next release of Python 3.12.

[egor-tensin]

Well, the change in the linked PR should resolve this, but it will have to wait for the next release of Python 3.12.

Thanks, sounds good!