Make logging code self-contained

[gtsiam]

• Rewrite logging code to use python's builting QueueListener, effectively moving the logging process into a thread of the Frigate app.
• Also, wrap this behaviour in a easy-to-use context manager to encourage some level of consistency for other processes.

---

@NickM-27 @hawkeye217 @blakeblackshear Sorry for pinging randomly, but I'm seeing my other PR (#13765) get forgotten, so I'm assuming it got buried by the dependabot noise.

TL;DR: Whom should I annoy to get things merged? Maintaining multiple branches gets old, fast. Unless you prefer huge, unreviewable, PRs.

I'm planning to deploy frigate in my own nvr, but I have quite a few fixes I want to contribute to get it working the way I want to. I figured you'd prefer contributions to me nagging you via issues.

Next is probably faster shutdown, assuming I don't get side tracked.

I'm also planning to improve the webui, to get it into a state where I can use that as a birdseye view with a simple web client. Is there somewhere I can reach you directly (matrix/discord/irc/pigeon)?

[netlify]

✅ Deploy Preview for frigate-docs canceled.

Name	Link
🔨 Latest commit	fa73c86
🔍 Latest deploy log	https://app.netlify.com/sites/frigate-docs/deploys/66e97beb474784000853dcc2

[blakeblackshear]

We definitely prefer contributions over issues. We have had several large PRs opened in the past that haven't been merged because there was context that the author wasn't aware of. I definitely encourage you to communicate your plans prior to investing a lot of effort as we are moving relatively fast.

A series of smaller PRs like this are preferred. We are all on the Home Assistant discord.

blakeblackshear
nick_mowen_the_lawn
hawkeye217

[blakeblackshear]

The logging implementation here was based on the recommended approach for multiprocessing logging in the Python docs. https://docs.python.org/3/howto/logging-cookbook.html#logging-to-a-single-file-from-multiple-processes

I'm guessing you are suggesting we switch to the variant where the logging is a thread of the main process.

I'm curious what problem you are solving by doing this? We have quite a few things happening in the main thread when users have a large number of cameras.

[gtsiam]

Oh; I didn't realize that was in the cookbook. Still, I find that particular implementation rather dubious. Mainly in the cleanup department.

The main point of the PR is the log_thread context manager: You can just do @log_thread() or with log_thread(): to use it, instead of having to join a process in the FrigateApp in various places.

I could move log_thread to use a separate process, but:

• QueueListener is in the standard library. The less code we have to write, the better.
• I don't see moving to a thread from a process as neither a win nor a loss. Besides if your bottleneck is the logger, you're doing something wrong (And currently, not enough things are logged for that to be the case).
• Fewer processes generally means easier lifecycle management (So maybe a minor win).

[gtsiam]

Oh, I forgot to run mypy. Let me fix that.

[NickM-27]

This PR re-introduces an issue that was fixed already which is that the onvif package forcefully introduces its own log handler which will repeat all logs without the formatting applied. This is why the

    if root.hasHandlers():
        root.handlers.clear()

is there. This can't be controlled from the onvif package

[NickM-27]

When testing inside the dev container the main process does not stop cleanly, it has to be killed

[gtsiam]

When testing inside the dev container the main process does not stop cleanly, it has to be killed.

I couldn't figure out how you're running it inside the dev container, so I'm just doing make run each time. What's the command that doesn't exit?

[NickM-27]

What's the command that doesn't exit?

the main python process that runs, python3 -m frigate

[NickM-27]

Okay, everything looks to be working correctly now

[gtsiam]

When testing inside the dev container the main process does not stop cleanly, it has to be killed.

I don't thing this is due to my changes. This seems to happen when FrigateApp crashes because of an exception and the relevant stop events are simply never raised. So non-daemon threads are never stopped.