Support filtering log messages by module at runtime.

[t-moe]

Our usecase:
We develop an embedded iot product where we use defmt to store the logs into the flash of the device. When we have a defective device, the customer sends it back, and we can extract the logs from the flash. Now, it would be nice, to also be able to request the logs from the device over the air. Currently we are only able to filter the stored logs by severity, based on the "msg id". In order to save bandwidth, it would be nice, if the device could also filter the stored logs by module.

To make that work, defmt would need to store a "module id" with each message. (Gated behind a feature?)
Quectel is using a similar approach on their modems (Quectel EPAT Format) and I think it allows for quite flexible usecases....

[BriocheBerlin]

Hi, thanks for your message. What you have going on there sounds neat!
I am not entirely sure I am understanding correctly. Are you offering to do the implementation or would you like us to do it?

[t-moe]

I just wanted to open a tracking issue for the feature for now and to get some feedback on the topic... Would adding such a feature even be in scope for defmt? After all, it changes the binary format...

And yes, I could imagine helping to develop this feature.... But let's first discuss if it's even possible/wanted...

EDIT:
To clarify: We don't have the "defmt store to flash"-feature implemented yet either, but that's definitely something we'll do and I'll contribute to defmt. But storing the logs to flash should be easy and there's no change required in the current API. However, being able to filter the logs by module is a bigger change, that's why I opened this issue to discuss it...

[jonathanpallant]

I like the idea of having both compile-time and run-time log levels, to better suit long-running binaries on embedded devices deployed in the field.

[t-moe]

Am I correct that the format strings indices are no longer sorted by log-level (or never were)?
This would effectively mean that the device has currently no way to filter a log msg at runtime based on its string-index.

(note to myself: Here's how pigweed does it: https://pigweed.dev/pw_log_tokenized/#configuring-metadata-bit-fields )

In the meantime I've implemented a defmt log sink which stores all logs in the flash memory of the device. I've also implemented the on-demand extraction of the stored logs in order to send them to the cloud. I plan to eventually open-source these components. I think with the current data-format I can filter the messages by timestamp at runtime. But I would be nice if I could also filter at least by log-level, better even by modules...