[Feature Request]: Retire native JSON MQTT functionality

[thebentern]

Platform

Cross-Platform

Description

This has been a problematic feature for several reasons:

1. It takes up precious space on NRF52 based devices which are approaching the limits of flash utilization
2. It has created potential security concerns in the past and there are likely undiscovered ones existing now.
3. It should be an application layer concern where the implementation can be less prescriptive and tailored to the use case.

Tagging @ianmcorvidae because he has some migration strategies to externalize the support while keeping things in the format they exist in currently for the firmware so that automations in Home Assistant or otherwise can continue working with a bit of extra sauce.

[rcarteraz]

I like this idea. It's definitely something that needs to be addressed and @ianmcorvidae has some great ideas around this. Looking forward to seeing the solution.

[GUVWAF]

Agree on this, although indeed with the "requirement" that it would need to be able to easily integrate with popular home automation systems like Home Assistant and OpenHAB. Ideally it would be a native integration/binding such that the end-user doesn't have to handle this glue code and run it in an additional daemon.

[pdxlocations]

It case it's useful, I've roughed out a python script for this purpose. Maybe it's possible to use something like this as a custom addon to HA?
https://github.com/pdxlocations/mqtt-protobuf-to-json

[asjmcguire]

From what I can tell having a quick look at this @pdxlocations it only seems to cope with decoding mesh packets and outputting them to JSON, which indeed is something I still need, but much more important for me, given I use Home Assistant to broadcast weather updates on both a dedicated channel more frequently, and longfast less frequently, is the ability to SEND a mqtt message to be broadcast. So code would need to be able to figure out which portnum to use, and what channel.

[GUVWAF]

@garthvh Why are we planning to retain the JSON functionality for the RAK Ethernet gateway? #5498 (comment)

Then someone would still need to maintain this (add new fields and fix bugs like the one linked), and it can only be tested on one specific hardware combo. Besides, once we have a suitable alternative it will likely be slightly different than the manually crafted JSON, so you can't use the same system with different hardware. In my opinion if we remove it from the firmware, we have to nuke it altogether.

[garthvh]

@garthvh Why are we planning to retain the JSON functionality for the RAK Ethernet gateway? #5498 (comment)

Then someone would still need to maintain this (add new fields and fix bugs like the one linked), and it can only be tested on one specific hardware combo. Besides, once we have a suitable alternative it will likely be slightly different than the manually crafted JSON, so you can't use the same system with different hardware. In my opinion if we remove it from the firmware, we have to nuke it altogether.

I don't anticipate it being something that the project maintains, that device is managed by @beegee-tokyo and RAK so my thought was that he could continue to maintain it if necessary, but I am not opposed to nuking it.

[beegee-tokyo]

@garthvh I am ok, if you replace it with something else that works.
MQTT with JSON might not be important for "messaging", but when using Meshtastic in other applications like sensor networks, control functions, it is essential, specially because you don't offer a working raw payload decoder at this point.
I can maintain MQTT until you come up with something else.

[leshniak]

It case it's useful, I've roughed out a python script for this purpose. Maybe it's possible to use something like this as a custom addon to HA? https://github.com/pdxlocations/mqtt-protobuf-to-json

I've started a similar project based on node.js a month ago https://github.com/leshniak/meshtastic-protobuf-decoder (for private use at the time). @pdxlocations but yours suits better for HA.

[beegee-tokyo]

@leshniak
Any plans to add your project to NodeRED official packages?
I am struggling to add your meshtastic-protobuf-decoder to my NodeRED installation.

[fifieldt]

@broglep is working on a home assistant integration at https://github.com/broglep/homeassistant-meshtastic that looks like it won't need json+mqtt

[pdxlocations]

An HA integration using protos and MQTT: https://github.com/kvj/hass_Mtastic_MQTT