Direct Messages no longer sent over MQTT in JSON

[JohnathonMohr]

Category

Other

Hardware

Heltec V3

Firmware Version

2.5.4.8d288d5

Description

I have my node configured to forward traffic to my own local MQTT server to use with Home Assistant, such as what is described at https://meshtastic.org/docs/software/integrations/mqtt/home-assistant/. Everything was working as expected with the 2.4 Beta firmware, and I had automations set up to notify me of channel and direct messages.

After updating to 2.5 Beta (excited for PKI), I was disappointed to find that DMs were no longer getting forwarded to the JSON topic in MQTT. I have found that traffic such as telemetry, nodeinfo, and even channel messages are still forwarded to MQTT using the settings I already had configured, but DMs are just missing.

I’ve confirmed it isn’t simply a matter of going to a different topic, as I’ve set up a listener for “#” (grabbing everything) and sending DMs, and they do not get sent.

Maybe they show up in protobufs (I don’t have anything setup to be able to deserialize that topic)? I just know it’s missing in JSON.

Relevant log output

No response

[Talie5in]

@JohnathonMohr Between two 2.5.x devices with PKC (Green Lock) on both sender/receiver in a Android/iOS App?
If so, that would make sense, as you would no longer have the decryption key (that was originally at the Channel Level for DM's) to be able to decode the packet.

2.4.x <-> 2.5.x where PKC is not been used for DM's should however still be through MQTT.

[JohnathonMohr]

Yes, this is between 2 nodes with 2.5.x, using their unique keys (green lock in the app). I originally observed this in the app, but also confirmed same behavior when messaging via the in-device hosted web sites (not sure if you meant specifically using the apps to message, or just that the apps display the green locks).
Is the decryption key not stored in the Meshtastic device? If it's stored in the device (my assumption) what's preventing decryption before sending to MQTT (just like decryption with the channel key was done previously)?

[Talie5in]

Do you have encryption enabled in your MQTT config?

https://meshtastic.org/docs/configuration/module/mqtt/#encryption-enabled

But, typically most MQTT based tooling decodes the protobuf and ChannelKey (eg. Default Public LongFast is a known key, so anyone could of before 2.5.x decrypted your DMs or channel messages over RF or MQTT)

[JohnathonMohr]

Prior to 2.5 I had encryption disabled. I tried enabling it in my tests with 2.5, but didn't observe any difference (still saw decrypted nodeinfo, telemetry, etc. in JSON, no DMs). Maybe I would only observe a difference if I was using the protobufs?

And about who can read: yes, I noted that I could also read DMs between other nodes, which is expected given they were only encrypted with the default LF shared key. I expect to see others' DMs less and less with broader adoption of 2.5.

Still, since the device can decrypt the received message to display to the user, why can't it forward that to the MQTT JSON topic?

Is it a shortcoming of the design? I.e. maybe MQTT processing is done in a logically different area before DM decryption. If this is the case, I would request the design be fixed. As it stands, a documented method of integration is broken.

[jcyrio]

This same issue has been preventing me from upgrading to 2.5.

I made some apps that are able to send DMs over MQTT between any two nodes on a private encrypted channel. I couldn't get it to work on 2.5. I read on the forums that MQTT is now expecting a "PKI" topic, but haven't seen any documentation on this yet.

I'm using protobufs, not JSON.

[garthvh]

This same issue has been preventing me from upgrading to 2.5.

I made some apps that are able to send DMs over MQTT between any two nodes on a private encrypted channel. I couldn't get it to work on 2.5. I read on the forums that MQTT is now expecting a "PKI" topic, but haven't seen any documentation on this yet.

I'm using protobufs, not JSON.

For protobufs there is a pki topic, JSON really need to leave the device and be decoded externally.

[GUVWAF]

@JohnathonMohr can it be that only DMs not directed to the MQTT node are not being forwarded?

I believe this is what @ianmcorvidae already pointed out, if the MQTT node cannot decrypt the packet (which is the case when it's PKI encrypted and directed to another node), it won't uplink it to MQTT:

firmware/src/mesh/Router.cpp

Lines 616 to 617 in ddd4a45

	if (decoded && moduleConfig.mqtt.enabled && !isFromUs(p) && mqtt)
	mqtt->onSend(*p_encrypted, *p, p->channel);

We need to find a way to allow PKI-encrypted packets to be uplinked to the "PKI" topic.

[JohnathonMohr]

can it be that only DMs not directed to the MQTT node are not being forwarded?

No, I was directing my DMs to the MQTT node. I could see in the node-hosted website that it was received, but it wasn't published to the JSON MQTT topic.

[GUVWAF]

Okay, that would also make more sense, because you would need to have the private key to be able to decrypt it anyway (although you could still see the unencrypted metadata).

However, I don't see why it wouldn't uplink it then, because indeed it first decrypts as before. Can you maybe catch serial logs when you receive the packet on the MQTT node?

[JohnathonMohr]

Can you maybe catch serial logs when you receive the packet on the MQTT node?

I'll try to do this a little later.

[JohnathonMohr]

@jp-bennett 2.5.6 doesn't seem to have fixed this. I did see an interesting serial log when receiving the message though (on both 2.5.4 and 2.5.6):
INFO | ??:??:?? 149 [Router] MQTT onSend - Not forwarding packet due to DontMqttMeBro flag

[jp-bennett]

@jp-bennett 2.5.6 doesn't seem to have fixed this. I did see an interesting serial log when receiving the message though (on both 2.5.4 and 2.5.6):
INFO | ??:??:?? 149 [Router] MQTT onSend - Not forwarding packet due to DontMqttMeBro flag

On the sending node, make sure the OkToMQTT setting is set to true.

[JohnathonMohr]

On the sending node, make sure the OkToMQTT setting is set to true.

@jp-bennett where do I find that setting? Is it newly introduced? I've used this (and other) nodes in the past to test MQTT delivery of DMs and they've worked fine. If it's something new I have to enable now (or it got disabled upgrading to 2.5.4?) how do I do it?

[jp-bennett]

@JohnathonMohr it's new in 2.5 firmwares. It's part of the LoRa config.

[JohnathonMohr]

Ok, wow, that did the trick - I had to enable "Ok to MQTT" on the sending node. 😳

So now I feel like this is even more complicated - I understand the intent of this config, but it's a major break in what was supported previously.

As I mentioned above, sending messages to a local (private) MQTT instance and configuring automations around that is a documented means of integration, but that no longer works if it's entirely controlled by the sending nodes.

I have my node connected to WiFi (so Bluetooth doesn't work) so automations that rely on MQTT to send notifications of received messages are the only means of any message notifications.

All this basically means that messages sent to most WiFi-connected nodes are lost unless the node web site is monitored frequently. 😕

[GUVWAF]

Not really a major break; it only holds for the default PSK, and it will only check this for nodes that have updated to 2.5. So, yes, the sending nodes control this (because it's also their data), but they can just enable the setting and it's only necessary after updating to >=2.5.

Nonetheless, the issue is fixed.

[fti7]

This issue is not really fixed. It breaks every Automation done via MQTT.
I've some MQTT -> NodeRed/Grafan Setup for Node Telemetry Data etc.
Now its broken unless the Nodes enable the Flag (Disabled by Default)

I understand that this Flag might be very useful for Bridges via MQTT to Opt-In.
But not for usecases where MQTT is used instead of normal API Access because its easier to use in 3rd Party Systems.

We need some "Push everything to MQTT" Flag for the Gateway Nodes.

[thebentern]

We need some "Push everything to MQTT" Flag for the Gateway Nodes.

That really defeats the purpose if anyone can just say "Screw your intentions! I'm uplinking your packets to MQTT!"

[fti7]

We need some "Push everything to MQTT" Flag for the Gateway Nodes.

That really defeats the purpose if anyone can just say "Screw your intentions! I'm uplinking your packets to MQTT!"

We should treat MQTT and the normal API equally as an Interface to Meshtastic. If not, people will build API->MQTT Bridges as external components, which only adds complexity, reinventing the Wheel and doesnt help.

What you really want is an "OkToBridge" Flag for Sending Nodes.
It could be added to the Message in MQTT and then ignored on Devices which recieved it via MQTT.

[JohnathonMohr]

That really defeats the purpose if anyone can just say "Screw your intentions! I'm uplinking your packets to MQTT!"

Well, yes, but on the flip side, this setting allows the sender to say "Screw your intentions, I only want notifications to work on nodes that are app-connected over Bluetooth".

The problem with the current implementation is there is no middle-ground: it's all or nothing for MQTT. But the use cases for MQTT (home automations for e.g.) are much broader than what this setting seems to be designed to protect against.

I feel like ultimately this needs a better design. But at the very least, can the setting be repurposed to "allow publishing to the public MQTT server"? This would still allow for home automation setups and not leak to the public.

[fti7]

Flags to control specific behavior is better than massacre a complete Interface Type.
Another pragmatic solution would be to enable OkToMQTT on 2.5er Devices by Default and let Users opt-out. But i guess that might break other Use-Cases in the Future etc.

[garthvh]

Users need to explicitly opt in, and their choice needs to be respected.

[jp-bennett]

We added a localhost exception. Currently considering a local network exception, for exactly this sort of concern. It's a delicate balancing act.

[JohnathonMohr]

Thanks @jp-bennett. Definitely looking forward to the local network exception - since I'm using ESP32-based devices, I can't run MQTT locally on any of them.