Add (optional) timestamps to (S)SerialReceived messages

[kobuki]

Have you looked for this feature in other issues and in the wiki?
Yes.

Is your feature request related to a problem? Please describe.
No.

Describe the solution you'd like
For certain protocols I'm using it would be very useful to add timestamps to remote serial messages coming in from SerialReceived/SSerialReceived. Sometimes network congestion or intermittent WiFi delays cause weird discrepancies in message processing that can be addressed when an original timestamp for message birth is available.

It could be made optional, and the ideal would be able to choose between: no timestamps, timestamps using ISO format or timestamps using epoch time (single integer number). The timestamp could be easily added as an extension to the JSON message already sent with serial text events.

Describe alternatives you've considered
I tried rules, but it has multiple problems: it forces all rule parameters uppercase (including the original serial message), and has issues with space and certain special characters like ':'. Most notably it only provides text forced to uppercase until the first space encountered.

Additional context
N/A

[Jason2866]

Mhh, i could not reproduce your issue. I use SerialReceive, i had never transmitting issues.
Tasmota has only a small serial buffer, so you will loose infos if not delivered at once.
If your wifi is bad you cant solve missed packets issues with timestamps.
If i where you i would try to solve network problems

[kobuki]

Well, it's not really possible to solve network problems I have no control over. Think of general internet congestion here. I have pretty good WiFi APs, they're not the issue. Sometimes I do everything within my possibilities otherwise.

It's not just the possible delays, but also I want a self-contained message with timestamps - I think it would be a useful addition for other users as well, hence the feature request. I want to add the processed message to a database with as precise timestamp as possible (within a second at least).

[ascillato2]

Closing this issue as it has been answered.

---

Support Information (Guide)

See Wiki for more information.
See FAQ for common questions/answers and links if none of your question is in the list.
See Chat for more user experience.
See Community for forum.
See Code of Conduct

[kobuki]

@ascillato2: Why did you close it? It hasn't been addressed yet. Is it possible to add this feature? What merits do you require besides being useful and a concrete user requesting it for concrete reasons?

[kobuki]

OK, let me put this another way. I requested this feature because I need it, not to create extra work for FOSS maintainers. If I create a PR, would you be willing to merge it? If so, what configuration method would you suggest implementing for controlling the requested behavior? Other than that, it seems fairly simple to implement.

[Jason2866]

The decision to merge a PR is @arendst
There is a chance if code size increase is minimal and the PR is usefull for others too.
Imho your arguments are not powerfull enough to convience arendst to merge.

[kobuki]

I'm adding this function to my private repo anyway, but it would be a waste not to include upstream. Well, let's hear what the boss says.

[ascillato]

Hi,

I have just connected for testing TX to RX on a device with Tasmota:

11:14:51 CMD: serialsend2 hi
11:14:51 MQT: stat/cocina/RESULT = {"SerialSend":"Done"}
11:14:51 MQT: tele/cocina/RESULT = {"SerialReceived":"hi"}

And I can see the time in both, Tasmota console and MQTT broker. So, then from the broker it can be used/stored in a database with that time of reception.

As Jason explained, if having a wifi issue or congestion, the serial message will be lost, not delayed. For that case, you will need extra logic (using rules) to store the data if there is no mqtt connection and resend it when mqtt is online again.

So, sorry, but IMHO seems there is no need for this feature and the extra code added.

Anyway, as we don't know how is your setup or what is exactly you want achieve, it would be useful if you can explain exactly how is your setup, devices, etc.

[kobuki]

There are no guarantees in MQTT for message delivery time, so they can be arbitrarily delayed for whatever technical or other reason. In cases the delay does occur, we don't know when they were really sent. In general, that's what I want to fix by adding time metadada. Intermittent WiFi delays and network congestion are just common examples of technical delays. The one convincing reason against this is how Tasmota (or its TCP stack) behaves (messages are dropped, not retransmitted). You might say I'm overthinking it a bit since with good, working hardware and WiFi it's a very rare issue, but while MQTT is a good fit for small devices and IOT, it has its drawbacks. Like the inability to provide (optional) timestamps natively.

My use case now is simply transmitting power and sensor data from my smart energy meter via MQTT, potentially over the internet (with TLS). The data is coming in serial, one line per message "packet".

Note that it's not possible to use rules to store incoming serial data as it is transformed for use in the rule. I think if I wanted to seriously address the issue, I'd need to add a simple pending queue storage, based on SPIFFS or some other extended storage available.

[Jason2866]

My2cents if you need a very reliable solution go for a expensive ready made one.
My experience with Tasmota, wifi and my own hosted mqtt broker. I never had delays greater 100ms...
My IoT devices at home (i have some ;-) ) just react nearly in real time. For example devices triggered via a sensor (wifi and mqtt) and doing a action (via rule in my HomeAutomation system OpenHab) transfered via wifi and mqtt are acting <20 ms. No noticable delay for humans when light is switched on...
You will get the greatest delays in using public IoT services.

[kobuki]

Yes, for most of the time my experience is the same. It's my smart home project, so no big promises are needed and if I needed ultra-reliable connection I'd pull a wire from end to end inside the house, really.

But I've seen unusual random delays with gate openers, for instance. Pushing a button in Home Assistant sends an MQTT message to a Sonoff RF Bridge. Sometimes it doesn't seem to react at all, but then all of a sudden the operation begins - gate is opening, something like 5..10 secs later. I've seen it sporadically and haven't dug too deep. The delay can be at many places, and not necessarily in a Tasmota-equipped device. But these can happen. This was only meant as an example about my concerns. And yes, my other concern is when using cloud MQTT services...

Well, let's put the issue to rest for now - thanks for all the feedback and thoughts given so far. If/when a related Tasmota issue comes up, I'll be back.

[ascillato]

@kobuki

Hi,

Your request has been added by Theo. Please, test it if this suits better to your setup. Thanks.

[kobuki]

I realized I removed a whole message instead of just the edit I made. So thanks again for the heads-up and for including serial in the timestamps feature.

I've flashed it to the planned target device, and yes, Time 3 fits my purposes just fine. I made a comment in the commit, but I'd also note here that I think for Time 3 the Epoch JSON name should stay Epoch instead of swapping it with Time when present alone in the output JSON.

EDIT: it got a NAK from @arendst - case closed.