LoRa Communication can't handle large packets

[maxchu765]

Describe the bug
LoRa communication will fail when trying to transmit data from more than 3 sensors. The packet is too large for the receive to parse.

Hardware in Use
Feather M0 LoRa
Hypnos Board
I2C Multiplexer

I2C and Teros sensors

To Reproduce
Transmit data with 3 or more I2C sensors over LoRa. The receiver will show a "Failed to parse MsgPack" error.

Code
https://github.com/OPEnSLab-OSU/SitkaNet-2/blob/master/Node/Node.ino

Config
https://github.com/OPEnSLab-OSU/SitkaNet-2/blob/master/Node/config.h

[Cogswatch]

This is almost certainly due to the limits on packet size imposed by Radiohead's (the library we use for LoRa) RF95 implementation which maxes out at 251 bytes. Some possible solutions include:

• abbreviating the packet tags
• micromanaging the types to ensure there isn't wasteful casting occurring (float over int)
• segmenting the payload across multiple transmissions

For the time being, we are going to have to recommend that users do their best to check that the data they are shipping out is the minimum viable if packet size is proving to be the limiting factor while our team figures out the best path forward and works to provide a fix.

[prototypicalpro]

This problem will be addressed specifically in SitkaNet code by using an external process to compress the JSON into the minimum values needed before transmission. I created a pull request that allows the sketch to send/receive raw payloads for this purpose (#73), and will be working with @maxchu765 on SitkaNet individually. This will not address this problem in the general case, and a general solution is something Loom will need to consider in the future.

[MrKangs]

Currently, there is an example code called Transmit_Raw and Receive_Blocking_Raw in the LoRa example that is based on the JSON translation function for all users for different projects. You can find this code in the SerialSensor Branch for now. We need to test it before we close the issue. It should be good. There are many comments that guide what to do so that people won't get lost with it. If there any help, they are going to make an issue on the GitHub page.

Updates!
The main issue has been solved. Therefore, you don't need to follow these code examples to do large package radio communication. However, these examples will not be deleted because it is a good alternative way to solve if there is an issue in this method that I created.

[MrKangs]

It will now able to send the larger package size over Radio communication in the Commplat branch. Currently, there are some parts that need to be added into the code to make the performance better such as two nodes into one hub sending with larger package size or with batch sending, but the main problem of sending a larger package will be available. The maximum size is now can send up to 2048 bytes rather than 251 bytes. This will be updated on master in the next coming month.

[MrKangs]

Updates for the changes that will happen for this issue and solve the main problem.

Current Situation

1. While in the master branch sends one package, this branch(Commplat) will automatically send multiple packages by the number of modules + 1, even though it is a small package that is able to send it by one send due to the situation 3.
2. The send package size and the receive package size are not the same: it shows that receive package size is bigger than sending size.
3. There is a vague limitation for each radio communication. According to the Adafruit with LoRa description, it says that it has a limit of 252 bytes per transaction, but when I observe, it changes a lot, which I am not sure where to look at it. For example, sometimes sending 20 modules with one data value set works, but sometimes not even 13 modules or less doesn't work while both JSON document Memory Usage and JSON document size are the same. Might be something with the Buffer size?
4. Batch sending works fine with sending large package size. Batch_Recieve doesn't influence with this current code.

If you are planning to send a large size package over LoRa or any kind of radio communication, please use this branch rather than the master branch because this will solve this current issue.

Future plans

1. Like the previous comment, I was working on the batch with large package transmission. However, I encounter an issue like issue number 2 and 3. As for now, I am planning to work on that part.
2. For those who are the people who are willing to have two nodes radio communication to one hub will not happen in the next month or so because this a totally different issue to handle. If you are interested in it, please wait for it. It will be coming with the name SleepyNet. You can checkout here.
3. As for a temporary solution for having multiple nodes to one hub, you can set that up with correct timing between each board transmission. If you need help with that, please contact one of the Loom staff members or post something here.
4. Once situation 2 and 3 is complete, testing will be in place for the remaining time unless an issue is found. If anyone is willing to test it, please let me know and give me some feedback on it.

Developer Notes

1. What is the correct way to determine when to send multiple packages size verse one package?
   • Find a way to check LoRa Buffer size rather than JSON Document Memory Usage or Memory Pool Size
   • If that case doesn't solve, convert back to the old calculation that is able to filter better
   • Gather more ideas to solve this issue
2. How can we make the JSON document size the same for both send and receive ends?
   • Currently, the sending size is smaller than the receive size: it shows about 200 ~ 400 bytes difference which can delete data during the transmission.
   • Maybe having different JSON documents might solve it, but that doesn't seem that helpful.
     • For example, in the code, we are using mergeJSON, but make more like sendJSON and receiceJSON so that it will remove waste. I hope this will work so that it will be simple to fix.
   • If that doesn't work, then find a new way to handle this entire issue.
     • Currently, we are copying and pasting to a new JSON document that might increase the chance of memory problems. Therefore, finding a new way could be better with memory management,

If you have any ideas on how to answer these questions, please let me know! That would be great!

[MrKangs]

Updates
In the last developer notes, I accomplish for finding the solution and fixing it properly for the first one. However, the second one will not be fixed for now. And here is why.

While COM7 is sending, COM10 is receiving. You can see a comparison between the two that the message pack size is different when you send verses when you receive by 30 bytes even though they are the same message data. I believe that it happens it increases during radio communication. Because of that, I can't change that part at all. Therefore, I will let the user know that they need to reduce the size of the package, but it has increased from 251 bytes up to 2000 bytes of radio communication, so I believe that it is better in what way. This will be as it is for now, but once we find a new way, this issue will be reopened. You can find the code in Commplat until September 25th, then it will be in master.

Developer Notes
If anyone has an idea where to fix it, please open the issue and send a comment here. I will try to look at it and discuss it together if that will work or not.

[MrKangs]

If you take a look at here, even with the general send and receive for the basic example of Loom shows a difference between JSON size that the receiving end gains more memory usage. We need to know what is going on between each radio communication.

[prototypicalpro]

The problem occurs when you measure the size of the received msgpack packet using this strlen call. Strlen is designed to iterate over a string until a null terminator (0x00) is found—a msgpack payload is not a string, however, and as a result it is perfectly valid behavior for a msgpack payload to have 0x00 values all over the place. Because of this, attempting to measure the msgpack with strlen sometimes results in the correct value, but usually results in a value significantly less than the actual length. This problem is demonstrated below using msgpack.org:

To fix this issue, you'll need to find an alternate method of measuring the size of the received msgpack. I would suggest deserializing it and then measuring the newly deserialized payload, but I'm sure there are other options.

[MrKangs]

After disgusting with other team members, we decided to let the end-user to limit the size of the JSON package sending and receiving. However, as I said before, the size has improved than before this was integrated. This will be merged into master soon!

[MrKangs]

After disgusting with other team members, we decided to let the end-user to limit the size of the JSON package sending and receiving. However, as I said before, the size has improved than before this was integrated. This will be merged into master soon!

This is now in the master branch,