Incomplete SysEx packets not supported?

[samspencer5991]

Not quite sure if this is an issue with the library or the RTP protocol. But I'm using MIDI as the transport protocol for configuration data for some devices, and the JSON data is serialised with ArduinoJSON, which because of the size of the data, parses out chunks of the data rather than a completed array. Therefor when sending via SysEx MIDI, it sends partial packets, only with the F0 and F7 framing bytes for the complete packet, not each chunk.

This approach avoids extra overhead and complexity of having to add the framing into every transfer (often 10K+ bytes in size). For other transports such as USB device, Serial, BLE, this is working well, but it seems that packets which are not fully framed are not sent via RTP. Could you advise if this is an underlying limitation of RTP which I will need to work around (incomplete packets not possible, and I will need to change my approach), or a library parsing limitation (which could be fixed in the library parsing)?

[lathoub]

Hi @samspencer5991

Have you looked at example AVR_SysEx? (Receiving large SysEx, is that what you mean? Or is it sending large SysEx?)

That contains code to check the start SysEx, middle parts and an ending part (if SysEx larger than buffer size)

Arduino-AppleMIDI-Library/examples/AVR_SysEx/AVR_SysEx.ino

Lines 100 to 110 in e0abb86

	char getSysExStatus(const byte* data, uint16_t length)
	{
	if (data[0] == 0xF0 && data[length - 1] == 0xF7)
	return 'F'; // Full SysEx Command
	else if (data[0] == 0xF0 && data[length - 1] != 0xF7)
	return 'S'; // Start of SysEx-Segment
	else if (data[0] != 0xF0 && data[length - 1] != 0xF7)
	return 'M'; // Middle of SysEx-Segment
	else
	return 'E'; // End of SysEx-Segment
	}

[samspencer5991]

It's sending which is causing the issue. Reception works fine. Within the same function calls I've tested a dummy array which I've constructed and that works both with the framing, and without (directing the SysEx send function accordingly).

The problem is not that I'm trying to send a packet larger than any internal buffer, it is sending the SysEx in chunks which don't all have the framing bytes (only the first and last obviously). I should note this approach has worked well across multiple transports including your own BLE library, so I wanted to check whether it's something unavoidable within the RTP protocol, or I may have missed something in the source files.

[lathoub]

Have you tried to see what the underlying TCP packets look like with WireShark?

[samspencer5991]

Yep, I've just taken a look. So what is interesting is that I'm using exactly the same transmission for USB Device, BLE, and RTP, and only changing the transmitting transport to test. The issue appears to be with longer packets and they become malformed. The transmitter breaks the SysEx packet up into a header (F0 3 address bytes, a command byte, F7), and then a terminating character (F0 7E F7) with the data payload in between (F0 data.... F7). The header and terminating packet are transmitted correctly, but the larger packet becomes malformed with extra sysex framing bytes inserted:

You can see in the first image the incomplete JSON packet, with a SysEx start tacked on to the end, and then the next packet begins with a SysEx stop byte. These do not appear on other transport layers.

[lathoub]

Thanks for the WireShark packets.

The transmitter breaks the SysEx packet up into a header (F0 3 address bytes, a command byte, F7), and then a terminating character (F0 7E F7) with the data payload in between (F0 data.... F7).

So in effect you are transmitting 3 SysEx messages in one go?

Also, The transmitter that is your software, right?

(What hardware are you using?)

[samspencer5991]

Yep that's right. They are sent as individual calls to the SysEx send function though, not lumped into a single packet containing the headers.

It's a custom ESP32-S3 board (16MF and 2MR)

[samspencer5991]

And yep when I was referring to the transmitter, I simply meant the packaging and preparing section of firmware for packet delivery.