Marlin Binary Protocol Mark II

[p3p]

This is not final, there will be major changes before it is stable

Marlin Binary Protocol

This is an update to the BinaryFileTransfer feature, generalising the transport layer and improving reliability, still more work to do but more useful than the current feature. Protocol feedback and testing required.
The transport layer overhead is 10 bytes.

31 - 28	27 - 24	23 - 20	19 - 16	15 - 12	11 - 8	7 - 4	3 - 0
Start Token (0xB5AD)				Sync Number		Protocol ID	Packet Type
Payload Length				Header Checksum

31 - 28	27 - 24	23 - 20	19 - 16	15 - 12	11 - 8	7 - 4	3 - 0
Payload Data		...
...				Packet Checksum

Start Token : 0xB5AD
Sync Number : Previous Packet Sync Number + 1
Protocol ID : 0 reserved for control packets
Packet Type : Packet type passed to App Protocol
Payload Length: Lengths of the optional Packet Payload
Header Checksum : 16bit Fletchers checksum of Header Bytes excluding Start Token
optional
Packet Payload: Payload Length bytes of data
Packet Checksum: 16bit Fletchers checksum of packet excluding Start Token

Description

The packets are designed for guaranteed data integrity and fast protocol recovery if the header is corrupted, packet stream is synchronous not allowing out of order transmission.

Acknowledging

All packets are acked (apart from the SYNC control packet)
ok<sync>

Connection Control <Protocol ID: 0>

Packets

SYNC <Packet Type: 1>

returns: ss<SYNC>,<BUFFER_SIZE>,<VERSION_MAJOR>.<VERSION_MINOR>.<VERSION_PATCH>
Syncronise Host to client and get connection info

CLOSE <Packet Type: 2>

returns: standard ok<SYNC>
Inform the client host is disconnecting, used to switch client back to ASCII

Binary File Transfer <Protocol ID: 1>

Binary File Transfer that supports stream compression

Packets

QUERY <Packet Type: 0>

returns: PFT:version:<VERSION_MAJOR>.<VERSION_MINOR>.<VERSION_PATCH>:compresion:<COMPRESSION_ALGO(<PARAMETERS,>)>
Query if the client has the protocol enabled and what compression is supported

OPEN <Packet Type: 1>

Payload:

31 - 28	27 - 24	23 - 20	19 - 16	15 - 12	11 - 8	7 - 4	3 - 0
Enable Compression		Dummy Transfer		Filename
...						NULL

returns:
PFT:success\n : File opened and ready for write
PFT:fail\n : The client couldn't open the file
PFT:busy\n : File already open for write?

CLOSE <Packet Type: 2>

Close File
returns:
PFT:success\n : Buffer flushed and file closed
PFT:ioerror\n : Client storage device failure
PFT:invalid\n : No file open

WRITE <Packet Type: 3>

Payload is upto BUFFER_SIZE bytes.
returns:
PFT:ioerror\n : Client storage device failure
PFT:invalid\n : No file open

ABORT <Packet Type: 4>

Close file and remove it
returns:
PFT:success\n : Transfer aborted

Peformance

These tests use the same sdcard, file transfer rates will depend on it (one of my cards hates Marlin and struggled for 3KiB/s writes)

All transfers use 512byte payload size.

USB -> Hardware UART converter 8N1 115200 baud (11.52 kB/s theoretical)

throughput test: 10.30KiB/s
gcode file transfer: 10.25KiB/s
compressed throughput test: 10.23KiB/s (24KiB/s uncompressed)
compressed gcode file transfer: 8.78KiB/s (21KiB/s uncompressed)

CDC Serial

throughput test: 181.89KiB/s
gcode file transfer: 105.83KiB/s
compressed throughput test: 135.63KiB/s (320.44KiB/s uncompressed)
compressed gcode file transfer: 47.63KiB/s (112.53KiB/s uncompressed)

Future work

The command response codes are a weak link currently only the host transmits using the transport protocol to simplify client implementation, ack packet losses are compensated for, but other replies may be lost.
Add a compressed gcode streaming protocol.
Fixing everything and cleaning up.

Example Host

usage: transfer.py [-h] [-p PORT] [-b BAUD] [-d BLOCKSIZE] [-r] [-t] [-c]
                   [-e ERROR_RATIO]
                   source [destination]

Send files over a serial port to Marlin

positional arguments:
  source                source path
  destination           destination path

optional arguments:
  -h, --help            show this help message and exit
  -p PORT, --port PORT  serial port to use
  -b BAUD, --baud BAUD  baud rate of serial connection
  -d BLOCKSIZE, --blocksize BLOCKSIZE
                        defaults to autodetect
  -r, --reset           Reset after transfer (firmware flash)
  -t, --test            Benchmark the serial link without storing the file
  -c, --compression     Enable compression
  -e ERROR_RATIO, --error_ratio ERROR_RATIO
                        Simulated corruption ratio

transfer.py.zip
requires pip package: https://github.com/p3p/pyheatshrink/releases/download/0.3.3/pyheatshrink-pip.zip

[rafaljot]

Great, it is little bit more complicated than previous version but looks promising.
How to configure Marlin to try this?
#define BINARY_FILE_TRANSFER
Anything else?

[rafaljot]

I had to add in transfer.py

import time

and time.sleep(3)
aTmega based boards reset after connect.

try:
    protocol = MarlinBinaryProtocol.Protocol(args.port, args.baud, args.blocksize, float(args.error_ratio), int(args.timeout))
    #echologger = MarlinBinaryProtocol.EchoProtocol(protocol)
    time.sleep(3)
    protocol.send_ascii("M117 Upload start\n")
    protocol.connect()

[p3p]

Glad you got it working, there has been little interest in this so testing by other people has been minimal (practically non existent?), I'm not sure what performance you will get on an 8bit platform it was primarily designed to take advantage of the more capable 32bit mcus and the native lossless usb cdc connections.

I was hoping to get the 'finalised' protocol merged before 2.0 but unfortunately didn't have the time to clean it up sufficiently, just so you know it is very incompatible with this version.

[rafaljot]

I'm trying to implement it for ESP3D. ESP3D is nice but useless without upload in my opinion...
I have unused 8bit Trigorilla board in drawer. It is enough for testing.

[rafaljot]

Unfortunately I' can't compile last Marlin in PlatformIO :
error:

in file included from Marlin/src/feature/binary_protocol.cpp:28:0:
Marlin/src/feature/binary_protocol.h:233:33: error: 'BinaryStream::Packet::Header::HEADER_TOKEN' may not be static because it is a member of a union
       static constexpr uint16_t HEADER_TOKEN = 0xB5AD;
                                 ^
Compiling .pio/build/DUE_USB/src/src/feature/caselight.cpp.o
*** [.pio/build/DUE_USB/src/src/feature/binary_protocol.cpp.o] Error 1

[trippwill]

I created a client python package based on @p3p 's sample code. It's on

• PyPi here: https://pypi.org/project/marlin-binary-protocol/
• GitHub here: https://github.com/CharlesWillis3/marlin-binary-protocol

It should work for both python 2 and python 3.

I'm using the package in an OctoPrint plugin I'm working on.

@p3p this is your code, so please let me know if you'd like to take ownership or want changes to the license, copyright, etc..

[p3p]

Hey Charles, The protocol has undergone major changes since this was merged that's why I added the warning at the top, I've recently started cleaning it up for the 'final' revision, I'm not sure it would be a good idea to have an Octoprint plugin designed for this alpha version, but then again as long as the version is detectable it may not be the end of the world.

If you would like to discuss the current version I'm on the Marlin Discord.

[ETE-Design]

@p3p Will it be possible to upload G-Code faster also? Would be great if this feature could be "Final" then @luc-github would mabye use it to transfer G-Code faster with ESP3D...
@rafaljot Did you manage to get it work with ESP3D?

[filimonic]

Currently there is packet loss problem while transferring data. Board silently swallows each 3rd-5th packet and does not respond anything, resulting timeout. Retransmission helps, but at read timeout = 100ms, and 20%-30% loss, it gives huge amount of time spent waiting for board response and timeout then.

I'm trying to transmit to MKS Robin nano v2 using USART1 (Wi-Fi socket) as SERIAL_2 at 250 Kbaud using 2 wires (RX-TX) and USB-TTL based on CH340G chip.

Any ideas ?

[sjasonsmith]

This is not a good place to post to get attention to your issue. You should probably document your issue very well in a bug report. Be sure to include everything requested in the template, as well as how you identified packet loss, a file you are trying to send, etc.