Sending video over RNS

[Rishi-jha]

Hi,
I was trying the Reticulum network stack to stream video from one device to another, but getting some problems doing so, I have modified the Buffer Example, below are client and server,

def client_loop():
    global server_link

    # Wait for the link to become active
    while not server_link:
        time.sleep(0.1)

    should_quit = False
    while not should_quit:
        try:
            print("> ", end=" ")
            text = input()
            print("Reading file.....")

            # Check if we should quit the example
            if text == "quit" or text == "q" or text == "exit":
                should_quit = True
                server_link.teardown()
            else:
                vid = cv2.VideoCapture(0)
                while (vid.isOpened()):
                    img, frame = vid.read()
                    frame = imutils.resize(frame, width=320)
                    data = pickle.dumps(frame)
                    message = struct.pack("Q", len(data)) + data
                    cv2.imshow('TRANSMITTING VIDEO', frame)
                    buffer.write(message)
                    buffer.flush()


        except Exception as e:
            RNS.log("Error while sending data over the link buffer: " + str(e))
            should_quit = True
            server_link.teardown()

def server_buffer_ready(ready_bytes: int):
    """
    Callback from buffer when buffer has data available

    :param ready_bytes: The number of bytes ready to read
    """
    global latest_buffer
    data = latest_buffer.read(ready_bytes)
    payload_size = struct.calcsize("Q")
    packed_msg_size = data[:payload_size]
    data = data[payload_size:]
    msg_size = struct.unpack("Q", packed_msg_size)[0]
    frame_data = data[:msg_size]
    data = data[msg_size:]
    frame = pickle.loads(frame_data)
    cv2.imshow("Showing", frame)
    # stream.write(data)
    RNS.log("Received data over the buffer: " + "Hello")

Getting Error like:

[2023-05-11 15:07:50] [Notice] Error calling RawChannelReader(0) callback: invalid load key, '\xa6'.
[2023-05-11 15:07:50] [Notice] Error calling RawChannelReader(0) callback: FRAME length exceeds system's maximum of 9223372036854775807 bytes

First thing is, Is Video/Audio streaming supported over this stack?

If so, then what would be the way to transmit large streams of data over the stack?

I tried sending a static file, (opening file, fetching bytes and sending over RNS chunk by chunk.) and it works fine,

Help is appreciated.

[faragher]

While the system technically supports any data, that's a lot of data, and the network is not designed to be low latency or high speed.

Looking at your error messages, I think there's something fundamentally wrong. The maximum transmission per packet is just under 500 bytes, and your error message lists an error due to exceeding 9,223,372,036,854,775,807 bytes, or 9,000 petabytes.

I was interested in an RNode connected camera since Esperif disabled the API calls I need for transmit-only WiFi, but the problem becomes bandwidth. I can expect around 3,000 bits per second over the radio, and I need a lower duty cycle for legal reasons. So at about 1,000 bits per second I can't expect more than a coupe frames per second, even at low resolutions. I assume a similar application here, since you're looking at 320 pixels.

I'm not following the code, I'm not terribly familiar with the cv2 library. I suggest making sure you get raw bytes, split them into like 450 byte packets, tag each one with an index and total number of packets, then reassemble them on the other end.

But first you need to figure out why the frame is listed as the maximum value of a 64 bit signed integer.

[markqvist]

Also, this is a really bad approach:

data = pickle.dumps(frame)
message = struct.pack("Q", len(data)) + data
buffer.write(message)

Don't do this. Seriously.

You are pickling (serialising) raw binary frame data, then sending it down a buffer... This is just several layers of redundancy and unneeded complexity to send an image frame. This is exactly the kind of stuff I feared people would start doing if the Buffer API was added... :S

A much more efficient and Reticulum-centric way to do this is to let the client query the video server every time it wants a new frame. When the client asks for a frame, the video server sends it using a Reticulum Resource. This is the most efficient, and correct, way. No need to manually pickle everything. Reticulum does all of that stuff for you ;) That is the whole point! Just pass the binary frame data to the Resource constructor and you are done.

I repeat: In Reticulum, NEVER just shove data down a path, that the sender did not ask for.

DON'T use a Buffer unless you ABSOLUTELY know you positively HAVE TO. It is a construction that seems familiar and "nice" for people coming from TCP. But, Reticulum has much better ways of doing this, and a Buffer is not efficient for doing stuff like this, especially not over low-bandwidth mediums.

Use the Resource and/or Request APIs. They take care of super-efficient sequencing, reliable transfer, compression and everything else for you, and does so in a bandwidth friendly way :)