Reticulum Application-Level Protocols

[4c3e]

Hello all! As Reticulum development continues (a big thank you to @markqvist!) I wanted to open up a discussion about an internet protocol, and other application-level protocols that can be designed on top of Reticulum. LXMF and Nomadnet already seem to implement a sort of internet protocol that allows for a node's pages to be requested by clients.

What I'm interested in is a Gemini-like protocol that creates connections using Reticulum's Link API. Gemini is enjoyable over the traditional internet, but it's simplicity and lightweight spec could actually be very useful on a high latency, low bandwidth systems, like Reticulum over packet radio, compared to traditional http and html.

Another application-level protocol I'm interested in is a telnet/ssh virtual terminal tool that is also built over Reticulum Links.

Currently I don't have much free time, but I'll be trying to work on a basic Reticulum Internet Protocol spec sheet here:
https://github.com/4c3e/rip-spec

Edit: HTTP/3 is adding an interesting streaming protocol that works over a UDP connection to multiplex different streams over a single connection. Certainly overkill for now, but a similar protocol could possibly work to enable multiple "Virtual Links" over a single underlying RNS Link.

[markqvist]

Thanks for starting this discussion @4c3e. I am not familiar with Gemini, but I just read your spec draft. A genereal purpose protocol like this would be very useful indeed. Since I am really busy right now, I am going to read it all again in more detail in a day or two, and also read up on Gemini and HTTP/3 to get a better basis for discussing this.

Am I correctly assuming that you intend to use the Request API as a carrier for the request/response flow?

Also, a remote shell utility is very high on my list of priorities :) It will probably be included as a bundled utility in one of the next releases :)

Link multiplexing is also a very interesting subject. For me the area of efficiently addressing multiple destinations over a single virtual link is especially interesting, since it will enable much better distributed communications systems. I'm working on the theory and methods for doing this with Reticulum, but it is probably a bit off still, since other things are more important to have working right now.

[4c3e]

Here's a mirror for the gemtext spec: https://gitlab.com/gemini-specification/gemini-text/-/blob/master/specification.gmi

Micron looks very interesting, is it utf-8 based? And can you write micron documents using normal text editors? I have been interested in extending the Gemini spec to allow for more formatting options, which seems closer to what micron is than gemini. Specifically I've been looking into adding forms and inline links on top of the Gemini spec.

Also, the browser/server minimal demo is working if anyone wants to check it out, note that the index.gem file has a hard-coded link that will need to be changed to the server's destination hash.

Edit: Just checked out the micron section of nomadnet's guides, really cool!

[4c3e]

Also, when writing request handlers for RNS Links, what is the recommended way to return static files? Currently I am just reading text files into one string variable, then returning that string variable, which runs into issues with serving non-text files.

[4c3e]

After messing around with the LXMF page system on Nomadnet, I think the best direction to go is to simply "open up" the protocols that nomadnet is using under the hood, instead of building a whole separate system. By "open up" I basically mean documenting what transactions happen under the hood for Nomadnet to request and display a page, so alternative stand-alone browsers and servers could be built and all talk to each other.

A side note, I think defining a standard message format would be useful as well, to allow for different messaging applications to communicate with each other as well. I'm pretty sure nomadnet conversations just use "raw" LXMF packets so I think there isn't much to this(?)

[4c3e]

@markqvist et al, I updated the RIP spec sheet to take into account the Request/Response API and Micron, would love some comments if anyone has any. The main differences between RIP and how pages are handled in Nomadnet (to my knowledge) is:

• URL (rip://<destination>/path/goes/here vs <destination>:/path/goes/here)
• RIP Response packets are msgpacked [header, body]. The header value allows clients to get a more granular response status and MIME type of the response body, and leaves extra room for custom protocols to be built using it.

I built a small class to make synchronous requests in this method: https://github.com/4c3e/riprequest

[markqvist]

This is starting to look like something :) Just taking a day off today, but will return to this shortly. I have a lot of comments, thoughts and ideas, and I think we can shape this into something very useful. Thanks for bringing up this discussion.

[markqvist]

I regret that I have not yet gotten back to you on this. I have been exceedingly busy with work on other parts of the project. This will be short and not go into full details as I would have liked, so it is just a few comments.

I have been known to be quite pedantic at times, so don't take this as critique of the idea or spec itself. It is aimed at the wording and how the naming communicates purpose, which I think is extremely important.

I think the naming of this specification could potentially be misleading. The closest analogue to this spec in the "world of established protocols" would be HTTP, HyperText Transfer Protocol. The proposed spec seeks to define a structure for page/text transfer over Reticulum, providing the basic mechanism for what could be analogous to the world wide web. Yet it is named an internet protocol. I find this inaccurate.

Internet != world wide web != page transfer protocols. One could say that Reticulum itself, at its base, is an alternative inter-net protocol. It can be used to build both isolated, separate network, and connect these networks together arbitrarily. It allows both networking and inter-networking.

To illustrate what this spec is really seeking to provide, I think it is essential to update the naming to reflect it.

[markqvist]

I like the simplicity of it, but I also get the feeling that it is a bit strange to add the headers a text line to the response body. What if the response body is not text? What if it is an ELF binary, or JPG image? Or what if the response is itself encrypted with a key external to reticulum. In such cases, parsing out the header line and reconstructing the binary blob of response data seems a bit cumbersome. Maybe it would be better to have the header information in a variable separate from the main response body? The request/response API already supports this, since the response can be any data type, including a dict or an array, which is then automatically reconstructed as such on the receiving end before reaching the receiving program.

[4c3e]

What if the response body is not text?

When I first experimented with response headers I used an array that was then packed with msgpack, because I didn't want to be bothered with parsing text 😆

If we use msgpack to encode a dict/array would an eventual C implementation of RNS be able to correctly unpack the data as well?

[markqvist]

Yes, msgpack has C implementations as well. But you don't need to use msgpack in your own code! Reticulum uses msgpack internally. You can just return a dict or array from the response callback, and Reticulum will transparently compress, pack, unpack and decompress it in the most efficient way.

[4c3e]

Oh awesome, I guess one other question I have is if you can return a python file object in the response callback and it will also just work?

For example, lets say we handle response's by using the form [header, body] which is a python array that will get packed automatically like you mentioned. Could the body var be a python file object? For example's sake, lets say the response body is a .png image, and the header is the string: 20 image/png

[markqvist]

As far as I remember, you can. Sorry, I'm travelling right now, so can't easily cgeck the source. That behaviour is the intention anyway, and I am pretty sure I did get around to implementing it. The request/response mechanism uses the Resource API internally, which does support passing file objects directly.

No matter what, you can count on this as the intended behaviour. If it's not working, I will fix it pronto :)

Edit: come to think of it, I am pretty sure it works when passing files directly as the response, but it might not work as members of an array. We should probably have that case working also though.

[4c3e]

I have been thinking on this for a bit more, and I came up with another route which might make more sense. Would love other's input.

Similar to the optional title field in LXMF, it might be convenient to just specify a "META" field in the request API. For most RNS requests and responses, it would be useful to have access to some sort of metadata about the attached data.

It would slightly bloat situations where this metadata is not needed, but applications could just ignore the field in these cases.

[markqvist]

This is actually already possible, and something that was the intention from the beginning. Just in a bit of a different way than you might be used to. Consider the Request API:

request(path, data=None, response_callback=None, failed_callback=None, progress_callback=None, timeout=None)

That data argument is not just for passing a string, or a blob of bytes, or something like that. Consider this:

...

filename = "the_file.txt"
file = open(filename, "rb")

request_data = {
  "command": "upload",
  "data": file.read(),
  "meta": {
    "filename": filename,
    "creator": "somebody",
    "expiration_date": None,
  }
}

...

You can just pass that request_data dict directly to the Request API. Reticulum will handle packing, compressing, sequencing and mapping the entire thing in the most efficient way possible, completely automatically. In the other end the reverse happens, and the receiver just gets the exact same kind of dict passed to its callback.

So you don't really need to add any meta fields. It's already there. It's infinitely expandable, and you don't have to do anything extra. Just define the structures you need and send them :)

[4c3e]

Apologies, I was having a bit of trouble understanding this bit of the request api, seeing the example you attached cleared things up. You're right, this already looks ideal!

[markqvist]

No worries, it absolutely bears elaborating on a bit, since it isn't at all obvious that Reticulum allows you to do this :)