Sockets API

[acehoss]

The ability for programmers to establish a link and open a bidirectional stream on each end of it will facilitate the development of interactive applications.

[markqvist]

As soon as this is added, someone will write a 10-line program that tunnels IP over Reticulum, and everyone will fall back into writing IP-based programs, that just happen to tunnel stuff over the underlying Reticulum transport medium, thus killing the entire purpose of Reticulum in the first place ;)

(Apart from some niche cases, like IP-over-Reticulum VPNs, and other useful utilities)

That being said, there is a Buffer class on the roadmap that will add functionality like that (can you guess why I didn't call it Stream?). I have been a bit hesitant to add it though, even though it's pretty simple...

[acehoss]

As soon as this is added, someone will write a 10-line program that tunnels IP over Reticulum, and everyone will fall back into writing IP-based programs

I don't think you're wrong--it's not what I am building, but a dozen different ideas like that came to mind (____, but over Reticulum). There's probably a significant "if you don't build it, they will" factor here, and it's not necessarily all that hard. The Packet API can do this already, just without reliable delivery. And if tunneling IP, adding reliability to the layer 2 protocol might actually worsen performance. But hopefully at least for now, by not making it easy to build something like an IP to RNS gateway, we don't have to worry about it.

It does bring up a concern with making it easier for developers to make software that uses Reticulum: it looks like Reticulum currently doesn't give users much ability to control the way that traffic flows in, out, or through their networks. If a DoS attack were to occur--whether malicious or accidental--network operators would have to rely on the capabilities of the transport medium to address it.

Access control, rate limiting, etc. might be important to have before it's terribly easy for developers to send data over the network. When Reticulum goes viral and we start to see thousands of announcements per minute on TestNet, it's too late. 😨

that just happen to tunnel stuff over the underlying Reticulum transport medium, thus killing the entire purpose of Reticulum in the first place ;)

(Apart from some niche cases, like IP-over-Reticulum VPNs, and other useful utilities)

My interest in Reticulum is much more in scratching my itch to build networks over packet radio than in concerns of security and privacy. But I understand that those concerns are central to the vision of the platform, and those concerns necessarily should have the strongest influence on where effort is spent.

[markqvist]

There's probably a significant "if you don't build it, they will" factor here, and it's not necessarily all that hard. The Packet API can do this already, just without reliable delivery. And if tunneling IP, adding reliability to the layer 2 protocol might actually worsen performance.

I think you're right there. And it's stupidly easy ;) Just for fun, I already wrote small program creating a full virtual ethernet device tunneled over Reticulum. Just showing up as a tap0 kernel dev, then actually transporting all the ethernet frames over Reticulum ;) Not gonna include that as a standard util yet though. But yeah, someone else will make it eventually if it's not added. And it is a useful tool, so it's coming at some point. First the Buffer API (which will mimick how streams work), and then related utilities for stringing that together with anything over stdio. But there's other stuff I want out first

it looks like Reticulum currently doesn't give users much ability to control the way that traffic flows in, out, or through their networks.

And intentionally so, though it is a little basic currently. But the design philosophy is that you will not need to handle a lot of this yourself. The stack should be able to mitigate almost all of this stuff by itself. There will be a few more controls in the future though.

Access control, rate limiting, etc. might be important to have before it's terribly easy for developers to send data over the network. When Reticulum goes viral and we start to see thousands of announcements per minute on TestNet, it's too late.

Reticulum can already handle this quite well. The only kind of traffic that can be flood-routed is announces, and those will by default be limited to 2% of link capacity (user configurable though). If that is exceeded, announces (and other management traffic) is queued and prioritised according to a set of rules that ensures that local network segments will still work even though a storm of annonces are coming in from a fast interface.

This is also why you can directly link fast and slow networks in Reticulum. It handles management traffic flow between them in a sensible manner.

My interest in Reticulum is much more in scratching my itch to build networks over packet radio than in concerns of security and privacy. But I understand that those concerns are central to the vision of the platform, and those concerns necessarily should have the strongest influence on where effort is spent.

I think the ability to function over low-bandwidth mediums is just as important actually. It's sort of a trifecta of features, once you have all three, some pretty great stuff is possible :)

[markqvist]

By the way, I completely went rambling, and forgot to ask, what kind of programs/services/apps do you want to build where you want bidirectional streams, instead of just using the bidirectional requests already in place? I honestly haven't found any usecase yet, where you can't do it better with requests.

I might have some weird bias or deficiency here, but I honestly feel streams are such an archaic way of doing things. People usually just end up implementing requests on top of them anyway ;)

[acehoss]

The thing that I'm tinkering with right now is shell over RNS. The rnx utility is okay for running some programs on another computer and viewing the output, but of course you can't interact with the remote programs. I'd like something a little more SSH-like.

I understand why rnx works the way it does--and it's not the transport. The tty stuff is not simple, and Windows support would require a platform-specific implementation. Heck, there are probably platform-specific tweaks needed for different *nixes.

I started with the rnx code and got it tweaked to be interactive, but then ran into some painful limitations with the subprocess.Popen approach. I pivoted to using a fork-exec model, and now I've got a nice, clean module for executing a child process with callbacks when the tty produces data. Color codes, arrow keys, etc. are working. Now it's just a matter of wiring it into the remote shell code and cleaning things up a bit.

A stream approach might look a little cleaner and require a couple fewer lines of code, but it wouldn't perform much different. And a programming benefit of the request model is it has built-in support for transmitting out-of-band data. A socket is just a pipe with the other end on another computer.

There's a tradeoff--sockets might be archaic, but they're familiar. The RNS request model is arcane and requires some amount of getting your hands dirty before it makes sense. With the one, developers can hit the ground running, but using 50 year old architecture. With the other, the architecture is more purposeful, but will require more time learning the concepts before putting the tech to work doing something.

There is perhaps another benefit to sockets, though, which is: if you implement channel capacity adaptation and reliable delivery for the socket, then any additional protocols built on top of it need not be concerned with that stuff. Whatever flaws TCP might have, this is the probably biggest reason it is the most ubiquitous transport-layer protocol in use today.

[resession]

i find myself using streams to move data a lot instead of request/response pattern. using streams for non-IP based data would be awesome if that is possible. just thinking about using something like websockets to send and receive non-IP stuff sounds cool.

[4c3e]

The new Channel and Buffer APIs are coming in the next release! The docs will be updated after the release but you can check out an example right now on the GitHub main branch:
https://github.com/markqvist/Reticulum/blob/master/Examples/Channel.py

[markqvist]

Yes, @acehoss has been doing a great job on this. You can already have a peek at the APIs here:

• https://markqvist.github.io/Reticulum/manual/reference.html#RNS.Buffer
• https://markqvist.github.io/Reticulum/manual/reference.html#RNS.Channel.Channel

Or, if you're feeling adventurous, you can download the latest version from the master branch, and start playing around with it.